Convert Figma logo to code with AI

ashnkumar logosketch-code

Keras model to generate HTML code from hand-drawn website mockups. Implements an image captioning architecture to drawn source images.

5,099
690
5,099
29

Top Related Projects

11,940

pix2code: Generating Code from a Graphical User Interface Screenshot

A neural network that transforms a design mock-up into a static website.

7,528

A tool for defining design systems and using them to generate cross-platform UI code, Sketch files, and other artifacts.

Efficient Image Captioning code in Torch, runs on GPU

10,784

PyTorch package for the discrete VAE used for DALL·E.

Quick Overview

Sketch-code is a machine learning project that converts hand-drawn website wireframes into HTML/CSS code. It uses a deep learning model to analyze sketches of web designs and generate corresponding front-end code, bridging the gap between design and development.

Pros

  • Accelerates the web development process by automating HTML/CSS generation
  • Enables designers to quickly prototype and visualize their ideas in code
  • Reduces the communication gap between designers and developers
  • Provides a novel approach to UI/UX design using machine learning

Cons

  • May produce imperfect or inconsistent code that requires manual refinement
  • Limited to basic HTML/CSS structures; complex layouts or interactions may not be accurately captured
  • Requires high-quality, consistent sketches for optimal results
  • Dependent on the training data, which may limit its adaptability to diverse design styles

Code Examples

As this is not a code library but a machine learning project, there are no specific code examples to showcase. The project focuses on the model architecture and training process rather than providing a programmable API.

Getting Started

Since sketch-code is a machine learning project and not a code library, there isn't a traditional "getting started" section with code examples. However, to use or contribute to the project, you can follow these steps:

  1. Clone the repository:

    git clone https://github.com/ashnkumar/sketch-code.git
    
  2. Install the required dependencies:

    pip install -r requirements.txt
    
  3. Follow the instructions in the repository's README for setting up the environment, preparing the dataset, and training the model.

  4. Use the trained model to generate HTML/CSS code from your hand-drawn wireframes according to the project documentation.

Competitor Comparisons

11,940

pix2code: Generating Code from a Graphical User Interface Screenshot

Pros of pix2code

  • More comprehensive documentation and detailed README
  • Supports multiple output formats (HTML/CSS, Android XML, iOS Storyboard)
  • Includes pre-trained models for faster implementation

Cons of pix2code

  • Less recent updates and maintenance
  • More complex setup and dependencies
  • Limited to specific UI element types and layouts

Code Comparison

sketch-code:

def build_model(input_shape):
    image_model = keras.applications.VGG16(weights='imagenet', include_top=False, input_shape=input_shape)
    x = image_model.output
    x = GlobalAveragePooling2D()(x)
    x = Dense(1024, activation='relu')(x)
    x = Dense(1024, activation='relu')(x)
    x = Dense(VOCAB_SIZE, activation='softmax')(x)
    return Model(inputs=image_model.input, outputs=x)

pix2code:

def build_model(input_shape, output_size, context_length):
    image_model = keras.applications.InceptionV3(weights=None, include_top=False, pooling='avg', input_shape=input_shape)
    image_features = image_model.output
    context = Input(shape=(context_length, output_size))
    features = concatenate([image_features, Flatten()(context)])
    dense1 = Dense(1024, activation='relu')(features)
    dense2 = Dense(1024, activation='relu')(dense1)
    output = Dense(output_size, activation='softmax')(dense2)
    return Model(inputs=[image_model.input, context], outputs=output)

Both projects use deep learning models to generate code from UI designs, but pix2code employs a more complex architecture with context input and supports multiple output formats. sketch-code focuses on simplicity and ease of use, making it more accessible for beginners or quick prototyping.

A neural network that transforms a design mock-up into a static website.

Pros of Screenshot-to-code

  • Supports multiple output formats (HTML/CSS, Android XML, iOS Storyboard)
  • Includes a pre-trained model for immediate use
  • Offers a web-based demo for easy testing

Cons of Screenshot-to-code

  • Less focus on generating clean, production-ready code
  • May require more post-processing of generated code
  • Limited customization options for the generated output

Code Comparison

Screenshot-to-code:

def generate_html(screenshot):
    model = load_model('model.h5')
    prediction = model.predict(screenshot)
    return convert_prediction_to_html(prediction)

Sketch-code:

def generate_code(sketch):
    encoder = load_encoder('encoder.h5')
    decoder = load_decoder('decoder.h5')
    features = encoder.predict(sketch)
    return decoder.predict(features)

Both projects aim to convert visual designs into code, but they differ in their approach and output formats. Screenshot-to-code offers more versatility in terms of output formats, while Sketch-code focuses on generating cleaner, more production-ready HTML/CSS code. The code comparison shows that Screenshot-to-code uses a single model for prediction, while Sketch-code employs an encoder-decoder architecture. Ultimately, the choice between these projects depends on the specific use case and desired output quality.

7,528

A tool for defining design systems and using them to generate cross-platform UI code, Sketch files, and other artifacts.

Pros of Lona

  • More comprehensive design system toolset, including a design editor and code generation
  • Supports multiple platforms (iOS, Android, React)
  • Active development and community support

Cons of Lona

  • Steeper learning curve due to its more complex architecture
  • Limited to specific design-to-code workflows
  • May require more setup and configuration

Code Comparison

Lona (JSON configuration):

{
  "type": "View",
  "parameters": {
    "backgroundColor": "blue100"
  },
  "children": [
    {
      "type": "Text",
      "parameters": {
        "text": "Hello, World!"
      }
    }
  ]
}

Sketch-Code (HTML output):

<div class="container">
  <h1>Hello, World!</h1>
</div>

While Sketch-Code focuses on converting design mockups to HTML/CSS, Lona provides a more comprehensive design system toolset with support for multiple platforms. Sketch-Code may be simpler to use for basic web design conversions, but Lona offers more flexibility and power for complex design systems across different platforms.

Efficient Image Captioning code in Torch, runs on GPU

Pros of neuraltalk2

  • More established project with a larger community and more citations
  • Focuses on image captioning, a well-defined and widely applicable task
  • Provides pre-trained models for immediate use

Cons of neuraltalk2

  • Less actively maintained, with fewer recent updates
  • More complex setup and dependencies compared to sketch-code
  • Limited to image captioning, while sketch-code offers broader design-to-code functionality

Code Comparison

neuraltalk2:

from neuraltalk2.caption_generator import CaptionGenerator

caption_generator = CaptionGenerator(model_path, vocab_path)
caption = caption_generator.generate(image_path)

sketch-code:

from sketch_code.model import SketchCodeModel

model = SketchCodeModel()
html_output = model.convert_sketch_to_code(sketch_image_path)

Both projects use Python and deep learning techniques, but neuraltalk2 focuses on generating image captions, while sketch-code converts design sketches into HTML/CSS code. neuraltalk2 has a more specialized use case, while sketch-code aims to streamline the design-to-development process.

10,784

PyTorch package for the discrete VAE used for DALL·E.

Pros of DALL-E

  • More advanced and versatile image generation capabilities
  • Backed by OpenAI's extensive research and resources
  • Capable of generating a wide range of image styles and content

Cons of DALL-E

  • Closed-source and not freely available for public use
  • Requires significant computational resources to run
  • Less focused on specific UI/UX design tasks

Code Comparison

While a direct code comparison is not relevant due to the different nature and availability of these projects, we can highlight the key differences in their implementation:

DALL-E (conceptual):

import dalle

prompt = "A futuristic user interface with holographic elements"
image = dalle.generate_image(prompt)
image.save("generated_ui.png")

sketch-code:

from sketch_code import SketchCodeModel

model = SketchCodeModel()
sketch_image = load_image("ui_sketch.png")
html_code = model.generate_html(sketch_image)

Summary

DALL-E is a more advanced and versatile image generation model, while sketch-code focuses specifically on converting UI sketches to HTML/CSS code. DALL-E offers broader capabilities but is not openly available, whereas sketch-code is open-source and tailored for a specific use case in UI/UX design workflows.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

SketchCode

Generating HTML Code from a hand-drawn wireframe

Preview

SketchCode is a deep learning model that takes hand-drawn web mockups and converts them into working HTML code. It uses an image captioning architecture to generate its HTML markup from hand-drawn website wireframes.

For more information, check out this post: Automating front-end development with deep learning

This project builds on the synthetically generated dataset and model architecture from pix2code by Tony Beltramelli and the Design Mockups project from Emil Wallner.

Note: This project is meant as a proof-of-concept; the model isn't (yet) built to generalize to the variability of sketches seen in actual wireframes, and thus its performance relies on wireframes resembling the core dataset.

Setup

Prerequisites

  • Python 3 (not compatible with python 2)
  • pip

Install dependencies

pip install -r requirements.txt

Example Usage

Download the data and pretrained weights:

# Getting the data, 1,700 images, 342mb
git clone https://github.com/ashnkumar/sketch-code.git
cd sketch-code
cd scripts

# Get the data and pretrained weights
sh get_data.sh
sh get_pretrained_model.sh

Converting an example drawn image into HTML code, using pretrained weights:

cd src

python convert_single_image.py --png_path ../examples/drawn_example1.png \
      --output_folder ./generated_html \
      --model_json_file ../bin/model_json.json \
      --model_weights_file ../bin/weights.h5

General Usage

Converting a single image into HTML code, using weights:

cd src

python convert_single_image.py --png_path {path/to/img.png} \
      --output_folder {folder/to/output/html} \
      --model_json_file {path/to/model/json_file.json} \
      --model_weights_file {path/to/model/weights.h5}

Converting a batch of images in a folder to HTML:

cd src

python convert_batch_of_images.py --pngs_path {path/to/folder/with/pngs} \
      --output_folder {folder/to/output/html} \
      --model_json_file {path/to/model/json_file.json} \
      --model_weights_file {path/to/model/weights.h5}

Train the model:

cd src

# training from scratch
# <augment_training_data> adds Keras ImageDataGenerator augmentation for training images
python train.py --data_input_path {path/to/folder/with/pngs/guis} \
      --validation_split 0.2 \
      --epochs 10 \
      --model_output_path {path/to/output/model}
      --augment_training_data 1

# training starting with pretrained model
python train.py --data_input_path {path/to/folder/with/pngs/guis} \
      --validation_split 0.2 \
      --epochs 10 \
      --model_output_path {path/to/output/model} \
      --model_json_file ../bin/model_json.json \
      --model_weights_file ../bin/pretrained_weights.h5 \
      --augment_training_data 1

Evalute the generated prediction using the BLEU score

cd src

# evaluate single GUI prediction
python evaluate_single_gui.py --original_gui_filepath  {path/to/original/gui/file} \
      --predicted_gui_filepath {path/to/predicted/gui/file}

# training starting with pretrained model
python evaluate_batch_guis.py --original_guis_filepath  {path/to/folder/with/original/guis} \
      --predicted_guis_filepath {path/to/folder/with/predicted/guis}

License

The MIT License (MIT)

Copyright (c) 2018 Ashwin Kumar<ash.nkumar@gmail.com@gmail.com>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.