FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

36,658

4,520

36,658

875

View on GitHub

Top Related Projects

text-generation-webui

40,103

A Gradio web UI for Large Language Models.

ChatGLM-6B

40,508

ChatGLM-6B: An Open Bilingual Dialogue Language Model | 开源双语对话语言模型

stanford_alpaca

29,424

Code and documentation to train Stanford's Alpaca models, and generate the data.

alpaca-lora

18,587

Instruct-tune LLaMA on consumer hardware

StableLM

15,836

StableLM: Stability AI Language Models

Quick Overview

FastChat is an open-source platform for training, serving, and evaluating large language models (LLMs). It provides a comprehensive toolkit for researchers and developers to work with various LLMs, including ChatGPT-style assistants. The project aims to facilitate the development and deployment of conversational AI systems.

Pros

Supports multiple LLMs, including Vicuna, Alpaca, LLaMA, and more
Offers a web UI for easy interaction with the models
Provides tools for model evaluation and comparison
Includes a distributed serving system for efficient deployment

Cons

Requires significant computational resources for training and serving large models
May have a steep learning curve for users new to LLMs
Documentation could be more comprehensive for some advanced features
Dependent on the availability and licensing of underlying models

Code Examples

Loading and using a model:

from fastchat.model import load_model, get_conversation_template

model, tokenizer = load_model("vicuna-7b")
conv = get_conversation_template("vicuna")
conv.append_message(conv.roles[0], "Hello, how are you?")
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

input_ids = tokenizer([prompt]).input_ids
output_ids = model.generate(
    input_ids,
    do_sample=True,
    temperature=0.7,
    max_new_tokens=100,
)
response = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Starting a model worker:

from fastchat.serve.model_worker import ModelWorker

worker = ModelWorker(
    controller_addr="http://localhost:21001",
    worker_addr="http://localhost:21002",
    model_path="vicuna-7b",
    model_names=["vicuna-7b"],
)
worker.start()

Using the OpenAI-compatible API:

import openai

openai.api_key = "EMPTY"
openai.api_base = "http://localhost:8000/v1"

completion = openai.ChatCompletion.create(
    model="vicuna-7b",
    messages=[
        {"role": "user", "content": "Hello, how are you?"}
    ]
)
print(completion.choices[0].message.content)

Getting Started

Install FastChat:

pip install fschat

Download a model (e.g., Vicuna-7B):

python3 -m fastchat.model.apply_delta --base /path/to/llama-7b --target /path/to/vicuna-7b --delta lmsys/vicuna-7b-delta-v1.1

Start the controller:

python3 -m fastchat.serve.controller

Start a model worker:

python3 -m fastchat.serve.model_worker --model-path /path/to/vicuna-7b

Start the API server:

python3 -m fastchat.serve.openai_api_server --host localhost --port 8000

Now you can interact with the model using the OpenAI-compatible API or the provided web UI.

Competitor Comparisons

text-generation-webui

40,103

A Gradio web UI for Large Language Models.

Pros of text-generation-webui

More extensive UI features, including chat, notebook, and character creation modes
Supports a wider range of models and architectures
Offers more customization options for inference settings

Cons of text-generation-webui

Less optimized for performance compared to FastChat
May be more complex to set up and configure for beginners
Lacks some of the advanced features for model fine-tuning present in FastChat

Code Comparison

text-generation-webui:

def generate_reply(
    question, history, state, regenerate=False, _continue=False, loading_message=False
):
    # Complex generation logic with multiple options and settings

FastChat:

def generate_stream(
    model, tokenizer, params, device, context_len=2048, stream_interval=2
):
    # Streamlined generation function focused on efficiency

The code snippets highlight the difference in approach: text-generation-webui offers more flexibility and options, while FastChat focuses on streamlined and efficient generation.

ChatGLM-6B

40,508

ChatGLM-6B: An Open Bilingual Dialogue Language Model | 开源双语对话语言模型

Pros of ChatGLM-6B

Smaller model size (6B parameters) makes it more accessible for deployment on consumer hardware
Bilingual support for Chinese and English, with strong performance in Chinese language tasks
Open-source model weights, allowing for fine-tuning and customization

Cons of ChatGLM-6B

Limited language support compared to FastChat's multi-lingual capabilities
Less extensive documentation and community support
Fewer pre-built tools and integrations for deployment and fine-tuning

Code Comparison

ChatGLM-6B:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()
response, history = model.chat(tokenizer, "你好", history=[])

FastChat:

from fastchat.model import load_model, get_conversation_template
model, tokenizer = load_model("lmsys/vicuna-7b-v1.3", device="cuda", num_gpus=1)
conv = get_conversation_template("vicuna")
conv.append_message(conv.roles[0], "Hello")
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()
output = model.generate(tokenizer(prompt, return_tensors="pt").to("cuda"))

stanford_alpaca

29,424

Code and documentation to train Stanford's Alpaca models, and generate the data.

Pros of Stanford Alpaca

Focuses on fine-tuning LLaMA models, providing a more specialized approach
Includes a dataset of 52K instruction-following demonstrations
Offers a simpler implementation, making it easier for researchers to understand and modify

Cons of Stanford Alpaca

Limited to LLaMA models, while FastChat supports multiple model architectures
Lacks a user-friendly interface for chatting with the model
Does not include features for model serving or API integration

Code Comparison

Stanford Alpaca (data generation):

def generate_prompt(instruction, input=None):
    if input:
        return f"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:"
    else:
        return f"Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Response:"

FastChat (model loading):

def load_model(model_path, device, num_gpus, max_gpu_memory):
    if device == "cpu":
        kwargs = {}
    elif device == "cuda":
        kwargs = {"torch_dtype": torch.float16}
        if num_gpus == "auto":
            kwargs["device_map"] = "auto"
        else:
            num_gpus = int(num_gpus)
            kwargs.update({
                "device_map": "auto",
                "max_memory": {i: f"{max_gpu_memory}GiB" for i in range(num_gpus)},
            })
    elif device == "mps":
        kwargs = {"torch_dtype": torch.float16}
    else:
        raise ValueError(f"Invalid device: {device}")

    tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=False)
    model = AutoModelForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, **kwargs)

    return model, tokenizer

alpaca-lora

18,587

Instruct-tune LLaMA on consumer hardware

Pros of Alpaca-LoRA

Focuses on fine-tuning LLMs using LoRA, which is more efficient and requires less computational resources
Provides a straightforward approach to creating custom language models based on existing pre-trained models
Includes scripts for data generation and model training, making it easier to get started

Cons of Alpaca-LoRA

Limited to LoRA-based fine-tuning, while FastChat offers a broader range of model training and deployment options
Less comprehensive documentation and community support compared to FastChat
Fewer built-in features for model serving and interaction

Code Comparison

Alpaca-LoRA (model loading):

model = LlamaForCausalLM.from_pretrained(
    base_model,
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
)

FastChat (model loading):

model, tokenizer = load_model(
    model_path,
    device=args.device,
    num_gpus=args.num_gpus,
    max_gpu_memory=args.max_gpu_memory,
    load_8bit=args.load_8bit,
    cpu_offloading=args.cpu_offloading,
)

Both repositories focus on working with large language models, but Alpaca-LoRA specializes in efficient fine-tuning using LoRA, while FastChat offers a more comprehensive suite of tools for training, serving, and interacting with various language models.

StableLM

15,836

StableLM: Stability AI Language Models

Pros of StableLM

Focuses on developing and releasing open-source language models
Provides pre-trained models of various sizes (3B, 7B) for different use cases
Offers more flexibility in model architecture and training approaches

Cons of StableLM

Less emphasis on chat-specific applications and interfaces
Fewer ready-to-use tools for deployment and fine-tuning
Smaller community and ecosystem compared to FastChat

Code Comparison

FastChat:

from fastchat.model import load_model, get_conversation_template

model, tokenizer = load_model("vicuna-7b")
conv = get_conversation_template("vicuna")
conv.append_message(conv.roles[0], "Hello!")
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

StableLM:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("stabilityai/stablelm-tuned-alpha-7b")
model = AutoModelForCausalLM.from_pretrained("stabilityai/stablelm-tuned-alpha-7b")
inputs = tokenizer("Hello, how are you?", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)

Both repositories offer powerful language model capabilities, but FastChat is more focused on chat applications and provides a more comprehensive ecosystem for deployment and fine-tuning. StableLM, on the other hand, offers more flexibility in model architecture and training approaches, making it suitable for a wider range of applications beyond chat interfaces.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

FastChat

| Demo | Discord | X |

FastChat is an open platform for training, serving, and evaluating large language model based chatbots.

FastChat powers Chatbot Arena (lmarena.ai), serving over 10 million chat requests for 70+ LLMs.
Chatbot Arena has collected over 1.5M human votes from side-by-side LLM battles to compile an online LLM Elo leaderboard.

FastChat's core features include:

The training and evaluation code for state-of-the-art models (e.g., Vicuna, MT-Bench).
A distributed multi-model serving system with web UI and OpenAI-compatible RESTful APIs.

News

[2024/03] ð¥ We released Chatbot Arena technical report.
[2023/09] We released LMSYS-Chat-1M, a large-scale real-world LLM conversation dataset. Read the report.
[2023/08] We released Vicuna v1.5 based on Llama 2 with 4K and 16K context lengths. Download weights.
[2023/07] We released Chatbot Arena Conversations, a dataset containing 33k conversations with human preferences. Download it here.

[2023/08] We released LongChat v1.5 based on Llama 2 with 32K context lengths. Download weights.
[2023/06] We introduced MT-bench, a challenging multi-turn question set for evaluating chatbots. Check out the blog post.
[2023/06] We introduced LongChat, our long-context chatbots and evaluation tools. Check out the blog post.
[2023/05] We introduced Chatbot Arena for battles among LLMs. Check out the blog post.
[2023/03] We released Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality. Check out the blog post.

Install
Model Weights
Inference with Command Line Interface
Serving with Web GUI
API
Evaluation
Fine-tuning
Citation

Install

Method 1: With pip

pip3 install "fschat[model_worker,webui]"

Method 2: From source

Clone this repository and navigate to the FastChat folder

git clone https://github.com/lm-sys/FastChat.git
cd FastChat

If you are running on Mac:

brew install rust cmake

Install Package

pip3 install --upgrade pip  # enable PEP 660 support
pip3 install -e ".[model_worker,webui]"

Model Weights

Vicuna Weights

Vicuna is based on Llama 2 and should be used under Llama's model license.

You can use the commands below to start chatting. It will automatically download the weights from Hugging Face repos. Downloaded weights are stored in a .cache folder in the user's home folder (e.g., ~/.cache/huggingface/hub/<model_name>).

See more command options and how to handle out-of-memory in the "Inference with Command Line Interface" section below.

NOTE: transformers>=4.31 is required for 16K versions.

Size	Chat Command	Hugging Face Repo
7B	`python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5`	lmsys/vicuna-7b-v1.5
7B-16k	`python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5-16k`	lmsys/vicuna-7b-v1.5-16k
13B	`python3 -m fastchat.serve.cli --model-path lmsys/vicuna-13b-v1.5`	lmsys/vicuna-13b-v1.5
13B-16k	`python3 -m fastchat.serve.cli --model-path lmsys/vicuna-13b-v1.5-16k`	lmsys/vicuna-13b-v1.5-16k
33B	`python3 -m fastchat.serve.cli --model-path lmsys/vicuna-33b-v1.3`	lmsys/vicuna-33b-v1.3

Old weights: see docs/vicuna_weights_version.md for all versions of weights and their differences.

Other Models

Besides Vicuna, we also released two additional models: LongChat and FastChat-T5. You can use the commands below to chat with them. They will automatically download the weights from Hugging Face repos.

Model	Chat Command	Hugging Face Repo
LongChat-7B	`python3 -m fastchat.serve.cli --model-path lmsys/longchat-7b-32k-v1.5`	lmsys/longchat-7b-32k
FastChat-T5-3B	`python3 -m fastchat.serve.cli --model-path lmsys/fastchat-t5-3b-v1.0`	lmsys/fastchat-t5-3b-v1.0

Inference with Command Line Interface

(Experimental Feature: You can specify --style rich to enable rich text output and better text streaming quality for some non-ASCII content. This may not work properly on certain terminals.)

Supported Models

FastChat supports a wide range of models, including LLama 2, Vicuna, Alpaca, Baize, ChatGLM, Dolly, Falcon, FastChat-T5, GPT4ALL, Guanaco, MTP, OpenAssistant, OpenChat, RedPajama, StableLM, WizardLM, xDAN-AI and more.

See a complete list of supported models and instructions to add a new model here.

Single GPU

The command below requires around 14GB of GPU memory for Vicuna-7B and 28GB of GPU memory for Vicuna-13B. See the "Not Enough Memory" section below if you do not have enough memory. --model-path can be a local folder or a Hugging Face repo name.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5

Multiple GPUs

You can use model parallelism to aggregate GPU memory from multiple GPUs on the same machine.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2

Tips: Sometimes the "auto" device mapping strategy in huggingface/transformers does not perfectly balance the memory allocation across multiple GPUs. You can use --max-gpu-memory to specify the maximum memory per GPU for storing model weights. This allows it to allocate more memory for activations, so you can use longer context lengths or larger batch sizes. For example,

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2 --max-gpu-memory 8GiB

CPU Only

This runs on the CPU only and does not require GPU. It requires around 30GB of CPU memory for Vicuna-7B and around 60GB of CPU memory for Vicuna-13B.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device cpu

Use Intel AI Accelerator AVX512_BF16/AMX to accelerate CPU inference.

CPU_ISA=amx python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device cpu

Metal Backend (Mac Computers with Apple Silicon or AMD GPUs)

Use --device mps to enable GPU acceleration on Mac computers (requires torch >= 2.0). Use --load-8bit to turn on 8-bit compression.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device mps --load-8bit

Vicuna-7B can run on a 32GB M1 Macbook with 1 - 2 words / second.

Intel XPU (Intel Data Center and Arc A-Series GPUs)

Install the Intel Extension for PyTorch. Set the OneAPI environment variables:

source /opt/intel/oneapi/setvars.sh

Use --device xpu to enable XPU/GPU acceleration.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device xpu

Vicuna-7B can run on an Intel Arc A770 16GB.

Ascend NPU

Install the Ascend PyTorch Adapter. Set the CANN environment variables:

source /usr/local/Ascend/ascend-toolkit/set_env.sh

Use --device npu to enable NPU acceleration.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device npu

Vicuna-7B/13B can run on an Ascend NPU.

Not Enough Memory

If you do not have enough memory, you can enable 8-bit compression by adding --load-8bit to commands above. This can reduce memory usage by around half with slightly degraded model quality. It is compatible with the CPU, GPU, and Metal backend.

Vicuna-13B with 8-bit compression can run on a single GPU with 16 GB of VRAM, like an Nvidia RTX 3090, RTX 4080, T4, V100 (16GB), or an AMD RX 6800 XT.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --load-8bit

In addition to that, you can add --cpu-offloading to commands above to offload weights that don't fit on your GPU onto the CPU memory. This requires 8-bit compression to be enabled and the bitsandbytes package to be installed, which is only available on linux operating systems.

More Platforms and Quantization

For AMD GPU users, please install ROCm and the ROCm version of PyTorch before you install FastChat. See also this post.
FastChat supports ExLlama V2. See docs/exllama_v2.md.
FastChat supports GPTQ 4bit inference with GPTQ-for-LLaMa. See docs/gptq.md.
FastChat supports AWQ 4bit inference with mit-han-lab/llm-awq. See docs/awq.md.
MLC LLM, backed by TVM Unity compiler, deploys Vicuna natively on phones, consumer-class GPUs and web browsers via Vulkan, Metal, CUDA and WebGPU.

Use models from modelscope

For Chinese users, you can use models from www.modelscope.cn via specify the following environment variables.

export FASTCHAT_USE_MODELSCOPE=True

Serving with Web GUI

To serve using the web UI, you need three main components: web servers that interface with users, model workers that host one or more models, and a controller to coordinate the webserver and model workers. You can learn more about the architecture here.

Here are the commands to follow in your terminal:

Launch the controller

python3 -m fastchat.serve.controller

This controller manages the distributed workers.

Launch the model worker(s)

python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5

Wait until the process finishes loading the model and you see "Uvicorn running on ...". The model worker will register itself to the controller .

To ensure that your model worker is connected to your controller properly, send a test message using the following command:

python3 -m fastchat.serve.test_message --model-name vicuna-7b-v1.5

You will see a short output.

Launch the Gradio web server

python3 -m fastchat.serve.gradio_web_server

This is the user interface that users will interact with.

By following these steps, you will be able to serve your models using the web UI. You can open your browser and chat with a model now. If the models do not show up, try to reboot the gradio web server.

(Optional): Advanced Features, Scalability, Third Party UI

You can register multiple model workers to a single controller, which can be used for serving a single model with higher throughput or serving multiple models at the same time. When doing so, please allocate different GPUs and ports for different model workers.

# worker 0
CUDA_VISIBLE_DEVICES=0 python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5 --controller http://localhost:21001 --port 31000 --worker http://localhost:31000
# worker 1
CUDA_VISIBLE_DEVICES=1 python3 -m fastchat.serve.model_worker --model-path lmsys/fastchat-t5-3b-v1.0 --controller http://localhost:21001 --port 31001 --worker http://localhost:31001

You can also launch a multi-tab gradio server, which includes the Chatbot Arena tabs.

python3 -m fastchat.serve.gradio_web_server_multi

The default model worker based on huggingface/transformers has great compatibility but can be slow. If you want high-throughput batched serving, you can try vLLM integration.
If you want to host it on your own UI or third party UI, see Third Party UI.

API

OpenAI-Compatible RESTful APIs & SDK

FastChat provides OpenAI-compatible APIs for its supported models, so you can use FastChat as a local drop-in replacement for OpenAI APIs. The FastChat server is compatible with both openai-python library and cURL commands. The REST API is capable of being executed from Google Colab free tier, as demonstrated in the FastChat_API_GoogleColab.ipynb notebook, available in our repository. See docs/openai_api.md.

Hugging Face Generation APIs

See fastchat/serve/huggingface_api.py.

LangChain Integration

See docs/langchain_integration.

Evaluation

We use MT-bench, a set of challenging multi-turn open-ended questions to evaluate models. To automate the evaluation process, we prompt strong LLMs like GPT-4 to act as judges and assess the quality of the models' responses. See instructions for running MT-bench at fastchat/llm_judge.

MT-bench is the new recommended way to benchmark your models. If you are still looking for the old 80 questions used in the vicuna blog post, please go to vicuna-blog-eval.

Fine-tuning

Data

Vicuna is created by fine-tuning a Llama base model using approximately 125K user-shared conversations gathered from ShareGPT.com with public APIs. To ensure data quality, we convert the HTML back to markdown and filter out some inappropriate or low-quality samples. Additionally, we divide lengthy conversations into smaller segments that fit the model's maximum context length. For detailed instructions to clean the ShareGPT data, check out here.

We will not release the ShareGPT dataset. If you would like to try the fine-tuning code, you can run it with some dummy conversations in dummy_conversation.json. You can follow the same format and plug in your own data.

Code and Hyperparameters

Our code is based on Stanford Alpaca with additional support for multi-turn conversations. We use similar hyperparameters as the Stanford Alpaca.

Hyperparameter	Global Batch Size	Learning rate	Epochs	Max length	Weight decay
Vicuna-13B	128	2e-5	3	2048	0

Fine-tuning Vicuna-7B with Local GPUs

Install dependency

pip3 install -e ".[train]"

You can use the following command to train Vicuna-7B with 4 x A100 (40GB). Update --model_name_or_path with the actual path to Llama weights and --data_path with the actual path to data.

torchrun --nproc_per_node=4 --master_port=20001 fastchat/train/train_mem.py \
    --model_name_or_path meta-llama/Llama-2-7b-hf \
    --data_path data/dummy_conversation.json \
    --bf16 True \
    --output_dir output_vicuna \
    --num_train_epochs 3 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --gradient_accumulation_steps 16 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 1200 \
    --save_total_limit 10 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True

Tips:

If you are using V100 which is not supported by FlashAttention, you can use the memory-efficient attention implemented in xFormers. Install xformers and replace fastchat/train/train_mem.py above with fastchat/train/train_xformers.py.
If you meet out-of-memory due to "FSDP Warning: When using FSDP, it is efficient and recommended... ", see solutions here.
If you meet out-of-memory during model saving, see solutions here.
To turn on logging to popular experiment tracking tools such as Tensorboard, MLFlow or Weights & Biases, use the report_to argument, e.g. pass --report_to wandb to turn on logging to Weights & Biases.

Other models, platforms and LoRA support

More instructions to train other models (e.g., FastChat-T5) and use LoRA are in docs/training.md.

Fine-tuning on Any Cloud with SkyPilot

SkyPilot is a framework built by UC Berkeley for easily and cost effectively running ML workloads on any cloud (AWS, GCP, Azure, Lambda, etc.). Find SkyPilot documentation here on using managed spot instances to train Vicuna and save on your cloud costs.

Citation

The code (training, serving, and evaluation) in this repository is mostly developed for or derived from the paper below. Please cite it if you find the repository helpful.

@misc{zheng2023judging,
      title={Judging LLM-as-a-judge with MT-Bench and Chatbot Arena},
      author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric. P Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},
      year={2023},
      eprint={2306.05685},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

We are also planning to add more of our research to this repository.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

Top Related Projects

Quick Overview

Pros

Cons

Code Examples

Getting Started

Competitor Comparisons

Pros of text-generation-webui

Cons of text-generation-webui

Code Comparison

Pros of ChatGLM-6B

Cons of ChatGLM-6B

Code Comparison

Pros of Stanford Alpaca

Cons of Stanford Alpaca

Code Comparison

Pros of Alpaca-LoRA

Cons of Alpaca-LoRA

Code Comparison

Pros of StableLM

Cons of StableLM

Code Comparison

Convert designs to code with AI

README

FastChat

News

Contents

Install

Method 1: With pip

Method 2: From source

Model Weights

Vicuna Weights

Other Models

Inference with Command Line Interface

Supported Models

Single GPU

Multiple GPUs

CPU Only

Metal Backend (Mac Computers with Apple Silicon or AMD GPUs)

Intel XPU (Intel Data Center and Arc A-Series GPUs)

Ascend NPU

Not Enough Memory

More Platforms and Quantization

Use models from modelscope

Serving with Web GUI

Launch the controller

Launch the model worker(s)

Launch the Gradio web server

(Optional): Advanced Features, Scalability, Third Party UI

API

OpenAI-Compatible RESTful APIs & SDK

Hugging Face Generation APIs

LangChain Integration

Evaluation

Fine-tuning

Data

Code and Hyperparameters

Fine-tuning Vicuna-7B with Local GPUs

Other models, platforms and LoRA support

Fine-tuning on Any Cloud with SkyPilot

Citation

Top Related Projects

Convert designs to code with AI