Convert Figma logo to code with AI

MinishLab logomodel2vec

Fast State-of-the-Art Static Embeddings

1,130
49
1,130
7
View on GitHub

Top Related Projects

explosion's avatar

spaCy

31,212

💫 Industrial-strength Natural Language Processing (NLP) in Python

piskvorky's avatar

gensim

15,797

Topic Modelling for Humans

facebookresearch's avatar

fastText

26,018

Library for fast text representation and classification.

huggingface's avatar

transformers

141,749

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

stanfordnlp's avatar

GloVe

6,991

Software in C and data files for the popular GloVe model for distributed word representations, a.k.a. word vectors or embeddings

Quick Overview

Model2Vec is a Python library that implements the Model2Vec algorithm, which is designed to generate vector representations of machine learning models. This approach allows for comparing and analyzing different models in a vector space, potentially enabling tasks like model similarity assessment and transfer learning.

Pros

  • Enables comparison and analysis of machine learning models in a vector space
  • Potentially useful for transfer learning and model selection tasks
  • Implements a novel approach to model representation

Cons

  • Limited documentation and examples available
  • May not be suitable for all types of machine learning models
  • Relatively new and untested in real-world applications

Code Examples

Here are a few code examples demonstrating the usage of Model2Vec:

  1. Creating a Model2Vec instance:
from model2vec import Model2Vec

# Initialize Model2Vec with default parameters
m2v = Model2Vec()
  1. Generating a vector representation for a model:
import tensorflow as tf

# Create a simple TensorFlow model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu', input_shape=(10,)),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

# Generate vector representation
model_vector = m2v.vectorize(model)
  1. Comparing two models:
# Assume we have two models: model1 and model2
vector1 = m2v.vectorize(model1)
vector2 = m2v.vectorize(model2)

# Calculate similarity between models
similarity = m2v.similarity(vector1, vector2)
print(f"Similarity between models: {similarity}")

Getting Started

To get started with Model2Vec, follow these steps:

  1. Install the library:

    pip install model2vec

  2. Import and initialize Model2Vec:

    from model2vec import Model2Vec
m2v = Model2Vec()

  3. Vectorize your machine learning models:

    model_vector = m2v.vectorize(your_model)

  4. Use the generated vectors for analysis or comparison tasks as needed.

Competitor Comparisons

explosion logo

spaCy

31,212

💫 Industrial-strength Natural Language Processing (NLP) in Python

Pros of spaCy

  • Comprehensive NLP library with extensive features and pre-trained models
  • Large and active community, extensive documentation, and regular updates
  • Optimized for production use with efficient performance

Cons of spaCy

  • Steeper learning curve due to its extensive feature set
  • Larger footprint and potentially slower for simple tasks
  • May be overkill for projects requiring only basic NLP functionality

Code Comparison

model2vec:

from model2vec import Model2Vec

model = Model2Vec(model_path="path/to/model")
vector = model.encode("Hello, world!")

spaCy:

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Hello, world!")
vector = doc.vector

Key Differences

  • model2vec focuses specifically on converting models to vectors, while spaCy is a full-featured NLP library
  • spaCy offers more comprehensive language processing capabilities, including tokenization, part-of-speech tagging, and named entity recognition
  • model2vec may be more lightweight and easier to integrate for projects only needing vector representations

Use Cases

  • Choose model2vec for projects requiring simple model-to-vector conversions
  • Opt for spaCy when building complex NLP applications or requiring a wide range of language processing features
piskvorky logo

gensim

15,797

Topic Modelling for Humans

Pros of gensim

  • More comprehensive and mature library with a wider range of NLP and topic modeling functionalities
  • Larger community support and more frequent updates
  • Better documentation and extensive tutorials

Cons of gensim

  • Heavier and more complex, which may be overkill for simpler projects
  • Steeper learning curve for beginners
  • Potentially slower for specific word embedding tasks

Code comparison

model2vec:

from model2vec import Model2Vec

model = Model2Vec(size=100, window=5, min_count=5)
model.train(sentences)

gensim:

from gensim.models import Word2Vec

model = Word2Vec(sentences, vector_size=100, window=5, min_count=5)
model.train(sentences, total_examples=len(sentences), epochs=10)

Summary

gensim is a more comprehensive and well-established library for various NLP tasks, including word embeddings. It offers a wider range of functionalities and better community support. However, it may be more complex and potentially slower for specific word embedding tasks compared to model2vec.

model2vec, on the other hand, appears to be a more focused and lightweight library specifically for word embeddings. It might be easier to use for beginners and potentially faster for simple word embedding tasks, but it lacks the extensive features and community support of gensim.

facebookresearch logo

fastText

26,018

Library for fast text representation and classification.

Pros of fastText

  • More mature and widely adopted project with extensive documentation
  • Supports multiple languages and pre-trained models
  • Offers both supervised and unsupervised learning capabilities

Cons of fastText

  • Larger codebase and potentially more complex to understand
  • May have higher memory requirements for large datasets
  • Less focused on specific model-to-vector conversion tasks

Code Comparison

model2vec:

def convert_model(model, output_file):
    vector = model_to_vector(model)
    np.save(output_file, vector)

fastText:

void FastText::loadModel(const std::string& filename) {
  std::ifstream ifs(filename, std::ifstream::binary);
  if (!ifs.is_open()) {
    throw std::invalid_argument(filename + " cannot be opened for loading!");
  }
  loadModel(ifs);
  ifs.close();
}

model2vec focuses on converting machine learning models to vector representations, while fastText provides a broader set of text classification and word embedding functionalities. model2vec is more specialized and potentially easier to use for specific model conversion tasks, whereas fastText offers a more comprehensive suite of text processing tools but may require more setup and configuration for specific use cases.

huggingface logo

transformers

141,749

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

Pros of transformers

  • Extensive library with support for numerous pre-trained models and architectures
  • Large community and frequent updates, ensuring compatibility with latest research
  • Comprehensive documentation and examples for various NLP tasks

Cons of transformers

  • Larger codebase and dependencies, potentially slower for simple tasks
  • Steeper learning curve for beginners due to its extensive features
  • May be overkill for projects requiring only basic word embeddings

Code comparison

model2vec:

from model2vec import Model2Vec

model = Model2Vec.load("path/to/model")
vector = model.get_vector("word")

transformers:

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("bert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
inputs = tokenizer("Hello, world!", return_tensors="pt")
outputs = model(**inputs)

Summary

transformers offers a comprehensive solution for various NLP tasks with extensive model support, while model2vec focuses specifically on word embeddings. transformers provides more flexibility and features but may be more complex for simple use cases, whereas model2vec offers a straightforward approach for basic word vector operations.

stanfordnlp logo

GloVe

6,991

Software in C and data files for the popular GloVe model for distributed word representations, a.k.a. word vectors or embeddings

Pros of GloVe

  • Well-established and widely used in NLP research and applications
  • Efficient training on large corpora with good performance on various tasks
  • Extensive documentation and community support

Cons of GloVe

  • Limited to word-level representations, not capturing subword information
  • May struggle with out-of-vocabulary words and rare terms
  • Requires retraining for new domains or languages

Code Comparison

GloVe:

for (a = 0; a < vocab_size; a++) {
    for (b = 0; b < vector_size; b++)
        fprintf(fout, "%lf ", W[a * vector_size + b]);
    fprintf(fout, "\n");
}

model2vec:

def save_embeddings(self, path):
    with open(path, 'w') as f:
        for word, vec in self.embeddings.items():
            f.write(f"{word} {' '.join(map(str, vec))}\n")

Both repositories provide methods for saving word embeddings, but GloVe uses C for efficiency, while model2vec uses Python for readability and ease of use. GloVe's implementation is more low-level and optimized for performance, whereas model2vec's approach is more Pythonic and easier to integrate into existing NLP pipelines.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Model2Vec logo

Fast State-of-the-Art Static Embeddings

Model2Vec is a technique to turn any sentence transformer into a really small static model, reducing model size by a factor up to 50 and making the models up to 500 times faster, with a small drop in performance. Our best model is the most performant static embedding model in the world. See our results here, or dive in to see how it works.

Quickstart

Install the lightweight base package with:

pip install model2vec

You can start using Model2Vec by loading one of our flagship models from the HuggingFace hub. These models are pre-trained and ready to use. The following code snippet shows how to load a model and make embeddings, which you can use for any task, such as text classification, retrieval, clustering, or building a RAG system:

from model2vec import StaticModel

# Load a model from the HuggingFace hub (in this case the potion-base-8M model)
model = StaticModel.from_pretrained("minishlab/potion-base-8M")

# Make embeddings
embeddings = model.encode(["It's dangerous to go alone!", "It's a secret to everybody."])

# Make sequences of token embeddings
token_embeddings = model.encode_as_sequence(["It's dangerous to go alone!", "It's a secret to everybody."])

Instead of using one of our models, you can also distill your own Model2Vec model from a Sentence Transformer model. First, install the distillation extras with:

pip install model2vec[distill]

Then, you can distill a model in ~30 seconds on a CPU with the following code snippet:

from model2vec.distill import distill

# Distill a Sentence Transformer model, in this case the BAAI/bge-base-en-v1.5 model
m2v_model = distill(model_name="BAAI/bge-base-en-v1.5", pca_dims=256)

# Save the model
m2v_model.save_pretrained("m2v_model")

After distillation, you can also fine-tune your own classification models on top of the distilled model, or on a pre-trained model. First, make sure you install the training extras with:

pip install model2vec[training]

Then, you can fine-tune a model as follows:

import numpy as np
from datasets import load_dataset
from model2vec.train import StaticModelForClassification

# Initialize a classifier from a pre-trained model
classifier = StaticModelForClassification.from_pretrained(model_name="minishlab/potion-base-32M")

# Load a dataset. Note: both single and multi-label classification datasets are supported
ds = load_dataset("setfit/subj")

# Train the classifier on text (X) and labels (y)
classifier.fit(ds["train"]["text"], ds["train"]["label"])

# Evaluate the classifier
classification_report = classifier.evaluate(ds["test"]["text"], ds["test"]["label"])

For advanced usage, please refer to our usage documentation.

Updates & Announcements

Main Features

  • State-of-the-Art Performance: Model2Vec models outperform any other static embeddings (such as GLoVe and BPEmb) by a large margin, as can be seen in our results.
  • Small: Model2Vec reduces the size of a Sentence Transformer model by a factor of up to 50. Our best model is just ~30 MB on disk, and our smallest model just ~8 MB (making it the smallest model on MTEB!).
  • Lightweight Dependencies: the base package's only major dependency is numpy.
  • Lightning-fast Inference: up to 500 times faster on CPU than the original model.
  • Fast, Dataset-free Distillation: distill your own model in 30 seconds on a CPU, without a dataset.
  • Fine-tuning: fine-tune your own classification models on top of Model2Vec models.
  • Integrated in many popular libraries: Model2Vec is integrated direclty into popular libraries such as Sentence Transformers and LangChain. For more information, see our integrations documentation.
  • Tightly integrated with HuggingFace hub: easily share and load models from the HuggingFace hub, using the familiar from_pretrained and push_to_hub. Our own models can be found here.

What is Model2Vec?

Model2vec creates a small, fast, and powerful model that outperforms other static embedding models by a large margin on all tasks we could find, while being much faster to create than traditional static embedding models such as GloVe. Like BPEmb, it can create subword embeddings, but with much better performance. Distillation doesn't need any data, just a vocabulary and a model.

The core idea is to forward pass a vocabulary through a sentence transformer model, creating static embeddings for the indiviudal tokens. After this, there are a number of post-processing steps we do that results in our best models. For a more extensive deepdive, please refer to the following resources:

  • Our initial Model2Vec blog post. Note that, while this post gives a good overview of the core idea, we've made a number of substantial improvements since then.
  • Our Tokenlearn blog post. This post describes the Tokenlearn method we used to train our potion models.
  • Our official documentation. This document provides a high-level overview of how Model2Vec works.

Documentation

Our official documentation can be found here. This includes:

Model List

We provide a number of models that can be used out of the box. These models are available on the HuggingFace hub and can be loaded using the from_pretrained method. The models are listed below.

ModelLanguageSentence TransformerParamsTask
potion-base-32MEnglishbge-base-en-v1.532.3MGeneral
potion-base-8MEnglishbge-base-en-v1.57.5MGeneral
potion-base-4MEnglishbge-base-en-v1.53.7MGeneral
potion-base-2MEnglishbge-base-en-v1.51.8MGeneral
potion-retrieval-32MEnglishbge-base-en-v1.532.3MRetrieval
M2V_multilingual_outputMultilingualLaBSE471MGeneral

Results

We have performed extensive experiments to evaluate the performance of Model2Vec models. The results are documented in the results folder. The results are presented in the following sections:

License

MIT

Citing

If you use Model2Vec in your research, please cite the following:

@software{minishlab2024model2vec,
  authors = {Stephan Tulkens and Thomas van Dongen},
  title = {Model2Vec: Fast State-of-the-Art Static Embeddings},
  year = {2024},
  url = {https://github.com/MinishLab/model2vec}
}