Convert Figma logo to code with AI

allenai logoallennlp

An open-source NLP research library, built on PyTorch.

11,750
2,244
11,750
91
View on GitHub

Top Related Projects

huggingface's avatar

transformers

131,798

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

pytorch's avatar

pytorch

82,049

Tensors and Dynamic neural networks in Python with strong GPU acceleration

tensorflow's avatar

tensorflow

185,446

An Open Source Machine Learning Framework for Everyone

explosion's avatar

spaCy

29,635

💫 Industrial-strength Natural Language Processing (NLP) in Python

facebookresearch's avatar

fairseq

30,331

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

stanfordnlp's avatar

stanza

7,206

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

Quick Overview

AllenNLP is an open-source NLP research library built on PyTorch. It provides high-level abstractions and APIs for developing and evaluating deep learning models for various NLP tasks. AllenNLP is designed to be flexible, modular, and easy to use for both researchers and practitioners in the field of natural language processing.

Pros

  • Extensive collection of pre-built models and datasets for common NLP tasks
  • Highly modular and extensible architecture, allowing easy customization
  • Comprehensive documentation and tutorials for beginners and advanced users
  • Active community and regular updates from the Allen Institute for AI

Cons

  • Steeper learning curve compared to some other NLP libraries
  • Can be resource-intensive for large-scale projects
  • Some advanced features may require in-depth knowledge of PyTorch
  • Limited support for non-English languages in some pre-built models

Code Examples

  1. Loading a pre-trained model and making predictions:
from allennlp.predictors.predictor import Predictor

predictor = Predictor.from_path("https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz")
result = predictor.predict(sentence="The cat sat on the mat.")
print(result)
  1. Training a custom text classification model:
from allennlp.data import DatasetReader, Instance
from allennlp.data.fields import TextField, LabelField
from allennlp.data.token_indexers import SingleIdTokenIndexer
from allennlp.models import Model
from allennlp.modules.text_field_embedders import BasicTextFieldEmbedder
from allennlp.modules.token_embedders import Embedding
from allennlp.nn.util import get_text_field_mask
from allennlp.training.trainer import Trainer
from allennlp.data.vocabulary import Vocabulary
from allennlp.predictors import TextClassifierPredictor
from allennlp.data.tokenizers import WhitespaceTokenizer

# Define your custom dataset reader, model, and training logic here
# ...

trainer = Trainer(model=model, optimizer=optimizer, data_loader=data_loader)
trainer.train()
  1. Using AllenNLP for named entity recognition:
from allennlp.predictors import Predictor

predictor = Predictor.from_path("https://storage.googleapis.com/allennlp-public-models/ner-model-2020.02.10.tar.gz")
result = predictor.predict(sentence="John Smith works at Google in New York.")
print(result)

Getting Started

To get started with AllenNLP, follow these steps:

  1. Install AllenNLP:
pip install allennlp
  1. Import the necessary modules:
from allennlp.predictors import Predictor
  1. Load a pre-trained model and make predictions:
predictor = Predictor.from_path("https://storage.googleapis.com/allennlp-public-models/fine-grained-ner.2021-02-11.tar.gz")
result = predictor.predict(sentence="AllenNLP is a great library for NLP tasks.")
print(result)

For more detailed instructions and advanced usage, refer to the official AllenNLP documentation and tutorials.

Competitor Comparisons

huggingface logo

transformers

131,798

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

Pros of Transformers

  • Broader model support, including many state-of-the-art architectures
  • Larger community and more frequent updates
  • Easier integration with popular deep learning frameworks (PyTorch, TensorFlow)

Cons of Transformers

  • Steeper learning curve for beginners
  • Less focus on traditional NLP tasks and linguistic annotations
  • Can be overwhelming due to the large number of models and options

Code Comparison

AllenNLP:

from allennlp.predictors.predictor import Predictor

predictor = Predictor.from_path("https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz")
result = predictor.predict(sentence="Did Uriah honestly think he could beat the game in under three hours?")

Transformers:

from transformers import pipeline

nlp = pipeline("sentiment-analysis")
result = nlp("I love this movie!")

Both libraries offer high-level APIs for common NLP tasks, but Transformers provides a more streamlined approach with its pipeline functionality. AllenNLP tends to be more verbose and requires more setup for specific tasks. Transformers focuses on transformer-based models, while AllenNLP supports a wider range of traditional NLP models and techniques.

pytorch logo

pytorch

82,049

Tensors and Dynamic neural networks in Python with strong GPU acceleration

Pros of PyTorch

  • More general-purpose deep learning framework, supporting a wider range of applications beyond NLP
  • Larger community and ecosystem, with more third-party libraries and resources
  • More flexible and dynamic computational graph, allowing for easier debugging and experimentation

Cons of PyTorch

  • Steeper learning curve for beginners, especially those focused solely on NLP tasks
  • Less specialized for NLP-specific tasks and models compared to AllenNLP
  • Requires more boilerplate code for common NLP operations

Code Comparison

PyTorch (basic neural network):

import torch.nn as nn

class SimpleNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(10, 1)

    def forward(self, x):
        return self.linear(x)

AllenNLP (text classification):

from allennlp.models import Model

@Model.register("text_classifier")
class TextClassifier(Model):
    def __init__(self, vocab, embedder, encoder):
        super().__init__(vocab)
        self.embedder = embedder
        self.encoder = encoder
        self.classifier = nn.Linear(encoder.get_output_dim(), vocab.get_vocab_size('labels'))
tensorflow logo

tensorflow

185,446

An Open Source Machine Learning Framework for Everyone

Pros of TensorFlow

  • Broader ecosystem and industry adoption
  • Supports multiple programming languages (Python, JavaScript, C++)
  • More extensive documentation and community resources

Cons of TensorFlow

  • Steeper learning curve for beginners
  • Can be more complex to set up and configure
  • Less focused on natural language processing tasks

Code Comparison

AllenNLP example:

from allennlp.predictors.predictor import Predictor

predictor = Predictor.from_path("https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz")
result = predictor.predict(sentence="Did Uriah honestly think he could beat the game in under three hours?")

TensorFlow example:

import tensorflow as tf

mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(10, activation='softmax')
])

AllenNLP is more focused on NLP tasks, providing pre-built models and easy-to-use interfaces for common NLP operations. TensorFlow offers a more general-purpose machine learning framework with greater flexibility but requires more setup for specific tasks.

explosion logo

spaCy

29,635

💫 Industrial-strength Natural Language Processing (NLP) in Python

Pros of spaCy

  • Faster processing speed and lower memory usage
  • More comprehensive built-in language models and support for multiple languages
  • Easier integration with production systems due to its focus on efficiency

Cons of spaCy

  • Less flexibility for custom model architectures
  • Smaller community and fewer third-party extensions
  • More limited support for deep learning tasks

Code Comparison

spaCy:

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")

for ent in doc.ents:
    print(ent.text, ent.label_)

AllenNLP:

from allennlp.predictors.predictor import Predictor

predictor = Predictor.from_path("https://storage.googleapis.com/allennlp-public-models/ner-model-2020.02.10.tar.gz")
result = predictor.predict(sentence="Apple is looking at buying U.K. startup for $1 billion")

for entity in result["tags"]:
    print(entity)
facebookresearch logo

fairseq

30,331

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

Pros of fairseq

  • More focused on sequence-to-sequence tasks and machine translation
  • Extensive support for various architectures like Transformer, CNN, RNN
  • Better optimized for performance and distributed training

Cons of fairseq

  • Steeper learning curve for beginners
  • Less comprehensive documentation compared to AllenNLP
  • More specialized, potentially less versatile for general NLP tasks

Code Comparison

AllenNLP example:

from allennlp.predictors import Predictor

predictor = Predictor.from_path("https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz")
result = predictor.predict(sentence="Did Uriah honestly think he could beat the game in under three hours?")

fairseq example:

from fairseq.models.transformer import TransformerModel

en2de = TransformerModel.from_pretrained('/path/to/model', checkpoint_file='model.pt')
en2de.translate('Hello world!')

Both libraries offer high-level APIs for common NLP tasks, but fairseq is more focused on sequence-to-sequence models, while AllenNLP provides a broader range of NLP tools and models. fairseq's code tends to be more low-level and performance-oriented, while AllenNLP emphasizes ease of use and extensibility.

stanfordnlp logo

stanza

7,206

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

Pros of Stanza

  • Supports a wider range of languages (over 60) out-of-the-box
  • Provides more comprehensive linguistic annotations, including dependency parsing and named entity recognition
  • Offers a simpler API for basic NLP tasks, making it more accessible for beginners

Cons of Stanza

  • Less flexible for custom model development compared to AllenNLP
  • Fewer pre-built models and tasks available
  • Limited support for deep learning frameworks (primarily uses PyTorch)

Code Comparison

Stanza:

import stanza
nlp = stanza.Pipeline('en')
doc = nlp("Hello world!")
print([(word.text, word.lemma, word.pos) for sent in doc.sentences for word in sent.words])

AllenNLP:

from allennlp.predictors.predictor import Predictor
predictor = Predictor.from_path("https://storage.googleapis.com/allennlp-public-models/structured-prediction-biaffine-parser-ontonotes-2020.02.10.tar.gz")
result = predictor.predict(sentence="Hello world!")
print([(token.text, token.lemma_, token.pos_) for token in result['tokens']])

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README


An Apache 2.0 NLP research library, built on PyTorch, for developing state-of-the-art deep learning models on a wide variety of linguistic tasks.

CI PyPI License Codecov Optuna

⚠️ NOTICE: The AllenNLP library is now in maintenance mode. That means we are no longer adding new features or upgrading dependencies. We will still respond to questions and address bugs as they arise up until December 16th, 2022. If you have any concerns or are interested in maintaining AllenNLP going forward, please open an issue on this repository.

AllenNLP has been a big success, but as the field is advancing quickly it's time to focus on new initiatives. We're working hard to make AI2 Tango the best way to organize research codebases. If you are an active user of AllenNLP, here are some suggested alternatives:

  • If you like the trainer, the configuration language, or are simply looking for a better way to manage your experiments, check out AI2 Tango.
  • If you like AllenNLP's modules and nn packages, check out delmaksym/allennlp-light. It's even compatible with AI2 Tango!
  • If you like the framework aspect of AllenNLP, check out flair. It has multiple state-of-art NLP models and allows you to easily use pretrained embeddings such as those from transformers.
  • If you like the AllenNLP metrics package, check out torchmetrics. It has the same API as AllenNLP, so it should be a quick learning curve to make the switch.
  • If you want to vectorize text, try the transformers library.
  • If you want to maintain the AllenNLP Fairness or Interpret components, please get in touch. There is no alternative to it, so we are looking for a dedicated maintainer.
  • If you are concerned about other AllenNLP functionality, please create an issue. Maybe we can find another way to continue supporting your use case.

Quick Links

In this README

Getting Started Using the Library

If you're interested in using AllenNLP for model development, we recommend you check out the AllenNLP Guide for a thorough introduction to the library, followed by our more advanced guides on GitHub Discussions.

When you're ready to start your project, we've created a couple of template repositories that you can use as a starting place:

  • If you want to use allennlp train and config files to specify experiments, use this template. We recommend this approach.
  • If you'd prefer to use python code to configure your experiments and run your training loop, use this template. There are a few things that are currently a little harder in this setup (loading a saved model, and using distributed training), but otherwise it's functionality equivalent to the config files setup.

In addition, there are external tutorials:

And others on the AI2 AllenNLP blog.

Plugins

AllenNLP supports loading "plugins" dynamically. A plugin is just a Python package that provides custom registered classes or additional allennlp subcommands.

There is ecosystem of open source plugins, some of which are maintained by the AllenNLP team here at AI2, and some of which are maintained by the broader community.

Plugin Maintainer CLI Description
allennlp-models AI2 No A collection of state-of-the-art models
allennlp-semparse AI2 No A framework for building semantic parsers
allennlp-server AI2 Yes A simple demo server for serving models
allennlp-optuna Makoto Hiramatsu Yes Optuna integration for hyperparameter optimization

AllenNLP will automatically find any official AI2-maintained plugins that you have installed, but for AllenNLP to find personal or third-party plugins you've installed, you also have to create either a local plugins file named .allennlp_plugins in the directory where you run the allennlp command, or a global plugins file at ~/.allennlp/plugins. The file should list the plugin modules that you want to be loaded, one per line.

To test that your plugins can be found and imported by AllenNLP, you can run the allennlp test-install command. Each discovered plugin will be logged to the terminal.

For more information about plugins, see the plugins API docs. And for information on how to create a custom subcommand to distribute as a plugin, see the subcommand API docs.

Package Overview

allennlp An open-source NLP research library, built on PyTorch
allennlp.commands Functionality for the CLI
allennlp.common Utility modules that are used across the library
allennlp.data A data processing module for loading datasets and encoding strings as integers for representation in matrices
allennlp.fairness A module for bias mitigation and fairness algorithms and metrics
allennlp.modules A collection of PyTorch modules for use with text
allennlp.nn Tensor utility functions, such as initializers and activation functions
allennlp.training Functionality for training models

Installation

AllenNLP requires Python 3.6.1 or later and PyTorch.

We support AllenNLP on Mac and Linux environments. We presently do not support Windows but are open to contributions.

Installing via conda-forge

The simplest way to install AllenNLP is using conda (you can choose a different python version):

conda install -c conda-forge python=3.8 allennlp

To install optional packages, such as checklist, use

conda install -c conda-forge allennlp-checklist

or simply install allennlp-all directly. The plugins mentioned above are similarly installable, e.g.

conda install -c conda-forge allennlp-models allennlp-semparse allennlp-server allennlp-optuna

Installing via pip

It's recommended that you install the PyTorch ecosystem before installing AllenNLP by following the instructions on pytorch.org.

After that, just run pip install allennlp.

⚠️ If you're using Python 3.7 or greater, you should ensure that you don't have the PyPI version of dataclasses installed after running the above command, as this could cause issues on certain platforms. You can quickly check this by running pip freeze | grep dataclasses. If you see something like dataclasses=0.6 in the output, then just run pip uninstall -y dataclasses.

If you need pointers on setting up an appropriate Python environment or would like to install AllenNLP using a different method, see below.

Setting up a virtual environment

Conda can be used set up a virtual environment with the version of Python required for AllenNLP. If you already have a Python 3 environment you want to use, you can skip to the 'installing via pip' section.

  1. Download and install Conda.

  2. Create a Conda environment with Python 3.8 (3.7 or 3.9 would work as well):

    conda create -n allennlp_env python=3.8

  3. Activate the Conda environment. You will need to activate the Conda environment in each terminal in which you want to use AllenNLP:

    conda activate allennlp_env

Installing the library and dependencies

Installing the library and dependencies is simple using pip.

pip install allennlp

To install the optional dependencies, such as checklist, run

pip install allennlp[checklist]

Or you can just install all optional dependencies with pip install allennlp[all].

Looking for bleeding edge features? You can install nightly releases directly from pypi

AllenNLP installs a script when you install the python package, so you can run allennlp commands just by typing allennlp into a terminal. For example, you can now test your installation with allennlp test-install.

You may also want to install allennlp-models, which contains the NLP constructs to train and run our officially supported models, many of which are hosted at https://demo.allennlp.org.

pip install allennlp-models

Installing using Docker

Docker provides a virtual machine with everything set up to run AllenNLP-- whether you will leverage a GPU or just run on a CPU. Docker provides more isolation and consistency, and also makes it easy to distribute your environment to a compute cluster.

AllenNLP provides official Docker images with the library and all of its dependencies installed.

Once you have installed Docker, you should also install the NVIDIA Container Toolkit if you have GPUs available.

Then run the following command to get an environment that will run on GPU:

mkdir -p $HOME/.allennlp/
docker run --rm --gpus all -v $HOME/.allennlp:/root/.allennlp allennlp/allennlp:latest

You can test the Docker environment with

docker run --rm --gpus all -v $HOME/.allennlp:/root/.allennlp allennlp/allennlp:latest test-install

If you don't have GPUs available, just omit the --gpus all flag.

Building your own Docker image

For various reasons you may need to create your own AllenNLP Docker image, such as if you need a different version of PyTorch. To do so, just run make docker-image from the root of your local clone of AllenNLP.

By default this builds an image with the tag allennlp/allennlp, but you can change this to anything you want by setting the DOCKER_IMAGE_NAME flag when you call make. For example, make docker-image DOCKER_IMAGE_NAME=my-allennlp.

If you want to use a different version of Python or PyTorch, set the flags DOCKER_PYTHON_VERSION and DOCKER_TORCH_VERSION to something like 3.9 and 1.9.0-cuda10.2, respectively. These flags together determine the base image that is used. You can see the list of valid combinations in this GitHub Container Registry: github.com/allenai/docker-images/pkgs/container/pytorch.

After building the image you should be able to see it listed by running docker images allennlp.

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
allennlp/allennlp   latest              b66aee6cb593        5 minutes ago       2.38GB

Installing from source

You can also install AllenNLP by cloning our git repository:

git clone https://github.com/allenai/allennlp.git

Create a Python 3.7 or 3.8 virtual environment, and install AllenNLP in editable mode by running:

pip install -U pip setuptools wheel
pip install --editable .[dev,all]

This will make allennlp available on your system but it will use the sources from the local clone you made of the source repository.

You can test your installation with allennlp test-install. See https://github.com/allenai/allennlp-models for instructions on installing allennlp-models from source.

Running AllenNLP

Once you've installed AllenNLP, you can run the command-line interface with the allennlp command (whether you installed from pip or from source). allennlp has various subcommands such as train, evaluate, and predict. To see the full usage information, run allennlp --help.

You can test your installation by running allennlp test-install.

Issues

Everyone is welcome to file issues with either feature requests, bug reports, or general questions. As a small team with our own internal goals, we may ask for contributions if a prompt fix doesn't fit into our roadmap. To keep things tidy we will often close issues we think are answered, but don't hesitate to follow up if further discussion is needed.

Contributions

The AllenNLP team at AI2 (@allenai) welcomes contributions from the community. If you're a first time contributor, we recommend you start by reading our CONTRIBUTING.md guide. Then have a look at our issues with the tag Good First Issue.

If you would like to contribute a larger feature, we recommend first creating an issue with a proposed design for discussion. This will prevent you from spending significant time on an implementation which has a technical limitation someone could have pointed out early on. Small contributions can be made directly in a pull request.

Pull requests (PRs) must have one approving review and no requested changes before they are merged. As AllenNLP is primarily driven by AI2 we reserve the right to reject or revert contributions that we don't think are good additions.

Citing

If you use AllenNLP in your research, please cite AllenNLP: A Deep Semantic Natural Language Processing Platform.

@inproceedings{Gardner2017AllenNLP,
  title={AllenNLP: A Deep Semantic Natural Language Processing Platform},
  author={Matt Gardner and Joel Grus and Mark Neumann and Oyvind Tafjord
    and Pradeep Dasigi and Nelson F. Liu and Matthew Peters and
    Michael Schmitz and Luke S. Zettlemoyer},
  year={2017},
  Eprint = {arXiv:1803.07640},
}

Team

AllenNLP is an open-source project backed by the Allen Institute for Artificial Intelligence (AI2). AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering. To learn more about who specifically contributed to this codebase, see our contributors page.