Convert Figma logo to code with AI

stanfordnlp logostanza

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

7,206
885
7,206
99
View on GitHub

Top Related Projects

huggingface's avatar

transformers

131,798

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

explosion's avatar

spaCy

29,635

💫 Industrial-strength Natural Language Processing (NLP) in Python

nltk's avatar

nltk

13,520

NLTK Source

facebookresearch's avatar

fairseq

30,331

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

piskvorky's avatar

gensim

15,616

Topic Modelling for Humans

allenai's avatar

allennlp

11,750

An open-source NLP research library, built on PyTorch.

Quick Overview

Stanza is a Python natural language processing toolkit developed by Stanford NLP Group. It provides tools for various NLP tasks such as tokenization, part-of-speech tagging, named entity recognition, and dependency parsing across multiple languages. Stanza is designed to have a simple API, high accuracy, and support for many human languages.

Pros

  • Supports over 60 human languages with pre-trained neural models
  • Offers a simple and consistent API for various NLP tasks
  • Provides seamless integration with PyTorch for easy customization and extension
  • Includes a Python interface to Stanford CoreNLP Java package

Cons

  • May have slower processing speed compared to some other NLP libraries
  • Requires more memory due to the use of neural models
  • Limited support for older Python versions (requires Python 3.6+)
  • Some advanced features may require additional setup or dependencies

Code Examples

  1. Basic text processing:
import stanza

# Download English model
stanza.download('en')

# Initialize English pipeline
nlp = stanza.Pipeline('en')

# Process text
doc = nlp("Hello world! This is a test.")

# Print sentences and tokens
for sent in doc.sentences:
    print([word.text for word in sent.words])
  1. Named Entity Recognition:
import stanza

nlp = stanza.Pipeline('en', processors='tokenize,ner')
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")

for ent in doc.ents:
    print(f"{ent.text}\t{ent.type}")
  1. Dependency parsing:
import stanza

nlp = stanza.Pipeline('en', processors='tokenize,pos,lemma,depparse')
doc = nlp("The quick brown fox jumps over the lazy dog.")

for sent in doc.sentences:
    print(sent.print_dependencies())

Getting Started

To get started with Stanza, follow these steps:

  1. Install Stanza using pip:
pip install stanza
  1. Download the desired language model:
import stanza
stanza.download('en')  # Download English model
  1. Create a pipeline and process text:
nlp = stanza.Pipeline('en')  # Initialize English pipeline
doc = nlp("Hello world!")  # Process text
print([(word.text, word.lemma) for sent in doc.sentences for word in sent.words])

Competitor Comparisons

huggingface logo

transformers

131,798

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

Pros of transformers

  • Broader scope, covering a wide range of NLP tasks and models
  • Larger community and more frequent updates
  • Extensive documentation and examples

Cons of transformers

  • Steeper learning curve due to its extensive features
  • Higher computational requirements for some models
  • Less specialized for specific NLP tasks like dependency parsing

Code comparison

Stanza:

import stanza
nlp = stanza.Pipeline('en')
doc = nlp("Hello world!")
print([(word.text, word.upos) for sent in doc.sentences for word in sent.words])

transformers:

from transformers import pipeline
nlp = pipeline("ner")
result = nlp("Hello world!")
print(result)

Summary

Stanza focuses on core NLP tasks with a simpler API, while transformers offers a broader range of models and tasks. Stanza excels in linguistic analysis, whereas transformers is more versatile for various NLP applications. Choose based on your specific needs and computational resources.

explosion logo

spaCy

29,635

💫 Industrial-strength Natural Language Processing (NLP) in Python

Pros of spaCy

  • Faster processing speed, especially for large-scale text analysis
  • More extensive pre-trained models and support for a wider range of languages
  • Better integration with deep learning frameworks and easier customization

Cons of spaCy

  • Less accurate for certain tasks, particularly in academic or research contexts
  • Steeper learning curve for beginners due to its more complex API
  • Larger memory footprint, which can be an issue for resource-constrained environments

Code Comparison

spaCy:

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")

for ent in doc.ents:
    print(ent.text, ent.label_)

Stanza:

import stanza

nlp = stanza.Pipeline("en")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")

for ent in doc.ents:
    print(ent.text, ent.type)

Both libraries offer similar functionality for basic NLP tasks, but spaCy's API is more concise and optimized for production environments, while Stanza's API is more aligned with academic research and offers higher accuracy for certain tasks.

nltk logo

nltk

13,520

NLTK Source

Pros of NLTK

  • Extensive documentation and educational resources
  • Broader range of NLP tasks and functionalities
  • Large, active community and long-standing reputation

Cons of NLTK

  • Slower performance for some tasks
  • Less focus on modern deep learning approaches
  • More complex setup for certain languages and tasks

Code Comparison

NLTK:

import nltk
nltk.download('punkt')
text = "Hello, world! This is NLTK."
tokens = nltk.word_tokenize(text)
print(tokens)

Stanza:

import stanza
nlp = stanza.Pipeline('en')
doc = nlp("Hello, world! This is Stanza.")
print([word.text for sent in doc.sentences for word in sent.words])

Both NLTK and Stanza are powerful NLP libraries, but they cater to different needs. NLTK offers a wide range of traditional NLP tools and is excellent for educational purposes, while Stanza focuses on providing state-of-the-art deep learning models for core NLP tasks. NLTK's extensive documentation and broader functionality make it a go-to choice for many researchers and students, but Stanza's modern approach and superior performance in certain tasks make it attractive for production environments and cutting-edge research.

facebookresearch logo

fairseq

30,331

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

Pros of fairseq

  • More comprehensive toolkit for sequence modeling, including machine translation, language modeling, and speech recognition
  • Supports a wider range of architectures and models, including Transformer-based models
  • Highly optimized for performance and scalability, suitable for large-scale training

Cons of fairseq

  • Steeper learning curve due to its extensive features and flexibility
  • Requires more setup and configuration compared to Stanza's simpler interface
  • May be overkill for basic NLP tasks that Stanza handles efficiently

Code Comparison

Stanza:

import stanza
nlp = stanza.Pipeline('en')
doc = nlp("Hello world!")
print([(word.text, word.upos) for sent in doc.sentences for word in sent.words])

fairseq:

from fairseq.models.transformer import TransformerModel
en2de = TransformerModel.from_pretrained('/path/to/model', checkpoint_file='model.pt')
en2de.translate('Hello world!')

The code snippets demonstrate the difference in complexity and use cases. Stanza is more straightforward for basic NLP tasks, while fairseq requires more setup but offers advanced sequence modeling capabilities.

piskvorky logo

gensim

15,616

Topic Modelling for Humans

Pros of Gensim

  • Specialized in topic modeling and document similarity
  • Efficient processing of large text corpora
  • Extensive documentation and tutorials

Cons of Gensim

  • Limited natural language processing capabilities
  • Less comprehensive language support
  • Steeper learning curve for beginners

Code Comparison

Gensim (Topic Modeling):

from gensim import corpora, models
dictionary = corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]
lda_model = models.LdaMulticore(corpus=corpus, num_topics=10)

Stanza (NLP Pipeline):

import stanza
nlp = stanza.Pipeline('en')
doc = nlp("Hello world!")
for sentence in doc.sentences:
    print([word.text for word in sentence.words])
allenai logo

allennlp

11,750

An open-source NLP research library, built on PyTorch.

Pros of AllenNLP

  • More extensive and feature-rich library with a wider range of NLP tasks and models
  • Better documentation and tutorials for beginners and advanced users
  • Active community and frequent updates

Cons of AllenNLP

  • Steeper learning curve due to its complexity and extensive features
  • Heavier and slower compared to Stanza for basic NLP tasks
  • Requires more setup and configuration for simple use cases

Code Comparison

Stanza:

import stanza
nlp = stanza.Pipeline('en')
doc = nlp("Hello world!")
print([(word.text, word.upos) for sent in doc.sentences for word in sent.words])

AllenNLP:

from allennlp.predictors.predictor import Predictor
predictor = Predictor.from_path("https://storage.googleapis.com/allennlp-public-models/structured-prediction-biaffine-parser-ontonotes-2020.02.10.tar.gz")
result = predictor.predict(sentence="Hello world!")
print([(token.text, token.pos) for token in result["tokens"]])

Both libraries offer powerful NLP capabilities, but AllenNLP provides a more comprehensive toolkit at the cost of increased complexity. Stanza is more straightforward for basic tasks, while AllenNLP excels in advanced NLP applications and research scenarios.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Stanza: A Python NLP Library for Many Human Languages

The Stanford NLP Group's official Python NLP library. It contains support for running various accurate natural language processing tools on 60+ languages and for accessing the Java Stanford CoreNLP software from Python. For detailed information please visit our official website.

🔥  A new collection of biomedical and clinical English model packages are now available, offering seamless experience for syntactic analysis and named entity recognition (NER) from biomedical literature text and clinical notes. For more information, check out our Biomedical models documentation page.

References

If you use this library in your research, please kindly cite our ACL2020 Stanza system demo paper:

@inproceedings{qi2020stanza,
    title={Stanza: A {Python} Natural Language Processing Toolkit for Many Human Languages},
    author={Qi, Peng and Zhang, Yuhao and Zhang, Yuhui and Bolton, Jason and Manning, Christopher D.},
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
    year={2020}
}

If you use our biomedical and clinical models, please also cite our Stanza Biomedical Models description paper:

@article{zhang2021biomedical,
    author = {Zhang, Yuhao and Zhang, Yuhui and Qi, Peng and Manning, Christopher D and Langlotz, Curtis P},
    title = {Biomedical and clinical {E}nglish model packages for the {S}tanza {P}ython {NLP} library},
    journal = {Journal of the American Medical Informatics Association},
    year = {2021},
    month = {06},
    issn = {1527-974X}
}

The PyTorch implementation of the neural pipeline in this repository is due to Peng Qi (@qipeng), Yuhao Zhang (@yuhaozhang), and Yuhui Zhang (@yuhui-zh15), with help from Jason Bolton (@j38), Tim Dozat (@tdozat) and John Bauer (@AngledLuffa). Maintenance of this repo is currently led by John Bauer.

If you use the CoreNLP software through Stanza, please cite the CoreNLP software package and the respective modules as described here ("Citing Stanford CoreNLP in papers"). The CoreNLP client is mostly written by Arun Chaganty, and Jason Bolton spearheaded merging the two projects together.

If you use the Semgrex or Ssurgeon part of CoreNLP, please cite our GURT paper on Semgrex and Ssurgeon:

@inproceedings{bauer-etal-2023-semgrex,
    title = "Semgrex and Ssurgeon, Searching and Manipulating Dependency Graphs",
    author = "Bauer, John  and
      Kiddon, Chlo{\'e}  and
      Yeh, Eric  and
      Shan, Alex  and
      D. Manning, Christopher",
    booktitle = "Proceedings of the 21st International Workshop on Treebanks and Linguistic Theories (TLT, GURT/SyntaxFest 2023)",
    month = mar,
    year = "2023",
    address = "Washington, D.C.",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.tlt-1.7",
    pages = "67--73",
    abstract = "Searching dependency graphs and manipulating them can be a time consuming and challenging task to get right. We document Semgrex, a system for searching dependency graphs, and introduce Ssurgeon, a system for manipulating the output of Semgrex. The compact language used by these systems allows for easy command line or API processing of dependencies. Additionally, integration with publicly released toolkits in Java and Python allows for searching text relations and attributes over natural text.",
}

Issues and Usage Q&A

To ask questions, report issues or request features 🤔, please use the GitHub Issue Tracker. Before creating a new issue, please make sure to search for existing issues that may solve your problem, or visit the Frequently Asked Questions (FAQ) page on our website.

Contributing to Stanza

We welcome community contributions to Stanza in the form of bugfixes 🛠️ and enhancements 💡! If you want to contribute, please first read our contribution guideline.

Installation

pip

Stanza supports Python 3.6 or later. We recommend that you install Stanza via pip, the Python package manager. To install, simply run:

pip install stanza

This should also help resolve all of the dependencies of Stanza, for instance PyTorch 1.3.0 or above.

If you currently have a previous version of stanza installed, use:

pip install stanza -U

Anaconda

To install Stanza via Anaconda, use the following conda command:

conda install -c stanfordnlp stanza

Note that for now installing Stanza via Anaconda does not work for Python 3.10. For Python 3.10 please use pip installation.

From Source

Alternatively, you can also install from source of this git repository, which will give you more flexibility in developing on top of Stanza. For this option, run

git clone https://github.com/stanfordnlp/stanza.git
cd stanza
pip install -e .

Running Stanza

Getting Started with the neural pipeline

To run your first Stanza pipeline, simply following these steps in your Python interactive interpreter:

>>> import stanza
>>> stanza.download('en')       # This downloads the English models for the neural pipeline
>>> nlp = stanza.Pipeline('en') # This sets up a default neural pipeline in English
>>> doc = nlp("Barack Obama was born in Hawaii.  He was elected president in 2008.")
>>> doc.sentences[0].print_dependencies()

If you encounter requests.exceptions.ConnectionError, please try to use a proxy:

>>> import stanza
>>> proxies = {'http': 'http://ip:port', 'https': 'http://ip:port'}
>>> stanza.download('en', proxies=proxies)  # This downloads the English models for the neural pipeline
>>> nlp = stanza.Pipeline('en')             # This sets up a default neural pipeline in English
>>> doc = nlp("Barack Obama was born in Hawaii.  He was elected president in 2008.")
>>> doc.sentences[0].print_dependencies()

The last command will print out the words in the first sentence in the input string (or Document, as it is represented in Stanza), as well as the indices for the word that governs it in the Universal Dependencies parse of that sentence (its "head"), along with the dependency relation between the words. The output should look like:

('Barack', '4', 'nsubj:pass')
('Obama', '1', 'flat')
('was', '4', 'aux:pass')
('born', '0', 'root')
('in', '6', 'case')
('Hawaii', '4', 'obl')
('.', '4', 'punct')

See our getting started guide for more details.

Accessing Java Stanford CoreNLP software

Aside from the neural pipeline, this package also includes an official wrapper for accessing the Java Stanford CoreNLP software with Python code.

There are a few initial setup steps.

  • Download Stanford CoreNLP and models for the language you wish to use
  • Put the model jars in the distribution folder
  • Tell the Python code where Stanford CoreNLP is located by setting the CORENLP_HOME environment variable (e.g., in *nix): export CORENLP_HOME=/path/to/stanford-corenlp-4.5.3

We provide comprehensive examples in our documentation that show how one can use CoreNLP through Stanza and extract various annotations from it.

Online Colab Notebooks

To get your started, we also provide interactive Jupyter notebooks in the demo folder. You can also open these notebooks and run them interactively on Google Colab. To view all available notebooks, follow these steps:

  • Go to the Google Colab website
  • Navigate to File -> Open notebook, and choose GitHub in the pop-up menu
  • Note that you do not need to give Colab access permission to your GitHub account
  • Type stanfordnlp/stanza in the search bar, and click enter

Trained Models for the Neural Pipeline

We currently provide models for all of the Universal Dependencies treebanks v2.8, as well as NER models for a few widely-spoken languages. You can find instructions for downloading and using these models here.

Batching To Maximize Pipeline Speed

To maximize speed performance, it is essential to run the pipeline on batches of documents. Running a for loop on one sentence at a time will be very slow. The best approach at this time is to concatenate documents together, with each document separated by a blank line (i.e., two line breaks \n\n). The tokenizer will recognize blank lines as sentence breaks. We are actively working on improving multi-document processing.

Training your own neural pipelines

All neural modules in this library can be trained with your own data. The tokenizer, the multi-word token (MWT) expander, the POS/morphological features tagger, the lemmatizer and the dependency parser require CoNLL-U formatted data, while the NER model requires the BIOES format. Currently, we do not support model training via the Pipeline interface. Therefore, to train your own models, you need to clone this git repository and run training from the source.

For detailed step-by-step guidance on how to train and evaluate your own models, please visit our training documentation.

LICENSE

Stanza is released under the Apache License, Version 2.0. See the LICENSE file for more details.