spark-nlp

State of the Art Natural Language Processing

3,802

707

3,802

View on GitHub

Top Related Projects

transformers

131,798

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

spaCy

29,635

💫 Industrial-strength Natural Language Processing (NLP) in Python

flair

13,877

A very simple framework for state-of-the-art Natural Language Processing (NLP)

stanza

7,206

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

Quick Overview

Spark NLP is an open-source Natural Language Processing (NLP) library built on top of Apache Spark and Scala. It provides high-performance NLP annotations for machine learning pipelines that scale easily in distributed environments. Spark NLP offers production-grade, scalable, and unified solutions for various NLP tasks.

Pros

High performance and scalability, leveraging Apache Spark's distributed computing capabilities
Comprehensive set of NLP tools and pre-trained models for various languages and tasks
Easy integration with existing Spark ML pipelines
Active development and community support

Cons

Steep learning curve for users unfamiliar with Apache Spark or Scala
Resource-intensive, requiring significant computational power for large-scale tasks
Limited flexibility compared to some other NLP libraries when working outside of Spark environments
Documentation can be overwhelming for beginners

Code Examples

Basic text classification pipeline:

from sparknlp.base import *
from sparknlp.annotator import *
from pyspark.ml import Pipeline

documentAssembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

tokenizer = Tokenizer() \
    .setInputCols(["document"]) \
    .setOutputCol("token")

classifier = ClassifierDLApproach() \
    .setInputCols(["document", "token"]) \
    .setOutputCol("class") \
    .setLabelColumn("label")

pipeline = Pipeline().setStages([
    documentAssembler,
    tokenizer,
    classifier
])

Named Entity Recognition:

from sparknlp.pretrained import PretrainedPipeline

pipeline = PretrainedPipeline("recognize_entities_dl", lang="en")

result = pipeline.annotate("John Doe works at Google in London")
print(result['ner'])

Sentiment analysis:

sentiment_pipeline = PretrainedPipeline("analyze_sentiment", lang="en")

result = sentiment_pipeline.annotate("I love this product! It's amazing.")
print(result['sentiment'])

Getting Started

To get started with Spark NLP, follow these steps:

Install Spark NLP:

pip install spark-nlp

Import required modules and start a Spark session:

import sparknlp
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("Spark NLP") \
    .master("local[*]") \
    .config("spark.driver.memory", "8G") \
    .config("spark.driver.maxResultSize", "2G") \
    .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:4.4.0") \
    .getOrCreate()

sparknlp.start()

Now you can use Spark NLP components in your code, as shown in the examples above.

Competitor Comparisons

transformers

131,798

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

Pros of transformers

Broader language support and more pre-trained models
Easier integration with PyTorch and TensorFlow
More frequent updates and active community contributions

Cons of transformers

Less optimized for distributed computing and big data processing
Steeper learning curve for beginners in NLP
Limited built-in support for traditional NLP tasks (e.g., tokenization, POS tagging)

Code Comparison

spark-nlp:

from sparknlp.pretrained import PretrainedPipeline

pipeline = PretrainedPipeline("explain_document_dl", lang="en")
result = pipeline.annotate("Hello world!")

transformers:

from transformers import pipeline

nlp = pipeline("sentiment-analysis")
result = nlp("Hello world!")[0]

Both libraries offer easy-to-use pipelines for NLP tasks, but spark-nlp is more tightly integrated with Apache Spark for distributed processing, while transformers focuses on providing access to state-of-the-art models and is more flexible in terms of deep learning frameworks.

spaCy

29,635

💫 Industrial-strength Natural Language Processing (NLP) in Python

Pros of spaCy

Lightweight and fast, optimized for CPU performance
Extensive language support with pre-trained models for many languages
User-friendly API with excellent documentation and community support

Cons of spaCy

Limited scalability for large-scale distributed processing
Less integration with big data ecosystems compared to Spark NLP
Fewer advanced NLP features out-of-the-box (e.g., limited named entity resolution)

Code Comparison

spaCy:

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
for ent in doc.ents:
    print(ent.text, ent.label_)

Spark NLP:

from sparknlp.base import *
from sparknlp.annotator import *

pipeline = PretrainedPipeline("recognize_entities_dl", lang="en")
result = pipeline.annotate("Apple is looking at buying U.K. startup for $1 billion")
print(result['entities'])

Both libraries offer powerful NLP capabilities, but spaCy is more suitable for smaller-scale projects and rapid prototyping, while Spark NLP excels in distributed processing and integration with big data ecosystems.

flair

13,877

A very simple framework for state-of-the-art Natural Language Processing (NLP)

Pros of flair

Lightweight and easy to use, with a simple API
Focuses on state-of-the-art NLP models and techniques
Supports a wide range of languages out-of-the-box

Cons of flair

Limited scalability for large-scale data processing
Fewer pre-built pipelines and components compared to spark-nlp
Less integration with big data ecosystems

Code Comparison

flair:

from flair.data import Sentence
from flair.models import SequenceTagger

tagger = SequenceTagger.load('ner')
sentence = Sentence('John Doe works at Apple Inc.')
tagger.predict(sentence)

spark-nlp:

from sparknlp.pretrained import PretrainedPipeline

pipeline = PretrainedPipeline('recognize_entities_dl')
result = pipeline.annotate('John Doe works at Apple Inc.')

Both libraries offer straightforward ways to perform NER, but flair's approach is more lightweight and focused on single sentences, while spark-nlp is designed for scalable processing within the Spark ecosystem.

stanza

7,206

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

Pros of Stanza

Lightweight and easy to install, with minimal dependencies
Supports over 60 languages out-of-the-box
Provides accurate and state-of-the-art NLP models

Cons of Stanza

Limited scalability for large-scale data processing
Lacks integration with big data ecosystems like Apache Spark
Fewer pre-trained pipelines and models compared to Spark NLP

Code Comparison

Stanza:

import stanza
nlp = stanza.Pipeline('en')
doc = nlp("Hello world!")
print([(word.text, word.lemma) for sent in doc.sentences for word in sent.words])

Spark NLP:

from sparknlp.pretrained import PretrainedPipeline
pipeline = PretrainedPipeline('explain_document_dl', lang='en')
result = pipeline.annotate("Hello world!")
print([(token.result, token.lemma) for token in result['token']])

Both libraries offer similar functionality for basic NLP tasks, but Spark NLP is designed for scalability and integration with Apache Spark, while Stanza focuses on providing accurate models for a wide range of languages in a lightweight package.

gensim

15,616

Topic Modelling for Humans

Pros of Gensim

Lightweight and easy to install, with fewer dependencies
Faster processing for smaller datasets and simpler NLP tasks
More extensive documentation and tutorials for beginners

Cons of Gensim

Limited scalability for large-scale distributed processing
Fewer pre-trained models and pipelines compared to Spark NLP
Less integration with big data ecosystems and tools

Code Comparison

Gensim:

from gensim.models import Word2Vec

sentences = [["cat", "say", "meow"], ["dog", "say", "woof"]]
model = Word2Vec(sentences, min_count=1)
print(model.wv.most_similar("dog"))

Spark NLP:

from sparknlp.base import *
from sparknlp.annotator import *

pipeline = Pipeline().setStages([
    DocumentAssembler(),
    TokenizerModel.pretrained(),
    WordEmbeddingsModel.pretrained()
])
result = pipeline.fit(data).transform(data)

Both libraries offer powerful NLP capabilities, but Gensim is more suitable for smaller-scale projects and quick prototyping, while Spark NLP excels in large-scale, distributed processing environments. Gensim's simpler API and extensive documentation make it more accessible for beginners, whereas Spark NLP provides more advanced features and integration with big data tools.

nltk

13,520

NLTK Source

Pros of NLTK

Extensive documentation and educational resources
Large community support and long-standing reputation
Comprehensive set of built-in corpora and lexical resources

Cons of NLTK

Slower performance for large-scale processing
Less suitable for production environments
Limited support for deep learning and advanced NLP tasks

Code Comparison

NLTK:

import nltk
from nltk.tokenize import word_tokenize
from nltk.tag import pos_tag

text = "NLTK is a leading platform for building Python programs to work with human language data."
tokens = word_tokenize(text)
pos_tags = pos_tag(tokens)

Spark NLP:

import sparknlp
from sparknlp.base import *
from sparknlp.annotator import *

spark = sparknlp.start()
pipeline = Pipeline().setStages([
    DocumentAssembler(),
    SentenceDetector(),
    Tokenizer(),
    POSTagger()
])
result = pipeline.fit(spark.createDataFrame([["Spark NLP is a library for large-scale NLP using Apache Spark."]]).toDF("text")).transform(data)

The code examples demonstrate basic tokenization and part-of-speech tagging in both libraries. NLTK offers a more straightforward approach, while Spark NLP leverages Apache Spark for distributed processing, making it more suitable for large-scale applications.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Spark NLP: State-of-the-Art Natural Language Processing & LLMs Library

Spark NLP is a state-of-the-art Natural Language Processing library built on top of Apache Spark. It provides simple, performant & accurate NLP annotations for machine learning pipelines that scale easily in a distributed environment. Spark NLP comes with 36000+ pretrained pipelines and models in more than 200+ languages. It also offers tasks such as Tokenization, Word Segmentation, Part-of-Speech Tagging, Word and Sentence Embeddings, Named Entity Recognition, Dependency Parsing, Spell Checking, Text Classification, Sentiment Analysis, Token Classification, Machine Translation (+180 languages), Summarization, Question Answering, Table Question Answering, Text Generation, Image Classification, Image to Text (captioning), Automatic Speech Recognition, Zero-Shot Learning, and many more NLP tasks.

Spark NLP is the only open-source NLP library in production that offers state-of-the-art transformers such as BERT, CamemBERT, ALBERT, ELECTRA, XLNet, DistilBERT, RoBERTa, DeBERTa, XLM-RoBERTa, Longformer, ELMO, Universal Sentence Encoder, Llama-2, M2M100, BART, Instructor, E5, Google T5, MarianMT, OpenAI GPT2, Vision Transformers (ViT), OpenAI Whisper, and many more not only to Python and R, but also to JVM ecosystem (Java, Scala, and Kotlin) at scale by extending Apache Spark natively.

Project's website

Take a look at our official Spark NLP page: https://sparknlp.org/ for user documentation and examples

Features

Quick Start

This is a quick example of how to use Spark NLP pre-trained pipeline in Python and PySpark:

$ java -version
# should be Java 8 or 11 (Oracle or OpenJDK)
$ conda create -n sparknlp python=3.7 -y
$ conda activate sparknlp
# spark-nlp by default is based on pyspark 3.x
$ pip install spark-nlp==5.4.0 pyspark==3.3.1

In Python console or Jupyter Python3 kernel:

# Import Spark NLP
from sparknlp.base import *
from sparknlp.annotator import *
from sparknlp.pretrained import PretrainedPipeline
import sparknlp

# Start SparkSession with Spark NLP
# start() functions has 3 parameters: gpu, apple_silicon, and memory
# sparknlp.start(gpu=True) will start the session with GPU support
# sparknlp.start(apple_silicon=True) will start the session with macOS M1 & M2 support
# sparknlp.start(memory="16G") to change the default driver memory in SparkSession
spark = sparknlp.start()

# Download a pre-trained pipeline
pipeline = PretrainedPipeline('explain_document_dl', lang='en')

# Your testing dataset
text = """
The Mona Lisa is a 16th century oil painting created by Leonardo.
It's held at the Louvre in Paris.
"""

# Annotate your testing dataset
result = pipeline.annotate(text)

# What's in the pipeline
list(result.keys())
Output: ['entities', 'stem', 'checked', 'lemma', 'document',
         'pos', 'token', 'ner', 'embeddings', 'sentence']

# Check the results
result['entities']
Output: ['Mona Lisa', 'Leonardo', 'Louvre', 'Paris']

For more examples, you can visit our dedicated examples to showcase all Spark NLP use cases!

Packages Cheatsheet

This is a cheatsheet for corresponding Spark NLP Maven package to Apache Spark / PySpark major version:

Apache Spark	Spark NLP on CPU	Spark NLP on GPU	Spark NLP on AArch64 (linux)	Spark NLP on Apple Silicon
3.0/3.1/3.2/3.3/3.4/3.5	`spark-nlp`	`spark-nlp-gpu`	`spark-nlp-aarch64`	`spark-nlp-silicon`
Start Function	`sparknlp.start()`	`sparknlp.start(gpu=True)`	`sparknlp.start(aarch64=True)`	`sparknlp.start(apple_silicon=True)`

NOTE: M1/M2 and AArch64 are under experimental support. Access and support to these architectures are limited by the community and we had to build most of the dependencies by ourselves to make them compatible. We support these two architectures, however, they may not work in some environments.

Pipelines and Models

For a quick example of using pipelines and models take a look at our official documentation

Please check out our Models Hub for the full list of pre-trained models with examples, demo, benchmark, and more

Platform and Ecosystem Support

Apache Spark Support

Spark NLP 5.4.0 has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x

Spark NLP	Apache Spark 3.5.x	Apache Spark 3.4.x	Apache Spark 3.3.x	Apache Spark 3.2.x	Apache Spark 3.1.x	Apache Spark 3.0.x	Apache Spark 2.4.x	Apache Spark 2.3.x
5.4.x	YES	YES	YES	YES	YES	YES	NO	NO
5.3.x	YES	YES	YES	YES	YES	YES	NO	NO
5.2.x	YES	YES	YES	YES	YES	YES	NO	NO
5.1.x	Partially	YES	YES	YES	YES	YES	NO	NO
5.0.x	YES	YES	YES	YES	YES	YES	NO	NO

Find out more about Spark NLP versions from our release notes.

Scala and Python Support

Spark NLP	Python 3.6	Python 3.7	Python 3.8	Python 3.9	Python 3.10	Scala 2.11	Scala 2.12
5.3.x	NO	YES	YES	YES	YES	NO	YES
5.2.x	NO	YES	YES	YES	YES	NO	YES
5.1.x	NO	YES	YES	YES	YES	NO	YES
5.0.x	NO	YES	YES	YES	YES	NO	YES

Find out more about 4.x SparkNLP versions in our official documentation

Databricks Support

Spark NLP 5.4.0 has been tested and is compatible with the following runtimes:

CPU	GPU
14.0 / 14.0 ML	14.0 ML & GPU
14.1 / 14.1 ML	14.1 ML & GPU
14.2 / 14.2 ML	14.2 ML & GPU
14.3 / 14.3 ML	14.3 ML & GPU

We are compatible with older runtimes. For a full list check databricks support in our official documentation

EMR Support

Spark NLP 5.4.0 has been tested and is compatible with the following EMR releases:

EMR Release
emr-6.13.0
emr-6.14.0
emr-6.15.0
emr-7.0.0

We are compatible with older EMR releases. For a full list check EMR support in our official documentation

Full list of Amazon EMR 6.x releases Full list 5.4.2mazon EMR 7.x releases](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-7x.html)

NOTE: The EMR 6.1.0 and 6.1.1 are not supported.

Installation

Command line (requires internet connection)

To install spark-nlp packages through command line follow these instructions from our official documentation

Scala

Spark NLP supports Scala 2.12.15 if you are using Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x versions. Our packages are deployed to Maven central. To add any of our packages as a dependency in your application you can follow these instructions from our official documentation.

If you are interested, there is a simple SBT project for Spark NLP to guide you on how to use it in your projects Spark NLP SBT S5.4.2r

Python

Spark NLP supports Python 3.7.x and above depending on your major PySpark version. Check all available installations for Python in our official documentation

Compiled JARs

To compile the jars from source follow these instructions from our official documenation

Platform-Specific Instructions

For detailed instructions on how to use Spark NLP on supported platforms, please refer to our official documentation:

Platform	Supported Language(s)
Apache Zeppelin	Scala, Python
Jupyter Notebook	Python
Google Colab Notebook	Python
Kaggle Kernel	Python
Databricks Cluster	Scala, Python
EMR Cluster	Scala, Python
GCP Dataproc Cluster	Scala, Python

Offline

Spark NLP library and all the pre-trained models/pipelines can be used entirely offline with no access to the Internet. Please check these instructions from our official documentation to use Spark NLP offline

Advanced Settings

You can change Spark NLP configurations via Spark properties configuration. Please check these instructions from our official documentation.

S3 Integration

In Spark NLP we can define S3 locations to:

Export log files of training models
Store tensorflow graphs used in NerDLApproach

Please check these instructions from our official documentation.

Document5.4.2

Examples

Need more examples? Check out our dedicated Spark NLP Examples repository to showcase all Spark NLP use cases!

Also, don't forget to check Spark NLP in Action built by Streamlit.

All examples: spark-nlp/examples

FAQ

Check our Articles and Videos page here

Citation

We have published a paper that you can cite for the Spark NLP library:

@article{KOCAMAN2021100058,
    title = {Spark NLP: Natural language understanding at scale},
    journal = {Software Impacts},
    pages = {100058},
    year = {2021},
    issn = {2665-9638},
    doi = {https://doi.org/10.1016/j.simpa.2021.100058},
    url = {https://www.sciencedirect.com/science/article/pii/S2665963.2.300063},
    author = {Veysel Kocaman and David Talby},
    keywords = {Spark, Natural language processing, Deep learning, Tensorflow, Cluster},
    abstract = {Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100+ pretrained pipelines and models in more than 192+ languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing 9x growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the worldâs most widely used NLP library in the enterprise.}
    }
}5.4.2

Community support

Slack For live discussion with the Spark NLP community and the team
GitHub Bug reports, feature requests, and contributions
Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
Medium Spark NLP articles
YouTube Spark NLP video tutorials

Contributing

We appreciate any sort of contributions:

ideas
feedback
documentation
bug reports
NLP training and testing corpora
Development and testing

Clone the repo and submit your pull-requests! Or directly create issues in this repo.

John Snow Labs

http://johnsnowlabs.com

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot