Convert Figma logo to code with AI

keon logoawesome-nlp

:book: A curated list of resources dedicated to Natural Language Processing (NLP)

16,566
2,572
16,566
9
View on GitHub

Top Related Projects

fendouai's avatar

Awesome-Chatbot

2,035

Awesome Chatbot Projects,Corpus,Papers,Tutorials.Chinese Chatbot =>:

graykode's avatar

nlp-tutorial

14,131

Natural Language Processing Tutorial for Deep Learning Researchers

thunlp's avatar

PLMpapers

3,319

Must-read Papers on pre-trained language models.

huggingface's avatar

transformers

131,798

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

explosion's avatar

spaCy

29,635

💫 Industrial-strength Natural Language Processing (NLP) in Python

microsoft's avatar

nlp-recipes

6,371

Natural Language Processing Best Practices & Examples

Quick Overview

Awesome-NLP is a curated list of resources dedicated to Natural Language Processing (NLP). It serves as a comprehensive collection of tools, libraries, datasets, and educational materials for NLP practitioners, researchers, and enthusiasts. The repository aims to provide a one-stop reference for anyone looking to explore or work in the field of NLP.

Pros

  • Extensive collection of NLP resources covering various aspects of the field
  • Regularly updated with new and relevant content
  • Well-organized structure, making it easy to find specific resources
  • Includes both open-source and commercial tools, catering to different needs

Cons

  • May be overwhelming for beginners due to the vast amount of information
  • Some links may become outdated over time
  • Lacks detailed descriptions or comparisons of the listed resources
  • Primarily focuses on English language resources, with limited coverage of other languages

Getting Started

As this is not a code library but a curated list of resources, there is no specific code or installation process. To get started with Awesome-NLP:

  1. Visit the GitHub repository: https://github.com/keon/awesome-nlp
  2. Browse through the different sections based on your interests or needs
  3. Click on the links to explore specific resources, tools, or libraries
  4. Consider starring or watching the repository to stay updated with new additions

Competitor Comparisons

fendouai logo

Awesome-Chatbot

2,035

Awesome Chatbot Projects,Corpus,Papers,Tutorials.Chinese Chatbot =>:

Pros of Awesome-Chatbot

  • More focused on chatbot-specific resources and technologies
  • Includes sections on commercial chatbot platforms and services
  • Provides links to chatbot-related communities and forums

Cons of Awesome-Chatbot

  • Less comprehensive coverage of general NLP topics
  • Fewer academic and research-oriented resources
  • Not as frequently updated as awesome-nlp

Code Comparison

While both repositories are curated lists and don't contain actual code, they differ in their organization. Here's an example of how they structure their content:

awesome-nlp:

## Contents
1. [Research Summaries and Trends](#research-summaries-and-trends)
2. [Tutorials](#tutorials)
3. [Datasets](#datasets)
4. [Techniques](#techniques)

Awesome-Chatbot:

## Chatbot Tutorials
* [Chatbot Tutorials](#chatbot-tutorials)
* [Deep Learning in Chatbots](#deep-learning-in-chatbots)
* [Flow Chart of Chatbot](#flow-chart-of-chatbot)

Both repositories serve as valuable resources for NLP and chatbot development, with awesome-nlp offering a broader scope of NLP topics and Awesome-Chatbot providing more specialized information for chatbot creators. The choice between them depends on the specific needs of the user and their focus area within NLP or chatbot development.

graykode logo

nlp-tutorial

14,131

Natural Language Processing Tutorial for Deep Learning Researchers

Pros of nlp-tutorial

  • Provides hands-on, practical implementations of NLP algorithms
  • Includes step-by-step tutorials with code examples
  • Focuses on PyTorch implementations, making it easier for deep learning practitioners

Cons of nlp-tutorial

  • Limited in scope compared to the comprehensive resource list in awesome-nlp
  • May not cover the latest advancements in NLP as frequently as awesome-nlp
  • Lacks the extensive categorization and organization found in awesome-nlp

Code Comparison

nlp-tutorial (PyTorch implementation):

class NNLM(nn.Module):
    def __init__(self):
        super(NNLM, self).__init__()
        self.C = nn.Embedding(n_class, m)
        self.H = nn.Linear(n_step * m, n_hidden, bias=False)
        self.d = nn.Parameter(torch.ones(n_hidden))
        self.U = nn.Linear(n_hidden, n_class, bias=False)
        self.W = nn.Linear(n_step * m, n_class, bias=False)

awesome-nlp (typically doesn't include code, but might link to implementations like):

from nltk import word_tokenize
from gensim.models import Word2Vec

sentences = [word_tokenize(sentence.lower()) for sentence in raw_sentences]
model = Word2Vec(sentences, size=100, window=5, min_count=1, workers=4)
thunlp logo

PLMpapers

3,319

Must-read Papers on pre-trained language models.

Pros of PLMpapers

  • Focused specifically on pre-trained language models (PLMs)
  • Regularly updated with recent papers and research
  • Well-organized by different aspects of PLMs (e.g., architecture, training, applications)

Cons of PLMpapers

  • Limited scope compared to the broader NLP field covered in awesome-nlp
  • Lacks resources like datasets, tools, and libraries
  • May be more challenging for beginners to navigate due to its research-oriented nature

Code comparison

Not applicable, as both repositories are primarily curated lists of resources and papers without significant code content.

Additional notes

awesome-nlp provides a comprehensive overview of NLP resources, including tutorials, courses, libraries, and datasets. It's suitable for both beginners and experienced practitioners looking for a wide range of NLP-related content.

PLMpapers, on the other hand, is a specialized repository focusing on the latest research in pre-trained language models. It's particularly useful for researchers and advanced practitioners who want to stay up-to-date with cutting-edge developments in this specific area of NLP.

Both repositories serve different purposes and can be complementary for those interested in NLP, depending on their specific needs and level of expertise.

huggingface logo

transformers

131,798

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

Pros of transformers

  • Provides ready-to-use implementations of state-of-the-art NLP models
  • Offers a unified API for various NLP tasks and model architectures
  • Includes extensive documentation and examples for easy integration

Cons of transformers

  • Focuses primarily on transformer-based models, limiting coverage of other NLP techniques
  • Requires more computational resources due to the complexity of transformer models
  • Has a steeper learning curve for beginners compared to a curated list of resources

Code comparison

transformers:

from transformers import pipeline

classifier = pipeline("sentiment-analysis")
result = classifier("I love this product!")[0]
print(f"Label: {result['label']}, Score: {result['score']:.4f}")

awesome-nlp:

# No direct code implementation
# Provides links to various NLP libraries and resources

Summary

transformers is a comprehensive library for implementing and using state-of-the-art NLP models, particularly focused on transformer architectures. It offers ready-to-use models and a unified API for various tasks.

awesome-nlp is a curated list of NLP resources, libraries, and tools. It serves as a valuable reference for exploring different aspects of NLP but doesn't provide direct implementations.

Choose transformers for hands-on work with cutting-edge models, and awesome-nlp for a broader overview of NLP resources and techniques.

explosion logo

spaCy

29,635

💫 Industrial-strength Natural Language Processing (NLP) in Python

Pros of spaCy

  • Comprehensive NLP library with ready-to-use models and pipelines
  • Optimized for production use with high performance and efficiency
  • Extensive documentation and active community support

Cons of spaCy

  • Steeper learning curve for beginners compared to curated resource lists
  • Focused on specific NLP tasks, may not cover all areas of NLP research
  • Requires more setup and installation compared to a simple resource list

Code Comparison

spaCy:

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")

for ent in doc.ents:
    print(ent.text, ent.label_)

awesome-nlp:

## Libraries

- [NLTK](http://www.nltk.org/) - Natural Language Toolkit
- [spaCy](https://spacy.io/) - Industrial-strength NLP
- [CoreNLP](https://stanfordnlp.github.io/CoreNLP/) - Stanford's NLP toolkit

Summary

spaCy is a comprehensive NLP library offering ready-to-use models and optimized performance, while awesome-nlp is a curated list of NLP resources. spaCy provides a more hands-on approach with actual code implementation, whereas awesome-nlp serves as a reference guide to various NLP tools and resources. The choice between them depends on whether you need a practical NLP tool (spaCy) or a broad overview of available NLP resources (awesome-nlp).

microsoft logo

nlp-recipes

6,371

Natural Language Processing Best Practices & Examples

Pros of nlp-recipes

  • Provides practical, ready-to-use code examples and notebooks for various NLP tasks
  • Offers end-to-end solutions, including data preparation, model training, and evaluation
  • Includes detailed documentation and explanations for each recipe

Cons of nlp-recipes

  • Focuses primarily on Microsoft's NLP tools and frameworks
  • May not cover as wide a range of NLP topics as awesome-nlp
  • Requires more technical knowledge to implement and adapt the recipes

Code Comparison

awesome-nlp:

# No code examples provided directly in the repository
# It's a curated list of NLP resources and links

nlp-recipes:

# Example from text classification recipe
from utils_nlp.models.transformers.sequence_classification import Classifier

classifier = Classifier(model_name="distilbert-base-uncased", num_labels=len(label_map))
classifier.fit(train_dataloader)

Summary

awesome-nlp is a comprehensive list of NLP resources, tools, and libraries, while nlp-recipes provides practical code examples and end-to-end solutions for various NLP tasks. awesome-nlp offers a broader overview of the NLP landscape, while nlp-recipes focuses on hands-on implementation using Microsoft's tools. Choose awesome-nlp for discovering resources and nlp-recipes for practical implementation guidance.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

awesome-nlp

Awesome

A curated list of resources dedicated to Natural Language Processing

Awesome NLP Logo

Read this in English, Traditional Chinese

Please read the contribution guidelines before contributing. Please add your favourite NLP resource by raising a pull request

Contents

Research Summaries and Trends

Prominent NLP Research Labs

Back to Top

Tutorials

Back to Top

Reading Content

General Machine Learning

Introductions and Guides to NLP

Blogs and Newsletters

Videos and Online Courses

Back to Top

Books

Libraries

Back to Top

  • Node.js and Javascript - Node.js Libaries for NLP | Back to Top

    • Twitter-text - A JavaScript implementation of Twitter's text processing library
    • Knwl.js - A Natural Language Processor in JS
    • Retext - Extensible system for analyzing and manipulating natural language
    • NLP Compromise - Natural Language processing in the browser
    • Natural - general natural language facilities for node
    • Poplar - A web-based annotation tool for natural language processing (NLP)
    • NLP.js - An NLP library for building bots
    • node-question-answering - Fast and production-ready question answering w/ DistilBERT in Node.js

  • Python - Python NLP Libraries | Back to Top

    • sentimental-onix Sentiment models for spacy using onnx
    • TextAttack - Adversarial attacks, adversarial training, and data augmentation in NLP
    • TextBlob - Providing a consistent API for diving into common natural language processing (NLP) tasks. Stands on the giant shoulders of Natural Language Toolkit (NLTK) and Pattern, and plays nicely with both :+1:
    • spaCy - Industrial strength NLP with Python and Cython :+1:
    • Speedster - Automatically apply SOTA optimization techniques to achieve the maximum inference speed-up on your hardware
      • textacy - Higher level NLP built on spaCy
    • gensim - Python library to conduct unsupervised semantic modelling from plain text :+1:
    • scattertext - Python library to produce d3 visualizations of how language differs between corpora
    • GluonNLP - A deep learning toolkit for NLP, built on MXNet/Gluon, for research prototyping and industrial deployment of state-of-the-art models on a wide range of NLP tasks.
    • AllenNLP - An NLP research library, built on PyTorch, for developing state-of-the-art deep learning models on a wide variety of linguistic tasks.
    • PyTorch-NLP - NLP research toolkit designed to support rapid prototyping with better data loaders, word vector loaders, neural network layer representations, common NLP metrics such as BLEU
    • Rosetta - Text processing tools and wrappers (e.g. Vowpal Wabbit)
    • PyNLPl - Python Natural Language Processing Library. General purpose NLP library for Python, handles some specific formats like ARPA language models, Moses phrasetables, GIZA++ alignments.
    • foliapy - Python library for working with FoLiA, an XML format for linguistic annotation.
    • PySS3 - Python package that implements a novel white-box machine learning model for text classification, called SS3. Since SS3 has the ability to visually explain its rationale, this package also comes with easy-to-use interactive visualizations tools (online demos).
    • jPTDP - A toolkit for joint part-of-speech (POS) tagging and dependency parsing. jPTDP provides pre-trained models for 40+ languages.
    • BigARTM - a fast library for topic modelling
    • Snips NLU - A production ready library for intent parsing
    • Chazutsu - A library for downloading&parsing standard NLP research datasets
    • Word Forms - Word forms can accurately generate all possible forms of an English word
    • Multilingual Latent Dirichlet Allocation (LDA) - A multilingual and extensible document clustering pipeline
    • Natural Language Toolkit (NLTK) - A library containing a wide variety of NLP functionality, supporting over 50 corpora.
    • NLP Architect - A library for exploring the state-of-the-art deep learning topologies and techniques for NLP and NLU
    • Flair - A very simple framework for state-of-the-art multilingual NLP built on PyTorch. Includes BERT, ELMo and Flair embeddings.
    • Kashgari - Simple, Keras-powered multilingual NLP framework, allows you to build your models in 5 minutes for named entity recognition (NER), part-of-speech tagging (PoS) and text classification tasks. Includes BERT and word2vec embedding.
    • FARM - Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.
    • Haystack - End-to-end Python framework for building natural language search interfaces to data. Leverages Transformers and the State-of-the-Art of NLP. Supports DPR, Elasticsearch, HuggingFace’s Modelhub, and much more!
    • Rita DSL - a DSL, loosely based on RUTA on Apache UIMA. Allows to define language patterns (rule-based NLP) which are then translated into spaCy, or if you prefer less features and lightweight - regex patterns.
    • Transformers - Natural Language Processing for TensorFlow 2.0 and PyTorch.
    • Tokenizers - Tokenizers optimized for Research and Production.
    • fairSeq Facebook AI Research implementations of SOTA seq2seq models in Pytorch.
    • corex_topic - Hierarchical Topic Modeling with Minimal Domain Knowledge
    • Sockeye - Neural Machine Translation (NMT) toolkit that powers Amazon Translate.
    • DL Translate - A deep learning-based translation library for 50 languages, built on transformers and Facebook's mBART Large.
    • Jury - Evaluation of NLP model outputs offering various automated metrics.
    • python-ucto - Unicode-aware regular-expression based tokenizer for various languages. Python binding to C++ library, supports FoLiA format.

Services

NLP as API with higher level functionality such as NER, Topic tagging and so on | Back to Top

  • Wit-ai - Natural Language Interface for apps and devices
  • IBM Watson's Natural Language Understanding - API and Github demo
  • Amazon Comprehend - NLP and ML suite covers most common tasks like NER, tagging, and sentiment analysis
  • Google Cloud Natural Language API - Syntax Analysis, NER, Sentiment Analysis, and Content tagging in atleast 9 languages include English and Chinese (Simplified and Traditional).
  • ParallelDots - High level Text Analysis API Service ranging from Sentiment Analysis to Intent Analysis
  • Microsoft Cognitive Service
  • TextRazor
  • Rosette
  • Textalytic - Natural Language Processing in the Browser with sentiment analysis, named entity extraction, POS tagging, word frequencies, topic modeling, word clouds, and more
  • NLP Cloud - SpaCy NLP models (custom and pre-trained ones) served through a RESTful API for named entity recognition (NER), POS tagging, and more.
  • Cloudmersive - Unified and free NLP APIs that perform actions such as speech tagging, text rephrasing, language translation/detection, and sentence parsing

Annotation Tools

  • GATE - General Architecture and Text Engineering is 15+ years old, free and open source
  • Anafora is free and open source, web-based raw text annotation tool
  • brat - brat rapid annotation tool is an online environment for collaborative text annotation
  • doccano - doccano is free, open-source, and provides annotation features for text classification, sequence labeling and sequence to sequence
  • INCEpTION - A semantic annotation platform offering intelligent assistance and knowledge management
  • tagtog, team-first web tool to find, create, maintain, and share datasets - costs $
  • prodigy is an annotation tool powered by active learning, costs $
  • LightTag - Hosted and managed text annotation tool for teams, costs $
  • rstWeb - open source local or online tool for discourse tree annotations
  • GitDox - open source server annotation tool with GitHub version control and validation for XML data and collaborative spreadsheet grids
  • Label Studio - Hosted and managed text annotation tool for teams, freemium based, costs $
  • Datasaur support various NLP tasks for individual or teams, freemium based
  • Konfuzio - team-first hosted and on-prem text, image and PDF annotation tool powered by active learning, freemium based, costs $
  • UBIAI - Easy-to-use text annotation tool for teams with most comprehensive auto-annotation features. Supports NER, relations and document classification as well as OCR annotation for invoice labeling, costs $
  • Shoonya - Shoonya is free and open source data annotation platform with wide varities of organization and workspace level management system. Shoonya is data agnostic, can be used by teams to annotate data with various level of verification stages at scale.
  • Annotation Lab - Free End-to-End No-Code platform for text annotation and DL model training/tuning. Out-of-the-box support for Named Entity Recognition, Classification, Relation extraction and Assertion Status Spark NLP models. Unlimited support for users, teams, projects, documents. Not FOSS.
  • FLAT - FLAT is a web-based linguistic annotation environment based around the FoLiA format, a rich XML-based format for linguistic annotation. Free and open source.

Techniques

Text Embeddings

Word Embeddings

Sentence and Language Model Based Word Embeddings

Back to Top

Question Answering and Knowledge Extraction

Back to Top

Datasets

Back to Top

  • nlp-datasets great collection of nlp datasets
  • gensim-data - Data repository for pretrained NLP models and NLP corpora.

Multilingual NLP Frameworks

Back to Top

  • UDPipe is a trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files. Primarily written in C++, offers a fast and reliable solution for multilingual NLP processing.
  • NLP-Cube : Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing. New platform, written in Python with Dynet 2.0. Offers standalone (CLI/Python bindings) and server functionality (REST API).
  • UralicNLP is an NLP library mostly for many endangered Uralic languages such as Sami languages, Mordvin languages, Mari languages, Komi languages and so on. Also some non-endangered languages are supported such as Finnish together with non-Uralic languages such as Swedish and Arabic. UralicNLP can do morphological analysis, generation, lemmatization and disambiguation.

NLP in Korean

Back to Top

Libraries

  • KoNLPy - Python package for Korean natural language processing.
  • Mecab (Korean) - C++ library for Korean NLP
  • KoalaNLP - Scala library for Korean Natural Language Processing.
  • KoNLP - R package for Korean Natural language processing

Blogs and Tutorials

Datasets

  • KAIST Corpus - A corpus from the Korea Advanced Institute of Science and Technology in Korean.
  • Naver Sentiment Movie Corpus in Korean
  • Chosun Ilbo archive - dataset in Korean from one of the major newspapers in South Korea, the Chosun Ilbo.
  • Chat data - Chatbot data in Korean
  • Petitions - Collect expired petition data from the Blue House National Petition Site.
  • Korean Parallel corpora - Neural Machine Translation(NMT) Dataset for Korean to French & Korean to English
  • KorQuAD - Korean SQuAD dataset with Wiki HTML source. Mentions both v1.0 and v2.1 at the time of adding to Awesome NLP

NLP in Arabic

Back to Top

Libraries

  • goarabic - Go package for Arabic text processing
  • jsastem - Javascript for Arabic stemming
  • PyArabic - Python libraries for Arabic
  • RFTokenizer - trainable Python segmenter for Arabic, Hebrew and Coptic

Datasets

  • Multidomain Datasets - Largest Available Multi-Domain Resources for Arabic Sentiment Analysis
  • LABR - LArge Arabic Book Reviews dataset
  • Arabic Stopwords - A list of Arabic stopwords from various resources

NLP in Chinese

Back to Top

Libraries

  • jieba - Python package for Words Segmentation Utilities in Chinese
  • SnowNLP - Python package for Chinese NLP
  • FudanNLP - Java library for Chinese text processing
  • HanLP - The multilingual NLP library

Anthology

  • funNLP - Collection of NLP tools and resources mainly for Chinese

NLP in German

  • German-NLP - Curated list of open-access/open-source/off-the-shelf resources and tools developed with a particular focus on German

NLP in Polish

  • Polish-NLP - A curated list of resources dedicated to Natural Language Processing (NLP) in polish. Models, tools, datasets.

NLP in Spanish

Back to Top

Libraries

  • spanlp - Python library to detect, censor and clean profanity, vulgarities, hateful words, racism, xenophobia and bullying in texts written in Spanish. It contains data of 21 Spanish-speaking countries.

Data

Word and Sentence Embeddings

NLP in Indic languages

Back to Top

Data, Corpora and Treebanks

Corpora/Datasets that need a login/access can be gained via email

Language Models and Word Embeddings

Libraries and Tooling

NLP in Thai

Back to Top

Libraries

  • PyThaiNLP - Thai NLP in Python Package
  • JTCC - A character cluster library in Java
  • CutKum - Word segmentation with deep learning in TensorFlow
  • Thai Language Toolkit - Based on a paper by Wirote Aroonmanakun in 2002 with included dataset
  • SynThai - Word segmentation and POS tagging using deep learning in Python

Data

  • Inter-BEST - A text corpus with 5 million words with word segmentation
  • Prime Minister 29 - Dataset containing speeches of the current Prime Minister of Thailand

NLP in Danish

NLP in Vietnamese

Libraries

  • underthesea - Vietnamese NLP Toolkit
  • vn.vitk - A Vietnamese Text Processing Toolkit
  • VnCoreNLP - A Vietnamese natural language processing toolkit
  • PhoBERT - Pre-trained language models for Vietnamese
  • pyvi - Python Vietnamese Core NLP Toolkit

Data

  • Vietnamese treebank - 10,000 sentences for the constituency parsing task
  • BKTreeBank - a Vietnamese Dependency Treebank
  • UD_Vietnamese - Vietnamese Universal Dependency Treebank
  • VIVOS - a free Vietnamese speech corpus consisting of 15 hours of recording speech by AILab
  • VNTQcorpus(big).txt - 1.75 million sentences in news
  • ViText2SQL - A dataset for Vietnamese Text-to-SQL semantic parsing (EMNLP-2020 Findings)
  • EVB Corpus - 20,000,000 words (20 million) from 15 bilingual books, 100 parallel English-Vietnamese / Vietnamese-English texts, 250 parallel law and ordinance texts, 5,000 news articles, and 2,000 film subtitles.

NLP for Dutch

Back to Top

  • python-frog - Python binding to Frog, an NLP suite for Dutch. (pos tagging, lemmatisation, dependency parsing, NER)
  • SimpleNLG_NL - Dutch surface realiser used for Natural Language Generation in Dutch, based on the SimpleNLG implementation for English and French.
  • Alpino - Dependency parser for Dutch (also does PoS tagging and Lemmatisation).
  • Kaldi NL - Dutch Speech Recognition models based on Kaldi.
  • spaCy - Dutch model available. - Industrial strength NLP with Python and Cython.

NLP in Indonesian

Datasets

Libraries & Embedding

NLP in Urdu

Datasets

Libraries

NLP in Persian

Back to Top

Libraries

  • Hazm - Persian NLP Toolkit.
  • Parsivar: A Language Processing Toolkit for Persian
  • Perke: Perke is a Python keyphrase extraction package for Persian language. It provides an end-to-end keyphrase extraction pipeline in which each component can be easily modified or extended to develop new models.
  • Perstem: Persian stemmer, morphological analyzer, transliterator, and partial part-of-speech tagger
  • ParsiAnalyzer: Persian Analyzer For Elasticsearch
  • virastar: Cleaning up Persian text!

Datasets

  • Bijankhan Corpus: Bijankhan corpus is a tagged corpus that is suitable for natural language processing research on the Persian (Farsi) language. This collection is gathered form daily news and common texts. In this collection all documents are categorized into different subjects such as political, cultural and so on. Totally, there are 4300 different subjects. The Bijankhan collection contains about 2.6 millions manually tagged words with a tag set that contains 40 Persian POS tags.
  • Uppsala Persian Corpus (UPC): Uppsala Persian Corpus (UPC) is a large, freely available Persian corpus. The corpus is a modified version of the Bijankhan corpus with additional sentence segmentation and consistent tokenization containing 2,704,028 tokens and annotated with 31 part-of-speech tags. The part-of-speech tags are listed with explanations in this table.
  • Large-Scale Colloquial Persian: Large Scale Colloquial Persian Dataset (LSCP) is hierarchically organized in asemantic taxonomy that focuses on multi-task informal Persian language understanding as a comprehensive problem. LSCP includes 120M sentences from 27M casual Persian tweets with its dependency relations in syntactic annotation, Part-of-speech tags, sentiment polarity and automatic translation of original Persian sentences in English (EN), German (DE), Czech (CS), Italian (IT) and Hindi (HI) spoken languages. Learn more about this project at LSCP webpage.
  • ArmanPersoNERCorpus: The dataset includes 250,015 tokens and 7,682 Persian sentences in total. It is available in 3 folds to be used in turn as training and test sets. Each file contains one token, along with its manually annotated named-entity tag, per line. Each sentence is separated with a newline. The NER tags are in IOB format.
  • FarsiYar PersianNER: The dataset includes about 25,000,000 tokens and about 1,000,000 Persian sentences in total based on Persian Wikipedia Corpus. The NER tags are in IOB format. More than 1000 volunteers contributed tag improvements to this dataset via web panel or android app. They release updated tags every two weeks.
  • PERLEX: The first Persian dataset for relation extraction, which is an expert translated version of the “Semeval-2010-Task-8” dataset. Link to the relevant publication.
  • Persian Syntactic Dependency Treebank: This treebank is supplied for free noncommercial use. For commercial uses feel free to contact us. The number of annotated sentences is 29,982 sentences including samples from almost all verbs of the Persian valency lexicon.
  • Uppsala Persian Dependency Treebank (UPDT): Dependency-based syntactically annotated corpus.
  • Hamshahri: Hamshahri collection is a standard reliable Persian text collection that was used at Cross Language Evaluation Forum (CLEF) during years 2008 and 2009 for evaluation of Persian information retrieval systems.

NLP in Ukrainian

Back to Top

  • awesome-ukrainian-nlp - a curated list of Ukrainian NLP datasets, models, etc.
  • UkrainianLT - another curated list with a focus on machine translation and speech processing

NLP in Hungarian

Back to Top

  • awesome-hungarian-nlp: A curated list of free resources dedicated to Hungarian Natural Language Processing.

NLP in Portuguese

Back to Top

  • Portuguese-nlp - a List of resources and tools developed with focus on Portuguese.

Other Languages

  • Russian: pymorphy2 - a good pos-tagger for Russian
  • Asian Languages: Thai, Lao, Chinese, Japanese, and Korean ICU Tokenizer implementation in ElasticSearch
  • Ancient Languages: CLTK: The Classical Language Toolkit is a Python library and collection of texts for doing NLP in ancient languages
  • Hebrew: NLPH_Resources - A collection of papers, corpora and linguistic resources for NLP in Hebrew

Back to Top

Credits for initial curators and sources

License

License - CC0