DeepPavlov
An open source library for deep learning end-to-end dialog systems and chatbots.
Top Related Projects
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
Bot Framework provides the most comprehensive experience for building conversation applications.
The open-source hub to build & deploy GPT/LLM Agents ⚡️
💫 Industrial-strength Natural Language Processing (NLP) in Python
A framework for training and evaluating AI models on a variety of openly available dialogue datasets.
Quick Overview
DeepPavlov is an open-source conversational AI framework that provides a set of pre-trained models and tools for building and deploying conversational AI applications. It is designed to simplify the process of developing and deploying conversational AI systems, making it easier for developers to create chatbots, virtual assistants, and other conversational interfaces.
Pros
- Extensive Pre-trained Models: DeepPavlov comes with a wide range of pre-trained models for various conversational AI tasks, such as intent recognition, named entity recognition, and dialogue management, which can be easily fine-tuned for specific use cases.
- Modular and Flexible Architecture: The framework's modular design allows developers to easily integrate and customize different components of the conversational AI system, making it highly flexible and adaptable.
- Active Community and Documentation: DeepPavlov has an active community of contributors and users, and the project is well-documented, making it easier for developers to get started and find support.
- Supports Multiple Languages: The framework supports multiple languages, including English, Russian, and Chinese, making it suitable for a wide range of use cases.
Cons
- Steep Learning Curve: While the framework is well-documented, the overall complexity of the system and the number of available components can make it challenging for beginners to get started.
- Limited Deployment Options: DeepPavlov primarily focuses on the development and training of conversational AI models, but it may not provide comprehensive deployment and hosting solutions, which could be a limitation for some users.
- Performance Concerns: Depending on the complexity of the models and the specific use case, the performance of the conversational AI system built with DeepPavlov may vary, and some users may encounter performance issues.
- Dependency on External Libraries: DeepPavlov relies on several external libraries and frameworks, such as TensorFlow and PyTorch, which can introduce additional dependencies and potential compatibility issues.
Code Examples
Here are a few code examples demonstrating the usage of DeepPavlov:
- Intent Recognition:
from deeppavlov import build_model, configs
intent_model = build_model(configs.classifiers.intents_dstc2, download=True)
intent = intent_model(['I would like to book a flight'])
print(intent) # Output: ['inform']
This code demonstrates how to use the pre-trained intent recognition model in DeepPavlov to classify the intent of a user's input.
- Named Entity Recognition:
from deeppavlov import build_model, configs
ner_model = build_model(configs.ner.ner_ontonotes_bert_mult, download=True)
entities = ner_model(['My name is John and I live in New York'])
print(entities) # Output: [('John', 'PER'), ('New York', 'LOC')]
This code shows how to use the pre-trained named entity recognition model in DeepPavlov to extract entities from a given text.
- Dialogue Management:
from deeppavlov import build_model, configs
dialogue_model = build_model(configs.dialog.dialog_system_seq2seq_atten, download=True)
response = dialogue_model(['Hi, how can I help you?', 'I would like to book a flight'])
print(response) # Output: ['Sure, let me help you with that. What are the details of your flight?']
This code demonstrates the usage of the pre-trained dialogue management model in DeepPavlov to generate a response based on the user's input.
Getting Started
To get started with DeepPavlov, follow these steps:
- Install the required dependencies:
pip install deeppavlov
- Download the pre-trained models:
from deeppavlov import download_model
download_model('intents_dstc2')
download_model('ner_ontonotes_bert_mult')
download_model('dialog_system_seq2seq_atten')
- Build and use the models:
from deeppavlov import build_model, configs
intent_model = build_model(configs.classifiers.intents_dstc2)
ner_model = build_model(
Competitor Comparisons
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
Pros of Transformers
- Larger community and more frequent updates
- Broader range of pre-trained models and tasks
- Seamless integration with PyTorch and TensorFlow
Cons of Transformers
- Steeper learning curve for beginners
- Less focus on end-to-end dialogue systems
Code Comparison
DeepPavlov:
from deeppavlov import build_model, configs
model = build_model(configs.squad.squad_bert, download=True)
result = model(["Text", "Question"])
Transformers:
from transformers import pipeline
qa_pipeline = pipeline("question-answering")
result = qa_pipeline(question="Question", context="Text")
Both repositories offer powerful NLP capabilities, but Transformers provides a wider range of models and tasks, while DeepPavlov focuses more on dialogue systems and offers a simpler API for certain tasks. Transformers has a larger community and more frequent updates, but may be more challenging for beginners. DeepPavlov provides an easier entry point for specific dialogue-related tasks but has a narrower scope overall.
Bot Framework provides the most comprehensive experience for building conversation applications.
Pros of Bot Framework SDK
- Comprehensive ecosystem with tools for building, testing, and deploying bots
- Strong integration with Azure services and other Microsoft technologies
- Extensive documentation and community support
Cons of Bot Framework SDK
- Steeper learning curve, especially for developers new to Microsoft technologies
- More complex setup and configuration compared to DeepPavlov
- Primarily focused on chatbots, while DeepPavlov offers a broader range of NLP tasks
Code Comparison
Bot Framework SDK (C#):
[LuisIntent("Greeting")]
public async Task Greeting(IDialogContext context, LuisResult result)
{
await context.PostAsync("Hello! How can I help you today?");
context.Wait(MessageReceived);
}
DeepPavlov (Python):
from deeppavlov import build_model, configs
model = build_model(configs.ner.ner_ontonotes_bert_mult, download=True)
result = model(["Hello, how are you?"])
print(result)
The Bot Framework SDK example shows a simple intent handler for a greeting, while the DeepPavlov example demonstrates how to build and use a pre-trained NER model. DeepPavlov's approach is more straightforward for specific NLP tasks, while Bot Framework SDK provides a more comprehensive structure for building conversational agents.
The open-source hub to build & deploy GPT/LLM Agents ⚡️
Pros of Botpress
- User-friendly visual interface for bot design and management
- Extensive built-in NLU capabilities and integrations
- Active community and regular updates
Cons of Botpress
- Steeper learning curve for developers
- Less flexibility for custom NLP models compared to DeepPavlov
Code Comparison
Botpress (JavaScript):
bp.hear(/hello/i, async (bp, event) => {
await bp.messaging.sendText(event.channel.id, 'Hello, human!')
})
DeepPavlov (Python):
from deeppavlov import build_model, configs
model = build_model(configs.ner.ner_ontonotes_bert_mult, download=True)
result = model(["Hello, how are you?"])
Botpress focuses on a more visual approach to bot creation, with a user-friendly interface and built-in NLU capabilities. It's suitable for both developers and non-technical users. However, it may have a steeper learning curve for developers accustomed to more code-centric approaches.
DeepPavlov, on the other hand, is more flexible for custom NLP model development and integration. It's primarily designed for researchers and developers working on advanced NLP tasks. While it offers powerful tools for building complex conversational AI systems, it may require more technical expertise to implement effectively.
💫 Industrial-strength Natural Language Processing (NLP) in Python
Pros of spaCy
- More mature and widely adopted in industry
- Faster processing speed for large-scale text analysis
- Extensive documentation and community support
Cons of spaCy
- Less flexibility for customizing deep learning models
- Fewer out-of-the-box task-specific models compared to DeepPavlov
- Limited support for languages other than English
Code Comparison
spaCy:
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
for ent in doc.ents:
print(ent.text, ent.label_)
DeepPavlov:
from deeppavlov import build_model, configs
ner_model = build_model(configs.ner.ner_ontonotes_bert_mult, download=True)
results = ner_model(["Apple is looking at buying U.K. startup for $1 billion"])
for entity in results[0]:
print(entity)
Both libraries offer NLP capabilities, but spaCy is more focused on efficiency and industrial applications, while DeepPavlov provides a wider range of pre-built models for various NLP tasks. spaCy's syntax is generally more concise, while DeepPavlov offers more flexibility in model selection and configuration.
A framework for training and evaluating AI models on a variety of openly available dialogue datasets.
Pros of ParlAI
- Extensive library of pre-built tasks and datasets for dialogue research
- Strong integration with Facebook's AI research ecosystem
- Active community and frequent updates
Cons of ParlAI
- Steeper learning curve for beginners
- More focused on research than production deployment
- Heavier resource requirements for some models
Code Comparison
ParlAI:
from parlai.core.agents import Agent
from parlai.core.worlds import DialogPartnerWorld
class MyAgent(Agent):
def act(self):
observation = self.observation
return {'text': 'Hello, ' + observation['text']}
DeepPavlov:
from deeppavlov import build_model, configs
model = build_model(configs.squad.squad_bert, download=True)
result = model(["Text", "Question"])
print(result)
ParlAI offers a more flexible, agent-based approach for dialogue systems, while DeepPavlov provides a simpler API for quick model deployment. ParlAI's code structure is geared towards research and experimentation, whereas DeepPavlov focuses on ease of use for production scenarios.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
DeepPavlov 1.0
DeepPavlov 1.0 is an open-source NLP framework built on PyTorch and transformers. DeepPavlov 1.0 is created for modular and configuration-driven development of state-of-the-art NLP models and supports a wide range of NLP model applications. DeepPavlov 1.0 is designed for practitioners with limited knowledge of NLP/ML.
Quick Links
name | Description |
---|---|
âï¸ Demo | Check out our NLP models in the online demo |
ð Documentation | How to use DeepPavlov 1.0 and its features |
ð Model List | Find the NLP model you need in the list of available models |
ðª Contribution Guide | Please read the contribution guidelines before making a contribution |
ð Issues | If you have an issue with DeepPavlov, please let us know |
â© Forum | Please let us know if you have a problem with DeepPavlov |
ð¦ Blogs | Read about our current development |
ð¦ Extended colab tutorials | Check out the code tutorials for our models |
ð Docker Hub | Check out the Docker images for rapid deployment |
ð©âð« Feedback | Please leave us your feedback to make DeepPavlov better |
Installation
-
DeepPavlov supports
Linux
,Windows 10+
(through WSL/WSL2),MacOS
(Big Sur+) platforms,Python 3.6
,3.7
,3.8
,3.9
and3.10
. Depending on the model used, you may need from 4 to 16 GB RAM. -
Create and activate a virtual environment:
Linux
python -m venv env source ./env/bin/activate
-
Install the package inside the environment:
pip install deeppavlov
QuickStart
There is a bunch of great pre-trained NLP models in DeepPavlov. Each model is determined by its config file.
List of models is available on
the doc page in
the deeppavlov.configs
(Python):
from deeppavlov import configs
When you're decided on the model (+ config file), there are two ways to train, evaluate and infer it:
- via Command line interface (CLI) and
- via Python.
GPU requirements
By default, DeepPavlov installs models requirements from PyPI. PyTorch from PyPI could not support your device CUDA capability. To run supported DeepPavlov models on GPU you should have CUDA compatible with used GPU and PyTorch version required by DeepPavlov models. See docs for details. GPU with Pascal or newer architecture and 4+ GB VRAM is recommended.
Command line interface (CLI)
To get predictions from a model interactively through CLI, run
python -m deeppavlov interact <config_path> [-d] [-i]
-d
downloads required data - pretrained model files and embeddings (optional).-i
installs model requirements (optional).
You can train it in the same simple way:
python -m deeppavlov train <config_path> [-d] [-i]
Dataset will be downloaded regardless of whether there was -d
flag or not.
To train on your own data you need to modify dataset reader path in the train config doc. The data format is specified in the corresponding model doc page.
There are even more actions you can perform with configs:
python -m deeppavlov <action> <config_path> [-d] [-i]
<action>
can beinstall
to install model requirements (same as-i
),download
to download model's data (same as-d
),train
to train the model on the data specified in the config file,evaluate
to calculate metrics on the same dataset,interact
to interact via CLI,riseapi
to run a REST API server (see doc),predict
to get prediction for samples from stdin or from <file_path> if-f <file_path>
is specified.
<config_path>
specifies path (or name) of model's config file-d
downloads required data-i
installs model requirements
Python
To get predictions from a model interactively through Python, run
from deeppavlov import build_model
model = build_model(<config_path>, install=True, download=True)
# get predictions for 'input_text1', 'input_text2'
model(['input_text1', 'input_text2'])
where
install=True
installs model requirements (optional),download=True
downloads required data from web - pretrained model files and embeddings (optional),<config_path>
is model name (e.g.'ner_ontonotes_bert_mult'
), path to the chosen model's config file (e.g."deeppavlov/configs/ner/ner_ontonotes_bert_mult.json"
), ordeeppavlov.configs
attribute (e.g.deeppavlov.configs.ner.ner_ontonotes_bert_mult
without quotation marks).
You can train it in the same simple way:
from deeppavlov import train_model
model = train_model(<config_path>, install=True, download=True)
To train on your own data you need to modify dataset reader path in the train config doc. The data format is specified in the corresponding model doc page.
You can also calculate metrics on the dataset specified in your config file:
from deeppavlov import evaluate_model
model = evaluate_model(<config_path>, install=True, download=True)
DeepPavlov also allows to build a model from components for inference using Python.
License
DeepPavlov is Apache 2.0 - licensed.
Top Related Projects
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
Bot Framework provides the most comprehensive experience for building conversation applications.
The open-source hub to build & deploy GPT/LLM Agents ⚡️
💫 Industrial-strength Natural Language Processing (NLP) in Python
A framework for training and evaluating AI models on a variety of openly available dialogue datasets.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot