jupytext

Jupyter Notebooks as Markdown Documents, Julia, Python or R scripts

6,580

386

6,580

135

View on GitHub

Top Related Projects

papermill

5,778

📚 Parameterize, execute, and analyze notebooks

nbdev

4,882

Create delightful software with Jupyter Notebooks

ipython

16,244

Official repository for IPython itself. Other repos in the IPython organization contain things like the website, documentation builds, etc.

Quick Overview

Jupytext is a Python package that provides two-way conversion between Jupyter notebooks and various text-based formats such as Markdown, R Markdown, and Python scripts. It allows users to edit, version control, and collaborate on Jupyter notebooks using their preferred text editor or IDE, while still being able to open and run them in Jupyter.

Pros

Enables version control of Jupyter notebooks using Git or other VCS
Allows editing notebooks in text editors or IDEs, improving collaboration
Supports multiple text formats, including Markdown and various programming languages
Integrates seamlessly with Jupyter Lab and Jupyter Notebook

Cons

May require additional setup and configuration for some workflows
Can introduce complexity when dealing with mixed content (e.g., code and markdown)
Potential for sync issues if not used carefully across different environments
Learning curve for users unfamiliar with text-based notebook formats

Code Examples

Converting a Jupyter notebook to a Python script:

import jupytext

# Convert notebook to Python script
jupytext.write(notebook, "notebook.py")

Pairing a Jupyter notebook with a Markdown file:

import jupytext

# Pair notebook with Markdown file
notebook = jupytext.read("notebook.ipynb")
jupytext.write(notebook, "notebook.md")

Reading a Python script as a Jupyter notebook:

import jupytext

# Read Python script as notebook
notebook = jupytext.read("script.py")
jupytext.write(notebook, "notebook.ipynb")

Getting Started

To get started with Jupytext, follow these steps:

Install Jupytext:

pip install jupytext

Configure Jupytext in Jupyter (optional):
- Create or edit ~/.jupyter/jupyter_notebook_config.py
- Add the following line:
```
c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
```
Use Jupytext from the command line:

jupytext --to py notebook.ipynb  # Convert to Python script
jupytext --to md notebook.ipynb  # Convert to Markdown

Or use Jupytext in Python scripts:

import jupytext

notebook = jupytext.read("notebook.ipynb")
jupytext.write(notebook, "notebook.md")

Competitor Comparisons

papermill

5,778

📚 Parameterize, execute, and analyze notebooks

Pros of Papermill

Focuses on parameterizing and executing notebooks
Supports multiple languages and kernel types
Integrates well with workflow orchestration tools

Cons of Papermill

Limited version control and collaboration features
Doesn't provide seamless conversion between notebook and script formats
May require additional setup for complex parameterization

Code Comparison

Papermill execution:

import papermill as pm

pm.execute_notebook(
    'input.ipynb',
    'output.ipynb',
    parameters=dict(alpha=0.6, ratio=0.1)
)

Jupytext conversion:

import jupytext

notebook = jupytext.read("notebook.ipynb")
jupytext.write(notebook, "script.py")

Key Differences

Jupytext focuses on version control and format conversion, while Papermill emphasizes notebook execution and parameterization
Jupytext provides bidirectional conversion between notebooks and various script formats, whereas Papermill primarily works with notebook files
Papermill is better suited for production workflows and batch processing, while Jupytext excels in development and collaboration scenarios

Both tools serve different primary purposes but can be complementary in a data science workflow, with Jupytext handling version control and Papermill managing execution and parameterization.

nbdev

4,882

Create delightful software with Jupyter Notebooks

Pros of nbdev

Integrated development environment for creating Python libraries
Supports literate programming with automatic documentation generation
Includes tools for testing, continuous integration, and packaging

Cons of nbdev

Steeper learning curve due to its comprehensive feature set
More opinionated workflow that may not suit all development styles
Requires adherence to specific conventions for optimal use

Code comparison

nbdev:

# %nbdev_export
def my_function(x):
    """
    This function does something.
    
    Args:
        x: An input value
    
    Returns:
        The processed value
    """
    return x * 2

Jupytext:

# %%
def my_function(x):
    """
    This function does something.
    
    Args:
        x: An input value
    
    Returns:
        The processed value
    """
    return x * 2

Key differences

nbdev focuses on creating complete Python libraries, while Jupytext primarily handles notebook-to-script conversion
Jupytext offers more flexibility in terms of supported formats and conversion options
nbdev provides a more comprehensive development ecosystem, including documentation and testing tools
Jupytext is generally easier to integrate into existing workflows without significant changes

nbconvert

1,714

Jupyter Notebook Conversion

Pros of nbconvert

Officially supported by the Jupyter project
Wider range of output formats (HTML, LaTeX, PDF, etc.)
More extensive documentation and community support

Cons of nbconvert

Less flexible for version control of notebooks
Primarily focused on converting notebooks to other formats, not bidirectional conversion
More complex setup and usage for simple conversions

Code comparison

nbconvert:

jupyter nbconvert --to python notebook.ipynb

Jupytext:

jupytext --to py notebook.ipynb

Additional notes

Jupytext offers seamless bidirectional conversion between notebooks and various text formats, making it ideal for version control and collaboration. It integrates well with Jupyter environments and allows for simultaneous editing of both notebook and text representations.

nbconvert, on the other hand, excels in producing polished output formats for presentation or publication. It's more suitable for final document generation rather than day-to-day notebook management.

Both tools have their strengths, and the choice between them depends on specific use cases and workflow requirements.

ipython

16,244

Official repository for IPython itself. Other repos in the IPython organization contain things like the website, documentation builds, etc.

Pros of IPython

More comprehensive interactive computing environment
Broader feature set including shell capabilities and magic commands
Larger community and ecosystem of extensions

Cons of IPython

Heavier and more complex system
Less focused on version control and collaboration for notebooks

Code Comparison

IPython:

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot([1, 2, 3, 4])
plt.show()

Jupytext:

# %% [markdown]
# # My Notebook
# %% [python]
import matplotlib.pyplot as plt
plt.plot([1, 2, 3, 4])
plt.show()

Key Differences

IPython provides a full interactive computing environment, while Jupytext focuses on converting notebooks to plain text formats
Jupytext enables better version control and collaboration for notebooks by converting them to scripts
IPython offers more built-in features for data exploration and visualization
Jupytext allows for easier integration with traditional development workflows and tools

Use Cases

IPython: Interactive data analysis, prototyping, and exploration
Jupytext: Version control of notebooks, integrating notebooks into software development processes

Both tools can be complementary, with Jupytext enhancing IPython/Jupyter notebooks' version control capabilities.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Jupytext

Have you always wished Jupyter notebooks were plain text documents? Wished you could edit them in your favorite IDE? And get clear and meaningful diffs when doing version control? Then, Jupytext may well be the tool you're looking for!

Text Notebooks

A Python notebook encoded in the py:percent format has a .py extension and looks like this:

# %% [markdown]
# This is a markdown cell

# %%
def f(x):
  return 3*x+1

Only the notebook inputs (and optionally, the metadata) are included. Text notebooks are well suited for version control. You can also edit or refactor them in an IDE - the .py notebook above is a regular Python file.

We recommend the percent format for notebooks that mostly contain code. The percent format is available for Julia, Python, R and many other languages.

If your notebook is documentation-oriented, a Markdown-based format (text notebooks with a .md extension) might be more appropriate. Depending on what you plan to do with your notebook, you might prefer the Myst Markdown format, which interoperates very well with Jupyter Book, or Quarto Markdown, or even Pandoc Markdown.

Installation

Install Jupytext in the Python environment that you use for Jupyter. Use either

pip install jupytext

conda install jupytext -c conda-forge

Then, restart your Jupyter Lab server, and make sure Jupytext is activated in Jupyter: .py and .md files have a Notebook icon, and you can open them as Notebooks with a right click in Jupyter Lab.

Paired Notebooks

Text notebooks with a .py or .md extension are well suited for version control. They can be edited or authored conveniently in an IDE. You can open and run them as notebooks in Jupyter Lab with a right click. However, the notebook outputs are lost when the notebook is closed, as only the notebook inputs are saved in text notebooks.

A convenient alternative to text notebooks are paired notebooks. These are a set of two files, say .ipynb and .py, that contain the same notebook, but in different formats.

You can edit the .py version of the paired notebook, and get the edits back in Jupyter by selecting reload notebook from disk. The outputs will be reloaded from the .ipynb file, if it exists. The .ipynb version will be updated or recreated the next time you save the notebook in Jupyter.

To pair a notebook in Jupyter Lab, use the command Pair Notebook with percent Script from the Command Palette:

To pair all the notebooks in a certain directory, create a configuration file with this content:

# jupytext.toml at the root of your notebook directory
formats = "ipynb,py:percent"

Command line

Jupytext is also available at the command line. You can

pair a notebook with jupytext --set-formats ipynb,py:percent notebook.ipynb
synchronize the paired files with jupytext --sync notebook.py (the inputs are loaded from the most recent paired file)
convert a notebook in one format to another with jupytext --to ipynb notebook.py (use -o if you want a specific output file)
pipe a notebook to a linter with e.g. jupytext --pipe black notebook.ipynb

Sample use cases

Notebooks under version control

This is a quick how-to:

Open your .ipynb notebook in Jupyter and pair it to a .py notebook, using either the pair command in Jupyter Lab, or a global configuration file
Save the notebook - this creates a .py notebook
Add this .py notebook to version control

You might exclude .ipynb files from version control (unless you want to see the outputs versioned!). Jupytext will recreate the .ipynb files locally when the users open and save the .py notebooks.

Collaborating on notebooks with Git

Collaborating on Jupyter notebooks through Git becomes as easy as collaborating on text files.

Assume that you have your .py notebooks under version control (see above). Then,

Your collaborator pulls the .py notebook
They open it as a notebook in Jupyter (right-click in Jupyter Lab)
At that stage the notebook has no outputs. They run the notebook and save it. Outputs are regenerated, and a local .ipynb file is created
They edit the notebook, and push the updated notebook.py file. The diff is nothing else than a standard diff on a Python script.
You pull the updated notebook.py script, and refresh your browser. The input cells are updated based on the new content of notebook.py. The outputs are reloaded from your local .ipynb file. Finally, the kernel variables are untouched, so you have the option to run only the modified cells to get the new outputs.

Editing or refactoring a notebook in an IDE

Once your notebook is paired with a .py file, you can easily edit or refactor the .py representation of the notebook in an IDE.

Once you are done editing the .py notebook, you will just have to reload the notebook in Jupyter to get the latest edits there.

Note: It is simpler to close the .ipynb notebook in Jupyter when you edit the paired .py file. There is no obligation to do so; however, if you don't, you should be prepared to read carefully the pop-up messages. If Jupyter tries to save the notebook while the paired .py file has also been edited on disk since the last reload, a conflict will be detected and you will be asked to decide which version of the notebook (in memory or on disk) is the appropriate one.

More resources

Read more about Jupytext in the documentation.

If you're new to Jupytext, you may want to start with the FAQ or with the Tutorials.

There is also this short introduction to Jupytext: .

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot