464

I'm looking at using the *.ipynb files as the source of truth and programmatically 'compiling' them into .py files for scheduled jobs/tasks.

The only way I understand to do this is via the GUI. Is there a way to do it via command line?

2
  • 1
    What do you mean by "source of truth"? IPython notebooks are just json files. You can load them and manipulate as Python dictionaries. For source code you should be iterating input keys where cell_type equals 'code'. Have a look at this scheme Commented Jun 13, 2013 at 1:03
  • 3
    Well I want to store the .ipynb in a repository and not the .py files. So then as a 'build step' I would convert the .ipynb to .py files for actual use by the automated system. You're right, I could just load the json and output only the code cells, but I was wondering whether there was something out there already that did that for me :) Commented Jun 13, 2013 at 1:17

22 Answers 22

669

If you don't want to output a Python script every time you save, or you don't want to restart the IPython kernel:

On the command line, you can use nbconvert:

$ jupyter nbconvert --to script [YOUR_NOTEBOOK].ipynb

As a bit of a hack, you can even call the above command in an IPython notebook by pre-pending ! (used for any command line argument). Inside a notebook:

!jupyter nbconvert --to script config_template.ipynb

Before --to script was added, the option was --to python or --to=python, but it was renamed in the move toward a language-agnostic notebook system.

Sign up to request clarification or add additional context in comments.

18 Comments

If you DO want one every save, in jupyter you can trigger nbconvert via pre- or post-save hooks: ContentsManager.pre_save_hook abd FileContentsManager.post_save_hook. You would add a post-save hook jupyter nbconvert --to script [notebook]
Is there a way to do the reverse i.e convert from a python script to a notebook. For ex - having some specialized docstrings that are parsed into cells ?
convert all notebooks in a folder jupyter nbconvert --to script /path/to/notebooks/*.ipynb
Thanks, it works!, but what if I don't want the # In[ ]: type stuff in the script, I want it to be clean. Is there any way to do that?
@RishabhAgrahari Following seems to work for me. !jupyter nbconvert --to script --no-prompt notebook.ipynb
|
159

If you want to convert all *.ipynb files from current directory to python script, you can run the command like this:

jupyter nbconvert --to script *.ipynb

2 Comments

Or you can run: ipython nbconvert --to script *.ipynb
to convert programmatically as in a function use this syntax : subprocess.run(['jupyter', 'nbconvert',ipynb_file,"--to", "script","--output",r'temp_converted_py'],shell=True) note that ipynb_file is a variable
29

Here is a quick and dirty way to extract the code from V3 or V4 ipynb without using ipython. It does not check cell types, etc.

import sys,json

f = open(sys.argv[1], 'r') #input.ipynb
j = json.load(f)
of = open(sys.argv[2], 'w') #output.py
if j["nbformat"] >=4:
        for i,cell in enumerate(j["cells"]):
                of.write("#cell "+str(i)+"\n")
                for line in cell["source"]:
                        of.write(line)
                of.write('\n\n')
else:
        for i,cell in enumerate(j["worksheets"][0]["cells"]):
                of.write("#cell "+str(i)+"\n")
                for line in cell["input"]:
                        of.write(line)
                of.write('\n\n')

of.close()

3 Comments

Best answer if you do not want to install any of the Jupyter tools.
I like this. But I found out when I download .py format from Jupyter notebooks, it uses UNIX line endings even though I'm on windows. To generate the same, add the newlines='\n' as a third argument in the open output file call. (Python 3.x)
This answer opens the possibility to read tags and extract only those cells that have it. Harder to do via the command line alone.
25

Jupytext is nice to have in your toolchain for such conversions. It allows not only conversion from a notebook to a script, but you can go back again from the script to notebook as well. And even have that notebook produced in executed form.

jupytext --to py notebook.ipynb                 # convert notebook.ipynb to a .py file
jupytext --to notebook notebook.py              # convert notebook.py to an .ipynb file with no outputs
jupytext --to notebook --execute notebook.py    # convert notebook.py to an .ipynb file and run it 

3 Comments

Apparently there is also ipynb-py-convert, see here.
'jupytext' is not recognized as an internal or external command, operable program or batch file.???
Have you installed it @AmineChadi . See here for how to do that. If you are using it via a notebook as your command line interface, you can just run %pip install jupytext in your notebook.
22

Following the previous example but with the new nbformat lib version :

import nbformat
from nbconvert import PythonExporter

def convertNotebook(notebookPath, modulePath):

  with open(notebookPath) as fh:
    nb = nbformat.reads(fh.read(), nbformat.NO_CONVERT)

  exporter = PythonExporter()
  source, meta = exporter.from_notebook_node(nb)

  with open(modulePath, 'w+') as fh:
    fh.writelines(source.encode('utf-8'))

4 Comments

's last line of code, fh.writelines(source.encode('utf-8')) gives 'TypeError: write() argument must be str, not int' fh.writelines(source) works though.
Had the same issue, fixed by dropping the .encode('utf-8') piece on the last line.
I solved the last line problem writing fh.write(source) instead of fh.writelines(...).
How do we understand when we need to use 'utf-8'?
12

I understand this is an old thread. I have faced the same issue and wanted to convert the .pynb file to .py file via command line.

My search took me to ipynb-py-convert

By following below steps I was able to get .py file

  1. Install "pip install ipynb-py-convert"
  2. Go to the directory where the ipynb file is saved via command prompt
  3. Enter the command

> ipynb-py-convert YourFileName.ipynb YourFilename.py

Eg:. ipynb-py-convert getting-started-with-kaggle-titanic-problem.ipynb getting-started-with-kaggle-titanic-problem.py

Above command will create a python script with the name "YourFileName.py" and as per our example it will create getting-started-with-kaggle-titanic-problem.py file

1 Comment

I updated to notebook 7 few days back and was surprised not to find the download as .py option. Hard to understand why this default option is removed to convert notebook to a .py file, it was so handy. After spending good amount of time, found that only this option is working for me to convert a notebook to a .py file.
9

You can do this from the IPython API.

from IPython.nbformat import current as nbformat
from IPython.nbconvert import PythonExporter

filepath = 'path/to/my_notebook.ipynb'
export_path = 'path/to/my_notebook.py'

with open(filepath) as fh:
    nb = nbformat.reads_json(fh.read())

exporter = PythonExporter()

# source is a tuple of python source code
# meta contains metadata
source, meta = exporter.from_notebook_node(nb)

with open(export_path, 'w+') as fh:
    fh.writelines(source)

Comments

7

Using nbconvert 6.07 and jupyter client 6.1.12:

Convert jupyter notebook to python script

$ jupyter nbconvert mynotebook.ipynb --to python

Convert jupyter notebook to python script specifying output filename

$ jupyter nbconvert mynotebook.ipnb --to python --output myscript.py

Comments

4

For converting all *.ipynb format files in current directory to python scripts recursively:

for i in *.ipynb **/*.ipynb; do 
    echo "$i"
    jupyter nbconvert  "$i" "$i"
done

1 Comment

I had to add the --to script argument to avoid the default HTML output in Jupiter 4.4.0.
4

Common unflexible way

Most would suggest the nbconvert method, but the output is also kinda ugly containing tons of comments:

jupyter nbconvert --to script [YOUR_NOTEBOOK].ipynb

Cell level extractions

As notebooks are basically JSON, then you can write your own module extracting specific cells of your notebooks using tags like "###EXTRACT###" as the first line of the cells you want to extract.

Here is how to use the module I wrote:

# Importing Custom Module
from ipynb_exporter import NotebookModuleBuilder as nmb

# Exporting labeled cells
nmb().ipynb_to_file(ipynb_path="mynotebook.ipynb",
                    label="###EXTRACT###",
                    py_path="mynotebookcells.py")

Here is the module:

import json

class NotebookModuleBuilder():
    """ Class helps you extract code cells from ipynb-files by using tags """
    
    @staticmethod
    def _read_json(path: str) -> dict:
        """ Reads a json-file and returns a dictionary
            
        Args:
            path: Path to jupyter notebook (.ipynb)
            
        Returns:
            dictionary representation of notebook
        """
        file = open(path, mode= "r", encoding= "utf-8")
        myfile = file.read()
        myjson = json.loads(myfile)
        file.close()
        return myjson

    @staticmethod
    def _get_code_cells(dictionary: dict) -> list:
        """ Finds cells of ipynb with code
        
        Args:
            dictionary: Dictionary from importing a ipynb notebook
            
        Returns:
            List of code cells
        """
        code_cells = [cell for cell in dictionary['cells'] if cell['cell_type']=='code']
        return code_cells

    @staticmethod
    def _get_labeled_cells(code_cells: dict, label="###EXPORT###") -> dict:
        """ Gets cells with the specified label
        
        Args:
            code_cells: Dictionary with code cells
            
        Returns:
            Dictionary with labeled cells
        """
        label = label + "\n"
        sourced_cells = [cell for cell in code_cells if len(cell['source']) > 0]
        labeled_cells = [cell['source'] for cell in sourced_cells if cell['source'][0]==label]
        return labeled_cells

    @staticmethod
    def _write_to_file(labeled_cells: dict, output_file: str) -> None:
        """ Writes the labeled cells to a file
        
        Args:
            labeled_cells: Dictionary with cells that should be written to a file
        """
        flattened_lists = '\n\n'.join([''.join(labeled_cell[1:]) for labeled_cell in labeled_cells])
        file = open(output_file, 'w')
        file.write(flattened_lists)
        file.close()
    
    def ipynb_to_file(self, ipynb_path: str, py_path: str, label: str = '###EXTRACT###') -> None:
        """ Writes cells labeled with ###EXTRACT### in ipynb into a py-file
        
        Args:
            label: Lable that in first line of a cell to match
            ipynb_path: Input path to ipynb-notebook
            py_path: Output path to py-file
        """
        json_file = self._read_json(ipynb_path)
        code_cells = self._get_code_cells(json_file)
        labeled_cells = self._get_labeled_cells(code_cells,label)
        self._write_to_file(labeled_cells, py_path)

Comments

3

The following example turns an Iron Python Notebook called a_notebook.ipynb into a python script called a_python_script.py leaving out the cells tagged with the keyword remove, which I add manually to the cells that I don't want to end up in the script, leaving out visualizations and other steps that once I am done with the notebook I don't need to be executed by the script.

import nbformat as nbf
from nbconvert.exporters import PythonExporter
from nbconvert.preprocessors import TagRemovePreprocessor

with open("a_notebook.ipynb", 'r', encoding='utf-8') as f:
    the_notebook_nodes = nbf.read(f, as_version = 4)

trp = TagRemovePreprocessor()

trp.remove_cell_tags = ("remove",)

pexp = PythonExporter()

pexp.register_preprocessor(trp, enabled= True)

the_python_script, meta = pexp.from_notebook_node(the_notebook_nodes)

with open("a_python_script.py", 'w', encoding='utf-8') as f:
    f.writelines(the_python_script)

Comments

3

no file/directory error

On my mint [ubuntu] system at work, even though jupyter was already installed and notebooks worked, jupyter nbconvert --to script gave the error no file/directory until I did a separate

sudo apt-get install jupyter-nbconvert

Then all was fine with the conversion. I just wanted to add this in case anyone hits the same error (for me it was confusing as I thought the no file error referred to the notebook, which was definitely there in the local directory, took me a while to realize the subcommand was not installed).

Comments

3

I found that there are 2 ways to convert Jupyter Notebook to plain Python script with command line. Below is the example Jupyter notebook and output from both tools.

enter image description here

1. Use nbconvert

The nbconvert is tool that is used in Jupyter Notebook User Interface in Download as functionality. It can be used as command line tool:

jupyter nbconvert --to python notebook.ipynb

Example Python script: enter image description here

2. Use jupytext

The jupytext is a package to keep .ipynb file synchronized with .py file. It can be also used to convert .ipynb files in command line. It supports several types of conversion:

Convert to Python script with light format

jupytext --to py notebook.ipynb             

Example python script:

enter image description here

Convert to Python script with percent format

jupytext --to py:percent notebook.ipynb 

Example Python script:

enter image description here

Comments

3

You can convert all Notebook files to python files by doing this:

pip install jupytext
jupytext --to py *.ipynb 

Comments

2

There's a very nice package called nb_dev which is designed for authoring Python packages in Jupyter Notebooks. Like nbconvert, it can turn a notebook into a .py file, but it is more flexible and powerful because it has a lot of nice additional authoring features to help you develop tests, documentation, and register packages on PyPI. It was developed by the fast.ai folks.

It has a bit of a learning curve, but the documentation is good and it is not difficult overall.

Comments

2

Here is a jq solution that could be situationally useful. Remember, notebooks are just json.

jq -r '.cells[] | select(.cell_type  == "code") | .source[] | rtrimstr("\n")' $filename

Comments

1

The given solution works only for converting a single .py file. Here is a solution that converts all .py files in a directory and sub-directories.

First, you need to install the tool that convert only one file at a time like ipynb-py-convert

pip install ipynb-py-convert

Then cd into the place of your folder having the .py files and directories. Then we run the tool recursively on all files in the directory and subdirectories:

In powershell:

foreach ($f in Get-ChildItem "." -Filter *.ipynb -Recurse){ ipynb-py-convert $f.FullName "$($f.FullName.Substring(0,$f.FullName.Length-6)).py"}

Now if you would like to convert the otherway around from .ipynb to .py also with batch convert, you can run:

foreach ($f in Get-ChildItem "." -Filter *.py -Recurse){ ipynb-py-convert $f.FullName "$($f.FullName.Substring(0,$f.FullName.Length-3)).ipynb"}

This helped me a lot while exploring .py files. I make a copy of the project, run this code and quickly in Jupiter test different parts of the code as cells and so on. I hope it helps more people.

Comments

1
jupyter nbconvert main.ipynb --to python

Comments

1

The following answer worked for me (based on Yaach's answer from October 25) but mine needed to convert several files at the same time:

  1. Launch Anaconda prompt
  2. CD to your directory which has several ipnyb files, for example, C:\myfolder
  3. Run: jupyter nbconvert coding-challenge-*.ipynb --to python

Note the wild card character. The result was several coding_challeng-*.py files in the same folder.

Comments

0

I have built a function to achieve this. The user does not have to install anything to use it.

#!/usr/bin/python


# A short routine to convert a Jupyter Notebook to a Python file

import json

def ipynb_to_py(input_ipynb_file,output_py_file=None):
    """
    Generate a Python script (.py) that includes all source code from the input Jupyter notebook (.ipynb).
    
    The user can input a Jupyter Notebook file from the current working directory or from a path.
    
    If the name for output Python file is not specified, 
    the output file name will copy the file name of the input Jupyter Notebook, 
    but the file exention will be changed from ".ipynb" chanegd to ".py".
    And the output Python file will be saved at the same directory of the input Jupyter Notebook.
    For example:
    ipynb_to_py("test-jupyternotebook.ipynb")
    ipynb_to_py("./test-input-dir/test-jupyternotebook.ipynb")
    
    The user can also specify an output file name that ends with ".py".
    If the output file name is provided, but no path to output file is added, 
    the file will be saved at the current working directory.
    For example:
    ipynb_to_py("test-jupyternotebook.ipynb","test1.py")
    ipynb_to_py("./test-input-dir/test-jupyternotebook.ipynb","test2.py")
        
    The user can save out the file at a target directory by adding a path to the output file.
    For example: 
    ipynb_to_py("test-jupyternotebook.ipynb","./test-outputdir/test3.py")
    ipynb_to_py("./test-input-dir/test-jupyternotebook.ipynb","./test-output-dir/test4.py")
    
    This function does not edit or delete the original input Jupyter Notebook file.
    
    Args:
    -----
        input_ipynb_file: The file name string for the Jupyter Notebook (ends with ".ipynb")
        output_py_file (optional): The file name for Python file to be created (ends with ".py"). 
    
    Returns:
    --------
        A Python file containing all source code in the Jupyter Notebook.
        
    Example usages:
    ---------------
        ipynb_to_py("test-jupyternotebook.ipynb")
        ipynb_to_py("./test-input-dir/test-jupyternotebook.ipynb")
        ipynb_to_py("test-jupyternotebook.ipynb","test1.py")
        ipynb_to_py("test-jupyternotebook.ipynb","./test-outputdir/test2.py")
        ipynb_to_py("test-jupyternotebook.ipynb","./test-outputdir/test3.py")
        ipynb_to_py("./test-input-dir/test-jupyternotebook.ipynb","./test-output-dir/test4.py")
             
    """
    # Check if the input file is a Jupyter Notebook
    if input_ipynb_file.endswith(".ipynb"):
        
        # Open the input Jupyter Notebook file
        notebook = open(input_ipynb_file)
        
        # Read its content in the json format
        notebook_content = json.load(notebook)

        # Only extract the source code snippet from each cell in the input Jupyter Notebook
        source_code_snippets = [cell['source'] for cell in notebook_content['cells']]
        
        # If the name for output Python file is not specified,
        # The name of input Jupyter Notebook will be used after changing ".ipynb" to ".py".
        if output_py_file == None:
            output_py_file = input_ipynb_file.split('.ipynb')[0]+".py"
        else:
            pass

        # Create a Python script to save out all the extracted source code snippets
        output_file = open(output_py_file,'w')

        # Print out each line in each source code snippet to the output file
        for snippet in source_code_snippets:
            for line in snippet:
                # Use end='' to avoid creating unwanted gaps between lines
                print(line,end = '',file = output_file)
            # At end of each snippet, move to the next line before printing the next one
            print('',sep = '\n',file=output_file)

        # Close the output file
        output_file.close()
        print("The path to output file:",output_py_file)
        
    else:
        print("The input file must be a Jupyter Notebook (in .ipynb format)!")
        
def main():
    pass

if __name__ == "__main__":
    main()

Comments

-1

I had this problem and tried to find the solution online. Though I found some solutions, they still have some problems, e.g., the annoying Untitled.txt auto-creation when you start a new notebook from the dashboard.

So eventually I wrote my own solution:

import io
import os
import re
from nbconvert.exporters.script import ScriptExporter
from notebook.utils import to_api_path


def script_post_save(model, os_path, contents_manager, **kwargs):
    """Save a copy of notebook to the corresponding language source script.

    For example, when you save a `foo.ipynb` file, a corresponding `foo.py`
    python script will also be saved in the same directory.

    However, existing config files I found online (including the one written in
    the official documentation), will also create an `Untitile.txt` file when
    you create a new notebook, even if you have not pressed the "save" button.
    This is annoying because we usually will rename the notebook with a more
    meaningful name later, and now we have to rename the generated script file,
    too!

    Therefore we make a change here to filter out the newly created notebooks
    by checking their names. For a notebook which has not been given a name,
    i.e., its name is `Untitled.*`, the corresponding source script will not be
    saved. Note that the behavior also applies even if you manually save an
    "Untitled" notebook. The rationale is that we usually do not want to save
    scripts with the useless "Untitled" names.
    """
    # only process for notebooks
    if model["type"] != "notebook":
        return

    script_exporter = ScriptExporter(parent=contents_manager)
    base, __ = os.path.splitext(os_path)

    # do nothing if the notebook name ends with `Untitled[0-9]*`
    regex = re.compile(r"Untitled[0-9]*$")
    if regex.search(base):
        return

    script, resources = script_exporter.from_filename(os_path)
    script_fname = base + resources.get('output_extension', '.txt')

    log = contents_manager.log
    log.info("Saving script at /%s",
             to_api_path(script_fname, contents_manager.root_dir))

    with io.open(script_fname, "w", encoding="utf-8") as f:
        f.write(script)

c.FileContentsManager.post_save_hook = script_post_save

To use this script, you can add it to ~/.jupyter/jupyter_notebook_config.py :)

Note that you may need to restart the jupyter notebook / lab for it to work.

1 Comment

The link is broken
-2

The %notebook foo.ipynb magic command will export the current IPython to "foo.ipynb".

More info by typing %notebook?

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.