How to avoid Jupyter cell-ids from changing all the time and thereby spamming the VCS diffs?

user_745903 0
Viewed 763

As discussed in q/66678305, newer Jupyter versions store in addition to the source code and output of cells an ID for the purpose of e.g. linking to a cell.

However, these IDs aren't stable but often change even when the cell's source code was not touched. As a result, if you have the .ipynb file under version control with e.g. git, the commits end up having lots of rather funny sounding “changed lines” that don't correspond to any actual change made in the commit. Like,

   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "respected-breach",
+   "id": "incident-winning",
    "metadata": {},
    "outputs": [],

Is there a way to prevent this?

jupyter-notebook ipython
1 Answers

Answer for Git on Linux. Probably also works on MacOS, but not Windows.

It is good practice to not VCS the .ipynb files as saved by Jupyter, but instead a filtered version that does not contain all the volatile information. For this purpose, various git hooks are available; the one I'm using is based on https://github.com/toobaz/ipynb_output_filter/blob/master/ipynb_output_filter.py.

Strangely enough, it turns out this script can not be modified to remove the "id" field from cells. Namely, if you try to remove that field in the filtering loop, like with

        for field in ("prompt_number", "execution_number", "id"):
            if field in cell:
                del cell[field]

then the write function from jupyter_nbformat will just put an id back in. It is possible to merely change the id to something constant, but then Jupyter will complain about nonunique ids.

As a hack to circumvent this, I now use this filter with a simple grep to delete the ID:

#!/bin/bash
grep -v '^ *"id": "[a-z\-]*",$'

Store that in e.g. ~/bin/ipynb_output_filter.sh, make it executable (chmod +x ~/bin/ipynb_output_filter.sh) and ensure you have the following ~/.gitattributes file:

*.ipynb filter=dropoutput_ipynb

and in your git config (either global ~/.gitconfig or project)

[core]
         attributesfile = ~/.gitattributes
[filter "dropoutput_ipynb"]
         clean = ~/bin/ipynb_output_filter.sh
         smudge = cat

If you want to use a standard python filter in addition to that, you can invoke it before the grep in ~/bin/ipynb_output_filter.sh, like

#!/bin/bash
~/bin/ipynb_output_filter.py | grep -v '^ *"id": "[a-z\-]*",$'
user_7459030
Related