Where are the logs for jupyterhub?

user_559827 0
Viewed 5219

IMPORTANT: I use Jupyter notebooks by visiting my employer's jupyterhub website, and logging in. I never actually run the jupyterhub command on the command line.

I mention this up-front because so much of the information I find online on jupyterhub presupposes that the user, at some point, runs the jupyterhub command on the command line, but I never do this, per company SOP. Please keep this in mind at the time of posting an answer.

I just discovered, after hours of troubleshooting and head-scratching, that the reason a particular conda virtual environment/kernel was not showing up among the choices revealed upon expanding the New button in my jupyterhub homepage was a 1-character typo in a kernel.json file.

As far as the user (me) was concerned, the jupyterhub startup was going through without a hitch.

In reality, this one small typo was causing the entire processing of this candidate kernel to be silently aborted.

A simple warning would have saved me hours and hours of work.

The next best thing would have been to be able to examine any log messages that jupyterhub may produce in such situations.

Does jupyterhub keep a log?

Assuming that the answer to this first question is "yes", I have a few follow-up questions:

  • How can I view this log?
  • Is there a way to make this log (if it exists) maximally verbose?
  • Is it possible to have the log messages also displayed directly on the web interface?
jupyterhub
1 Answers

TL;DR: JupyterHub does not keep a log. The log messages are printed to stdout/stderr and have to be captured in a file, preferably using redirection.

Hidden in the jupyterhub_config.py file I see:

## DEPRECATED: use output redirection instead, e.g.
#  
#  jupyterhub &>> /var/log/jupyterhub.log
#  Default: ''
# c.JupyterHub.extra_log_file = ''

Note also this GitHub issue comment from 2017 which says that

If you want logging to a file, output redirection is a more robust approach that works for JupyterHub, as well as every other program:

jupyterhub &>> /var/log/jupyterhub.log This will capture all output from jupyterhub and its subprocesses, including the proxy and single-user servers.

Or, if you are using systemd/supervisor/etc. to launch your process, there is always a stdout/stderr capture option, which is the right way to go.

It looks like Jupyter just prints its messages to stdout/stderr, and it is the administrator's responsibility to save the messages into a log file.

Following this other GitHub issue comment with modifications, I could start JupyterHub on a Linux system that runs systemd (e.g. modern Debian, Ubuntu) as follows:

  1. Create the file /lib/systemd/system/jupyterhub.service with the following contents:
[Unit]
Description=Jupyterhub

[Service]
User=root
ExecStart=/usr/local/bin/jupyterhub -f /etc/jupyterhub/jupyterhub_config.py

[Install]
WantedBy=multi-user.target
  1. Reload the systemctl daemon and then you can start/stop the service as usual:
sudo systemctl daemon-reload
sudo service jupyterhub start

Now for the bonus: you can see the JupyterHub logs with the command sudo journalctl -u jupyter.

Since all this magic requires sudo privileges, you would like to talk to your employer's IT department / sysadmin person to set up JupyterHub like this. Your user must be member of the systemd-journal group to see all logs.

user_4651390
Related