Jupyter notebook on EMR not printing output while code is running Pyspark

user_9908952 0
Viewed 520

I am running a very, very simple script in a Jupyter pyspark notebook, but it is not printing results as it runs, it just spits out the output when it's done. Here is the code:

import time
import sys

for i in range(10):
    print(i)
    time.sleep(1)

This waits 10 seconds and then prints:

0
1
2
3
4
5
6
7
8
9

I would like to print results as they happen. I have tried to flush them using

for i in range(10):
    print(i)
    sys.stdout.flush()

and print(i, flush=True) to no avail. Any suggestions?

amazon-emr jupyter-notebook pyspark python
2 Answers

Depending on what you are doing, if you are running pyspark code and are hoping to see results before the job is complete, this may not work.

You may be running into an issue with how spark/pyspark runs your code. Spark is designed to efficiently divide up your task into parts, and distribute those parts to the nodes of your EMR cluster.

This means that the actual work does not happen on the machine where your notebook is running. The main node, where your notebook is running, sends tasks out to all the worker nodes, then collects the results as they are sent back, and only displays them once the job is complete. This can be troublesome for someone used to debugging plain python, but is a big part of what makes pyspark so fast when working with large amounts of data.

user_73862260

It is a buffering issue. You could also use python -u command or set PYTHONUNBUFFERED envvar. python uses line-buffering if it is run interactively (in a terminal) and block buffering (e.g., ~4K bytes buffers) if the output is redirected

user_135312190
Related