Modin with ray for pandas working in command prompt but not on Idle, no error code

user_19250322 0
Viewed 160

I try to use modin unstead of pandas to "parallelize by changing a single line of code" I'm using IDLE and when I run this code :

import os
os.environ["MODIN_ENGINE"] = "ray"    
import ray
ray.init()
import modin.pandas as pd
pd.read_csv("some_path")

Some command prompt windows open and close (their path refers to ray), then the line : ================================ RESTART: Shell ================================ is shown (with no error code so I can't know what went wrong) After that whatever the pandas command I try to run in the IDLE window, I get the error "NameError : name 'pd' is not defined".

The problem seems to come from IDLE, because I tried to run it directly from command prompt, and the code worked as intended.

So i tried theese solutions, that all failed :

-Computer reboots

-Check if there was several python installations

-Uninstalling, redownloading an reinstalling all modules

-Uninstalled completely python and reinstalling (3.9)

I found log saying the error comes from ray, and that the root cause is logged in dashboard_agent.log

The refered log is not saved at each run, but I found 2 of them and they warn about a missing module.

I installed the missing module, re-ran the script multiple times, and the script is still not working, the logs are still referring to a log that is no longer generated when I try to run the code,at least in 20 attempts .

modin ray pandas python-idle python
2 Answers

It appears IDLE gives the RESTART message when a subprocess fails, see https://stackoverflow.com/a/29216224/19027728. To clarify, this happens when you are running the IDLE shell, but not from command prompt? In the command prompt run, are you running a script ie python script.py or are you running python interactively? Have you tried running this with another shell, like IPython?

From your debugging steps, it appears it might be a Ray issue. To confirm, can you try running with Dask instead? If you installed with modin[all] you should have that backend. If not, pip install modin[dask] should work.

If using Ray is a necessity, could you perhaps try sharing some of those Ray debug logs? Also, make sure to call ray.shutdown() when appropriate to avoid spawning redundant Ray instances, which might cause issues.

user_190277280

Jeffrey Li is correct about the restart message indicating that IDLE's execution subprocess crashed. The difference from command line Python may be that the execution environment is slightly different when execution via IDLE. (This will be true of any GUI alternative.) My additional suggestions.

  1. Read the Running user code section of the IDLE doc. It is available on IDLE's help menu. One can use sys to investigate the differences mentioned.

  2. Run IDLE from a command line, if not doing so already, with python -m idlelib, where python is adjusted as appropriate for your OS and Python version. The makes the execution environment more similar to the standard environment. So a) you may get an error message on the terminal where you start IDLE and b) you program might behave better.

  3. For the present, develop your pandas application by running pandas directly, without modin. All I know of the latter is that it appears to be an optional optimizer. I know that at least some parts of pandas run fine with IDLE when run directly.

user_7228040
Related