Check failed: has_owner Object IDs generated randomly or out-of-band cannot be passed as a task argument. Python interpreter killed

useriBg5 0
Viewed 9

I am running a Ray application, and as it is running, the Python interpreter simply dies and the application stops running.

After debugging a little further, I find the following error message along with what looks like a C++ stack trace:

core_worker.cc:820:  Check failed: has_owner Object IDs generated randomly (ObjectID.from_random()) or out-of-band (ObjectID.from_binary(...)) cannot be passed as a task argument because Ray does not know which task created them. If this was not how your object ID was generated, please file an issue at https://github.com/ray-project/ray/issues/
*** StackTrace Information ***
/tmp/tmp.wyliNWTsWo/venv/lib/python3.8/site-packages/ray/_raylet.so(+0xc4bb3a) [0x7f4891a1bb3a] ray::operator<<()
/tmp/tmp.wyliNWTsWo/venv/lib/python3.8/site-packages/ray/_raylet.so(+0xc4d642) [0x7f4891a1d642] ray::SpdLogMessage::Flush()
/tmp/tmp.wyliNWTsWo/venv/lib/python3.8/site-packages/ray/_raylet.so(_ZN3ray6RayLogD1Ev+0x37) [0x7f4891a1d957] ray::RayLog::~RayLog()
/tmp/tmp.wyliNWTsWo/venv/lib/python3.8/site-packages/ray/_raylet.so(_ZNK3ray4core10CoreWorker15GetOwnerAddressERKNS_8ObjectIDE+0xd9) [0x7f489139bc19] ray::core::CoreWorker::GetOwnerAddress()
/tmp/tmp.wyliNWTsWo/venv/lib/python3.8/site-packages/ray/_raylet.so(+0x51d502) [0x7f48912ed502] __pyx_f_3ray_7_raylet_prepare_args_and_increment_put_refs()
/tmp/tmp.wyliNWTsWo/venv/lib/python3.8/site-packages/ray/_raylet.so(+0x527384) [0x7f48912f7384] __pyx_pw_3ray_7_raylet_10CoreWorker_51submit_task()
/tmp/tmp.wyliNWTsWo/venv/bin/python(+0x1ff05e) [0x55ffe9dc705e] method_vectorcall_VARARGS_KEYWORDS
/tmp/tmp.wyliNWTsWo/venv/bin/python(+0x5db6e) [0x55ffe9c25b6e] call_function
/tmp/tmp.wyliNWTsWo/venv/bin/python(_PyEval_EvalFrameDefault+0x19c4) [0x55ffe9c279d4] _PyEval_EvalFrameDefault
ray python
1 Answers

This happens because you are trying to pass an unknown object as a task argument to Ray.

What do you mean by unknown object?

A way to reproduce this is with the following set of commands:

In [1]: import ray

In [2]: ray.init()
   ...: my_objref = ray.put('MY TEXT')
   ...: ray.shutdown()
   ...: 
   ...: ray.init()
   ...: 
   ...: @ray.remote
   ...: def my_fn(obj):
   ...:   print(obj)
   ...: 
2022-09-12 09:50:29,991 INFO worker.py:1518 -- Started a local Ray instance.
2022-09-12 09:50:35,126 INFO worker.py:1518 -- Started a local Ray instance.

In [3]: my_fn.remote(my_objref)
[2022-09-12 09:50:45,982 C 881351 881351] core_worker.cc:820:  Check failed: has_owner Object IDs generated randomly (ObjectID.from_random()) or out-of-band (ObjectID.from_binary(...)) cannot be passed as a task argument because Ray does not know which task created them. If this was not how your object ID was generated, please file an issue at https://github.com/ray-project/ray/issues/
*** StackTrace Information ***

In this example, we are creating an object in one Ray session, and then passing it as an argument to a function within a different Ray session. This object is unknown to the second Ray session.

How can I solve it?

If you are using a library built on top of Ray (RLib, Ray Serve, Ray Datasets, etc...), then this is most likely a bug on the library, so please file an issue: https://github.com/ray-project/ray/issues/

If you are building your application on top of Ray Core (using ray.remote calls), then this is likely a bug in your application. Can you see places where you may be passing objects that may be unknown?

useriBg50
Related