"can't pickle _thread.RLock objects" when using HuggingFace Trainer with Ray Tune

user_15236206 0
Viewed 329

I try to use the Trainer function from HuggingFace's transformers to train the model and use ray tune for hyperparameters searching. I can run the code in Google Colab without the error, but when I use a remote machine to run the code via ssh, It gives an error:

Traceback (most recent call last):
  File "main.py", line 130, in <module>
    backend="ray")
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/transformers/trainer.py", line 1668, in hyperparameter_search
    best_run = run_hp_search(self, n_trials, direction, **kwargs)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/transformers/integrations.py", line 236, in run_hp_search_ray
    **kwargs,
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/tune/tune.py", line 297, in run
    _ray_auto_init()
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/tune/tune.py", line 670, in _ray_auto_init
    ray.init()
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/_private/client_mode_hook.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/worker.py", line 940, in init
    hook()
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/tune/registry.py", line 197, in flush
    self.references[k] = ray.put(v)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/_private/client_mode_hook.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/worker.py", line 1597, in put
    object_ref = worker.put_object(value)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/worker.py", line 287, in put_object
    serialized_value = self.get_serialization_context().serialize(value)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/serialization.py", line 331, in serialize
    return self._serialize_to_msgpack(value)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/serialization.py", line 311, in _serialize_to_msgpack
    self._serialize_to_pickle5(metadata, python_objects)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/serialization.py", line 271, in _serialize_to_pickle5
    raise e
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/serialization.py", line 268, in _serialize_to_pickle5
    value, protocol=5, buffer_callback=writer.buffer_callback)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
    return Pickler.dump(self, obj)
TypeError: can't pickle _thread.RLock objects

Thanks a lot!

ray huggingface-transformers python
1 Answers

This is due to the fact that the memory tracker that the Trainer uses is not pickleable.

In the more recent transformer versions (4.6 and above I believe), the memory tracker is disabled by default.

Or you can manually disable it by setting skip_memory_metrics=True in your TrainingArguments.

See https://github.com/huggingface/transformers/issues/12177

user_112496910
Related