Im trying to start inference GPT NEOX on single a100(40gb) gpu with deepspeed zero offload(https://www.deepspeed.ai/tutorials/zero-offload/), i know its possible to use it on training, but any solution to use it on inference?
0 Answers
Related
Im trying to start inference GPT NEOX on single a100(40gb) gpu with deepspeed zero offload(https://www.deepspeed.ai/tutorials/zero-offload/), i know its possible to use it on training, but any solution to use it on inference?