"Run client channel backup poller" GRPC client error in Celery worker

user_7052159 0
Viewed 485

I am working with GRPC and Celery in Python. When running functions which call to GRPC server in a celery worker, the celery worker log shows a lot of lines:

E1018 16:17:46.449758455     101 backup_poller.cc:133]       Run client channel backup poller: {"created":"@1634573866.449735058","description":"pollset_work","file":"src/core/lib/iomgr/ev_epollex_linux.cc","file_line":321,"referenced_errors":[{"created":"@1634573866.449730704","description":"Bad file descriptor","errno":9,"file":"src/core/lib/iomgr/ev_epollex_linux.cc","file_line":952,"os_error":"Bad file descriptor","syscall":"epoll_wait"}]}
E1018 16:17:51.449779383      34 backup_poller.cc:133]       Run client channel backup poller: {"created":"@1634573871.449756248","description":"pollset_work","file":"src/core/lib/iomgr/ev_epollex_linux.cc","file_line":321,"referenced_errors":[{"created":"@1634573871.449754215","description":"Bad file descriptor","errno":9,"file":"src/core/lib/iomgr/ev_epollex_linux.cc","file_line":952,"os_error":"Bad file descriptor","syscall":"epoll_wait"}]}
E1018 16:17:56.449816371     101 backup_poller.cc:133]       Run client channel backup poller: {"created":"@1634573876.449790405","description":"pollset_work","file":"src/core/lib/iomgr/ev_epollex_linux.cc","file_line":321,"referenced_errors":[{"created":"@1634573876.449786443","description":"Bad file descriptor","errno":9,"file":"src/core/lib/iomgr/ev_epollex_linux.cc","file_line":952,"os_error":"Bad file descriptor","syscall":"epoll_wait"}]}

While running, Celery Worker crashes many times with Segmentation fault (core dumped) error. I cannot find out the root cause of this error.

E1018 16:10:18.593682050      33 backup_poller.cc:133]       Run client channel backup poller: {"created":"@1634573418.593660900","description":"pollset_work","file":"src/core/lib/iomgr/ev_epollex_linux.cc","file_line":321,"referenced_errors":[{"created":"@1634573418.593658189","description":"Bad file descriptor","errno":9,"file":"src/core/lib/iomgr/ev_epollex_linux.cc","file_line":952,"os_error":"Bad file descriptor","syscall":"epoll_wait"}]}
Segmentation fault (core dumped)
grpc celery python
1 Answers

I had a similar error recently trying to call gRPC client as the part of the celery worker task. The common command: celery -A app worker -l info -c 1 resulted in:

[2022-05-23 14:44:44,585: INFO/MainProcess] celery.worker.strategy: Task io_bound:get_landmarks_via_grpc[ebb65e3d-cef5-4b22-88d6-947a2560f4c1] received
Segmentation fault (core dumped)

According to this GH issue the problem is that celery uses prefork pool by default while gRPC doesn't support forking (more details here).

So if you choose a different pool option, e.g. 'gevent', it works.
celery -A app worker -l info --pool=gevent -c 1 results in the succeeded task.

More about celery execution pools: https://www.distributedpython.com/2018/10/26/celery-execution-pool/

user_191820230
Related