Unable to install confluent-kafka: "fatal error: librdkafka/rdkafka.h: No such file or directory"

user_19070243 0
Viewed 4253

I am using the confluent-kafka Python client in my project. I'm trying to create a Docker image with this client.

I am facing the following error:-

#11 8.015 [pipenv.exceptions.InstallError]:       In file included from /tmp/pip-install-so_whhii/confluent-kafka_9d9553bf46cf489bb25fcb2ac7698747/src/confluent_kafka/src/Admin.c:17:
#11 8.015 [pipenv.exceptions.InstallError]:       /tmp/pip-install-so_whhii/confluent-kafka_9d9553bf46cf489bb25fcb2ac7698747/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
#11 8.015 [pipenv.exceptions.InstallError]:          23 | #include <librdkafka/rdkafka.h>
#11 8.015 [pipenv.exceptions.InstallError]:             |          ^~~~~~~~~~~~~~~~~~~~~~
#11 8.015 [pipenv.exceptions.InstallError]:       compilation terminated.
#11 8.015 [pipenv.exceptions.InstallError]:       error: command '/usr/bin/gcc' failed with exit code 1
#11 8.016 [pipenv.exceptions.InstallError]:       [end of output]

Based on my search it is related to Apple M1 build for librdkafka.

librdkafka confluent-kafka-python apache-kafka docker python
6 Answers

Please refer to this Github issue. The problem with M1 is that Homebrew is installed in a different location and so these variables need to be added to the environment by including these lines in your .zshrc file

export C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include
export LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib
user_70150860

The Confluent engineers are obviously very focused on their paying customers, and many, many months after the release of Python 3.10, they still haven't released 3.10 wheels that include the binaries for the package.

https://github.com/confluentinc/confluent-kafka-python/issues/1219

You have two choices. 1, run Python 3.9; 2, install librdkafka-dev via apt before installing confluent-kafka (for a debian base image).

The issue is the same with Apple M1 - i.e, there are no pre-built wheels for the platform containing the binaries, but it is much easier to fix (by installing librdkafka-dev) if you are trying to build a amd64 Linux image.

user_9039660

On a non-M1 Mac (intel), Python 3.10.6, I solved this with

export C_INCLUDE_PATH=/usr/local/Cellar/librdkafka/1.9.2/include     
export LIBRARY_PATH=/usr/local/Cellar/librdkafka/1.9.2/lib
user_43579380

Prebuilt binary wheels of confluent-kafka-python for your Apple M1 seems to not exist.

Thus, in order to install the package, pip tries to build it itself from source. But librdkafka looks like to not be installed as per librdkafka/rdkafka.h: No such file or directory error you got.

You can find specific instructions to install from source for different platforms here.

user_80533700

I solved this problem trying with Python 3.9.13 on a virtual environment.

user_51566750

On a non-M1 mac, Python 3.10.2, I solved this with brew install librdkafka.

user_17546530
Related