I have set up a three nodes Hadoop cluster for dev environment wherein one act as a master node and other being as a datanodes. cluster has been set up with below configurations/version:
Operating System: Red Hat Enterprise Linux Server 7.7
python 3.7.3
anaconda 2
spark 2.45
I would like to install and configure airflow with mysql as a backend database on this hadoop cluster set up. I thought of asking something else regarding configuring MySQL as a backend database but receiving an error while running pip on master node itself.
pip install apache-airflow
Error Message:
Running setup.py install for setproctitle ... error
ERROR: Complete output from command /root/anaconda2/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-TUrT4x/setproctitle/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-GdyZzq/install-record.txt --single-version-externally-managed --compile:
ERROR: running install
running build
running build_ext
building 'setproctitle' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -pthread -B /root/anaconda2/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DHAVE_SYS_PRCTL_H=1 -DSPT_VERSION=1.1.10 -I/root/anaconda2/include/python2.7 -c src/setproctitle.c -o build/temp.linux-x86_64-2.7/src/setproctitle.o
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command "/root/anaconda2/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-TUrT4x/setproctitle/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-GdyZzq/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-TUrT4x/setproctitle/
when I typed which gcc, I get this.
which gcc
/usr/bin/which: no gcc in (/home/xyz/anaconda2/envs/python3.7.2/bin:/home/xyz/anaconda2/bin:/home/xyz/spark-2.4.5-bin-hadoop2.7/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:
/usr/sbin:/home/xyz/jdk1.8.0_241/bin:/home/xyz/hadoop-2.7.7/bin:/home/xyz/.local/bin:/home/xyz/bin)
'xyz' is user name
I am looking for some help to solve above issue and suggestion regarding airflow for the above mentioned configurations.