Install pandas on EMR cluster

Unfortunately this is not fixable at this moment for EMR clusters. I know its not popular solution, but "there is no solution" which you are looking for.

You can see the issue reported on official aws forum here for more details.

I will sum issue bellow.

If you want to have pandas installed, you need to have numpy. Issue at this moment is that (based on the current conclusion), regardless of what happens in the bootstrap.sh the system python37-numpy package gets installed later automatically after bootstrap.sh

This means that, if you install nympy again through bootstrap, importing numpy within the python will result with the importing older version, which is enforced by steps which are happening after boostrap during the emr cluster setup.

You have two solutions:

1. Either manually fix things after cluster is bootstrapped (this means that you will need to do this manually across all of the cluster, or invoke the bash script via ssh to automate this process)
2. You dont install the numpy at all, and roll with the current version, but use pandas which is compatible with that version.

I personally used second solution and installed pandas 1.1.5 until numpy version is upgraded by aws/emr.

I would just add, regarding your statement "don't work, and it's impossible to ssh into the slave nodes".You can ssh into the master and into the executors when ever you want. The default security group for emr is not exposing port 22, hence if you are not able to ssh to the executors, i would take a look there first.

OK, it ain't pretty, but this is what I did to solve:
First of all I used EMR version 5.33.0 and installed the following applications on the cluster: Spark, JupyterEnterpriseGateway (the latter because I am using EMR notebooks).
I used the following bootstrap actions to install necessary things for python:

#!/bin/bash
sudo yes | sudo yum install python3-devel
sudo pip3 install cython
sudo pip3 install setuptools --upgrade

After the cluster has spun up I run this external python script, which has to be given the master node's public DNS name, the master and slave nodes' private IP addresses and the private key filepath. What it does is uninstalls numpy, installs it again and then installs pandas. I have to say this solution feels very hacky and I would very much prefer to do it via bootstrap actions and I still don't really understand why I can't.

import argparse
import paramiko
from scp import SCPClient
import os

INSTALL_COMMANDS = [
    'sudo yes | sudo pip3 uninstall numpy',
    'sudo pip3 install numpy',
    'sudo pip3 install pandas',
]


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--master_address')
    parser.add_argument('--master_ip')
    parser.add_argument('--slave_ips', nargs='+')
    parser.add_argument('--username', default='hadoop')
    parser.add_argument('--pk_filepath')
    args = parser.parse_args()

    key = paramiko.RSAKey.from_private_key_file(args.pk_filepath)
    client = paramiko.SSHClient()
    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    client.connect(hostname=args.master_address, username=args.username, pkey=key)
    scp = SCPClient(client.get_transport())
    scp.put(args.pk_filepath, remote_path='~')
    command_chmod = f'chmod 600 {os.path.split(args.pk_filepath)[-1]}'
    print(f'executing: {command_chmod}')
    stdin, stdout, stderr = client.exec_command(command_chmod)

    all_clients = [client]
    for slave_ip in args.slave_ips:
        slave_channel = client.get_transport().open_channel('direct-tcpip',
                                                            (slave_ip, 22),
                                                            (args.master_ip, 22))
        slave_client = paramiko.SSHClient()
        slave_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        slave_client.connect(hostname=slave_ip,
                             username=args.username,
                             pkey=key,
                             sock=slave_channel)
        all_clients.append(slave_client)

    for client in all_clients:
        for command in INSTALL_COMMANDS:
            print(f'executing {command} on {client}')
            stdin, stdout, stderr = client.exec_command(command)
            print(stderr.read())