Raise limitations in CPU use when running singularity containers

user_13678312 0
Viewed 459

I developed some (complex) code in python 3. When I run it on my laptop from the shell (Ubuntu 18.04), the CPU use is 550% (from the "top" command). When I run it from a singularity container (based on Ubuntu 16.04), the CPU use is 250% and execution time is increased. I cannot figure out why singularity cannot use more CPU.

I read the manual on https://sylabs.io/guides/3.0/admin-guide/configfiles.html#singularity-conf but my singularity.conf is a default file and I did not create any /sys/fs/cgroup file either. Singularity version is 3.0.3.

Does anyone has a clue regarding this issue?

Thank you!

JB

edit: this case can be reproduced with the simple example below:

use: python3 nb_cpu_singularity.py 300000 10000

nb_cpu_singularity.py:

import numpy as np
import numba as nb
import argparse


parser = argparse.ArgumentParser(description="compute dot products")
parser.add_argument("sample_size", type=int, default=10000,
                    help="number of dot products")
parser.add_argument("dim", type=int, default=1000,
                    help="dimension of vectors")

args = parser.parse_args()

# Inputs
sample_size = args.sample_size
dim = args.dim

@nb.jit(nopython=True, nogil=True, fastmath=True, parallel=False)
def build_vector(offset, dim):

    v = np.zeros(dim, dtype=np.float64)
    for i in range(dim):
        v[i] += i+offset
    return(v)


@nb.jit(nopython=True, nogil=True, fastmath=True, parallel=False)
def dot_products(sample_size, dim):

    for i in range(sample_size):
        np.dot(build_vector(i, dim), build_vector(i+1, dim))        


dot_products(sample_size, dim)

Edit: following the answer by Jakub, I added two singularity recipes yielding different behaviours.

Bootstrap: docker
From: ubuntu:18.04

# .def files for Singularity image to be used with bnp-mrf for count data.
# Includes R packages for post-processing

# Tips:
#   + Use export TMPDIR=my_tmp_dir to specify the directory for temporary files
#   + Build images as root: sudo singularity build ...

# Tested with singularity 3.0.3

%help
This singularity image contains python libraries to run BNP MRF models without tensorflow.
You may run the image by using
singularity run --app jupyter -e -B /my_scratch:/scratch:rw notensorflow-1-4-1_minimal_count.simg
where /my_scratch is the name of a host directory containing some jupyter notebook(s) you want to run withing the container and assuming notensorflow-1-4-1_minimal_count.simg is the name of the file produce by singularity build on the present definition file.
If you just want to run an ipython console, use
singularity run --app console notensorflow-1-4-1_minimal_count.simg


%labels
BUILD.CMD="sudo singularity build notensorflow-1-4-1_minimal_count.simg make_simg_count_data_minimal.singularity"

%setup

# Just an example, not used here
%files
#basic_classification.py     /opt/scripts/

%environment
export LANG="C.UTF-8" LC_ALL="C.UTF-8"

%post

export TZ=Europe/Minsk

apt update && DEBIAN_FRONTEND=noninteractive apt install -y gedit python3-pip llvm software-properties-common apt-transport-https

# R installation
export R_REPOS="https://cloud.r-project.org"
# apt-key adv --keyserver keys.gnupg.net --recv-key 'E19F5F87128899B192B1A2C2AD5F960A256A04AF' ?
# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 51716619E084DAB9
# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 51716619E084DAB9
add-apt-repository "deb $R_REPOS/bin/linux/ubuntu bionic-cran35/"
export R_VERSION="3.6.3-1bionic"
apt update && DEBIAN_FRONTEND=noninteractive apt install -y r-base=$R_VERSION libudunits2-dev libgdal-dev

# Python env
python3 -m pip install --upgrade pip

python3 -m pip install llvmlite matplotlib numba numpy opencv-python pandas scikit-image scikit-learn scipy ipython jupyterlab rpy2 tbb

rm -rf /var/lib/apt/lists/*

%runscript
cd /scratch
ipython3

%apprun console
cd /scratch
ipython

%apprun jupyter
cd /scratch/
jupyter lab

The other image is

Bootstrap: docker
From: ubuntu:16.04

# .def files for Singularity image to be used with bnp-mrf for count data.
# Includes R packages for post-processing

# Tips:
#   + Use export TMPDIR=my_tmp_dir to specify the directory for temporary files
#   + Build images as root: sudo singularity build ...

# Tested with singularity 3.0.3

% help
This singularity image contains python libraries to run BNP MRF models without tensorflow.
You may run the image by using
singularity run --app jupyter -e -B /my_scratch:/scratch:rw notensorflow-1-4-1_cpu_count.simg
where /my_scratch is the name of a host directory containing some jupyter notebook(s) you want to run withing the container and assuming notensorflow-1-4-1_cpu_count.simg is the name of the file produce by singularity build on the present definition file.
If you just want to run an ipython console, use
singularity run --app console notensorflow-1-4-1_cpu_count.simg


%labels
BUILD.CMD="sudo singularity build notensorflow-1-4-1_cpu_count.simg make_simg_count_data_cpu.singularity"

%setup
%mkdir -p ${SINGULARITY_ROOTFS}/r_analysis

# Just an example, not used here
%files
%basic_classification.py     /opt/scripts/

%environment
export LANG="C.UTF-8" LC_ALL="C.UTF-8"

%post

apt update && apt install -y gedit 

apt install -y --no-install-recommends \
    ca-certificates apt-transport-https gnupg curl dirmngr vim cmake

apt update --allow-insecure-repositories

apt update && apt install -y python3-pip

apt install -y llvm

python3 -m pip install --upgrade pip

python3 -m pip install py
python3 -m pip install urllib3
python3 -m pip install pylint
python3 -m pip install wordcloud
python3 -m pip install tornado
python3 -m pip install theano
python3 -m pip install cython
python3 -m pip install dlib
python3 -m pip install h5py
python3 -m pip install html5lib
python3 -m pip install jupyter
python3 -m pip install joblib
python3 -m pip install llvmlite==0.30.0
python3 -m pip install nltk
python3 -m pip install jupyter 
python3 -m pip install notebook 
python3 -m pip install matplotlib
python3 -m pip install numba==0.46.0
python3 -m pip install numpy
python3 -m pip install opencv-python
python3 -m pip install pandas
python3 -m pip install pillow
python3 -m pip install scikit-image
python3 -m pip install scikit-learn
python3 -m pip install scipy
python3 -m pip install seaborn
python3 -m pip install simplegeneric

# # Install pymc3 from source
apt install -y git
cd /root
git clone --branch v3.6 https://github.com/pymc-devs/pymc3/ /root/pymc3

# Note that 
# cd /root/pymc3/ 
# /usr/bin/python3 setup.py install
# is not necessarily equivalent to 
# cd /root/pymc3/ && \
#    /usr/bin/python3 setup.py install
# since the side effect of cd might be lost in subsequent instructions

cd /root/pymc3/ && \
    /usr/bin/python3 setup.py install && \
    cd ../ && \
    rm -Rf pymc3

# R packages

apt install -y apt-transport-https software-properties-common

apt update

export R_REPOS="https://cloud.r-project.org"

add-apt-repository "deb $R_REPOS/bin/linux/ubuntu xenial-cran35/"

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 51716619E084DAB9

apt update

# To find R versions
# apt-cache policy r-base

export R_VERSION="3.6.3-1xenial"
apt install -y r-base=$R_VERSION
apt install -y libudunits2-dev
apt install -y libgdal-dev

mkdir /root/r_analysis
cd /root/r_analysis
export R_CONTRIBS="https://cloud.r-project.org"
echo 'install.packages("INLA", repos=c("'$R_CONTRIBS'", INLA="https://inla.r-inla-download.org/R/testing"), dep=TRUE)' >> r_install.txt
echo 'install.packages("diseasemapping", repos="'$R_CONTRIBS'")' >> r_install.txt
echo 'install.packages("sp", repos="'$R_CONTRIBS'")' >> r_install.txt
echo 'install.packages("spdep", repos="'$R_CONTRIBS'")' >> r_install.txt
echo 'install.packages("geostatsp", repos="'$R_CONTRIBS'")' >> r_install.txt
echo 'install.packages("mapmisc", repos="'$R_CONTRIBS'")' >> r_install.txt
Rscript r_install.txt

rm -Rf /root/r_analysis

# Install rpy2
apt install -y python3-rpy2=2.9.3-1xenial0

%runscript
python3 /opt/scripts/basic_classification.py


%apprun console
ipython

%apprun jupyter
jupyter notebook --ip 0.0.0.0 --no-browser --allow-root
singularity-container cpu-usage python-3.x
1 Answers

I am not able to reproduce this, also using the default singularity.conf and singularity version 3.5.1. Could the difference be due to differences between your host environment and the python environment in the container? Can you please share your Singularity definition file?

My computer has 8 cores, and I saw the usage get up to 798% when running in the container and on my host. Here are the code and commands I used.

Singularity

Bootstrap: docker
From: continuumio/miniconda3:4.8.2

%post
/opt/conda/bin/conda install --yes nomkl numba numpy scipy

%runscript
/opt/conda/bin/python3 "$@"

sudo singularity build conda.sif Singularity
singularity run conda.sif nb_cpu_singularity.py 30000 100000

Host

conda create --yes -n sotest nomkl numba numpy scipy
conda activate sotest
python nb_cpu_singularity.py 30000 100000
user_56660870
Related