podman returns "Error processing tar file(exit status 1): operation not supported" (rootless)

user_17556074 0
Viewed 476

When starting the images alpine and ubuntu, podman works. If starting the images ubi8 and grafana/grafana-oss, it fails with

Error: writing blob: adding layer with blob "sha256:de63ba066b7c0c23e2434efebcda7800d50d60f33803af9c500f75a69fb76ffa": Error processing tar file(exit status 1): operation not supported

Why is it failing with some images but not others? This is being done rootless, but a network file system is not in play. Full output:

$ whoami
foo
$ echo $HOME
/home/foo
$ df -h /home
Filesystem      Size  Used Avail Use% Mounted on
rootfs          7.9G  6.8G  1.2G  86% /
$ podman run -it ubi8
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 1b890c73c3cf skipped: already exists  
Copying blob de63ba066b7c done  
Error: writing blob: adding layer with blob "sha256:de63ba066b7c0c23e2434efebcda7800d50d60f33803af9c500f75a69fb76ffa": Error processing tar file(exit status 1): operation not supported

It looks similar to but is not the error given when trying to store data on a network file system. ($HOME is a local fs.) The host system is an OS image running in memory on a VMware VM. The OS is RHEL8.6. Kernel=4.18.0.

podman
1 Answers

This did not work because tmpfs (rootfs) was being used as the root filesystem of the host system. This was being used to store the container's backing store (graphroot), i.e. to store images, container-local file systems, etc. When an image is run to create a container, the image's file system and the running container's file system are merged into an overlay file system. The image's file system functions in that overlayfs as the "lower directory" and the container's file system functions as the "upper directory." Tmpfs (and so rootfs) does not allow all the xattrs (user.* and trusted.*) necessary to function as the upper directory. Specifically, the ability to create opaque directories on the upper file system is absent due to the xattr issue. Some images, such as ubi8 and grafana-oss, start up with pre-existing opaque directories, so those cannot be run at all, nor their images pulled. Other images such as alpine and ubuntu do not have this limitation. However, once started, those containers cannot install software due to the xattr issue. It's best to run all these containers using a backing store that is on an XFS file system. However, it's possible to run them on tmpfs using --storage-opt "overlay.mount_program=/usr/bin/fuse-overlayfs" A consequence of using the fuse-overlayfs driver is slower i/o due to data having to go through the FUSE layer on the way to and back from the kernel. If performance to the container's local file system isn't an issue, this would be an acceptable workaround in my view. It's possible to get around this issue by creating an XFS file system on the host system and pointing the backing store to that directory, e.g. by setting XDG_DATA_HOME. Alternatively this can be accomplished by creating a file on the host system's tmpfs that contains an XFS file system and mounting that. Incidentally, NFS v3 has a similar problem. Redhat has updated podman to allow NFS v4 to be used as the backing store.

user_175560740
Related