Error response from daemon: Get https://docker.intra/v2/main/manifests/3.64.0: unknown: Bad credentials

user_229864 0
Viewed 473

We use Artifactory as a hub for all external docker registries. We have different enviornments, all pull form the same url https://docker.intra. We suddenly have one case where a certain image is not pulled anymore but get this error

ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://docker.intra/v2/main/manifests/3.64.0: unknown: Bad credentials

Actually

Warning     Failed  Error: ImagePullBackOff     2 minutes ago
Normal  BackOff     Back-off pulling image "docker.intra/main:3.64.0"   5 minutes ago
Normal  Pulling     Pulling image "docker.intra/main:3.64.0"    6 minutes ago
Warning     Failed  Error: ErrImagePull     6 minutes ago
Warning     Failed  Failed to pull image "docker.intra/main:3.64.0": rpc error: code = Unknown desc = Error response from daemon: Get https://docker.intra/v2/main/manifests/3.64.0: unknown: This request is blocked due to recurrent login failures, please try again in 4 seconds    6 minutes ago
Warning     Failed  Failed to pull image "docker.intra/main:3.64.0": rpc error: code = Unknown desc = Error response from daemon: Get https://docker.intra/v2/main/manifests/3.64.0: unknown: Bad credentials   6 minutes ago
Warning     Failed  Failed to pull image "docker.intra/main:3.64.0": rpc error: code = Unknown desc = Error response from daemon: Get https://docker.intra/v2/main/manifests/3.64.0: unknown: This request is blocked due to recurrent login failures, please try again in 3 seconds    7 minutes ago
Normal  Scheduled   Successfully assigned stackrox/central-6487fdc867-jq4j5 to k8s0001  7 minutes ago

I however, can successfully download the image from within the mentioned node k8s0001

[papanito@k8s0001 ~]$ docker pull docker.intra/main:3.64.0

    3.64.0: Pulling from main
    29291e31a76a: Pulling fs layer
    ...
    bcabcd2816e8: Pull complete
    Digest: sha256:d03c7cf13b296d8e75529651edae6f08c56144bc090ad79baddccebc07c5a491
    Status: Downloaded newer image for docker.intra/main:3.64.0

There is no authentication defined and should not be required. Here the daemon.json

{
    "insecure-registries": null,
    "log-driver": "json-file",
    "data-root": "/mnt/data/docker",
    "log-opts": {
    "max-size": "50m"
    },
    "log-level": "info",
    "storage-driver": "overlay2",
    "icc": false,
    "registry-mirrors": [
    "https://docker.intra"
    ],
    "userland-proxy": false,
    "no-new-privileges": false,
    "live-restore": true
}

I wonder where the "Bad Credentials" come from and why other images in the same cluster are downloaded successfully!?

kubernetes docker artifactory
2 Answers

The root cause for this behavior is not clear, however it seems it's related to the namespace. Pulling the docker image within another namespace works fine. Same works if one deploys the application in a new namespace.

Alternatively you can also delete the ns completely and then re-create it.

user_2298640

In my case it was due to the password issue., where in my repo password I had $0 in it so when I created k8s image pull secret to connect to the repository , k8s created it as $0 as -bash , so its wrong password, due to that imagepullback error I was getting continuously in pod events and pod was not able to run.,

I changed the password without $0 and that fixed the issue.,

Using below command you can find out whether the image pull secret set has been take the correct password or not.,

Decoding Kubernetes secret

kubectl -n namespace get secret registry-key -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'
user_180274180
Related