Problem
I am running an Azure Pipeline to deploy specifically tagged images from Azure Container Registry (ACR) into an AKS cluster. For some image tags, the deployment succeeds. For other image tags, the deployment fails to pull the image. The deployed pod shows ImagePullBackOff.
Setup
An Azure Container Registry (ACR) contains a repository named my_repo_name Repository my_repo_name contains several image tags, examples: my_repo_name:0.4.0-51775-custom-hostaliases.485, my_repo_name:0.4.0-51775-custom-hostaliases.473, my_repo_name:0.4.0-alpha.449 An Azure Pipeline builds/publishes a Docker image to ACR and/or deploys an image to AKS
Strangeness
The deployment pipeline succeeds for my_repo_name:0.4.0-51775-custom-hostaliases.473, my_repo_name:0.4.0-alpha.459, and several other image tags. However, the deployment pipeline fails for my_repo_name:0.4.0-51775-custom-hostaliases.485 and several other image tags.
Error
kubectl describe shows: Warning Failed 26s (x3 over 73s) kubelet Failed to pull image "my_repo_name.azurecr.io/my_repo_name:0.4.0-51775-custom-hostaliases.485": rpc error: code = Unknown desc = failed to pull and unpack image "my_repo_name.azurecr.io/my_repo_name:0.4.0-51775-custom-hostaliases.485": failed to resolve reference "my_repo_name.azurecr.io/my_repo_name:0.4.0-51775-custom-hostaliases.485": failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized
I checked the ACR. All image tags that I am trying to deploy exist in ACR.
Questions
What would cause K8s to have the authorization to pull some images tags and lack authorization for other image tags?
What tools, logs, etc. can help me troubleshoot this issue? I turned Azure Pipeline logging to debug mode, looked through App Insights logging at pod-level, used kubectl describe.