"unknown blob" errors while pulling new images from JFrog

Viewed 3058

Since 2 days ago, we have been getting "unknown blob" errors when pulling from jfrog. I am attaching a sample log:

Command ['ssh', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', 'localhost', 'docker', 'pull', '<redacted>.jfrog.io/<redacted>:latest'] failed with exit code 1 and output 'latest: Pulling from <redacted>
f5d23c7fed46: Pulling fs layer
3f4aa1d1dde5: Pulling fs layer
52c4bf0b6229: Pulling fs layer
fe61f8f5a308: Pulling fs layer
ebeed9e8b27e: Pulling fs layer
89831686aa31: Pulling fs layer
2e2c5baec652: Pulling fs layer
b6fa760c79e4: Pulling fs layer
2e2c5baec652: Waiting
ebeed9e8b27e: Waiting
b6fa760c79e4: Waiting
fe61f8f5a308: Waiting
3f4aa1d1dde5: Verifying Checksum
3f4aa1d1dde5: Download complete
f5d23c7fed46: Verifying Checksum
f5d23c7fed46: Download complete
fe61f8f5a308: Download complete
ebeed9e8b27e: Download complete
89831686aa31: Download complete
f5d23c7fed46: Pull complete
3f4aa1d1dde5: Pull complete
2e2c5baec652: Verifying Checksum
2e2c5baec652: Download complete
b6fa760c79e4: Downloading
unknown blob

This seems to have started during the kinesis outage. We first noticed it while we were trying to deploy a workaround during the outage. However the problem still persists.

The image pulls fine from docker hub, so it's not corrupted. This is currently breaking out automated deploy/provisioning process, as we have manually pull failed imaged from dockerhub.

Thanks, -Caius

1 Answers

With @John's suggestion, I zapped the cache on JFrog side, and that removed the issue.

It seems that it's stale/invalid cache issue.

Also, while looking at the JFrog logs, I did find this, which might be relavant:

2020-11-28T18:55:24.493Z [jfrt ] [ERROR] [b66d3ae308977fb1] [o.a.r.RemoteRepoBase:858      ] [ttp-nio-8081-exec-17] - IO error while trying to download resource '<redacted>: org.artifactory.request.RemoteRequestException: Error fetching <redacted>/blobs/sha256:9c11dabbdc3a450cd1d9e15b016d455250606d78eecb33c92eebfa657549787f (remote response: 429: Too Many Requests)

TL;DR: zapping the cache fixed the problem.

Related