We are running into issues with one of Ubuntu's repo's, and are wondering if others are experiencing something similar.
Background:
We run several autoscaling groups within AWS EC2 and our automated server provisioning has been failing consistently for the past several days between 1:30am EST and 2:30am EST.
The failures are the result of "apt-get update" timing out when pulling updated package lists from us-east-1.ec2.archive.ubuntu.com/(various). When the task fails, several corrupt lists are left in /var/lib/apt/lists/partial/ which prevent further apt-get-update's from running.
We setup a monitor to check the access state of each package that "apt-get update" pulls down (i.e. curl -Is apt-package-url | head -1). This ran minutely overnight and confirmed what we were experiencing.
From 1:39am EST to 2:11 EST we started to encounter 408 Request Timeout issues against some (but not all) files within the ubuntu repo.
Here is the list of affected packages:
Fri Mar 11 07:11:04 UTC 2016 - http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/precise/main/binary-i386/Packages.gz - HTTP/1.0 408 Request Time-out
Fri Mar 11 07:04:54 UTC 2016 - http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/precise-updates/universe/source/Sources.gz - HTTP/1.0 408 Request Time-out
Fri Mar 11 07:03:43 UTC 2016 - http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/precise-updates/universe/binary-i386/Packages.gz - HTTP/1.0 408 Request Time-out
Fri Mar 11 07:00:07 UTC 2016 - http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/precise/universe/i18n/Translation-en.gz -
Fri Mar 11 06:59:22 UTC 2016 - http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/precise-updates/universe/binary-i386/Packages.gz - HTTP/1.0 408 Request Time-out
Fri Mar 11 06:58:24 UTC 2016 - http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/precise/universe/i18n/Index - HTTP/1.0 408 Request Time-out
Fri Mar 11 06:49:11 UTC 2016 - http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/precise/main/binary-i386/Packages.gz - HTTP/1.0 408 Request Time-out
Fri Mar 11 06:47:18 UTC 2016 - http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/precise/main/i18n/Index - HTTP/1.0 408 Request Time-out
Fri Mar 11 06:39:26 UTC 2016 - http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/precise/Release - HTTP/1.0 408 Request Time-out}
As a reminder, these endpoints were monitored minutely, and these are the only times that they timed out.
The complete list of monitored endpoints contains about 25 different files on us-east-1.ec2.archive.ubuntu.com, and as you can see not all of them were affected. So it doesn't look like the repository goes down, nor that we are experiencing networking issues from our hosts. It's as if the transfer from the repo simply times out leading to a corrupt partial file (slow? bad cache? etc?)
Questions:
Has anyone else within the community run into issues recently with the older Precise ubuntu repositories?
While apt-cache options already exist, is anyone aware of any option that would also cache the source lists?
Any thoughts/suggestions?
I'm primarily interested if others are running into the same or similar issues with Apt. We are actively working on a couple different apt cache solutions to see if we can mitigate this problem.
Thanks!