Amazon S3: s3cmd put ([Errno 32] Broken pipe)

Recently I decided to use Amazon’s S3 as another location to store some of my server backups. However I found when testing that I was unable to upload my backup tarballs to S3. I ended up with the following errors.

[email protected]:backups$ s3cmd put backup-20140731.tar.gz s3://test-bucket/backup-20140731.tar.gz
backup-20140731.tar.gz -> s3://test-bucket/backup-20140731.tar.gz  [1 of 1]
    253952 of 250487870     0% in    1s   159.37 kB/s  failed
WARNING: Upload failed: /backup-20140731.tar.gz ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
backup-20140731.tar.gz -> s3://test-bucket/backup-20140731.tar.gz  [1 of 1]
    569344 of 250487870     0% in    1s   357.37 kB/s  failed
WARNING: Upload failed: /backup-20140731.tar.gz ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
backup-20140731.tar.gz -> s3://test-bucket/backup-20140731.tar.gz  [1 of 1]
    405504 of 250487870     0% in    1s   282.39 kB/s  failed
WARNING: Upload failed: /backup-20140731.tar.gz ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
backup-20140731.tar.gz -> s3://test-bucket/backup-20140731.tar.gz  [1 of 1]
     98304 of 250487870     0% in    1s    64.06 kB/s  failed
WARNING: Upload failed: /backup-20140731.tar.gz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
backup-20140731.tar.gz -> s3://test-bucket/backup-20140731.tar.gz  [1 of 1]
     28672 of 250487870     0% in    1s    14.69 kB/s  failed
WARNING: Upload failed: /backup-20140731.tar.gz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
backup-20140731.tar.gz -> s3://test-bucket/backup-20140731.tar.gz  [1 of 1]
     16384 of 250487870     0% in    4s     3.84 kB/s  failed
ERROR: Upload of 'backup-20140731.tar.gz' failed too many times. Skipping that file.

The file I’m uploading isn’t huge (238MB). So it wasn’t the known limit of trying to upload files which are greater than 5GB.

[email protected]:backups$ ll
total 238M
-rw------- 1 jacob 238M Jul 31 06:00 backup-20140731.tar.gz

It seems that this is a common issue when uploading files to a new bucket. Something to do with the DNS propagation means it takes a little while before your bucket can successfully accept files. More info here.

After waiting a few hours running the same command completes successfully.