toffer / minecloud-ami Goto Github PK

Rare, intermittent "BadRequest" errors leads to corrupted backup on S3.

Here's an example error message from running msm-backup-working-files-to-s3.py when it calls boto_rsync:

boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>BadDigest</Code>
    <Message>The Content-MD5 you specified did not match what we received.</Message>
    <ExpectedDigest>VJXmhwy/D/WTNmj8oPUWhA==</ExpectedDigest>
    <CalculatedDigest>1L1i4pARxmfJaEzIV0JVag==</CalculatedDigest>   
    <RequestId>951A188CE57205D4</RequestId>  
    <HostId>FsGh/wbmZDOw7kRk9hGZ078NVvBe1VV7p3U1ac0b8Xp9dakzvVyVrg344xQqvWEy</HostId>
</Error>

Any recommendations for a well-crafted IAM user policy?

Out of laziness I just went for Power User, but ideally the policy would be narrowly specified for subsets of EC2 and S3 functionality -- but exactly which permissions is always tricky business :)

Do you happen to have a policy already?

Seem to be getting this error whenever I run ./build-ami.py. I'm in the venv and everything, I think it's because the instance takes longer to boot, and perhaps boto has a timeout?

Launching instance of ami-c08dbc85...
Instance state: pending
Instance state: pending
Public DNS: ec2-54-193-66-121.us-west-1.compute.amazonaws.com
Public IP address: 54.193.66.121
Instance state: running
Instance ID: i-22b8ca7f
Waiting 30 seconds for instance to boot...
[ec2-54-193-66-121.us-west-1.compute.amazonaws.com] Executing task 'check_instance_availability'

Fatal error: Timed out trying to connect to ec2-54-193-66-121.us-west-1.compute.amazonaws.com (tried 1 time)

Underlying exception:
    timed out

Aborting.

The /opt/msm directory should be copied from the EC2 instance to the S3 bucket during shutdown, but the backup scripts called by /etc/init.d/msm-s3-sync stop fail, since the MSM_S3_BUCKET environment variable is not available on the EC2 instance.

Usually, the EC2 instance.public_dns_name is available as soon as the instance state changes from "pending" to "running", but sometimes there is a delay. When there is a delay, the build-ami.py script fails.

Launching instance of ami-23d9a94a...
Instance state: pending
Instance state: pending
Instance state: pending
Public DNS: ec2-174-129-151-62.compute-1.amazonaws.com
Public IP address: 174.129.151.62
Instance state: running
Instance ID: i-9ab7dbf9
Waiting 30 seconds for instance to boot...
[ec2-174-129-151-62.compute-1.amazonaws.com] Executing task 'check_instance_availability'
Waiting for cloud-init to finish running...
Waiting for cloud-init to finish running...
Waiting for cloud-init to finish running...
Instance is ready.
[ec2-174-129-151-62.compute-1.amazonaws.com] Executing task 'copy_manifests'
Copying puppet manifests...
[localhost] local: git archive --prefix=puppet-minecraft/ --output=puppet-minecraft.tar.gz HEAD
[ec2-174-129-151-62.compute-1.amazonaws.com] put: puppet-minecraft.tar.gz -> /home/ubuntu/puppet-minecraft.tar.gz
[ec2-174-129-151-62.compute-1.amazonaws.com] run: tar xzf puppet-minecraft.tar.gz
[localhost] local: rm puppet-minecraft.tar.gz
[ec2-174-129-151-62.compute-1.amazonaws.com] Executing task 'apply_manifests'
Running puppet apply...
[ec2-174-129-151-62.compute-1.amazonaws.com] sudo: puppet apply -v --modulepath=/home/ubuntu/puppet-minecraft/modules /home/ubuntu/puppet-minecraft/manifests/base.pp
[ec2-174-129-151-62.compute-1.amazonaws.com] out: info: Applying configuration version '1374717354'
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[first]/Apt/Exec[apt-update]/returns: executed successfully
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[first]/Apt/Exec[hold_grub]/returns: executed successfully
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[first]/Apt/Exec[apt-upgrade]/returns: executed successfully
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[first]/Apt/Package[tree]/ensure: ensure changed 'purged' to 'present'
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[first]/Apt/Package[zip]/ensure: ensure changed 'purged' to 'present'
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[first]/Apt/Package[libpq-dev]/ensure: ensure changed 'purged' to 'present'
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[second]/Virtualenv/Package[python-dev]/ensure: ensure changed 'purged' to 'present'
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[second]/Virtualenv/Package[python-setuptools]/ensure: ensure changed 'purged' to 'present'
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[second]/Virtualenv/Exec[easy_install pip]/returns: executed successfully
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[second]/Oracle_java/User[oab]/ensure: created
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[second]/Oracle_java/Exec[git_clone_oab]/returns: executed successfully
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[second]/Oracle_java/Exec[build_java]/returns: executed successfully
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[second]/Oracle_java/Package[oracle-java7-jre]/ensure: ensure changed 'purged' to 'present'
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[second]/Virtualenv/Package[virtualenv]/ensure: created
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[second]/Virtualenv/Exec[install-virtualenv]/returns: executed successfully
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[second]/Virtualenv/File[/usr/local/venv/requirements.txt]/ensure: defined content as '{md5}1c58a1ef9b23ac010406f7f2167a5b9c'
[ec2-174-129-151-62.compute-1.amazonaws.com] out: err: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns: change from notrun to 0 failed: /usr/local/venv/bin/pip install -r /usr/local/venv/requirements.txt returned 1 instead of one of [0] at /home/ubuntu/puppet-minecraft/modules/virtualenv/manifests/init.pp:43
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/File[/etc/init.d/msm-s3-sync]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/File[/etc/init.d/msm-s3-sync]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/File[/usr/local/bin]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/File[/usr/local/bin]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/msm-backup-archives-to-s3.sh]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/msm-backup-archives-to-s3.sh]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/File[/etc/init.d/msm-update-instance-state]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/File[/etc/init.d/msm-update-instance-state]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/pip-2.7]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/pip-2.7]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/msm-update-auth-lists.py]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/msm-update-auth-lists.py]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/msm-update-instance-state.py]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/msm-update-instance-state.py]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/msm-manage-restore-working-files-from-s3.py]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/msm-manage-restore-working-files-from-s3.py]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/virtualenv]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/virtualenv]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/pip]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/pip]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/File[/etc/init.d/msm-update-auth-lists]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/File[/etc/init.d/msm-update-auth-lists]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/Exec[update-rc.d -f msm-update-auth-lists remove]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/Exec[update-rc.d -f msm-update-auth-lists remove]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/msm-pre-shutdown-backup.sh]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/msm-pre-shutdown-backup.sh]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/Exec[update-rc.d msm-update-auth-lists defaults 96 04]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/Exec[update-rc.d msm-update-auth-lists defaults 96 04]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/virtualenv-2.7]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/virtualenv-2.7]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/msm-manage-pre-shutdown-backup.py]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/msm-manage-pre-shutdown-backup.py]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/msm-redis-listener.py]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/msm-redis-listener.py]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/msm-update-download-url.py]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/msm-update-download-url.py]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/File[/etc/init.d/supervisor]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/File[/etc/init.d/supervisor]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/Exec[update-rc.d -f msm-update-instance-state remove]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/Exec[update-rc.d -f msm-update-instance-state remove]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/Exec[update-rc.d msm-update-instance-state defaults 98 06]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/Exec[update-rc.d msm-update-instance-state defaults 98 06]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/File[/etc/init.d/msm-jar-update]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/File[/etc/init.d/msm-jar-update]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/Exec[update-rc.d -f msm-jar-update remove]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/Exec[update-rc.d -f msm-jar-update remove]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/Exec[update-rc.d msm-jar-update defaults 96 04]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/Exec[update-rc.d msm-jar-update defaults 96 04]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/msm-track-active-players.py]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/msm-track-active-players.py]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/File[/etc/cron.d]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/File[/etc/cron.d]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/etc/cron.d/.placeholder]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/etc/cron.d/.placeholder]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/etc/cron.d/msm-backup-working-files-to-s3]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/etc/cron.d/msm-backup-working-files-to-s3]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/etc/cron.d/msm-track-active-players]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/etc/cron.d/msm-track-active-players]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/Group[minecraft]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/Group[minecraft]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/User[minecraft]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/User[minecraft]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/opt/msm]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/opt/msm]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/dev/shm/msm]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/dev/shm/msm]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/Exec[git_clone_msm]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/Exec[git_clone_msm]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/etc/init.d/msm]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/etc/init.d/msm]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/Exec[update-rc.d -f msm remove]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/Exec[update-rc.d -f msm remove]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/etc/msm.conf]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/etc/msm.conf]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/Exec[msm_update]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/Exec[msm_update]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/Exec[update-rc.d msm defaults 97 03]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/Exec[update-rc.d msm defaults 97 03]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/Exec[msm_jargroup_create]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/Exec[msm_jargroup_create]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/Exec[msm_server_create]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/Exec[msm_server_create]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/opt/msm/servers/default/active]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/opt/msm/servers/default/active]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/Exec[msm_jar]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/Exec[msm_jar]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/opt/msm/servers/default/worldstorage/world]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/opt/msm/servers/default/worldstorage/world]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/opt/msm/servers/default/world]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/opt/msm/servers/default/world]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/opt/msm/archives]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/opt/msm/archives]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/opt/msm/archives/logs]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/opt/msm/archives/logs]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/opt/msm/archives/worlds]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/opt/msm/archives/worlds]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/opt/msm/archives/backups]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/opt/msm/archives/backups]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/Exec[update-rc.d -f msm-s3-sync remove]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/Exec[update-rc.d -f msm-s3-sync remove]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/Exec[update-rc.d msm-s3-sync defaults 95 05]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/Exec[update-rc.d msm-s3-sync defaults 95 05]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/opt/msm/servers/default/server.properties]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/opt/msm/servers/default/server.properties]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/Service[msm]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/Service[msm]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/msm-backup-working-files-to-s3.py]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/msm-backup-working-files-to-s3.py]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /File[/usr/local/bin/msm-restore-working-files-from-s3.sh]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /File[/usr/local/bin/msm-restore-working-files-from-s3.sh]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm/File[/usr/local/bin/msm]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm/File[/usr/local/bin/msm]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/File[/etc/init.d/msm-log-rotate]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/File[/etc/init.d/msm-log-rotate]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/Exec[update-rc.d -f msm-log-rotate remove]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/Exec[update-rc.d -f msm-log-rotate remove]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/Exec[update-rc.d msm-log-rotate defaults 96 04]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/Exec[update-rc.d msm-log-rotate defaults 96 04]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/Package[supervisor]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/Package[supervisor]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/File[/etc/supervisor/conf.d/msm-redis-listener.conf]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/File[/etc/supervisor/conf.d/msm-redis-listener.conf]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: /Stage[main]/Msm_backup_restore/Service[supervisor]: Dependency Exec[pip-install-requirements] has failures: true
[ec2-174-129-151-62.compute-1.amazonaws.com] out: warning: /Stage[main]/Msm_backup_restore/Service[supervisor]: Skipping because of failed dependencies
[ec2-174-129-151-62.compute-1.amazonaws.com] out: info: Creating state file /var/lib/puppet/state/state.yaml
[ec2-174-129-151-62.compute-1.amazonaws.com] out: notice: Finished catalog run in 254.11 seconds
[ec2-174-129-151-62.compute-1.amazonaws.com] out:

Disconnecting from ec2-174-129-151-62.compute-1.amazonaws.com... done.
[localhost] local: git rev-parse --verify --short HEAD
Building AMI...
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI state: pending
AMI is ready.
AMI ID: ami-49017a20
AMI Name: Minecraft-Server-001
AMI Description: Built on 2013-07-24 from cbccb8b
Terminating instance...
Instance state: shutting-down
Instance state: shutting-down
Instance terminated.

I've pulled a few of the changes from dmotles into my fork. Makes me realize that, short of building a whole new AMI, I'm not clear on recommended practice for updating the deployed instance. Did I miss something in the notes / obvious ?

Currently, the build-ami.py script creates a Minecraft world, then saves it to S3, so that the Minecraft world is restored next time a Minecloud EC2 instance is launched. That works fine the first time you run the build-ami.py script.

The problem arises when you've already been using Minecloud for awhile. When you play, you launch a server and it restores the world from S3. When you shut down the server, it saves the world to S3.

Now, you want to build an updated AMI, perhaps because the Minecloud-AMI project has been updated, or Ubuntu security patches have been released. But, when you run build-ami.py to create the new AMI, it creates a new default Minecraft world, and then saves it to your S3 bucket, overwriting your already existing world.

(Note that this doesn't overwrite the backups saved to /opt/msm/archives in the S3 bucket, so it is possible to recover your previous world.)

Small data files seem to be backed up to S3 with no problem, but the data files for large Minecraft worlds are sometimes incomplete on S3.

Following the instructions in the README, when I get to the point where I run build-ami.py, I get 400 Bad Request from AWS.

It is possible this is related to IAM. I did not set up a new account for this, because I already had one. I tried to set up a new user, but I am unsure what the policy should be. I gave my minecloud user (by way of my minecraft group) the AmazonEC2FullAccess policy. Is that sufficient?

(venv) $ ./build-ami.py 

Launching instance of ami-fb68f8cb...
Traceback (most recent call last):
  File "./build-ami.py", line 147, in <module>
    main()
  File "./build-ami.py", line 135, in main
    instance = launch_instance()
  File "./build-ami.py", line 33, in launch_instance
    user_data=env.ec2_userdata)
  File "/Users/jon/development/minecloud/minecloud/venv/lib/python2.7/site-packages/boto/ec2/connection.py", line 722, in run_instances
    verb='POST')
  File "/Users/jon/development/minecloud/minecloud/venv/lib/python2.7/site-packages/boto/connection.py", line 1062, in get_object
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Value () for parameter groupId is invalid. The value cannot be empty</Message></Error></Errors><RequestID>78c86641-9fa9-4a9e-b922-3e71a5f91759</RequestID></Response>

I'm trying to start a Minecloud server for the first time. The Minecloud AMI is up and running (I can see it in my AWS console) and the instance's public IP corresponds to the server IP listed in the Django app. However, the app is stuck at "Now, it's restoring saved game data. Almost ready..." and I can't join the server in Minecraft. However, I can SSH in. And yess, the minecraft security group in EC2 allows all traffic on port 22 and port 25565. The associated S3 bucket is empty, so I can only assume that the server isn't initializing the game world. Is this a problem with how I configured the server, or a bug? How should I go about initializing game data on the AMI instance ?

I'm trying to run build-ami.py using Python 2.7 in Linux on Windows (Ubuntu). I had to do some extra steps that may be relevant, like installing PIP manually, but I wouldn't expect that to be a problem. Here's the error I received (pastebin since github's formatting is all janky):

Launching instance of ami-fb68f8cb... Traceback (most recent call last): File "build-ami.py", line 147, in <module> main() File "build-ami.py", line 135, in main instance = launch_instance() File "build-ami.py", line 33, in launch_instance user_data=env.ec2_userdata) File "/home/super/minecloud-project/venv/lib/python2.7/site-packages/boto/ec2/connection.py", line 722, in run_instances verb='POST') File "/home/super/minecloud-project/venv/lib/python2.7/site-packages/boto/connection.py", line 1048, in get_object response = self.make_request(action, params, path, verb) File "/home/super/minecloud-project/venv/lib/python2.7/site-packages/boto/connection.py", line 974, in make_request return self._mexe(http_request) File "/home/super/minecloud-project/venv/lib/python2.7/site-packages/boto/connection.py", line 894, in _mexe raise e ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)

Thoughts?

Because of the new way Mojang distributes updated versions of the Minecraft server, the minecraft-server-manager script we rely on is not getting the latest version of Minecraft.

See issue 168 for minecraft-server-manager for details: msmhq/msm#168

I first noticed this error after the release of pip v. 1.4.

Here's the relevant issue on the pip issue tracker:

• pypa/pip#1074

Here's the debug info:

debug: Exec[pip-install-requirements](provider=posix): Executing '/usr/local/venv/bin/pip install -r /usr/local/venv/requirements.txt'
debug: Executing '/usr/local/venv/bin/pip install -r /usr/local/venv/requirements.txt'
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns: Traceback (most recent call last):
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/local/venv/bin/pip", line 9, in <module>
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     load_entry_point('pip==1.4', 'console_scripts', 'pip')()
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/local/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 378, in load_entry_point
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     return get_distribution(dist).load_entry_point(group, name)
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/local/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 2566, in load_entry_point
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     return ep.load()
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/local/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     entry = __import__(self.module_name, globals(),globals(), ['__name__'])
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/local/venv/local/lib/python2.7/site-packages/pip/__init__.py", line 11, in <module>
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     from pip.vcs import git, mercurial, subversion, bazaar  # noqa
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/local/venv/local/lib/python2.7/site-packages/pip/vcs/subversion.py", line 4, in <module>
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     from pip.index import Link
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/local/venv/local/lib/python2.7/site-packages/pip/index.py", line 32, in <module>
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     from pip.wheel import Wheel, wheel_ext, wheel_setuptools_support, setuptools_requirement
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/local/venv/local/lib/python2.7/site-packages/pip/wheel.py", line 18, in <module>
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     from pip import pep425tags
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/local/venv/local/lib/python2.7/site-packages/pip/pep425tags.py", line 98, in <module>
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     supported_tags = get_supported()
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/local/venv/local/lib/python2.7/site-packages/pip/pep425tags.py", line 61, in get_supported
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     soabi = sysconfig.get_config_var('SOABI')
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/lib/python2.7/sysconfig.py", line 577, in get_config_var
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     return get_config_vars().get(name)
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/lib/python2.7/sysconfig.py", line 476, in get_config_vars
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     _init_posix(_CONFIG_VARS)
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:   File "/usr/lib/python2.7/sysconfig.py", line 344, in _init_posix
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns:     raise IOError(msg)
notice: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns: IOError: invalid Python installation: unable to open /usr/venv/lib/python2.7/config/Makefile (No such file or directory)
err: /Stage[second]/Virtualenv/Exec[pip-install-requirements]/returns: change from notrun to 0 failed: /usr/local/venv/bin/pip install -v -r /usr/local/venv/requirements.txt returned 1 instead of one of [0] at /home/ubuntu/puppet-minecraft/modules/virtualenv/manifests/init.pp:44