This is an archive of the discontinued Mercurial Phabricator instance.

Differential D6288

automation: shore up rebooting behavior
ClosedPublic

Authored by indygreg on Apr 19 2019, 10:41 AM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGe570106beda1: automation: shore up rebooting behavior
Summary

There was a race condition in the old code. Use
instance.stop()/instance.start() to eliminate it.

As part of debugging this, I also found another race condition
related to PowerShell permissions after the reboot. Unfortunately,
I'm not sure the best way to work around it. I've added a comment
for now.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Apr 19 2019, 10:41 AM
Closed by commit rHGe570106beda1: automation: shore up rebooting behavior (authored by indygreg). · Explain WhyMay 15 2019, 11:46 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 14867

contrib/automation/hgautomation/aws.py

[instance], [instance],
'AWS-RunPowerShellScript', 'AWS-RunPowerShellScript',
{ {
'commands': WINDOWS_BOOTSTRAP_POWERSHELL.split('\n'), 'commands': WINDOWS_BOOTSTRAP_POWERSHELL.split('\n'),
}, },
) )
# Reboot so all updates are fully applied. # Reboot so all updates are fully applied.
#
# We don't use instance.reboot() here because it is asynchronous and
# we don't know when exactly the instance has rebooted. It could take
# a while to stop and we may start trying to interact with the instance
# before it has rebooted.
print('rebooting instance %s' % instance.id) print('rebooting instance %s' % instance.id)
ec2client.reboot_instances(InstanceIds=[instance.id]) instance.stop()
ec2client.get_waiter('instance_stopped').wait(
InstanceIds=[instance.id],
WaiterConfig={
'Delay': 5,
})
instance.start()
wait_for_ip_addresses([instance])
time.sleep(15) # There is a race condition here between the User Data PS script running
# and us connecting to WinRM. This can manifest as
# "AuthorizationManager check failed" failures during run_powershell().
# TODO figure out a workaround.
print('waiting for Windows Remote Management to come back...') print('waiting for Windows Remote Management to come back...')
client = wait_for_winrm(instance.public_ip_address, 'Administrator', client = wait_for_winrm(instance.public_ip_address, 'Administrator',
c.automation.default_password()) c.automation.default_password())
print('established WinRM connection to %s' % instance.id) print('established WinRM connection to %s' % instance.id)
instance.winrm_client = client instance.winrm_client = client
print('bootstrapping instance...') print('bootstrapping instance...')