( )⚙ D8465 automation: always use latest Windows AMI

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8465

automation: always use latest Windows AMI
ClosedPublic

Authored by indygreg on Apr 21 2020, 10:39 PM.

Details

Reviewers
marmoute
Group Reviewers
hg-reviewers
Commits
rHG828d3277618a: automation: always use latest Windows AMI
Summary

The old AMI isn't available any more.

We seem to run into this problem every few months. Amazon (or
Microsoft) appears to be removing the old AMIs when they are
superseded or something. Let's give up on tracking known images
and switch the image selection logic to use the latest published
image.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Apr 21 2020, 10:39 PM
marmoute accepted this revision.Apr 22 2020, 12:25 PM
marmoute added a subscriber: marmoute.

looks fine

indygreg added a commit: rHG828d3277618a: automation: always use latest Windows AMI.Apr 23 2020, 2:04 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG828d3277618a: automation: always use latest Windows AMI (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 21197

contrib/automation/hgautomation/aws.py

AMAZON_ACCOUNT_ID = '801119661308' AMAZON_ACCOUNT_ID = '801119661308'
DEBIAN_ACCOUNT_ID = '379101102735' DEBIAN_ACCOUNT_ID = '379101102735'
DEBIAN_ACCOUNT_ID_2 = '136693071363' DEBIAN_ACCOUNT_ID_2 = '136693071363'
UBUNTU_ACCOUNT_ID = '099720109477' UBUNTU_ACCOUNT_ID = '099720109477'
WINDOWS_BASE_IMAGE_NAME = 'Windows_Server-2019-English-Full-Base-2019.11.13' WINDOWS_BASE_IMAGE_NAME = 'Windows_Server-2019-English-Full-Base-*'
KEY_PAIRS = { KEY_PAIRS = {
'automation', 'automation',
} }
SECURITY_GROUPS = { SECURITY_GROUPS = {
print('removing role %s from %s' % (role, profile.name)) print('removing role %s from %s' % (role, profile.name))
profile.remove_role(RoleName=role) profile.remove_role(RoleName=role)
for role in sorted(wanted - have): for role in sorted(wanted - have):
print('adding role %s to %s' % (role, profile.name)) print('adding role %s to %s' % (role, profile.name))
profile.add_role(RoleName=role) profile.add_role(RoleName=role)
def find_image(ec2resource, owner_id, name): def find_image(ec2resource, owner_id, name, reverse_sort_field=None):
"""Find an AMI by its owner ID and name.""" """Find an AMI by its owner ID and name."""
images = ec2resource.images.filter( images = ec2resource.images.filter(
Filters=[ Filters=[
{'Name': 'owner-id', 'Values': [owner_id],}, {'Name': 'owner-id', 'Values': [owner_id],},
{'Name': 'state', 'Values': ['available'],}, {'Name': 'state', 'Values': ['available'],},
{'Name': 'image-type', 'Values': ['machine'],}, {'Name': 'image-type', 'Values': ['machine'],},
{'Name': 'name', 'Values': [name],}, {'Name': 'name', 'Values': [name],},
] ]
) )
if reverse_sort_field:
images = sorted(
images,
key=lambda image: getattr(image, reverse_sort_field),
reverse=True,
)
for image in images: for image in images:
return image return image
raise Exception('unable to find image for %s' % name) raise Exception('unable to find image for %s' % name)
def ensure_security_groups(ec2resource, prefix='hg-'): def ensure_security_groups(ec2resource, prefix='hg-'):
"""Ensure all necessary Mercurial security groups are present. """Ensure all necessary Mercurial security groups are present.
try: try:
yield instances yield instances
finally: finally:
for instance in instances: for instance in instances:
instance.ssh_client.close() instance.ssh_client.close()
def ensure_windows_dev_ami( def ensure_windows_dev_ami(
c: AWSConnection, prefix='hg-', base_image_name=WINDOWS_BASE_IMAGE_NAME c: AWSConnection, prefix='hg-', base_image_name=WINDOWS_BASE_IMAGE_NAME,
): ):
"""Ensure Windows Development AMI is available and up-to-date. """Ensure Windows Development AMI is available and up-to-date.
If necessary, a modern AMI will be built by starting a temporary EC2 If necessary, a modern AMI will be built by starting a temporary EC2
instance and bootstrapping it. instance and bootstrapping it.
Obsolete AMIs will be deleted so there is only a single AMI having the Obsolete AMIs will be deleted so there is only a single AMI having the
desired name. desired name.
Returns an ``ec2.Image`` of either an existing AMI or a newly-built Returns an ``ec2.Image`` of either an existing AMI or a newly-built
one. one.
""" """
ec2client = c.ec2client ec2client = c.ec2client
ec2resource = c.ec2resource ec2resource = c.ec2resource
ssmclient = c.session.client('ssm') ssmclient = c.session.client('ssm')
name = '%s%s' % (prefix, 'windows-dev') name = '%s%s' % (prefix, 'windows-dev')
image = find_image(ec2resource, AMAZON_ACCOUNT_ID, base_image_name) image = find_image(
ec2resource,
AMAZON_ACCOUNT_ID,
base_image_name,
reverse_sort_field="name",
)
config = { config = {
'BlockDeviceMappings': [ 'BlockDeviceMappings': [
{ {
'DeviceName': '/dev/sda1', 'DeviceName': '/dev/sda1',
'Ebs': { 'Ebs': {
'DeleteOnTermination': True, 'DeleteOnTermination': True,
'VolumeSize': 32, 'VolumeSize': 32,