This is an archive of the discontinued Mercurial Phabricator instance.

Differential D6983

automation: add a command to submit to a Try server
ClosedPublic

Authored by indygreg on Oct 5 2019, 12:03 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGc5c502bd1f70: automation: add a command to submit to a Try server
Summary

The CI code for running the Try Server requires more thorough review.
Let's add just the client-side bits for submitting to Try so others
can start using it.

Diff Detail

Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Oct 5 2019, 12:03 PM
indygreg updated this revision to Diff 16839.Oct 5 2019, 1:10 PM
indygreg added a commit: rHGc5c502bd1f70: automation: add a command to submit to a Try server.Oct 5 2019, 2:29 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGc5c502bd1f70: automation: add a command to submit to a Try server (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 16839

contrib/automation/README.rst

in the ``~/.hgautomation`` directory. in the ``~/.hgautomation`` directory.
We attempt to limit persistent state to this directory. Even when We attempt to limit persistent state to this directory. Even when
performing tasks that may have side-effects, we try to limit those performing tasks that may have side-effects, we try to limit those
side-effects so they don't impact the local system. e.g. when we SSH side-effects so they don't impact the local system. e.g. when we SSH
into a remote machine, we create a temporary directory for the SSH into a remote machine, we create a temporary directory for the SSH
config so the user's known hosts file isn't updated. config so the user's known hosts file isn't updated.
Try Server
==========
There exists a *Try Server* which allows automation to run against
an arbitrary Mercurial changeset and displays results via the web.
.. note::
The *Try Server* is still experimental infrastructure.
To use the *Try Server*::
$ ./automation.py try
With a custom AWS profile::
$ AWS_PROFILE=hg contrib/automation/automation.py try
By default, the ``.`` revision is submitted. **Any uncommitted changes
are not submitted.**
To switch which revision is used::
$ ./automation.py try -r abcdef
Access to the *Try Server* requires access to a special AWS account.
This account is currently run by Gregory Szorc. Here is the procedure
for accessing the *Try Server*:
1. Email Gregory Szorc at gregory.szorc@gmail.com and request a
username. This username will be stored in the public domain.
2. Wait for an email reply containing your temporary AWS credentials.
3. Log in at https://gregoryszorc-hg.signin.aws.amazon.com/console
and set a new, secure password.
4. Go to https://console.aws.amazon.com/iam/home?region=us-west-2#/security_credentials
5. Under ``Access keys for CLI, SDK, & API access``, click the
``Create access key`` button.
6. See the ``AWS Integration`` section for instructions on
configuring your local client to use the generated credentials.
AWS Integration AWS Integration
=============== ===============
Various automation tasks integrate with AWS to provide access to Various automation tasks integrate with AWS to provide access to
resources such as EC2 instances for generic compute. resources such as EC2 instances for generic compute.
This obviously requires an AWS account and credentials to work. This obviously requires an AWS account and credentials to work.

contrib/automation/hgautomation/cli.py

import os import os
import pathlib import pathlib
import time import time
from . import ( from . import (
aws, aws,
HGAutomation, HGAutomation,
linux, linux,
try_server,
windows, windows,
) )
SOURCE_ROOT = pathlib.Path(os.path.abspath(__file__)).parent.parent.parent.parent SOURCE_ROOT = pathlib.Path(os.path.abspath(__file__)).parent.parent.parent.parent
DIST_PATH = SOURCE_ROOT / 'dist' DIST_PATH = SOURCE_ROOT / 'dist'
def publish_windows_artifacts(hg: HGAutomation, aws_region, version: str, def publish_windows_artifacts(hg: HGAutomation, aws_region, version: str,
pypi: bool, mercurial_scm_org: bool, pypi: bool, mercurial_scm_org: bool,
ssh_username: str): ssh_username: str):
windows.publish_artifacts(DIST_PATH, version, windows.publish_artifacts(DIST_PATH, version,
pypi=pypi, mercurial_scm_org=mercurial_scm_org, pypi=pypi, mercurial_scm_org=mercurial_scm_org,
ssh_username=ssh_username) ssh_username=ssh_username)
def run_try(hga: HGAutomation, aws_region: str, rev: str):
c = hga.aws_connection(aws_region, ensure_ec2_state=False)
try_server.trigger_try(c, rev=rev)
def get_parser(): def get_parser():
parser = argparse.ArgumentParser() parser = argparse.ArgumentParser()
parser.add_argument( parser.add_argument(
'--state-path', '--state-path',
default='~/.hgautomation', default='~/.hgautomation',
help='Path for local state files', help='Path for local state files',
) )
help='SSH username for mercurial-scm.org', help='SSH username for mercurial-scm.org',
) )
sp.add_argument( sp.add_argument(
'version', 'version',
help='Mercurial version string to locate local packages', help='Mercurial version string to locate local packages',
) )
sp.set_defaults(func=publish_windows_artifacts) sp.set_defaults(func=publish_windows_artifacts)
sp = subparsers.add_parser(
'try',
help='Run CI automation against a custom changeset'
)
sp.add_argument('-r', '--rev',
default='.',
help='Revision to run CI on')
sp.set_defaults(func=run_try)
return parser return parser
def main(): def main():
parser = get_parser() parser = get_parser()
args = parser.parse_args() args = parser.parse_args()
local_state_path = pathlib.Path(os.path.expanduser(args.state_path)) local_state_path = pathlib.Path(os.path.expanduser(args.state_path))

contrib/automation/hgautomation/try_server.py

  • This file was added.
# try_server.py - Interact with Try server
#
# Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
# no-check-code because Python 3 native.
import base64
import json
import os
import subprocess
import tempfile
from .aws import AWSConnection
LAMBDA_FUNCTION = "ci-try-server-upload"
def trigger_try(c: AWSConnection, rev="."):
"""Trigger a new Try run."""
lambda_client = c.session.client("lambda")
cset, bundle = generate_bundle(rev=rev)
payload = {
"bundle": base64.b64encode(bundle).decode("utf-8"),
"node": cset["node"],
"branch": cset["branch"],
"user": cset["user"],
"message": cset["desc"],
}
print("resolved revision:")
print("node: %s" % cset["node"])
print("branch: %s" % cset["branch"])
print("user: %s" % cset["user"])
print("desc: %s" % cset["desc"].splitlines()[0])
print()
print("sending to Try...")
res = lambda_client.invoke(
FunctionName=LAMBDA_FUNCTION,
InvocationType="RequestResponse",
Payload=json.dumps(payload).encode("utf-8"),
)
body = json.load(res["Payload"])
for message in body:
print("remote: %s" % message)
def generate_bundle(rev="."):
"""Generate a bundle suitable for use by the Try service.
Returns a tuple of revision metadata and raw Mercurial bundle data.
"""
# `hg bundle` doesn't support streaming to stdout. So we use a temporary
# file.
path = None
try:
fd, path = tempfile.mkstemp(prefix="hg-bundle-", suffix=".hg")
os.close(fd)
args = [
"hg",
"bundle",
"--type",
"gzip-v2",
"--base",
"public()",
"--rev",
rev,
path,
]
print("generating bundle...")
subprocess.run(args, check=True)
with open(path, "rb") as fh:
bundle_data = fh.read()
finally:
if path:
os.unlink(path)
args = [
"hg",
"log",
"-r",
rev,
# We have to upload as JSON, so it won't matter if we emit binary
# since we need to normalize to UTF-8.
"-T",
"json",
]
res = subprocess.run(args, check=True, capture_output=True)
return json.loads(res.stdout)[0], bundle_data

tests/test-check-code.t

> -X mercurial/thirdparty \ > -X mercurial/thirdparty \
> | sed 's-\\-/-g' | "$check_code" --warnings --per-file=0 - || false > | sed 's-\\-/-g' | "$check_code" --warnings --per-file=0 - || false
Skipping contrib/automation/hgautomation/__init__.py it has no-che?k-code (glob) Skipping contrib/automation/hgautomation/__init__.py it has no-che?k-code (glob)
Skipping contrib/automation/hgautomation/aws.py it has no-che?k-code (glob) Skipping contrib/automation/hgautomation/aws.py it has no-che?k-code (glob)
Skipping contrib/automation/hgautomation/cli.py it has no-che?k-code (glob) Skipping contrib/automation/hgautomation/cli.py it has no-che?k-code (glob)
Skipping contrib/automation/hgautomation/linux.py it has no-che?k-code (glob) Skipping contrib/automation/hgautomation/linux.py it has no-che?k-code (glob)
Skipping contrib/automation/hgautomation/pypi.py it has no-che?k-code (glob) Skipping contrib/automation/hgautomation/pypi.py it has no-che?k-code (glob)
Skipping contrib/automation/hgautomation/ssh.py it has no-che?k-code (glob) Skipping contrib/automation/hgautomation/ssh.py it has no-che?k-code (glob)
Skipping contrib/automation/hgautomation/try_server.py it has no-che?k-code (glob)
Skipping contrib/automation/hgautomation/windows.py it has no-che?k-code (glob) Skipping contrib/automation/hgautomation/windows.py it has no-che?k-code (glob)
Skipping contrib/automation/hgautomation/winrm.py it has no-che?k-code (glob) Skipping contrib/automation/hgautomation/winrm.py it has no-che?k-code (glob)
Skipping contrib/packaging/hgpackaging/downloads.py it has no-che?k-code (glob) Skipping contrib/packaging/hgpackaging/downloads.py it has no-che?k-code (glob)
Skipping contrib/packaging/hgpackaging/inno.py it has no-che?k-code (glob) Skipping contrib/packaging/hgpackaging/inno.py it has no-che?k-code (glob)
Skipping contrib/packaging/hgpackaging/py2exe.py it has no-che?k-code (glob) Skipping contrib/packaging/hgpackaging/py2exe.py it has no-che?k-code (glob)
Skipping contrib/packaging/hgpackaging/util.py it has no-che?k-code (glob) Skipping contrib/packaging/hgpackaging/util.py it has no-che?k-code (glob)
Skipping contrib/packaging/hgpackaging/wix.py it has no-che?k-code (glob) Skipping contrib/packaging/hgpackaging/wix.py it has no-che?k-code (glob)
Skipping contrib/packaging/inno/build.py it has no-che?k-code (glob) Skipping contrib/packaging/inno/build.py it has no-che?k-code (glob)