tl;dr a new CI system built on top of AWS using mostly serverless
technology is implemented. It can be seen at
Mercurial's existing CI infrastructure is under-loved. This commit
implements an in-repository CI system for Mercurial.
My main motiviation for doing this is that the transition to Python 3
will require a substantial investment in CI and I think this is a good
opportunity to revisit how the project currently does CI. The existing
CI system is rooted in CI technology from yesterday and I think we can
iterate faster going forward by implementing CI in the main repository
on top of infrastructure that anyone can stand up if they have a credit
Everything in the new CI system is built on top of AWS. There are no
persistent EC2 instances: all compute is ephemeral in "serverless"
Lambda functions or performed on EC2 instances spun up to run
I rejected building on top of existing CI platforms like Azure Pipelines
or GitHub Actions because a) support for Mercurial is poor b) they don't
provide enough CPU horsepower to run Mercurial's test harness quickly c)
if you want to buy enough CPU horsepower to do so, you need to run
persistent servers which end up sitting around idle most of the time,
accruing large bills. Standing up a basic CI system built on top of
simple event triggers (such as database row insertions triggering the
launch of an EC2 instance to run a job) is not a lot of lines of code,
surprisingly. And the benefits we get from containing the entire stack
and having everything be open source can be substantial.
The code in this commit is currently running at
https://ci.hg.gregoryszorc.com/ and is running out of an AWS account
I currently have chained up to my personal AWS account. That includes
billing: I'm currently running this out of my own pocket. I'm fine
with that for now. But we'll probably want to move account ownership
and billing to the official Mercurial Project at some point. Although
if you catch me in a generous mood, I may foot the bill semi-permanently.
Although it would be great to get the 501(c)(3) tax status hooked up
so I can write off the expense. That may mean transferring account
ownership to the Mercurial Project. We can discuss offline...
A lot of functionality is very rough around the edges. And there's a
lot that could be implemented. The web interface in particular is
rather primitive (I'm not great at web development). But it gets the
job done. I've already found the functionality very useful for tracking
down Python 3 issues.
This is a lot of code to review at once. I suggest starting by reading
the added README.rst. Most of the business logic lives in the Lambda
functions in ci.py. I tried to isolate the Terraform logic to
per-component files to make everything easier to grok.