This is an archive of the discontinued Mercurial Phabricator instance.

ci: implement a new CI system for Mercurial
Changes PlannedPublic

Authored by indygreg on Sep 28 2019, 11:50 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Summary

tl;dr a new CI system built on top of AWS using mostly serverless
technology is implemented. It can be seen at
https://ci.hg.gregoryszorc.com/

Mercurial's existing CI infrastructure is under-loved. This commit
implements an in-repository CI system for Mercurial.

My main motiviation for doing this is that the transition to Python 3
will require a substantial investment in CI and I think this is a good
opportunity to revisit how the project currently does CI. The existing
CI system is rooted in CI technology from yesterday and I think we can
iterate faster going forward by implementing CI in the main repository
on top of infrastructure that anyone can stand up if they have a credit
card.

Everything in the new CI system is built on top of AWS. There are no
persistent EC2 instances: all compute is ephemeral in "serverless"
Lambda functions or performed on EC2 instances spun up to run
individual jobs.

I rejected building on top of existing CI platforms like Azure Pipelines
or GitHub Actions because a) support for Mercurial is poor b) they don't
provide enough CPU horsepower to run Mercurial's test harness quickly c)
if you want to buy enough CPU horsepower to do so, you need to run
persistent servers which end up sitting around idle most of the time,
accruing large bills. Standing up a basic CI system built on top of
simple event triggers (such as database row insertions triggering the
launch of an EC2 instance to run a job) is not a lot of lines of code,
surprisingly. And the benefits we get from containing the entire stack
and having everything be open source can be substantial.

The code in this commit is currently running at
https://ci.hg.gregoryszorc.com/ and is running out of an AWS account
I currently have chained up to my personal AWS account. That includes
billing: I'm currently running this out of my own pocket. I'm fine
with that for now. But we'll probably want to move account ownership
and billing to the official Mercurial Project at some point. Although
if you catch me in a generous mood, I may foot the bill semi-permanently.
Although it would be great to get the 501(c)(3) tax status hooked up
so I can write off the expense. That may mean transferring account
ownership to the Mercurial Project. We can discuss offline...

A lot of functionality is very rough around the edges. And there's a
lot that could be implemented. The web interface in particular is
rather primitive (I'm not great at web development). But it gets the
job done. I've already found the functionality very useful for tracking
down Python 3 issues.

This is a lot of code to review at once. I suggest starting by reading
the added README.rst. Most of the business logic lives in the Lambda
functions in ci.py. I tried to isolate the Terraform logic to
per-component files to make everything easier to grok.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Sep 28 2019, 11:50 PM
indygreg updated this revision to Diff 16677.Sep 29 2019, 12:24 AM
indygreg updated this revision to Diff 16678.Sep 29 2019, 12:50 PM
indygreg updated this revision to Diff 16682.Sep 29 2019, 3:07 PM
indygreg updated this revision to Diff 16695.Sep 29 2019, 3:57 PM
indygreg updated this revision to Diff 16697.Sep 29 2019, 11:31 PM
indygreg updated this revision to Diff 16716.Sep 30 2019, 11:57 PM
indygreg planned changes to this revision.Oct 16 2019, 10:50 PM

Since nobody seems to have plans to review this, I'll hold off on requesting review at this time.