Page MenuHomePhabricator

procutil: avoid using os.fork() to implement runbgcommand

Authored by valentin.gatienbaron on Sep 14 2020, 11:07 AM.



We ran into the following deadlock:

  • some command creates an ssh peer, then raises without explicitly closing the peer (hg id + extension in our case)
  • dispatch catches the exception, calls ui.log('commandfinish', ..) (the sshpeer is still not closed), which calls logtoprocess, which calls procutil.runbgcommand.
  • in the child of runbgcommand's fork(), between the fork and the exec, the opening of file descriptors triggers a gc which runs the destructor for sshpeer, which waits on ssh's stderr being closed, which never happens since ssh's stderr is held open by the parent of the fork where said destructor hasn't run

Remotefilelog appears to have a hack around this deadlock as well.

I don't know if there's more subtlety to it, because even though the
problem is determistic, it is very fragile, so I didn't manage to
reduce it.

I can imagine three ways of tackling this problem:

  1. don't run any python between fork and exec in runbgcommand
  2. make the finalizer harmless after the fork
  3. close the peer without relying on gc behavior

This commit goes with 1, as forking without exec'ing is tricky in
general in a language with gc finalizers. And maybe it's better in the
presence of rust threads. A future commit will try 2 or 3.

Performance wise: at low memory usage, it's an improvement. At higher
memory usage, it's about 2x faster than before when ensurestart=True,
but 2x slower when ensurestart=False. Not sure if that matters. The
reason for that last bit is that the subprocess.Popen always waits for
the execve to finish, and at high memory usage, execve is slow because
it deallocates the large page table. Numbers and script:

				 before after
    mem=1.0GB, ensurestart=True  52.1ms 26.0ms
    mem=1.0GB, ensurestart=False 14.7ms 26.0ms
    mem=0.5GB, ensurestart=True  23.2ms 11.2ms
    mem=0.5GB, ensurestart=False  6.2ms 11.3ms
    mem=0.2GB, ensurestart=True  15.7ms 7.4ms
    mem=0.2GB, ensurestart=False  4.3ms 8.1ms
    mem=0.0GB, ensurestart=True   2.3ms 0.7ms
    mem=0.0GB, ensurestart=False  0.8ms 0.8ms

    import time
    for memsize in [1_000_000_000, 500_000_000, 250_000_000, 0]:
        mem = 'a' * memsize
        for ensurestart in [True, False]:
            now = time.time()
            n = 100
            for i in range(n):
                procutil.runbgcommand([b'true'], {}, ensurestart=ensurestart)
            after = time.time()
            ms = (after - now) / float(n) * 1000
            print(f'mem={memsize / 1e9:.1f}GB, ensurestart={ensurestart} -> {ms:.1f}ms')

Diff Detail

rHG Mercurial
Automatic diff as part of commit; lint not applicable.
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

This mostly goes over my head. @durin42 @indygreg seems like something which needs your eyes.

I don't love the performance loss when ensurestart=False (that'll show up in some performance metrics at Google, but 10ms is livable for the correctness win).

Is there some way we could make this work on both Python 2 and 3 and just let it be suboptimally slow on Python 2? I don't think we're (despite my desires) ready to cut Python 2 loose. :(

Ok. When I sent this, I thought the py3 switch was further along than it seems to be. I should be able to make this work on py2 (although I think it would merely reduce the chance of deadlock on py2).

The slowdown is a bit annoying, and while it doesn't have to happen in principle, I looked in the implementation of subprocess and what I remember is that there was no way to make python not wait for the child to finish its execve.

I rebased, but there's nothing new here (apart from conflict resolution).
The discussion for py2 support was not super conclusive. But even given that D7258 uses py3 by default except on windows, and the fact the code changed here doesn't run on windows, I think I still need to make this change work with py2, so py2 issues from windows users can still be investigated from linux.

Gentle ping on this. I read the discussion and it does not seems clear to me what is holding it back.

What's holding this back is lack of py2 compatibility. Although maybe I want to wait it out at this point. Do we expect to drop py2 in a few days, after 5.7 is released for instance?

dropping of python2 compatibility is still stuck on having proper packaging (and compatibility) for Tortoise HG.

valentin.gatienbaron edited the summary of this revision. (Show Details)Feb 14 2021, 8:47 PM
valentin.gatienbaron updated this revision to Diff 25626.

I made the command trivially compatible with py2, by keeping both implementations around. It seems ok, under the assumption that the the py2 has a limited lifetime (until py2 support is dropped), and that it doesn't affect the windows code path (so it affects OSes that should only be using py3).

Alphare added a subscriber: Alphare.Apr 1 2021, 8:59 AM

@durin42 Does this patch now look good with the py2 hard fix? I can take the time to look it over if you want an extra pair of eyes.

It looks superficially correct - I didn't give it a detailed
inspection, but I trust you and Valentin on this.

Alphare accepted this revision.Apr 9 2021, 5:22 AM

This looks right to the best of my knowledge. Regarding the py2/py3 split, duplication like this is fine, we'll be dropping py2 soon enough.
Thanks for the detailed report and fix Valentin.

pulkit accepted this revision.Apr 11 2021, 5:28 PM
This revision is now accepted and ready to land.Apr 11 2021, 5:28 PM