( )⚙ D11982 stream-clone: stop considering working copy only requirements

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11982

stream-clone: stop considering working copy only requirements
ClosedPublic

Authored by marmoute on Jan 12 2022, 9:08 AM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHGeb5c33f1d08f: stream-clone: stop considering working copy only requirements
Summary

Now that the ducks are properly aligned on there shelve, it becomes easy to fix
the list of requirements to consider for stream. We remove the working copy
specific ones and add some missing revlog specific ones.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

marmoute created this revision.Jan 12 2022, 9:08 AM
marmoute planned changes to this revision.Jan 16 2022, 6:54 PM

While working on this code further I noticed something odd with the supportedformat/_basesupported differences and this changes might not be correct. At least not until I reach the end of this rabbit hole. The series depending on it is still fine to review as I do not expect changes to theses patches.

marmoute added a parent revision: D12084: stream-clone: also filter the requirement we put in the bundle 2.Jan 24 2022, 9:51 AM
marmoute retitled this revision from stream-clone: only consider store requirements to stream-clone: stop considering working copy only requirements.
marmoute edited the summary of this revision. (Show Details)
marmoute updated this revision to Diff 31788.
Alphare accepted this revision.Jan 24 2022, 11:21 AM
This revision is now accepted and ready to land.Jan 24 2022, 11:21 AM
baymax updated this revision to Diff 31850.Jan 24 2022, 1:11 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Revision Contents
Changeset List

Diff 31637

mercurial/bundle2.py

for n in sorted(nodes): for n in sorted(nodes):
yield n yield n
for n in sorted(closed): for n in sorted(closed):
yield n yield n
bundler.newpart(b'cache:rev-branch-cache', data=generate(), mandatory=False) bundler.newpart(b'cache:rev-branch-cache', data=generate(), mandatory=False)
def _formatrequirementsspec(requirements): def _formatrequirementsspec(requs):
requirements = [req for req in requirements if req != b"shared"] requs = [
return urlreq.quote(b','.join(sorted(requirements))) req for req in requs if req not in requirements.WORKING_DIR_REQUIREMENTS
]
return urlreq.quote(b','.join(sorted(requs)))
def _formatrequirementsparams(requirements): def _formatrequirementsparams(requirements):
requirements = _formatrequirementsspec(requirements) requirements = _formatrequirementsspec(requirements)
params = b"%s%s" % (urlreq.quote(b"requirements="), requirements) params = b"%s%s" % (urlreq.quote(b"requirements="), requirements)
return params return params

mercurial/streamclone.py

filecount, bytecount = map(int, l.split(b' ', 1)) filecount, bytecount = map(int, l.split(b' ', 1))
except (ValueError, TypeError): except (ValueError, TypeError):
raise error.ResponseError( raise error.ResponseError(
_(b'unexpected response from remote server:'), l _(b'unexpected response from remote server:'), l
) )
with repo.lock(): with repo.lock():
consumev1(repo, fp, filecount, bytecount) consumev1(repo, fp, filecount, bytecount)
# new requirements = old non-(store)-format requirements +
# new requirements = old non-format requirements + # new (store) format-related remote requirements
# new format-related remote requirements kept_requirements = set(repo.requirements)
# requirements from the streamed-in repository kept_requirements -= repo.supportedformats
repo.requirements = requirements | ( wc_req = requirementsmod.WORKING_DIR_REQUIREMENTS
repo.requirements - repo.supportedformats kept_requirements |= repo.requirements & wc_req
) repo.requirements = kept_requirements | set(requirements)
repo.svfs.options = localrepo.resolvestorevfsoptions( repo.svfs.options = localrepo.resolvestorevfsoptions(
repo.ui, repo.requirements, repo.features repo.ui, repo.requirements, repo.features
) )
scmutil.writereporequirements(repo) scmutil.writereporequirements(repo)
if rbranchmap: if rbranchmap:
repo._branchcaches.replace(repo, rbranchmap) repo._branchcaches.replace(repo, rbranchmap)
compressed in the future). compressed in the future).
Returns a tuple of (requirements, data generator). Returns a tuple of (requirements, data generator).
""" """
if compression != b'UN': if compression != b'UN':
raise ValueError(b'we do not support the compression argument yet') raise ValueError(b'we do not support the compression argument yet')
requirements = repo.requirements & repo.supportedformats requirements = repo.requirements & repo.supportedformats
requirements -= requirementsmod.WORKING_DIR_REQUIREMENTS
requires = b','.join(sorted(requirements)) requires = b','.join(sorted(requirements))
def gen(): def gen():
yield b'HGS1' yield b'HGS1'
yield compression yield compression
filecount, bytecount, it = generatev1(repo) filecount, bytecount, it = generatev1(repo)
repo.ui.status( repo.ui.status(
if missingreqs: if missingreqs:
raise error.Abort( raise error.Abort(
_(b'unable to apply stream clone: unsupported format: %s') _(b'unable to apply stream clone: unsupported format: %s')
% b', '.join(sorted(missingreqs)) % b', '.join(sorted(missingreqs))
) )
consumev2(repo, fp, filecount, filesize) consumev2(repo, fp, filecount, filesize)
# new requirements = old non-format requirements + # new requirements = old non-(store)-format requirements +
# new format-related remote requirements # new (store) format-related remote requirements
# requirements from the streamed-in repository # requirements from the streamed-in repository
repo.requirements = set(requirements) | ( kept_requirements = set(repo.requirements)
repo.requirements - repo.supportedformats kept_requirements -= repo.supportedformats
) wc_req = requirementsmod.WORKING_DIR_REQUIREMENTS
kept_requirements |= repo.requirements & wc_req
repo.requirements = kept_requirements | set(requirements)
repo.svfs.options = localrepo.resolvestorevfsoptions( repo.svfs.options = localrepo.resolvestorevfsoptions(
repo.ui, repo.requirements, repo.features repo.ui, repo.requirements, repo.features
) )
scmutil.writereporequirements(repo) scmutil.writereporequirements(repo)
def _copy_files(src_vfs_map, dst_vfs_map, entries, progress): def _copy_files(src_vfs_map, dst_vfs_map, entries, progress):
hardlink = [True] hardlink = [True]

mercurial/wireprotov1server.py

# transports. So conditionally add it. # transports. So conditionally add it.
if commands.commandavailable(b'changegroupsubset', proto): if commands.commandavailable(b'changegroupsubset', proto):
caps.append(b'changegroupsubset') caps.append(b'changegroupsubset')
if streamclone.allowservergeneration(repo): if streamclone.allowservergeneration(repo):
if repo.ui.configbool(b'server', b'preferuncompressed'): if repo.ui.configbool(b'server', b'preferuncompressed'):
caps.append(b'stream-preferred') caps.append(b'stream-preferred')
requiredformats = repo.requirements & repo.supportedformats requiredformats = repo.requirements & repo.supportedformats
requiredformats -= requirementsmod.WORKING_DIR_REQUIREMENTS
# if our local revlogs are just revlogv1, add 'stream' cap # if our local revlogs are just revlogv1, add 'stream' cap
if not requiredformats - {requirementsmod.REVLOGV1_REQUIREMENT}: if not requiredformats - {requirementsmod.REVLOGV1_REQUIREMENT}:
caps.append(b'stream') caps.append(b'stream')
# otherwise, add 'streamreqs' detailing our local revlog format # otherwise, add 'streamreqs' detailing our local revlog format
else: else:
caps.append(b'streamreqs=%s' % b','.join(sorted(requiredformats))) caps.append(b'streamreqs=%s' % b','.join(sorted(requiredformats)))
if repo.ui.configbool(b'experimental', b'bundle2-advertise'): if repo.ui.configbool(b'experimental', b'bundle2-advertise'):
capsblob = bundle2.encodecaps(bundle2.getrepocaps(repo, role=b'server')) capsblob = bundle2.encodecaps(bundle2.getrepocaps(repo, role=b'server'))

tests/test-clone-stream.t

streaming all changes streaming all changes
remote: abort: server has obsolescence markers, but client cannot receive them via stream clone remote: abort: server has obsolescence markers, but client cannot receive them via stream clone
abort: pull failed on remote abort: pull failed on remote
[100] [100]
$ killdaemons.py $ killdaemons.py
#endif #endif
Stream clone regarding non-store format variant
-----------------------------------------------
$ cat << EOF >> $HGRCPATH
> [storage]
> dirstate-v2.slow-path=allow
> EOF
We can add dirstate-v2 when cloning (or any other non-store format change actually)
$ hg init server-no-dirstate-v2 --config format.exp-rc-dirstate-v2=no
$ hg debugbuilddag .+5 -R server-no-dirstate-v2
$ hg serve -p $HGPORT -d --pid-file=hg.pid -R server-no-dirstate-v2
$ cat hg.pid > $DAEMON_PIDS
$ hg debugcapabilities http://localhost:$HGPORT | grep streamreqs
streamreqs=generaldelta,revlogv1,sparserevlog (no-rust !)
streamreqs=generaldelta,persistent-nodemap,revlogv1,sparserevlog (rust !)
$ hg clone --quiet -U --stream --config format.exp-rc-dirstate-v2=yes http://localhost:$HGPORT clone-added-d2
$ hg debugrequirement -R server-no-dirstate-v2 | grep dirstate-v2
[1]
$ hg debugformat -R server-no-dirstate-v2 | grep dirstate-v2
dirstate-v2: no
$ hg debugrequirement -R clone-added-d2 | grep dirstate-v2
dirstate-v2
$ hg debugformat -R clone-added-d2 | grep dirstate-v2
dirstate-v2: yes
$ killdaemons.py
We can drop it when cloning too.
$ hg init server-dirstate-v2 --config format.exp-rc-dirstate-v2=yes
$ hg debugbuilddag .+5 -R server-dirstate-v2
$ hg serve -p $HGPORT -d --pid-file=hg.pid -R server-dirstate-v2
$ cat hg.pid > $DAEMON_PIDS
$ hg debugcapabilities http://localhost:$HGPORT | grep streamreqs
streamreqs=generaldelta,revlogv1,sparserevlog (no-rust !)
streamreqs=generaldelta,persistent-nodemap,revlogv1,sparserevlog (rust !)
$ hg clone --quiet -U --stream --config format.exp-rc-dirstate-v2=no http://localhost:$HGPORT clone-dropped-d2
$ hg debugrequirement -R server-dirstate-v2 | grep dirstate-v2
dirstate-v2
$ hg debugformat -R server-dirstate-v2 | grep dirstate-v2
dirstate-v2: yes
$ hg debugrequirement -R clone-dropped-d2 | grep dirstate-v2
[1]
$ hg debugformat -R clone-dropped-d2 | grep dirstate-v2
dirstate-v2: no
$ killdaemons.py