( )⚙ D4713 largefiles: automatically load largefiles extension when required (BC)

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4713

largefiles: automatically load largefiles extension when required (BC)
ClosedPublic

Authored by indygreg on Sep 24 2018, 12:12 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG823a580448d7: largefiles: automatically load largefiles extension when required (BC)
Summary

This is very similar to what we just did for LFS but for largefiles.
See recent commit messages for the rationale here.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Sep 24 2018, 12:12 PM
Closed by commit rHG823a580448d7: largefiles: automatically load largefiles extension when required (BC) (authored by indygreg). · Explain WhySep 26 2018, 1:24 PM
This revision was automatically updated to reflect the committed changes.
yuja added a subscriber: yuja.Sep 28 2018, 10:25 PM
  1. Map of requirements to list of extensions to load automatically when
  2. requirement is present. autoextensions = {

+ b'largefiles': [b'largefiles'],

    b'lfs': [b'lfs'],
}

Can we add some warnings here? The largefiles is IMHO one of the most buggy
extensions, and loading it has a side effect (e.g. fsmonitor is disabled.)
It shouldn't be silently loaded into the process.

indygreg added a comment.Oct 1 2018, 12:18 PM

I didn't realize largefiles is riddled with bugs. I'd be happy to drop this changeset if there are concerns about its behavior.

mharbison72 added a subscriber: mharbison72.Oct 2 2018, 1:22 AM
In D4713#72604, @yuja wrote:
  1. Map of requirements to list of extensions to load automatically when
  2. requirement is present. autoextensions = {

+ b'largefiles': [b'largefiles'],

    b'lfs': [b'lfs'],
}

Can we add some warnings here? The largefiles is IMHO one of the most buggy
extensions, and loading it has a side effect (e.g. fsmonitor is disabled.)
It shouldn't be silently loaded into the process.

I get what you’re thinking, but maybe a debug message is better than a warning, even though it’s less visible. I’m concerned about every invocation being unnecessarily noisy now that .hg/hgrc isn’t updated, and noise on stderr triggers that thgw.exe dialog when UAC is enabled. OTOH, updating the hgrc with this would make for a quick and dirty one shot warning.

yuja added a comment.Oct 2 2018, 10:59 AM
> Can we add some warnings here? The largefiles is IMHO one of the most buggy
>  extensions, and loading it has a side effect (e.g. fsmonitor is disabled.)
>  It shouldn't be silently loaded into the process.
I get what you’re thinking, but maybe a debug message is better than a warning, even though it’s less visible. I’m concerned about every invocation being unnecessarily noisy now that .hg/hgrc isn’t updated,

I don't think a debug message would work because ordinary users will never
see it. Is that common to work on largefiles repo without enabling the
extension globally?

and noise on stderr triggers that thgw.exe dialog when UAC is enabled.

Well, that's the bug of TortoiseHg if it doesn't redirect ui.ferr.

FWIW, I like the idea of enabling storage extensions automatically.

mharbison72 added a comment.Oct 2 2018, 11:19 PM
In D4713#73077, @yuja wrote:
> Can we add some warnings here? The largefiles is IMHO one of the most buggy
>  extensions, and loading it has a side effect (e.g. fsmonitor is disabled.)
>  It shouldn't be silently loaded into the process.
I get what you’re thinking, but maybe a debug message is better than a warning, even though it’s less visible. I’m concerned about every invocation being unnecessarily noisy now that .hg/hgrc isn’t updated,

I don't think a debug message would work because ordinary users will never
see it. Is that common to work on largefiles repo without enabling the
extension globally?

The only data point I have is what I did, which was to only enable it on the command line when cloning. The clone command modifying the local hgrc if necessary handled the rest. I only started to do that because I got annoyed at how simply having it loaded could alter certain commands. IDK if command line enabling was ever documented as the preferred alternative, but that's how I described it in e1dbe0b215ae.

At this point, I can't recall specific issues with globally enabling, and they might be mostly fixed. They definitely aren't all fixed. That said, this implicit loading on demand seems roughly equivalent to the previous behavior, so I'm not sure that this would be any more surprising. Repos on the command line would be isolated, but not with commandserver because of the global function wrapping. But I'm guessing most people don't understand that about commandserver.

yuja added a comment.Oct 3 2018, 10:26 AM
The only data point I have is what I did, which was to only enable it on the command line when cloning.  The clone command modifying the local hgrc if necessary handled the rest.  I only started to do that because I got annoyed at how simply having it loaded could alter certain commands.  IDK if command line enabling was ever documented as the preferred alternative, but that's how I described it in https://phab.mercurial-scm.org/rHGe1dbe0b215ae137eec53ceb12440536d570a83d2.
At this point, I can't recall specific issues with globally enabling, and they might be mostly fixed.  They definitely aren't all fixed.  That said, this implicit loading on demand seems roughly equivalent to the previous behavior, so I'm not sure that this would be any more surprising.  Repos on the command line would be isolated, but not with commandserver because of the global function wrapping.  But I'm guessing most people don't understand that about commandserver.

IIUC, we would have to enable the largefiles extension once to clone the
repo, but that's no longer needed. So you can enable the extension without
hearing its name at all.

Perhaps, a warning can be displayed when the non-core repository requirement
is added?

mharbison72 added a comment.Oct 4 2018, 12:36 AM
In D4713#73260, @yuja wrote:

IIUC, we would have to enable the largefiles extension once to clone the
repo, but that's no longer needed. So you can enable the extension without
hearing its name at all.
Perhaps, a warning can be displayed when the non-core repository requirement
is added?

I'll take a look and see what I can figure out, but it likely won't be until next week. I think it should maybe be limited to a clone/unbundle adding a requirement with an implicitly loaded extension.

Revision Contents
Changeset List

Diff 11385

hgext/largefiles/overrides.py

repo = destrepo.local() repo = destrepo.local()
# When cloning to a remote repo (like through SSH), no repo is available # When cloning to a remote repo (like through SSH), no repo is available
# from the peer. Therefore the largefiles can't be downloaded and the # from the peer. Therefore the largefiles can't be downloaded and the
# hgrc can't be updated. # hgrc can't be updated.
if not repo: if not repo:
return result return result
# If largefiles is required for this repo, permanently enable it locally
if 'largefiles' in repo.requirements:
repo.vfs.append('hgrc',
util.tonativeeol('\n[extensions]\nlargefiles=\n'))
# Caching is implicitly limited to 'rev' option, since the dest repo was # Caching is implicitly limited to 'rev' option, since the dest repo was
# truncated at that point. The user may expect a download count with # truncated at that point. The user may expect a download count with
# this option, so attempt whether or not this is a largefile repo. # this option, so attempt whether or not this is a largefile repo.
if opts.get('all_largefiles'): if opts.get('all_largefiles'):
success, missing = lfcommands.downloadlfiles(ui, repo, None) success, missing = lfcommands.downloadlfiles(ui, repo, None)
if missing != 0: if missing != 0:
return None return None
return result return result
def hgpostshare(orig, sourcerepo, destrepo, defaultpath=None):
orig(sourcerepo, destrepo, defaultpath=defaultpath)
# If largefiles is required for this repo, permanently enable it locally
if 'largefiles' in destrepo.requirements:
destrepo.vfs.append('hgrc',
util.tonativeeol('\n[extensions]\nlargefiles=\n'))
def overriderebase(orig, ui, repo, **opts): def overriderebase(orig, ui, repo, **opts):
if not util.safehasattr(repo, '_largefilesenabled'): if not util.safehasattr(repo, '_largefilesenabled'):
return orig(ui, repo, **opts) return orig(ui, repo, **opts)
resuming = opts.get(r'continue') resuming = opts.get(r'continue')
repo._lfcommithooks.append(lfutil.automatedcommithook(resuming)) repo._lfcommithooks.append(lfutil.automatedcommithook(resuming))
repo._lfstatuswriters.append(lambda *msg, **opts: None) repo._lfstatuswriters.append(lambda *msg, **opts: None)
try: try:

hgext/largefiles/uisetup.py

overrides.exchangepushoperation) overrides.exchangepushoperation)
entry = extensions.wrapcommand(commands.table, 'clone', entry = extensions.wrapcommand(commands.table, 'clone',
overrides.overrideclone) overrides.overrideclone)
cloneopt = [('', 'all-largefiles', None, cloneopt = [('', 'all-largefiles', None,
_('download all versions of all largefiles'))] _('download all versions of all largefiles'))]
entry[1].extend(cloneopt) entry[1].extend(cloneopt)
extensions.wrapfunction(hg, 'clone', overrides.hgclone) extensions.wrapfunction(hg, 'clone', overrides.hgclone)
extensions.wrapfunction(hg, 'postshare', overrides.hgpostshare)
entry = extensions.wrapcommand(commands.table, 'cat', entry = extensions.wrapcommand(commands.table, 'cat',
overrides.overridecat) overrides.overridecat)
extensions.wrapfunction(merge, '_checkunknownfile', extensions.wrapfunction(merge, '_checkunknownfile',
overrides.overridecheckunknownfile) overrides.overridecheckunknownfile)
extensions.wrapfunction(merge, 'calculateupdates', extensions.wrapfunction(merge, 'calculateupdates',
overrides.overridecalculateupdates) overrides.overridecalculateupdates)
extensions.wrapfunction(merge, 'recordupdates', extensions.wrapfunction(merge, 'recordupdates',

mercurial/localrepo.py

The function can be used to validate configs, automatically add The function can be used to validate configs, automatically add
options (including extensions) based on requirements, etc. options (including extensions) based on requirements, etc.
""" """
# Map of requirements to list of extensions to load automatically when # Map of requirements to list of extensions to load automatically when
# requirement is present. # requirement is present.
autoextensions = { autoextensions = {
b'largefiles': [b'largefiles'],
b'lfs': [b'lfs'], b'lfs': [b'lfs'],
} }
for requirement, names in sorted(autoextensions.items()): for requirement, names in sorted(autoextensions.items()):
if requirement not in requirements: if requirement not in requirements:
continue continue
for name in names: for name in names:

tests/test-largefiles-misc.t

This file contains testcases that tend to be related to special cases or less This file contains testcases that tend to be related to special cases or less
common commands affecting largefile. common commands affecting largefile.
$ hg init requirements
$ cd requirements
# largefiles not loaded by default.
$ hg config extensions
[1]
# Adding largefiles to requires file will auto-load largefiles extension.
$ echo largefiles >> .hg/requires
$ hg config extensions
extensions.largefiles=
# But only if there is no config entry for the extension already.
$ cat > .hg/hgrc << EOF
> [extensions]
> largefiles=!
> EOF
$ hg config extensions
abort: repository requires features unknown to this Mercurial: largefiles!
(see https://mercurial-scm.org/wiki/MissingRequirement for more information)
[255]
$ cat > .hg/hgrc << EOF
> [extensions]
> largefiles=
> EOF
$ hg config extensions
extensions.largefiles=
$ cat > .hg/hgrc << EOF
> [extensions]
> largefiles = missing.py
> EOF
$ hg config extensions
*** failed to import extension largefiles from missing.py: [Errno 2] $ENOENT$: 'missing.py'
abort: repository requires features unknown to this Mercurial: largefiles!
(see https://mercurial-scm.org/wiki/MissingRequirement for more information)
[255]
$ cd ..
Each sections should be independent of each others. Each sections should be independent of each others.
$ USERCACHE="$TESTTMP/cache"; export USERCACHE $ USERCACHE="$TESTTMP/cache"; export USERCACHE
$ mkdir "${USERCACHE}" $ mkdir "${USERCACHE}"
$ cat >> $HGRCPATH <<EOF $ cat >> $HGRCPATH <<EOF
> [extensions] > [extensions]
> largefiles= > largefiles=
> purge= > purge=
sharing a largefile repo automatically enables largefiles on the share sharing a largefile repo automatically enables largefiles on the share
$ hg share --config extensions.share= . ../shared_lfrepo $ hg share --config extensions.share= . ../shared_lfrepo
updating working directory updating working directory
getting changed largefiles getting changed largefiles
1 largefiles updated, 0 removed 1 largefiles updated, 0 removed
3 files updated, 0 files merged, 0 files removed, 0 files unresolved 3 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat ../shared_lfrepo/.hg/hgrc $ grep largefiles ../shared_lfrepo/.hg/requires
largefiles
[extensions]
largefiles=
verify that large files in subrepos handled properly verify that large files in subrepos handled properly
$ hg init subrepo $ hg init subrepo
$ echo "subrepo = subrepo" > .hgsub $ echo "subrepo = subrepo" > .hgsub
$ hg add .hgsub $ hg add .hgsub
$ hg ci -m "add subrepo" $ hg ci -m "add subrepo"
Invoking status precommit hook Invoking status precommit hook
A .hgsub A .hgsub

tests/test-largefiles.t

$ hg pull --rebase --all-largefiles --config paths.default-push=bogus/path --config paths.default=../b $ hg pull --rebase --all-largefiles --config paths.default-push=bogus/path --config paths.default=../b
pulling from $TESTTMP/b pulling from $TESTTMP/b
searching for changes searching for changes
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 1 changesets with 2 changes to 2 files (+1 heads) added 1 changesets with 2 changes to 2 files (+1 heads)
new changesets a381d2c8c80e (1 drafts) new changesets a381d2c8c80e (1 drafts)
0 largefiles cached
rebasing 8:f574fb32bb45 "modify normal file largefile in repo d" rebasing 8:f574fb32bb45 "modify normal file largefile in repo d"
Invoking status precommit hook Invoking status precommit hook
M sub/normal4 M sub/normal4
M sub2/large6 M sub2/large6
saved backup bundle to $TESTTMP/d/.hg/strip-backup/f574fb32bb45-dd1d9f80-rebase.hg saved backup bundle to $TESTTMP/d/.hg/strip-backup/f574fb32bb45-dd1d9f80-rebase.hg
0 largefiles cached
$ [ -f .hg/largefiles/e166e74c7303192238d60af5a9c4ce9bef0b7928 ] $ [ -f .hg/largefiles/e166e74c7303192238d60af5a9c4ce9bef0b7928 ]
$ hg log --template '{rev}:{node|short} {desc|firstline}\n' $ hg log --template '{rev}:{node|short} {desc|firstline}\n'
9:598410d3eb9a modify normal file largefile in repo d 9:598410d3eb9a modify normal file largefile in repo d
8:a381d2c8c80e modify normal file and largefile in repo b 8:a381d2c8c80e modify normal file and largefile in repo b
7:daea875e9014 add/edit more largefiles 7:daea875e9014 add/edit more largefiles
6:4355d653f84f edit files yet again 6:4355d653f84f edit files yet again
5:9d5af5072dbd edit files again 5:9d5af5072dbd edit files again
4:74c02385b94c move files 4:74c02385b94c move files

tests/test-subrepo-deep-nested-change.t

updating [===========================================>] 1/1\r (no-eol) (esc) updating [===========================================>] 1/1\r (no-eol) (esc)
\r (no-eol) (esc) \r (no-eol) (esc)
updating to branch default updating to branch default
cloning subrepo sub1 from $TESTTMP/sub1 cloning subrepo sub1 from $TESTTMP/sub1
cloning subrepo sub1/sub2 from $TESTTMP/sub2 cloning subrepo sub1/sub2 from $TESTTMP/sub2
3 files updated, 0 files merged, 0 files removed, 0 files unresolved 3 files updated, 0 files merged, 0 files removed, 0 files unresolved
Largefiles is NOT enabled in the clone if the source repo doesn't require it Largefiles is NOT enabled in the clone if the source repo doesn't require it
$ cat cloned/.hg/hgrc $ grep largefiles cloned/.hg/hgrc
# example repository config (see 'hg help config' for more info) [1]
[paths]
default = $TESTTMP/main
# path aliases to other clones of this repo in URLs or filesystem paths
# (see 'hg help config.paths' for more info)
#
# default:pushurl = ssh://jdoe@example.net/hg/jdoes-fork
# my-fork = ssh://jdoe@example.net/hg/jdoes-fork
# my-clone = /home/jdoe/jdoes-clone
[ui]
# name and email (local to this repository, optional), e.g.
# username = Jane Doe <jdoe@example.com>
Checking cloned repo ids Checking cloned repo ids
$ printf "cloned " ; hg id -R cloned $ printf "cloned " ; hg id -R cloned
cloned 7f491f53a367 tip cloned 7f491f53a367 tip
$ printf "cloned/sub1 " ; hg id -R cloned/sub1 $ printf "cloned/sub1 " ; hg id -R cloned/sub1
cloned/sub1 fc3b4ce2696f tip cloned/sub1 fc3b4ce2696f tip
$ printf "cloned/sub1/sub2 " ; hg id -R cloned/sub1/sub2 $ printf "cloned/sub1/sub2 " ; hg id -R cloned/sub1/sub2
$ find ../archive_lf | sort $ find ../archive_lf | sort
../archive_lf ../archive_lf
../archive_lf/sub1 ../archive_lf/sub1
../archive_lf/sub1/sub2 ../archive_lf/sub1/sub2
../archive_lf/sub1/sub2/large.bin ../archive_lf/sub1/sub2/large.bin
$ rm -rf ../archive_lf $ rm -rf ../archive_lf
The local repo enables largefiles if a largefiles repo is cloned The local repo enables largefiles if a largefiles repo is cloned
$ hg showconfig extensions $ hg showconfig extensions
abort: repository requires features unknown to this Mercurial: largefiles! extensions.largefiles=
(see https://mercurial-scm.org/wiki/MissingRequirement for more information)
[255]
$ hg --config extensions.largefiles= clone -qU . ../lfclone
$ cat ../lfclone/.hg/hgrc
# example repository config (see 'hg help config' for more info)
[paths]
default = $TESTTMP/cloned
# path aliases to other clones of this repo in URLs or filesystem paths
# (see 'hg help config.paths' for more info)
#
# default:pushurl = ssh://jdoe@example.net/hg/jdoes-fork
# my-fork = ssh://jdoe@example.net/hg/jdoes-fork
# my-clone = /home/jdoe/jdoes-clone
[ui]
# name and email (local to this repository, optional), e.g.
# username = Jane Doe <jdoe@example.com>
[extensions] $ hg --config extensions.largefiles= clone -qU . ../lfclone
largefiles= $ grep largefiles ../lfclone/.hg/requires
largefiles
Find an exact match to a standin (should archive nothing) Find an exact match to a standin (should archive nothing)
$ hg --config extensions.largefiles= archive -S -I 'sub/sub2/.hglf/large.bin' ../archive_lf $ hg --config extensions.largefiles= archive -S -I 'sub/sub2/.hglf/large.bin' ../archive_lf
$ find ../archive_lf 2> /dev/null | sort $ find ../archive_lf 2> /dev/null | sort
$ cat >> $HGRCPATH <<EOF $ cat >> $HGRCPATH <<EOF
> [extensions] > [extensions]
> largefiles= > largefiles=