This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5493

match: support rooted globs in hgignore
ClosedPublic

Authored by valentin.gatienbaron on Jan 4 2019, 3:34 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG4fab8a7d2d72: match: support rooted globs in hgignore
Summary

In a .hgignore, "glob:foo" always means "/foo". This cannot be
avoided because there is no syntax like "^" in regexes to say you
don't want the implied "/" (of course one can use regexes, but glob
syntax is nice).

When you have a long list of fairly specific globs like
path/to/some/thing, this has two consequences:

  1. unintended files may be ignored (not too common though)
  2. matching performance can suffer significantly Here is vanilla hg status timing on a private repository:

    Using syntax:glob everywhere real 0m2.199s user 0m1.545s sys 0m0.619s

    When rooting the appropriate globs real 0m1.434s user 0m0.847s sys 0m0.565s

    (tangentially, none of this shows up in --profile's output. It seems that C code doesn't play well with profiling)

The code already supports this but there is no syntax to make use of
it, so it seems reasonable to create such syntax. I create a new
hgignore syntax "rootglob".

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

yuja added subscribers: foozy, yuja.Jan 9 2019, 8:50 AM
The code already supports this but there is no syntax to make use of
it, so it seems reasonable to create such syntax. I create a new
hgignore syntax "rootedglob". There might be a better name, but
"rooted" is the terminology in use in user-facing documentation.

Perhaps, it can be called a "rootglob".

https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089604.html

I don't remember how the discussion ended, but since we have new "rootfilesin"
matcher, "root<whatever>" should be the new naming convention.

+ syntaxes = collections.OrderedDict([
+ ('re', 'relre:'),
+ ('regexp', 'relre:'),
+ ('glob', 'relglob:'),
+ ('rootedglob', 'glob:'), # after 'glob' line, so glob:a means relglob:a

Can you add a new matcher kind 'rootglob'? I understand 'glob' can be abused
for .hgignore, but it's the pattern relative to cwd.

valentin.gatienbaron edited the summary of this revision. (Show Details)Jan 15 2019, 7:02 PM
valentin.gatienbaron updated this revision to Diff 13229.
valentin.gatienbaron added a comment.Jan 15 2019, 11:57 PM

Sorry, it took me a while to come back to this. I renamed the syntax to rootglob.

I haven't handled your other remark, however I noticed that globs in .gitignore can be rooted [1]. I had looked for a precedent for syntax for rooting a glob last week without success. Now that I see this, their choice looks sensible enough to me, so I think I'll try that, as I think that'd be nicer overall, and easier to implement (and side-step your remark about abusing the glob syntax).

[1] From https://git-scm.com/docs/gitignore: A leading slash matches the beginning of the pathname. For example, "/*.c" matches "cat-file.c" but not "mozilla-sha1/sha1.c".

yuja added a comment.Jan 16 2019, 8:24 AM

I haven't handled your other remark,

I think rootglob: can be easily added. As we already have cwd-relative
glob:, we'll just need some "if"s to not rewrite a repo.root-relative path
to cwd-relative path.

This patch works, but it's technically incorrect since glob: isn't a
repo.root-relative pattern. That's my concern.

however I noticed that globs in .gitignore can be rooted [1]. I had looked for a precedent for syntax for rooting a glob last week without success. Now that I see this, their choice looks sensible enough to me, so I think I'll try that, as I think that'd be nicer overall, and easier to implement (and side-step your remark about abusing the glob syntax).

[1] From https://git-scm.com/docs/gitignore: A leading slash matches the beginning of the pathname. For example, "/*.c" matches "cat-file.c" but not "mozilla-sha1/sha1.c".

Do you mean you're planning to copy the git's syntax? I'm -1 on it because
it would look quite different from the other hg's pattern syntax.

valentin.gatienbaron added a comment.Jan 16 2019, 11:30 PM

Alright, I think I did what you asked for.

yuja added a comment.Jan 17 2019, 8:17 AM

Queued this, many thanks.

Closed by commit rHG4fab8a7d2d72: match: support rooted globs in hgignore (authored by valentin.gatienbaron). · Explain WhyJan 17 2019, 8:42 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 13316

mercurial/help/hgignore.txt

syntax: NAME syntax: NAME
where ``NAME`` is one of the following: where ``NAME`` is one of the following:
``regexp`` ``regexp``
Regular expression, Python/Perl syntax. Regular expression, Python/Perl syntax.
``glob`` ``glob``
Shell-style glob. Shell-style glob.
``rootglob``
A variant of ``glob`` that is rooted (see below).
The chosen syntax stays in effect when parsing all patterns that The chosen syntax stays in effect when parsing all patterns that
follow, until another syntax is selected. follow, until another syntax is selected.
Neither glob nor regexp patterns are rooted. A glob-syntax pattern of Neither ``glob`` nor regexp patterns are rooted. A glob-syntax
the form ``*.c`` will match a file ending in ``.c`` in any directory, pattern of the form ``*.c`` will match a file ending in ``.c`` in any
and a regexp pattern of the form ``\.c$`` will do the same. To root a directory, and a regexp pattern of the form ``\.c$`` will do the
regexp pattern, start it with ``^``. same. To root a regexp pattern, start it with ``^``. To get the same
effect with glob-syntax, you have to use ``rootglob``.
Subdirectories can have their own .hgignore settings by adding Subdirectories can have their own .hgignore settings by adding
``subinclude:path/to/subdir/.hgignore`` to the root ``.hgignore``. See ``subinclude:path/to/subdir/.hgignore`` to the root ``.hgignore``. See
:hg:`help patterns` for details on ``subinclude:`` and ``include:``. :hg:`help patterns` for details on ``subinclude:`` and ``include:``.
.. note:: .. note::
Patterns specified in other than ``.hgignore`` are always rooted. Patterns specified in other than ``.hgignore`` are always rooted.

mercurial/help/patterns.txt

``path:``. These path names must completely match starting at the ``path:``. These path names must completely match starting at the
current repository root, and when the path points to a directory, it is matched current repository root, and when the path points to a directory, it is matched
recursively. To match all files in a directory non-recursively (not including recursively. To match all files in a directory non-recursively (not including
any files in subdirectories), ``rootfilesin:`` can be used, specifying an any files in subdirectories), ``rootfilesin:`` can be used, specifying an
absolute path (relative to the repository root). absolute path (relative to the repository root).
To use an extended glob, start a name with ``glob:``. Globs are rooted To use an extended glob, start a name with ``glob:``. Globs are rooted
at the current directory; a glob such as ``*.c`` will only match files at the current directory; a glob such as ``*.c`` will only match files
in the current directory ending with ``.c``. in the current directory ending with ``.c``. ``rootglob:`` can be used
instead of ``glob:`` for a glob that is rooted at the root of the
repository.
The supported glob syntax extensions are ``**`` to match any string The supported glob syntax extensions are ``**`` to match any string
across path separators and ``{a,b}`` to mean "a or b". across path separators and ``{a,b}`` to mean "a or b".
To use a Perl/Python regular expression, start a name with ``re:``. To use a Perl/Python regular expression, start a name with ``re:``.
Regexp pattern matching is anchored at the root of the repository. Regexp pattern matching is anchored at the root of the repository.
To read name patterns from a file, use ``listfile:`` or ``listfile0:``. To read name patterns from a file, use ``listfile:`` or ``listfile0:``.
**.c any name ending in ".c" in any subdirectory of the **.c any name ending in ".c" in any subdirectory of the
current directory including itself. current directory including itself.
foo/* any file in directory foo foo/* any file in directory foo
foo/** any file in directory foo plus all its subdirectories, foo/** any file in directory foo plus all its subdirectories,
recursively recursively
foo/*.c any name ending in ".c" in the directory foo foo/*.c any name ending in ".c" in the directory foo
foo/**.c any name ending in ".c" in any subdirectory of foo foo/**.c any name ending in ".c" in any subdirectory of foo
including itself. including itself.
rootglob:*.c any name ending in ".c" in the root of the repository
Regexp examples:: Regexp examples::
re:.*\.c$ any name ending in ".c", anywhere in the repository re:.*\.c$ any name ending in ".c", anywhere in the repository
File examples:: File examples::
listfile:list.txt read list from list.txt with one file pattern per line listfile:list.txt read list from list.txt with one file pattern per line

mercurial/match.py

pycompat, pycompat,
util, util,
) )
from .utils import ( from .utils import (
stringutil, stringutil,
) )
allpatternkinds = ('re', 'glob', 'path', 'relglob', 'relpath', 'relre', allpatternkinds = ('re', 'glob', 'path', 'relglob', 'relpath', 'relre',
'rootglob',
'listfile', 'listfile0', 'set', 'include', 'subinclude', 'listfile', 'listfile0', 'set', 'include', 'subinclude',
'rootfilesin') 'rootfilesin')
cwdrelativepatternkinds = ('relpath', 'glob') cwdrelativepatternkinds = ('relpath', 'glob')
propertycache = util.propertycache propertycache = util.propertycache
def _rematcher(regex): def _rematcher(regex):
'''compile the regexp with the best available regexp engine and return a '''compile the regexp with the best available regexp engine and return a
def _donormalize(patterns, default, root, cwd, auditor, warn): def _donormalize(patterns, default, root, cwd, auditor, warn):
'''Convert 'kind:pat' from the patterns list to tuples with kind and '''Convert 'kind:pat' from the patterns list to tuples with kind and
normalized and rooted patterns and with listfiles expanded.''' normalized and rooted patterns and with listfiles expanded.'''
kindpats = [] kindpats = []
for kind, pat in [_patsplit(p, default) for p in patterns]: for kind, pat in [_patsplit(p, default) for p in patterns]:
if kind in cwdrelativepatternkinds: if kind in cwdrelativepatternkinds:
pat = pathutil.canonpath(root, cwd, pat, auditor) pat = pathutil.canonpath(root, cwd, pat, auditor)
elif kind in ('relglob', 'path', 'rootfilesin'): elif kind in ('relglob', 'path', 'rootfilesin', 'rootglob'):
pat = util.normpath(pat) pat = util.normpath(pat)
elif kind in ('listfile', 'listfile0'): elif kind in ('listfile', 'listfile0'):
try: try:
files = util.readfile(pat) files = util.readfile(pat)
if kind == 'listfile0': if kind == 'listfile0':
files = files.split('\0') files = files.split('\0')
else: else:
files = files.splitlines() files = files.splitlines()
# Anything after the pattern must be a non-directory. # Anything after the pattern must be a non-directory.
return escaped + '[^/]+$' return escaped + '[^/]+$'
if kind == 'relglob': if kind == 'relglob':
return '(?:|.*/)' + _globre(pat) + globsuffix return '(?:|.*/)' + _globre(pat) + globsuffix
if kind == 'relre': if kind == 'relre':
if pat.startswith('^'): if pat.startswith('^'):
return pat return pat
return '.*' + pat return '.*' + pat
if kind == 'glob': if kind in ('glob', 'rootglob'):
return _globre(pat) + globsuffix return _globre(pat) + globsuffix
raise error.ProgrammingError('not a regex pattern: %s:%s' % (kind, pat)) raise error.ProgrammingError('not a regex pattern: %s:%s' % (kind, pat))
def _buildmatch(kindpats, globsuffix, listsubrepos, root): def _buildmatch(kindpats, globsuffix, listsubrepos, root):
'''Return regexp string and a matcher function for kindpats. '''Return regexp string and a matcher function for kindpats.
globsuffix is appended to the regexp of globs.''' globsuffix is appended to the regexp of globs.'''
matchfuncs = [] matchfuncs = []
This calculates the roots and directories exactly matching the patterns and This calculates the roots and directories exactly matching the patterns and
returns a tuple of (roots, dirs) for each. It does not return other returns a tuple of (roots, dirs) for each. It does not return other
directories which may also need to be considered, like the parent directories which may also need to be considered, like the parent
directories. directories.
''' '''
r = [] r = []
d = [] d = []
for kind, pat, source in kindpats: for kind, pat, source in kindpats:
if kind == 'glob': # find the non-glob prefix if kind in ('glob', 'rootglob'): # find the non-glob prefix
root = [] root = []
for p in pat.split('/'): for p in pat.split('/'):
if '[' in p or '{' in p or '*' in p or '?' in p: if '[' in p or '{' in p or '*' in p or '?' in p:
break break
root.append(p) root.append(p)
r.append('/'.join(root) or '.') r.append('/'.join(root) or '.')
elif kind in ('relpath', 'path'): elif kind in ('relpath', 'path'):
r.append(pat or '.') r.append(pat or '.')
empty lines are skipped. empty lines are skipped.
lines can be of the following formats: lines can be of the following formats:
syntax: regexp # defaults following lines to non-rooted regexps syntax: regexp # defaults following lines to non-rooted regexps
syntax: glob # defaults following lines to non-rooted globs syntax: glob # defaults following lines to non-rooted globs
re:pattern # non-rooted regular expression re:pattern # non-rooted regular expression
glob:pattern # non-rooted glob glob:pattern # non-rooted glob
rootglob:pat # rooted glob (same root as ^ in regexps)
pattern # pattern of the current default type pattern # pattern of the current default type
if sourceinfo is set, returns a list of tuples: if sourceinfo is set, returns a list of tuples:
(pattern, lineno, originalline). This is useful to debug ignore patterns. (pattern, lineno, originalline). This is useful to debug ignore patterns.
''' '''
syntaxes = { syntaxes = {
're': 'relre:', 're': 'relre:',
'regexp': 'relre:', 'regexp': 'relre:',
'glob': 'relglob:', 'glob': 'relglob:',
'rootglob': 'rootglob:',
'include': 'include', 'include': 'include',
'subinclude': 'subinclude', 'subinclude': 'subinclude',
} }
syntax = 'relre:' syntax = 'relre:'
patterns = [] patterns = []
fp = open(filepath, 'rb') fp = open(filepath, 'rb')
for lineno, line in enumerate(util.iterfile(fp), start=1): for lineno, line in enumerate(util.iterfile(fp), start=1):

tests/test-hgignore.t

? a.o ? a.o
? syntax ? syntax
$ hg debugignore a.c $ hg debugignore a.c
a.c is not ignored a.c is not ignored
$ hg debugignore dir/c.o $ hg debugignore dir/c.o
dir/c.o is ignored dir/c.o is ignored
(ignore rule in $TESTTMP/ignorerepo/.hgignore, line 2: 'dir/**/c.o') (glob) (ignore rule in $TESTTMP/ignorerepo/.hgignore, line 2: 'dir/**/c.o') (glob)
Check rooted globs
$ hg purge --all --config extensions.purge=
$ echo "syntax: rootglob" > .hgignore
$ echo "a/*.ext" >> .hgignore
$ for p in a b/a aa; do mkdir -p $p; touch $p/b.ext; done
$ hg status -A 'set:**.ext'
? aa/b.ext
? b/a/b.ext
I a/b.ext
Check using 'include:' in ignore file Check using 'include:' in ignore file
$ hg purge --all --config extensions.purge= $ hg purge --all --config extensions.purge=
$ touch foo.included $ touch foo.included
$ echo ".*.included" > otherignore $ echo ".*.included" > otherignore
$ hg status -I "include:otherignore" $ hg status -I "include:otherignore"
? foo.included ? foo.included
$ echo "include:otherignore" >> .hgignore $ echo "include:otherignore" >> .hgignore
$ hg status $ hg status
A dir/b.o A dir/b.o
? .hgignore ? .hgignore
? otherignore ? otherignore
Check recursive uses of 'include:' Check recursive uses of 'include:'
$ echo "include:nested/ignore" >> otherignore $ echo "include:nested/ignore" >> otherignore
$ mkdir nested $ mkdir nested nested/more
$ echo "glob:*ignore" > nested/ignore $ echo "glob:*ignore" > nested/ignore
$ echo "rootglob:a" >> nested/ignore
$ touch a nested/a nested/more/a
$ hg status $ hg status
A dir/b.o A dir/b.o
? nested/a
? nested/more/a
$ rm a nested/a nested/more/a
$ cp otherignore goodignore $ cp otherignore goodignore
$ echo "include:badignore" >> otherignore $ echo "include:badignore" >> otherignore
$ hg status $ hg status
skipping unreadable pattern file 'badignore': $ENOENT$ skipping unreadable pattern file 'badignore': $ENOENT$
A dir/b.o A dir/b.o
$ mv goodignore otherignore $ mv goodignore otherignore
$ touch dir1/file1 dir1/file2 dir2/file1 dir2/file2 $ touch dir1/file1 dir1/file2 dir2/file1 dir2/file2
$ echo "subinclude:dir2/.hgignore" >> .hgignore $ echo "subinclude:dir2/.hgignore" >> .hgignore
$ echo "glob:file*2" > dir2/.hgignore $ echo "glob:file*2" > dir2/.hgignore
$ hg status $ hg status
? dir1/file1 ? dir1/file1
? dir1/file2 ? dir1/file2
? dir2/file1 ? dir2/file1
Check including subincludes with regexs Check including subincludes with other patterns
$ echo "subinclude:dir1/.hgignore" >> .hgignore $ echo "subinclude:dir1/.hgignore" >> .hgignore
$ echo "regexp:f.le1" > dir1/.hgignore
$ mkdir dir1/subdir
$ touch dir1/subdir/file1
$ echo "rootglob:f?le1" > dir1/.hgignore
$ hg status
? dir1/file2
? dir1/subdir/file1
? dir2/file1
$ rm dir1/subdir/file1
$ echo "regexp:f.le1" > dir1/.hgignore
$ hg status $ hg status
? dir1/file2 ? dir1/file2
? dir2/file1 ? dir2/file1
Check multiple levels of sub-ignores Check multiple levels of sub-ignores
$ mkdir dir1/subdir
$ touch dir1/subdir/subfile1 dir1/subdir/subfile3 dir1/subdir/subfile4 $ touch dir1/subdir/subfile1 dir1/subdir/subfile3 dir1/subdir/subfile4
$ echo "subinclude:subdir/.hgignore" >> dir1/.hgignore $ echo "subinclude:subdir/.hgignore" >> dir1/.hgignore
$ echo "glob:subfil*3" >> dir1/subdir/.hgignore $ echo "glob:subfil*3" >> dir1/subdir/.hgignore
$ hg status $ hg status
? dir1/file2 ? dir1/file2
? dir1/subdir/subfile4 ? dir1/subdir/subfile4
? dir2/file1 ? dir2/file1