This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5493

match: support rooted globs in hgignore
ClosedPublic

Authored by valentin.gatienbaron on Jan 4 2019, 3:34 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG4fab8a7d2d72: match: support rooted globs in hgignore
Summary

In a .hgignore, "glob:foo" always means "/foo". This cannot be
avoided because there is no syntax like "^" in regexes to say you
don't want the implied "/" (of course one can use regexes, but glob
syntax is nice).

When you have a long list of fairly specific globs like
path/to/some/thing, this has two consequences:

  1. unintended files may be ignored (not too common though)
  2. matching performance can suffer significantly Here is vanilla hg status timing on a private repository:

    Using syntax:glob everywhere real 0m2.199s user 0m1.545s sys 0m0.619s

    When rooting the appropriate globs real 0m1.434s user 0m0.847s sys 0m0.565s

    (tangentially, none of this shows up in --profile's output. It seems that C code doesn't play well with profiling)

The code already supports this but there is no syntax to make use of
it, so it seems reasonable to create such syntax. I create a new
hgignore syntax "rootglob".

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

yuja added subscribers: foozy, yuja.Jan 9 2019, 8:50 AM
The code already supports this but there is no syntax to make use of
it, so it seems reasonable to create such syntax. I create a new
hgignore syntax "rootedglob". There might be a better name, but
"rooted" is the terminology in use in user-facing documentation.

Perhaps, it can be called a "rootglob".

https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089604.html

I don't remember how the discussion ended, but since we have new "rootfilesin"
matcher, "root<whatever>" should be the new naming convention.

+ syntaxes = collections.OrderedDict([
+ ('re', 'relre:'),
+ ('regexp', 'relre:'),
+ ('glob', 'relglob:'),
+ ('rootedglob', 'glob:'), # after 'glob' line, so glob:a means relglob:a

Can you add a new matcher kind 'rootglob'? I understand 'glob' can be abused
for .hgignore, but it's the pattern relative to cwd.

valentin.gatienbaron edited the summary of this revision. (Show Details)Jan 15 2019, 7:02 PM
valentin.gatienbaron updated this revision to Diff 13229.
valentin.gatienbaron added a comment.Jan 15 2019, 11:57 PM

Sorry, it took me a while to come back to this. I renamed the syntax to rootglob.

I haven't handled your other remark, however I noticed that globs in .gitignore can be rooted [1]. I had looked for a precedent for syntax for rooting a glob last week without success. Now that I see this, their choice looks sensible enough to me, so I think I'll try that, as I think that'd be nicer overall, and easier to implement (and side-step your remark about abusing the glob syntax).

[1] From https://git-scm.com/docs/gitignore: A leading slash matches the beginning of the pathname. For example, "/*.c" matches "cat-file.c" but not "mozilla-sha1/sha1.c".

yuja added a comment.Jan 16 2019, 8:24 AM

I haven't handled your other remark,

I think rootglob: can be easily added. As we already have cwd-relative
glob:, we'll just need some "if"s to not rewrite a repo.root-relative path
to cwd-relative path.

This patch works, but it's technically incorrect since glob: isn't a
repo.root-relative pattern. That's my concern.

however I noticed that globs in .gitignore can be rooted [1]. I had looked for a precedent for syntax for rooting a glob last week without success. Now that I see this, their choice looks sensible enough to me, so I think I'll try that, as I think that'd be nicer overall, and easier to implement (and side-step your remark about abusing the glob syntax).

[1] From https://git-scm.com/docs/gitignore: A leading slash matches the beginning of the pathname. For example, "/*.c" matches "cat-file.c" but not "mozilla-sha1/sha1.c".

Do you mean you're planning to copy the git's syntax? I'm -1 on it because
it would look quite different from the other hg's pattern syntax.

valentin.gatienbaron added a comment.Jan 16 2019, 11:30 PM

Alright, I think I did what you asked for.

yuja added a comment.Jan 17 2019, 8:17 AM

Queued this, many thanks.

Closed by commit rHG4fab8a7d2d72: match: support rooted globs in hgignore (authored by valentin.gatienbaron). · Explain WhyJan 17 2019, 8:42 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/help/hgignore.txt (11 lines)
Mmercurial/match.py (17 lines)
Mtests/test-hgignore.t (32 lines)

Diff 13011

mercurial/help/hgignore.txt

syntax: NAME syntax: NAME
where ``NAME`` is one of the following: where ``NAME`` is one of the following:
``regexp`` ``regexp``
Regular expression, Python/Perl syntax. Regular expression, Python/Perl syntax.
``glob`` ``glob``
Shell-style glob. Shell-style glob.
``rootedglob``
A variant of ``glob`` that is rooted (see below).
The chosen syntax stays in effect when parsing all patterns that The chosen syntax stays in effect when parsing all patterns that
follow, until another syntax is selected. follow, until another syntax is selected.
Neither glob nor regexp patterns are rooted. A glob-syntax pattern of Neither ``glob`` nor regexp patterns are rooted. A glob-syntax
the form ``*.c`` will match a file ending in ``.c`` in any directory, pattern of the form ``*.c`` will match a file ending in ``.c`` in any
and a regexp pattern of the form ``\.c$`` will do the same. To root a directory, and a regexp pattern of the form ``\.c$`` will do the
regexp pattern, start it with ``^``. same. To root a regexp pattern, start it with ``^``. To get the same
effect with glob-syntax, you have to use ``rootedglob``.
Subdirectories can have their own .hgignore settings by adding Subdirectories can have their own .hgignore settings by adding
``subinclude:path/to/subdir/.hgignore`` to the root ``.hgignore``. See ``subinclude:path/to/subdir/.hgignore`` to the root ``.hgignore``. See
:hg:`help patterns` for details on ``subinclude:`` and ``include:``. :hg:`help patterns` for details on ``subinclude:`` and ``include:``.
.. note:: .. note::
Patterns specified in other than ``.hgignore`` are always rooted. Patterns specified in other than ``.hgignore`` are always rooted.

mercurial/match.py

# match.py - filename matching # match.py - filename matching
# #
# Copyright 2008, 2009 Matt Mackall <mpm@selenic.com> and others # Copyright 2008, 2009 Matt Mackall <mpm@selenic.com> and others
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import, print_function from __future__ import absolute_import, print_function
import collections
import copy import copy
import itertools import itertools
import os import os
import re import re
from .i18n import _ from .i18n import _
from . import ( from . import (
encoding, encoding,
empty lines are skipped. empty lines are skipped.
lines can be of the following formats: lines can be of the following formats:
syntax: regexp # defaults following lines to non-rooted regexps syntax: regexp # defaults following lines to non-rooted regexps
syntax: glob # defaults following lines to non-rooted globs syntax: glob # defaults following lines to non-rooted globs
re:pattern # non-rooted regular expression re:pattern # non-rooted regular expression
glob:pattern # non-rooted glob glob:pattern # non-rooted glob
rootedglob:pat # rooted glob (same root as ^ in regexps)
pattern # pattern of the current default type pattern # pattern of the current default type
if sourceinfo is set, returns a list of tuples: if sourceinfo is set, returns a list of tuples:
(pattern, lineno, originalline). This is useful to debug ignore patterns. (pattern, lineno, originalline). This is useful to debug ignore patterns.
''' '''
syntaxes = { syntaxes = collections.OrderedDict([
're': 'relre:', ('re', 'relre:'),
'regexp': 'relre:', ('regexp', 'relre:'),
'glob': 'relglob:', ('glob', 'relglob:'),
'include': 'include', ('rootedglob', 'glob:'), # after 'glob' line, so glob:a means relglob:a
'subinclude': 'subinclude', ('include', 'include'),
} ('subinclude', 'subinclude'),
])
syntax = 'relre:' syntax = 'relre:'
patterns = [] patterns = []
fp = open(filepath, 'rb') fp = open(filepath, 'rb')
for lineno, line in enumerate(util.iterfile(fp), start=1): for lineno, line in enumerate(util.iterfile(fp), start=1):
if "#" in line: if "#" in line:
global _commentre global _commentre
if not _commentre: if not _commentre:

tests/test-hgignore.t

? a.o ? a.o
? syntax ? syntax
$ hg debugignore a.c $ hg debugignore a.c
a.c is not ignored a.c is not ignored
$ hg debugignore dir/c.o $ hg debugignore dir/c.o
dir/c.o is ignored dir/c.o is ignored
(ignore rule in $TESTTMP/ignorerepo/.hgignore, line 2: 'dir/**/c.o') (glob) (ignore rule in $TESTTMP/ignorerepo/.hgignore, line 2: 'dir/**/c.o') (glob)
Check rooted globs
$ hg purge --all --config extensions.purge=
$ echo "syntax: rootedglob" > .hgignore
$ echo "a/*.ext" >> .hgignore
$ for p in a b/a aa; do mkdir -p $p; touch $p/b.ext; done
$ hg status -A 'set:**.ext'
? aa/b.ext
? b/a/b.ext
I a/b.ext
Check using 'include:' in ignore file Check using 'include:' in ignore file
$ hg purge --all --config extensions.purge= $ hg purge --all --config extensions.purge=
$ touch foo.included $ touch foo.included
$ echo ".*.included" > otherignore $ echo ".*.included" > otherignore
$ hg status -I "include:otherignore" $ hg status -I "include:otherignore"
? foo.included ? foo.included
$ echo "include:otherignore" >> .hgignore $ echo "include:otherignore" >> .hgignore
$ hg status $ hg status
A dir/b.o A dir/b.o
? .hgignore ? .hgignore
? otherignore ? otherignore
Check recursive uses of 'include:' Check recursive uses of 'include:'
$ echo "include:nested/ignore" >> otherignore $ echo "include:nested/ignore" >> otherignore
$ mkdir nested $ mkdir nested nested/more
$ echo "glob:*ignore" > nested/ignore $ echo "glob:*ignore" > nested/ignore
$ echo "rootedglob:a" >> nested/ignore
$ touch a nested/a nested/more/a
$ hg status $ hg status
A dir/b.o A dir/b.o
? nested/a
? nested/more/a
$ rm a nested/a nested/more/a
$ cp otherignore goodignore $ cp otherignore goodignore
$ echo "include:badignore" >> otherignore $ echo "include:badignore" >> otherignore
$ hg status $ hg status
skipping unreadable pattern file 'badignore': $ENOENT$ skipping unreadable pattern file 'badignore': $ENOENT$
A dir/b.o A dir/b.o
$ mv goodignore otherignore $ mv goodignore otherignore
$ touch dir1/file1 dir1/file2 dir2/file1 dir2/file2 $ touch dir1/file1 dir1/file2 dir2/file1 dir2/file2
$ echo "subinclude:dir2/.hgignore" >> .hgignore $ echo "subinclude:dir2/.hgignore" >> .hgignore
$ echo "glob:file*2" > dir2/.hgignore $ echo "glob:file*2" > dir2/.hgignore
$ hg status $ hg status
? dir1/file1 ? dir1/file1
? dir1/file2 ? dir1/file2
? dir2/file1 ? dir2/file1
Check including subincludes with regexs Check including subincludes with other patterns
$ echo "subinclude:dir1/.hgignore" >> .hgignore $ echo "subinclude:dir1/.hgignore" >> .hgignore
$ echo "regexp:f.le1" > dir1/.hgignore
$ mkdir dir1/subdir
$ touch dir1/subdir/file1
$ echo "rootedglob:f?le1" > dir1/.hgignore
$ hg status
? dir1/file2
? dir1/subdir/file1
? dir2/file1
$ rm dir1/subdir/file1
$ echo "regexp:f.le1" > dir1/.hgignore
$ hg status $ hg status
? dir1/file2 ? dir1/file2
? dir2/file1 ? dir2/file1
Check multiple levels of sub-ignores Check multiple levels of sub-ignores
$ mkdir dir1/subdir
$ touch dir1/subdir/subfile1 dir1/subdir/subfile3 dir1/subdir/subfile4 $ touch dir1/subdir/subfile1 dir1/subdir/subfile3 dir1/subdir/subfile4
$ echo "subinclude:subdir/.hgignore" >> dir1/.hgignore $ echo "subinclude:subdir/.hgignore" >> dir1/.hgignore
$ echo "glob:subfil*3" >> dir1/subdir/.hgignore $ echo "glob:subfil*3" >> dir1/subdir/.hgignore
$ hg status $ hg status
? dir1/file2 ? dir1/file2
? dir1/subdir/subfile4 ? dir1/subdir/subfile4
? dir2/file1 ? dir2/file1