This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10284

re2: feed unicode string to re2 module when necessary
ClosedPublic

Authored by marmoute on Mar 28 2021, 6:52 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG112826b59476: re2: feed unicode string to re2 module when necessary
rHG3ff353820fc4: re2: feed unicode string to re2 module when necessary
Summary

My previous test were using the pyre2 Python project, that wrap the Google RE2 library
in python as a re2 module and accept bytes as input. However the fb-re2
Python project is also offering a wrapping of the Google RE2 library in python
as a re2 module ans accept only unicode on python3. So we detect this
situation and convert thing to unicode when necessary.

Hooray…

We should consider using a rust wrapping for regexp handling. We needs regexps
in Rust anyway and this give use more control with less variants and more
sanity.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Mar 28 2021, 6:52 PM
marmoute added a commit: rHG3ff353820fc4: re2: feed unicode string to re2 module when necessary.Mar 29 2021, 3:51 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG3ff353820fc4: re2: feed unicode string to re2 module when necessary (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/util.py (15 lines)

Diff 26625

mercurial/util.py

s2 = os.lstat(p2) s2 = os.lstat(p2)
if s2 == s1: if s2 == s1:
return False return False
return True return True
except OSError: except OSError:
return True return True
_re2_input = lambda x: x
try: try:
import re2 # pytype: disable=import-error import re2 # pytype: disable=import-error
_re2 = None _re2 = None
except ImportError: except ImportError:
_re2 = False _re2 = False
class _re(object): class _re(object):
def _checkre2(self): def _checkre2(self):
global _re2 global _re2
global _re2_input
try: try:
# check if match works, see issue3964 # check if match works, see issue3964
_re2 = bool(re2.match(br'\[([^\[]+)\]', b'[ui]')) check_pattern = br'\[([^\[]+)\]'
check_input = b'[ui]'
_re2 = bool(re2.match(check_pattern, check_input))
except ImportError: except ImportError:
_re2 = False _re2 = False
except TypeError:
# the `pyre-2` project provides a re2 module that accept bytes
# the `fb-re2` project provides a re2 module that acccept sysstr
check_pattern = pycompat.sysstr(check_pattern)
check_input = pycompat.sysstr(check_input)
_re2 = bool(re2.match(check_pattern, check_input))
_re2_input = pycompat.sysstr
def compile(self, pat, flags=0): def compile(self, pat, flags=0):
"""Compile a regular expression, using re2 if possible """Compile a regular expression, using re2 if possible
For best performance, use only re2-compatible regexp features. The For best performance, use only re2-compatible regexp features. The
only flags from the re module that are re2-compatible are only flags from the re module that are re2-compatible are
IGNORECASE and MULTILINE.""" IGNORECASE and MULTILINE."""
if _re2 is None: if _re2 is None:
self._checkre2() self._checkre2()
if _re2 and (flags & ~(remod.IGNORECASE | remod.MULTILINE)) == 0: if _re2 and (flags & ~(remod.IGNORECASE | remod.MULTILINE)) == 0:
if flags & remod.IGNORECASE: if flags & remod.IGNORECASE:
pat = b'(?i)' + pat pat = b'(?i)' + pat
if flags & remod.MULTILINE: if flags & remod.MULTILINE:
pat = b'(?m)' + pat pat = b'(?m)' + pat
try: try:
return re2.compile(pat) return re2.compile(_re2_input(pat))
except re2.error: except re2.error:
pass pass
return remod.compile(pat, flags) return remod.compile(pat, flags)
@propertycache @propertycache
def escape(self): def escape(self):
"""Return the version of escape corresponding to self.compile. """Return the version of escape corresponding to self.compile.