This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8148

revlog-compression: update the config to be a list
ClosedPublic

Authored by marmoute on Feb 25 2020, 5:15 AM.

Details

Reviewers
mharbison72
Group Reviewers
hg-reviewers
Commits
rHGf0027a3dd7cb: revlog-compression: update the config to be a list
rHG54b9238c27be: revlog-compression: update the config to be a list
Summary

format.revlog-compression is now a list of engine, the first supported one is to
be used. Doing this have several benefits:

  1. this is fully backward compatible, config using a single entry will be read

as a single item list, not changing any behavior.

  1. This open the way to use zstd by default without impacting platform were it

is not available. This will be done in a later changesets.

Using zstd provide a significant performance boost explained in : bb271ec2fbfb.
However zstd is not available in some cases, A notable example is the --pure
version of Mercurial which doesn't come with zstd support.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Feb 25 2020, 5:15 AM
pulkit added a subscriber: pulkit.Feb 25 2020, 6:12 AM

Looks good to me, can you add an entry to relnotes/next?

mharbison72 requested changes to this revision.Feb 25 2020, 12:45 PM
mharbison72 added a subscriber: mharbison72.
mharbison72 added inline comments.
mercurial/helptext/config.txt
891–892

s/value/values/

892

s/wille/will/

tests/test-repo-compengines.t
25

Nit: It might help readability (especially with a list) if the engines were delimited with quotes or parenthesis or something. It's mostly obvious here, but not if it's in a config file. (And actually, I missed that "unknown" was used above, and thought it was an awkward sentence at first.)

This revision now requires changes to proceed.Feb 25 2020, 12:45 PM
marmoute marked an inline comment as done.Feb 25 2020, 6:18 PM
marmoute updated this revision to Diff 20303.
mharbison72 accepted this revision.Feb 25 2020, 6:58 PM

LGTM, thanks for the update.

pulkit added a comment.Feb 26 2020, 4:31 AM
In D8148#121295, @pulkit wrote:

Looks good to me, can you add an entry to relnotes/next?

still missing the release notes entry.

marmoute added a commit: rHG54b9238c27be: revlog-compression: update the config to be a list.Feb 26 2020, 10:32 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG54b9238c27be: revlog-compression: update the config to be a list (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.
ryancoop987 added a subscriber: ryancoop987.Oct 21 2021, 4:04 AM

Really glad for this online alarmsinwindows10 amazing post forever

Revision Contents
Changeset List

Diff 20307

mercurial/configitems.py

b'format', b'obsstore-version', default=None, b'format', b'obsstore-version', default=None,
) )
coreconfigitem( coreconfigitem(
b'format', b'sparse-revlog', default=True, b'format', b'sparse-revlog', default=True,
) )
coreconfigitem( coreconfigitem(
b'format', b'format',
b'revlog-compression', b'revlog-compression',
default=b'zlib', default=lambda: [b'zlib'],
alias=[(b'experimental', b'format.compression')], alias=[(b'experimental', b'format.compression')],
) )
coreconfigitem( coreconfigitem(
b'format', b'usefncache', default=True, b'format', b'usefncache', default=True,
) )
coreconfigitem( coreconfigitem(
b'format', b'usegeneraldelta', default=True, b'format', b'usegeneraldelta', default=True,
) )

mercurial/helptext/config.txt

Repositories with this on-disk format require Mercurial version 4.7 Repositories with this on-disk format require Mercurial version 4.7
Enabled by default. Enabled by default.
``revlog-compression`` ``revlog-compression``
Compression algorithm used by revlog. Supported values are `zlib` and Compression algorithm used by revlog. Supported values are `zlib` and
`zstd`. The `zlib` engine is the historical default of Mercurial. `zstd` is `zstd`. The `zlib` engine is the historical default of Mercurial. `zstd` is
a newer format that is usually a net win over `zlib`, operating faster at a newer format that is usually a net win over `zlib`, operating faster at
better compression rates. Use `zstd` to reduce CPU usage. better compression rates. Use `zstd` to reduce CPU usage. Multiple values
can be specified, the first available one will be used.
mharbison72Unsubmitted
Not Done

s/value/values/

mharbison72: s/value/values/
mharbison72Unsubmitted
Not Done

s/wille/will/

mharbison72: s/wille/will/
On some systems, the Mercurial installation may lack `zstd` support. On some systems, the Mercurial installation may lack `zstd` support.
Default is `zlib`. Default is `zlib`.
``bookmarks-in-store`` ``bookmarks-in-store``
Store bookmarks in .hg/store/. This means that bookmarks are shared when Store bookmarks in .hg/store/. This means that bookmarks are shared when
using `hg share` regardless of the `-B` option. using `hg share` regardless of the `-B` option.

mercurial/localrepo.py

requirements = {b'revlogv1'} requirements = {b'revlogv1'}
if ui.configbool(b'format', b'usestore'): if ui.configbool(b'format', b'usestore'):
requirements.add(b'store') requirements.add(b'store')
if ui.configbool(b'format', b'usefncache'): if ui.configbool(b'format', b'usefncache'):
requirements.add(b'fncache') requirements.add(b'fncache')
if ui.configbool(b'format', b'dotencode'): if ui.configbool(b'format', b'dotencode'):
requirements.add(b'dotencode') requirements.add(b'dotencode')
compengine = ui.config(b'format', b'revlog-compression') compengines = ui.configlist(b'format', b'revlog-compression')
if compengine not in util.compengines: for compengine in compengines:
if compengine in util.compengines:
break
else:
raise error.Abort( raise error.Abort(
_( _(
b'compression engine %s defined by ' b'compression engines %s defined by '
b'format.revlog-compression not available' b'format.revlog-compression not available'
) )
% compengine, % b', '.join(b'"%s"' % e for e in compengines),
hint=_( hint=_(
b'run "hg debuginstall" to list available ' b'run "hg debuginstall" to list available '
b'compression engines' b'compression engines'
), ),
) )
# zlib is the historical default and doesn't need an explicit requirement. # zlib is the historical default and doesn't need an explicit requirement.
elif compengine == b'zstd': if compengine == b'zstd':
requirements.add(b'revlog-compression-zstd') requirements.add(b'revlog-compression-zstd')
elif compengine != b'zlib': elif compengine != b'zlib':
requirements.add(b'exp-compression-%s' % compengine) requirements.add(b'exp-compression-%s' % compengine)
if scmutil.gdinitconfig(ui): if scmutil.gdinitconfig(ui):
requirements.add(b'generaldelta') requirements.add(b'generaldelta')
if ui.configbool(b'format', b'sparse-revlog'): if ui.configbool(b'format', b'sparse-revlog'):
requirements.add(SPARSEREVLOG_REQUIREMENT) requirements.add(SPARSEREVLOG_REQUIREMENT)

mercurial/upgrade.py

for req in repo.requirements: for req in repo.requirements:
prefix = req.startswith prefix = req.startswith
if prefix(b'revlog-compression-') or prefix(b'exp-compression-'): if prefix(b'revlog-compression-') or prefix(b'exp-compression-'):
compression = req.split(b'-', 2)[2] compression = req.split(b'-', 2)[2]
return compression return compression
@classmethod @classmethod
def fromconfig(cls, repo): def fromconfig(cls, repo):
return repo.ui.config(b'format', b'revlog-compression') compengines = repo.ui.configlist(b'format', b'revlog-compression')
# return the first valid value as the selection code would do
for comp in compengines:
if comp in util.compengines:
return comp
# no valide compression found lets display it all for clarity
return b','.join(compengines)
@registerformatvariant @registerformatvariant
class compressionlevel(formatvariant): class compressionlevel(formatvariant):
name = b'compression-level' name = b'compression-level'
default = b'default' default = b'default'
description = _(b'compression level') description = _(b'compression level')

tests/test-repo-compengines.t

0x78 (x) : 1 (100.00%) 0x78 (x) : 1 (100.00%)
0x78 (x) : 110 (100.00%) 0x78 (x) : 110 (100.00%)
$ cd .. $ cd ..
Unknown compression engine to format.compression aborts Unknown compression engine to format.compression aborts
$ hg --config format.revlog-compression=unknown init unknown $ hg --config format.revlog-compression=unknown init unknown
abort: compression engine unknown defined by format.revlog-compression not available abort: compression engines "unknown" defined by format.revlog-compression not available
mharbison72Unsubmitted
Done

Nit: It might help readability (especially with a list) if the engines were delimited with quotes or parenthesis or something. It's mostly obvious here, but not if it's in a config file. (And actually, I missed that "unknown" was used above, and thought it was an awkward sentence at first.)

mharbison72: Nit: It might help readability (especially with a list) if the engines were delimited with…
(run "hg debuginstall" to list available compression engines) (run "hg debuginstall" to list available compression engines)
[255] [255]
unknown compression engine in a list with known one works fine
$ hg --config format.revlog-compression=zlib,unknown init zlib-before-unknow
$ hg --config format.revlog-compression=unknown,zlib init unknown-before-zlib
A requirement specifying an unknown compression engine results in bail A requirement specifying an unknown compression engine results in bail
$ hg init unknownrequirement $ hg init unknownrequirement
$ cd unknownrequirement $ cd unknownrequirement
$ echo exp-compression-unknown >> .hg/requires $ echo exp-compression-unknown >> .hg/requires
$ hg log $ hg log
abort: repository requires features unknown to this Mercurial: exp-compression-unknown! abort: repository requires features unknown to this Mercurial: exp-compression-unknown!
(see https://mercurial-scm.org/wiki/MissingRequirement for more information) (see https://mercurial-scm.org/wiki/MissingRequirement for more information)