This is an archive of the discontinued Mercurial Phabricator instance.

copies: add a HASCOPIESINFO flag to highlight rev with useful data
ClosedPublic

Authored by marmoute on Oct 1 2020, 10:09 AM.

Download Raw Diff

Details

Reviewers

indygreg
Alphare

Group Reviewers

hg-reviewers

Commits

rHG2d6aea053153: copies: add a HASCOPIESINFO flag to highlight rev with useful data
rHGc398c7982424: copies: add a HASCOPIESINFO flag to highlight rev with useful data

Summary

If some files changes that may impact copy tracing are detected, we set this
flag. This helps the copy tracing algorithm to skip fetching possibly expensive
data when unnecessary.

Diff Detail

Repository

rHG Mercurial

Branch

default

Lint

No Linters Available

Unit

No Unit Test Coverage

Event Timeline

marmoute created this revision.Oct 1 2020, 10:09 AM

Herald added a reviewer: indygreg. · View Herald TranscriptOct 1 2020, 10:09 AM

Herald added a reviewer: hg-reviewers. · View Herald Transcript

Herald added a subscriber: mercurial-patches. · View Herald Transcript

marmoute added a child revision: D9140: copies: return None instead of ChangingFiles when relevant.Oct 1 2020, 10:09 AM

marmoute added a parent revision: D9130: salvaged: properly deal with salvaged file during copy tracing.Oct 1 2020, 10:15 AM

Alphare accepted this revision.Oct 2 2020, 9:11 AM

marmoute added a commit: rHGc398c7982424: copies: add a HASCOPIESINFO flag to highlight rev with useful data.Oct 8 2020, 12:17 PM

This revision was not accepted when it landed; it landed in state Needs Review.

Closed by commit rHGc398c7982424: copies: add a HASCOPIESINFO flag to highlight rev with useful data (authored by marmoute). · Explain Why

This revision was automatically updated to reflect the committed changes.

marmoute added a commit: rHG2d6aea053153: copies: add a HASCOPIESINFO flag to highlight rev with useful data.Oct 9 2020, 2:29 AM

@marmoute: both REVISION_FLAG_SIDEDATA and REVISION_FLAG_HASCOPIESINFO need documented in mercurial/helptext/internals/revlogs.txt. Could you please send a patch to ensure the technical specs stay in sync with reality?

Revision Contents
Changeset List

			Path	Packages
M			mercurial/changelog.py (10 lines)
M			mercurial/interfaces/repository.py (2 lines)
M			mercurial/revlog.py (2 lines)
M			mercurial/revlogutils/constants.py (3 lines)
M			mercurial/revlogutils/flagutil.py (3 lines)
M			tests/test-lfs-serve.t (7 lines)

Commit	Parents	Author	Summary	Date
2313f2393ef4	67f18b3b750f	Pierre-Yves David		Oct 1 2020, 3:29 AM

Status	Author	Revision
Closed	marmoute	D9113 copies: rename some function to the new naming scheme
Closed	marmoute	D9112 changing-files: cache the various property
Closed	marmoute	D9111 changing-files: always use `mark_touched` to update the touched set
Closed	marmoute	D9141 copies: move `merged` testing sooner
Closed	marmoute	D9140 copies: return None instead of ChangingFiles when relevant
Closed	marmoute	D9139 copies: add a HASCOPIESINFO flag to highlight rev with useful data
Closed	marmoute	D9130 salvaged: properly deal with salvaged file during copy tracing
Closed	marmoute	D9129 salvaged: persist the salvaged set on disk
Closed	marmoute	D9128 changing-files: add clean computation of changed file for merges
Closed	marmoute	D9127 changing-files: add clean computation of changed files for linear changesets
Closed	marmoute	D9126 changing-files: add clean computation of changed files for roots
Closed	marmoute	D9125 changing-files: add a debug command display changed files
Closed	marmoute	D9124 side-data: add a test to check sidedata upgrade
Closed	marmoute	D9123 changing-files: split the changing files computation from encoding
Closed	marmoute	D9120 salvaged: record salvaged in ChangingFiles at commit time
Closed	marmoute	D9119 salvaged: track removal-candidates in more cases
Closed	marmoute	D9122 salvaged: explicitly skip salvaged file while encoding
Closed	marmoute	D9118 changing-files: add a "salvaged" set to track file that were not removed
Closed	marmoute	D9117 copies: directly pass a changes object to the copy tracing code
Closed	marmoute	D9116 copies: no longer change the sidedata flag
Closed	marmoute	D9115 copies: use dedicated `_revinfo_getter` function and call
Closed	marmoute	D9114 copies: make two version of the changeset centric algorithm
Closed	marmoute	D9092 changing-files: retrieve changelogrevision.files from the sidedata block
Closed	marmoute	D9091 changing-files: drop the now useless changelogrevision argument
Closed	marmoute	D9090 changing-files: rework the way we store changed files in side-data
Closed	marmoute	D9143 changing-files: fix docstring

Diff 22970

mercurial/changelog.py

	metadata,			metadata,
	pycompat,			pycompat,
	revlog,			revlog,
	)			)
	from .utils import (			from .utils import (
	dateutil,			dateutil,
	stringutil,			stringutil,
	)			)
				from .revlogutils import flagutil

	_defaultextra = {b'branch': b'default'}			_defaultextra = {b'branch': b'default'}


	def _string_escape(text):			def _string_escape(text):
	"""			"""
	>>> from .pycompat import bytechr as chr			>>> from .pycompat import bytechr as chr
	>>> d = {b'nl': chr(10), b'bs': chr(92), b'cr': chr(13), b'nul': chr(0)}			>>> d = {b'nl': chr(10), b'bs': chr(92), b'cr': chr(13), b'nul': chr(0)}
	branch = extra.get(b"branch")			branch = extra.get(b"branch")
	if branch in (b"default", b""):			if branch in (b"default", b""):
	del extra[b"branch"]			del extra[b"branch"]
	elif branch in (b".", b"null", b"tip"):			elif branch in (b".", b"null", b"tip"):
	raise error.StorageError(			raise error.StorageError(
	_(b'the name \'%s\' is reserved') % branch			_(b'the name \'%s\' is reserved') % branch
	)			)
	sortedfiles = sorted(files.touched)			sortedfiles = sorted(files.touched)
				flags = 0
	sidedata = None			sidedata = None
	if self._copiesstorage == b'changeset-sidedata':			if self._copiesstorage == b'changeset-sidedata':
				if (
				files.removed
				or files.merged
				or files.salvaged
				or files.copied_from_p1
				or files.copied_from_p2
				):
				flags \|= flagutil.REVIDX_HASCOPIESINFO
	sidedata = metadata.encode_files_sidedata(files)			sidedata = metadata.encode_files_sidedata(files)

	if extra:			if extra:
	extra = encodeextra(extra)			extra = encodeextra(extra)
	parseddate = b"%s %s" % (parseddate, extra)			parseddate = b"%s %s" % (parseddate, extra)
	l = [hex(manifest), user, parseddate] + sortedfiles + [b"", desc]			l = [hex(manifest), user, parseddate] + sortedfiles + [b"", desc]
	text = b"\n".join(l)			text = b"\n".join(l)
	return self.addrevision(			return self.addrevision(

mercurial/interfaces/repository.py

	REPO_FEATURE_STREAM_CLONE = b'streamclone'			REPO_FEATURE_STREAM_CLONE = b'streamclone'
	# Files storage may lack data for all ancestors.			# Files storage may lack data for all ancestors.
	REPO_FEATURE_SHALLOW_FILE_STORAGE = b'shallowfilestorage'			REPO_FEATURE_SHALLOW_FILE_STORAGE = b'shallowfilestorage'

	REVISION_FLAG_CENSORED = 1 << 15			REVISION_FLAG_CENSORED = 1 << 15
	REVISION_FLAG_ELLIPSIS = 1 << 14			REVISION_FLAG_ELLIPSIS = 1 << 14
	REVISION_FLAG_EXTSTORED = 1 << 13			REVISION_FLAG_EXTSTORED = 1 << 13
	REVISION_FLAG_SIDEDATA = 1 << 12			REVISION_FLAG_SIDEDATA = 1 << 12
				REVISION_FLAG_HASCOPIESINFO = 1 << 11

	REVISION_FLAGS_KNOWN = (			REVISION_FLAGS_KNOWN = (
	REVISION_FLAG_CENSORED			REVISION_FLAG_CENSORED
	\| REVISION_FLAG_ELLIPSIS			\| REVISION_FLAG_ELLIPSIS
	\| REVISION_FLAG_EXTSTORED			\| REVISION_FLAG_EXTSTORED
	\| REVISION_FLAG_SIDEDATA			\| REVISION_FLAG_SIDEDATA
				\| REVISION_FLAG_HASCOPIESINFO
	)			)

	CG_DELTAMODE_STD = b'default'			CG_DELTAMODE_STD = b'default'
	CG_DELTAMODE_PREV = b'previous'			CG_DELTAMODE_PREV = b'previous'
	CG_DELTAMODE_FULL = b'fulltext'			CG_DELTAMODE_FULL = b'fulltext'
	CG_DELTAMODE_P1 = b'p1'			CG_DELTAMODE_P1 = b'p1'

mercurial/revlog.py

	REVLOG_DEFAULT_FORMAT,			REVLOG_DEFAULT_FORMAT,
	REVLOG_DEFAULT_VERSION,			REVLOG_DEFAULT_VERSION,
	)			)
	from .revlogutils.flagutil import (			from .revlogutils.flagutil import (
	REVIDX_DEFAULT_FLAGS,			REVIDX_DEFAULT_FLAGS,
	REVIDX_ELLIPSIS,			REVIDX_ELLIPSIS,
	REVIDX_EXTSTORED,			REVIDX_EXTSTORED,
	REVIDX_FLAGS_ORDER,			REVIDX_FLAGS_ORDER,
				REVIDX_HASCOPIESINFO,
	REVIDX_ISCENSORED,			REVIDX_ISCENSORED,
	REVIDX_RAWTEXT_CHANGING_FLAGS,			REVIDX_RAWTEXT_CHANGING_FLAGS,
	REVIDX_SIDEDATA,			REVIDX_SIDEDATA,
	)			)
	from .thirdparty import attr			from .thirdparty import attr
	from . import (			from . import (
	ancestor,			ancestor,
	dagop,			dagop,
	REVLOG_DEFAULT_FLAGS			REVLOG_DEFAULT_FLAGS
	REVLOG_DEFAULT_FORMAT			REVLOG_DEFAULT_FORMAT
	REVLOG_DEFAULT_VERSION			REVLOG_DEFAULT_VERSION
	REVLOGV1_FLAGS			REVLOGV1_FLAGS
	REVLOGV2_FLAGS			REVLOGV2_FLAGS
	REVIDX_ISCENSORED			REVIDX_ISCENSORED
	REVIDX_ELLIPSIS			REVIDX_ELLIPSIS
	REVIDX_SIDEDATA			REVIDX_SIDEDATA
				REVIDX_HASCOPIESINFO
	REVIDX_EXTSTORED			REVIDX_EXTSTORED
	REVIDX_DEFAULT_FLAGS			REVIDX_DEFAULT_FLAGS
	REVIDX_FLAGS_ORDER			REVIDX_FLAGS_ORDER
	REVIDX_RAWTEXT_CHANGING_FLAGS			REVIDX_RAWTEXT_CHANGING_FLAGS

	parsers = policy.importmod('parsers')			parsers = policy.importmod('parsers')
	rustancestor = policy.importrust('ancestor')			rustancestor = policy.importrust('ancestor')
	rustdagop = policy.importrust('dagop')			rustdagop = policy.importrust('dagop')

mercurial/revlogutils/constants.py

	# revision has censor metadata, must be verified			# revision has censor metadata, must be verified
	REVIDX_ISCENSORED = repository.REVISION_FLAG_CENSORED			REVIDX_ISCENSORED = repository.REVISION_FLAG_CENSORED
	# revision hash does not match data (narrowhg)			# revision hash does not match data (narrowhg)
	REVIDX_ELLIPSIS = repository.REVISION_FLAG_ELLIPSIS			REVIDX_ELLIPSIS = repository.REVISION_FLAG_ELLIPSIS
	# revision data is stored externally			# revision data is stored externally
	REVIDX_EXTSTORED = repository.REVISION_FLAG_EXTSTORED			REVIDX_EXTSTORED = repository.REVISION_FLAG_EXTSTORED
	# revision data contains extra metadata not part of the official digest			# revision data contains extra metadata not part of the official digest
	REVIDX_SIDEDATA = repository.REVISION_FLAG_SIDEDATA			REVIDX_SIDEDATA = repository.REVISION_FLAG_SIDEDATA
				# revision changes files in a way that could affect copy tracing.
				REVIDX_HASCOPIESINFO = repository.REVISION_FLAG_HASCOPIESINFO
	REVIDX_DEFAULT_FLAGS = 0			REVIDX_DEFAULT_FLAGS = 0
	# stable order in which flags need to be processed and their processors applied			# stable order in which flags need to be processed and their processors applied
	REVIDX_FLAGS_ORDER = [			REVIDX_FLAGS_ORDER = [
	REVIDX_ISCENSORED,			REVIDX_ISCENSORED,
	REVIDX_ELLIPSIS,			REVIDX_ELLIPSIS,
	REVIDX_EXTSTORED,			REVIDX_EXTSTORED,
	REVIDX_SIDEDATA,			REVIDX_SIDEDATA,
				REVIDX_HASCOPIESINFO,
	]			]

	# bitmark for flags that could cause rawdata content change			# bitmark for flags that could cause rawdata content change
	REVIDX_RAWTEXT_CHANGING_FLAGS = (			REVIDX_RAWTEXT_CHANGING_FLAGS = (
	REVIDX_ISCENSORED \| REVIDX_EXTSTORED \| REVIDX_SIDEDATA			REVIDX_ISCENSORED \| REVIDX_EXTSTORED \| REVIDX_SIDEDATA
	)			)

	SPARSE_REVLOG_MAX_CHAIN_LENGTH = 1000			SPARSE_REVLOG_MAX_CHAIN_LENGTH = 1000

mercurial/revlogutils/flagutil.py


	from ..i18n import _			from ..i18n import _

	from .constants import (			from .constants import (
	REVIDX_DEFAULT_FLAGS,			REVIDX_DEFAULT_FLAGS,
	REVIDX_ELLIPSIS,			REVIDX_ELLIPSIS,
	REVIDX_EXTSTORED,			REVIDX_EXTSTORED,
	REVIDX_FLAGS_ORDER,			REVIDX_FLAGS_ORDER,
				REVIDX_HASCOPIESINFO,
	REVIDX_ISCENSORED,			REVIDX_ISCENSORED,
	REVIDX_RAWTEXT_CHANGING_FLAGS,			REVIDX_RAWTEXT_CHANGING_FLAGS,
	REVIDX_SIDEDATA,			REVIDX_SIDEDATA,
	)			)

	from .. import error, util			from .. import error, util

	# blanked usage of all the name to prevent pyflakes constraints			# blanked usage of all the name to prevent pyflakes constraints
	# We need these name available in the module for extensions.			# We need these name available in the module for extensions.
	REVIDX_ISCENSORED			REVIDX_ISCENSORED
	REVIDX_ELLIPSIS			REVIDX_ELLIPSIS
	REVIDX_EXTSTORED			REVIDX_EXTSTORED
	REVIDX_SIDEDATA			REVIDX_SIDEDATA
				REVIDX_HASCOPIESINFO,
	REVIDX_DEFAULT_FLAGS			REVIDX_DEFAULT_FLAGS
	REVIDX_FLAGS_ORDER			REVIDX_FLAGS_ORDER
	REVIDX_RAWTEXT_CHANGING_FLAGS			REVIDX_RAWTEXT_CHANGING_FLAGS

	REVIDX_KNOWN_FLAGS = util.bitsfrom(REVIDX_FLAGS_ORDER)			REVIDX_KNOWN_FLAGS = util.bitsfrom(REVIDX_FLAGS_ORDER)

	# Store flag processors (cf. 'addflagprocessor()' to register)			# Store flag processors (cf. 'addflagprocessor()' to register)
	flagprocessors = {			flagprocessors = {
	REVIDX_ISCENSORED: None,			REVIDX_ISCENSORED: None,
				REVIDX_HASCOPIESINFO: None,
	}			}


	def addflagprocessor(flag, processor):			def addflagprocessor(flag, processor):
	"""Register a flag processor on a revision data flag.			"""Register a flag processor on a revision data flag.

	Invariant:			Invariant:
	- Flags need to be defined in REVIDX_KNOWN_FLAGS and REVIDX_FLAGS_ORDER,			- Flags need to be defined in REVIDX_KNOWN_FLAGS and REVIDX_FLAGS_ORDER,

tests/test-lfs-serve.t

	... bprint(b"\n# LFS not enabled- revlogs don't have 0x2000 flag")			... bprint(b"\n# LFS not enabled- revlogs don't have 0x2000 flag")
	... stdout.flush()			... stdout.flush()
	... runcommand(server, [b'debugprocessors', b'nonlfs3.txt'])			... runcommand(server, [b'debugprocessors', b'nonlfs3.txt'])
	... runcommand(server, [b'config', b'extensions'])			... runcommand(server, [b'config', b'extensions'])

	# LFS required- both lfs and non-lfs revlogs have 0x2000 flag			# LFS required- both lfs and non-lfs revlogs have 0x2000 flag
	*** runcommand debugprocessors lfs.bin -R ../server			*** runcommand debugprocessors lfs.bin -R ../server
	registered processor '0x8000'			registered processor '0x8000'
				registered processor '0x800'
	registered processor '0x2000'			registered processor '0x2000'
	*** runcommand debugprocessors nonlfs2.txt -R ../server			*** runcommand debugprocessors nonlfs2.txt -R ../server
	registered processor '0x8000'			registered processor '0x8000'
				registered processor '0x800'
	registered processor '0x2000'			registered processor '0x2000'
	*** runcommand config extensions --cwd ../server			*** runcommand config extensions --cwd ../server
	extensions.debugprocessors=$TESTTMP/debugprocessors.py			extensions.debugprocessors=$TESTTMP/debugprocessors.py
	extensions.lfs=			extensions.lfs=

	# LFS not enabled- revlogs don't have 0x2000 flag			# LFS not enabled- revlogs don't have 0x2000 flag
	*** runcommand debugprocessors nonlfs3.txt			*** runcommand debugprocessors nonlfs3.txt
	registered processor '0x8000'			registered processor '0x8000'
				registered processor '0x800'
	*** runcommand config extensions			*** runcommand config extensions
	extensions.debugprocessors=$TESTTMP/debugprocessors.py			extensions.debugprocessors=$TESTTMP/debugprocessors.py

	$ rm $HGRCPATH			$ rm $HGRCPATH
	$ mv $HGRCPATH.tmp $HGRCPATH			$ mv $HGRCPATH.tmp $HGRCPATH

	$ hg clone $TESTTMP/client $TESTTMP/nonlfs -qr 0 --config extensions.lfs=			$ hg clone $TESTTMP/client $TESTTMP/nonlfs -qr 0 --config extensions.lfs=
	$ cat >> $TESTTMP/nonlfs/.hg/hgrc <<EOF			$ cat >> $TESTTMP/nonlfs/.hg/hgrc <<EOF
	... runcommand(server, [b'debugprocessors', b'nonlfs.txt', b'-R',			... runcommand(server, [b'debugprocessors', b'nonlfs.txt', b'-R',
	... b'../nonlfs'])			... b'../nonlfs'])
	... runcommand(server, [b'config', b'extensions', b'--cwd',			... runcommand(server, [b'config', b'extensions', b'--cwd',
	... b'../nonlfs'])			... b'../nonlfs'])

	# LFS enabled- both lfs and non-lfs revlogs have 0x2000 flag			# LFS enabled- both lfs and non-lfs revlogs have 0x2000 flag
	*** runcommand debugprocessors lfs.bin -R ../server			*** runcommand debugprocessors lfs.bin -R ../server
	registered processor '0x8000'			registered processor '0x8000'
				registered processor '0x800'
	registered processor '0x2000'			registered processor '0x2000'
	*** runcommand debugprocessors nonlfs2.txt -R ../server			*** runcommand debugprocessors nonlfs2.txt -R ../server
	registered processor '0x8000'			registered processor '0x8000'
				registered processor '0x800'
	registered processor '0x2000'			registered processor '0x2000'
	*** runcommand config extensions --cwd ../server			*** runcommand config extensions --cwd ../server
	extensions.debugprocessors=$TESTTMP/debugprocessors.py			extensions.debugprocessors=$TESTTMP/debugprocessors.py
	extensions.lfs=			extensions.lfs=

	# LFS enabled without requirement- revlogs have 0x2000 flag			# LFS enabled without requirement- revlogs have 0x2000 flag
	*** runcommand debugprocessors nonlfs3.txt			*** runcommand debugprocessors nonlfs3.txt
	registered processor '0x8000'			registered processor '0x8000'
				registered processor '0x800'
	registered processor '0x2000'			registered processor '0x2000'
	*** runcommand config extensions			*** runcommand config extensions
	extensions.debugprocessors=$TESTTMP/debugprocessors.py			extensions.debugprocessors=$TESTTMP/debugprocessors.py
	extensions.lfs=			extensions.lfs=

	# LFS disabled locally- revlogs don't have 0x2000 flag			# LFS disabled locally- revlogs don't have 0x2000 flag
	*** runcommand debugprocessors nonlfs.txt -R ../nonlfs			*** runcommand debugprocessors nonlfs.txt -R ../nonlfs
	registered processor '0x8000'			registered processor '0x8000'
				registered processor '0x800'
	*** runcommand config extensions --cwd ../nonlfs			*** runcommand config extensions --cwd ../nonlfs
	extensions.debugprocessors=$TESTTMP/debugprocessors.py			extensions.debugprocessors=$TESTTMP/debugprocessors.py
	extensions.lfs=!			extensions.lfs=!

	--------------------------------------------------------------------------------			--------------------------------------------------------------------------------
	Case #6: client with lfs content and the extension enabled; server with			Case #6: client with lfs content and the extension enabled; server with
	lfs content, and the extension enabled.			lfs content, and the extension enabled.

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	22970		Oct 1 2020, 10:09 AM	★	★
Diff 2	23108	rHGc398c79824248fecee352af14b0bd6d3fe8d4995	Oct 1 2020, 3:29 AM	★	★