mercurial/revlogutils/censor.py
154	We should replace `cut_off` to `cutoff` in all of those signatures, since it's the noun and not the verb, to avoid confusion
225	Should be `all_files`
302	`data_delta_base, start, end, rewritten_entries[rev]` should be `data_delta_base, start, end = rewritten_entries[rev]`. This makes me think that the censor tests do not look at the case where a revs have a censored node as delta base, which wouldn't be surprising since the last version didn't handle this. But we should add a test case for this.

mercurial/revlogutils/censor.py
190	I'm not really sure I understand what we're doing here with the compression settings. (also, the comment about `tell()` is not at the right place anymore)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	28552		Jun 11 2021, 6:56 AM	★	★
Diff 2	28569		Jun 15 2021, 7:39 AM	★	★
Diff 3	28617		Jun 20 2021, 5:22 PM	★	★
Diff 4	28629		Jun 21 2021, 1:14 AM	★	★
Diff 5	28647		Jun 21 2021, 9:10 AM	★	★
Diff 6	28665	rHGf8330a3fc39fa87bbd4f6215068282d2b7bbe091	Jun 7 2021, 5:59 AM	★	★

Commit	Parents	Author	Summary	Date
bfca867576ca	fb455d73e924	Pierre-Yves David		Jun 7 2021, 5:59 AM

Status	Author	Revision
Closed	marmoute	D10869 censor: implement censoring for revlogv2
Closed	marmoute	D10887 deltas: at a `target_rev` parameter to finddeltainfo
Closed	marmoute	D10868 revlog: list older-but-still-around file in `files`
Closed	marmoute	D10867 revlog: add docket method to request new content files
Closed	marmoute	D10866 revlog: add a way to keep track of older uids in the docket
Closed	marmoute	D10889 revlog: factor the logic to determine the delta compression out
Closed	marmoute	D10865 nodemap: fix some comment formatting
Closed	marmoute	D10858 revlogv2: fix `hg verify` with revlog v2
Closed	marmoute	D10857 revlogv2: also test that local clone works
Closed	marmoute	D10856 clone: also report the bookmark file as copied
Closed	marmoute	D10855 clone: reuse the stream clone logic for local clone
Closed	marmoute	D10854 copyfiles: add a way to relax the file system checking for hardlink
Closed	marmoute	D10853 copyfile: add a option callback for failed hardlinking
Closed	marmoute	D10852 streamingclone: extract the scanning part from the generation part
Closed	marmoute	D10851 vfs: add a `register_file` method on the vfs class
Closed	marmoute	D10850 clone: use "official" API to create local clone destination
Closed	marmoute	D10849 localrepo: introduce a clone_requirements function
Closed	marmoute	D10848 createrepository: allow to directly pass the target requirements
Abandoned	marmoute	D10847 copyfile: add overlay in the whitelist for hardlink
Closed	marmoute	D10846 transaction: simplify `undo.backupfiles` file creation with a variable
Closed	marmoute	D10845 transaction: narrow the error filtering when failing to rename undo file
Closed	marmoute	D10844 transaction: clarify a conditionnal about version check
Closed	marmoute	D10843 transaction: extract message about different version in a constants
Closed	marmoute	D10842 transaction: explain why some recovery failed
Closed	marmoute	D10841 copyfiles: deal with existing file when hardlinking
Closed	marmoute	D10840 clone: cleanup the "cleanup dir" logic used during local clone

Diff 28629

mercurial/revlog.py

	if self._format_version == REVLOGV0:			if self._format_version == REVLOGV0:
	raise error.RevlogError(			raise error.RevlogError(
	_(b'cannot censor with version %d revlogs')			_(b'cannot censor with version %d revlogs')
	% self._format_version			% self._format_version
	)			)
	elif self._format_version == REVLOGV1:			elif self._format_version == REVLOGV1:
	censor.v1_censor(self, tr, censornode, tombstone)			censor.v1_censor(self, tr, censornode, tombstone)
	else:			else:
	# revlog v2			censor.v2_censor(self, tr, censornode, tombstone)
	raise error.RevlogError(
	_(b'cannot censor with version %d revlogs')
	% self._format_version
	)

	def verifyintegrity(self, state):			def verifyintegrity(self, state):
	"""Verifies the integrity of the revlog.			"""Verifies the integrity of the revlog.

	Yields ``revlogproblem`` instances describing problems that are			Yields ``revlogproblem`` instances describing problems that are
	found.			found.
	"""			"""
	dd, di = self.checksize()			dd, di = self.checksize()

mercurial/revlogutils/censor.py

	# censor code related to censoring revision			# censor code related to censoring revision
				# coding: utf8
	#			#
	# Copyright 2021 Pierre-Yves David <pierre-yves.david@octobus.net>			# Copyright 2021 Pierre-Yves David <pierre-yves.david@octobus.net>
	# Copyright 2015 Google, Inc <martinvonz@google.com>			# Copyright 2015 Google, Inc <martinvonz@google.com>
	#			#
	# This software may be used and distributed according to the terms of the			# This software may be used and distributed according to the terms of the
	# GNU General Public License version 2 or any later version.			# GNU General Public License version 2 or any later version.

				import contextlib
				import os

	from ..node import (			from ..node import (
	nullrev,			nullrev,
	)			)
				from .constants import (
				COMP_MODE_DEFAULT,
				COMP_MODE_PLAIN,
				ENTRY_DATA_COMPRESSED_LENGTH,
				ENTRY_DATA_COMPRESSION_MODE,
				ENTRY_DATA_OFFSET,
				ENTRY_DATA_UNCOMPRESSED_LENGTH,
				ENTRY_DELTA_BASE,
				ENTRY_LINK_REV,
				ENTRY_NODE_ID,
				ENTRY_PARENT_1,
				ENTRY_PARENT_2,
				ENTRY_SIDEDATA_COMPRESSED_LENGTH,
				ENTRY_SIDEDATA_COMPRESSION_MODE,
				ENTRY_SIDEDATA_OFFSET,
				REVLOGV0,
				REVLOGV1,
				)
	from ..i18n import _			from ..i18n import _

	from .. import (			from .. import (
	error,			error,
				pycompat,
				revlogutils,
				util,
	)			)
	from ..utils import (			from ..utils import (
	storageutil,			storageutil,
	)			)
	from . import constants			from . import (
				constants,
				deltas,
				)


	def v1_censor(rl, tr, censornode, tombstone=b''):			def v1_censor(rl, tr, censornode, tombstone=b''):
	"""censors a revision in a "version 1" revlog"""			"""censors a revision in a "version 1" revlog"""
	assert rl._format_version == constants.REVLOGV1, rl._format_version			assert rl._format_version == constants.REVLOGV1, rl._format_version

	# avoid cycle			# avoid cycle
	from .. import revlog			from .. import revlog
	tr.addbackup(rl._datafile, location=b'store')			tr.addbackup(rl._datafile, location=b'store')

	rl.opener.rename(newrl._indexfile, rl._indexfile)			rl.opener.rename(newrl._indexfile, rl._indexfile)
	if not rl._inline:			if not rl._inline:
	rl.opener.rename(newrl._datafile, rl._datafile)			rl.opener.rename(newrl._datafile, rl._datafile)

	rl.clearcaches()			rl.clearcaches()
	rl._loadindex()			rl._loadindex()


				def v2_censor(rl, tr, censornode, tombstone=b''):
				"""censors a revision in a "version 2" revlog"""
				# General principle
				#
				# We create new revlog files (index/data/sidedata) to copy the content of
				# the existing data without the censored data.
				#
				# We need to recompute new delta for any revision that used the censored
				# revision as delta base. As the cumulative size of the new delta may be
				# large, we store them in a temporary file until they are stored in their
				# final destination.
				#
				# All data before the censored data can be blindly copied. The rest needs
				# to be copied as we go and the associated index entry needs adjustement.

				assert rl._format_version != REVLOGV0, rl._format_version
				assert rl._format_version != REVLOGV1, rl._format_version

				old_index = rl.index
				docket = rl._docket

				censor_rev = rl.rev(censornode)
				tombstone = storageutil.packmeta({b'censored': tombstone}, b'')

				censored_entry = rl.index[censor_rev]
				index_cutoff = rl.index.entry_size * censor_rev
				AlphareUnsubmitted Not Done We should replace `cut_off` to `cutoff` in all of those signatures, since it's the noun and not the verb, to avoid confusion Alphare: We should replace `cut_off` to `cutoff` in all of those signatures, since it's the noun and not…
				data_cutoff = censored_entry[ENTRY_DATA_OFFSET] >> 16
				sidedata_cutoff = rl.sidedata_cut_off(censor_rev)

				# rev → (new_base, data_start, data_end)
				rewritten_entries = {}

				dc = deltas.deltacomputer(rl)
				excl = [censor_rev]

				with pycompat.unnamedtempfile(mode=b"w+b") as tmp_storage:
				with rl._segmentfile._open_read() as dfh:
				for rev in range(censor_rev + 1, len(old_index)):
				entry = old_index[rev]
				if censor_rev != entry[ENTRY_DELTA_BASE]:
				continue
				# This is a revision that use the censored revision as the base
				# for its delta. We need a need new deltas
				if entry[ENTRY_DATA_UNCOMPRESSED_LENGTH] == 0:
				# this revision is empty, we can delta against nullrev
				rewritten_entries[rev] = (nullrev, 0, 0)
				else:

				text = rl.rawdata(rev, _df=dfh)
				info = revlogutils.revisioninfo(
				node=entry[ENTRY_NODE_ID],
				p1=rl.node(entry[ENTRY_PARENT_1]),
				p2=rl.node(entry[ENTRY_PARENT_2]),
				btext=[text],
				textlen=len(text),
				cachedelta=None,
				flags=entry[ENTRY_DATA_OFFSET] & 0xFFFF,
				)
				d = dc.finddeltainfo(
				info, dfh, excluded_bases=excl, target_rev=rev
				)
				# using `tell` is a bit lazy, but we are not here for speed
				AlphareUnsubmitted Not Done I'm not really sure I understand what we're doing here with the compression settings. (also, the comment about `tell()` is not at the right place anymore) Alphare: I'm not really sure I understand what we're doing here with the compression settings. (also…
				h, data = d.data
				if not h and not data:
				# not data to store at all... declare them uncompressed
				comp_mode = COMP_MODE_PLAIN
				elif not h:
				t = data[0:1]
				if t == b'\0':
				comp_mode = COMP_MODE_PLAIN
				elif t == rl._docket.default_compression_header:
				comp_mode = COMP_MODE_DEFAULT
				elif h == b'u':
				# we have a more efficient way to declare uncompressed
				h = b''
				comp_mode = COMP_MODE_PLAIN
				d = deltas.drop_u_compression(d)
				start = tmp_storage.tell()
				tmp_storage.write(d.data[1])
				end = tmp_storage.tell()
				rewritten_entries[rev] = (d.base, start, end, comp_mode)

				old_index_filepath = rl.opener.join(docket.index_filepath())
				old_data_filepath = rl.opener.join(docket.data_filepath())
				old_sidedata_filepath = rl.opener.join(docket.sidedata_filepath())

				new_index_filepath = rl.opener.join(docket.new_index_file())
				new_data_filepath = rl.opener.join(docket.new_data_file())
				new_sidedata_filepath = rl.opener.join(docket.new_sidedata_file())

				util.copyfile(
				old_index_filepath, new_index_filepath, nb_bytes=index_cutoff
				)
				util.copyfile(
				old_data_filepath, new_data_filepath, nb_bytes=data_cutoff
				)
				util.copyfile(
				AlphareUnsubmitted Not Done Should be `all_files` Alphare: Should be `all_files`
				old_sidedata_filepath,
				new_sidedata_filepath,
				nb_bytes=sidedata_cutoff,
				)
				rl.opener.register_file(docket.index_filepath())
				rl.opener.register_file(docket.data_filepath())
				rl.opener.register_file(docket.sidedata_filepath())

				docket.index_end = index_cutoff
				docket.data_end = data_cutoff
				docket.sidedata_end = sidedata_cutoff

				# reload the revlog internal information
				rl.clearcaches()
				rl._loadindex(docket=docket)

				@contextlib.contextmanager
				def all_files():
				# hide opening in an helper function to please check-code, black
				# and various python ersion at the same time
				with open(old_data_filepath, 'rb') as old_data_file:
				with open(old_sidedata_filepath, 'rb') as old_sidedata_file:
				with open(new_index_filepath, 'r+b') as new_index_file:
				with open(new_data_filepath, 'r+b') as new_data_file:
				with open(
				new_sidedata_filepath, 'r+b'
				) as new_sidedata_file:
				yield (
				old_data_file,
				old_sidedata_file,
				new_index_file,
				new_data_file,
				new_sidedata_file,
				)

				# we dont need to open the old index file since its content already
				# exist in a usable form in `old_index`.
				with all_files() as (
				old_data_file,
				old_sidedata_file,
				new_index_file,
				new_data_file,
				new_sidedata_file,
				):
				new_index_file.seek(0, os.SEEK_END)
				assert new_index_file.tell() == index_cutoff
				new_data_file.seek(0, os.SEEK_END)
				assert new_data_file.tell() == data_cutoff
				new_sidedata_file.seek(0, os.SEEK_END)
				assert new_sidedata_file.tell() == sidedata_cutoff

				### writing the censored revision
				entry = old_index[censor_rev]

				# XXX consider trying the default compression too
				new_data_size = len(tombstone)
				new_data_offset = new_data_file.tell()
				new_data_file.write(tombstone)

				# we are not adding any sidedata as they might leak info about the censored version

				new_entry = revlogutils.entry(
				flags=constants.REVIDX_ISCENSORED,
				data_offset=new_data_offset,
				data_compressed_length=new_data_size,
				data_uncompressed_length=new_data_size,
				data_delta_base=censor_rev,
				link_rev=entry[ENTRY_LINK_REV],
				parent_rev_1=entry[ENTRY_PARENT_1],
				parent_rev_2=entry[ENTRY_PARENT_2],
				node_id=entry[ENTRY_NODE_ID],
				sidedata_offset=0,
				sidedata_compressed_length=0,
				data_compression_mode=COMP_MODE_PLAIN,
				sidedata_compression_mode=COMP_MODE_PLAIN,
				)
				rl.index.append(new_entry)
				AlphareUnsubmitted Not Done `data_delta_base, start, end, rewritten_entries[rev]` should be `data_delta_base, start, end = rewritten_entries[rev]`. This makes me think that the censor tests do not look at the case where a revs have a censored node as delta base, which wouldn't be surprising since the last version didn't handle this. But we should add a test case for this. Alphare: `data_delta_base, start, end, rewritten_entries[rev]` should be `data_delta_base, start, end =…
				entry_bin = rl.index.entry_binary(censor_rev)
				new_index_file.write(entry_bin)
				docket.index_end = new_index_file.tell()
				docket.data_end = new_data_file.tell()

				#### Writing all subsequent revisions
				for rev in range(censor_rev + 1, len(old_index)):
				entry = old_index[rev]
				flags = entry[ENTRY_DATA_OFFSET] & 0xFFFF
				old_data_offset = entry[ENTRY_DATA_OFFSET] >> 16

				if rev not in rewritten_entries:
				old_data_file.seek(old_data_offset)
				new_data_size = entry[ENTRY_DATA_COMPRESSED_LENGTH]
				new_data = old_data_file.read(new_data_size)
				data_delta_base = entry[ENTRY_DELTA_BASE]
				d_comp_mode = entry[ENTRY_DATA_COMPRESSION_MODE]
				else:
				(
				data_delta_base,
				start,
				end,
				d_comp_mode,
				) = rewritten_entries[rev]
				new_data_size = end - start
				tmp_storage.seek(start)
				new_data = tmp_storage.read(new_data_size)

				# It might be faster to group continuous read/write operation,
				# however, this is censor, an operation that is not focussed
				# around stellar performance. So I have not written this
				# optimisation yet.
				new_data_offset = new_data_file.tell()
				new_data_file.write(new_data)

				sidedata_size = entry[ENTRY_SIDEDATA_COMPRESSED_LENGTH]
				new_sidedata_offset = new_sidedata_file.tell()
				if 0 < sidedata_size:
				old_sidedata_offset = entry[ENTRY_SIDEDATA_OFFSET]
				old_sidedata_file.seek(old_sidedata_offset)
				new_sidedata = old_sidedata_file.read(sidedata_size)
				new_sidedata_file.write(new_sidedata)

				data_uncompressed_length = entry[ENTRY_DATA_UNCOMPRESSED_LENGTH]
				sd_com_mode = entry[ENTRY_SIDEDATA_COMPRESSION_MODE]
				assert data_delta_base <= rev, (data_delta_base, rev)

				new_entry = revlogutils.entry(
				flags=flags,
				data_offset=new_data_offset,
				data_compressed_length=new_data_size,
				data_uncompressed_length=data_uncompressed_length,
				data_delta_base=data_delta_base,
				link_rev=entry[ENTRY_LINK_REV],
				parent_rev_1=entry[ENTRY_PARENT_1],
				parent_rev_2=entry[ENTRY_PARENT_2],
				node_id=entry[ENTRY_NODE_ID],
				sidedata_offset=new_sidedata_offset,
				sidedata_compressed_length=sidedata_size,
				data_compression_mode=d_comp_mode,
				sidedata_compression_mode=sd_com_mode,
				)
				rl.index.append(new_entry)
				entry_bin = rl.index.entry_binary(rev)
				new_index_file.write(entry_bin)

				docket.index_end = new_index_file.tell()
				docket.data_end = new_data_file.tell()
				docket.sidedata_end = new_sidedata_file.tell()

				docket.write(transaction=None, stripping=True)

mercurial/revlogutils/deltas.py

	If no suitable deltabase is found, we return delta info for a full			If no suitable deltabase is found, we return delta info for a full
	snapshot.			snapshot.

	`excluded_bases` is an optional set of revision that cannot be used as			`excluded_bases` is an optional set of revision that cannot be used as
	a delta base. Use this to recompute delta suitable in censor or strip			a delta base. Use this to recompute delta suitable in censor or strip
	context.			context.
	"""			"""
	if target_rev is None:			if target_rev is None:
	curr = len(self.revlog)			target_rev = len(self.revlog)

	if not revinfo.textlen:			if not revinfo.textlen:
	return self._fullsnapshotinfo(fh, revinfo, target_rev)			return self._fullsnapshotinfo(fh, revinfo, target_rev)

	if excluded_bases is None:			if excluded_bases is None:
	excluded_bases = set()			excluded_bases = set()

	# no delta for flag processor revision (see "candelta" for why)			# no delta for flag processor revision (see "candelta" for why)

tests/test-censor.t

	#require no-reposimplestore			#require no-reposimplestore
				#testcases revlogv1 revlogv2

				#if revlogv2

				$ cat >> $HGRCPATH <<EOF
				> [experimental]
				> revlogv2=enable-unstable-format-and-corrupt-my-data
				> EOF

				#endif

	$ cat >> $HGRCPATH <<EOF			$ cat >> $HGRCPATH <<EOF
	> [extensions]			> [extensions]
	> censor=			> censor=
	> EOF			> EOF
	$ cp $HGRCPATH $HGRCPATH.orig			$ cp $HGRCPATH $HGRCPATH.orig

	Create repo with unimpeachable content			Create repo with unimpeachable content
	$ hg unbundle initbundle			$ hg unbundle initbundle
	adding changesets			adding changesets
	adding manifests			adding manifests
	adding file changes			adding file changes
	added 1 changesets with 2 changes to 2 files			added 1 changesets with 2 changes to 2 files
	new changesets e97f55b2665a (1 drafts)			new changesets e97f55b2665a (1 drafts)
	(run 'hg update' to get a working copy)			(run 'hg update' to get a working copy)
	$ hg cat -r 0 target \| head -n 10			$ hg cat -r 0 target \| head -n 10

				#if revlogv2

				Testing feature that does not work in revlog v1
				===============================================

				Censoring a revision that is used as delta base
				-----------------------------------------------

				$ cd ..
				$ hg init censor-with-delta
				$ cd censor-with-delta
				$ echo root > target
				$ hg add target
				$ hg commit -m root
				$ B0=`hg id --debug -i`
				$ for x in `"$PYTHON" $TESTDIR/seq.py 0 50000`
				> do
				> echo "Password: hunter$x" >> target
				> done
				$ hg ci -m 'write a long file'
				$ B1=`hg id --debug -i`
				$ echo 'small change (should create a delta)' >> target
				$ hg ci -m 'create a delta over the password'
				(should show that the last revision is a delta, not a snapshot)
				$ B2=`hg id --debug -i`

				Make sure the last revision is a delta against the revision we will censor

				$ hg debugdeltachain target -T '{rev} {chainid} {chainlen} {prevrev}\n'
				0 1 1 -1
				1 2 1 -1
				2 2 2 1

				Censor the file

				$ hg cat -r $B1 target \| wc -l
				50002 (re)
				$ hg censor -r $B1 target
				$ hg cat -r $B1 target \| wc -l
				0 (re)

				Check the children is fine

				$ hg cat -r $B2 target \| wc -l
				50003 (re)

				#endif

			Path	Packages
M			mercurial/revlog.py (6 lines)
M			mercurial/revlogutils/censor.py (278 lines)
M			mercurial/revlogutils/deltas.py (2 lines)
M			tests/test-censor.t (58 lines)

This is an archive of the discontinued Mercurial Phabricator instance.

censor: implement censoring for revlogv2
ClosedPublic

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 28629

mercurial/revlog.py

mercurial/revlogutils/censor.py

mercurial/revlogutils/deltas.py

tests/test-censor.t

This is an archive of the discontinued Mercurial Phabricator instance.

censor: implement censoring for revlogv2ClosedPublic

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 28629

mercurial/revlog.py

mercurial/revlogutils/censor.py

mercurial/revlogutils/deltas.py

tests/test-censor.t

censor: implement censoring for revlogv2
ClosedPublic

Revision Contents
Changeset List