This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10788

censor: reduce risk of censor test blowing up output limit
ClosedPublic

Authored by marmoute on May 28 2021, 6:56 PM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHG65b86f516ba2: censor: reduce risk of censor test blowing up output limit
Summary

The censored file is really big, and we keep using hg cat on it, with the
assumption that it is censored. However, when it is not censored, the amount of
extra line output is huge and quickly saturate the backlog of terminal and CI
system.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.May 28 2021, 6:56 PM
baymax updated this revision to Diff 28253.May 30 2021, 12:07 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

baymax updated this revision to Diff 28301.May 31 2021, 6:51 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Alphare accepted this revision as: Alphare.May 31 2021, 10:31 AM
baymax updated this revision to Diff 28316.May 31 2021, 4:21 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

baymax updated this revision to Diff 28398.Jun 1 2021, 11:17 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

baymax updated this revision to Diff 28413.Jun 2 2021, 7:43 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

baymax updated this revision to Diff 28447.Jun 4 2021, 11:04 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

baymax updated this revision to Diff 28469.Jun 7 2021, 5:16 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

marmoute added a commit: rHG65b86f516ba2: censor: reduce risk of censor test blowing up output limit.Jun 7 2021, 5:23 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG65b86f516ba2: censor: reduce risk of censor test blowing up output limit (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mtests/test-censor.t (128 lines)

Diff 28491

tests/test-censor.t

1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ echo 'Tainted file now super sanitized' > target $ echo 'Tainted file now super sanitized' > target
$ hg ci -m 'super sanitized' target $ hg ci -m 'super sanitized' target
created new head created new head
$ H2=`hg id --debug -i` $ H2=`hg id --debug -i`
Verify target contents before censorship at each revision Verify target contents before censorship at each revision
$ hg cat -r $H1 target $ hg cat -r $H1 target | head -n 10
Tainted file is now sanitized Tainted file is now sanitized
$ hg cat -r $H2 target $ hg cat -r $H2 target | head -n 10
Tainted file now super sanitized Tainted file now super sanitized
$ hg cat -r $C2 target $ hg cat -r $C2 target | head -n 10
Tainted file Tainted file
Passwords: hunter2 Passwords: hunter2
hunter3 hunter3
$ hg cat -r $C1 target $ hg cat -r $C1 target | head -n 10
Tainted file Tainted file
Passwords: hunter2 Passwords: hunter2
$ hg cat -r 0 target $ hg cat -r 0 target | head -n 10
Initially untainted file Initially untainted file
Try to censor revision with too large of a tombstone message Try to censor revision with too large of a tombstone message
$ hg censor -r $C1 -t 'blah blah blah blah blah blah blah blah bla' target $ hg censor -r $C1 -t 'blah blah blah blah blah blah blah blah bla' target
abort: censor tombstone must be no longer than censored data abort: censor tombstone must be no longer than censored data
[255] [255]
Censor revision with 2 offenses Censor revision with 2 offenses
(this also tests file pattern matching: path relative to cwd case) (this also tests file pattern matching: path relative to cwd case)
$ mkdir -p foo/bar/baz $ mkdir -p foo/bar/baz
$ hg --cwd foo/bar/baz censor -r $C2 -t "remove password" ../../../target $ hg --cwd foo/bar/baz censor -r $C2 -t "remove password" ../../../target
$ hg cat -r $H1 target $ hg cat -r $H1 target | head -n 10
Tainted file is now sanitized Tainted file is now sanitized
$ hg cat -r $H2 target $ hg cat -r $H2 target | head -n 10
Tainted file now super sanitized Tainted file now super sanitized
$ hg cat -r $C2 target $ hg cat -r $C2 target | head -n 10
abort: censored node: 1e0247a9a4b7 abort: censored node: 1e0247a9a4b7
(set censor.policy to ignore errors) (set censor.policy to ignore errors)
[255] $ hg cat -r $C1 target | head -n 10
$ hg cat -r $C1 target
Tainted file Tainted file
Passwords: hunter2 Passwords: hunter2
$ hg cat -r 0 target $ hg cat -r 0 target | head -n 10
Initially untainted file Initially untainted file
Censor revision with 1 offense Censor revision with 1 offense
(this also tests file pattern matching: with 'path:' scheme) (this also tests file pattern matching: with 'path:' scheme)
$ hg --cwd foo/bar/baz censor -r $C1 path:target $ hg --cwd foo/bar/baz censor -r $C1 path:target
$ hg cat -r $H1 target $ hg cat -r $H1 target | head -n 10
Tainted file is now sanitized Tainted file is now sanitized
$ hg cat -r $H2 target $ hg cat -r $H2 target | head -n 10
Tainted file now super sanitized Tainted file now super sanitized
$ hg cat -r $C2 target $ hg cat -r $C2 target | head -n 10
abort: censored node: 1e0247a9a4b7 abort: censored node: 1e0247a9a4b7
(set censor.policy to ignore errors) (set censor.policy to ignore errors)
[255] $ hg cat -r $C1 target | head -n 10
$ hg cat -r $C1 target
abort: censored node: 613bc869fceb abort: censored node: 613bc869fceb
(set censor.policy to ignore errors) (set censor.policy to ignore errors)
[255] $ hg cat -r 0 target | head -n 10
$ hg cat -r 0 target
Initially untainted file Initially untainted file
Can only checkout target at uncensored revisions, -X is workaround for --all Can only checkout target at uncensored revisions, -X is workaround for --all
$ hg revert -r $C2 target $ hg revert -r $C2 target | head -n 10
abort: censored node: 1e0247a9a4b7 abort: censored node: 1e0247a9a4b7
(set censor.policy to ignore errors) (set censor.policy to ignore errors)
[255] $ hg revert -r $C1 target | head -n 10
$ hg revert -r $C1 target
abort: censored node: 613bc869fceb abort: censored node: 613bc869fceb
(set censor.policy to ignore errors) (set censor.policy to ignore errors)
[255]
$ hg revert -r $C1 --all $ hg revert -r $C1 --all
reverting bystander reverting bystander
reverting target reverting target
abort: censored node: 613bc869fceb abort: censored node: 613bc869fceb
(set censor.policy to ignore errors) (set censor.policy to ignore errors)
[255] [255]
$ hg revert -r $C1 --all -X target $ hg revert -r $C1 --all -X target
$ cat target $ cat target | head -n 10
Tainted file now super sanitized Tainted file now super sanitized
$ hg revert -r 0 --all $ hg revert -r 0 --all
reverting target reverting target
$ cat target $ cat target | head -n 10
Initially untainted file Initially untainted file
$ hg revert -r $H2 --all $ hg revert -r $H2 --all
reverting bystander reverting bystander
reverting target reverting target
$ cat target $ cat target | head -n 10
Tainted file now super sanitized Tainted file now super sanitized
Uncensored file can be viewed at any revision Uncensored file can be viewed at any revision
$ hg cat -r $H1 bystander $ hg cat -r $H1 bystander | head -n 10
Normal file v2 Normal file v2
$ hg cat -r $C2 bystander $ hg cat -r $C2 bystander | head -n 10
Normal file v2 Normal file v2
$ hg cat -r $C1 bystander $ hg cat -r $C1 bystander | head -n 10
Normal file here Normal file here
$ hg cat -r 0 bystander $ hg cat -r 0 bystander | head -n 10
Normal file here Normal file here
Can update to children of censored revision Can update to children of censored revision
$ hg update -r $H1 $ hg update -r $H1
1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat target $ cat target | head -n 10
Tainted file is now sanitized Tainted file is now sanitized
$ hg update -r $H2 $ hg update -r $H2
1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat target $ cat target | head -n 10
Tainted file now super sanitized Tainted file now super sanitized
Set censor policy to abort in trusted $HGRC so hg verify fails Set censor policy to abort in trusted $HGRC so hg verify fails
$ cp $HGRCPATH.orig $HGRCPATH $ cp $HGRCPATH.orig $HGRCPATH
$ cat >> $HGRCPATH <<EOF $ cat >> $HGRCPATH <<EOF
> [censor] > [censor]
> policy = abort > policy = abort
crosschecking files in changesets and manifests crosschecking files in changesets and manifests
checking files checking files
checked 5 changesets with 7 changes to 2 files checked 5 changesets with 7 changes to 2 files
May update to revision with censored data with explicit config May update to revision with censored data with explicit config
$ hg update -r $C2 $ hg update -r $C2
1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat target $ cat target | head -n 10
$ hg update -r $C1 $ hg update -r $C1
2 files updated, 0 files merged, 0 files removed, 0 files unresolved 2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat target $ cat target | head -n 10
$ hg update -r 0 $ hg update -r 0
1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat target $ cat target | head -n 10
Initially untainted file Initially untainted file
$ hg update -r $H2 $ hg update -r $H2
2 files updated, 0 files merged, 0 files removed, 0 files unresolved 2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat target $ cat target | head -n 10
Tainted file now super sanitized Tainted file now super sanitized
Can merge in revision with censored data. Test requires one branch of history Can merge in revision with censored data. Test requires one branch of history
with the file censored, but we can't censor at a head, so advance H1. with the file censored, but we can't censor at a head, so advance H1.
$ hg update -r $H1 $ hg update -r $H1
1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ C3=$H1 $ C3=$H1
Can re-add file after being deleted + censored Can re-add file after being deleted + censored
$ C4=$H2 $ C4=$H2
$ hg rm target $ hg rm target
$ hg ci -m 'delete target so it may be censored' $ hg ci -m 'delete target so it may be censored'
$ H2=`hg id --debug -i` $ H2=`hg id --debug -i`
$ hg censor -r $C4 target $ hg censor -r $C4 target
$ hg cat -r $C4 target $ hg cat -r $C4 target | head -n 10
$ hg cat -r "$H2^^" target $ hg cat -r "$H2^^" target | head -n 10
Tainted file now super sanitized Tainted file now super sanitized
$ echo 'fresh start' > target $ echo 'fresh start' > target
$ hg add target $ hg add target
$ hg ci -m reincarnated target $ hg ci -m reincarnated target
$ H2=`hg id --debug -i` $ H2=`hg id --debug -i`
$ hg cat -r $H2 target $ hg cat -r $H2 target | head -n 10
fresh start fresh start
$ hg cat -r "$H2^" target $ hg cat -r "$H2^" target | head -n 10
target: no such file in rev 452ec1762369 target: no such file in rev 452ec1762369
[1] $ hg cat -r $C4 target | head -n 10
$ hg cat -r $C4 target $ hg cat -r "$H2^^^" target | head -n 10
$ hg cat -r "$H2^^^" target
Tainted file now super sanitized Tainted file now super sanitized
Can censor after revlog has expanded to no longer permit inline storage Can censor after revlog has expanded to no longer permit inline storage
$ for x in `"$PYTHON" $TESTDIR/seq.py 0 50000` $ for x in `"$PYTHON" $TESTDIR/seq.py 0 50000`
> do > do
> echo "Password: hunter$x" >> target > echo "Password: hunter$x" >> target
> done > done
$ hg ci -m 'add 100k passwords' $ hg ci -m 'add 100k passwords'
$ H2=`hg id --debug -i` $ H2=`hg id --debug -i`
$ C5=$H2 $ C5=$H2
$ hg revert -r "$H2^" target $ hg revert -r "$H2^" target
$ hg ci -m 'cleaned 100k passwords' $ hg ci -m 'cleaned 100k passwords'
$ H2=`hg id --debug -i` $ H2=`hg id --debug -i`
$ hg censor -r $C5 target $ hg censor -r $C5 target
$ hg cat -r $C5 target $ hg cat -r $C5 target | head -n 10
$ hg cat -r $H2 target $ hg cat -r $H2 target | head -n 10
fresh start fresh start
Repo with censored nodes can be cloned and cloned nodes are censored Repo with censored nodes can be cloned and cloned nodes are censored
$ cd .. $ cd ..
$ hg clone r rclone $ hg clone r rclone
updating to branch default updating to branch default
2 files updated, 0 files merged, 0 files removed, 0 files unresolved 2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd rclone $ cd rclone
$ hg cat -r $H1 target $ hg cat -r $H1 target | head -n 10
advanced head H1 advanced head H1
$ hg cat -r $H2~5 target $ hg cat -r $H2~5 target | head -n 10
Tainted file now super sanitized Tainted file now super sanitized
$ hg cat -r $C2 target $ hg cat -r $C2 target | head -n 10
$ hg cat -r $C1 target $ hg cat -r $C1 target | head -n 10
$ hg cat -r 0 target $ hg cat -r 0 target | head -n 10
Initially untainted file Initially untainted file
$ hg verify $ hg verify
checking changesets checking changesets
checking manifests checking manifests
crosschecking files in changesets and manifests crosschecking files in changesets and manifests
checking files checking files
checked 12 changesets with 13 changes to 2 files checked 12 changesets with 13 changes to 2 files
Repo cloned before tainted content introduced can pull censored nodes Repo cloned before tainted content introduced can pull censored nodes
$ cd ../rpull $ cd ../rpull
$ hg cat -r tip target $ hg cat -r tip target | head -n 10
Initially untainted file Initially untainted file
$ hg verify $ hg verify
checking changesets checking changesets
checking manifests checking manifests
crosschecking files in changesets and manifests crosschecking files in changesets and manifests
checking files checking files
checked 1 changesets with 2 changes to 2 files checked 1 changesets with 2 changes to 2 files
$ hg pull -r $H1 -r $H2 $ hg pull -r $H1 -r $H2
pulling from $TESTTMP/r pulling from $TESTTMP/r
searching for changes searching for changes
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 11 changesets with 11 changes to 2 files (+1 heads) added 11 changesets with 11 changes to 2 files (+1 heads)
new changesets 186fb27560c3:683e4645fded new changesets 186fb27560c3:683e4645fded
(run 'hg heads' to see heads, 'hg merge' to merge) (run 'hg heads' to see heads, 'hg merge' to merge)
$ hg update 4 $ hg update 4
2 files updated, 0 files merged, 0 files removed, 0 files unresolved 2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat target $ cat target | head -n 10
Tainted file now super sanitized Tainted file now super sanitized
$ hg cat -r $H1 target $ hg cat -r $H1 target | head -n 10
advanced head H1 advanced head H1
$ hg cat -r $H2~5 target $ hg cat -r $H2~5 target | head -n 10
Tainted file now super sanitized Tainted file now super sanitized
$ hg cat -r $C2 target $ hg cat -r $C2 target | head -n 10
$ hg cat -r $C1 target $ hg cat -r $C1 target | head -n 10
$ hg cat -r 0 target $ hg cat -r 0 target | head -n 10
Initially untainted file Initially untainted file
$ hg verify $ hg verify
checking changesets checking changesets
checking manifests checking manifests
crosschecking files in changesets and manifests crosschecking files in changesets and manifests
checking files checking files
checked 12 changesets with 13 changes to 2 files checked 12 changesets with 13 changes to 2 files
Censored nodes can be pushed if they censor previously unexchanged nodes Censored nodes can be pushed if they censor previously unexchanged nodes
$ echo 'Passwords: hunter2hunter2' > target $ echo 'Passwords: hunter2hunter2' > target
$ hg ci -m 're-add password from clone' target $ hg ci -m 're-add password from clone' target
created new head created new head
$ H3=`hg id --debug -i` $ H3=`hg id --debug -i`
$ REV=$H3 $ REV=$H3
$ echo 'Re-sanitized; nothing to see here' > target $ echo 'Re-sanitized; nothing to see here' > target
$ hg ci -m 're-sanitized' target $ hg ci -m 're-sanitized' target
$ H2=`hg id --debug -i` $ H2=`hg id --debug -i`
$ CLEANREV=$H2 $ CLEANREV=$H2
$ hg cat -r $REV target $ hg cat -r $REV target | head -n 10
Passwords: hunter2hunter2 Passwords: hunter2hunter2
$ hg censor -r $REV target $ hg censor -r $REV target
$ hg cat -r $REV target $ hg cat -r $REV target | head -n 10
$ hg cat -r $CLEANREV target $ hg cat -r $CLEANREV target | head -n 10
Re-sanitized; nothing to see here Re-sanitized; nothing to see here
$ hg push -f -r $H2 $ hg push -f -r $H2
pushing to $TESTTMP/r pushing to $TESTTMP/r
searching for changes searching for changes
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 2 changesets with 2 changes to 1 files (+1 heads) added 2 changesets with 2 changes to 1 files (+1 heads)
$ cd ../r $ cd ../r
$ hg cat -r $REV target $ hg cat -r $REV target | head -n 10
$ hg cat -r $CLEANREV target $ hg cat -r $CLEANREV target | head -n 10
Re-sanitized; nothing to see here Re-sanitized; nothing to see here
$ hg update $CLEANREV $ hg update $CLEANREV
2 files updated, 0 files merged, 0 files removed, 0 files unresolved 2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat target $ cat target | head -n 10
Re-sanitized; nothing to see here Re-sanitized; nothing to see here
Censored nodes can be bundled up and unbundled in another repo Censored nodes can be bundled up and unbundled in another repo
$ hg bundle --base 0 ../pwbundle $ hg bundle --base 0 ../pwbundle
13 changesets found 13 changesets found
$ cd ../rclone $ cd ../rclone
$ hg unbundle ../pwbundle $ hg unbundle ../pwbundle
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 2 changesets with 2 changes to 2 files (+1 heads) added 2 changesets with 2 changes to 2 files (+1 heads)
new changesets 075be80ac777:dcbaf17bf3a1 (2 drafts) new changesets 075be80ac777:dcbaf17bf3a1 (2 drafts)
(run 'hg heads .' to see heads, 'hg merge' to merge) (run 'hg heads .' to see heads, 'hg merge' to merge)
$ hg cat -r $REV target $ hg cat -r $REV target | head -n 10
$ hg cat -r $CLEANREV target $ hg cat -r $CLEANREV target | head -n 10
Re-sanitized; nothing to see here Re-sanitized; nothing to see here
$ hg update $CLEANREV $ hg update $CLEANREV
2 files updated, 0 files merged, 0 files removed, 0 files unresolved 2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat target $ cat target | head -n 10
Re-sanitized; nothing to see here Re-sanitized; nothing to see here
$ hg verify $ hg verify
checking changesets checking changesets
checking manifests checking manifests
crosschecking files in changesets and manifests crosschecking files in changesets and manifests
checking files checking files
checked 14 changesets with 15 changes to 2 files checked 14 changesets with 15 changes to 2 files
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 6 changesets with 5 changes to 2 files (+1 heads) added 6 changesets with 5 changes to 2 files (+1 heads)
new changesets efbe78065929:683e4645fded (6 drafts) new changesets efbe78065929:683e4645fded (6 drafts)
(run 'hg heads .' to see heads, 'hg merge' to merge) (run 'hg heads .' to see heads, 'hg merge' to merge)
$ hg update $H2 $ hg update $H2
2 files updated, 0 files merged, 0 files removed, 0 files unresolved 2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat target $ cat target | head -n 10
Re-sanitized; nothing to see here Re-sanitized; nothing to see here
$ hg verify $ hg verify
checking changesets checking changesets
checking manifests checking manifests
crosschecking files in changesets and manifests crosschecking files in changesets and manifests
checking files checking files
checked 14 changesets with 15 changes to 2 files checked 14 changesets with 15 changes to 2 files
$ cd ../r $ cd ../r
Can import bundle where first revision of a file is censored Can import bundle where first revision of a file is censored
$ hg init ../rinit $ hg init ../rinit
$ hg censor -r 0 target $ hg censor -r 0 target
$ hg bundle -r 0 --base null ../rinit/initbundle $ hg bundle -r 0 --base null ../rinit/initbundle
1 changesets found 1 changesets found
$ cd ../rinit $ cd ../rinit
$ hg unbundle initbundle $ hg unbundle initbundle
adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 1 changesets with 2 changes to 2 files added 1 changesets with 2 changes to 2 files
new changesets e97f55b2665a (1 drafts) new changesets e97f55b2665a (1 drafts)
(run 'hg update' to get a working copy) (run 'hg update' to get a working copy)
$ hg cat -r 0 target $ hg cat -r 0 target | head -n 10