This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11688

dirstate-v2: preserve the fallback values on disk
ClosedPublic

Authored by marmoute on Oct 18 2021, 9:20 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGb874e8d81a98: dirstate-v2: preserve the fallback values on disk
rHG5f8ee8157dcc: dirstate-v2: preserve the fallback values on disk
Summary

When the fallback values are set, they are now read and written to disk.

See format documentation for details.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Oct 18 2021, 9:20 PM
mharbison72 added a subscriber: mharbison72.Oct 18 2021, 11:52 PM

LGTM

mercurial/helptext/internals/dirstate-v2.txt
576–579

Looking further ahead to the UI, it would feel weird if it was a command that only worked on Windows. Is the plan to simply chmod+x the file on !windows, and then continue on looking to the filesystem like is done now? Is that too complicated with symlinks (where you'd have to delete the existing file/symlink and create a symlink/file when the filesystem supports symlinks)?

Alphare added a subscriber: Alphare.Oct 19 2021, 4:29 AM
Alphare added inline comments.
mercurial/helptext/internals/dirstate-v2.txt
567

Maybe the entry carries "fallback" information?

582

Same change as above

marmoute updated this revision to Diff 30893.Oct 19 2021, 7:49 AM
marmoute updated this revision to Diff 30895.Oct 19 2021, 9:20 AM
marmoute added inline comments.Oct 19 2021, 9:21 AM
mercurial/helptext/internals/dirstate-v2.txt
576–579

The plan will always be to look at things in this order:

  1. the file system
  2. the fallback value
  3. the parents manifests

(this series introduce a (2) here).

The first user of this logic will be the merge code so that they can properly apply mode changes on system where the mode is unsupported.

It will also make sense to add some command to manually set this. In such case, this would also update -on-disk- status on the file for consistency.

I don't think either of the above will be present in 6.0 for time constraint reason, but it is important to have the semantic added early.

Does this address your question ?

marmoute added a commit: rHG5f8ee8157dcc: dirstate-v2: preserve the fallback values on disk.Oct 19 2021, 1:01 PM
marmoute added a commit: rHGb874e8d81a98: dirstate-v2: preserve the fallback values on disk.
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG5f8ee8157dcc: dirstate-v2: preserve the fallback values on disk (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 30903

mercurial/helptext/internals/dirstate-v2.txt

HAS_MODE_AND_SIZE = 1 << 3 HAS_MODE_AND_SIZE = 1 << 3
HAS_FILE_MTIME = 1 << 4 HAS_FILE_MTIME = 1 << 4
HAS_DIRECTORY_MTIME = 1 << 5 HAS_DIRECTORY_MTIME = 1 << 5
MODE_EXEC_PERM = 1 << 6 MODE_EXEC_PERM = 1 << 6
MODE_IS_SYMLINK = 1 << 7 MODE_IS_SYMLINK = 1 << 7
EXPECTED_STATE_IS_MODIFIED = 1 << 8 EXPECTED_STATE_IS_MODIFIED = 1 << 8
ALL_UNKNOWN_RECORDED = 1 << 9 ALL_UNKNOWN_RECORDED = 1 << 9
ALL_IGNORED_RECORDED = 1 << 10 ALL_IGNORED_RECORDED = 1 << 10
HAS_FALLBACK_EXEC = 1 << 11
FALLBACK_EXEC = 1 << 12
HAS_FALLBACK_SYMLINK = 1 << 13
FALLBACK_SYMLINK = 1 << 14
The meaning of each bit is described below. The meaning of each bit is described below.
Other bits are unset. Other bits are unset.
They may be assigned meaning if the future, They may be assigned meaning if the future,
with the limitation that Mercurial versions that pre-date such meaning with the limitation that Mercurial versions that pre-date such meaning
will always reset those bits to unset when writing nodes. will always reset those bits to unset when writing nodes.
(A new node is written for any mutation in its subtree, (A new node is written for any mutation in its subtree,
are listed. are listed.
Note that the amount recorded "ignored" children can still be zero if None Note that the amount recorded "ignored" children can still be zero if None
where present. where present.
Also note that having this flag unset does not imply that no "ignored" Also note that having this flag unset does not imply that no "ignored"
children have been recorded. Some might be present, but there is no garantee children have been recorded. Some might be present, but there is no garantee
that is will be all of them. that is will be all of them.
`HAS_FALLBACK_EXEC`
If this flag is set, the entry carries as "fallback" information for the
AlphareUnsubmitted
Not Done

Maybe the entry carries "fallback" information?

Alphare: Maybe `the entry carries "fallback" information`?
executable bit in the `FALLBACK_EXEC` flag.
Fallback information can be stored in the dirstate to keep track of
filesystem attribute tracked by Mercurial when the underlying file
system or operating system does not support that property, (e.g.
Windows).
`FALLBACK_EXEC`
Should be ignored if `HAS_FALLBACK_EXEC` is unset. If set the file for this
entry should be considered executable if that information cannot be
extracted from the file system. If unset it should be considered
non-executable instead.
mharbison72Unsubmitted
Not Done

Looking further ahead to the UI, it would feel weird if it was a command that only worked on Windows. Is the plan to simply chmod+x the file on !windows, and then continue on looking to the filesystem like is done now? Is that too complicated with symlinks (where you'd have to delete the existing file/symlink and create a symlink/file when the filesystem supports symlinks)?

mharbison72: Looking further ahead to the UI, it would feel weird if it was a command that only worked on…
marmouteAuthorUnsubmitted
Done

The plan will always be to look at things in this order:

  1. the file system
  2. the fallback value
  3. the parents manifests

(this series introduce a (2) here).

The first user of this logic will be the merge code so that they can properly apply mode changes on system where the mode is unsupported.

It will also make sense to add some command to manually set this. In such case, this would also update -on-disk- status on the file for consistency.

I don't think either of the above will be present in 6.0 for time constraint reason, but it is important to have the semantic added early.

Does this address your question ?

marmoute: The plan will always be to look at things in this order: 1) the file system 2) the fallback…
`HAS_FALLBACK_SYMLINK`
If this flag is set, the entry carries as "fallback" information for symbolic
AlphareUnsubmitted
Not Done

Same change as above

Alphare: Same change as above
link status in the `FALLBACK_SYMLINK` flag.
Fallback information can be stored in the dirstate to keep track of
filesystem attribute tracked by Mercurial when the underlying file
system or operating system does not support that property, (e.g.
Windows).
`FALLBACK_SYMLINK`
Should be ignored if `HAS_FALLBACK_SYMLINK` is unset. If set the file for
this entry should be considered a symlink if that information cannot be
extracted from the file system. If unset it should be considered a normal
file instead.

mercurial/pure/parsers.py

DIRSTATE_V2_HAS_MODE_AND_SIZE = 1 << 3 DIRSTATE_V2_HAS_MODE_AND_SIZE = 1 << 3
DIRSTATE_V2_HAS_FILE_MTIME = 1 << 4 DIRSTATE_V2_HAS_FILE_MTIME = 1 << 4
_DIRSTATE_V2_HAS_DIRCTORY_MTIME = 1 << 5 # Unused when Rust is not available _DIRSTATE_V2_HAS_DIRCTORY_MTIME = 1 << 5 # Unused when Rust is not available
DIRSTATE_V2_MODE_EXEC_PERM = 1 << 6 DIRSTATE_V2_MODE_EXEC_PERM = 1 << 6
DIRSTATE_V2_MODE_IS_SYMLINK = 1 << 7 DIRSTATE_V2_MODE_IS_SYMLINK = 1 << 7
DIRSTATE_V2_EXPECTED_STATE_IS_MODIFIED = 1 << 8 DIRSTATE_V2_EXPECTED_STATE_IS_MODIFIED = 1 << 8
DIRSTATE_V2_ALL_UNKNOWN_RECORDED = 1 << 9 DIRSTATE_V2_ALL_UNKNOWN_RECORDED = 1 << 9
DIRSTATE_V2_ALL_IGNORED_RECORDED = 1 << 10 DIRSTATE_V2_ALL_IGNORED_RECORDED = 1 << 10
DIRSTATE_V2_HAS_FALLBACK_EXEC = 1 << 11
DIRSTATE_V2_FALLBACK_EXEC = 1 << 12
DIRSTATE_V2_HAS_FALLBACK_SYMLINK = 1 << 13
DIRSTATE_V2_FALLBACK_SYMLINK = 1 << 14
@attr.s(slots=True, init=False) @attr.s(slots=True, init=False)
class DirstateItem(object): class DirstateItem(object):
"""represent a dirstate entry """represent a dirstate entry
It hold multiple attributes It hold multiple attributes
mode = None mode = None
if flags & +DIRSTATE_V2_EXPECTED_STATE_IS_MODIFIED: if flags & +DIRSTATE_V2_EXPECTED_STATE_IS_MODIFIED:
# we do not have support for this flag in the code yet, # we do not have support for this flag in the code yet,
# force a lookup for this file. # force a lookup for this file.
has_mode_size = False has_mode_size = False
has_meaningful_mtime = False has_meaningful_mtime = False
fallback_exec = None
if flags & DIRSTATE_V2_HAS_FALLBACK_EXEC:
fallback_exec = flags & DIRSTATE_V2_FALLBACK_EXEC
fallback_symlink = None
if flags & DIRSTATE_V2_HAS_FALLBACK_SYMLINK:
fallback_symlink = flags & DIRSTATE_V2_FALLBACK_SYMLINK
if has_mode_size: if has_mode_size:
assert stat.S_IXUSR == 0o100 assert stat.S_IXUSR == 0o100
if flags & DIRSTATE_V2_MODE_EXEC_PERM: if flags & DIRSTATE_V2_MODE_EXEC_PERM:
mode = 0o755 mode = 0o755
else: else:
mode = 0o644 mode = 0o644
if flags & DIRSTATE_V2_MODE_IS_SYMLINK: if flags & DIRSTATE_V2_MODE_IS_SYMLINK:
mode |= stat.S_IFLNK mode |= stat.S_IFLNK
else: else:
mode |= stat.S_IFREG mode |= stat.S_IFREG
return cls( return cls(
wc_tracked=bool(flags & DIRSTATE_V2_WDIR_TRACKED), wc_tracked=bool(flags & DIRSTATE_V2_WDIR_TRACKED),
p1_tracked=bool(flags & DIRSTATE_V2_P1_TRACKED), p1_tracked=bool(flags & DIRSTATE_V2_P1_TRACKED),
p2_info=bool(flags & DIRSTATE_V2_P2_INFO), p2_info=bool(flags & DIRSTATE_V2_P2_INFO),
has_meaningful_data=has_mode_size, has_meaningful_data=has_mode_size,
has_meaningful_mtime=has_meaningful_mtime, has_meaningful_mtime=has_meaningful_mtime,
parentfiledata=(mode, size, mtime), parentfiledata=(mode, size, mtime),
fallback_exec=fallback_exec,
fallback_symlink=fallback_symlink,
) )
@classmethod @classmethod
def from_v1_data(cls, state, mode, size, mtime): def from_v1_data(cls, state, mode, size, mtime):
"""Build a new DirstateItem object from V1 data """Build a new DirstateItem object from V1 data
Since the dirstate-v1 format is frozen, the signature of this function Since the dirstate-v1 format is frozen, the signature of this function
is not expected to change, unlike the __init__ one. is not expected to change, unlike the __init__ one.
if self._mode is not None and self._size is not None: if self._mode is not None and self._size is not None:
flags |= DIRSTATE_V2_HAS_MODE_AND_SIZE flags |= DIRSTATE_V2_HAS_MODE_AND_SIZE
if self.mode & stat.S_IXUSR: if self.mode & stat.S_IXUSR:
flags |= DIRSTATE_V2_MODE_EXEC_PERM flags |= DIRSTATE_V2_MODE_EXEC_PERM
if stat.S_ISLNK(self.mode): if stat.S_ISLNK(self.mode):
flags |= DIRSTATE_V2_MODE_IS_SYMLINK flags |= DIRSTATE_V2_MODE_IS_SYMLINK
if self._mtime is not None: if self._mtime is not None:
flags |= DIRSTATE_V2_HAS_FILE_MTIME flags |= DIRSTATE_V2_HAS_FILE_MTIME
if self._fallback_exec is not None:
flags |= DIRSTATE_V2_HAS_FALLBACK_EXEC
if self._fallback_exec:
flags |= DIRSTATE_V2_FALLBACK_EXEC
if self._fallback_symlink is not None:
flags |= DIRSTATE_V2_HAS_FALLBACK_SYMLINK
if self._fallback_symlink:
flags |= DIRSTATE_V2_FALLBACK_SYMLINK
# Note: we do not need to do anything regarding # Note: we do not need to do anything regarding
# DIRSTATE_V2_ALL_UNKNOWN_RECORDED and DIRSTATE_V2_ALL_IGNORED_RECORDED # DIRSTATE_V2_ALL_UNKNOWN_RECORDED and DIRSTATE_V2_ALL_IGNORED_RECORDED
# since we never set _DIRSTATE_V2_HAS_DIRCTORY_MTIME # since we never set _DIRSTATE_V2_HAS_DIRCTORY_MTIME
return (flags, self._size or 0, self._mtime or 0) return (flags, self._size or 0, self._mtime or 0)
def v1_state(self): def v1_state(self):
"""return a "state" suitable for v1 serialization""" """return a "state" suitable for v1 serialization"""
if not self.any_tracked: if not self.any_tracked:

rust/hg-core/src/dirstate/entry.rs

self.flags.intersects( self.flags.intersects(
Flags::WDIR_TRACKED | Flags::P1_TRACKED | Flags::P2_INFO, Flags::WDIR_TRACKED | Flags::P1_TRACKED | Flags::P2_INFO,
) )
} }
/// Returns `(wdir_tracked, p1_tracked, p2_info, mode_size, mtime)` /// Returns `(wdir_tracked, p1_tracked, p2_info, mode_size, mtime)`
pub(crate) fn v2_data( pub(crate) fn v2_data(
&self, &self,
) -> (bool, bool, bool, Option<(u32, u32)>, Option<u32>) { ) -> (
bool,
bool,
bool,
Option<(u32, u32)>,
Option<u32>,
Option<bool>,
Option<bool>,
) {
if !self.any_tracked() { if !self.any_tracked() {
// TODO: return an Option instead? // TODO: return an Option instead?
panic!("Accessing v1_state of an untracked DirstateEntry") panic!("Accessing v1_state of an untracked DirstateEntry")
} }
let wdir_tracked = self.flags.contains(Flags::WDIR_TRACKED); let wdir_tracked = self.flags.contains(Flags::WDIR_TRACKED);
let p1_tracked = self.flags.contains(Flags::P1_TRACKED); let p1_tracked = self.flags.contains(Flags::P1_TRACKED);
let p2_info = self.flags.contains(Flags::P2_INFO); let p2_info = self.flags.contains(Flags::P2_INFO);
let mode_size = self.mode_size; let mode_size = self.mode_size;
let mtime = self.mtime; let mtime = self.mtime;
(wdir_tracked, p1_tracked, p2_info, mode_size, mtime) (
wdir_tracked,
p1_tracked,
p2_info,
mode_size,
mtime,
self.get_fallback_exec(),
self.get_fallback_symlink(),
)
} }
fn v1_state(&self) -> EntryState { fn v1_state(&self) -> EntryState {
if !self.any_tracked() { if !self.any_tracked() {
// TODO: return an Option instead? // TODO: return an Option instead?
panic!("Accessing v1_state of an untracked DirstateEntry") panic!("Accessing v1_state of an untracked DirstateEntry")
} }
if self.removed() { if self.removed() {

rust/hg-core/src/dirstate_tree/on_disk.rs

const HAS_MODE_AND_SIZE = 1 << 3; const HAS_MODE_AND_SIZE = 1 << 3;
const HAS_FILE_MTIME = 1 << 4; const HAS_FILE_MTIME = 1 << 4;
const HAS_DIRECTORY_MTIME = 1 << 5; const HAS_DIRECTORY_MTIME = 1 << 5;
const MODE_EXEC_PERM = 1 << 6; const MODE_EXEC_PERM = 1 << 6;
const MODE_IS_SYMLINK = 1 << 7; const MODE_IS_SYMLINK = 1 << 7;
const EXPECTED_STATE_IS_MODIFIED = 1 << 8; const EXPECTED_STATE_IS_MODIFIED = 1 << 8;
const ALL_UNKNOWN_RECORDED = 1 << 9; const ALL_UNKNOWN_RECORDED = 1 << 9;
const ALL_IGNORED_RECORDED = 1 << 10; const ALL_IGNORED_RECORDED = 1 << 10;
const HAS_FALLBACK_EXEC = 1 << 11;
const FALLBACK_EXEC = 1 << 12;
const HAS_FALLBACK_SYMLINK = 1 << 13;
const FALLBACK_SYMLINK = 1 << 14;
} }
} }
/// Duration since the Unix epoch /// Duration since the Unix epoch
#[derive(BytesCast, Copy, Clone)] #[derive(BytesCast, Copy, Clone)]
#[repr(C)] #[repr(C)]
struct PackedTruncatedTimestamp { struct PackedTruncatedTimestamp {
truncated_seconds: U32Be, truncated_seconds: U32Be,
.get(), .get(),
tracked_descendants_count: self.tracked_descendants_count.get(), tracked_descendants_count: self.tracked_descendants_count.get(),
}) })
} }
fn from_dirstate_entry( fn from_dirstate_entry(
entry: &DirstateEntry, entry: &DirstateEntry,
) -> (Flags, U32Be, PackedTruncatedTimestamp) { ) -> (Flags, U32Be, PackedTruncatedTimestamp) {
let (wdir_tracked, p1_tracked, p2_info, mode_size_opt, mtime_opt) = let (
entry.v2_data(); wdir_tracked,
p1_tracked,
p2_info,
mode_size_opt,
mtime_opt,
fallback_exec,
fallback_symlink,
) = entry.v2_data();
// TODO: convert throug raw flag bits instead? // TODO: convert throug raw flag bits instead?
let mut flags = Flags::empty(); let mut flags = Flags::empty();
flags.set(Flags::WDIR_TRACKED, wdir_tracked); flags.set(Flags::WDIR_TRACKED, wdir_tracked);
flags.set(Flags::P1_TRACKED, p1_tracked); flags.set(Flags::P1_TRACKED, p1_tracked);
flags.set(Flags::P2_INFO, p2_info); flags.set(Flags::P2_INFO, p2_info);
let size = if let Some((m, s)) = mode_size_opt { let size = if let Some((m, s)) = mode_size_opt {
let exec_perm = m & libc::S_IXUSR != 0; let exec_perm = m & libc::S_IXUSR != 0;
let is_symlink = m & libc::S_IFMT == libc::S_IFLNK; let is_symlink = m & libc::S_IFMT == libc::S_IFLNK;
flags.set(Flags::MODE_EXEC_PERM, exec_perm); flags.set(Flags::MODE_EXEC_PERM, exec_perm);
flags.set(Flags::MODE_IS_SYMLINK, is_symlink); flags.set(Flags::MODE_IS_SYMLINK, is_symlink);
flags.insert(Flags::HAS_MODE_AND_SIZE); flags.insert(Flags::HAS_MODE_AND_SIZE);
s.into() s.into()
} else { } else {
0.into() 0.into()
}; };
let mtime = if let Some(m) = mtime_opt { let mtime = if let Some(m) = mtime_opt {
flags.insert(Flags::HAS_FILE_MTIME); flags.insert(Flags::HAS_FILE_MTIME);
PackedTruncatedTimestamp { PackedTruncatedTimestamp {
truncated_seconds: m.into(), truncated_seconds: m.into(),
nanoseconds: 0.into(), nanoseconds: 0.into(),
} }
} else { } else {
PackedTruncatedTimestamp::null() PackedTruncatedTimestamp::null()
}; };
if let Some(f_exec) = fallback_exec {
flags.insert(Flags::HAS_FALLBACK_EXEC);
if f_exec {
flags.insert(Flags::FALLBACK_EXEC);
}
}
if let Some(f_symlink) = fallback_symlink {
flags.insert(Flags::HAS_FALLBACK_SYMLINK);
if f_symlink {
flags.insert(Flags::FALLBACK_SYMLINK);
}
}
(flags, size, mtime) (flags, size, mtime)
} }
} }
fn read_hg_path( fn read_hg_path(
on_disk: &[u8], on_disk: &[u8],
slice: PathSlice, slice: PathSlice,
) -> Result<&HgPath, DirstateV2ParseError> { ) -> Result<&HgPath, DirstateV2ParseError> {