( )⚙ D11635 dirstate-v2: Replace the 32-bit `mode` field with two bits

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11635

dirstate-v2: Replace the 32-bit `mode` field with two bits
ClosedPublic

Authored by SimonSapin on Oct 12 2021, 1:00 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG4d5a13253d34: dirstate-v2: Replace the 32-bit `mode` field with two bits
Summary

Previously we stored the entire value from stat_result.st_mode,
like dirstate-v1 does. However only the executable permission
and type of file (only symbolic links and normal files are supported)
are relevant to Mecurial.

So replace this field with two bits in the existing bitfield byte.
For now the unused space is left as padding, as it will be used
for something else soon.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

SimonSapin created this revision.Oct 12 2021, 1:00 PM
SimonSapin added a commit: rHG4d5a13253d34: dirstate-v2: Replace the 32-bit `mode` field with two bits.Oct 12 2021, 2:03 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG4d5a13253d34: dirstate-v2: Replace the 32-bit `mode` field with two bits (authored by SimonSapin). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 30723

mercurial/helptext/internals/dirstate-v2.txt

Some boolean values packed as bits of a single byte. Some boolean values packed as bits of a single byte.
Starting from least-significant, bit masks are:: Starting from least-significant, bit masks are::
WDIR_TRACKED = 1 << 0 WDIR_TRACKED = 1 << 0
P1_TRACKED = 1 << 1 P1_TRACKED = 1 << 1
P2_INFO = 1 << 2 P2_INFO = 1 << 2
HAS_MODE_AND_SIZE = 1 << 3 HAS_MODE_AND_SIZE = 1 << 3
HAS_MTIME = 1 << 4 HAS_MTIME = 1 << 4
MODE_EXEC_PERM = 1 << 5
MODE_IS_SYMLINK = 1 << 7
Other bits are unset. The meaning of these bits are: Other bits are unset. The meaning of these bits are:
`WDIR_TRACKED` `WDIR_TRACKED`
Set if the working directory contains a tracked file at this node’s path. Set if the working directory contains a tracked file at this node’s path.
This is typically set and unset by `hg add` and `hg rm`. This is typically set and unset by `hg add` and `hg rm`.
`P1_TRACKED` `P1_TRACKED`
Additional untracked nodes may also exist. Additional untracked nodes may also exist.
Although implementations should strive to clean up nodes Although implementations should strive to clean up nodes
that are entirely unused, other untracked nodes may also exist. that are entirely unused, other untracked nodes may also exist.
For example, a future version of Mercurial might in some cases For example, a future version of Mercurial might in some cases
add nodes for untracked files or/and ignored files in the working directory add nodes for untracked files or/and ignored files in the working directory
in order to optimize `hg status` in order to optimize `hg status`
by enabling it to skip `readdir` in more cases. by enabling it to skip `readdir` in more cases.
When a node is for a file tracked anywhere, When a node is for a file tracked anywhere:
the rest of the node data is three fields: - If `HAS_MODE_AND_SIZE` is set, the file is expected
to be a symbolic link or a normal file based on `MODE_IS_SYMLINK`.
- If `HAS_MODE_AND_SIZE` is set, the file’s owner is expected
to have execute permission or not based on `MODE_EXEC_PERM`.
- If `HAS_MODE_AND_SIZE` is unset,
the expected type of file and permission are unknown.
The rest of the node data is three fields:
* Offset 31: * Offset 31:
If `HAS_MODE_AND_SIZE` is unset, four zero bytes. 4 unused bytes, set to zero
Otherwise, a 32-bit integer for the Unix mode (as in `stat_result.st_mode`)
expected for this file to be considered clean.
Only the `S_IXUSR` bit (owner has execute permission) is considered.
* Offset 35: * Offset 35:
If `HAS_MODE_AND_SIZE` is unset, four zero bytes. If `HAS_MODE_AND_SIZE` is unset, four zero bytes.
Otherwise, a 32-bit integer for expected size of the file Otherwise, a 32-bit integer for expected size of the file
truncated to its 31 least-significant bits. truncated to its 31 least-significant bits.
Unlike in dirstate-v1, negative values are not used. Unlike in dirstate-v1, negative values are not used.
* Offset 39: * Offset 39:

rust/Cargo.lock

# This file is automatically @generated by Cargo. # This file is automatically @generated by Cargo.
# It is not intended for manual editing. # It is not intended for manual editing.
version = 3
[[package]] [[package]]
name = "adler" name = "adler"
version = "0.2.3" version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ee2a4ec343196209d6594e19543ae87a39f96d5534d7174822a3ad825dd6ed7e" checksum = "ee2a4ec343196209d6594e19543ae87a39f96d5534d7174822a3ad825dd6ed7e"
[[package]] [[package]]
name = "aho-corasick" name = "aho-corasick"
"crossbeam-channel 0.4.4", "crossbeam-channel 0.4.4",
"derive_more", "derive_more",
"flate2", "flate2",
"format-bytes", "format-bytes",
"home", "home",
"im-rc", "im-rc",
"itertools", "itertools",
"lazy_static", "lazy_static",
"libc",
"log", "log",
"memmap2", "memmap2",
"micro-timer", "micro-timer",
"pretty_assertions", "pretty_assertions",
"rand", "rand",
"rand_distr", "rand_distr",
"rand_pcg", "rand_pcg",
"rayon", "rayon",

rust/hg-core/Cargo.toml

bitflags = "1.2" bitflags = "1.2"
bytes-cast = "0.2" bytes-cast = "0.2"
byteorder = "1.3.4" byteorder = "1.3.4"
derive_more = "0.99" derive_more = "0.99"
home = "0.5" home = "0.5"
im-rc = "15.0.*" im-rc = "15.0.*"
itertools = "0.9" itertools = "0.9"
lazy_static = "1.4.0" lazy_static = "1.4.0"
libc = "0.2"
rand = "0.7.3" rand = "0.7.3"
rand_pcg = "0.2.1" rand_pcg = "0.2.1"
rand_distr = "0.2.2" rand_distr = "0.2.2"
rayon = "1.3.0" rayon = "1.3.0"
regex = "1.3.9" regex = "1.3.9"
sha-1 = "0.9.6" sha-1 = "0.9.6"
twox-hash = "1.5.0" twox-hash = "1.5.0"
same-file = "1.0.6" same-file = "1.0.6"

rust/hg-core/src/dirstate_tree/on_disk.rs

#[derive(BytesCast)] #[derive(BytesCast)]
#[repr(C)] #[repr(C)]
struct Flags: u8 { struct Flags: u8 {
const WDIR_TRACKED = 1 << 0; const WDIR_TRACKED = 1 << 0;
const P1_TRACKED = 1 << 1; const P1_TRACKED = 1 << 1;
const P2_INFO = 1 << 2; const P2_INFO = 1 << 2;
const HAS_MODE_AND_SIZE = 1 << 3; const HAS_MODE_AND_SIZE = 1 << 3;
const HAS_MTIME = 1 << 4; const HAS_MTIME = 1 << 4;
const MODE_EXEC_PERM = 1 << 5;
const MODE_IS_SYMLINK = 1 << 7;
} }
} }
#[derive(BytesCast, Copy, Clone, Debug)] #[derive(BytesCast, Copy, Clone, Debug)]
#[repr(C)] #[repr(C)]
struct Entry { struct Entry {
mode: U32Be, _padding: U32Be,
size: U32Be, size: U32Be,
mtime: U32Be, mtime: U32Be,
} }
/// Duration since the Unix epoch /// Duration since the Unix epoch
#[derive(BytesCast, Copy, Clone)] #[derive(BytesCast, Copy, Clone)]
#[repr(C)] #[repr(C)]
struct PackedTimestamp { struct PackedTimestamp {
if self.flags.contains(Flags::HAS_MTIME) && !self.has_entry() { if self.flags.contains(Flags::HAS_MTIME) && !self.has_entry() {
Some(self.data.as_timestamp()?) Some(self.data.as_timestamp()?)
} else { } else {
None None
}, },
) )
} }
fn synthesize_unix_mode(&self) -> u32 {
let file_type = if self.flags.contains(Flags::MODE_IS_SYMLINK) {
libc::S_IFLNK
} else {
libc::S_IFREG
};
let permisions = if self.flags.contains(Flags::MODE_EXEC_PERM) {
0o755
} else {
0o644
};
file_type | permisions
}
fn assume_entry(&self) -> DirstateEntry { fn assume_entry(&self) -> DirstateEntry {
// TODO: convert through raw bits instead? // TODO: convert through raw bits instead?
let wdir_tracked = self.flags.contains(Flags::WDIR_TRACKED); let wdir_tracked = self.flags.contains(Flags::WDIR_TRACKED);
let p1_tracked = self.flags.contains(Flags::P1_TRACKED); let p1_tracked = self.flags.contains(Flags::P1_TRACKED);
let p2_info = self.flags.contains(Flags::P2_INFO); let p2_info = self.flags.contains(Flags::P2_INFO);
let mode_size = if self.flags.contains(Flags::HAS_MODE_AND_SIZE) { let mode_size = if self.flags.contains(Flags::HAS_MODE_AND_SIZE) {
Some((self.data.mode.into(), self.data.size.into())) Some((self.synthesize_unix_mode(), self.data.size.into()))
} else { } else {
None None
}; };
let mtime = if self.flags.contains(Flags::HAS_MTIME) { let mtime = if self.flags.contains(Flags::HAS_MTIME) {
Some(self.data.mtime.into()) Some(self.data.mtime.into())
} else { } else {
None None
}; };
fn from_dirstate_entry(entry: &DirstateEntry) -> (Flags, Self) { fn from_dirstate_entry(entry: &DirstateEntry) -> (Flags, Self) {
let (wdir_tracked, p1_tracked, p2_info, mode_size_opt, mtime_opt) = let (wdir_tracked, p1_tracked, p2_info, mode_size_opt, mtime_opt) =
entry.v2_data(); entry.v2_data();
// TODO: convert throug raw flag bits instead? // TODO: convert throug raw flag bits instead?
let mut flags = Flags::empty(); let mut flags = Flags::empty();
flags.set(Flags::WDIR_TRACKED, wdir_tracked); flags.set(Flags::WDIR_TRACKED, wdir_tracked);
flags.set(Flags::P1_TRACKED, p1_tracked); flags.set(Flags::P1_TRACKED, p1_tracked);
flags.set(Flags::P2_INFO, p2_info); flags.set(Flags::P2_INFO, p2_info);
let (mode, size, mtime); let (size, mtime);
if let Some((m, s)) = mode_size_opt { if let Some((m, s)) = mode_size_opt {
mode = m; let exec_perm = m & libc::S_IXUSR != 0;
let is_symlink = m & libc::S_IFMT == libc::S_IFLNK;
flags.set(Flags::MODE_EXEC_PERM, exec_perm);
flags.set(Flags::MODE_IS_SYMLINK, is_symlink);
size = s; size = s;
flags.insert(Flags::HAS_MODE_AND_SIZE) flags.insert(Flags::HAS_MODE_AND_SIZE)
} else { } else {
mode = 0;
size = 0; size = 0;
} }
if let Some(m) = mtime_opt { if let Some(m) = mtime_opt {
mtime = m; mtime = m;
flags.insert(Flags::HAS_MTIME); flags.insert(Flags::HAS_MTIME);
} else { } else {
mtime = 0; mtime = 0;
} }
let raw_entry = Entry { let raw_entry = Entry {
mode: mode.into(), _padding: 0.into(),
size: size.into(), size: size.into(),
mtime: mtime.into(), mtime: mtime.into(),
}; };
(flags, raw_entry) (flags, raw_entry)
} }
fn from_timestamp(timestamp: TruncatedTimestamp) -> Self { fn from_timestamp(timestamp: TruncatedTimestamp) -> Self {
let packed = PackedTimestamp { let packed = PackedTimestamp {
Entry::from_dirstate_entry(entry) Entry::from_dirstate_entry(entry)
} }
dirstate_map::NodeData::CachedDirectory { mtime } => { dirstate_map::NodeData::CachedDirectory { mtime } => {
(Flags::HAS_MTIME, Entry::from_timestamp(*mtime)) (Flags::HAS_MTIME, Entry::from_timestamp(*mtime))
} }
dirstate_map::NodeData::None => ( dirstate_map::NodeData::None => (
Flags::empty(), Flags::empty(),
Entry { Entry {
mode: 0.into(), _padding: 0.into(),
size: 0.into(), size: 0.into(),
mtime: 0.into(), mtime: 0.into(),
}, },
), ),
}; };
Node { Node {
children, children,
copy_source, copy_source,

rust/hg-cpython/Cargo.toml

# Enable one of these features to build a test executable linked to libpython: # Enable one of these features to build a test executable linked to libpython:
# e.g. cargo test --no-default-features --features python27-bin # e.g. cargo test --no-default-features --features python27-bin
python27-bin = ["cpython/python27-sys"] python27-bin = ["cpython/python27-sys"]
python3-bin = ["cpython/python3-sys"] python3-bin = ["cpython/python3-sys"]
[dependencies] [dependencies]
crossbeam-channel = "0.4" crossbeam-channel = "0.4"
hg-core = { path = "../hg-core"} hg-core = { path = "../hg-core"}
libc = '*' libc = "0.2"
log = "0.4.8" log = "0.4.8"
env_logger = "0.7.1" env_logger = "0.7.1"
stable_deref_trait = "1.2.0" stable_deref_trait = "1.2.0"
[dependencies.cpython] [dependencies.cpython]
version = "0.6.0" version = "0.6.0"
default-features = false default-features = false
mercurial/helptext/internals/dirstate-v2.txt