( )⚙ D11965 rhg: desambiguate status without decompressing filelog if possible

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11965

rhg: desambiguate status without decompressing filelog if possible
ClosedPublic

Authored by SimonSapin on Jan 6 2022, 1:56 PM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHGe91aa800ae5b: rhg: desambiguate status without decompressing filelog if possible
rHG2082dd346d56: rhg: desambiguate status without decompressing filelog if possible
rHG340a484b65a7: rhg: desambiguate status without decompressing filelog if possible
Summary

When status is unsure based on stat() and the dirstate if a file is clean
or modified, we need to compare it against the filelog.

This comparison can skip looking at contents if the lengths differ.
This changeset optimize this further to deduce what we can about the length
if the filelog without decompressing it or resolving deltas.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Revision Contents
Changeset List

Diff 31616

mercurial/revlogutils/flagutil.py

REVIDX_ISCENSORED REVIDX_ISCENSORED
REVIDX_ELLIPSIS REVIDX_ELLIPSIS
REVIDX_EXTSTORED REVIDX_EXTSTORED
REVIDX_HASCOPIESINFO, REVIDX_HASCOPIESINFO,
REVIDX_DEFAULT_FLAGS REVIDX_DEFAULT_FLAGS
REVIDX_FLAGS_ORDER REVIDX_FLAGS_ORDER
REVIDX_RAWTEXT_CHANGING_FLAGS REVIDX_RAWTEXT_CHANGING_FLAGS
# Keep this in sync with REVIDX_KNOWN_FLAGS in rust/hg-core/src/revlog/revlog.rs
REVIDX_KNOWN_FLAGS = util.bitsfrom(REVIDX_FLAGS_ORDER) REVIDX_KNOWN_FLAGS = util.bitsfrom(REVIDX_FLAGS_ORDER)
# Store flag processors (cf. 'addflagprocessor()' to register) # Store flag processors (cf. 'addflagprocessor()' to register)
flagprocessors = { flagprocessors = {
REVIDX_ISCENSORED: None, REVIDX_ISCENSORED: None,
REVIDX_HASCOPIESINFO: None, REVIDX_HASCOPIESINFO: None,
} }

rust/hg-core/src/revlog/filelog.rs

let encoded_bytes = let encoded_bytes =
path_encode(&[b"data/", hg_path.as_bytes(), suffix].concat()); path_encode(&[b"data/", hg_path.as_bytes(), suffix].concat());
get_path_from_bytes(&encoded_bytes).into() get_path_from_bytes(&encoded_bytes).into()
} }
pub struct FilelogEntry<'a>(RevlogEntry<'a>); pub struct FilelogEntry<'a>(RevlogEntry<'a>);
impl FilelogEntry<'_> { impl FilelogEntry<'_> {
/// `self.data()` can be expensive, with decompression and delta
/// resolution.
///
/// *Without* paying this cost, based on revlog index information
/// including `RevlogEntry::uncompressed_len`:
///
/// * Returns `true` if the length that `self.data().file_data().len()`
/// would return is definitely **not equal** to `other_len`.
/// * Returns `false` if available information is inconclusive.
pub fn file_data_len_not_equal_to(&self, other_len: u64) -> bool {
// Relevant code that implement this behavior in Python code:
// basefilectx.cmp, filelog.size, storageutil.filerevisioncopied,
// revlog.size, revlog.rawsize
// Let’s call `file_data_len` what would be returned by
// `self.data().file_data().len()`.
if self.0.is_cencored() {
let file_data_len = 0;
return other_len != file_data_len;
}
if self.0.has_length_affecting_flag_processor() {
// We can’t conclude anything about `file_data_len`.
return false;
}
// Revlog revisions (usually) have metadata for the size of
// their data after decompression and delta resolution
// as would be returned by `Revlog::get_rev_data`.
//
// For filelogs this is the file’s contents preceded by an optional
// metadata block.
let uncompressed_len = if let Some(l) = self.0.uncompressed_len() {
l as u64
} else {
// The field was set to -1, the actual uncompressed len is unknown.
// We need to decompress to say more.
return false;
};
// `uncompressed_len = file_data_len + optional_metadata_len`,
// so `file_data_len <= uncompressed_len`.
if uncompressed_len < other_len {
// Transitively, `file_data_len < other_len`.
// So `other_len != file_data_len` definitely.
return true;
}
if uncompressed_len == other_len + 4 {
// It’s possible that `file_data_len == other_len` with an empty
// metadata block (2 start marker bytes + 2 end marker bytes).
// This happens when there wouldn’t otherwise be metadata, but
// the first 2 bytes of file data happen to match a start marker
// and would be ambiguous.
return false;
}
if !self.0.has_p1() {
// There may or may not be copy metadata, so we can’t deduce more
// about `file_data_len` without computing file data.
return false;
}
// Filelog ancestry is not meaningful in the way changelog ancestry is.
// It only provides hints to delta generation.
// p1 and p2 are set to null when making a copy or rename since
// contents are likely unrelatedto what might have previously existed
// at the destination path.
//
// Conversely, since here p1 is non-null, there is no copy metadata.
// Note that this reasoning may be invalidated in the presence of
// merges made by some previous versions of Mercurial that
// swapped p1 and p2. See <https://bz.mercurial-scm.org/show_bug.cgi?id=6528>
// and `tests/test-issue6528.t`.
//
// Since copy metadata is currently the only kind of metadata
// kept in revlog data of filelogs,
// this `FilelogEntry` does not have such metadata:
let file_data_len = uncompressed_len;
return file_data_len != other_len;
}
pub fn data(&self) -> Result<FilelogRevisionData, HgError> { pub fn data(&self) -> Result<FilelogRevisionData, HgError> {
Ok(FilelogRevisionData(self.0.data()?.into_owned())) Ok(FilelogRevisionData(self.0.data()?.into_owned()))
} }
} }
/// The data for one revision in a filelog, uncompressed and delta-resolved. /// The data for one revision in a filelog, uncompressed and delta-resolved.
pub struct FilelogRevisionData(Vec<u8>); pub struct FilelogRevisionData(Vec<u8>);

rust/hg-core/src/revlog/index.rs

offset_override offset_override
} else { } else {
let mut bytes = [0; 8]; let mut bytes = [0; 8];
bytes[2..8].copy_from_slice(&self.bytes[0..=5]); bytes[2..8].copy_from_slice(&self.bytes[0..=5]);
BigEndian::read_u64(&bytes[..]) as usize BigEndian::read_u64(&bytes[..]) as usize
} }
} }
pub fn flags(&self) -> u16 {
BigEndian::read_u16(&self.bytes[6..=7])
}
/// Return the compressed length of the data. /// Return the compressed length of the data.
pub fn compressed_len(&self) -> u32 { pub fn compressed_len(&self) -> u32 {
BigEndian::read_u32(&self.bytes[8..=11]) BigEndian::read_u32(&self.bytes[8..=11])
} }
/// Return the uncompressed length of the data. /// Return the uncompressed length of the data.
pub fn uncompressed_len(&self) -> i32 { pub fn uncompressed_len(&self) -> i32 {
BigEndian::read_i32(&self.bytes[12..=15]) BigEndian::read_i32(&self.bytes[12..=15])

rust/hg-core/src/revlog/revlog.rs

use super::nodemap::{NodeMap, NodeMapError}; use super::nodemap::{NodeMap, NodeMapError};
use super::nodemap_docket::NodeMapDocket; use super::nodemap_docket::NodeMapDocket;
use super::patch; use super::patch;
use crate::errors::HgError; use crate::errors::HgError;
use crate::repo::Repo; use crate::repo::Repo;
use crate::revlog::Revision; use crate::revlog::Revision;
use crate::{Node, NULL_REVISION}; use crate::{Node, NULL_REVISION};
const REVISION_FLAG_CENSORED: u16 = 1 << 15;
const REVISION_FLAG_ELLIPSIS: u16 = 1 << 14;
const REVISION_FLAG_EXTSTORED: u16 = 1 << 13;
const REVISION_FLAG_HASCOPIESINFO: u16 = 1 << 12;
// Keep this in sync with REVIDX_KNOWN_FLAGS in
// mercurial/revlogutils/flagutil.py
const REVIDX_KNOWN_FLAGS: u16 = REVISION_FLAG_CENSORED
| REVISION_FLAG_ELLIPSIS
| REVISION_FLAG_EXTSTORED
| REVISION_FLAG_HASCOPIESINFO;
#[derive(derive_more::From)] #[derive(derive_more::From)]
pub enum RevlogError { pub enum RevlogError {
InvalidRevision, InvalidRevision,
/// Working directory is not supported /// Working directory is not supported
WDirUnsupported, WDirUnsupported,
/// Found more than one entry whose ID match the requested prefix /// Found more than one entry whose ID match the requested prefix
AmbiguousPrefix, AmbiguousPrefix,
#[from] #[from]
== rev == rev
{ {
None None
} else { } else {
Some(index_entry.base_revision_or_base_of_delta_chain()) Some(index_entry.base_revision_or_base_of_delta_chain())
}, },
p1: index_entry.p1(), p1: index_entry.p1(),
p2: index_entry.p2(), p2: index_entry.p2(),
flags: index_entry.flags(),
hash: *index_entry.hash(), hash: *index_entry.hash(),
}; };
Ok(entry) Ok(entry)
} }
/// when resolving internal references within revlog, any errors /// when resolving internal references within revlog, any errors
/// should be reported as corruption, instead of e.g. "invalid revision" /// should be reported as corruption, instead of e.g. "invalid revision"
fn get_entry_internal( fn get_entry_internal(
revlog: &'a Revlog, revlog: &'a Revlog,
rev: Revision, rev: Revision,
bytes: &'a [u8], bytes: &'a [u8],
compressed_len: u32, compressed_len: u32,
uncompressed_len: i32, uncompressed_len: i32,
base_rev_or_base_of_delta_chain: Option<Revision>, base_rev_or_base_of_delta_chain: Option<Revision>,
p1: Revision, p1: Revision,
p2: Revision, p2: Revision,
flags: u16,
hash: Node, hash: Node,
} }
impl<'a> RevlogEntry<'a> { impl<'a> RevlogEntry<'a> {
pub fn revision(&self) -> Revision { pub fn revision(&self) -> Revision {
self.rev self.rev
} }
pub fn uncompressed_len(&self) -> Option<u32> { pub fn uncompressed_len(&self) -> Option<u32> {
u32::try_from(self.uncompressed_len).ok() u32::try_from(self.uncompressed_len).ok()
} }
pub fn has_p1(&self) -> bool {
self.p1 != NULL_REVISION
}
pub fn is_cencored(&self) -> bool {
(self.flags & REVISION_FLAG_CENSORED) != 0
}
pub fn has_length_affecting_flag_processor(&self) -> bool {
// Relevant Python code: revlog.size()
// note: ELLIPSIS is known to not change the content
(self.flags & (REVIDX_KNOWN_FLAGS ^ REVISION_FLAG_ELLIPSIS)) != 0
}
/// The data for this entry, after resolving deltas if any. /// The data for this entry, after resolving deltas if any.
pub fn data(&self) -> Result<Cow<'a, [u8]>, HgError> { pub fn data(&self) -> Result<Cow<'a, [u8]>, HgError> {
let mut entry = self.clone(); let mut entry = self.clone();
let mut delta_chain = vec![]; let mut delta_chain = vec![];
// The meaning of `base_rev_or_base_of_delta_chain` depends on // The meaning of `base_rev_or_base_of_delta_chain` depends on
// generaldelta. See the doc on `ENTRY_DELTA_BASE` in // generaldelta. See the doc on `ENTRY_DELTA_BASE` in
// `mercurial/revlogutils/constants.py` and the code in // `mercurial/revlogutils/constants.py` and the code in

rust/rhg/src/commands/status.rs

return Ok(true); return Ok(true);
} }
let filelog = repo.filelog(hg_path)?; let filelog = repo.filelog(hg_path)?;
let fs_len = fs_metadata.len(); let fs_len = fs_metadata.len();
let filelog_entry = let filelog_entry =
filelog.entry_for_node(entry.node_id()?).map_err(|_| { filelog.entry_for_node(entry.node_id()?).map_err(|_| {
HgError::corrupted("filelog missing node from manifest") HgError::corrupted("filelog missing node from manifest")
})?; })?;
// TODO: check `fs_len` here like below, but based on if filelog_entry.file_data_len_not_equal_to(fs_len) {
// `RevlogEntry::uncompressed_len` without decompressing the full filelog // No need to read file contents:
// contents where possible. This is only valid if the revlog data does not // it cannot be equal if it has a different length.
// contain metadata. See how Python’s `revlog.rawsize` calls return Ok(true);
// `storageutil.filerevisioncopied`. }
// (Maybe also check for content-modifying flags? See `revlog.size`.)
let filelog_data = filelog_entry.data()?; let p1_filelog_data = filelog_entry.data()?;
let contents_in_p1 = filelog_data.file_data()?; let p1_contents = p1_filelog_data.file_data()?;
if contents_in_p1.len() as u64 != fs_len { if p1_contents.len() as u64 != fs_len {
// No need to read the file contents: // No need to read file contents:
// it cannot be equal if it has a different length. // it cannot be equal if it has a different length.
return Ok(true); return Ok(true);
} }
let fs_contents = if is_symlink { let fs_contents = if is_symlink {
get_bytes_from_os_string(vfs.read_link(fs_path)?.into_os_string()) get_bytes_from_os_string(vfs.read_link(fs_path)?.into_os_string())
} else { } else {
vfs.read(fs_path)? vfs.read(fs_path)?
}; };
Ok(contents_in_p1 != &*fs_contents) Ok(p1_contents != &*fs_contents)
} }