( )⚙ D7928 rust-status: add function for sequential traversal of the working directory

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D7928

rust-status: add function for sequential traversal of the working directory
ClosedPublic

Authored by Alphare on Jan 17 2020, 10:33 AM.

Details

Reviewers
marmoute
Group Reviewers
hg-reviewers
Commits
rHG1debb5894b39: rust-status: add function for sequential traversal of the working directory
Summary

This change also introduces helper structs to make things clearer.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Alphare created this revision.Jan 17 2020, 10:33 AM
marmoute requested changes to this revision.Feb 5 2020, 5:40 PM
marmoute added a subscriber: marmoute.

There is a lot going on in this changeset. I am counting at least:

  • new BadType/BadMatch structs,
  • StatusResult → DirstateStatus change,
  • new StatusOptions struct
  • new traverse_worker function.

Can we get the smaller cleanup as reparated chagneset before hand? It would make the thing clearer in my opinion.

rust/hg-core/src/dirstate/status.rs
64–65

nits: It might be worth adding the same commentan extra comment to the BadType/BadMatch declaration to clarify their intent.

This revision now requires changes to proceed.Feb 5 2020, 5:40 PM
Alphare marked an inline comment as done.Feb 6 2020, 5:07 AM
In D7928#119428, @marmoute wrote:

There is a lot going on in this changeset. I am counting at least:

  • new BadType/BadMatch structs,
  • StatusResult → DirstateStatus change,
  • new StatusOptions struct
  • new traverse_worker function.

Can we get the smaller cleanup as reparated chagneset before hand? It would make the thing clearer in my opinion.

I don't think the BadType/BadMatch struct warrant a new changeset. I'll split into 3 then.

Alphare updated this revision to Diff 20047.Feb 10 2020, 11:30 AM
Alphare updated this revision to Diff 20187.Feb 13 2020, 12:37 PM
Alphare updated this revision to Diff 20453.Mar 4 2020, 10:08 AM
Alphare updated this revision to Diff 20461.Mar 4 2020, 11:55 AM
Alphare added a commit: rHG1debb5894b39: rust-status: add function for sequential traversal of the working directory.Mar 11 2020, 12:23 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG1debb5894b39: rust-status: add function for sequential traversal of the working directory (authored by Alphare). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mrust/hg-core/src/dirstate/status.rs (181 lines)

Diff 20720

rust/hg-core/src/dirstate/status.rs

// status.rs // status.rs
// //
// Copyright 2019 Raphaël Gomès <rgomes@octobus.net> // Copyright 2019 Raphaël Gomès <rgomes@octobus.net>
// //
// This software may be used and distributed according to the terms of the // This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version. // GNU General Public License version 2 or any later version.
//! Rust implementation of dirstate.status (dirstate.py). //! Rust implementation of dirstate.status (dirstate.py).
//! It is currently missing a lot of functionality compared to the Python one //! It is currently missing a lot of functionality compared to the Python one
//! and will only be triggered in narrow cases. //! and will only be triggered in narrow cases.
use crate::{ use crate::{
dirstate::SIZE_FROM_OTHER_PARENT, dirstate::SIZE_FROM_OTHER_PARENT,
matchers::Matcher, matchers::{Matcher, VisitChildrenSet},
utils::{ utils::{
files::HgMetadata, files::HgMetadata,
hg_path::{ hg_path::{
hg_path_to_path_buf, os_string_to_hg_path_buf, HgPath, HgPathBuf, hg_path_to_path_buf, os_string_to_hg_path_buf, HgPath, HgPathBuf,
}, },
}, },
CopyMap, DirstateEntry, DirstateMap, EntryState, CopyMap, DirstateEntry, DirstateMap, EntryState, FastHashMap,
}; };
use rayon::prelude::*; use rayon::prelude::*;
use std::collections::HashSet; use std::borrow::Cow;
use std::collections::{HashSet, VecDeque};
use std::fs::{read_dir, DirEntry}; use std::fs::{read_dir, DirEntry};
use std::io::ErrorKind;
use std::ops::Deref;
use std::path::Path; use std::path::Path;
/// Wrong type of file from a `BadMatch` /// Wrong type of file from a `BadMatch`
/// Note: a lot of those don't exist on all platforms. /// Note: a lot of those don't exist on all platforms.
#[derive(Debug)] #[derive(Debug)]
pub enum BadType { pub enum BadType {
CharacterDevice, CharacterDevice,
BlockDevice, BlockDevice,
Added, Added,
Removed, Removed,
Deleted, Deleted,
Clean, Clean,
Unknown, Unknown,
Ignored, Ignored,
/// Empty dispatch, the file is not worth listing /// Empty dispatch, the file is not worth listing
None, None,
/// Was explicitly matched but cannot be found/accessed /// Was explicitly matched but cannot be found/accessed
Bad(BadMatch), Bad(BadMatch),
marmouteUnsubmitted
Done

nits: It might be worth adding the same commentan extra comment to the BadType/BadMatch declaration to clarify their intent.

marmoute: nits: It might be worth adding the same commentan extra comment to the BadType/BadMatch…
Directory { Directory {
/// True if the directory used to be a file in the dmap so we can say /// True if the directory used to be a file in the dmap so we can say
/// that it's been removed. /// that it's been removed.
was_file: bool, was_file: bool,
}, },
} }
type IoResult<T> = std::io::Result<T>; type IoResult<T> = std::io::Result<T>;
pub struct StatusOptions { pub struct StatusOptions {
/// Remember the most recent modification timeslot for status, to make /// Remember the most recent modification timeslot for status, to make
/// sure we won't miss future size-preserving file content modifications /// sure we won't miss future size-preserving file content modifications
/// that happen within the same timeslot. /// that happen within the same timeslot.
pub last_normal_time: i64, pub last_normal_time: i64,
/// Whether we are on a filesystem with UNIX-like exec flags /// Whether we are on a filesystem with UNIX-like exec flags
pub check_exec: bool, pub check_exec: bool,
pub list_clean: bool, pub list_clean: bool,
pub list_unknown: bool,
pub list_ignored: bool,
}
/// Dispatch a single file found during `traverse`.
/// If `file` is a folder that needs to be traversed, it will be pushed into
/// `work`.
fn traverse_worker<'a>(
work: &mut VecDeque<HgPathBuf>,
matcher: &impl Matcher,
dmap: &DirstateMap,
filename: impl AsRef<HgPath>,
dir_entry: &DirEntry,
ignore_fn: &impl for<'r> Fn(&'r HgPath) -> bool,
dir_ignore_fn: &impl for<'r> Fn(&'r HgPath) -> bool,
options: StatusOptions,
) -> Option<IoResult<(Cow<'a, HgPath>, Dispatch)>> {
let file_type = match dir_entry.file_type() {
Ok(x) => x,
Err(e) => return Some(Err(e.into())),
};
let filename = filename.as_ref();
let entry_option = dmap.get(filename);
if file_type.is_dir() {
// Do we need to traverse it?
if !ignore_fn(&filename) || options.list_ignored {
work.push_front(filename.to_owned());
}
// Nested `if` until `rust-lang/rust#53668` is stable
if let Some(entry) = entry_option {
// Used to be a file, is now a folder
if matcher.matches_everything() || matcher.matches(&filename) {
return Some(Ok((
Cow::Owned(filename.to_owned()),
dispatch_missing(entry.state),
)));
}
}
} else if file_type.is_file() || file_type.is_symlink() {
if let Some(entry) = entry_option {
if matcher.matches_everything() || matcher.matches(&filename) {
let metadata = match dir_entry.metadata() {
Ok(x) => x,
Err(e) => return Some(Err(e.into())),
};
return Some(Ok((
Cow::Owned(filename.to_owned()),
dispatch_found(
&filename,
*entry,
HgMetadata::from_metadata(metadata),
&dmap.copy_map,
options,
),
)));
}
} else if (matcher.matches_everything() || matcher.matches(&filename))
&& !ignore_fn(&filename)
{
if (options.list_ignored || matcher.exact_match(&filename))
&& dir_ignore_fn(&filename)
{
if options.list_ignored {
return Some(Ok((
Cow::Owned(filename.to_owned()),
Dispatch::Ignored,
)));
}
} else {
return Some(Ok((
Cow::Owned(filename.to_owned()),
Dispatch::Unknown,
)));
}
}
} else if let Some(entry) = entry_option {
// Used to be a file or a folder, now something else.
if matcher.matches_everything() || matcher.matches(&filename) {
return Some(Ok((
Cow::Owned(filename.to_owned()),
dispatch_missing(entry.state),
)));
}
}
None
}
/// Walk the working directory recursively to look for changes compared to the
/// current `DirstateMap`.
fn traverse<'a>(
matcher: &(impl Matcher + Sync),
root_dir: impl AsRef<Path>,
dmap: &DirstateMap,
path: impl AsRef<HgPath>,
old_results: FastHashMap<Cow<'a, HgPath>, Dispatch>,
ignore_fn: &(impl for<'r> Fn(&'r HgPath) -> bool + Sync),
dir_ignore_fn: &(impl for<'r> Fn(&'r HgPath) -> bool + Sync),
options: StatusOptions,
) -> IoResult<FastHashMap<Cow<'a, HgPath>, Dispatch>> {
let root_dir = root_dir.as_ref();
let mut new_results = FastHashMap::default();
let mut work = VecDeque::new();
work.push_front(path.as_ref().to_owned());
while let Some(ref directory) = work.pop_front() {
if directory.as_bytes() == b".hg" {
continue;
}
let visit_entries = match matcher.visit_children_set(directory) {
VisitChildrenSet::Empty => continue,
VisitChildrenSet::This | VisitChildrenSet::Recursive => None,
VisitChildrenSet::Set(set) => Some(set),
};
let buf = hg_path_to_path_buf(directory)?;
let dir_path = root_dir.join(buf);
let skip_dot_hg = !directory.as_bytes().is_empty();
let entries = match list_directory(dir_path, skip_dot_hg) {
Err(e) => match e.kind() {
ErrorKind::NotFound | ErrorKind::PermissionDenied => {
new_results.insert(
Cow::Owned(directory.to_owned()),
Dispatch::Bad(BadMatch::OsError(
// Unwrapping here is OK because the error always
// is a real os error
e.raw_os_error().unwrap(),
)),
);
continue;
}
_ => return Err(e),
},
Ok(entries) => entries,
};
for (filename, dir_entry) in entries {
if let Some(ref set) = visit_entries {
if !set.contains(filename.deref()) {
continue;
}
}
// TODO normalize
let filename = if directory.is_empty() {
filename.to_owned()
} else {
directory.join(&filename)
};
if !old_results.contains_key(filename.deref()) {
if let Some((res, dispatch)) = traverse_worker(
&mut work,
matcher,
&dmap,
&filename,
&dir_entry,
&ignore_fn,
&dir_ignore_fn,
options,
)
.transpose()?
{
new_results.insert(res, dispatch);
}
}
}
}
new_results.extend(old_results.into_iter());
Ok(new_results)
} }
/// Stat all entries in the `DirstateMap` and mark them for dispatch into /// Stat all entries in the `DirstateMap` and mark them for dispatch into
/// the relevant collections. /// the relevant collections.
fn stat_dmap_entries( fn stat_dmap_entries(
dmap: &DirstateMap, dmap: &DirstateMap,
root_dir: impl AsRef<Path> + Sync + Send, root_dir: impl AsRef<Path> + Sync + Send,
options: StatusOptions, options: StatusOptions,