VisitChildrenSet has no need to own the set, this will save allocations.
The file_set return type change is motivated by both ergonomics and... being
able to compile code.
The AlwaysMatcher does not store a file_set, which requires it to return an
owned HashSet, which in turn would change our return type to Cow<&HgPath>
(lifetimes omitted). This is both un-ergonomic and troublesome for more
complex lifetime issues (especially with the upcoming FileMatcher in the
following patch).
Another option here would just to have a global empty hash set.