This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1974

narrow: import experimental extension from narrowhg revision cb51d673e9c5
ClosedPublic

Authored by durin42 on Feb 1 2018, 6:00 PM.

Download Raw Diff

Details

Reviewers

indygreg

Group Reviewers

hg-reviewers

Commits

rHGa2a6e724d61a: narrow: import experimental extension from narrowhg revision cb51d673e9c5

Summary

Adjustments:

renamed src to hgext/narrow
marked extension experimental
added correct copyright header where it was missing
updated hgrc extension enable line in library.sh
renamed library.sh to narrow-library.sh
dropped all files from repo root as they're not interesting
dropped test-pyflakes.t, test-check-code.t and test-check-py3-compat.t
renamed remaining tests to all be test-narrow-* when they didn't already
fixed test-narrow-expanddirstate.t to refer to narrow and not narrowhg
fixed tests that wanted update -C . instead of merge --abort
corrected a two-space indent in narrowspec.py
added a missing _() in narrowcommands.py
fixed imports to pass the import checker
narrow only adds its --include and --exclude to clone if sparse isn't enabled to avoid breaking test-duplicateoptions.py. This is a kludge, and we'll need to come up with a better solution in the future.

These were more or less the minimum to import something that would
pass tests and not create a bunch of files we'll never use.

Changes I intend to make as followups:

rework the test-narrow-*-tree.t tests to use the new testcases functionality in run-tests.py
remove lots of monkeypatches of core things

Diff Detail

Repository

rHG Mercurial

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

durin42 created this revision.Feb 1 2018, 6:00 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptFeb 1 2018, 6:00 PM

Herald added a subscriber: mercurial-devel. · View Herald Transcript

durin42 added a child revision: D1975: narrow: remove support for old match modules.Feb 1 2018, 6:00 PM

I glanced at the code and found the first set of needed follow-ups.

hgext/narrow/__init__.py
12–15	This seems inappropriate.
hgext/narrow/narrowbundle2.py
38–47	I think we'll want to UPPERCASE these since they are constants.
224	If this is referring to Mercurial versions, then this code is likely not needed, as a client with narrow support is newer than 3.1 or 3.2.
hgext/narrow/narrowrepo.py
27	I think we'll want to put `exp` somewhere in the requirement name until we're ready to mark the feature as non-experimental in core.
hgext/narrow/narrowrevlog.py
17–21	This needs to move to core because revision flags need to be reserved there.
hgext/narrow/narrowtemplates.py
46–51	We should be using the `registrar` API for declaring revsets and templates.

Should I add follow-up commits for the feedback? I think I'd slightly prefer doing that so that we import something that's as minimally modified as possible, and then fix it up all at once, but if you'd rather fix things here that's fine too.

(I agree with the things you caught, I just don't have time for them tonight - maybe tomorrow.)

hgext/narrow/__init__.py
12–15	Yeah, I noticed this after doing the first big extraction. It's done in D1980, but we can probably trivially fold that here. Do you have a preference?

In D1974#33514, @durin42 wrote:

Should I add follow-up commits for the feedback? I think I'd slightly prefer doing that so that we import something that's as minimally modified as possible, and then fix it up all at once, but if you'd rather fix things here that's fine too.

I think we should keep the import as similar to upstream as possible. That creates a cleaner transition of the code. So please make fixes in follow-up commits. We may want to check in a todo.rst file or similar in this initial commit to facilitate tracking known problems from the inception of the code.

Alrighty, I've added some follow-ups. There's one question for Martin here, which I think will result in one more follow-up to improve a comment. Take another look?

hgext/narrow/narrowbundle2.py
224	Hmm, I think this might be talking about client versions in the server code. @martinvonz do you remember what's going on here? If not I can dig more.

martinvonz added inline comments.Feb 2 2018, 11:29 AM

hgext/narrow/narrowbundle2.py
224	Copied from here: https://www.mercurial-scm.org/repo/hg/file/5cfdf6137af8/mercurial/exchange.py#l1811. I'd say leave it since it's harmless and it makes the import simpler. We can drop it after, but that's probably also unnecessary, since the idea is to move this into core and then it will be naturally lost anyway (because it's already in the destination function).

I'm still reviewing. Just thought I'd flush another batch of feedback :)

hgext/narrow/narrowrepo.py
29–34	Is this really the best way to influence the content of the `shared` file? If it doesn't exist, a good follow-up would be a hook point in core to allow extensions to modify the content of this file so it is only written once.
36–44	Again, my reaction is "eww." But this one is a bit special since we're dealing with a non-standard file. So unless we create a "also write these files with this content" mechanism (which doesn't feel necessary), this is probably the best we can do. This code should live in core eventually though. So it should work itself out.

indygreg added inline comments.Feb 2 2018, 2:15 PM

hgext/narrow/narrowrepo.py
76	Please add a docstring, since this is a new public method.
86–88	I agree with the inline todo :)
90–92	See comment in `narrowspec.py` about needing to hold the `wlock` somewhere.
94–97	I agree with the comment. I'm not an expert on dirstate/status, but this does seem like the wrong place. I'm pretty sure we have code paths that call lower-level `status()` functions. I would expect this to live in a similar place to where sparse lives. I think we can overlook this for the initial landing. This feels like a bit of work to resolve and can be deferred to the stabilization period.
hgext/narrow/narrowrevlog.py
2	I'd like to see this file's content moved into core sooner rather than later. There are a lot of implications for storage that need to be in people's minds when they are touching code in core.
32	This condition should always be true.
41	Please add docstring.
58	docstring
67–68	I think we prefer `error.ProgrammingError` these days.
70	docstring
75–88	`error.ProgrammingError`
91–95	Consider doing that and changing this to raise if called.
106	Nit: `dir` is a builtin. If this matches core, fine. But I'd prefer avoiding the name collision.
115–116	In-place mutation of low-level types. Yummy. Please add a todo for the post-landing list to construct the proper type from the beginning. This likely requires some API changes in core. I'm thinking some function should return the type to use for new revlogs. Or we should spawn this type and call super.init from its init.
129–132	I think this wants a comment explaining why we lie about rename metadata when the destination is outside of the narrow spec. Also, we'll want to flag this for further review, since there could be some interesting implications to lying here.
135–140	This is making assumptions about code that hasn't landed yet. Not sure if we should replace this with a TODO or what.
hgext/narrow/narrowspec.py
147–151	This is called from `narrowrepo.setnarrowparts()` and neither of them cares about locking. Somewhere we should ensure we hold the wlock.

Another batch before I head off to a meeting...

hgext/narrow/narrowcopies.py
2	Copy code is complex and has performance concerns. I'd like to see this moved into core sooner rather than later. Perhaps a good first step would be to add the `narrowmatch` method to `localrepository`? Or something on `localrepository` in core so we can detect presence of narrow in a one-liner.
hgext/narrow/narrowdirstate.py
2	I'm not an expert on dirstate. But I know enough to know that "here be dragons." I'd like to see everything dirstate moved into core ASAP so it can be better integrated with sparse, etc. People hacking on this code in core need to be aware of the implications for narrow.
hgext/narrow/narrowmerge.py
2	More code that should be on the fast track to core.
26	The more I see this pattern, the more I think there needs to be an attribute on `localrepository` that can be used for quickly testing if narrow is in play. Actually, should we assume all repos are narrow (with the all matcher) and fast path the special case of the default?
hgext/narrow/narrowspec.py
27	The format needs documentation somewhere inline.
32–43	Oh, hey, this looks just like sparse profiles! I sense some code conversion in our future...
50–55	Playing devil's advocate, do we really need two formats doing the same thing?
84	Can we use `str.count()` to avoid creating objects for each line?
94–96	What about `\` as a path separator? Should we also ban that? I assume we're interpreting the value here and paths as bytes, so `\` will never be used as an escape character?
111–116	It is better to build a list of lines and `'\n'.join()` them.
134–139	This reinvents `repo.vfs.tryread()`.
155–163	The purpose of this function was not clear on initial reading. Could you please elaborate a bit more in the docstring?
180	We use `set(repo_includes)` below to copy a set. Let's make things consistent.
186–193	I feel like there's a more efficient mechanism lurking here. But I'm not sure if we care about the performance implications of this function.
198	Mutating an argument feels a bit wonky. Should this be returned instead?
hgext/narrow/narrowwirepeer.py
28	There is no server-side implementation of this capability/command yet. We may want to remove the reference from core. Also, the wire protocol capability should have `exp` or `experimental` in it until we formalize the API.

I'm still only partly through the changegroup and bundle code. Haven't looked at commands or tests yet. I think I'm going to take a break for a bit because this is a lot of code to absorb!

hgext/narrow/narrowbundle2.py
38	This wants an `exp` or `experimental` in its name. Once we start advertising it, there's no going back. So we shouldn't squat on final names until we're sure.
127	`map()` returns an iterator on Python 3.
141–143	No, we can't. There were bugs in e.g. rebase that caused changed files to be dropped from the files list in the changeset. Walking manifests is the only way to be sure you are getting all changes.
261	Would it make sense to send the narrow spec before the changegroup part?
263–267	Part parameters are effectively header data for bundle2 parts. As such, they are read and written as atomic units. And they are URL quoted for transport safety. Since narrow specs are unbound in size, I think they should be transferred as part of the part payload, not in parameters. This allows for streaming and doesn't add overhead for encoding/decoding their content. This is obviously a BC change to the wire protocol and bundle storage.
hgext/narrow/narrowchangegroup.py
48	Might want a comment here that the wrapped method will get inherited by later versions of the changegroup packer. It feels weird to see this definition on `cg1packer` since code above removes support for changegroup versions `01` and `02`.
61–82	I don't fully understand linknodes. This should be scrutinized by someone who does, especially since there are likely performance concerns here.
103–106	I'm not a fan of `getattr()` and `del` here. Is there no way to always set these attributes on instance creation? (One solution is to move this code into `changegroup.py`.)
hgext/narrow/narrowwirepeer.py
46–49	This will blindly add `includepats` and `excludepats` as wire protocol arguments to the `unbundle` command. In the Python server, wire protocol arguments have to be defined as part of the command handler or the server will refuse to serve the request. So, this code is missing the server-side declaration of these arguments. And, the client/peer code here is buggy for not conditionally adding these arguments based on the presence of a server capability advertising support for receiving these arguments.

OK. I've now looked at all the Python code. I haven't yet spent much time on tests. But the amount of tests is encouraging.

I did glance over some of the complicated parts of the Python. Namely some of the algorithms around changegroup and bundle semantics. I will want to understand that mechanism someday. But I think I can delay that to when functionality is moved into core.

As it stands, I'm overall pretty happy with the state of things. Perfect is the enemy of good and I'm totally on board with landing mostly intact and refactoring and applying higher scrutiny once it lands.

The biggest concerns are around the parts that touch storage and wire protocol compatibility. If we write files or advertise things over the wire protocol, we need to be extra careful about the strings used for requirements, capabilities, bundle part names, etc as any future Mercurial client could encounter a requirement, capability, bundle, etc from an old Mercurial and the peers may disagree about the semantics of a given name. As long as these issues are addressed before 4.6 is released, we should be fine. But once something is used in the wild (which I equate to shipping a release), we have backwards and forwards compatibility risks to consider. I'm confident we'll address these before 4.6 is released. I'd like to see more judicial use of experimental in these names as soon as possible though, as that will keep the window for considering compatibility concerns more... narrow.

I'm really tempted to sign off on this review right now. However, I left a pile of comments. And the more of those that can be addressed in the initial landing, the better I'll feel about things. I'm find with this series growing to a few dozen in-flight changesets before anything lands if you are. Although I do want to get the code in as quickly as possible, as that is the best way to bake a feature.

hgext/narrow/narrowbundle2.py
139–152	I'm not an expert on the manifest APIs. There may be a more optimal way to implement this...
271–275	Should this code be above the `if narrowservebrokenellipses` block above? It seems like it is doing generic argument validation and that validation should be near the top of the function.
337	We may want to remove this feature for the time being, as it is very Google specific. I'm happy to provide a mechanism for extensions to hook in here to enforce ACLs. But there's nothing in core that deals with path-based access control and I think things should remain that way in order to keep the code simple. For the time being at least. I do acknowledge value in the feature and it is something I'd like to see in core some day. I just think we should stabilize the partial clone functionality first before we start forcing people to think about path-based access control.
379–381	Silently adding a repo requirement during a pull or unbundle operation doesn't feel right to me. At least not at this juncture. For at least as long as the feature is experimental, I think we should refuse to apply (or fetch) a narrow bundle if the repo isn't already narrow. We can provide an `hg debug*` command to add the requirement to make people's lives easier.
416–417	This compatibility with older Mercurial versions can be removed.
419–420	Doing strip-based operations as part of bundle application breaks read-only consistency of the repository during those operations. IMO a blocking issue to eventually making this feature non-experimental will be either a refactored store (so we don't need to perform strips to backfill data) or reader locks on the store (to lock out readers when operations such as strip that make the store inconsistent are being performed). Either of those will require a new repo requirement. That requirement is centered around the store and not narrow/shallow/partial clone support. So it needs to be considered a separate feature from all this clone feature work.
444–445	This should be using a context manager instead of `try..finally`.
451–454	I think `strip` should be something else to denote this from an actual `strip` operation.
462–467	This reinvents `vfs.tryunlink()`. Although you'll still want to catch the `OSError` from `tryunlink()` to turn it into a warning. But at least you can not worry about `ENOENT`.
476–482	I'm not super keen on tacking all these arguments onto the existing `getbundle` wire protocol command. Especially without experimental labels. I'd rather we invent new wire protocol commands when we add major new features like this. That's something I hope to address in the new wire protocol. I'm fine with this for now. Just as long as the client does some form of capabilities screening to know which set of arguments the server-side wire protocol command accepts. As long as we fix this during the `4.6` release period, we should be fine. Once version 2 of the wire protocol is done, I'll likely establish a new `getbundle` like command that is version 2 only and will remove narrow support from the existing `getbundle` command.
488–490	More ACL code that we may remove from core.
hgext/narrow/narrowchangegroup.py
144–152	Eek - this fully reimplements `generate()`?! This is probably the hackiest part of this code I've seen so far. I know I've said I want a lot of things moved into core. But considering the complexity of this duplicate code, I think this belongs as front as possible in the queue.
158	This should change to `cl.changelogrevision(x)` so we return an object with named attributes instead of a tuple. That will make e.g. `c[0]` change to `cl.manifest`. `cl.read()` is just a proxy around `changelogrevision()` anyway. If the changegroup code uses `read()`, it should be changed as well.
166	Nit: `cl.rev()` is called here and inside the conditional. My recollection is this loop is pretty tight, so aliasing `clrev = cl.rev` and saving the result to avoid another call might have an impact.
314–323	This code is rather difficult to reason about. I'm inclined to defer on a comprehensive review until this code is moved to core. It isn't helping that this file reinvents so much code already in core and it is difficult to reason about how everything plays together. Although the inline comments noting the narrow changes are very useful and appreciated.
hgext/narrow/narrowcommands.py
172–173	This function does a lot and wants a docstring.
172–182	I believe a few operations in this function do obtain a write lock. However, the repo needs to be consistent for the duration of this function. So I think a write lock needs to be taken explicitly as part of the function.
190	Use of `repo.changelog` in a loop creates perf problems. It's recommended to alias `repo.changelog.node` to a local and call that, as that also avoids the attribute lookup overhead, which does matter in tight loops.
210	`.changelog.node` in a tight loop.
219–236	This is extremely low-level code to have in this file. It belongs somewhere in the store API, since it assumes a specific filesystem layout in the store.
254	This function does a lot. It wants a docstring.
tests/test-narrow-archive.t
31	I'm not sure if this invocation of `tar` is portable. I'm sure others will fix things later if it breaks tests on other platforms.
tests/test-narrow-clone-non-narrow-server.t
22–24	The `capabilities` command isn't used by the SSH/stdio protocol. This should be using `hello` instead. I suspect I'll soon write a patch that doesn't expose the `capabilities` command to the SSH protocol because it isn't supposed to be used...

I know it isn't a viable feature for Google, but I have an idea that may greatly reduce the burden to getting narrow non-experimental...

What do you think about adding a mode that transfers the full changelog, all manifest revisions for relevant trees, and all filelog revisions for subscribed files. There would be no shallow cloning, ellipsis nodes, linkrev rewriting, etc. This would be a purely narrow clone. I think this would eliminate all the technical problems around having to backfill a store (via strip operations).

If this is technically viable, I think it would allow us to put non-shallow narrow clone on a faster track for being non-experimental, since a purely narrow repo would operate much like a traditional repository: we'd just be interacting with a subset of filelogs on client and server.

This line of thought causes me to ask questions like whether we should split the repo requirement, wire protocol capabilities, etc to more differentiate narrow support from shallow.

None of this should block initial landing. But it's definitely something I'd like the maintainers of narrowhg to think about.

idlsoft added a subscriber: idlsoft.Feb 8 2018, 1:16 PM

idlsoft added inline comments.Feb 8 2018, 1:52 PM

hgext/narrow/narrowbundle2.py
337	I've contributed the ACL code originally. It's a relatively trivial piece of code, at least compared to the rest of the codebase, and it is called conditionally. I hope it can stay, my company is about to migrate from git to hg because of it.
379–381	This was added to support seamless clone from an ACLed repo. The user doesn't know ahead of time about server-side narrowing.

durin42 marked 12 inline comments as done.Feb 12 2018, 3:02 PM

durin42 updated this revision to Diff 5498.

Okay, I've pushed more followups. The bulk of the TODOs are recorded in a file introduced in D2196, but there are also some added inline in the code.

Let me know how I can help - I'll make time to video conference this week if it'd facilitate getting this landed with minimal additional lost sanity.

Thanks!

hgext/narrow/narrowbundle2.py
139–152	I wrote lazymanifest, and this was the best adgar and I could come up with...
hgext/narrow/narrowrepo.py
86–88	Okay if we come back to that, since it's already logged as a TODO?
hgext/narrow/narrowrevlog.py
2	Yes, noted this in a TODO.rst.
91–95	Added to a TODO file.
106	Ouch, good catch. This does match core, so I fixed both in my followup.
115–116	Yeah, this is one of several places where I want to just hoist internals changes to core, with the only customer being narrow (for now, at least). It's super gross the way it currently is.
129–132	Yeah, all I remember about this (https://bitbucket.org/Google/narrowhg/commits/3cd72b1a1b41c9e46f12eba78c253da169277374) is that git-diffs break in the case of a rename from outside->inside. There are almost certainly other problems lurking in the weeds here, but to my knowledge we've not seen them at Google...
135–140	Added an explicit TODO. We can make this a TON cleaner when narrowhg can be formally aware of remotefilelog, this was mostly a kludge to work around them being in disjoint repos and not wanting to assume we could find remotefilelog cleanly.
hgext/narrow/narrowspec.py
32–43	Yes, the goal is that the two extensions will share a fair amount of logic. :)
50–55	That's a great question. I don't really know the answer offhand. Added a TODO...
84	Gave it a shot, and things broke all over the place. :(
94–96	@martinvonz please correct me if this is wrong. The paths are stored in the narrowspec with canonical (that is /) path separators, just like the rest of the internals. We're reading things as bytes, so yes, \ should never occur.
111–116	It shouldn't be: this particular "str with refcount of 1" behavior was optimized a long time ago in cpython to not be a horrendous stack of copies. If it shows up in a profile, we can obviously fix it.
134–139	Yep, but it also does some extra cache invalidation, so I don't think I can fuse them.
147–151	Added to my TODO doc because I don't want reasoning about locking to block getting out of this pile-of-comments hell.
hgext/narrow/narrowwirepeer.py
46–49	Yeah, this is a mess. I've added a TODO, as I'm not entirely sure how to work around this for now (calling capabilities in here seems...fraught.)
tests/test-narrow-archive.t
31	Noted. AFAIK this should be portable at least to bsdtar, but probably solaris or someone will eventually complain?

I think I've seen enough follow-ups to feel comfortable taking this in core. There's still a ton of work that needs to get done. But it will be easier to iterate and for others to get involved when the code is committed than when it is sitting around in review.

This revision is now accepted and ready to land.Feb 12 2018, 6:02 PM

Closed by commit rHGa2a6e724d61a: narrow: import experimental extension from narrowhg revision cb51d673e9c5 (authored by durin42). · Explain WhyFeb 12 2018, 6:17 PM

This revision was automatically updated to reflect the committed changes.

mharbison72 added a subscriber: mharbison72.Feb 15 2018, 10:37 PM

mharbison72 added inline comments.

tests/test-narrow-strip.t
81	The test failures here on Windows seem to be a case of incoming and unbundle getting confused. Here's some debug info (starting right after the `rm` on line 77), but I'm not sure what to do with it: $ hg log -G o changeset: 2:f505d5e96aa8 \| tag: tip \| parent: 0:a99f4d53924d \| user: test \| date: Thu Jan 01 00:00:00 1970 +0000 \| summary: modify inside \| \| @ changeset: 1:9e48d953700d \|/ user: test \| date: Thu Jan 01 00:00:00 1970 +0000 \| summary: modify outside again \| o changeset: 0:a99f4d53924d user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: initial $ hg strip . 0 files updated, 0 files merged, 0 files removed, 0 files unresolved saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/-backup.hg (glob) $ hg log -G o changeset: 1:f505d5e96aa8 \| tag: tip \| user: test \| date: Thu Jan 01 00:00:00 1970 +0000 \| summary: modify inside \| @ changeset: 0:a99f4d53924d user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: initial $ hg in --debug --bundle .hg/strip-backup/-backup.hg bundle2-input-part: total payload size 521 bundle2-input-part: total payload size 24 comparing with .hg/strip-backup/f505d5e96aa8-2b55e253-backup.hg query 1; heads searching for changes all remote heads known locally no changes found [1] $ hg unbundle --debug .hg/strip-backup/*-backup.hg bundle2-input-bundle: 1 params with-transaction bundle2-input-part: "changegroup" (params: 1 mandatory 1 advisory) supported adding changesets add changeset f505d5e96aa8 adding manifests adding file changes adding inside/f1 revisions added 0 changesets with 0 changes to 1 files bundle2-input-part: total payload size 521 bundle2-input-part: "phase-heads" supported bundle2-input-part: total payload size 24 bundle2-input-bundle: 1 parts total (run 'hg update' to get a working copy) $ hg log -G -T '{rev} {desc}\n' o 1 modify inside \| @ 0 initial

tests/test-narrow-strip.t

The test failures here on Windows seem to be a case of incoming and unbundle getting confused. Here's some debug info (starting right after the rm on line 77), but I'm not sure what to do with it:

$ hg log -G
o  changeset:   2:f505d5e96aa8
|  tag:         tip
|  parent:      0:a99f4d53924d
|  user:        test
|  date:        Thu Jan 01 00:00:00 1970 +0000
|  summary:     modify inside
|
| @  changeset:   1:9e48d953700d
|/   user:        test
|    date:        Thu Jan 01 00:00:00 1970 +0000
|    summary:     modify outside again
|
o  changeset:   0:a99f4d53924d
   user:        test
   date:        Thu Jan 01 00:00:00 1970 +0000
   summary:     initial

$ hg strip .
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-backup.hg (glob)
$ hg log -G
o  changeset:   1:f505d5e96aa8
|  tag:         tip
|  user:        test
|  date:        Thu Jan 01 00:00:00 1970 +0000
|  summary:     modify inside
|
@  changeset:   0:a99f4d53924d
   user:        test
   date:        Thu Jan 01 00:00:00 1970 +0000
   summary:     initial

$ hg in --debug --bundle .hg/strip-backup/*-backup.hg
bundle2-input-part: total payload size 521
bundle2-input-part: total payload size 24
comparing with .hg/strip-backup/f505d5e96aa8-2b55e253-backup.hg
query 1; heads
searching for changes
all remote heads known locally
no changes found
[1]
$ hg unbundle --debug .hg/strip-backup/*-backup.hg
bundle2-input-bundle: 1 params with-transaction
bundle2-input-part: "changegroup" (params: 1 mandatory 1 advisory) supported
adding changesets
add changeset f505d5e96aa8
adding manifests
adding file changes
adding inside/f1 revisions
added 0 changesets with 0 changes to 1 files
bundle2-input-part: total payload size 521
bundle2-input-part: "phase-heads" supported
bundle2-input-part: total payload size 24
bundle2-input-bundle: 1 parts total
(run 'hg update' to get a working copy)
$ hg log -G -T '{rev} {desc}\n'
o  1 modify inside
|
@  0 initial

Revision Contents
Changeset List

		Path
A	M	hgext/narrow/__init__.py (111 lines)
A	M	hgext/narrow/narrowbundle2.py (503 lines)
A	M	hgext/narrow/narrowchangegroup.py (385 lines)
A	M	hgext/narrow/narrowcommands.py (402 lines)
A	M	hgext/narrow/narrowcopies.py (35 lines)
A	M	hgext/narrow/narrowdirstate.py (80 lines)
A	M	hgext/narrow/narrowmerge.py (76 lines)
A	M	hgext/narrow/narrowpatch.py (42 lines)
A	M	hgext/narrow/narrowrepo.py (110 lines)
A	M	hgext/narrow/narrowrevlog.py (163 lines)
A	M	hgext/narrow/narrowspec.py (204 lines)
A	M	hgext/narrow/narrowtemplates.py (50 lines)
A	M	hgext/narrow/narrowwirepeer.py (51 lines)
M		setup.py (3 lines)
A	M	tests/narrow-library.sh (9 lines)
M		tests/test-help.t (2 lines)
A	M	tests/test-narrow-acl.t (42 lines)
A	M	tests/test-narrow-archive.t (32 lines)
A	M	tests/test-narrow-clone-no-ellipsis.t (130 lines)
A	M	tests/test-narrow-clone-non-narrow-server.t (53 lines)
A	M	tests/test-narrow-clone-nonlinear.t (148 lines)
A	M	tests/test-narrow-clone.t (225 lines)
A	M	tests/test-narrow-commit-tree.t (21 lines)
A	M	tests/test-narrow-commit.t (79 lines)
A	M	tests/test-narrow-copies.t (57 lines)
A	M	tests/test-narrow-debugcommands.t (43 lines)
A	M	tests/test-narrow-debugrebuilddirstate.t (31 lines)
A	M	tests/test-narrow-exchange-merges.t (207 lines)
A	M	tests/test-narrow-exchange.t (210 lines)
A	M	tests/test-narrow-expanddirstate.t (170 lines)
A	M	tests/test-narrow-merge-tree.t (28 lines)
A	M	tests/test-narrow-merge.t (94 lines)
A	M	tests/test-narrow-patch-tree.t (5 lines)
A	M	tests/test-narrow-patch.t (76 lines)
A	M	tests/test-narrow-patterns.t (418 lines)
A	M	tests/test-narrow-pull.t (175 lines)
A	M	tests/test-narrow-rebase.t (93 lines)
A	M	tests/test-narrow-shallow-merges.t (345 lines)
A	M	tests/test-narrow-shallow.t (122 lines)
A	M	tests/test-narrow-strip-tree.t (52 lines)
A	M	tests/test-narrow-strip.t (148 lines)
A	M	tests/test-narrow-tree.t (68 lines)
A	M	tests/test-narrow-update.t (76 lines)
A	M	tests/test-narrow-widen-tree.t (28 lines)
A	M	tests/test-narrow-widen.t (355 lines)
A	M	tests/test-narrow.t (358 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	5095		Feb 1 2018, 6:00 PM	★	★
Diff 2	5498		Feb 12 2018, 3:02 PM	★	★
Diff 3	5541	rHGa2a6e724d61a148ebec81722924b1c6e92c2217a	Jan 29 2018, 4:19 PM	★	★

Status	Author	Revision
Closed	durin42	D2196 narrow: add a TODO document
Closed	durin42	D2195 tests: use `hello` not `capabilities` over ssh
Closed	durin42	D2194 narrowwirepeer: add TODO about how we add wireproto args to unbundle :(
Closed	durin42	D2193 narrowwirepeer: rename expandnarrow capability to exp-expandnarrow
Closed	durin42	D2192 narrow: make restrictpatterns a little more idiomatic
Closed	durin42	D2191 narrowspec: consistently use set() to copy sets
Closed	durin42	D2190 narrowspec: document constraints when validating patterns
Closed	durin42	D2189 narrowrevlog: add a TODO around remotefilelog moving to core
Closed	durin42	D2188 narrowrevlog: add what little I can remember about rename filtering
Closed	durin42	D2187 manifest: clean up dirlog() to take a d parameter to avoid shadowing dir()
Closed	durin42	D2186 narrowrevlog: replace AssertionError with ProgrammingError
Closed	durin42	D2185 narrowrevlog: document excludeddir class and friends
Closed	durin42	D2184 narrowrepo: add docstring for narrowpats
Closed	durin42	D2009 narrowtemplates: update to use registrar mechanism
Closed	durin42	D2008 narrow: move from ELLIPSIS_NODE_FLAG to revlog.REVIDX_ELLIPSIS
Closed	durin42	D2007 narrowrepo: make repo requirement include the string 'experimental'
Closed	durin42	D2006 narrow: mark requirement as a constant
Closed	durin42	D2005 narrowbundle2: mark most constants as module-private
Closed	durin42	D2004 narrowbundle2: make constants ALLCAPS to be a bit more readable
Closed	durin42	D1980 narrow: remove old version-checking logic and declare internal
Closed	durin42	D1979 tests: fold narrow treemanifest tests into main test file using testcases
Closed	durin42	D1978 narrow: assume addflagprocessor will always exist on revlog module
Closed	durin42	D1977 narrow: this code should assume REVIDX_FLAGS_ORDER exists
Closed	durin42	D1976 narrow: drop legacy support for getsubsetraw
Closed	durin42	D1975 narrow: remove support for old match modules
Closed	durin42	D1974 narrow: import experimental extension from narrowhg revision cb51d673e9c5

Diff 5541

hgext/narrow/init.py

This file was added.

				# __init__.py - narrowhg extension
				#
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.
				'''create clones which fetch history data for subset of files (EXPERIMENTAL)'''

				from __future__ import absolute_import

				from mercurial import __version__
				if __version__.version < '3.7':
				raise ImportError(
				'narrowhg requires mercurial 3.7 or newer')

				indygregUnsubmitted Done This seems inappropriate. indygreg: This seems inappropriate.
				durin42AuthorUnsubmitted Done Yeah, I noticed this after doing the first big extraction. It's done in D1980, but we can probably trivially fold that here. Do you have a preference? durin42: Yeah, I noticed this after doing the first big extraction. It's done in D1980, but we can…
				try:
				from .__versionnum__ import version
				__version__ = version
				except ImportError:
				pass

				from mercurial import (
				extensions,
				hg,
				localrepo,
				registrar,
				util,
				verify as verifymod,
				)

				from . import (
				narrowbundle2,
				narrowchangegroup,
				narrowcommands,
				narrowcopies,
				narrowdirstate,
				narrowmerge,
				narrowpatch,
				narrowrepo,
				narrowrevlog,
				narrowtemplates,
				narrowwirepeer,
				)

				configtable = {}
				configitem = registrar.configitem(configtable)
				# Narrowhg has support for serving ellipsis nodes (which are used at
				# least by Google's internal server), but that support is pretty
				# fragile and has a lot of problems on real-world repositories that
				# have complex graph topologies. This could probably be corrected, but
				# absent someone needing the full support for ellipsis nodes in
				# repositories with merges, it's unlikely this work will get done. As
				# of this writining in late 2017, all repositories large enough for
				# ellipsis nodes to be a hard requirement also enforce strictly linear
				# history for other scaling reasons.
				configitem('experimental', 'narrowservebrokenellipses',
				default=False,
				alias=[('narrow', 'serveellipses')],
				)

				# Export the commands table for Mercurial to see.
				cmdtable = narrowcommands.table

				localrepo.localrepository._basesupported.add(narrowrepo.requirement)

				def uisetup(ui):
				"""Wraps user-facing mercurial commands with narrow-aware versions."""
				narrowrevlog.setup()
				narrowbundle2.setup()
				narrowmerge.setup()
				narrowtemplates.setup()
				narrowcommands.setup()
				narrowchangegroup.setup()
				narrowwirepeer.uisetup()

				def reposetup(ui, repo):
				"""Wraps local repositories with narrow repo support."""
				if not isinstance(repo, localrepo.localrepository):
				return

				if narrowrepo.requirement in repo.requirements:
				narrowrepo.wraprepo(repo, True)
				narrowcopies.setup(repo)
				narrowdirstate.setup(repo)
				narrowpatch.setup(repo)
				narrowwirepeer.reposetup(repo)

				def _narrowvalidpath(orig, repo, path):
				matcher = getattr(repo, 'narrowmatch', None)
				if matcher is None:
				return orig(repo, path)
				matcher = matcher()
				if matcher.visitdir(path) or matcher(path):
				return orig(repo, path)
				return False

				def _verifierinit(orig, self, repo, matcher=None):
				# The verifier's matcher argument was desgined for narrowhg, so it should
				# be None from core. If another extension passes a matcher (unlikely),
				# we'll have to fail until matchers can be composed more easily.
				assert matcher is None
				matcher = getattr(repo, 'narrowmatch', lambda: None)()
				orig(self, repo, matcher)

				def extsetup(ui):
				if util.safehasattr(verifymod, '_validpath'):
				extensions.wrapfunction(verifymod, '_validpath', _narrowvalidpath)
				else:
				extensions.wrapfunction(verifymod.verifier, '__init__', _verifierinit)
				extensions.wrapfunction(hg, 'postshare', narrowrepo.wrappostshare)
				extensions.wrapfunction(hg, 'copystore', narrowrepo.unsharenarrowspec)

hgext/narrow/narrowbundle2.py

This file was added.

				# narrowbundle2.py - bundle2 extensions for narrow repository support
				#
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				from __future__ import absolute_import

				import collections
				import errno
				import struct

				from mercurial.i18n import _
				from mercurial.node import (
				bin,
				nullid,
				nullrev,
				)
				from mercurial import (
				bundle2,
				changegroup,
				dagutil,
				error,
				exchange,
				extensions,
				repair,
				util,
				wireproto,
				)

				from . import (
				narrowrepo,
				narrowspec,
				)

				narrowcap = 'narrow'
				narrowacl_section = 'narrowhgacl'
				indygregUnsubmitted Not Done This wants an `exp` or `experimental` in its name. Once we start advertising it, there's no going back. So we shouldn't squat on final names until we're sure. indygreg: This wants an `exp` or `experimental` in its name. Once we start advertising it, there's no…
				changespecpart = narrowcap + ':changespec'
				specpart = narrowcap + ':spec'
				specpart_include = 'include'
				specpart_exclude = 'exclude'
				killnodesignal = 'KILL'
				donesignal = 'DONE'
				elidedcsheader = '>20s20s20sl' # cset id, p1, p2, len(text)
				elidedmfheader = '>20s20s20s20sl' # manifest id, p1, p2, link id, len(text)
				csheadersize = struct.calcsize(elidedcsheader)
				indygregUnsubmitted Done I think we'll want to UPPERCASE these since they are constants. indygreg: I think we'll want to UPPERCASE these since they are constants.
				mfheadersize = struct.calcsize(elidedmfheader)

				# When advertising capabilities, always include narrow clone support.
				def getrepocaps_narrow(orig, repo, **kwargs):
				caps = orig(repo, **kwargs)
				caps[narrowcap] = ['v0']
				return caps

				def _computeellipsis(repo, common, heads, known, match, depth=None):
				"""Compute the shape of a narrowed DAG.

				Args:
				repo: The repository we're transferring.
				common: The roots of the DAG range we're transferring.
				May be just [nullid], which means all ancestors of heads.
				heads: The heads of the DAG range we're transferring.
				match: The narrowmatcher that allows us to identify relevant changes.
				depth: If not None, only consider nodes to be full nodes if they are at
				most depth changesets away from one of heads.

				Returns:
				A tuple of (visitnodes, relevant_nodes, ellipsisroots) where:

				visitnodes: The list of nodes (either full or ellipsis) which
				need to be sent to the client.
				relevant_nodes: The set of changelog nodes which change a file inside
				the narrowspec. The client needs these as non-ellipsis nodes.
				ellipsisroots: A dict of {rev: parents} that is used in
				narrowchangegroup to produce ellipsis nodes with the
				correct parents.
				"""
				cl = repo.changelog
				mfl = repo.manifestlog

				cldag = dagutil.revlogdag(cl)
				# dagutil does not like nullid/nullrev
				commonrevs = cldag.internalizeall(common - set([nullid])) \| set([nullrev])
				headsrevs = cldag.internalizeall(heads)
				if depth:
				revdepth = {h: 0 for h in headsrevs}

				ellipsisheads = collections.defaultdict(set)
				ellipsisroots = collections.defaultdict(set)

				def addroot(head, curchange):
				"""Add a root to an ellipsis head, splitting heads with 3 roots."""
				ellipsisroots[head].add(curchange)
				# Recursively split ellipsis heads with 3 roots by finding the
				# roots' youngest common descendant which is an elided merge commit.
				# That descendant takes 2 of the 3 roots as its own, and becomes a
				# root of the head.
				while len(ellipsisroots[head]) > 2:
				child, roots = splithead(head)
				splitroots(head, child, roots)
				head = child # Recurse in case we just added a 3rd root

				def splitroots(head, child, roots):
				ellipsisroots[head].difference_update(roots)
				ellipsisroots[head].add(child)
				ellipsisroots[child].update(roots)
				ellipsisroots[child].discard(child)

				def splithead(head):
				r1, r2, r3 = sorted(ellipsisroots[head])
				for nr1, nr2 in ((r2, r3), (r1, r3), (r1, r2)):
				mid = repo.revs('sort(merge() & %d::%d & %d::%d, -rev)',
				nr1, head, nr2, head)
				for j in mid:
				if j == nr2:
				return nr2, (nr1, nr2)
				if j not in ellipsisroots or len(ellipsisroots[j]) < 2:
				return j, (nr1, nr2)
				raise error.Abort('Failed to split up ellipsis node! head: %d, '
				'roots: %d %d %d' % (head, r1, r2, r3))

				missing = list(cl.findmissingrevs(common=commonrevs, heads=headsrevs))
				visit = reversed(missing)
				relevant_nodes = set()
				visitnodes = map(cl.node, missing)
				required = set(headsrevs) \| known
				indygregUnsubmitted Not Done `map()` returns an iterator on Python 3. indygreg: `map()` returns an iterator on Python 3.
				for rev in visit:
				clrev = cl.changelogrevision(rev)
				ps = cldag.parents(rev)
				if depth is not None:
				curdepth = revdepth[rev]
				for p in ps:
				revdepth[p] = min(curdepth + 1, revdepth.get(p, depth + 1))
				needed = False
				shallow_enough = depth is None or revdepth[rev] <= depth
				if shallow_enough:
				curmf = mfl[clrev.manifest].read()
				if ps:
				# We choose to not trust the changed files list in
				# changesets because it's not always correct. TODO: could
				# we trust it for the non-merge case?
				p1mf = mfl[cl.changelogrevision(ps[0]).manifest].read()
				indygregUnsubmitted Not Done No, we can't. There were bugs in e.g. rebase that caused changed files to be dropped from the files list in the changeset. Walking manifests is the only way to be sure you are getting all changes. indygreg: No, we can't. There were bugs in e.g. rebase that caused changed files to be dropped from the…
				needed = any(match(f) for f in curmf.diff(p1mf).iterkeys())
				if not needed and len(ps) > 1:
				# For merge changes, the list of changed files is not
				# helpful, since we need to emit the merge if a file
				# in the narrow spec has changed on either side of the
				# merge. As a result, we do a manifest diff to check.
				p2mf = mfl[cl.changelogrevision(ps[1]).manifest].read()
				needed = any(match(f) for f in curmf.diff(p2mf).iterkeys())
				else:
				indygregUnsubmitted Not Done I'm not an expert on the manifest APIs. There may be a more optimal way to implement this... indygreg: I'm not an expert on the manifest APIs. There //may// be a more optimal way to implement this...
				durin42AuthorUnsubmitted Not Done I wrote lazymanifest, and this was the best adgar and I could come up with... durin42: I wrote lazymanifest, and this was the best adgar and I could come up with...
				# For a root node, we need to include the node if any
				# files in the node match the narrowspec.
				needed = any(match(f) for f in curmf)

				if needed:
				for head in ellipsisheads[rev]:
				addroot(head, rev)
				for p in ps:
				required.add(p)
				relevant_nodes.add(cl.node(rev))
				else:
				if not ps:
				ps = [nullrev]
				if rev in required:
				for head in ellipsisheads[rev]:
				addroot(head, rev)
				for p in ps:
				ellipsisheads[p].add(rev)
				else:
				for p in ps:
				ellipsisheads[p] \|= ellipsisheads[rev]

				# add common changesets as roots of their reachable ellipsis heads
				for c in commonrevs:
				for head in ellipsisheads[c]:
				addroot(head, c)
				return visitnodes, relevant_nodes, ellipsisroots

				def _packellipsischangegroup(repo, common, match, relevant_nodes,
				ellipsisroots, visitnodes, depth, source, version):
				if version in ('01', '02'):
				raise error.Abort(
				'ellipsis nodes require at least cg3 on client and server, '
				'but negotiated version %s' % version)
				# We wrap cg1packer.revchunk, using a side channel to pass
				# relevant_nodes into that area. Then if linknode isn't in the
				# set, we know we have an ellipsis node and we should defer
				# sending that node's data. We override close() to detect
				# pending ellipsis nodes and flush them.
				packer = changegroup.getbundler(version, repo)
				# Let the packer have access to the narrow matcher so it can
				# omit filelogs and dirlogs as needed
				packer._narrow_matcher = lambda : match
				# Give the packer the list of nodes which should not be
				# ellipsis nodes. We store this rather than the set of nodes
				# that should be an ellipsis because for very large histories
				# we expect this to be significantly smaller.
				packer.full_nodes = relevant_nodes
				# Maps ellipsis revs to their roots at the changelog level.
				packer.precomputed_ellipsis = ellipsisroots
				# Maps CL revs to per-revlog revisions. Cleared in close() at
				# the end of each group.
				packer.clrev_to_localrev = {}
				packer.next_clrev_to_localrev = {}
				# Maps changelog nodes to changelog revs. Filled in once
				# during changelog stage and then left unmodified.
				packer.clnode_to_rev = {}
				packer.changelog_done = False
				# If true, informs the packer that it is serving shallow content and might
				# need to pack file contents not introduced by the changes being packed.
				packer.is_shallow = depth is not None

				return packer.generate(common, visitnodes, False, source)

				# Serve a changegroup for a client with a narrow clone.
				def getbundlechangegrouppart_narrow(bundler, repo, source,
				bundlecaps=None, b2caps=None, heads=None,
				common=None, **kwargs):
				cgversions = b2caps.get('changegroup')
				getcgkwargs = {}
				if cgversions: # 3.1 and 3.2 ship with an empty value
				cgversions = [v for v in cgversions
				indygregUnsubmitted Not Done If this is referring to Mercurial versions, then this code is likely not needed, as a client with narrow support is newer than 3.1 or 3.2. indygreg: If this is referring to Mercurial versions, then this code is likely not needed, as a client…
				durin42AuthorUnsubmitted Not Done Hmm, I think this might be talking about client versions in the server code. @martinvonz do you remember what's going on here? If not I can dig more. durin42: Hmm, I think this might be talking about client versions in the server code. @martinvonz do you…
				martinvonzUnsubmitted Not Done Copied from here: https://www.mercurial-scm.org/repo/hg/file/5cfdf6137af8/mercurial/exchange.py#l1811. I'd say leave it since it's harmless and it makes the import simpler. We can drop it after, but that's probably also unnecessary, since the idea is to move this into core and then it will be naturally lost anyway (because it's already in the destination function). martinvonz: Copied from here: https://www.mercurial-scm.org/repo/hg/file/5cfdf6137af8/mercurial/exchange.
				if v in changegroup.supportedoutgoingversions(repo)]
				if not cgversions:
				raise ValueError(_('no common changegroup version'))
				version = getcgkwargs['version'] = max(cgversions)
				else:
				raise ValueError(_("server does not advertise changegroup version,"
				" can't negotiate support for ellipsis nodes"))

				include = sorted(filter(bool, kwargs.get('includepats', [])))
				exclude = sorted(filter(bool, kwargs.get('excludepats', [])))
				newmatch = narrowspec.match(repo.root, include=include, exclude=exclude)
				if not repo.ui.configbool("experimental", "narrowservebrokenellipses"):
				outgoing = exchange._computeoutgoing(repo, heads, common)
				if not outgoing.missing:
				return
				if util.safehasattr(changegroup, 'getsubsetraw'):
				# getsubsetraw was replaced with makestream in hg in 92f1e2be8ab6
				# (2017/09/10).
				packer = changegroup.getbundler(version, repo)
				packer._narrow_matcher = lambda : newmatch
				cg = changegroup.getsubsetraw(repo, outgoing, packer, source)
				else:
				def wrappedgetbundler(orig, args, *kwargs):
				bundler = orig(args, *kwargs)
				bundler._narrow_matcher = lambda : newmatch
				return bundler
				with extensions.wrappedfunction(changegroup, 'getbundler',
				wrappedgetbundler):
				cg = changegroup.makestream(repo, outgoing, version, source)
				part = bundler.newpart('changegroup', data=cg)
				part.addparam('version', version)
				if 'treemanifest' in repo.requirements:
				part.addparam('treemanifest', '1')

				if include or exclude:
				narrowspecpart = bundler.newpart(specpart)
				if include:
				indygregUnsubmitted Not Done Would it make sense to send the narrow spec before the changegroup part? indygreg: Would it make sense to send the narrow spec before the changegroup part?
				narrowspecpart.addparam(
				specpart_include, '\n'.join(include), mandatory=True)
				if exclude:
				narrowspecpart.addparam(
				specpart_exclude, '\n'.join(exclude), mandatory=True)

				indygregUnsubmitted Not Done Part parameters are effectively header data for bundle2 parts. As such, they are read and written as atomic units. And they are URL quoted for transport safety. Since narrow specs are unbound in size, I think they should be transferred as part of the part payload, not in parameters. This allows for streaming and doesn't add overhead for encoding/decoding their content. This is obviously a BC change to the wire protocol and bundle storage. indygreg: Part parameters are effectively header data for bundle2 parts. As such, they are read and…
				return

				depth = kwargs.get('depth', None)
				if depth is not None:
				depth = int(depth)
				if depth < 1:
				raise error.Abort(_('depth must be positive, got %d') % depth)

				indygregUnsubmitted Not Done Should this code be above the `if narrowservebrokenellipses` block above? It seems like it is doing generic argument validation and that validation should be near the top of the function. indygreg: Should this code be above the `if narrowservebrokenellipses` block above? It seems like it is…
				heads = set(heads or repo.heads())
				common = set(common or [nullid])
				oldinclude = sorted(filter(bool, kwargs.get('oldincludepats', [])))
				oldexclude = sorted(filter(bool, kwargs.get('oldexcludepats', [])))
				known = {bin(n) for n in kwargs.get('known', [])}
				if known and (oldinclude != include or oldexclude != exclude):
				# Steps:
				# 1. Send kill for "$known & ::common"
				#
				# 2. Send changegroup for ::common
				#
				# 3. Proceed.
				#
				# In the future, we can send kills for only the specific
				# nodes we know should go away or change shape, and then
				# send a data stream that tells the client something like this:
				#
				# a) apply this changegroup
				# b) apply nodes XXX, YYY, ZZZ that you already have
				# c) goto a
				#
				# until they've built up the full new state.
				# Convert to revnums and intersect with "common". The client should
				# have made it a subset of "common" already, but let's be safe.
				known = set(repo.revs("%ln & ::%ln", known, common))
				# TODO: we could send only roots() of this set, and the
				# list of nodes in common, and the client could work out
				# what to strip, instead of us explicitly sending every
				# single node.
				deadrevs = known
				def genkills():
				for r in deadrevs:
				yield killnodesignal
				yield repo.changelog.node(r)
				yield donesignal
				bundler.newpart(changespecpart, data=genkills())
				newvisit, newfull, newellipsis = _computeellipsis(
				repo, set(), common, known, newmatch)
				if newvisit:
				cg = _packellipsischangegroup(
				repo, common, newmatch, newfull, newellipsis,
				newvisit, depth, source, version)
				part = bundler.newpart('changegroup', data=cg)
				part.addparam('version', version)
				if 'treemanifest' in repo.requirements:
				part.addparam('treemanifest', '1')

				visitnodes, relevant_nodes, ellipsisroots = _computeellipsis(
				repo, common, heads, set(), newmatch, depth=depth)

				repo.ui.debug('Found %d relevant revs\n' % len(relevant_nodes))
				if visitnodes:
				cg = _packellipsischangegroup(
				repo, common, newmatch, relevant_nodes, ellipsisroots,
				visitnodes, depth, source, version)
				part = bundler.newpart('changegroup', data=cg)
				part.addparam('version', version)
				if 'treemanifest' in repo.requirements:
				part.addparam('treemanifest', '1')

				def applyacl_narrow(repo, kwargs):
				username = repo.ui.shortuser(repo.ui.username())
				indygregUnsubmitted Not Done We may want to remove this feature for the time being, as it is very Google specific. I'm happy to provide a mechanism for extensions to hook in here to enforce ACLs. But there's nothing in core that deals with path-based access control and I think things should remain that way in order to keep the code simple. For the time being at least. I do acknowledge value in the feature and it is something I'd like to see in core some day. I just think we should stabilize the partial clone functionality first before we start forcing people to think about path-based access control. indygreg: We may want to remove this feature for the time being, as it is very Google specific. I'm happy…
				idlsoftUnsubmitted Not Done I've contributed the ACL code originally. It's a relatively trivial piece of code, at least compared to the rest of the codebase, and it is called conditionally. I hope it can stay, my company is about to migrate from git to hg because of it. idlsoft: I've contributed the ACL code originally. It's a relatively trivial piece of code, at least…
				user_includes = repo.ui.configlist(
				narrowacl_section, username + '.includes',
				repo.ui.configlist(narrowacl_section, 'default.includes'))
				user_excludes = repo.ui.configlist(
				narrowacl_section, username + '.excludes',
				repo.ui.configlist(narrowacl_section, 'default.excludes'))
				if not user_includes:
				raise error.Abort(_("{} configuration for user {} is empty")
				.format(narrowacl_section, username))

				user_includes = [
				'path:.' if p == '*' else 'path:' + p for p in user_includes]
				user_excludes = [
				'path:.' if p == '*' else 'path:' + p for p in user_excludes]

				req_includes = set(kwargs.get('includepats', []))
				req_excludes = set(kwargs.get('excludepats', []))

				invalid_includes = []
				req_includes, req_excludes = narrowspec.restrictpatterns(
				req_includes, req_excludes,
				user_includes, user_excludes, invalid_includes)

				if invalid_includes:
				raise error.Abort(
				_("The following includes are not accessible for {}: {}")
				.format(username, invalid_includes))

				new_args = {}
				new_args.update(kwargs)
				new_args['includepats'] = req_includes
				if req_excludes:
				new_args['excludepats'] = req_excludes
				return new_args

				@bundle2.parthandler(specpart, (specpart_include, specpart_exclude))
				def _handlechangespec_2(op, inpart):
				includepats = set(inpart.params.get(specpart_include, '').splitlines())
				excludepats = set(inpart.params.get(specpart_exclude, '').splitlines())
				narrowspec.save(op.repo, includepats, excludepats)
				if not narrowrepo.requirement in op.repo.requirements:
				op.repo.requirements.add(narrowrepo.requirement)
				op.repo._writerequirements()
				op.repo.invalidate(clearfilecache=True)
				indygregUnsubmitted Not Done Silently adding a repo requirement during a pull or unbundle operation doesn't feel right to me. At least not at this juncture. For at least as long as the feature is experimental, I think we should refuse to apply (or fetch) a narrow bundle if the repo isn't already narrow. We can provide an `hg debug` command to add the requirement to make people's lives easier. indygreg:* Silently adding a repo requirement during a pull or unbundle operation doesn't feel right to me.
				idlsoftUnsubmitted Not Done This was added to support seamless clone from an ACLed repo. The user doesn't know ahead of time about server-side narrowing. idlsoft: This was added to support seamless clone from an ACLed repo. The user doesn't know ahead of…

				@bundle2.parthandler(changespecpart)
				def _handlechangespec(op, inpart):
				repo = op.repo
				cl = repo.changelog

				# changesets which need to be stripped entirely. either they're no longer
				# needed in the new narrow spec, or the server is sending a replacement
				# in the changegroup part.
				clkills = set()

				# A changespec part contains all the updates to ellipsis nodes
				# that will happen as a result of widening or narrowing a
				# repo. All the changes that this block encounters are ellipsis
				# nodes or flags to kill an existing ellipsis.
				chunksignal = changegroup.readexactly(inpart, 4)
				while chunksignal != donesignal:
				if chunksignal == killnodesignal:
				# a node used to be an ellipsis but isn't anymore
				ck = changegroup.readexactly(inpart, 20)
				if cl.hasnode(ck):
				clkills.add(ck)
				else:
				raise error.Abort(
				_('unexpected changespec node chunk type: %s') % chunksignal)
				chunksignal = changegroup.readexactly(inpart, 4)

				if clkills:
				# preserve bookmarks that repair.strip() would otherwise strip
				bmstore = repo._bookmarks
				class dummybmstore(dict):
				def applychanges(self, repo, tr, changes):
				pass
				def recordchange(self, tr): # legacy version
				pass
				repo._bookmarks = dummybmstore()
				indygregUnsubmitted Not Done This compatibility with older Mercurial versions can be removed. indygreg: This compatibility with older Mercurial versions can be removed.
				chgrpfile = repair.strip(op.ui, repo, list(clkills), backup=True,
				topic='widen')
				repo._bookmarks = bmstore
				indygregUnsubmitted Not Done Doing strip-based operations as part of bundle application breaks read-only consistency of the repository during those operations. IMO a blocking issue to eventually making this feature non-experimental will be either a refactored store (so we don't need to perform strips to backfill data) or reader locks on the store (to lock out readers when operations such as strip that make the store inconsistent are being performed). Either of those will require a new repo requirement. That requirement is centered around the store and not narrow/shallow/partial clone support. So it needs to be considered a separate feature from all this clone feature work. indygreg: Doing strip-based operations as part of bundle application breaks read-only consistency of the…
				if chgrpfile:
				# presence of _widen_bundle attribute activates widen handler later
				op._widen_bundle = chgrpfile
				# Set the new narrowspec if we're widening. The setnewnarrowpats() method
				# will currently always be there when using the core+narrowhg server, but
				# other servers may include a changespec part even when not widening (e.g.
				# because we're deepening a shallow repo).
				if util.safehasattr(repo, 'setnewnarrowpats'):
				repo.setnewnarrowpats()

				def handlechangegroup_widen(op, inpart):
				"""Changegroup exchange handler which restores temporarily-stripped nodes"""
				# We saved a bundle with stripped node data we must now restore.
				# This approach is based on mercurial/repair.py@6ee26a53c111.
				repo = op.repo
				ui = op.ui

				chgrpfile = op._widen_bundle
				del op._widen_bundle
				vfs = repo.vfs

				ui.note(_("adding branch\n"))
				f = vfs.open(chgrpfile, "rb")
				try:
				gen = exchange.readbundle(ui, f, chgrpfile, vfs)
				indygregUnsubmitted Not Done This should be using a context manager instead of `try..finally`. indygreg: This should be using a context manager instead of `try..finally`.
				if not ui.verbose:
				# silence internal shuffling chatter
				ui.pushbuffer()
				if isinstance(gen, bundle2.unbundle20):
				with repo.transaction('strip') as tr:
				bundle2.processbundle(repo, gen, lambda: tr)
				else:
				gen.apply(repo, 'strip', 'bundle:' + vfs.join(chgrpfile), True)
				if not ui.verbose:
				indygregUnsubmitted Not Done I think `strip` should be something else to denote this from an actual `strip` operation. indygreg: I think `strip` should be something else to denote this from an actual `strip` operation.
				ui.popbuffer()
				finally:
				f.close()

				# remove undo files
				for undovfs, undofile in repo.undofiles():
				try:
				undovfs.unlink(undofile)
				except OSError as e:
				if e.errno != errno.ENOENT:
				ui.warn(_('error removing %s: %s\n') %
				(undovfs.join(undofile), str(e)))

				indygregUnsubmitted Not Done This reinvents `vfs.tryunlink()`. Although you'll still want to catch the `OSError` from `tryunlink()` to turn it into a warning. But at least you can not worry about `ENOENT`. indygreg: This reinvents `vfs.tryunlink()`. Although you'll still want to catch the `OSError` from…
				# Remove partial backup only if there were no exceptions
				vfs.unlink(chgrpfile)

				def setup():
				"""Enable narrow repo support in bundle2-related extension points."""
				extensions.wrapfunction(bundle2, 'getrepocaps', getrepocaps_narrow)

				wireproto.gboptsmap['narrow'] = 'boolean'
				wireproto.gboptsmap['depth'] = 'plain'
				wireproto.gboptsmap['oldincludepats'] = 'csv'
				wireproto.gboptsmap['oldexcludepats'] = 'csv'
				wireproto.gboptsmap['includepats'] = 'csv'
				wireproto.gboptsmap['excludepats'] = 'csv'
				wireproto.gboptsmap['known'] = 'csv'

				indygregUnsubmitted Not Done I'm not super keen on tacking all these arguments onto the existing `getbundle` wire protocol command. Especially without experimental labels. I'd rather we invent new wire protocol commands when we add major new features like this. That's something I hope to address in the new wire protocol. I'm fine with this for now. Just as long as the client does some form of capabilities screening to know which set of arguments the server-side wire protocol command accepts. As long as we fix this during the `4.6` release period, we should be fine. Once version 2 of the wire protocol is done, I'll likely establish a new `getbundle` like command that is version 2 only and will remove narrow support from the existing `getbundle` command. indygreg: I'm not super keen on tacking all these arguments onto the existing `getbundle` wire protocol…
				# Extend changegroup serving to handle requests from narrow clients.
				origcgfn = exchange.getbundle2partsmapping['changegroup']
				def wrappedcgfn(args, *kwargs):
				repo = args[1]
				if repo.ui.has_section(narrowacl_section):
				getbundlechangegrouppart_narrow(
				args, *applyacl_narrow(repo, kwargs))
				elif kwargs.get('narrow', False):
				indygregUnsubmitted Not Done More ACL code that we may remove from core. indygreg: More ACL code that we may remove from core.
				getbundlechangegrouppart_narrow(args, *kwargs)
				else:
				origcgfn(args, *kwargs)
				exchange.getbundle2partsmapping['changegroup'] = wrappedcgfn

				# Extend changegroup receiver so client can fixup after widen requests.
				origcghandler = bundle2.parthandlermapping['changegroup']
				def wrappedcghandler(op, inpart):
				origcghandler(op, inpart)
				if util.safehasattr(op, '_widen_bundle'):
				handlechangegroup_widen(op, inpart)
				wrappedcghandler.params = origcghandler.params
				bundle2.parthandlermapping['changegroup'] = wrappedcghandler

hgext/narrow/narrowchangegroup.py

This file was added.

				# narrowchangegroup.py - narrow clone changegroup creation and consumption
				#
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				from __future__ import absolute_import

				from mercurial.i18n import _
				from mercurial import (
				changegroup,
				error,
				extensions,
				manifest,
				mdiff,
				node,
				util,
				)

				from . import (
				narrowrepo,
				narrowrevlog,
				)

				def setup():

				def supportedoutgoingversions(orig, repo):
				versions = orig(repo)
				if narrowrepo.requirement in repo.requirements:
				versions.discard('01')
				versions.discard('02')
				return versions

				extensions.wrapfunction(changegroup, 'supportedoutgoingversions',
				supportedoutgoingversions)

				def prune(orig, self, revlog, missing, commonrevs):
				if isinstance(revlog, manifest.manifestrevlog):
				matcher = getattr(self._repo, 'narrowmatch',
				getattr(self, '_narrow_matcher', None))
				if (matcher is not None and
				not matcher().visitdir(revlog._dir[:-1] or '.')):
				return []
				return orig(self, revlog, missing, commonrevs)

				extensions.wrapfunction(changegroup.cg1packer, 'prune', prune)

				indygregUnsubmitted Not Done Might want a comment here that the wrapped method will get inherited by later versions of the changegroup packer. It feels weird to see this definition on `cg1packer` since code above removes support for changegroup versions `01` and `02`. indygreg: Might want a comment here that the wrapped method will get inherited by later versions of the…
				def generatefiles(orig, self, changedfiles, linknodes, commonrevs,
				source):
				matcher = getattr(self._repo, 'narrowmatch',
				getattr(self, '_narrow_matcher', None))
				if matcher is not None:
				narrowmatch = matcher()
				changedfiles = filter(narrowmatch, changedfiles)
				if getattr(self, 'is_shallow', False):
				# See comment in generate() for why this sadness is a thing.
				mfdicts = self._mfdicts
				del self._mfdicts
				# In a shallow clone, the linknodes callback needs to also include
				# those file nodes that are in the manifests we sent but weren't
				# introduced by those manifests.
				commonctxs = [self._repo[c] for c in commonrevs]
				oldlinknodes = linknodes
				clrev = self._repo.changelog.rev
				def linknodes(flog, fname):
				for c in commonctxs:
				try:
				fnode = c.filenode(fname)
				self.clrev_to_localrev[c.rev()] = flog.rev(fnode)
				except error.ManifestLookupError:
				pass
				links = oldlinknodes(flog, fname)
				if len(links) != len(mfdicts):
				for mf, lr in mfdicts:
				fnode = mf.get(fname, None)
				if fnode in links:
				links[fnode] = min(links[fnode], lr, key=clrev)
				elif fnode:
				links[fnode] = lr
				return links
				return orig(self, changedfiles, linknodes, commonrevs, source)
				indygregUnsubmitted Not Done I don't fully understand linknodes. This should be scrutinized by someone who does, especially since there are likely performance concerns here. indygreg: I don't fully understand linknodes. This should be scrutinized by someone who does, especially…
				extensions.wrapfunction(
				changegroup.cg1packer, 'generatefiles', generatefiles)

				def ellipsisdata(packer, rev, revlog, p1, p2, data, linknode):
				n = revlog.node(rev)
				p1n, p2n = revlog.node(p1), revlog.node(p2)
				flags = revlog.flags(rev)
				flags \|= narrowrevlog.ELLIPSIS_NODE_FLAG
				meta = packer.builddeltaheader(
				n, p1n, p2n, node.nullid, linknode, flags)
				# TODO: try and actually send deltas for ellipsis data blocks
				diffheader = mdiff.trivialdiffheader(len(data))
				l = len(meta) + len(diffheader) + len(data)
				return ''.join((changegroup.chunkheader(l),
				meta,
				diffheader,
				data))

				def close(orig, self):
				getattr(self, 'clrev_to_localrev', {}).clear()
				if getattr(self, 'next_clrev_to_localrev', {}):
				self.clrev_to_localrev = self.next_clrev_to_localrev
				del self.next_clrev_to_localrev
				self.changelog_done = True
				indygregUnsubmitted Not Done I'm not a fan of `getattr()` and `del` here. Is there no way to always set these attributes on instance creation? (One solution is to move this code into `changegroup.py`.) indygreg: I'm not a fan of `getattr()` and `del` here. Is there no way to always set these attributes on…
				return orig(self)
				extensions.wrapfunction(changegroup.cg1packer, 'close', close)

				# In a perfect world, we'd generate better ellipsis-ified graphs
				# for non-changelog revlogs. In practice, we haven't started doing
				# that yet, so the resulting DAGs for the manifestlog and filelogs
				# are actually full of bogus parentage on all the ellipsis
				# nodes. This has the side effect that, while the contents are
				# correct, the individual DAGs might be completely out of whack in
				# a case like 882681bc3166 and its ancestors (back about 10
				# revisions or so) in the main hg repo.
				#
				# The one invariant we know holds is that the new (potentially
				# bogus) DAG shape will be valid if we order the nodes in the
				# order that they're introduced in dramatis personae by the
				# changelog, so what we do is we sort the non-changelog histories
				# by the order in which they are used by the changelog.
				def _sortgroup(orig, self, revlog, nodelist, lookup):
				if not util.safehasattr(self, 'full_nodes') or not self.clnode_to_rev:
				return orig(self, revlog, nodelist, lookup)
				key = lambda n: self.clnode_to_rev[lookup(n)]
				return [revlog.rev(n) for n in sorted(nodelist, key=key)]

				extensions.wrapfunction(changegroup.cg1packer, '_sortgroup', _sortgroup)

				def generate(orig, self, commonrevs, clnodes, fastpathlinkrev, source):
				'''yield a sequence of changegroup chunks (strings)'''
				# Note: other than delegating to orig, the only deviation in
				# logic from normal hg's generate is marked with BEGIN/END
				# NARROW HACK.
				if not util.safehasattr(self, 'full_nodes'):
				# not sending a narrow bundle
				for x in orig(self, commonrevs, clnodes, fastpathlinkrev, source):
				yield x
				return

				repo = self._repo
				cl = repo.changelog
				mfl = repo.manifestlog
				mfrevlog = mfl._revlog

				clrevorder = {}
				mfs = {} # needed manifests
				fnodes = {} # needed file nodes
				changedfiles = set()

				indygregUnsubmitted Not Done Eek - this fully reimplements `generate()`?! This is probably the hackiest part of this code I've seen so far. I know I've said I want a lot of things moved into core. But considering the complexity of this duplicate code, I think this belongs as front as possible in the queue. indygreg: Eek - this fully reimplements `generate()`?! This is probably the hackiest part of this code…
				# Callback for the changelog, used to collect changed files and manifest
				# nodes.
				# Returns the linkrev node (identity in the changelog case).
				def lookupcl(x):
				c = cl.read(x)
				clrevorder[x] = len(clrevorder)
				indygregUnsubmitted Not Done This should change to `cl.changelogrevision(x)` so we return an object with named attributes instead of a tuple. That will make e.g. `c[0]` change to `cl.manifest`. `cl.read()` is just a proxy around `changelogrevision()` anyway. If the changegroup code uses `read()`, it should be changed as well. indygreg: This should change to `cl.changelogrevision(x)` so we return an object with named attributes…
				# BEGIN NARROW HACK
				#
				# Only update mfs if x is going to be sent. Otherwise we
				# end up with bogus linkrevs specified for manifests and
				# we skip some manifest nodes that we should otherwise
				# have sent.
				if x in self.full_nodes or cl.rev(x) in self.precomputed_ellipsis:
				n = c[0]
				indygregUnsubmitted Not Done Nit: `cl.rev()` is called here and inside the conditional. My recollection is this loop is pretty tight, so aliasing `clrev = cl.rev` and saving the result to avoid another call might have an impact. indygreg: Nit: `cl.rev()` is called here and inside the conditional. My recollection is this loop is…
				# record the first changeset introducing this manifest version
				mfs.setdefault(n, x)
				# Set this narrow-specific dict so we have the lowest manifest
				# revnum to look up for this cl revnum. (Part of mapping
				# changelog ellipsis parents to manifest ellipsis parents)
				self.next_clrev_to_localrev.setdefault(cl.rev(x),
				mfrevlog.rev(n))
				# We can't trust the changed files list in the changeset if the
				# client requested a shallow clone.
				if self.is_shallow:
				changedfiles.update(mfl[c[0]].read().keys())
				else:
				changedfiles.update(c[3])
				# END NARROW HACK
				# Record a complete list of potentially-changed files in
				# this manifest.
				return x

				self._verbosenote(_('uncompressed size of bundle content:\n'))
				size = 0
				for chunk in self.group(clnodes, cl, lookupcl, units=_('changesets')):
				size += len(chunk)
				yield chunk
				self._verbosenote(_('%8.i (changelog)\n') % size)

				# We need to make sure that the linkrev in the changegroup refers to
				# the first changeset that introduced the manifest or file revision.
				# The fastpath is usually safer than the slowpath, because the filelogs
				# are walked in revlog order.
				#
				# When taking the slowpath with reorder=None and the manifest revlog
				# uses generaldelta, the manifest may be walked in the "wrong" order.
				# Without 'clrevorder', we would get an incorrect linkrev (see fix in
				# cc0ff93d0c0c).
				#
				# When taking the fastpath, we are only vulnerable to reordering
				# of the changelog itself. The changelog never uses generaldelta, so
				# it is only reordered when reorder=True. To handle this case, we
				# simply take the slowpath, which already has the 'clrevorder' logic.
				# This was also fixed in cc0ff93d0c0c.
				fastpathlinkrev = fastpathlinkrev and not self._reorder
				# Treemanifests don't work correctly with fastpathlinkrev
				# either, because we don't discover which directory nodes to
				# send along with files. This could probably be fixed.
				fastpathlinkrev = fastpathlinkrev and (
				'treemanifest' not in repo.requirements)
				# Shallow clones also don't work correctly with fastpathlinkrev
				# because file nodes may need to be sent for a manifest even if they
				# weren't introduced by that manifest.
				fastpathlinkrev = fastpathlinkrev and not self.is_shallow

				moreargs = []
				if self.generatemanifests.func_code.co_argcount == 7:
				# The source argument was added to generatemanifests in hg in
				# 75cc1f1e11f2 (2017/09/11).
				moreargs.append(source)
				for chunk in self.generatemanifests(commonrevs, clrevorder,
				fastpathlinkrev, mfs, fnodes, *moreargs):
				yield chunk
				# BEGIN NARROW HACK
				mfdicts = None
				if self.is_shallow:
				mfdicts = [(self._repo.manifestlog[n].read(), lr)
				for (n, lr) in mfs.iteritems()]
				# END NARROW HACK
				mfs.clear()
				clrevs = set(cl.rev(x) for x in clnodes)

				if not fastpathlinkrev:
				def linknodes(unused, fname):
				return fnodes.get(fname, {})
				else:
				cln = cl.node
				def linknodes(filerevlog, fname):
				llr = filerevlog.linkrev
				fln = filerevlog.node
				revs = ((r, llr(r)) for r in filerevlog)
				return dict((fln(r), cln(lr)) for r, lr in revs if lr in clrevs)

				# BEGIN NARROW HACK
				#
				# We need to pass the mfdicts variable down into
				# generatefiles(), but more than one command might have
				# wrapped generatefiles so we can't modify the function
				# signature. Instead, we pass the data to ourselves using an
				# instance attribute. I'm sorry.
				self._mfdicts = mfdicts
				# END NARROW HACK
				for chunk in self.generatefiles(changedfiles, linknodes, commonrevs,
				source):
				yield chunk

				yield self.close()

				if clnodes:
				repo.hook('outgoing', node=node.hex(clnodes[0]), source=source)
				extensions.wrapfunction(changegroup.cg1packer, 'generate', generate)

				def revchunk(orig, self, revlog, rev, prev, linknode):
				if not util.safehasattr(self, 'full_nodes'):
				# not sending a narrow changegroup
				for x in orig(self, revlog, rev, prev, linknode):
				yield x
				return
				# build up some mapping information that's useful later. See
				# the local() nested function below.
				if not self.changelog_done:
				self.clnode_to_rev[linknode] = rev
				linkrev = rev
				self.clrev_to_localrev[linkrev] = rev
				else:
				linkrev = self.clnode_to_rev[linknode]
				self.clrev_to_localrev[linkrev] = rev
				# This is a node to send in full, because the changeset it
				# corresponds to was a full changeset.
				if linknode in self.full_nodes:
				for x in orig(self, revlog, rev, prev, linknode):
				yield x
				return
				# At this point, a node can either be one we should skip or an
				# ellipsis. If it's not an ellipsis, bail immediately.
				if linkrev not in self.precomputed_ellipsis:
				return
				linkparents = self.precomputed_ellipsis[linkrev]
				def local(clrev):
				"""Turn a changelog revnum into a local revnum.

				The ellipsis dag is stored as revnums on the changelog,
				but when we're producing ellipsis entries for
				non-changelog revlogs, we need to turn those numbers into
				something local. This does that for us, and during the
				changelog sending phase will also expand the stored
				mappings as needed.
				"""
				if clrev == node.nullrev:
				return node.nullrev
				if not self.changelog_done:
				# If we're doing the changelog, it's possible that we
				# have a parent that is already on the client, and we
				# need to store some extra mapping information so that
				# our contained ellipsis nodes will be able to resolve
				# their parents.
				if clrev not in self.clrev_to_localrev:
				clnode = revlog.node(clrev)
				self.clnode_to_rev[clnode] = clrev
				return clrev
				# Walk the ellipsis-ized changelog breadth-first looking for a
				# change that has been linked from the current revlog.
				#
				# For a flat manifest revlog only a single step should be necessary
				# as all relevant changelog entries are relevant to the flat
				# manifest.
				#
				# For a filelog or tree manifest dirlog however not every changelog
				# entry will have been relevant, so we need to skip some changelog
				# nodes even after ellipsis-izing.
				walk = [clrev]
				indygregUnsubmitted Not Done This code is rather difficult to reason about. I'm inclined to defer on a comprehensive review until this code is moved to core. It isn't helping that this file reinvents so much code already in core and it is difficult to reason about how everything plays together. Although the inline comments noting the narrow changes are very useful and appreciated. indygreg: This code is rather difficult to reason about. I'm inclined to defer on a comprehensive review…
				while walk:
				p = walk[0]
				walk = walk[1:]
				if p in self.clrev_to_localrev:
				return self.clrev_to_localrev[p]
				elif p in self.full_nodes:
				walk.extend([pp for pp in self._repo.changelog.parentrevs(p)
				if pp != node.nullrev])
				elif p in self.precomputed_ellipsis:
				walk.extend([pp for pp in self.precomputed_ellipsis[p]
				if pp != node.nullrev])
				else:
				# In this case, we've got an ellipsis with parents
				# outside the current bundle (likely an
				# incremental pull). We "know" that we can use the
				# value of this same revlog at whatever revision
				# is pointed to by linknode. "Know" is in scare
				# quotes because I haven't done enough examination
				# of edge cases to convince myself this is really
				# a fact - it works for all the (admittedly
				# thorough) cases in our testsuite, but I would be
				# somewhat unsurprised to find a case in the wild
				# where this breaks down a bit. That said, I don't
				# know if it would hurt anything.
				for i in xrange(rev, 0, -1):
				if revlog.linkrev(i) == clrev:
				return i
				# We failed to resolve a parent for this node, so
				# we crash the changegroup construction.
				raise error.Abort(
				'unable to resolve parent while packing %r %r'
				' for changeset %r' % (revlog.indexfile, rev, clrev))
				return node.nullrev

				if not linkparents or (
				revlog.parentrevs(rev) == (node.nullrev, node.nullrev)):
				p1, p2 = node.nullrev, node.nullrev
				elif len(linkparents) == 1:
				p1, = sorted(local(p) for p in linkparents)
				p2 = node.nullrev
				else:
				p1, p2 = sorted(local(p) for p in linkparents)
				yield ellipsisdata(
				self, rev, revlog, p1, p2, revlog.revision(rev), linknode)
				extensions.wrapfunction(changegroup.cg1packer, 'revchunk', revchunk)

				def deltaparent(orig, self, revlog, rev, p1, p2, prev):
				if util.safehasattr(self, 'full_nodes'):
				# TODO: send better deltas when in narrow mode.
				#
				# changegroup.group() loops over revisions to send,
				# including revisions we'll skip. What this means is that
				# `prev` will be a potentially useless delta base for all
				# ellipsis nodes, as the client likely won't have it. In
				# the future we should do bookkeeping about which nodes
				# have been sent to the client, and try to be
				# significantly smarter about delta bases. This is
				# slightly tricky because this same code has to work for
				# all revlogs, and we don't have the linkrev/linknode here.
				return p1
				return orig(self, revlog, rev, p1, p2, prev)
				extensions.wrapfunction(changegroup.cg2packer, 'deltaparent', deltaparent)

hgext/narrow/narrowcommands.py

This file was added.

				# narrowcommands.py - command modifications for narrowhg extension
				#
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.
				from __future__ import absolute_import

				import itertools

				from mercurial.i18n import _
				from mercurial import (
				cmdutil,
				commands,
				discovery,
				error,
				exchange,
				extensions,
				hg,
				merge,
				node,
				registrar,
				repair,
				repoview,
				util,
				)

				from . import (
				narrowbundle2,
				narrowrepo,
				narrowspec,
				)

				table = {}
				command = registrar.command(table)

				def setup():
				"""Wraps user-facing mercurial commands with narrow-aware versions."""

				entry = extensions.wrapcommand(commands.table, 'clone', clonenarrowcmd)
				entry[1].append(('', 'narrow', None,
				_("create a narrow clone of select files")))
				entry[1].append(('', 'depth', '',
				_("limit the history fetched by distance from heads")))
				# TODO(durin42): unify sparse/narrow --include/--exclude logic a bit
				if 'sparse' not in extensions.enabled():
				entry[1].append(('', 'include', [],
				_("specifically fetch this file/directory")))
				entry[1].append(
				('', 'exclude', [],
				_("do not fetch this file/directory, even if included")))

				entry = extensions.wrapcommand(commands.table, 'pull', pullnarrowcmd)
				entry[1].append(('', 'depth', '',
				_("limit the history fetched by distance from heads")))

				extensions.wrapcommand(commands.table, 'archive', archivenarrowcmd)

				def expandpull(pullop, includepats, excludepats):
				if not narrowspec.needsexpansion(includepats):
				return includepats, excludepats

				heads = pullop.heads or pullop.rheads
				includepats, excludepats = pullop.remote.expandnarrow(
				includepats, excludepats, heads)
				pullop.repo.ui.debug('Expanded narrowspec to inc=%s, exc=%s\n' % (
				includepats, excludepats))
				return set(includepats), set(excludepats)

				def clonenarrowcmd(orig, ui, repo, args, *opts):
				"""Wraps clone command, so 'hg clone' first wraps localrepo.clone()."""
				wrappedextraprepare = util.nullcontextmanager()
				opts_narrow = opts['narrow']
				if opts_narrow:
				def pullbundle2extraprepare_widen(orig, pullop, kwargs):
				# Create narrow spec patterns from clone flags
				includepats = narrowspec.parsepatterns(opts['include'])
				excludepats = narrowspec.parsepatterns(opts['exclude'])

				# If necessary, ask the server to expand the narrowspec.
				includepats, excludepats = expandpull(
				pullop, includepats, excludepats)

				if not includepats and excludepats:
				# If nothing was included, we assume the user meant to include
				# everything, except what they asked to exclude.
				includepats = {'path:.'}

				narrowspec.save(pullop.repo, includepats, excludepats)

				# This will populate 'includepats' etc with the values from the
				# narrowspec we just saved.
				orig(pullop, kwargs)

				if opts.get('depth'):
				kwargs['depth'] = opts['depth']
				wrappedextraprepare = extensions.wrappedfunction(exchange,
				'_pullbundle2extraprepare', pullbundle2extraprepare_widen)

				def pullnarrow(orig, repo, args, *kwargs):
				narrowrepo.wraprepo(repo.unfiltered(), opts_narrow)
				if isinstance(repo, repoview.repoview):
				repo.__class__.__bases__ = (repo.__class__.__bases__[0],
				repo.unfiltered().__class__)
				if opts_narrow:
				repo.requirements.add(narrowrepo.requirement)
				repo._writerequirements()

				return orig(repo, args, *kwargs)

				wrappedpull = extensions.wrappedfunction(exchange, 'pull', pullnarrow)

				with wrappedextraprepare, wrappedpull:
				return orig(ui, repo, args, *opts)

				def pullnarrowcmd(orig, ui, repo, args, *opts):
				"""Wraps pull command to allow modifying narrow spec."""
				wrappedextraprepare = util.nullcontextmanager()
				if narrowrepo.requirement in repo.requirements:

				def pullbundle2extraprepare_widen(orig, pullop, kwargs):
				orig(pullop, kwargs)
				if opts.get('depth'):
				kwargs['depth'] = opts['depth']
				wrappedextraprepare = extensions.wrappedfunction(exchange,
				'_pullbundle2extraprepare', pullbundle2extraprepare_widen)

				with wrappedextraprepare:
				return orig(ui, repo, args, *opts)

				def archivenarrowcmd(orig, ui, repo, args, *opts):
				"""Wraps archive command to narrow the default includes."""
				if narrowrepo.requirement in repo.requirements:
				repo_includes, repo_excludes = repo.narrowpats
				includes = set(opts.get('include', []))
				excludes = set(opts.get('exclude', []))
				includes, excludes = narrowspec.restrictpatterns(
				includes, excludes, repo_includes, repo_excludes)
				if includes:
				opts['include'] = includes
				if excludes:
				opts['exclude'] = excludes
				return orig(ui, repo, args, *opts)

				def pullbundle2extraprepare(orig, pullop, kwargs):
				repo = pullop.repo
				if narrowrepo.requirement not in repo.requirements:
				return orig(pullop, kwargs)

				if narrowbundle2.narrowcap not in pullop.remotebundle2caps:
				raise error.Abort(_("server doesn't support narrow clones"))
				orig(pullop, kwargs)
				kwargs['narrow'] = True
				include, exclude = repo.narrowpats
				kwargs['oldincludepats'] = include
				kwargs['oldexcludepats'] = exclude
				kwargs['includepats'] = include
				kwargs['excludepats'] = exclude
				kwargs['known'] = [node.hex(ctx.node()) for ctx in
				repo.set('::%ln', pullop.common)
				if ctx.node() != node.nullid]
				if not kwargs['known']:
				# Mercurial serialized an empty list as '' and deserializes it as
				# [''], so delete it instead to avoid handling the empty string on the
				# server.
				del kwargs['known']

				extensions.wrapfunction(exchange,'_pullbundle2extraprepare',
				pullbundle2extraprepare)

				def _narrow(ui, repo, remote, commoninc, oldincludes, oldexcludes,
				newincludes, newexcludes, force):
				oldmatch = narrowspec.match(repo.root, oldincludes, oldexcludes)
				indygregUnsubmitted Not Done This function does a lot and wants a docstring. indygreg: This function does a lot and wants a docstring.
				newmatch = narrowspec.match(repo.root, newincludes, newexcludes)

				# This is essentially doing "hg outgoing" to find all local-only
				# commits. We will then check that the local-only commits don't
				# have any changes to files that will be untracked.
				unfi = repo.unfiltered()
				outgoing = discovery.findcommonoutgoing(unfi, remote,
				commoninc=commoninc)
				ui.status(_('looking for local changes to affected paths\n'))
				indygregUnsubmitted Not Done I believe a few operations in this function do obtain a write lock. However, the repo needs to be consistent for the duration of this function. So I think a write lock needs to be taken explicitly as part of the function. indygreg: I believe a few operations in this function do obtain a write lock. However, the repo needs to…
				localnodes = []
				for n in itertools.chain(outgoing.missing, outgoing.excluded):
				if any(oldmatch(f) and not newmatch(f) for f in unfi[n].files()):
				localnodes.append(n)
				revstostrip = unfi.revs('descendants(%ln)', localnodes)
				hiddenrevs = repoview.filterrevs(repo, 'visible')
				visibletostrip = list(repo.changelog.node(r)
				for r in (revstostrip - hiddenrevs))
				indygregUnsubmitted Not Done Use of `repo.changelog` in a loop creates perf problems. It's recommended to alias `repo.changelog.node` to a local and call that, as that also avoids the attribute lookup overhead, which does matter in tight loops. indygreg: Use of `repo.changelog` in a loop creates perf problems. It's recommended to alias `repo.
				if visibletostrip:
				ui.status(_('The following changeset(s) or their ancestors have '
				'local changes not on the remote:\n'))
				maxnodes = 10
				if ui.verbose or len(visibletostrip) <= maxnodes:
				for n in visibletostrip:
				ui.status('%s\n' % node.short(n))
				else:
				for n in visibletostrip[:maxnodes]:
				ui.status('%s\n' % node.short(n))
				ui.status(_('...and %d more, use --verbose to list all\n') %
				(len(visibletostrip) - maxnodes))
				if not force:
				raise error.Abort(_('local changes found'),
				hint=_('use --force-delete-local-changes to '
				'ignore'))

				if revstostrip:
				tostrip = [unfi.changelog.node(r) for r in revstostrip]
				if repo['.'].node() in tostrip:
				indygregUnsubmitted Not Done `.changelog.node` in a tight loop. indygreg: `.changelog.node` in a tight loop.
				# stripping working copy, so move to a different commit first
				urev = max(repo.revs('(::%n) - %ln + null',
				repo['.'].node(), visibletostrip))
				hg.clean(repo, urev)
				repair.strip(ui, unfi, tostrip, topic='narrow')

				todelete = []
				for f, f2, size in repo.store.datafiles():
				if f.startswith('data/'):
				file = f[5:-2]
				if not newmatch(file):
				todelete.append(f)
				elif f.startswith('meta/'):
				dir = f[5:-13]
				dirs = ['.'] + sorted(util.dirs({dir})) + [dir]
				include = True
				for d in dirs:
				visit = newmatch.visitdir(d)
				if not visit:
				include = False
				break
				if visit == 'all':
				break
				if not include:
				todelete.append(f)

				indygregUnsubmitted Not Done This is extremely low-level code to have in this file. It belongs somewhere in the store API, since it assumes a specific filesystem layout in the store. indygreg: This is extremely low-level code to have in this file. It belongs somewhere in the store API…
				repo.destroying()

				with repo.transaction("narrowing"):
				for f in todelete:
				ui.status(_('deleting %s\n') % f)
				util.unlinkpath(repo.svfs.join(f))
				repo.store.markremoved(f)

				for f in repo.dirstate:
				if not newmatch(f):
				repo.dirstate.drop(f)
				repo.wvfs.unlinkpath(f)
				repo.setnarrowpats(newincludes, newexcludes)

				repo.destroyed()

				def _widen(ui, repo, remote, commoninc, newincludes, newexcludes):
				newmatch = narrowspec.match(repo.root, newincludes, newexcludes)
				indygregUnsubmitted Not Done This function does a lot. It wants a docstring. indygreg: This function does a lot. It wants a docstring.

				# TODO(martinvonz): Get expansion working with widening/narrowing.
				if narrowspec.needsexpansion(newincludes):
				raise error.Abort('Expansion not yet supported on pull')

				def pullbundle2extraprepare_widen(orig, pullop, kwargs):
				orig(pullop, kwargs)
				# The old{in,ex}cludepats have already been set by orig()
				kwargs['includepats'] = newincludes
				kwargs['excludepats'] = newexcludes
				wrappedextraprepare = extensions.wrappedfunction(exchange,
				'_pullbundle2extraprepare', pullbundle2extraprepare_widen)

				# define a function that narrowbundle2 can call after creating the
				# backup bundle, but before applying the bundle from the server
				def setnewnarrowpats():
				repo.setnarrowpats(newincludes, newexcludes)
				repo.setnewnarrowpats = setnewnarrowpats

				ds = repo.dirstate
				p1, p2 = ds.p1(), ds.p2()
				with ds.parentchange():
				ds.setparents(node.nullid, node.nullid)
				common = commoninc[0]
				with wrappedextraprepare:
				exchange.pull(repo, remote, heads=common)
				with ds.parentchange():
				ds.setparents(p1, p2)

				actions = {k: [] for k in 'a am f g cd dc r dm dg m e k p pr'.split()}
				addgaction = actions['g'].append

				mf = repo['.'].manifest().matches(newmatch)
				for f, fn in mf.iteritems():
				if f not in repo.dirstate:
				addgaction((f, (mf.flags(f), False),
				"add from widened narrow clone"))

				merge.applyupdates(repo, actions, wctx=repo[None],
				mctx=repo['.'], overwrite=False)
				merge.recordupdates(repo, actions, branchmerge=False)

				# TODO(rdamazio): Make new matcher format and update description
				@command('tracked',
				[('', 'addinclude', [], _('new paths to include')),
				('', 'removeinclude', [], _('old paths to no longer include')),
				('', 'addexclude', [], _('new paths to exclude')),
				('', 'removeexclude', [], _('old paths to no longer exclude')),
				('', 'clear', False, _('whether to replace the existing narrowspec')),
				('', 'force-delete-local-changes', False,
				_('forces deletion of local changes when narrowing')),
				] + commands.remoteopts,
				_('[OPTIONS]... [REMOTE]'),
				inferrepo=True)
				def trackedcmd(ui, repo, remotepath=None, pats, *opts):
				"""show or change the current narrowspec

				With no argument, shows the current narrowspec entries, one per line. Each
				line will be prefixed with 'I' or 'X' for included or excluded patterns,
				respectively.

				The narrowspec is comprised of expressions to match remote files and/or
				directories that should be pulled into your client.
				The narrowspec has include and exclude expressions, with excludes always
				trumping includes: that is, if a file matches an exclude expression, it will
				be excluded even if it also matches an include expression.
				Excluding files that were never included has no effect.

				Each included or excluded entry is in the format described by
				'hg help patterns'.

				The options allow you to add or remove included and excluded expressions.

				If --clear is specified, then all previous includes and excludes are DROPPED
				and replaced by the new ones specified to --addinclude and --addexclude.
				If --clear is specified without any further options, the narrowspec will be
				empty and will not match any files.
				"""
				if narrowrepo.requirement not in repo.requirements:
				ui.warn(_('The narrow command is only supported on respositories cloned'
				' with --narrow.\n'))
				return 1

				# Before supporting, decide whether it "hg tracked --clear" should mean
				# tracking no paths or all paths.
				if opts['clear']:
				ui.warn(_('The --clear option is not yet supported.\n'))
				return 1

				if narrowspec.needsexpansion(opts['addinclude'] + opts['addexclude']):
				raise error.Abort('Expansion not yet supported on widen/narrow')

				addedincludes = narrowspec.parsepatterns(opts['addinclude'])
				removedincludes = narrowspec.parsepatterns(opts['removeinclude'])
				addedexcludes = narrowspec.parsepatterns(opts['addexclude'])
				removedexcludes = narrowspec.parsepatterns(opts['removeexclude'])
				widening = addedincludes or removedexcludes
				narrowing = removedincludes or addedexcludes
				only_show = not widening and not narrowing

				# Only print the current narrowspec.
				if only_show:
				include, exclude = repo.narrowpats

				ui.pager('tracked')
				fm = ui.formatter('narrow', opts)
				for i in sorted(include):
				fm.startitem()
				fm.write('status', '%s ', 'I', label='narrow.included')
				fm.write('pat', '%s\n', i, label='narrow.included')
				for i in sorted(exclude):
				fm.startitem()
				fm.write('status', '%s ', 'X', label='narrow.excluded')
				fm.write('pat', '%s\n', i, label='narrow.excluded')
				fm.end()
				return 0

				with repo.wlock(), repo.lock():
				cmdutil.bailifchanged(repo)

				# Find the revisions we have in common with the remote. These will
				# be used for finding local-only changes for narrowing. They will
				# also define the set of revisions to update for widening.
				remotepath = ui.expandpath(remotepath or 'default')
				url, branches = hg.parseurl(remotepath)
				ui.status(_('comparing with %s\n') % util.hidepassword(url))
				remote = hg.peer(repo, opts, url)
				commoninc = discovery.findcommonincoming(repo, remote)

				oldincludes, oldexcludes = repo.narrowpats
				if narrowing:
				newincludes = oldincludes - removedincludes
				newexcludes = oldexcludes \| addedexcludes
				_narrow(ui, repo, remote, commoninc, oldincludes, oldexcludes,
				newincludes, newexcludes,
				opts['force_delete_local_changes'])
				# _narrow() updated the narrowspec and _widen() below needs to
				# use the updated values as its base (otherwise removed includes
				# and addedexcludes will be lost in the resulting narrowspec)
				oldincludes = newincludes
				oldexcludes = newexcludes

				if widening:
				newincludes = oldincludes \| addedincludes
				newexcludes = oldexcludes - removedexcludes
				_widen(ui, repo, remote, commoninc, newincludes, newexcludes)

				return 0

hgext/narrow/narrowcopies.py

This file was added.

				# narrowcopies.py - extensions to mercurial copies module to support narrow
				# clones
				indygregUnsubmitted Not Done Copy code is complex and has performance concerns. I'd like to see this moved into core sooner rather than later. Perhaps a good first step would be to add the `narrowmatch` method to `localrepository`? Or something on `localrepository` in core so we can detect presence of narrow in a one-liner. indygreg: Copy code is complex and has performance concerns. I'd like to see this moved into core sooner…
				#
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				from __future__ import absolute_import

				from mercurial import (
				copies,
				extensions,
				util,
				)

				def setup(repo):
				def _computeforwardmissing(orig, a, b, match=None):
				missing = orig(a, b, match)
				if util.safehasattr(repo, 'narrowmatch'):
				narrowmatch = repo.narrowmatch()
				missing = filter(narrowmatch, missing)
				return missing

				def _checkcopies(orig, srcctx, dstctx, f, base, tca, remotebase, limit,
				data):
				if util.safehasattr(repo, 'narrowmatch'):
				narrowmatch = repo.narrowmatch()
				if not narrowmatch(f):
				return
				orig(srcctx, dstctx, f, base, tca, remotebase, limit, data)

				extensions.wrapfunction(copies, '_computeforwardmissing',
				_computeforwardmissing)
				extensions.wrapfunction(copies, '_checkcopies', _checkcopies)

hgext/narrow/narrowdirstate.py

This file was added.

				# narrowdirstate.py - extensions to mercurial dirstate to support narrow clones
				#
				indygregUnsubmitted Not Done I'm not an expert on dirstate. But I know enough to know that "here be dragons." I'd like to see everything dirstate moved into core ASAP so it can be better integrated with sparse, etc. People hacking on this code in core need to be aware of the implications for narrow. indygreg: I'm not an expert on dirstate. But I know enough to know that "here be dragons." I'd like to…
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				from __future__ import absolute_import

				from mercurial.i18n import _
				from mercurial import (
				dirstate,
				error,
				extensions,
				match as matchmod,
				util as hgutil,
				)

				from . import narrowspec

				def setup(repo):
				"""Add narrow spec dirstate ignore, block changes outside narrow spec."""

				def walk(orig, self, match, subrepos, unknown, ignored, full=True,
				narrowonly=True):
				if narrowonly:
				narrowmatch = repo.narrowmatch()
				match = matchmod.intersectmatchers(match, narrowmatch)
				return orig(self, match, subrepos, unknown, ignored, full)

				extensions.wrapfunction(dirstate.dirstate, 'walk', walk)

				# Prevent adding files that are outside the sparse checkout
				editfuncs = ['normal', 'add', 'normallookup', 'copy', 'remove', 'merge']
				for func in editfuncs:
				def _wrapper(orig, self, *args):
				dirstate = repo.dirstate
				narrowmatch = repo.narrowmatch()
				for f in args:
				if f is not None and not narrowmatch(f) and f not in dirstate:
				raise error.Abort(_("cannot track '%s' - it is outside " +
				"the narrow clone") % f)
				return orig(self, *args)
				extensions.wrapfunction(dirstate.dirstate, func, _wrapper)

				def filterrebuild(orig, self, parent, allfiles, changedfiles=None):
				if changedfiles is None:
				# Rebuilding entire dirstate, let's filter allfiles to match the
				# narrowspec.
				allfiles = [f for f in allfiles if repo.narrowmatch()(f)]
				orig(self, parent, allfiles, changedfiles)

				extensions.wrapfunction(dirstate.dirstate, 'rebuild', filterrebuild)

				def _narrowbackupname(backupname):
				assert 'dirstate' in backupname
				return backupname.replace('dirstate', narrowspec.FILENAME)

				def restorebackup(orig, self, tr, backupname):
				self._opener.rename(_narrowbackupname(backupname), narrowspec.FILENAME,
				checkambig=True)
				orig(self, tr, backupname)

				extensions.wrapfunction(dirstate.dirstate, 'restorebackup', restorebackup)

				def savebackup(orig, self, tr, backupname):
				orig(self, tr, backupname)

				narrowbackupname = _narrowbackupname(backupname)
				self._opener.tryunlink(narrowbackupname)
				hgutil.copyfile(self._opener.join(narrowspec.FILENAME),
				self._opener.join(narrowbackupname), hardlink=True)

				extensions.wrapfunction(dirstate.dirstate, 'savebackup', savebackup)

				def clearbackup(orig, self, tr, backupname):
				orig(self, tr, backupname)
				self._opener.unlink(_narrowbackupname(backupname))

				extensions.wrapfunction(dirstate.dirstate, 'clearbackup', clearbackup)

hgext/narrow/narrowmerge.py

This file was added.

				# narrowmerge.py - extensions to mercurial merge module to support narrow clones
				#
				indygregUnsubmitted Not Done More code that should be on the fast track to core. indygreg: More code that should be on the fast track to core.
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				from __future__ import absolute_import

				from mercurial.i18n import _
				from mercurial import (
				copies,
				error,
				extensions,
				merge,
				util,
				)

				def setup():
				def _manifestmerge(orig, repo, wctx, p2, pa, branchmerge, args, *kwargs):
				"""Filter updates to only lay out files that match the narrow spec."""
				actions, diverge, renamedelete = orig(
				repo, wctx, p2, pa, branchmerge, args, *kwargs)

				if not util.safehasattr(repo, 'narrowmatch'):
				return actions, diverge, renamedelete
				indygregUnsubmitted Not Done The more I see this pattern, the more I think there needs to be an attribute on `localrepository` that can be used for quickly testing if narrow is in play. Actually, should we assume all repos are narrow (with the all matcher) and fast path the special case of the default? indygreg: The more I see this pattern, the more I think there needs to be an attribute on…

				nooptypes = set(['k']) # TODO: handle with nonconflicttypes
				nonconflicttypes = set('a am c cm f g r e'.split())
				narrowmatch = repo.narrowmatch()
				for f, action in actions.items():
				if narrowmatch(f):
				pass
				elif not branchmerge:
				del actions[f] # just updating, ignore changes outside clone
				elif action[0] in nooptypes:
				del actions[f] # merge does not affect file
				elif action[0] in nonconflicttypes:
				raise error.Abort(_('merge affects file \'%s\' outside narrow, '
				'which is not yet supported') % f,
				hint=_('merging in the other direction '
				'may work'))
				else:
				raise error.Abort(_('conflict in file \'%s\' is outside '
				'narrow clone') % f)

				return actions, diverge, renamedelete

				extensions.wrapfunction(merge, 'manifestmerge', _manifestmerge)

				def _checkcollision(orig, repo, wmf, actions):
				if util.safehasattr(repo, 'narrowmatch'):
				narrowmatch = repo.narrowmatch()
				wmf = wmf.matches(narrowmatch)
				if actions:
				narrowactions = {}
				for m, actionsfortype in actions.iteritems():
				narrowactions[m] = []
				for (f, args, msg) in actionsfortype:
				if narrowmatch(f):
				narrowactions[m].append((f, args, msg))
				actions = narrowactions
				return orig(repo, wmf, actions)

				extensions.wrapfunction(merge, '_checkcollision', _checkcollision)

				def _computenonoverlap(orig, repo, args, *kwargs):
				u1, u2 = orig(repo, args, *kwargs)
				if not util.safehasattr(repo, 'narrowmatch'):
				return u1, u2

				narrowmatch = repo.narrowmatch()
				u1 = [f for f in u1 if narrowmatch(f)]
				u2 = [f for f in u2 if narrowmatch(f)]
				return u1, u2
				extensions.wrapfunction(copies, '_computenonoverlap', _computenonoverlap)

hgext/narrow/narrowpatch.py

This file was added.

				# narrowpatch.py - extensions to mercurial patch module to support narrow clones
				#
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				from __future__ import absolute_import

				from mercurial import (
				extensions,
				patch,
				util,
				)

				def setup(repo):
				def _filepairs(orig, *args):
				"""Only includes files within the narrow spec in the diff."""
				if util.safehasattr(repo, 'narrowmatch'):
				narrowmatch = repo.narrowmatch()
				for x in orig(*args):
				f1, f2, copyop = x
				if ((not f1 or narrowmatch(f1)) and
				(not f2 or narrowmatch(f2))):
				yield x
				else:
				for x in orig(*args):
				yield x

				def trydiff(orig, repo, revs, ctx1, ctx2, modified, added, removed,
				copy, getfilectx, args, *kwargs):
				if util.safehasattr(repo, 'narrowmatch'):
				narrowmatch = repo.narrowmatch()
				modified = filter(narrowmatch, modified)
				added = filter(narrowmatch, added)
				removed = filter(narrowmatch, removed)
				copy = {k: v for k, v in copy.iteritems() if narrowmatch(k)}
				return orig(repo, revs, ctx1, ctx2, modified, added, removed, copy,
				getfilectx, args, *kwargs)

				extensions.wrapfunction(patch, '_filepairs', _filepairs)
				extensions.wrapfunction(patch, 'trydiff', trydiff)

hgext/narrow/narrowrepo.py

This file was added.

				# narrowrepo.py - repository which supports narrow revlogs, lazy loading
				#
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				from __future__ import absolute_import

				from mercurial import (
				bundlerepo,
				localrepo,
				match as matchmod,
				scmutil,
				)

				from .. import (
				share,
				)

				from . import (
				narrowrevlog,
				narrowspec,
				)

				requirement = 'narrowhg'

				indygregUnsubmitted Done I think we'll want to put `exp` somewhere in the requirement name until we're ready to mark the feature as non-experimental in core. indygreg: I think we'll want to put `exp` somewhere in the requirement name until we're ready to mark the…
				def wrappostshare(orig, sourcerepo, destrepo, **kwargs):
				orig(sourcerepo, destrepo, **kwargs)
				if requirement in sourcerepo.requirements:
				with destrepo.wlock():
				with destrepo.vfs('shared', 'a') as fp:
				fp.write(narrowspec.FILENAME + '\n')

				indygregUnsubmitted Not Done Is this really the best way to influence the content of the `shared` file? If it doesn't exist, a good follow-up would be a hook point in core to allow extensions to modify the content of this file so it is only written once. indygreg: Is this really the best way to influence the content of the `shared` file? If it doesn't exist…
				def unsharenarrowspec(orig, ui, repo, repopath):
				if (requirement in repo.requirements
				and repo.path == repopath and repo.shared()):
				srcrepo = share._getsrcrepo(repo)
				with srcrepo.vfs(narrowspec.FILENAME) as f:
				spec = f.read()
				with repo.vfs(narrowspec.FILENAME, 'w') as f:
				f.write(spec)
				return orig(ui, repo, repopath)

				indygregUnsubmitted Not Done Again, my reaction is "eww." But this one is a bit special since we're dealing with a non-standard file. So unless we create a "also write these files with this content" mechanism (which doesn't feel necessary), this is probably the best we can do. This code should live in core eventually though. So it should work itself out. indygreg: Again, my reaction is "eww." But this one is a bit special since we're dealing with a non…
				def wraprepo(repo, opts_narrow):
				"""Enables narrow clone functionality on a single local repository."""

				cacheprop = localrepo.storecache
				if isinstance(repo, bundlerepo.bundlerepository):
				# We have to use a different caching property decorator for
				# bundlerepo because storecache blows up in strange ways on a
				# bundlerepo. Fortunately, there's no risk of data changing in
				# a bundlerepo.
				cacheprop = lambda name: localrepo.unfilteredpropertycache

				class narrowrepository(repo.__class__):

				def _constructmanifest(self):
				manifest = super(narrowrepository, self)._constructmanifest()
				narrowrevlog.makenarrowmanifestrevlog(manifest, repo)
				return manifest

				@cacheprop('00manifest.i')
				def manifestlog(self):
				mfl = super(narrowrepository, self).manifestlog
				narrowrevlog.makenarrowmanifestlog(mfl, self)
				return mfl

				def file(self, f):
				fl = super(narrowrepository, self).file(f)
				narrowrevlog.makenarrowfilelog(fl, self.narrowmatch())
				return fl

				@localrepo.repofilecache(narrowspec.FILENAME)
				def narrowpats(self):
				return narrowspec.load(self)
				indygregUnsubmitted Done Please add a docstring, since this is a new public method. indygreg: Please add a docstring, since this is a new public method.

				@localrepo.repofilecache(narrowspec.FILENAME)
				def _narrowmatch(self):
				include, exclude = self.narrowpats
				if not opts_narrow and not include and not exclude:
				return matchmod.always(self.root, '')
				return narrowspec.match(self.root, include=include, exclude=exclude)

				# TODO(martinvonz): make this property-like instead?
				def narrowmatch(self):
				return self._narrowmatch

				indygregUnsubmitted Not Done I agree with the inline todo :) indygreg: I agree with the inline todo :)
				durin42AuthorUnsubmitted Not Done Okay if we come back to that, since it's already logged as a TODO? durin42: Okay if we come back to that, since it's already logged as a TODO?
				def setnarrowpats(self, newincludes, newexcludes):
				narrowspec.save(self, newincludes, newexcludes)
				self.invalidate(clearfilecache=True)

				indygregUnsubmitted Not Done See comment in `narrowspec.py` about needing to hold the `wlock` somewhere. indygreg: See comment in `narrowspec.py` about needing to hold the `wlock` somewhere.
				# I'm not sure this is the right place to do this filter.
				# context._manifestmatches() would probably be better, or perhaps
				# move it to a later place, in case some of the callers do want to know
				# which directories changed. This seems to work for now, though.
				def status(self, args, *kwargs):
				indygregUnsubmitted Not Done I agree with the comment. I'm not an expert on dirstate/status, but this does seem like the wrong place. I'm pretty sure we have code paths that call lower-level `status()` functions. I would expect this to live in a similar place to where sparse lives. I think we can overlook this for the initial landing. This feels like a bit of work to resolve and can be deferred to the stabilization period. indygreg: I agree with the comment. I'm not an expert on dirstate/status, but this does seem like the…
				s = super(narrowrepository, self).status(args, *kwargs)
				narrowmatch = self.narrowmatch()
				modified = filter(narrowmatch, s.modified)
				added = filter(narrowmatch, s.added)
				removed = filter(narrowmatch, s.removed)
				deleted = filter(narrowmatch, s.deleted)
				unknown = filter(narrowmatch, s.unknown)
				ignored = filter(narrowmatch, s.ignored)
				clean = filter(narrowmatch, s.clean)
				return scmutil.status(modified, added, removed, deleted, unknown,
				ignored, clean)

				repo.__class__ = narrowrepository

hgext/narrow/narrowrevlog.py

This file was added.

				# narrowrevlog.py - revlog storing irrelevant nodes as "ellipsis" nodes
				#
				indygregUnsubmitted Not Done I'd like to see this file's content moved into core sooner rather than later. There are a lot of implications for storage that need to be in people's minds when they are touching code in core. indygreg: I'd like to see this file's content moved into core sooner rather than later. There are a lot…
				durin42AuthorUnsubmitted Not Done Yes, noted this in a TODO.rst. durin42: Yes, noted this in a TODO.rst.
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				from __future__ import absolute_import

				from mercurial import (
				manifest,
				revlog,
				util,
				)

				ELLIPSIS_NODE_FLAG = 1 << 14
				revlog.REVIDX_KNOWN_FLAGS \|= ELLIPSIS_NODE_FLAG
				if (util.safehasattr(revlog, 'REVIDX_FLAGS_ORDER') and
				ELLIPSIS_NODE_FLAG not in revlog.REVIDX_FLAGS_ORDER):
				revlog.REVIDX_FLAGS_ORDER.append(ELLIPSIS_NODE_FLAG)

				indygregUnsubmitted Done This needs to move to core because revision flags need to be reserved there. indygreg: This needs to move to core because revision flags need to be reserved there.
				def readtransform(self, text):
				return text, False

				def writetransform(self, text):
				return text, False

				def rawtransform(self, text):
				return False

				if util.safehasattr(revlog, 'addflagprocessor'):
				revlog.addflagprocessor(ELLIPSIS_NODE_FLAG,
				indygregUnsubmitted Done This condition should always be true. indygreg: This condition should always be true.
				(readtransform, writetransform, rawtransform))

				def setup():
				# We just wanted to add the flag processor, which is done at module
				# load time.
				pass

				class excludeddir(manifest.treemanifest):
				def __init__(self, dir, node):
				indygregUnsubmitted Done Please add docstring. indygreg: Please add docstring.
				super(excludeddir, self).__init__(dir)
				self._node = node
				# Add an empty file, which will be included by iterators and such,
				# appearing as the directory itself (i.e. something like "dir/")
				self._files[''] = node
				self._flags[''] = 't'

				# Manifests outside the narrowspec should never be modified, so avoid
				# copying. This makes a noticeable difference when there are very many
				# directories outside the narrowspec. Also, it makes sense for the copy to
				# be of the same type as the original, which would not happen with the
				# super type's copy().
				def copy(self):
				return self

				class excludeddirmanifestctx(manifest.treemanifestctx):
				def __init__(self, dir, node):
				indygregUnsubmitted Done docstring indygreg: docstring
				self._dir = dir
				self._node = node

				def read(self):
				return excludeddir(self._dir, self._node)

				def write(self, *args):
				raise AssertionError('Attempt to write manifest from excluded dir %s' %
				self._dir)

				indygregUnsubmitted Done I think we prefer `error.ProgrammingError` these days. indygreg: I think we prefer `error.ProgrammingError` these days.
				class excludedmanifestrevlog(manifest.manifestrevlog):
				def __init__(self, dir):
				indygregUnsubmitted Done docstring indygreg: docstring
				self._dir = dir

				def __len__(self):
				raise AssertionError('Attempt to get length of excluded dir %s' %
				self._dir)

				def rev(self, node):
				raise AssertionError('Attempt to get rev from excluded dir %s' %
				self._dir)

				def linkrev(self, node):
				raise AssertionError('Attempt to get linkrev from excluded dir %s' %
				self._dir)

				def node(self, rev):
				raise AssertionError('Attempt to get node from excluded dir %s' %
				self._dir)

				indygregUnsubmitted Done `error.ProgrammingError` indygreg: `error.ProgrammingError`
				def add(self, args, *kwargs):
				# We should never write entries in dirlogs outside the narrow clone.
				# However, the method still gets called from writesubtree() in
				# _addtree(), so we need to handle it. We should possibly make that
				# avoid calling add() with a clean manifest (_dirty is always False
				# in excludeddir instances).
				pass
				indygregUnsubmitted Not Done Consider doing that and changing this to raise if called. indygreg: Consider doing that and changing this to raise if called.
				durin42AuthorUnsubmitted Not Done Added to a TODO file. durin42: Added to a TODO file.

				def makenarrowmanifestrevlog(mfrevlog, repo):
				if util.safehasattr(mfrevlog, '_narrowed'):
				return

				class narrowmanifestrevlog(mfrevlog.__class__):
				# This function is called via debug{revlog,index,data}, but also during
				# at least some push operations. This will be used to wrap/exclude the
				# child directories when using treemanifests.
				def dirlog(self, dir):
				if dir and not dir.endswith('/'):
				indygregUnsubmitted Done Nit: `dir` is a builtin. If this matches core, fine. But I'd prefer avoiding the name collision. indygreg: Nit: `dir` is a builtin. If this matches core, fine. But I'd prefer avoiding the name collision.
				durin42AuthorUnsubmitted Not Done Ouch, good catch. This does match core, so I fixed both in my followup. durin42: Ouch, good catch. This does match core, so I fixed both in my followup.
				dir = dir + '/'
				if not repo.narrowmatch().visitdir(dir[:-1] or '.'):
				return excludedmanifestrevlog(dir)
				result = super(narrowmanifestrevlog, self).dirlog(dir)
				makenarrowmanifestrevlog(result, repo)
				return result

				mfrevlog.__class__ = narrowmanifestrevlog
				mfrevlog._narrowed = True

				indygregUnsubmitted Not Done In-place mutation of low-level types. Yummy. Please add a todo for the post-landing list to construct the proper type from the beginning. This likely requires some API changes in core. I'm thinking some function should return the type to use for new revlogs. Or we should spawn this type and call super.init from its init. indygreg: In-place mutation of low-level types. Yummy. Please add a todo for the post-landing list to…
				durin42AuthorUnsubmitted Not Done Yeah, this is one of several places where I want to just hoist internals changes to core, with the only customer being narrow (for now, at least). It's super gross the way it currently is. durin42: Yeah, this is one of several places where I want to just hoist internals changes to core, with…
				def makenarrowmanifestlog(mfl, repo):
				class narrowmanifestlog(mfl.__class__):
				def get(self, dir, node, verify=True):
				if not repo.narrowmatch().visitdir(dir[:-1] or '.'):
				return excludeddirmanifestctx(dir, node)
				return super(narrowmanifestlog, self).get(dir, node, verify=verify)
				mfl.__class__ = narrowmanifestlog

				def makenarrowfilelog(fl, narrowmatch):
				class narrowfilelog(fl.__class__):
				def renamed(self, node):
				m = super(narrowfilelog, self).renamed(node)
				if m and not narrowmatch(m[0]):
				return None
				return m

				indygregUnsubmitted Not Done I think this wants a comment explaining why we lie about rename metadata when the destination is outside of the narrow spec. Also, we'll want to flag this for further review, since there could be some interesting implications to lying here. indygreg: I think this wants a comment explaining why we lie about rename metadata when the destination…
				durin42AuthorUnsubmitted Not Done Yeah, all I remember about this (https://bitbucket.org/Google/narrowhg/commits/3cd72b1a1b41c9e46f12eba78c253da169277374) is that git-diffs break in the case of a rename from outside->inside. There are almost certainly other problems lurking in the weeds here, but to my knowledge we've not seen them at Google... durin42: Yeah, all I remember about this (https://bitbucket.
				def size(self, rev):
				# We take advantage of the fact that remotefilelog
				# lacks a node() method to just skip the
				# rename-checking logic when on remotefilelog. This
				# might be incorrect on other non-revlog-based storage
				# engines, but for now this seems to be fine.
				if util.safehasattr(self, 'node'):
				node = self.node(rev)
				indygregUnsubmitted Not Done This is making assumptions about code that hasn't landed yet. Not sure if we should replace this with a TODO or what. indygreg: This is making assumptions about code that hasn't landed yet. Not sure if we should replace…
				durin42AuthorUnsubmitted Not Done Added an explicit TODO. We can make this a TON cleaner when narrowhg can be formally aware of remotefilelog, this was mostly a kludge to work around them being in disjoint repos and not wanting to assume we could find remotefilelog cleanly. durin42: Added an explicit TODO. We can make this a TON cleaner when narrowhg can be formally aware of…
				# Because renamed() is overridden above to
				# sometimes return None even if there is metadata
				# in the revlog, size can be incorrect for
				# copies/renames, so we need to make sure we call
				# the super class's implementation of renamed()
				# for the purpose of size calculation.
				if super(narrowfilelog, self).renamed(node):
				return len(self.read(node))
				return super(narrowfilelog, self).size(rev)

				def cmp(self, node, text):
				different = super(narrowfilelog, self).cmp(node, text)
				if different:
				# Similar to size() above, if the file was copied from
				# a file outside the narrowspec, the super class's
				# would have returned True because we tricked it into
				# thinking that the file was not renamed.
				if super(narrowfilelog, self).renamed(node):
				t2 = self.read(node)
				return t2 != text
				return different

				fl.__class__ = narrowfilelog

hgext/narrow/narrowspec.py

This file was added.

				# narrowspec.py - methods for working with a narrow view of a repository
				#
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				from __future__ import absolute_import

				import errno

				from mercurial.i18n import _
				from mercurial import (
				error,
				match as matchmod,
				util,
				)

				from .. import (
				share,
				)

				FILENAME = 'narrowspec'

				def _parsestoredpatterns(text):
				"""Parses the narrowspec format that's stored on disk."""
				patlist = None
				indygregUnsubmitted Not Done The format needs documentation somewhere inline. indygreg: The format needs documentation somewhere inline.
				includepats = []
				excludepats = []
				for l in text.splitlines():
				if l == '[includes]':
				if patlist is None:
				patlist = includepats
				else:
				raise error.Abort(_('narrowspec includes section must appear '
				'at most once, before excludes'))
				elif l == '[excludes]':
				if patlist is not excludepats:
				patlist = excludepats
				else:
				raise error.Abort(_('narrowspec excludes section must appear '
				'at most once'))
				else:
				indygregUnsubmitted Not Done Oh, hey, this looks just like sparse profiles! I sense some code conversion in our future... indygreg: Oh, hey, this looks just like sparse profiles! I sense some code conversion in our future...
				durin42AuthorUnsubmitted Not Done Yes, the goal is that the two extensions will share a fair amount of logic. :) durin42: Yes, the goal is that the two extensions will share a fair amount of logic. :)
				patlist.append(l)

				return set(includepats), set(excludepats)

				def parseserverpatterns(text):
				"""Parses the narrowspec format that's returned by the server."""
				includepats = set()
				excludepats = set()

				# We get one entry per line, in the format "<key> <value>".
				# It's OK for value to contain other spaces.
				for kp in (l.split(' ', 1) for l in text.splitlines()):
				indygregUnsubmitted Not Done Playing devil's advocate, do we really need two formats doing the same thing? indygreg: Playing devil's advocate, do we really need two formats doing the same thing?
				durin42AuthorUnsubmitted Not Done That's a great question. I don't really know the answer offhand. Added a TODO... durin42: That's a great question. I don't really know the answer offhand. Added a TODO...
				if len(kp) != 2:
				raise error.Abort(_('Invalid narrowspec pattern line: "%s"') % kp)
				key = kp[0]
				pat = kp[1]
				if key == 'include':
				includepats.add(pat)
				elif key == 'exclude':
				excludepats.add(pat)
				else:
				raise error.Abort(_('Invalid key "%s" in server response') % key)

				return includepats, excludepats

				def normalizesplitpattern(kind, pat):
				"""Returns the normalized version of a pattern and kind.

				Returns a tuple with the normalized kind and normalized pattern.
				"""
				pat = pat.rstrip('/')
				_validatepattern(pat)
				return kind, pat

				def _numlines(s):
				"""Returns the number of lines in s, including ending empty lines."""
				# We use splitlines because it is Unicode-friendly and thus Python 3
				# compatible. However, it does not count empty lines at the end, so trick
				# it by adding a character at the end.
				return len((s + 'x').splitlines())

				indygregUnsubmitted Not Done Can we use `str.count()` to avoid creating objects for each line? indygreg: Can we use `str.count()` to avoid creating objects for each line?
				durin42AuthorUnsubmitted Not Done Gave it a shot, and things broke all over the place. :( durin42: Gave it a shot, and things broke all over the place. :(
				def _validatepattern(pat):
				"""Validates the pattern and aborts if it is invalid."""

				# We use newlines as separators in the narrowspec file, so don't allow them
				# in patterns.
				if _numlines(pat) > 1:
				raise error.Abort('newlines are not allowed in narrowspec paths')

				components = pat.split('/')
				if '.' in components or '..' in components:
				raise error.Abort(_('"." and ".." are not allowed in narrowspec paths'))

				indygregUnsubmitted Not Done What about `\` as a path separator? Should we also ban that? I assume we're interpreting the value here and paths as bytes, so `\` will never be used as an escape character? indygreg: What about `\` as a path separator? Should we also ban that? I assume we're interpreting the…
				durin42AuthorUnsubmitted Not Done @martinvonz please correct me if this is wrong. The paths are stored in the narrowspec with canonical (that is /) path separators, just like the rest of the internals. We're reading things as bytes, so yes, \ should never occur. durin42: @martinvonz please correct me if this is wrong. The paths are stored in the narrowspec with…
				def normalizepattern(pattern, defaultkind='path'):
				"""Returns the normalized version of a text-format pattern.

				If the pattern has no kind, the default will be added.
				"""
				kind, pat = matchmod._patsplit(pattern, defaultkind)
				return '%s:%s' % normalizesplitpattern(kind, pat)

				def parsepatterns(pats):
				"""Parses a list of patterns into a typed pattern set."""
				return set(normalizepattern(p) for p in pats)

				def format(includes, excludes):
				output = '[includes]\n'
				for i in sorted(includes - excludes):
				output += i + '\n'
				output += '[excludes]\n'
				for e in sorted(excludes):
				output += e + '\n'
				return output
				indygregUnsubmitted Not Done It is better to build a list of lines and `'\n'.join()` them. indygreg: It is better to build a list of lines and `'\n'.join()` them.
				durin42AuthorUnsubmitted Not Done It shouldn't be: this particular "str with refcount of 1" behavior was optimized a long time ago in cpython to not be a horrendous stack of copies. If it shows up in a profile, we can obviously fix it. durin42: It shouldn't be: this particular "str with refcount of 1" behavior was optimized a long time…

				def match(root, include=None, exclude=None):
				if not include:
				# Passing empty include and empty exclude to matchmod.match()
				# gives a matcher that matches everything, so explicitly use
				# the nevermatcher.
				return matchmod.never(root, '')
				return matchmod.match(root, '', [], include=include or [],
				exclude=exclude or [])

				def needsexpansion(includes):
				return [i for i in includes if i.startswith('include:')]

				def load(repo):
				if repo.shared():
				repo = share._getsrcrepo(repo)
				try:
				spec = repo.vfs.read(FILENAME)
				except IOError as e:
				# Treat "narrowspec does not exist" the same as "narrowspec file exists
				# and is empty".
				if e.errno == errno.ENOENT:
				# Without this the next call to load will use the cached
				indygregUnsubmitted Not Done This reinvents `repo.vfs.tryread()`. indygreg: This reinvents `repo.vfs.tryread()`.
				durin42AuthorUnsubmitted Not Done Yep, but it also does some extra cache invalidation, so I don't think I can fuse them. durin42: Yep, but it also does some extra cache invalidation, so I don't think I can fuse them.
				# non-existence of the file, which can cause some odd issues.
				repo.invalidate(clearfilecache=True)
				return set(), set()
				raise
				return _parsestoredpatterns(spec)

				def save(repo, includepats, excludepats):
				spec = format(includepats, excludepats)
				if repo.shared():
				repo = share._getsrcrepo(repo)
				repo.vfs.write(FILENAME, spec)

				indygregUnsubmitted Not Done This is called from `narrowrepo.setnarrowparts()` and neither of them cares about locking. Somewhere we should ensure we hold the wlock. indygreg: This is called from `narrowrepo.setnarrowparts()` and neither of them cares about locking.
				durin42AuthorUnsubmitted Not Done Added to my TODO doc because I don't want reasoning about locking to block getting out of this pile-of-comments hell. durin42: Added to my TODO doc because I don't want reasoning about locking to block getting out of this…
				def restrictpatterns(req_includes, req_excludes, repo_includes, repo_excludes,
				invalid_includes=None):
				r""" Restricts the patterns according to repo settings,
				results in a logical AND operation

				:param req_includes: requested includes
				:param req_excludes: requested excludes
				:param repo_includes: repo includes
				:param repo_excludes: repo excludes
				:param invalid_includes: an array to collect invalid includes
				:return: include and exclude patterns

				indygregUnsubmitted Not Done The purpose of this function was not clear on initial reading. Could you please elaborate a bit more in the docstring? indygreg: The purpose of this function was not clear on initial reading. Could you please elaborate a bit…
				>>> restrictpatterns({'f1','f2'}, {}, ['f1'], [])
				(set(['f1']), {})
				>>> restrictpatterns({'f1'}, {}, ['f1','f2'], [])
				(set(['f1']), {})
				>>> restrictpatterns({'f1/fc1', 'f3/fc3'}, {}, ['f1','f2'], [])
				(set(['f1/fc1']), {})
				>>> restrictpatterns({'f1_fc1'}, {}, ['f1','f2'], [])
				([], set(['path:.']))
				>>> restrictpatterns({'f1/../f2/fc2'}, {}, ['f1','f2'], [])
				(set(['f2/fc2']), {})
				>>> restrictpatterns({'f1/../f3/fc3'}, {}, ['f1','f2'], [])
				([], set(['path:.']))
				>>> restrictpatterns({'f1/$non_exitent_var'}, {}, ['f1','f2'], [])
				(set(['f1/$non_exitent_var']), {})
				"""
				res_excludes = req_excludes.copy()
				res_excludes.update(repo_excludes)
				indygregUnsubmitted Done We use `set(repo_includes)` below to copy a set. Let's make things consistent. indygreg: We use `set(repo_includes)` below to copy a set. Let's make things consistent.
				if not req_includes:
				res_includes = set(repo_includes)
				elif 'path:.' not in repo_includes:
				res_includes = []
				for req_include in req_includes:
				req_include = util.expandpath(util.normpath(req_include))
				if req_include in repo_includes:
				res_includes.append(req_include)
				continue
				valid = False
				for repo_include in repo_includes:
				if req_include.startswith(repo_include + '/'):
				valid = True
				indygregUnsubmitted Not Done I feel like there's a more efficient mechanism lurking here. But I'm not sure if we care about the performance implications of this function. indygreg: I feel like there's a more efficient mechanism lurking here. But I'm not sure if we care about…
				res_includes.append(req_include)
				break
				if not valid and invalid_includes is not None:
				invalid_includes.append(req_include)
				if len(res_includes) == 0:
				indygregUnsubmitted Done Mutating an argument feels a bit wonky. Should this be returned instead? indygreg: Mutating an argument feels a bit wonky. Should this be returned instead?
				res_excludes = {'path:.'}
				else:
				res_includes = set(res_includes)
				else:
				res_includes = set(req_includes)
				return res_includes, res_excludes

hgext/narrow/narrowtemplates.py

This file was added.

				# narrowtemplates.py - added template keywords for narrow clones
				#
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				from __future__ import absolute_import

				from mercurial import (
				revset,
				templatekw,
				util,
				)

				from . import narrowrevlog

				def _isellipsis(repo, rev):
				if repo.changelog.flags(rev) & narrowrevlog.ELLIPSIS_NODE_FLAG:
				return True
				return False

				def ellipsis(repo, ctx, templ, **args):
				""":ellipsis: String. 'ellipsis' if the change is an ellipsis node,
				else ''."""
				if _isellipsis(repo, ctx.rev()):
				return 'ellipsis'
				return ''

				def outsidenarrow(repo, ctx, templ, **args):
				""":outsidenarrow: String. 'outsidenarrow' if the change affects no
				tracked files, else ''."""
				if util.safehasattr(repo, 'narrowmatch'):
				m = repo.narrowmatch()
				if not any(m(f) for f in ctx.files()):
				return 'outsidenarrow'
				return ''

				def ellipsisrevset(repo, subset, x):
				"""``ellipsis()``
				Changesets that are ellipsis nodes.
				"""
				return subset.filter(lambda r: _isellipsis(repo, r))

				def setup():
				templatekw.keywords['ellipsis'] = ellipsis
				templatekw.keywords['outsidenarrow'] = outsidenarrow

				revset.symbols['ellipsis'] = ellipsisrevset
				revset.safesymbols.add('ellipsis')

hgext/narrow/narrowwirepeer.py

This file was added.

				# narrowwirepeer.py - passes narrow spec with unbundle command
				#
				# Copyright 2017 Google, Inc.
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				from __future__ import absolute_import

				from mercurial.i18n import _
				from mercurial import (
				error,
				extensions,
				hg,
				node,
				)

				from . import narrowspec

				def uisetup():
				def peersetup(ui, peer):
				# We must set up the expansion before reposetup below, since it's used
				# at clone time before we have a repo.
				class expandingpeer(peer.__class__):
				def expandnarrow(self, narrow_include, narrow_exclude, nodes):
				ui.status(_("expanding narrowspec\n"))
				if not self.capable('expandnarrow'):
				raise error.Abort(
				indygregUnsubmitted Done There is no server-side implementation of this capability/command yet. We may want to remove the reference from core. Also, the wire protocol capability should have `exp` or `experimental` in it until we formalize the API. indygreg: There is no server-side implementation of this capability/command yet. We may want to remove…
				'peer does not support expanding narrowspecs')

				hex_nodes = (node.hex(n) for n in nodes)
				new_narrowspec = self._call(
				'expandnarrow',
				includepats=','.join(narrow_include),
				excludepats=','.join(narrow_exclude),
				nodes=','.join(hex_nodes))

				return narrowspec.parseserverpatterns(new_narrowspec)
				peer.__class__ = expandingpeer
				hg.wirepeersetupfuncs.append(peersetup)

				def reposetup(repo):
				def wirereposetup(ui, peer):
				def wrapped(orig, cmd, args, *kwargs):
				if cmd == 'unbundle':
				include, exclude = repo.narrowpats
				kwargs["includepats"] = ','.join(include)
				kwargs["excludepats"] = ','.join(exclude)
				return orig(cmd, args, *kwargs)
				indygregUnsubmitted Not Done This will blindly add `includepats` and `excludepats` as wire protocol arguments to the `unbundle` command. In the Python server, wire protocol arguments have to be defined as part of the command handler or the server will refuse to serve the request. So, this code is missing the server-side declaration of these arguments. And, the client/peer code here is buggy for not conditionally adding these arguments based on the presence of a server capability advertising support for receiving these arguments. indygreg: This will blindly add `includepats` and `excludepats` as wire protocol arguments to the…
				durin42AuthorUnsubmitted Not Done Yeah, this is a mess. I've added a TODO, as I'm not entirely sure how to work around this for now (calling capabilities in here seems...fraught.) durin42: Yeah, this is a mess. I've added a TODO, as I'm not entirely sure how to work around this for…
				extensions.wrapfunction(peer, '_calltwowaystream', wrapped)
				hg.wirepeersetupfuncs.append(wirereposetup)

setup.py

	'mercurial.cffi',			'mercurial.cffi',
	'mercurial.hgweb',			'mercurial.hgweb',
	'mercurial.httpclient',			'mercurial.httpclient',
	'mercurial.pure',			'mercurial.pure',
	'mercurial.thirdparty',			'mercurial.thirdparty',
	'mercurial.thirdparty.attr',			'mercurial.thirdparty.attr',
	'hgext', 'hgext.convert', 'hgext.fsmonitor',			'hgext', 'hgext.convert', 'hgext.fsmonitor',
	'hgext.fsmonitor.pywatchman', 'hgext.highlight',			'hgext.fsmonitor.pywatchman', 'hgext.highlight',
	'hgext.largefiles', 'hgext.lfs', 'hgext.zeroconf', 'hgext3rd',			'hgext.largefiles', 'hgext.lfs', 'hgext.narrow',
				'hgext.zeroconf', 'hgext3rd',
	'hgdemandimport']			'hgdemandimport']

	common_depends = ['mercurial/bitmanipulation.h',			common_depends = ['mercurial/bitmanipulation.h',
	'mercurial/compat.h',			'mercurial/compat.h',
	'mercurial/cext/util.h']			'mercurial/cext/util.h']
	common_include_dirs = ['mercurial']			common_include_dirs = ['mercurial']

	osutil_cflags = []			osutil_cflags = []

tests/narrow-library.sh

This file was added.

				cat >> $HGRCPATH <<EOF
				[extensions]
				narrow=
				[ui]
				ssh=python "$TESTDIR/dummyssh"
				[experimental]
				bundle2-exp = True
				changegroup3 = True
				EOF

tests/test-help.t

	bookmarks create a new bookmark or list existing bookmarks			bookmarks create a new bookmark or list existing bookmarks
	clone make a copy of an existing repository			clone make a copy of an existing repository
	paths show aliases for remote repositories			paths show aliases for remote repositories
	update update working directory (or switch revisions)			update update working directory (or switch revisions)

	Extensions:			Extensions:

	clonebundles advertise pre-generated bundles to seed clones			clonebundles advertise pre-generated bundles to seed clones
				narrow create clones which fetch history data for subset of files
				(EXPERIMENTAL)
	prefixedname matched against word "clone"			prefixedname matched against word "clone"
	relink recreates hardlinks between repository clones			relink recreates hardlinks between repository clones

	Extension Commands:			Extension Commands:

	qclone clone main and patch repository at same time			qclone clone main and patch repository at same time

	Test unfound topic			Test unfound topic

tests/test-narrow-acl.t

This file was added.

				Make a narrow clone then archive it
				$ . "$TESTDIR/narrow-library.sh"

				$ hg init master
				$ cd master

				$ for x in `$TESTDIR/seq.py 3`; do
				> echo $x > "f$x"
				> hg add "f$x"
				> hg commit -m "Add $x"
				> done
				$ cat >> .hg/hgrc << EOF
				> [narrowhgacl]
				> default.includes=f1 f2
				> EOF
				$ hg serve -a localhost -p $HGPORT1 -d --pid-file=hg.pid
				$ cat hg.pid >> "$DAEMON_PIDS"

				$ cd ..
				$ hg clone http://localhost:$HGPORT1 narrowclone1
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 3 changesets with 2 changes to 2 files
				new changesets * (glob)
				updating to branch default
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				The clone directory should only contain f1 and f2
				$ ls -1 narrowclone1 \| sort
				f1
				f2

				Requirements should contain narrowhg
				$ cat narrowclone1/.hg/requires \| grep narrowhg
				narrowhg

				NarrowHG should track f1 and f2
				$ hg -R narrowclone1 tracked
				I path:f1
				I path:f2

tests/test-narrow-archive.t

This file was added.

				Make a narrow clone then archive it
				$ . "$TESTDIR/narrow-library.sh"

				$ hg init master
				$ cd master

				$ for x in `$TESTDIR/seq.py 3`; do
				> echo $x > "f$x"
				> hg add "f$x"
				> hg commit -m "Add $x"
				> done

				$ hg serve -a localhost -p $HGPORT1 -d --pid-file=hg.pid
				$ cat hg.pid >> "$DAEMON_PIDS"

				$ cd ..
				$ hg clone --narrow --include f1 --include f2 http://localhost:$HGPORT1/ narrowclone1
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 3 changesets with 2 changes to 2 files
				new changesets * (glob)
				updating to branch default
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				The tar should only contain f1 and f2
				$ cd narrowclone1
				$ hg archive -t tgz repo.tgz
				$ tar tfz repo.tgz
				repo/f1
				indygregUnsubmitted Not Done I'm not sure if this invocation of `tar` is portable. I'm sure others will fix things later if it breaks tests on other platforms. indygreg: I'm not sure if this invocation of `tar` is portable. I'm sure others will fix things later if…
				durin42AuthorUnsubmitted Not Done Noted. AFAIK this should be portable at least to bsdtar, but probably solaris or someone will eventually complain? durin42: Noted. AFAIK this should be portable at least to bsdtar, but probably solaris or someone will…
				repo/f2

tests/test-narrow-clone-no-ellipsis.t

This file was added.

				$ . "$TESTDIR/narrow-library.sh"

				$ hg init master
				$ cd master
				$ mkdir dir
				$ mkdir dir/src
				$ cd dir/src
				$ for x in `$TESTDIR/seq.py 20`; do echo $x > "f$x"; hg add "f$x"; hg commit -m "Commit src $x"; done
				$ cd ..
				$ mkdir tests
				$ cd tests
				$ for x in `$TESTDIR/seq.py 20`; do echo $x > "t$x"; hg add "t$x"; hg commit -m "Commit test $x"; done
				$ cd ../../..

				narrow clone a file, f10

				$ hg clone --narrow ssh://user@dummy/master narrow --noupdate --include "dir/src/f10"
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 40 changesets with 1 changes to 1 files
				new changesets : (glob)
				$ cd narrow
				$ cat .hg/requires \| grep -v generaldelta
				dotencode
				fncache
				narrowhg
				revlogv1
				store

				$ cat .hg/narrowspec
				[includes]
				path:dir/src/f10
				[excludes]
				$ hg update
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ find * \| sort
				dir
				dir/src
				dir/src/f10
				$ cat dir/src/f10
				10

				$ cd ..

				narrow clone a directory, tests/, except tests/t19

				$ hg clone --narrow ssh://user@dummy/master narrowdir --noupdate --include "dir/tests/" --exclude "dir/tests/t19"
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 40 changesets with 19 changes to 19 files
				new changesets : (glob)
				$ cd narrowdir
				$ cat .hg/narrowspec
				[includes]
				path:dir/tests
				[excludes]
				path:dir/tests/t19
				$ hg update
				19 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ find * \| sort
				dir
				dir/tests
				dir/tests/t1
				dir/tests/t10
				dir/tests/t11
				dir/tests/t12
				dir/tests/t13
				dir/tests/t14
				dir/tests/t15
				dir/tests/t16
				dir/tests/t17
				dir/tests/t18
				dir/tests/t2
				dir/tests/t20
				dir/tests/t3
				dir/tests/t4
				dir/tests/t5
				dir/tests/t6
				dir/tests/t7
				dir/tests/t8
				dir/tests/t9

				$ cd ..

				narrow clone everything but a directory (tests/)

				$ hg clone --narrow ssh://user@dummy/master narrowroot --noupdate --exclude "dir/tests"
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 40 changesets with 20 changes to 20 files
				new changesets : (glob)
				$ cd narrowroot
				$ cat .hg/narrowspec
				[includes]
				path:.
				[excludes]
				path:dir/tests
				$ hg update
				20 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ find * \| sort
				dir
				dir/src
				dir/src/f1
				dir/src/f10
				dir/src/f11
				dir/src/f12
				dir/src/f13
				dir/src/f14
				dir/src/f15
				dir/src/f16
				dir/src/f17
				dir/src/f18
				dir/src/f19
				dir/src/f2
				dir/src/f20
				dir/src/f3
				dir/src/f4
				dir/src/f5
				dir/src/f6
				dir/src/f7
				dir/src/f8
				dir/src/f9

				$ cd ..

tests/test-narrow-clone-non-narrow-server.t

This file was added.

				Test attempting a narrow clone against a server that doesn't support narrowhg.

				$ . "$TESTDIR/narrow-library.sh"

				$ hg init master
				$ cd master

				$ for x in `$TESTDIR/seq.py 10`; do
				> echo $x > "f$x"
				> hg add "f$x"
				> hg commit -m "Add $x"
				> done

				$ hg serve -a localhost -p $HGPORT1 --config extensions.narrow=! -d \
				> --pid-file=hg.pid
				$ cat hg.pid >> "$DAEMON_PIDS"
				$ hg serve -a localhost -p $HGPORT2 -d --pid-file=hg.pid
				$ cat hg.pid >> "$DAEMON_PIDS"

				Verify that narrow is advertised in the bundle2 capabilities:
				$ echo capabilities \| hg -R . serve --stdio \| \
				> python -c "import sys, urllib; print urllib.unquote_plus(list(sys.stdin)[1])" \| grep narrow
				narrow=v0

				indygregUnsubmitted Done The `capabilities` command isn't used by the SSH/stdio protocol. This should be using `hello` instead. I suspect I'll soon write a patch that doesn't expose the `capabilities` command to the SSH protocol because it isn't supposed to be used... indygreg: The `capabilities` command isn't used by the SSH/stdio protocol. This should be using `hello`…
				$ cd ..

				$ hg clone --narrow --include f1 http://localhost:$HGPORT1/ narrowclone
				requesting all changes
				abort: server doesn't support narrow clones
				[255]

				Make a narrow clone (via HGPORT2), then try to narrow and widen
				into it (from HGPORT1) to prove that narrowing is fine and widening fails
				gracefully:
				$ hg clone -r 0 --narrow --include f1 http://localhost:$HGPORT2/ narrowclone
				adding changesets
				adding manifests
				adding file changes
				added 1 changesets with 1 changes to 1 files
				new changesets * (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrowclone
				$ hg tracked --addexclude f2 http://localhost:$HGPORT1/
				comparing with http://localhost:$HGPORT1/
				searching for changes
				looking for local changes to affected paths
				$ hg tracked --addinclude f1 http://localhost:$HGPORT1/
				comparing with http://localhost:$HGPORT1/
				searching for changes
				no changes found
				abort: server doesn't support narrow clones
				[255]

tests/test-narrow-clone-nonlinear.t

This file was added.

				Testing narrow clones when changesets modifying a matching file exist on
				multiple branches

				$ . "$TESTDIR/narrow-library.sh"

				$ hg init master
				$ cd master
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF

				$ hg branch default
				marked working directory as branch default
				(branches are permanent and global, did you want a bookmark?)
				$ for x in `$TESTDIR/seq.py 10`; do
				> echo $x > "f$x"
				> hg add "f$x"
				> hg commit -m "Add $x"
				> done

				$ hg branch release-v1
				marked working directory as branch release-v1
				(branches are permanent and global, did you want a bookmark?)
				$ hg commit -m "Start release for v1"

				$ hg update default
				0 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ for x in `$TESTDIR/seq.py 10`; do
				> echo "$x v2" > "f$x"
				> hg commit -m "Update $x to v2"
				> done

				$ hg update release-v1
				10 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ hg branch release-v1
				marked working directory as branch release-v1
				$ for x in `$TESTDIR/seq.py 1 5`; do
				> echo "$x v1 hotfix" > "f$x"
				> hg commit -m "Hotfix $x in v1"
				> done

				$ hg update default
				10 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ hg branch release-v2
				marked working directory as branch release-v2
				$ hg commit -m "Start release for v2"

				$ hg update default
				0 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ hg branch default
				marked working directory as branch default
				$ for x in `$TESTDIR/seq.py 10`; do
				> echo "$x v3" > "f$x"
				> hg commit -m "Update $x to v3"
				> done

				$ hg update release-v2
				10 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ hg branch release-v2
				marked working directory as branch release-v2
				$ for x in `$TESTDIR/seq.py 4 9`; do
				> echo "$x v2 hotfix" > "f$x"
				> hg commit -m "Hotfix $x in v2"
				> done

				$ hg heads -T '{rev} <- {p1rev} ({branch}): {desc}\n'
				42 <- 41 (release-v2): Hotfix 9 in v2
				36 <- 35 (default): Update 10 to v3
				25 <- 24 (release-v1): Hotfix 5 in v1

				$ cd ..

				We now have 3 branches: default, which has v3 of all files, release-v1 which
				has v1 of all files, and release-v2 with v2 of all files.

				Narrow clone which should get all branches

				$ hg clone --narrow ssh://user@dummy/master narrow --include "f5"
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 12 changesets with 5 changes to 1 files (+2 heads)
				new changesets : (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow
				$ hg log -G -T "{if(ellipsis, '...')}{node\|short} ({branch}): {desc}\n"
				o ...031f516143fe (release-v2): Hotfix 9 in v2
				\|
				o 9cd7f7bb9ca1 (release-v2): Hotfix 5 in v2
				\|
				o ...37bbc88f3ef0 (release-v2): Hotfix 4 in v2
				\|
				\| @ ...dae2f368ca07 (default): Update 10 to v3
				\| \|
				\| o 9c224e89cb31 (default): Update 5 to v3
				\| \|
				\| o ...04fb59c7c9dc (default): Update 4 to v3
				\|/
				\| o b2253e82401f (release-v1): Hotfix 5 in v1
				\| \|
				\| o ...960ac37d74fd (release-v1): Hotfix 4 in v1
				\| \|
				o \| 986298e3f347 (default): Update 5 to v2
				\| \|
				o \| ...75d539c667ec (default): Update 4 to v2
				\|/
				o 04c71bd5707f (default): Add 5
				\|
				o ...881b3891d041 (default): Add 4


				Narrow clone the first file, hitting edge condition where unaligned
				changeset and manifest revnums cross branches.

				$ hg clone --narrow ssh://user@dummy/master narrow --include "f1"
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 10 changesets with 4 changes to 1 files (+2 heads)
				new changesets : (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow
				$ hg log -G -T "{if(ellipsis, '...')}{node\|short} ({branch}): {desc}\n"
				o ...031f516143fe (release-v2): Hotfix 9 in v2
				\|
				\| @ ...dae2f368ca07 (default): Update 10 to v3
				\| \|
				\| o 1f5d184b8e96 (default): Update 1 to v3
				\|/
				\| o ...b2253e82401f (release-v1): Hotfix 5 in v1
				\| \|
				\| o 133502f6b7e5 (release-v1): Hotfix 1 in v1
				\| \|
				o \| ...79165c83d644 (default): Update 10 to v2
				\| \|
				o \| c7b7a5f2f088 (default): Update 1 to v2
				\| \|
				\| o ...f0531a3db7a9 (release-v1): Start release for v1
				\|/
				o ...6a3f0f0abef3 (default): Add 10
				\|
				o e012ac15eaaa (default): Add 1

tests/test-narrow-clone.t

This file was added.

				$ . "$TESTDIR/narrow-library.sh"

				$ hg init master
				$ cd master
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF
				$ mkdir dir
				$ mkdir dir/src
				$ cd dir/src
				$ for x in `$TESTDIR/seq.py 20`; do echo $x > "f$x"; hg add "f$x"; hg commit -m "Commit src $x"; done
				$ cd ..
				$ mkdir tests
				$ cd tests
				$ for x in `$TESTDIR/seq.py 20`; do echo $x > "t$x"; hg add "t$x"; hg commit -m "Commit test $x"; done
				$ cd ../../..

				narrow clone a file, f10

				$ hg clone --narrow ssh://user@dummy/master narrow --noupdate --include "dir/src/f10"
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 3 changesets with 1 changes to 1 files
				new changesets : (glob)
				$ cd narrow
				$ cat .hg/requires \| grep -v generaldelta
				dotencode
				fncache
				narrowhg
				revlogv1
				store

				$ cat .hg/narrowspec
				[includes]
				path:dir/src/f10
				[excludes]
				$ hg tracked
				I path:dir/src/f10
				$ hg update
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ find * \| sort
				dir
				dir/src
				dir/src/f10
				$ cat dir/src/f10
				10

				$ cd ..

				narrow clone with a newline should fail

				$ hg clone --narrow ssh://user@dummy/master narrow_fail --noupdate --include 'dir/src/f10
				> '
				requesting all changes
				abort: newlines are not allowed in narrowspec paths
				[255]

				narrow clone a directory, tests/, except tests/t19

				$ hg clone --narrow ssh://user@dummy/master narrowdir --noupdate --include "dir/tests/" --exclude "dir/tests/t19"
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 21 changesets with 19 changes to 19 files
				new changesets : (glob)
				$ cd narrowdir
				$ cat .hg/narrowspec
				[includes]
				path:dir/tests
				[excludes]
				path:dir/tests/t19
				$ hg tracked
				I path:dir/tests
				X path:dir/tests/t19
				$ hg update
				19 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ find * \| sort
				dir
				dir/tests
				dir/tests/t1
				dir/tests/t10
				dir/tests/t11
				dir/tests/t12
				dir/tests/t13
				dir/tests/t14
				dir/tests/t15
				dir/tests/t16
				dir/tests/t17
				dir/tests/t18
				dir/tests/t2
				dir/tests/t20
				dir/tests/t3
				dir/tests/t4
				dir/tests/t5
				dir/tests/t6
				dir/tests/t7
				dir/tests/t8
				dir/tests/t9

				$ cd ..

				narrow clone everything but a directory (tests/)

				$ hg clone --narrow ssh://user@dummy/master narrowroot --noupdate --exclude "dir/tests"
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 21 changesets with 20 changes to 20 files
				new changesets : (glob)
				$ cd narrowroot
				$ cat .hg/narrowspec
				[includes]
				path:.
				[excludes]
				path:dir/tests
				$ hg tracked
				I path:.
				X path:dir/tests
				$ hg update
				20 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ find * \| sort
				dir
				dir/src
				dir/src/f1
				dir/src/f10
				dir/src/f11
				dir/src/f12
				dir/src/f13
				dir/src/f14
				dir/src/f15
				dir/src/f16
				dir/src/f17
				dir/src/f18
				dir/src/f19
				dir/src/f2
				dir/src/f20
				dir/src/f3
				dir/src/f4
				dir/src/f5
				dir/src/f6
				dir/src/f7
				dir/src/f8
				dir/src/f9

				$ cd ..

				narrow clone no paths at all

				$ hg clone --narrow ssh://user@dummy/master narrowempty --noupdate
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 1 changesets with 0 changes to 0 files
				new changesets * (glob)
				$ cd narrowempty
				$ hg tracked
				$ hg update
				0 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ ls

				$ cd ..

				simple clone
				$ hg clone ssh://user@dummy/master simpleclone
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 40 changesets with 40 changes to 40 files
				new changesets * (glob)
				updating to branch default
				40 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd simpleclone
				$ find * \| sort
				dir
				dir/src
				dir/src/f1
				dir/src/f10
				dir/src/f11
				dir/src/f12
				dir/src/f13
				dir/src/f14
				dir/src/f15
				dir/src/f16
				dir/src/f17
				dir/src/f18
				dir/src/f19
				dir/src/f2
				dir/src/f20
				dir/src/f3
				dir/src/f4
				dir/src/f5
				dir/src/f6
				dir/src/f7
				dir/src/f8
				dir/src/f9
				dir/tests
				dir/tests/t1
				dir/tests/t10
				dir/tests/t11
				dir/tests/t12
				dir/tests/t13
				dir/tests/t14
				dir/tests/t15
				dir/tests/t16
				dir/tests/t17
				dir/tests/t18
				dir/tests/t19
				dir/tests/t2
				dir/tests/t20
				dir/tests/t3
				dir/tests/t4
				dir/tests/t5
				dir/tests/t6
				dir/tests/t7
				dir/tests/t8
				dir/tests/t9

				$ cd ..

tests/test-narrow-commit-tree.t

This file was added.

				$ cd $TESTDIR && python $RUNTESTDIR/run-tests.py \
				> --extra-config-opt experimental.treemanifest=1 test-narrow-commit.t 2>&1 \| \
				> grep -v 'unexpected mercurial lib' \| egrep -v '\(expected'

				--- */tests/test-narrow-commit.t (glob)
				+++ */tests/test-narrow-commit.t.err (glob)
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)
				created new head
				$ hg files -r .
				inside/f1
				- outside/f1
				+ outside/
				Some filesystems (notably FAT/exFAT only store timestamps with 2
				seconds of precision, so by sleeping for 3 seconds, we can ensure that
				the timestamps of files stored by dirstate will appear older than the

				ERROR: test-narrow-commit.t output changed
				!
				Failed test-narrow-commit.t: output changed
				# Ran 1 tests, 0 skipped, 1 failed.
				python hash seed: * (glob)

tests/test-narrow-commit.t

This file was added.

				$ . "$TESTDIR/narrow-library.sh"

				create full repo

				$ hg init master
				$ cd master

				$ mkdir inside
				$ echo inside > inside/f1
				$ mkdir outside
				$ echo outside > outside/f1
				$ hg ci -Aqm 'initial'

				$ echo modified > inside/f1
				$ hg ci -qm 'modify inside'

				$ echo modified > outside/f1
				$ hg ci -qm 'modify outside'

				$ cd ..

				$ hg clone --narrow ssh://user@dummy/master narrow --include inside
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 3 changesets with 2 changes to 1 files
				new changesets : (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow

				$ hg update -q 0

				Can not modify dirstate outside

				$ mkdir outside
				$ touch outside/f1
				$ hg debugwalk -I 'relglob:f1'
				matcher: <includematcher includes='(?:(?:\|.*/)f1(?:/\|$))'>
				f inside/f1 inside/f1
				$ hg add outside/f1
				abort: cannot track 'outside/f1' - it is outside the narrow clone
				[255]
				$ touch outside/f3
				$ hg add outside/f3
				abort: cannot track 'outside/f3' - it is outside the narrow clone
				[255]
				$ rm -r outside

				Can modify dirstate inside

				$ echo modified > inside/f1
				$ touch inside/f3
				$ hg add inside/f3
				$ hg status
				M inside/f1
				A inside/f3
				$ hg revert -qC .
				$ rm inside/f3

				Can commit changes inside. Leaves outside unchanged.

				$ hg update -q 'desc("initial")'
				$ echo modified2 > inside/f1
				$ hg commit -m 'modify inside/f1'
				created new head
				$ hg files -r .
				inside/f1
				outside/f1
				Some filesystems (notably FAT/exFAT only store timestamps with 2
				seconds of precision, so by sleeping for 3 seconds, we can ensure that
				the timestamps of files stored by dirstate will appear older than the
				dirstate file, and therefore we'll be able to get stable output from
				debugdirstate. If we don't do this, the test can be slightly flaky.
				$ sleep 3
				$ hg status
				$ hg debugdirstate --nodates
				n 644 10 set inside/f1

tests/test-narrow-copies.t

This file was added.


				$ . "$TESTDIR/narrow-library.sh"

				create full repo

				$ hg init master
				$ cd master

				$ mkdir inside
				$ echo inside > inside/f1
				$ mkdir outside
				$ echo outside > outside/f2
				$ hg ci -Aqm 'initial'

				$ hg mv outside/f2 inside/f2
				$ hg ci -qm 'move f2 from outside'

				$ echo modified > inside/f2
				$ hg ci -qm 'modify inside/f2'

				$ cd ..

				$ hg clone --narrow ssh://user@dummy/master narrow --include inside
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 3 changesets with 3 changes to 2 files
				new changesets : (glob)
				updating to branch default
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow

				$ hg co 'desc("move f2")'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ hg status
				$ hg diff
				$ hg diff --change . --git
				diff --git a/inside/f2 b/inside/f2
				new file mode 100644
				--- /dev/null
				+++ b/inside/f2
				@@ -0,0 +1,1 @@
				+outside

				$ hg log --follow inside/f2 -r tip
				changeset: 2:bcfb756e0ca9
				tag: tip
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: modify inside/f2

				changeset: 1:5a016133b2bb
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: move f2 from outside

tests/test-narrow-debugcommands.t

This file was added.

				$ . "$TESTDIR/narrow-library.sh"
				$ hg init repo
				$ cd repo
				$ cat << EOF > .hg/narrowspec
				> [includes]
				> path:foo
				> [excludes]
				> EOF
				$ echo treemanifest >> .hg/requires
				$ echo narrowhg >> .hg/requires
				$ mkdir -p foo/bar
				$ echo b > foo/f
				$ echo c > foo/bar/f
				$ hg commit -Am hi
				adding foo/bar/f
				adding foo/f
				$ hg debugindex -m
				rev offset length delta linkrev nodeid p1 p2
				0 0 47 -1 0 14a5d056d75a 000000000000 000000000000
				$ hg debugindex --dir foo
				rev offset length delta linkrev nodeid p1 p2
				0 0 77 -1 0 e635c7857aef 000000000000 000000000000
				$ hg debugindex --dir foo/
				rev offset length delta linkrev nodeid p1 p2
				0 0 77 -1 0 e635c7857aef 000000000000 000000000000
				$ hg debugindex --dir foo/bar
				rev offset length delta linkrev nodeid p1 p2
				0 0 44 -1 0 e091d4224761 000000000000 000000000000
				$ hg debugindex --dir foo/bar/
				rev offset length delta linkrev nodeid p1 p2
				0 0 44 -1 0 e091d4224761 000000000000 000000000000
				$ hg debugdata -m 0
				foo\x00e635c7857aef92ac761ce5741a99da159abbbb24t (esc)
				$ hg debugdata --dir foo 0
				bar\x00e091d42247613adff5d41b67f15fe7189ee97b39t (esc)
				f\x001e88685f5ddec574a34c70af492f95b6debc8741 (esc)
				$ hg debugdata --dir foo/ 0
				bar\x00e091d42247613adff5d41b67f15fe7189ee97b39t (esc)
				f\x001e88685f5ddec574a34c70af492f95b6debc8741 (esc)
				$ hg debugdata --dir foo/bar 0
				f\x00149da44f2a4e14f488b7bd4157945a9837408c00 (esc)
				$ hg debugdata --dir foo/bar/ 0
				f\x00149da44f2a4e14f488b7bd4157945a9837408c00 (esc)

tests/test-narrow-debugrebuilddirstate.t

This file was added.

				$ . "$TESTDIR/narrow-library.sh"
				$ hg init master
				$ cd master
				$ echo treemanifest >> .hg/requires
				$ echo 'contents of file' > file
				$ mkdir foo
				$ echo 'contents of foo/bar' > foo/bar
				$ hg ci -Am 'some change'
				adding file
				adding foo/bar

				$ cd ..
				$ hg clone --narrow ssh://user@dummy/master copy --include=foo
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 1 changesets with 1 changes to 1 files
				new changesets * (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd copy

				$ hg debugdirstate
				n * 20 unset foo/bar (glob)
				$ mv .hg/dirstate .hg/old_dirstate
				$ dd bs=40 count=1 if=.hg/old_dirstate of=.hg/dirstate 2>/dev/null
				$ hg debugdirstate
				$ hg debugrebuilddirstate
				$ hg debugdirstate
				n * * unset foo/bar (glob)

tests/test-narrow-exchange-merges.t

This file was added.


				$ . "$TESTDIR/narrow-library.sh"

				create full repo

				$ hg init master
				$ cd master
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF

				$ mkdir inside
				$ echo 1 > inside/f
				$ hg commit -Aqm 'initial inside'

				$ mkdir outside
				$ echo 1 > outside/f
				$ hg commit -Aqm 'initial outside'

				$ echo 2a > outside/f
				$ hg commit -Aqm 'outside 2a'
				$ echo 3 > inside/f
				$ hg commit -Aqm 'inside 3'
				$ echo 4a > outside/f
				$ hg commit -Aqm 'outside 4a'
				$ hg update '.~3'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				$ echo 2b > outside/f
				$ hg commit -Aqm 'outside 2b'
				$ echo 3 > inside/f
				$ hg commit -Aqm 'inside 3'
				$ echo 4b > outside/f
				$ hg commit -Aqm 'outside 4b'
				$ hg update '.~3'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				$ echo 2c > outside/f
				$ hg commit -Aqm 'outside 2c'
				$ echo 3 > inside/f
				$ hg commit -Aqm 'inside 3'
				$ echo 4c > outside/f
				$ hg commit -Aqm 'outside 4c'
				$ hg update '.~3'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				$ echo 2d > outside/f
				$ hg commit -Aqm 'outside 2d'
				$ echo 3 > inside/f
				$ hg commit -Aqm 'inside 3'
				$ echo 4d > outside/f
				$ hg commit -Aqm 'outside 4d'

				$ hg update -r 'desc("outside 4a")'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ hg merge -r 'desc("outside 4b")' 2>&1 \| egrep -v '(warning:\|incomplete!)'
				merging outside/f
				0 files updated, 0 files merged, 0 files removed, 1 files unresolved
				use 'hg resolve' to retry unresolved file merges or 'hg merge --abort' to abandon
				$ echo 5 > outside/f
				$ rm outside/f.orig
				$ hg resolve --mark outside/f
				(no more unresolved files)
				$ hg commit -m 'merge a/b 5'
				$ echo 6 > outside/f
				$ hg commit -Aqm 'outside 6'

				$ hg merge -r 'desc("outside 4c")' 2>&1 \| egrep -v '(warning:\|incomplete!)'
				merging outside/f
				0 files updated, 0 files merged, 0 files removed, 1 files unresolved
				use 'hg resolve' to retry unresolved file merges or 'hg merge --abort' to abandon
				$ echo 7 > outside/f
				$ rm outside/f.orig
				$ hg resolve --mark outside/f
				(no more unresolved files)
				$ hg commit -Aqm 'merge a/b/c 7'
				$ echo 8 > outside/f
				$ hg commit -Aqm 'outside 8'

				$ hg merge -r 'desc("outside 4d")' 2>&1 \| egrep -v '(warning:\|incomplete!)'
				merging outside/f
				0 files updated, 0 files merged, 0 files removed, 1 files unresolved
				use 'hg resolve' to retry unresolved file merges or 'hg merge --abort' to abandon
				$ echo 9 > outside/f
				$ rm outside/f.orig
				$ hg resolve --mark outside/f
				(no more unresolved files)
				$ hg commit -Aqm 'merge a/b/c/d 9'
				$ echo 10 > outside/f
				$ hg commit -Aqm 'outside 10'

				$ echo 11 > inside/f
				$ hg commit -Aqm 'inside 11'
				$ echo 12 > outside/f
				$ hg commit -Aqm 'outside 12'

				$ hg log -G -T '{rev} {node\|short} {desc}\n'
				@ 21 8d874d57adea outside 12
				\|
				o 20 7ef88b4dd4fa inside 11
				\|
				o 19 2a20009de83e outside 10
				\|
				o 18 3ac1f5779de3 merge a/b/c/d 9
				\|\
				\| o 17 38a9c2f7e546 outside 8
				\| \|
				\| o 16 094aa62fc898 merge a/b/c 7
				\| \|\
				\| \| o 15 f29d083d32e4 outside 6
				\| \| \|
				\| \| o 14 2dc11382541d merge a/b 5
				\| \| \|\
				o \| \| \| 13 27d07ef97221 outside 4d
				\| \| \| \|
				o \| \| \| 12 465567bdfb2d inside 3
				\| \| \| \|
				o \| \| \| 11 d1c61993ec83 outside 2d
				\| \| \| \|
				\| o \| \| 10 56859a8e33b9 outside 4c
				\| \| \| \|
				\| o \| \| 9 bb96a08b062a inside 3
				\| \| \| \|
				\| o \| \| 8 b844052e7b3b outside 2c
				\|/ / /
				\| \| o 7 9db2d8fcc2a6 outside 4b
				\| \| \|
				\| \| o 6 6418167787a6 inside 3
				\| \| \|
				+---o 5 77344f344d83 outside 2b
				\| \|
				\| o 4 9cadde08dc9f outside 4a
				\| \|
				\| o 3 019ef06f125b inside 3
				\| \|
				\| o 2 75e40c075a19 outside 2a
				\|/
				o 1 906d6c682641 initial outside
				\|
				o 0 9f8e82b51004 initial inside


				Now narrow clone this and get a hopefully correct graph

				$ cd ..
				$ hg clone --narrow ssh://user@dummy/master narrow --include inside
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 14 changesets with 3 changes to 1 files
				new changesets : (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow

				To make updating the tests easier, we print the emitted nodes
				sorted. This makes it easier to identify when the same node structure
				has been emitted, just in a different order.

				$ hg log -T '{if(ellipsis,"...")}{node\|short} {p1node\|short} {p2node\|short} {desc}\n' \| sort
				...094aa62fc898 6418167787a6 bb96a08b062a merge a/b/c 7
				...2a20009de83e 019ef06f125b 3ac1f5779de3 outside 10
				...3ac1f5779de3 465567bdfb2d 094aa62fc898 merge a/b/c/d 9
				...75e40c075a19 9f8e82b51004 000000000000 outside 2a
				...77344f344d83 9f8e82b51004 000000000000 outside 2b
				...8d874d57adea 7ef88b4dd4fa 000000000000 outside 12
				...b844052e7b3b 9f8e82b51004 000000000000 outside 2c
				...d1c61993ec83 9f8e82b51004 000000000000 outside 2d
				019ef06f125b 75e40c075a19 000000000000 inside 3
				465567bdfb2d d1c61993ec83 000000000000 inside 3
				6418167787a6 77344f344d83 000000000000 inside 3
				7ef88b4dd4fa 2a20009de83e 000000000000 inside 11
				9f8e82b51004 000000000000 000000000000 initial inside
				bb96a08b062a b844052e7b3b 000000000000 inside 3

				But seeing the graph is also nice:
				$ hg log -G -T '{if(ellipsis,"...")}{node\|short} {desc}\n'
				@ ...8d874d57adea outside 12
				\|
				o 7ef88b4dd4fa inside 11
				\|
				o ...2a20009de83e outside 10
				\|\
				\| o ...3ac1f5779de3 merge a/b/c/d 9
				\| \|\
				\| \| o ...094aa62fc898 merge a/b/c 7
				\| \| \|\
				\| o \| \| 465567bdfb2d inside 3
				\| \| \| \|
				\| o \| \| ...d1c61993ec83 outside 2d
				\| \| \| \|
				\| \| \| o bb96a08b062a inside 3
				\| \| \| \|
				\| +---o ...b844052e7b3b outside 2c
				\| \| \|
				\| \| o 6418167787a6 inside 3
				\| \| \|
				\| \| o ...77344f344d83 outside 2b
				\| \|/
				o \| 019ef06f125b inside 3
				\| \|
				o \| ...75e40c075a19 outside 2a
				\|/
				o 9f8e82b51004 initial inside

tests/test-narrow-exchange.t

This file was added.


				$ . "$TESTDIR/narrow-library.sh"

				create full repo

				$ hg init master
				$ cd master
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF

				$ mkdir inside
				$ echo 1 > inside/f
				$ mkdir inside2
				$ echo 1 > inside2/f
				$ mkdir outside
				$ echo 1 > outside/f
				$ hg ci -Aqm 'initial'

				$ echo 2 > inside/f
				$ hg ci -qm 'inside 2'

				$ echo 2 > inside2/f
				$ hg ci -qm 'inside2 2'

				$ echo 2 > outside/f
				$ hg ci -qm 'outside 2'

				$ cd ..

				$ hg clone --narrow ssh://user@dummy/master narrow --include inside
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 3 changesets with 2 changes to 1 files
				new changesets : (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved

				$ hg clone --narrow ssh://user@dummy/master narrow2 --include inside --include inside2
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 4 changesets with 4 changes to 2 files
				new changesets : (glob)
				updating to branch default
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				Can push to wider repo if change does not affect paths in wider repo that are
				not also in narrower repo

				$ cd narrow
				$ echo 3 > inside/f
				$ hg ci -m 'inside 3'
				$ hg push ssh://user@dummy/narrow2
				pushing to ssh://user@dummy/narrow2
				searching for changes
				remote: adding changesets
				remote: adding manifests
				remote: adding file changes
				remote: added 1 changesets with 1 changes to 1 files

				Can push to narrower repo if change affects only paths within remote's
				narrow spec

				$ cd ../narrow2
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF
				$ hg co -r 'desc("inside 3")'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ echo 4 > inside/f
				$ hg ci -m 'inside 4'
				$ hg push ssh://user@dummy/narrow
				pushing to ssh://user@dummy/narrow
				searching for changes
				remote: adding changesets
				remote: adding manifests
				remote: adding file changes
				remote: added 1 changesets with 1 changes to 1 files

				Can push to narrow repo if change affects only paths outside remote's
				narrow spec

				$ echo 3 > inside2/f
				$ hg ci -m 'inside2 3'
				TODO: this should be successful
				$ hg push ssh://user@dummy/narrow
				pushing to ssh://user@dummy/narrow
				searching for changes
				remote: adding changesets
				remote: adding manifests
				remote: adding file changes
				remote: transaction abort!
				remote: rollback completed
				remote: abort: data/inside2/f.i@4a1aa07735e6: unknown parent!
				abort: stream ended unexpectedly (got 0 bytes, expected 4)
				[255]

				Can pull from wider repo if change affects only paths outside remote's
				narrow spec
				$ echo 4 > inside2/f
				$ hg ci -m 'inside2 4'
				$ hg log -G -T '{rev} {node\|short} {files}\n'
				@ 7 d78a96df731d inside2/f
				\|
				o 6 8c26f5218962 inside2/f
				\|
				o 5 ba3480e2f9de inside/f
				\|
				o 4 4e5edd526618 inside/f
				\|
				o 3 81e7e07b7ab0 outside/f
				\|
				o 2 f3993b8c0c2b inside2/f
				\|
				o 1 8cd66ca966b4 inside/f
				\|
				o 0 c8057d6f53ab inside/f inside2/f outside/f

				$ cd ../narrow
				$ hg log -G -T '{rev} {node\|short} {files}\n'
				o 4 ba3480e2f9de inside/f
				\|
				@ 3 4e5edd526618 inside/f
				\|
				o 2 81e7e07b7ab0 outside/f
				\|
				o 1 8cd66ca966b4 inside/f
				\|
				o 0 c8057d6f53ab inside/f inside2/f outside/f

				$ hg pull ssh://user@dummy/narrow2
				pulling from ssh://user@dummy/narrow2
				searching for changes
				remote: abort: unable to resolve parent while packing 'data/inside2/f.i' 3 for changeset 5 (?)
				adding changesets
				remote: abort: unexpected error: unable to resolve parent while packing 'data/inside2/f.i' 3 for changeset 5
				transaction abort!
				rollback completed
				abort: pull failed on remote
				[255]

				Check that the resulting history is valid in the full repo

				$ cd ../narrow2
				$ hg push ssh://user@dummy/master
				pushing to ssh://user@dummy/master
				searching for changes
				remote: adding changesets
				remote: adding manifests
				remote: adding file changes
				remote: added 4 changesets with 4 changes to 2 files
				$ cd ../master
				$ hg verify
				checking changesets
				checking manifests
				crosschecking files in changesets and manifests
				checking files
				3 files, 8 changesets, 10 total revisions

				Can not push to wider repo if change affects paths in wider repo that are
				not also in narrower repo
				$ cd ../master
				$ hg co -r 'desc("inside2 4")'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ echo 5 > inside2/f
				$ hg ci -m 'inside2 5'
				$ hg log -G -T '{rev} {node\|short} {files}\n'
				@ 8 5970befb64ba inside2/f
				\|
				o 7 d78a96df731d inside2/f
				\|
				o 6 8c26f5218962 inside2/f
				\|
				o 5 ba3480e2f9de inside/f
				\|
				o 4 4e5edd526618 inside/f
				\|
				o 3 81e7e07b7ab0 outside/f
				\|
				o 2 f3993b8c0c2b inside2/f
				\|
				o 1 8cd66ca966b4 inside/f
				\|
				o 0 c8057d6f53ab inside/f inside2/f outside/f

				$ cd ../narrow
				$ hg pull
				pulling from ssh://user@dummy/master
				searching for changes
				adding changesets
				adding manifests
				adding file changes
				added 1 changesets with 0 changes to 0 files
				new changesets * (glob)
				(run 'hg update' to get a working copy)
				TODO: this should tell the user that their narrow clone does not have the
				necessary content to be able to push to the target
				$ hg push ssh://user@dummy/narrow2
				pushing to ssh://user@dummy/narrow2
				searching for changes
				remote has heads on branch 'default' that are not known locally: d78a96df731d
				abort: push creates new remote head 5970befb64ba!
				(pull and merge or see 'hg help push' for details about pushing new heads)
				[255]

tests/test-narrow-expanddirstate.t

This file was added.

				$ . "$TESTDIR/narrow-library.sh"

				$ hg init master
				$ cd master

				$ mkdir inside
				$ echo inside > inside/f1
				$ mkdir outside
				$ echo outside > outside/f2
				$ mkdir patchdir
				$ echo patch_this > patchdir/f3
				$ hg ci -Aqm 'initial'

				$ cd ..

				$ hg clone --narrow ssh://user@dummy/master narrow --include inside
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 1 changesets with 1 changes to 1 files
				new changesets dff6a2a6d433
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved

				$ cd narrow

				$ mkdir outside
				$ echo other_contents > outside/f2
				$ grep outside .hg/narrowspec
				[1]
				$ grep outside .hg/dirstate
				[1]
				$ hg status

				`hg status` did not add outside.
				$ grep outside .hg/narrowspec
				[1]
				$ grep outside .hg/dirstate
				[1]

				Unfortunately this is not really a candidate for adding to narrowhg proper,
				since it depends on some other source for providing the manifests (when using
				treemanifests) and file contents. Something like a virtual filesystem and/or
				remotefilelog. We want to be useful when not using those systems, so we do not
				have this method available in narrowhg proper at the moment.
				$ cat > "$TESTTMP/expand_extension.py" <<EOF
				> import os
				> import sys
				>
				> from mercurial import extensions
				> from mercurial import localrepo
				> from mercurial import match as matchmod
				> from mercurial import patch
				> from mercurial import util as hgutil
				>
				> def expandnarrowspec(ui, repo, newincludes=None):
				> if not newincludes:
				> return
				> import sys
				> newincludes = set([newincludes])
				> narrowhg = extensions.find('narrow')
				> includes, excludes = repo.narrowpats
				> currentmatcher = narrowhg.narrowspec.match(repo.root, includes, excludes)
				> includes = includes \| newincludes
				> if not repo.currenttransaction():
				> ui.develwarn('expandnarrowspec called outside of transaction!')
				> repo.setnarrowpats(includes, excludes)
				> newmatcher = narrowhg.narrowspec.match(repo.root, includes, excludes)
				> added = matchmod.differencematcher(newmatcher, currentmatcher)
				> for f in repo['.'].manifest().walk(added):
				> repo.dirstate.normallookup(f)
				>
				> def makeds(ui, repo):
				> def wrapds(orig, self):
				> ds = orig(self)
				> class expandingdirstate(ds.__class__):
				> # Mercurial 4.4 uses this version.
				> @hgutil.propertycache
				> def _map(self):
				> ret = super(expandingdirstate, self)._map
				> with repo.wlock(), repo.lock(), repo.transaction(
				> 'expandnarrowspec'):
				> expandnarrowspec(ui, repo, os.environ.get('DIRSTATEINCLUDES'))
				> return ret
				> # Mercurial 4.3.3 and earlier uses this version. It seems that
				> # narrowhg does not currently support this version, but we include
				> # it just in case backwards compatibility is restored.
				> def _read(self):
				> ret = super(expandingdirstate, self)._read()
				> with repo.wlock(), repo.lock(), repo.transaction(
				> 'expandnarrowspec'):
				> expandnarrowspec(ui, repo, os.environ.get('DIRSTATEINCLUDES'))
				> return ret
				> ds.__class__ = expandingdirstate
				> return ds
				> return wrapds
				>
				> def reposetup(ui, repo):
				> extensions.wrapfilecache(localrepo.localrepository, 'dirstate',
				> makeds(ui, repo))
				> def overridepatch(orig, args, *kwargs):
				> with repo.wlock():
				> expandnarrowspec(ui, repo, os.environ.get('PATCHINCLUDES'))
				> return orig(args, *kwargs)
				>
				> extensions.wrapfunction(patch, 'patch', overridepatch)
				> EOF
				$ cat >> ".hg/hgrc" <<EOF
				> [extensions]
				> expand_extension = $TESTTMP/expand_extension.py
				> EOF

				Since we do not have the ability to rely on a virtual filesystem or
				remotefilelog in the test, we just fake it by copying the data from the 'master'
				repo.
				$ cp -a ../master/.hg/store/data/* .hg/store/data
				Do that for patchdir as well.
				$ cp -a ../master/patchdir .

				`hg status` will now add outside, but not patchdir.
				$ DIRSTATEINCLUDES=path:outside hg status
				M outside/f2
				$ grep outside .hg/narrowspec
				path:outside
				$ grep outside .hg/dirstate > /dev/null
				$ grep patchdir .hg/narrowspec
				[1]
				$ grep patchdir .hg/dirstate
				[1]

				Get rid of the modification to outside/f2.
				$ hg update -C .
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved

				This patch will not apply cleanly at the moment, so `hg import` will break
				$ cat > "$TESTTMP/foo.patch" <<EOF
				> --- patchdir/f3
				> +++ patchdir/f3
				> @@ -1,1 +1,1 @@
				> -this should be "patch_this", but its not, so patch fails
				> +this text is irrelevant
				> EOF
				$ PATCHINCLUDES=path:patchdir hg import -p0 -e "$TESTTMP/foo.patch" -m ignored
				applying $TESTTMP/foo.patch
				patching file patchdir/f3
				Hunk #1 FAILED at 0
				1 out of 1 hunks FAILED -- saving rejects to file patchdir/f3.rej
				abort: patch failed to apply
				[255]
				$ grep patchdir .hg/narrowspec
				[1]
				$ grep patchdir .hg/dirstate > /dev/null
				[1]

				Let's make it apply cleanly and see that it did expand properly
				$ cat > "$TESTTMP/foo.patch" <<EOF
				> --- patchdir/f3
				> +++ patchdir/f3
				> @@ -1,1 +1,1 @@
				> -patch_this
				> +patched_this
				> EOF
				$ PATCHINCLUDES=path:patchdir hg import -p0 -e "$TESTTMP/foo.patch" -m message
				applying $TESTTMP/foo.patch
				$ cat patchdir/f3
				patched_this
				$ grep patchdir .hg/narrowspec
				path:patchdir
				$ grep patchdir .hg/dirstate > /dev/null

tests/test-narrow-merge-tree.t

This file was added.

				$ cd $TESTDIR && python $RUNTESTDIR/run-tests.py \
				> --extra-config-opt experimental.treemanifest=1 test-narrow-merge.t 2>&1 \| \
				> grep -v 'unexpected mercurial lib' \| egrep -v '\(expected'

				--- */tests/test-narrow-merge.t (glob)
				+++ */tests/test-narrow-merge.t.err (glob)
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)

				$ hg update -q 'desc("modify inside/f1")'
				$ hg merge 'desc("modify outside/f1")'
				- abort: merge affects file 'outside/f1' outside narrow, which is not yet supported
				+ abort: merge affects file 'outside/' outside narrow, which is not yet supported
				(merging in the other direction may work)
				[255]

				@@ -\d+,\d+ \+\d+,\d+ @@ (re)

				$ hg update -q 'desc("modify outside/f1")'
				$ hg merge 'desc("conflicting outside/f1")'
				- abort: conflict in file 'outside/f1' is outside narrow clone
				+ abort: conflict in file 'outside/' is outside narrow clone
				[255]

				ERROR: test-narrow-merge.t output changed
				!
				Failed test-narrow-merge.t: output changed
				# Ran 1 tests, 0 skipped, 1 failed.
				python hash seed: * (glob)

tests/test-narrow-merge.t

This file was added.


				$ . "$TESTDIR/narrow-library.sh"

				create full repo

				$ hg init master
				$ cd master
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF

				$ mkdir inside
				$ echo inside1 > inside/f1
				$ echo inside2 > inside/f2
				$ mkdir outside
				$ echo outside1 > outside/f1
				$ echo outside2 > outside/f2
				$ hg ci -Aqm 'initial'

				$ echo modified > inside/f1
				$ hg ci -qm 'modify inside/f1'

				$ hg update -q 0
				$ echo modified > inside/f2
				$ hg ci -qm 'modify inside/f2'

				$ hg update -q 0
				$ echo modified2 > inside/f1
				$ hg ci -qm 'conflicting inside/f1'

				$ hg update -q 0
				$ echo modified > outside/f1
				$ hg ci -qm 'modify outside/f1'

				$ hg update -q 0
				$ echo modified2 > outside/f1
				$ hg ci -qm 'conflicting outside/f1'

				$ cd ..

				$ hg clone --narrow ssh://user@dummy/master narrow --include inside
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 6 changesets with 5 changes to 2 files (+4 heads)
				new changesets : (glob)
				updating to branch default
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow

				$ hg update -q 0

				Can merge in when no files outside narrow spec are involved

				$ hg update -q 'desc("modify inside/f1")'
				$ hg merge 'desc("modify inside/f2")'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				(branch merge, don't forget to commit)
				$ hg commit -m 'merge inside changes'

				Can merge conflicting changes inside narrow spec

				$ hg update -q 'desc("modify inside/f1")'
				$ hg merge 'desc("conflicting inside/f1")' 2>&1 \| egrep -v '(warning:\|incomplete!)'
				merging inside/f1
				0 files updated, 0 files merged, 0 files removed, 1 files unresolved
				use 'hg resolve' to retry unresolved file merges or 'hg merge --abort' to abandon
				$ echo modified3 > inside/f1
				$ hg resolve -m
				(no more unresolved files)
				$ hg commit -m 'merge inside/f1'

				TODO: Can merge non-conflicting changes outside narrow spec

				$ hg update -q 'desc("modify inside/f1")'
				$ hg merge 'desc("modify outside/f1")'
				abort: merge affects file 'outside/f1' outside narrow, which is not yet supported
				(merging in the other direction may work)
				[255]

				$ hg update -q 'desc("modify outside/f1")'
				$ hg merge 'desc("modify inside/f1")'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				(branch merge, don't forget to commit)
				$ hg ci -m 'merge from inside to outside'

				Refuses merge of conflicting outside changes

				$ hg update -q 'desc("modify outside/f1")'
				$ hg merge 'desc("conflicting outside/f1")'
				abort: conflict in file 'outside/f1' is outside narrow clone
				[255]

tests/test-narrow-patch-tree.t

This file was added.

				$ cd $TESTDIR && python $RUNTESTDIR/run-tests.py \
				> --extra-config-opt experimental.treemanifest=1 test-patch.t 2>&1 \| \
				> grep -v 'unexpected mercurial lib' \| egrep -v '\(expected'
				.
				# Ran 1 tests, 0 skipped, 0 failed.

tests/test-narrow-patch.t

This file was added.


				$ . "$TESTDIR/narrow-library.sh"

				create full repo

				$ hg init master
				$ cd master

				$ mkdir inside
				$ echo inside > inside/f1
				$ mkdir outside
				$ echo outside > outside/f1
				$ hg ci -Aqm 'initial'

				$ echo modified > inside/f1
				$ hg ci -qm 'modify inside'

				$ echo modified > outside/f1
				$ hg ci -qm 'modify outside'

				$ cd ..

				$ hg clone --narrow ssh://user@dummy/master narrow --include inside
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 3 changesets with 2 changes to 1 files
				new changesets : (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow

				Can show patch touching paths outside

				$ hg log -p
				changeset: 2:* (glob)
				tag: tip
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: modify outside


				changeset: 1:* (glob)
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: modify inside

				diff -r * -r * inside/f1 (glob)
				--- a/inside/f1 Thu Jan 01 00:00:00 1970 +0000
				+++ b/inside/f1 Thu Jan 01 00:00:00 1970 +0000
				@@ -1,1 +1,1 @@
				-inside
				+modified

				changeset: 0:* (glob)
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: initial

				diff -r 000000000000 -r * inside/f1 (glob)
				--- /dev/null Thu Jan 01 00:00:00 1970 +0000
				+++ b/inside/f1 Thu Jan 01 00:00:00 1970 +0000
				@@ -0,0 +1,1 @@
				+inside


				$ hg status --rev 1 --rev 2

				Can show copies inside the narrow clone

				$ hg cp inside/f1 inside/f2
				$ hg diff --git
				diff --git a/inside/f1 b/inside/f2
				copy from inside/f1
				copy to inside/f2

tests/test-narrow-patterns.t

This file was added.

				$ . "$TESTDIR/narrow-library.sh"

				initialize nested directories to validate complex include/exclude patterns

				$ hg init master
				$ cd master
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF

				$ echo root > root
				$ hg add root
				$ hg commit -m 'add root'

				$ for d in dir1 dir2 dir1/dirA dir1/dirB dir2/dirA dir2/dirB
				> do
				> mkdir -p $d
				> echo $d/foo > $d/foo
				> hg add $d/foo
				> hg commit -m "add $d/foo"
				> echo $d/bar > $d/bar
				> hg add $d/bar
				> hg commit -m "add $d/bar"
				> done
				$ chmod +x dir1/dirA/foo
				$ hg commit -m "make dir1/dirA/foo executable"
				$ hg log -G -T '{rev} {node\|short} {files}\n'
				@ 13 c87ca422d521 dir1/dirA/foo
				\|
				o 12 951b8a83924e dir2/dirB/bar
				\|
				o 11 01ae5a51b563 dir2/dirB/foo
				\|
				o 10 5eababdf0ac5 dir2/dirA/bar
				\|
				o 9 99d690663739 dir2/dirA/foo
				\|
				o 8 8e80155d5445 dir1/dirB/bar
				\|
				o 7 406760310428 dir1/dirB/foo
				\|
				o 6 623466a5f475 dir1/dirA/bar
				\|
				o 5 06ff3a5be997 dir1/dirA/foo
				\|
				o 4 33227af02764 dir2/bar
				\|
				o 3 5e1f9d8d7c69 dir2/foo
				\|
				o 2 594bc4b13d4a dir1/bar
				\|
				o 1 47f480a08324 dir1/foo
				\|
				o 0 2a4f0c3b67da root

				$ cd ..

				clone a narrow portion of the master, such that we can widen it later

				$ hg clone --narrow ssh://user@dummy/master narrow \
				> --include dir1 \
				> --include dir2 \
				> --exclude dir1/dirA \
				> --exclude dir1/dirB \
				> --exclude dir2/dirA \
				> --exclude dir2/dirB
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 6 changesets with 4 changes to 4 files
				new changesets : (glob)
				updating to branch default
				4 files updated, 0 files merged, 0 files removed, 0 files unresolved

				$ cd narrow
				$ cat .hg/narrowspec
				[includes]
				path:dir1
				path:dir2
				[excludes]
				path:dir1/dirA
				path:dir1/dirB
				path:dir2/dirA
				path:dir2/dirB
				$ hg manifest -r tip
				dir1/bar
				dir1/dirA/bar
				dir1/dirA/foo
				dir1/dirB/bar
				dir1/dirB/foo
				dir1/foo
				dir2/bar
				dir2/dirA/bar
				dir2/dirA/foo
				dir2/dirB/bar
				dir2/dirB/foo
				dir2/foo
				root
				$ find * \| sort
				dir1
				dir1/bar
				dir1/foo
				dir2
				dir2/bar
				dir2/foo
				$ hg log -G -T '{rev} {node\|short}{if(ellipsis, "...")} {files}\n'
				@ 5 c87ca422d521... dir1/dirA/foo
				\|
				o 4 33227af02764 dir2/bar
				\|
				o 3 5e1f9d8d7c69 dir2/foo
				\|
				o 2 594bc4b13d4a dir1/bar
				\|
				o 1 47f480a08324 dir1/foo
				\|
				o 0 2a4f0c3b67da... root


				widen the narrow checkout

				$ hg tracked --removeexclude dir1/dirA
				comparing with ssh://user@dummy/master
				searching for changes
				no changes found
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
				adding changesets
				adding manifests
				adding file changes
				added 9 changesets with 6 changes to 6 files
				new changesets : (glob)
				$ cat .hg/narrowspec
				[includes]
				path:dir1
				path:dir2
				[excludes]
				path:dir1/dirB
				path:dir2/dirA
				path:dir2/dirB
				$ find * \| sort
				dir1
				dir1/bar
				dir1/dirA
				dir1/dirA/bar
				dir1/dirA/foo
				dir1/foo
				dir2
				dir2/bar
				dir2/foo
				$ test -x dir1/dirA/foo && echo executable
				executable
				$ test -x dir1/dirA/bar \|\| echo not executable
				not executable
				$ hg log -G -T '{rev} {node\|short}{if(ellipsis, "...")} {files}\n'
				@ 8 c87ca422d521 dir1/dirA/foo
				\|
				o 7 951b8a83924e... dir2/dirB/bar
				\|
				o 6 623466a5f475 dir1/dirA/bar
				\|
				o 5 06ff3a5be997 dir1/dirA/foo
				\|
				o 4 33227af02764 dir2/bar
				\|
				o 3 5e1f9d8d7c69 dir2/foo
				\|
				o 2 594bc4b13d4a dir1/bar
				\|
				o 1 47f480a08324 dir1/foo
				\|
				o 0 2a4f0c3b67da... root


				widen narrow spec again, but exclude a file in previously included spec

				$ hg tracked --removeexclude dir2/dirB --addexclude dir1/dirA/bar
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				deleting data/dir1/dirA/bar.i
				no changes found
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
				adding changesets
				adding manifests
				adding file changes
				added 11 changesets with 7 changes to 7 files
				new changesets : (glob)
				$ cat .hg/narrowspec
				[includes]
				path:dir1
				path:dir2
				[excludes]
				path:dir1/dirA/bar
				path:dir1/dirB
				path:dir2/dirA
				$ find * \| sort
				dir1
				dir1/bar
				dir1/dirA
				dir1/dirA/foo
				dir1/foo
				dir2
				dir2/bar
				dir2/dirB
				dir2/dirB/bar
				dir2/dirB/foo
				dir2/foo
				$ hg log -G -T '{rev} {node\|short}{if(ellipsis, "...")} {files}\n'
				@ 10 c87ca422d521 dir1/dirA/foo
				\|
				o 9 951b8a83924e dir2/dirB/bar
				\|
				o 8 01ae5a51b563 dir2/dirB/foo
				\|
				o 7 5eababdf0ac5... dir2/dirA/bar
				\|
				o 6 623466a5f475... dir1/dirA/bar
				\|
				o 5 06ff3a5be997 dir1/dirA/foo
				\|
				o 4 33227af02764 dir2/bar
				\|
				o 3 5e1f9d8d7c69 dir2/foo
				\|
				o 2 594bc4b13d4a dir1/bar
				\|
				o 1 47f480a08324 dir1/foo
				\|
				o 0 2a4f0c3b67da... root


				widen narrow spec yet again, excluding a directory in previous spec

				$ hg tracked --removeexclude dir2/dirA --addexclude dir1/dirA
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				deleting data/dir1/dirA/foo.i
				no changes found
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
				adding changesets
				adding manifests
				adding file changes
				added 13 changesets with 8 changes to 8 files
				new changesets : (glob)
				$ cat .hg/narrowspec
				[includes]
				path:dir1
				path:dir2
				[excludes]
				path:dir1/dirA
				path:dir1/dirA/bar
				path:dir1/dirB
				$ find * \| sort
				dir1
				dir1/bar
				dir1/foo
				dir2
				dir2/bar
				dir2/dirA
				dir2/dirA/bar
				dir2/dirA/foo
				dir2/dirB
				dir2/dirB/bar
				dir2/dirB/foo
				dir2/foo
				$ hg log -G -T '{rev} {node\|short}{if(ellipsis, "...")} {files}\n'
				@ 12 c87ca422d521... dir1/dirA/foo
				\|
				o 11 951b8a83924e dir2/dirB/bar
				\|
				o 10 01ae5a51b563 dir2/dirB/foo
				\|
				o 9 5eababdf0ac5 dir2/dirA/bar
				\|
				o 8 99d690663739 dir2/dirA/foo
				\|
				o 7 8e80155d5445... dir1/dirB/bar
				\|
				o 6 623466a5f475... dir1/dirA/bar
				\|
				o 5 06ff3a5be997... dir1/dirA/foo
				\|
				o 4 33227af02764 dir2/bar
				\|
				o 3 5e1f9d8d7c69 dir2/foo
				\|
				o 2 594bc4b13d4a dir1/bar
				\|
				o 1 47f480a08324 dir1/foo
				\|
				o 0 2a4f0c3b67da... root


				include a directory that was previously explicitly excluded

				$ hg tracked --removeexclude dir1/dirA
				comparing with ssh://user@dummy/master
				searching for changes
				no changes found
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
				adding changesets
				adding manifests
				adding file changes
				added 13 changesets with 9 changes to 9 files
				new changesets : (glob)
				$ cat .hg/narrowspec
				[includes]
				path:dir1
				path:dir2
				[excludes]
				path:dir1/dirA/bar
				path:dir1/dirB
				$ find * \| sort
				dir1
				dir1/bar
				dir1/dirA
				dir1/dirA/foo
				dir1/foo
				dir2
				dir2/bar
				dir2/dirA
				dir2/dirA/bar
				dir2/dirA/foo
				dir2/dirB
				dir2/dirB/bar
				dir2/dirB/foo
				dir2/foo
				$ hg log -G -T '{rev} {node\|short}{if(ellipsis, "...")} {files}\n'
				@ 12 c87ca422d521 dir1/dirA/foo
				\|
				o 11 951b8a83924e dir2/dirB/bar
				\|
				o 10 01ae5a51b563 dir2/dirB/foo
				\|
				o 9 5eababdf0ac5 dir2/dirA/bar
				\|
				o 8 99d690663739 dir2/dirA/foo
				\|
				o 7 8e80155d5445... dir1/dirB/bar
				\|
				o 6 623466a5f475... dir1/dirA/bar
				\|
				o 5 06ff3a5be997 dir1/dirA/foo
				\|
				o 4 33227af02764 dir2/bar
				\|
				o 3 5e1f9d8d7c69 dir2/foo
				\|
				o 2 594bc4b13d4a dir1/bar
				\|
				o 1 47f480a08324 dir1/foo
				\|
				o 0 2a4f0c3b67da... root


				$ cd ..

				clone a narrow portion of the master, such that we can widen it later

				$ hg clone --narrow ssh://user@dummy/master narrow2 --include dir1/dirA
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 5 changesets with 2 changes to 2 files
				new changesets : (glob)
				updating to branch default
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow2
				$ find * \| sort
				dir1
				dir1/dirA
				dir1/dirA/bar
				dir1/dirA/foo
				$ hg tracked --addinclude dir1
				comparing with ssh://user@dummy/master
				searching for changes
				no changes found
				saved backup bundle to $TESTTMP/narrow2/.hg/strip-backup/*-widen.hg (glob)
				adding changesets
				adding manifests
				adding file changes
				added 10 changesets with 6 changes to 6 files
				new changesets : (glob)
				$ find * \| sort
				dir1
				dir1/bar
				dir1/dirA
				dir1/dirA/bar
				dir1/dirA/foo
				dir1/dirB
				dir1/dirB/bar
				dir1/dirB/foo
				dir1/foo
				$ hg log -G -T '{rev} {node\|short}{if(ellipsis, "...")} {files}\n'
				@ 9 c87ca422d521 dir1/dirA/foo
				\|
				o 8 951b8a83924e... dir2/dirB/bar
				\|
				o 7 8e80155d5445 dir1/dirB/bar
				\|
				o 6 406760310428 dir1/dirB/foo
				\|
				o 5 623466a5f475 dir1/dirA/bar
				\|
				o 4 06ff3a5be997 dir1/dirA/foo
				\|
				o 3 33227af02764... dir2/bar
				\|
				o 2 594bc4b13d4a dir1/bar
				\|
				o 1 47f480a08324 dir1/foo
				\|
				o 0 2a4f0c3b67da... root

tests/test-narrow-pull.t

This file was added.

				$ . "$TESTDIR/narrow-library.sh"

				$ hg init master
				$ cd master
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF
				$ for x in `$TESTDIR/seq.py 10`
				> do
				> echo $x > "f$x"
				> hg add "f$x"
				> hg commit -m "Commit f$x"
				> done
				$ cd ..

				narrow clone a couple files, f2 and f8

				$ hg clone --narrow ssh://user@dummy/master narrow --include "f2" --include "f8"
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 5 changesets with 2 changes to 2 files
				new changesets : (glob)
				updating to branch default
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow
				$ ls
				f2
				f8
				$ cat f2 f8
				2
				8

				$ cd ..

				change every upstream file twice

				$ cd master
				$ for x in `$TESTDIR/seq.py 10`
				> do
				> echo "update#1 $x" >> "f$x"
				> hg commit -m "Update#1 to f$x" "f$x"
				> done
				$ for x in `$TESTDIR/seq.py 10`
				> do
				> echo "update#2 $x" >> "f$x"
				> hg commit -m "Update#2 to f$x" "f$x"
				> done
				$ cd ..

				look for incoming changes

				$ cd narrow
				$ hg incoming --limit 3
				comparing with ssh://user@dummy/master
				searching for changes
				changeset: 5:ddc055582556
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: Update#1 to f1

				changeset: 6:f66eb5ad621d
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: Update#1 to f2

				changeset: 7:c42ecff04e99
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: Update#1 to f3


				Interrupting the pull is safe
				$ hg --config hooks.pretxnchangegroup.bad=false pull -q
				transaction abort!
				rollback completed
				abort: pretxnchangegroup.bad hook exited with status 1
				[255]
				$ hg id
				223311e70a6f tip

				pull new changes down to the narrow clone. Should get 8 new changesets: 4
				relevant to the narrow spec, and 4 ellipsis nodes gluing them all together.

				$ hg pull
				pulling from ssh://user@dummy/master
				searching for changes
				adding changesets
				adding manifests
				adding file changes
				added 9 changesets with 4 changes to 2 files
				new changesets : (glob)
				(run 'hg update' to get a working copy)
				$ hg log -T '{rev}: {desc}\n'
				13: Update#2 to f10
				12: Update#2 to f8
				11: Update#2 to f7
				10: Update#2 to f2
				9: Update#2 to f1
				8: Update#1 to f8
				7: Update#1 to f7
				6: Update#1 to f2
				5: Update#1 to f1
				4: Commit f10
				3: Commit f8
				2: Commit f7
				1: Commit f2
				0: Commit f1
				$ hg update tip
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				add a change and push it

				$ echo "update#3 2" >> f2
				$ hg commit -m "Update#3 to f2" f2
				$ hg log f2 -T '{rev}: {desc}\n'
				14: Update#3 to f2
				10: Update#2 to f2
				6: Update#1 to f2
				1: Commit f2
				$ hg push
				pushing to ssh://user@dummy/master
				searching for changes
				remote: adding changesets
				remote: adding manifests
				remote: adding file changes
				remote: added 1 changesets with 1 changes to 1 files
				$ cd ..

				$ cd master
				$ hg log f2 -T '{rev}: {desc}\n'
				30: Update#3 to f2
				21: Update#2 to f2
				11: Update#1 to f2
				1: Commit f2
				$ hg log -l 3 -T '{rev}: {desc}\n'
				30: Update#3 to f2
				29: Update#2 to f10
				28: Update#2 to f9

				Can pull into repo with a single commit

				$ cd ..
				$ hg clone -q --narrow ssh://user@dummy/master narrow2 --include "f1" -r 0
				$ cd narrow2
				$ hg pull -q -r 1
				transaction abort!
				rollback completed
				abort: pull failed on remote
				[255]

				Can use 'hg share':
				$ cat >> $HGRCPATH <<EOF
				> [extensions]
				> share=
				> EOF

				$ cd ..
				$ hg share narrow2 narrow2-share
				updating working directory
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow2-share
				$ hg status

				We should also be able to unshare without breaking everything:
				$ hg unshare
				devel-warn: write with no wlock: "narrowspec" at: */hgext/narrow/narrowrepo.py:41 (unsharenarrowspec) (glob)
				$ hg verify
				checking changesets
				checking manifests
				crosschecking files in changesets and manifests
				checking files
				1 files, 1 changesets, 1 total revisions

tests/test-narrow-rebase.t

This file was added.


				$ . "$TESTDIR/narrow-library.sh"

				create full repo

				$ hg init master
				$ cd master

				$ mkdir inside
				$ echo inside1 > inside/f1
				$ echo inside2 > inside/f2
				$ mkdir outside
				$ echo outside1 > outside/f1
				$ echo outside2 > outside/f2
				$ hg ci -Aqm 'initial'

				$ echo modified > inside/f1
				$ hg ci -qm 'modify inside/f1'

				$ hg update -q 0
				$ echo modified2 > inside/f2
				$ hg ci -qm 'modify inside/f2'

				$ hg update -q 0
				$ echo modified > outside/f1
				$ hg ci -qm 'modify outside/f1'

				$ hg update -q 0
				$ echo modified2 > outside/f1
				$ hg ci -qm 'conflicting outside/f1'

				$ cd ..

				$ hg clone --narrow ssh://user@dummy/master narrow --include inside
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 5 changesets with 4 changes to 2 files (+3 heads)
				new changesets : (glob)
				updating to branch default
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow
				$ cat >> $HGRCPATH <<EOF
				> [extensions]
				> rebase=
				> EOF

				$ hg update -q 0

				Can rebase onto commit where no files outside narrow spec are involved

				$ hg update -q 0
				$ echo modified > inside/f2
				$ hg ci -qm 'modify inside/f2'
				$ hg rebase -d 'desc("modify inside/f1")'
				rebasing 5:c2f36d04e05d "modify inside/f2" (tip)
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-rebase.hg (glob)

				Can rebase onto conflicting changes inside narrow spec

				$ hg update -q 0
				$ echo conflicting > inside/f1
				$ hg ci -qm 'conflicting inside/f1'
				$ hg rebase -d 'desc("modify inside/f1")' 2>&1 \| egrep -v '(warning:\|incomplete!)'
				rebasing 6:cdce97fbf653 "conflicting inside/f1" (tip)
				merging inside/f1
				unresolved conflicts (see hg resolve, then hg rebase --continue)
				$ echo modified3 > inside/f1
				$ hg resolve -m 2>&1 \| grep -v continue:
				(no more unresolved files)
				$ hg rebase --continue
				rebasing 6:cdce97fbf653 "conflicting inside/f1" (tip)
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-rebase.hg (glob)

				Can rebase onto non-conflicting changes outside narrow spec

				$ hg update -q 0
				$ echo modified > inside/f2
				$ hg ci -qm 'modify inside/f2'
				$ hg rebase -d 'desc("modify outside/f1")'
				rebasing 7:c2f36d04e05d "modify inside/f2" (tip)
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-rebase.hg (glob)

				Rebase interrupts on conflicting changes outside narrow spec

				$ hg update -q 'desc("conflicting outside/f1")'
				$ hg phase -f -d .
				no phases changed
				$ hg rebase -d 'desc("modify outside/f1")'
				rebasing 4:707c035aadb6 "conflicting outside/f1"
				abort: conflict in file 'outside/f1' is outside narrow clone
				[255]

tests/test-narrow-shallow-merges.t

This file was added.

				$ . "$TESTDIR/narrow-library.sh"

				create full repo

				$ hg init master
				$ cd master
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF

				$ mkdir inside
				$ echo 1 > inside/f
				$ hg commit -Aqm 'initial inside'

				$ mkdir outside
				$ echo 1 > outside/f
				$ hg commit -Aqm 'initial outside'

				$ echo 2a > outside/f
				$ hg commit -Aqm 'outside 2a'
				$ echo 3 > inside/f
				$ hg commit -Aqm 'inside 3'
				$ echo 4a > outside/f
				$ hg commit -Aqm 'outside 4a'
				$ hg update '.~3'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				$ echo 2b > outside/f
				$ hg commit -Aqm 'outside 2b'
				$ echo 3 > inside/f
				$ hg commit -Aqm 'inside 3'
				$ echo 4b > outside/f
				$ hg commit -Aqm 'outside 4b'
				$ hg update '.~3'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				$ echo 2c > outside/f
				$ hg commit -Aqm 'outside 2c'
				$ echo 3 > inside/f
				$ hg commit -Aqm 'inside 3'
				$ echo 4c > outside/f
				$ hg commit -Aqm 'outside 4c'
				$ hg update '.~3'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				$ echo 2d > outside/f
				$ hg commit -Aqm 'outside 2d'
				$ echo 3 > inside/f
				$ hg commit -Aqm 'inside 3'
				$ echo 4d > outside/f
				$ hg commit -Aqm 'outside 4d'

				$ hg update -r 'desc("outside 4a")'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ hg merge -r 'desc("outside 4b")' 2>&1 \| egrep -v '(warning:\|incomplete!)'
				merging outside/f
				0 files updated, 0 files merged, 0 files removed, 1 files unresolved
				use 'hg resolve' to retry unresolved file merges or 'hg merge --abort' to abandon
				$ echo 5 > outside/f
				$ rm outside/f.orig
				$ hg resolve --mark outside/f
				(no more unresolved files)
				$ hg commit -m 'merge a/b 5'
				$ echo 6 > outside/f
				$ hg commit -Aqm 'outside 6'

				$ hg merge -r 'desc("outside 4c")' 2>&1 \| egrep -v '(warning:\|incomplete!)'
				merging outside/f
				0 files updated, 0 files merged, 0 files removed, 1 files unresolved
				use 'hg resolve' to retry unresolved file merges or 'hg merge --abort' to abandon
				$ echo 7 > outside/f
				$ rm outside/f.orig
				$ hg resolve --mark outside/f
				(no more unresolved files)
				$ hg commit -Aqm 'merge a/b/c 7'
				$ echo 8 > outside/f
				$ hg commit -Aqm 'outside 8'

				$ hg merge -r 'desc("outside 4d")' 2>&1 \| egrep -v '(warning:\|incomplete!)'
				merging outside/f
				0 files updated, 0 files merged, 0 files removed, 1 files unresolved
				use 'hg resolve' to retry unresolved file merges or 'hg merge --abort' to abandon
				$ echo 9 > outside/f
				$ rm outside/f.orig
				$ hg resolve --mark outside/f
				(no more unresolved files)
				$ hg commit -Aqm 'merge a/b/c/d 9'
				$ echo 10 > outside/f
				$ hg commit -Aqm 'outside 10'

				$ echo 11 > inside/f
				$ hg commit -Aqm 'inside 11'
				$ echo 12 > outside/f
				$ hg commit -Aqm 'outside 12'

				$ hg log -G -T '{rev} {node\|short} {desc}\n'
				@ 21 8d874d57adea outside 12
				\|
				o 20 7ef88b4dd4fa inside 11
				\|
				o 19 2a20009de83e outside 10
				\|
				o 18 3ac1f5779de3 merge a/b/c/d 9
				\|\
				\| o 17 38a9c2f7e546 outside 8
				\| \|
				\| o 16 094aa62fc898 merge a/b/c 7
				\| \|\
				\| \| o 15 f29d083d32e4 outside 6
				\| \| \|
				\| \| o 14 2dc11382541d merge a/b 5
				\| \| \|\
				o \| \| \| 13 27d07ef97221 outside 4d
				\| \| \| \|
				o \| \| \| 12 465567bdfb2d inside 3
				\| \| \| \|
				o \| \| \| 11 d1c61993ec83 outside 2d
				\| \| \| \|
				\| o \| \| 10 56859a8e33b9 outside 4c
				\| \| \| \|
				\| o \| \| 9 bb96a08b062a inside 3
				\| \| \| \|
				\| o \| \| 8 b844052e7b3b outside 2c
				\|/ / /
				\| \| o 7 9db2d8fcc2a6 outside 4b
				\| \| \|
				\| \| o 6 6418167787a6 inside 3
				\| \| \|
				+---o 5 77344f344d83 outside 2b
				\| \|
				\| o 4 9cadde08dc9f outside 4a
				\| \|
				\| o 3 019ef06f125b inside 3
				\| \|
				\| o 2 75e40c075a19 outside 2a
				\|/
				o 1 906d6c682641 initial outside
				\|
				o 0 9f8e82b51004 initial inside


				Now narrow and shallow clone this and get a hopefully correct graph

				$ cd ..
				$ hg clone --narrow ssh://user@dummy/master narrow --include inside --depth 7
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 8 changesets with 3 changes to 1 files
				new changesets : (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow

				To make updating the tests easier, we print the emitted nodes
				sorted. This makes it easier to identify when the same node structure
				has been emitted, just in a different order.

				$ hg log -G -T '{rev} {node\|short}{if(ellipsis,"...")} {desc}\n'
				@ 7 8d874d57adea... outside 12
				\|
				o 6 7ef88b4dd4fa inside 11
				\|
				o 5 2a20009de83e... outside 10
				\|
				o 4 3ac1f5779de3... merge a/b/c/d 9
				\|\
				\| o 3 465567bdfb2d inside 3
				\| \|
				\| o 2 d1c61993ec83... outside 2d
				\|
				o 1 bb96a08b062a inside 3
				\|
				o 0 b844052e7b3b... outside 2c


				$ hg log -T '{if(ellipsis,"...")}{node\|short} {p1node\|short} {p2node\|short} {desc}\n' \| sort
				...2a20009de83e 000000000000 3ac1f5779de3 outside 10
				...3ac1f5779de3 bb96a08b062a 465567bdfb2d merge a/b/c/d 9
				...8d874d57adea 7ef88b4dd4fa 000000000000 outside 12
				...b844052e7b3b 000000000000 000000000000 outside 2c
				...d1c61993ec83 000000000000 000000000000 outside 2d
				465567bdfb2d d1c61993ec83 000000000000 inside 3
				7ef88b4dd4fa 2a20009de83e 000000000000 inside 11
				bb96a08b062a b844052e7b3b 000000000000 inside 3

				$ cd ..

				Incremental test case: show a pull can pull in a conflicted merge even if elided

				$ hg init pullmaster
				$ cd pullmaster
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF
				$ mkdir inside outside
				$ echo v1 > inside/f
				$ echo v1 > outside/f
				$ hg add inside/f outside/f
				$ hg commit -m init

				$ for line in a b c d
				> do
				> hg update -r 0
				> echo v2$line > outside/f
				> hg commit -m "outside 2$line"
				> echo v2$line > inside/f
				> hg commit -m "inside 2$line"
				> echo v3$line > outside/f
				> hg commit -m "outside 3$line"
				> echo v4$line > outside/f
				> hg commit -m "outside 4$line"
				> done
				0 files updated, 0 files merged, 0 files removed, 0 files unresolved
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				created new head
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				created new head
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				created new head

				$ cd ..
				$ hg clone --narrow ssh://user@dummy/pullmaster pullshallow \
				> --include inside --depth 3
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 12 changesets with 5 changes to 1 files (+3 heads)
				new changesets : (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd pullshallow

				$ hg log -G -T '{rev} {node\|short}{if(ellipsis,"...")} {desc}\n'
				@ 11 0ebbd712a0c8... outside 4d
				\|
				o 10 0d4c867aeb23 inside 2d
				\|
				o 9 e932969c3961... outside 2d

				o 8 33d530345455... outside 4c
				\|
				o 7 0ce6481bfe07 inside 2c
				\|
				o 6 caa65c940632... outside 2c

				o 5 3df233defecc... outside 4b
				\|
				o 4 7162cc6d11a4 inside 2b
				\|
				o 3 f2a632f0082d... outside 2b

				o 2 b8a3da16ba49... outside 4a
				\|
				o 1 53f543eb8e45 inside 2a
				\|
				o 0 1be3e5221c6a... outside 2a

				$ hg log -T '{if(ellipsis,"...")}{node\|short} {p1node\|short} {p2node\|short} {desc}\n' \| sort
				...0ebbd712a0c8 0d4c867aeb23 000000000000 outside 4d
				...1be3e5221c6a 000000000000 000000000000 outside 2a
				...33d530345455 0ce6481bfe07 000000000000 outside 4c
				...3df233defecc 7162cc6d11a4 000000000000 outside 4b
				...b8a3da16ba49 53f543eb8e45 000000000000 outside 4a
				...caa65c940632 000000000000 000000000000 outside 2c
				...e932969c3961 000000000000 000000000000 outside 2d
				...f2a632f0082d 000000000000 000000000000 outside 2b
				0ce6481bfe07 caa65c940632 000000000000 inside 2c
				0d4c867aeb23 e932969c3961 000000000000 inside 2d
				53f543eb8e45 1be3e5221c6a 000000000000 inside 2a
				7162cc6d11a4 f2a632f0082d 000000000000 inside 2b

				$ cd ../pullmaster
				$ hg update -r 'desc("outside 4a")'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ hg merge -r 'desc("outside 4b")' 2>&1 \| egrep -v '(warning:\|incomplete!)'
				merging inside/f
				merging outside/f
				0 files updated, 0 files merged, 0 files removed, 2 files unresolved
				use 'hg resolve' to retry unresolved file merges or 'hg merge --abort' to abandon
				$ echo 3 > inside/f
				$ echo 5 > outside/f
				$ rm -f {in,out}side/f.orig
				$ hg resolve --mark inside/f outside/f
				(no more unresolved files)
				$ hg commit -m 'merge a/b 5'

				$ hg update -r 'desc("outside 4c")'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ hg merge -r 'desc("outside 4d")' 2>&1 \| egrep -v '(warning:\|incomplete!)'
				merging inside/f
				merging outside/f
				0 files updated, 0 files merged, 0 files removed, 2 files unresolved
				use 'hg resolve' to retry unresolved file merges or 'hg merge --abort' to abandon
				$ echo 3 > inside/f
				$ echo 5 > outside/f
				$ rm -f {in,out}side/f.orig
				$ hg resolve --mark inside/f outside/f
				(no more unresolved files)
				$ hg commit -m 'merge c/d 5'

				$ hg update -r 'desc("merge a/b 5")'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ hg merge -r 'desc("merge c/d 5")'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				(branch merge, don't forget to commit)
				$ echo 6 > outside/f
				$ hg commit -m 'outside 6'
				$ echo 7 > outside/f
				$ hg commit -m 'outside 7'
				$ echo 8 > outside/f
				$ hg commit -m 'outside 8'

				$ cd ../pullshallow
				$ hg pull --depth 3
				pulling from ssh://user@dummy/pullmaster
				searching for changes
				adding changesets
				adding manifests
				adding file changes
				added 4 changesets with 3 changes to 1 files (-3 heads)
				new changesets : (glob)
				(run 'hg update' to get a working copy)

				$ hg log -T '{if(ellipsis,"...")}{node\|short} {p1node\|short} {p2node\|short} {desc}\n' \| sort
				...0ebbd712a0c8 0d4c867aeb23 000000000000 outside 4d
				...1be3e5221c6a 000000000000 000000000000 outside 2a
				...33d530345455 0ce6481bfe07 000000000000 outside 4c
				...3df233defecc 7162cc6d11a4 000000000000 outside 4b
				...b8a3da16ba49 53f543eb8e45 000000000000 outside 4a
				...bf545653453e 968003d40c60 000000000000 outside 8
				...caa65c940632 000000000000 000000000000 outside 2c
				...e932969c3961 000000000000 000000000000 outside 2d
				...f2a632f0082d 000000000000 000000000000 outside 2b
				0ce6481bfe07 caa65c940632 000000000000 inside 2c
				0d4c867aeb23 e932969c3961 000000000000 inside 2d
				53f543eb8e45 1be3e5221c6a 000000000000 inside 2a
				67d49c0bdbda b8a3da16ba49 3df233defecc merge a/b 5
				7162cc6d11a4 f2a632f0082d 000000000000 inside 2b
				968003d40c60 67d49c0bdbda e867021d52c2 outside 6
				e867021d52c2 33d530345455 0ebbd712a0c8 merge c/d 5

tests/test-narrow-shallow.t

This file was added.

				$ . "$TESTDIR/narrow-library.sh"

				$ hg init master
				$ cd master
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF
				$ for x in `$TESTDIR/seq.py 10`
				> do
				> echo $x > "f$x"
				> hg add "f$x"
				> done
				$ hg commit -m "Add root files"
				$ mkdir d1 d2
				$ for x in `$TESTDIR/seq.py 10`
				> do
				> echo d1/$x > "d1/f$x"
				> hg add "d1/f$x"
				> echo d2/$x > "d2/f$x"
				> hg add "d2/f$x"
				> done
				$ hg commit -m "Add d1 and d2"
				$ for x in `$TESTDIR/seq.py 10`
				> do
				> echo f$x rev2 > "f$x"
				> echo d1/f$x rev2 > "d1/f$x"
				> echo d2/f$x rev2 > "d2/f$x"
				> hg commit -m "Commit rev2 of f$x, d1/f$x, d2/f$x"
				> done
				$ cd ..

				narrow and shallow clone the d2 directory

				$ hg clone --narrow ssh://user@dummy/master shallow --include "d2" --depth 2
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 4 changesets with 13 changes to 10 files
				new changesets : (glob)
				updating to branch default
				10 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd shallow
				$ hg log -T '{rev}{if(ellipsis,"...")}: {desc}\n'
				3: Commit rev2 of f10, d1/f10, d2/f10
				2: Commit rev2 of f9, d1/f9, d2/f9
				1: Commit rev2 of f8, d1/f8, d2/f8
				0...: Commit rev2 of f7, d1/f7, d2/f7
				$ hg update 0
				3 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cat d2/f7 d2/f8
				d2/f7 rev2
				d2/8

				$ cd ..

				change every upstream file once

				$ cd master
				$ for x in `$TESTDIR/seq.py 10`
				> do
				> echo f$x rev3 > "f$x"
				> echo d1/f$x rev3 > "d1/f$x"
				> echo d2/f$x rev3 > "d2/f$x"
				> hg commit -m "Commit rev3 of f$x, d1/f$x, d2/f$x"
				> done
				$ cd ..

				pull new changes with --depth specified. There were 10 changes to the d2
				directory but the shallow pull should only fetch 3.

				$ cd shallow
				$ hg pull --depth 2
				pulling from ssh://user@dummy/master
				searching for changes
				adding changesets
				adding manifests
				adding file changes
				added 4 changesets with 10 changes to 10 files
				new changesets : (glob)
				(run 'hg update' to get a working copy)
				$ hg log -T '{rev}{if(ellipsis,"...")}: {desc}\n'
				7: Commit rev3 of f10, d1/f10, d2/f10
				6: Commit rev3 of f9, d1/f9, d2/f9
				5: Commit rev3 of f8, d1/f8, d2/f8
				4...: Commit rev3 of f7, d1/f7, d2/f7
				3: Commit rev2 of f10, d1/f10, d2/f10
				2: Commit rev2 of f9, d1/f9, d2/f9
				1: Commit rev2 of f8, d1/f8, d2/f8
				0...: Commit rev2 of f7, d1/f7, d2/f7
				$ hg update 4
				merging d2/f1
				merging d2/f2
				merging d2/f3
				merging d2/f4
				merging d2/f5
				merging d2/f6
				merging d2/f7
				3 files updated, 7 files merged, 0 files removed, 0 files unresolved
				$ cat d2/f7 d2/f8
				d2/f7 rev3
				d2/f8 rev2
				$ hg update 7
				3 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cat d2/f10
				d2/f10 rev3

				$ cd ..

				cannot clone with zero or negative depth

				$ hg clone --narrow ssh://user@dummy/master bad --include "d2" --depth 0
				requesting all changes
				remote: abort: depth must be positive, got 0
				abort: pull failed on remote
				[255]
				$ hg clone --narrow ssh://user@dummy/master bad --include "d2" --depth -1
				requesting all changes
				remote: abort: depth must be positive, got -1
				abort: pull failed on remote
				[255]

tests/test-narrow-strip-tree.t

This file was added.

				$ cd $TESTDIR && python $RUNTESTDIR/run-tests.py \
				> --extra-config-opt experimental.treemanifest=1 test-narrow-strip.t 2>&1 \| \
				> grep -v 'unexpected mercurial lib' \| egrep -v '\(expected'

				--- */test-narrow-strip.t (glob)
				+++ */test-narrow-strip.t.err (glob)
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)
				o 0 initial

				$ hg debugdata -m 1
				- inside/f1\x004d6a634d5ba06331a60c29ee0db8412490a54fcd (esc)
				- outside/f1\x0084ba604d54dee1f13310ce3d4ac2e8a36636691a (esc)
				+ inside\x006a8bc41df94075d501f9740587a0c0e13c170dc5t (esc)
				+ outside\x00255c2627ebdd3c7dcaa6945246f9b9f02bd45a09t (esc)

				$ rm -f $TESTTMP/narrow/.hg/strip-backup/*-backup.hg
				$ hg strip .
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)

				Check that hash of file outside narrow spec got restored
				$ hg debugdata -m 2
				- inside/f1\x004d6a634d5ba06331a60c29ee0db8412490a54fcd (esc)
				- outside/f1\x0084ba604d54dee1f13310ce3d4ac2e8a36636691a (esc)
				+ inside\x006a8bc41df94075d501f9740587a0c0e13c170dc5t (esc)
				+ outside\x00255c2627ebdd3c7dcaa6945246f9b9f02bd45a09t (esc)

				Also verify we can apply the bundle with 'hg pull':
				$ hg co -r 'desc("modify inside")'
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: initial

				- changeset: 1:9e48d953700d
				+ changeset: 1:3888164bccf0
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: modify outside again

				- changeset: 2:f505d5e96aa8
				+ changeset: 2:40b66f95a209
				tag: tip
				- parent: 0:a99f4d53924d
				+ parent: 0:c2a5fabcca3c
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: modify inside

				ERROR: test-narrow-strip.t output changed
				!
				Failed test-narrow-strip.t: output changed
				# Ran 1 tests, 0 skipped, 1 failed.
				python hash seed: * (glob)

tests/test-narrow-strip.t

This file was added.


				$ . "$TESTDIR/narrow-library.sh"

				create full repo

				$ hg init master
				$ cd master
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF

				$ mkdir inside
				$ echo inside > inside/f1
				$ mkdir outside
				$ echo outside > outside/f1
				$ hg ci -Aqm 'initial'

				$ echo modified > inside/f1
				$ hg ci -qm 'modify inside'

				$ hg co -q 0
				$ echo modified > outside/f1
				$ hg ci -qm 'modify outside'

				$ echo modified again >> outside/f1
				$ hg ci -qm 'modify outside again'

				$ cd ..

				$ hg clone --narrow ssh://user@dummy/master narrow --include inside
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 3 changesets with 2 changes to 1 files (+1 heads)
				new changesets : (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow
				$ cat >> $HGRCPATH <<EOF
				> [extensions]
				> strip=
				> EOF

				Can strip and recover changesets affecting only files within narrow spec

				$ hg co -r 'desc("modify inside")'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ rm -f $TESTTMP/narrow/.hg/strip-backup/*-backup.hg
				$ hg strip .
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-backup.hg (glob)
				$ hg unbundle .hg/strip-backup/*-backup.hg
				adding changesets
				adding manifests
				adding file changes
				added 1 changesets with 1 changes to 1 files (+1 heads)
				new changesets * (glob)
				(run 'hg heads' to see heads, 'hg merge' to merge)

				Can strip and recover changesets affecting files outside of narrow spec

				$ hg co -r 'desc("modify outside")'
				0 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ hg log -G -T '{rev} {desc}\n'
				o 2 modify inside
				\|
				\| @ 1 modify outside again
				\|/
				o 0 initial

				$ hg debugdata -m 1
				inside/f1\x004d6a634d5ba06331a60c29ee0db8412490a54fcd (esc)
				outside/f1\x0084ba604d54dee1f13310ce3d4ac2e8a36636691a (esc)

				$ rm -f $TESTTMP/narrow/.hg/strip-backup/*-backup.hg
				$ hg strip .
				0 files updated, 0 files merged, 0 files removed, 0 files unresolved
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-backup.hg (glob)
				$ hg unbundle .hg/strip-backup/*-backup.hg
				mharbison72Unsubmitted Not Done The test failures here on Windows seem to be a case of incoming and unbundle getting confused. Here's some debug info (starting right after the `rm` on line 77), but I'm not sure what to do with it: $ hg log -G o changeset: 2:f505d5e96aa8 \| tag: tip \| parent: 0:a99f4d53924d \| user: test \| date: Thu Jan 01 00:00:00 1970 +0000 \| summary: modify inside \| \| @ changeset: 1:9e48d953700d \|/ user: test \| date: Thu Jan 01 00:00:00 1970 +0000 \| summary: modify outside again \| o changeset: 0:a99f4d53924d user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: initial $ hg strip . 0 files updated, 0 files merged, 0 files removed, 0 files unresolved saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/-backup.hg (glob) $ hg log -G o changeset: 1:f505d5e96aa8 \| tag: tip \| user: test \| date: Thu Jan 01 00:00:00 1970 +0000 \| summary: modify inside \| @ changeset: 0:a99f4d53924d user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: initial $ hg in --debug --bundle .hg/strip-backup/-backup.hg bundle2-input-part: total payload size 521 bundle2-input-part: total payload size 24 comparing with .hg/strip-backup/f505d5e96aa8-2b55e253-backup.hg query 1; heads searching for changes all remote heads known locally no changes found [1] $ hg unbundle --debug .hg/strip-backup/-backup.hg bundle2-input-bundle: 1 params with-transaction bundle2-input-part: "changegroup" (params: 1 mandatory 1 advisory) supported adding changesets add changeset f505d5e96aa8 adding manifests adding file changes adding inside/f1 revisions added 0 changesets with 0 changes to 1 files bundle2-input-part: total payload size 521 bundle2-input-part: "phase-heads" supported bundle2-input-part: total payload size 24 bundle2-input-bundle: 1 parts total (run 'hg update' to get a working copy) $ hg log -G -T '{rev} {desc}\n' o 1 modify inside \| @ 0 initial mharbison72:* The test failures here on Windows seem to be a case of incoming and unbundle getting confused.
				adding changesets
				adding manifests
				adding file changes
				added 1 changesets with 0 changes to 0 files (+1 heads)
				new changesets * (glob)
				(run 'hg heads' to see heads, 'hg merge' to merge)
				$ hg log -G -T '{rev} {desc}\n'
				o 2 modify outside again
				\|
				\| o 1 modify inside
				\|/
				@ 0 initial

				Check that hash of file outside narrow spec got restored
				$ hg debugdata -m 2
				inside/f1\x004d6a634d5ba06331a60c29ee0db8412490a54fcd (esc)
				outside/f1\x0084ba604d54dee1f13310ce3d4ac2e8a36636691a (esc)

				Also verify we can apply the bundle with 'hg pull':
				$ hg co -r 'desc("modify inside")'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ rm .hg/strip-backup/*-backup.hg
				$ hg strip .
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-backup.hg (glob)
				$ hg pull .hg/strip-backup/*-backup.hg
				pulling from .hg/strip-backup/*-backup.hg (glob)
				searching for changes
				adding changesets
				adding manifests
				adding file changes
				added 1 changesets with 1 changes to 1 files (+1 heads)
				new changesets * (glob)
				(run 'hg heads' to see heads, 'hg merge' to merge)

				$ rm .hg/strip-backup/*-backup.hg
				$ hg strip 0
				0 files updated, 0 files merged, 1 files removed, 0 files unresolved
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-backup.hg (glob)
				$ hg incoming .hg/strip-backup/*-backup.hg
				comparing with .hg/strip-backup/*-backup.hg (glob)
				changeset: 0:* (glob)
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: initial

				changeset: 1:9e48d953700d
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: modify outside again

				changeset: 2:f505d5e96aa8
				tag: tip
				parent: 0:a99f4d53924d
				user: test
				date: Thu Jan 01 00:00:00 1970 +0000
				summary: modify inside

				$ hg pull .hg/strip-backup/*-backup.hg
				pulling from .hg/strip-backup/*-backup.hg (glob)
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 3 changesets with 2 changes to 1 files (+1 heads)
				new changesets : (glob)
				(run 'hg heads' to see heads, 'hg merge' to merge)

tests/test-narrow-tree.t

This file was added.

				$ cd $TESTDIR && python $RUNTESTDIR/run-tests.py \
				> --extra-config-opt experimental.treemanifest=1 test-narrow-narrow.t 2>&1 \| \
				> grep -v 'unexpected mercurial lib' \| egrep -v '\(expected'

				--- /*/tests/test-narrow-narrow.t (glob)
				+++ /*/tests/test-narrow-narrow.t.err (glob)
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)
				* (glob)
				* (glob)
				deleting data/d0/f.i
				+ deleting meta/d0/00manifest.i
				$ hg log -T "{node\|short}: {desc} {outsidenarrow}\n"
				*: local change to d3 (glob)
				*: add d10/f outsidenarrow (glob)
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)
				looking for local changes to affected paths
				saved backup bundle to $TESTTMP/narrow-local-changes/.hg/strip-backup/*-narrow.hg (glob)
				deleting data/d0/f.i
				+ deleting meta/d0/00manifest.i
				Updates off of stripped commit if necessary
				$ hg co -r 'desc("local change to d3")' -q
				$ echo local change >> d6/f
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				saved backup bundle to $TESTTMP/narrow-local-changes/.hg/strip-backup/*-narrow.hg (glob)
				deleting data/d3/f.i
				+ deleting meta/d3/00manifest.i
				$ hg log -T '{desc}\n' -r .
				add d10/f
				Updates to nullid if necessary
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)
				0 files updated, 0 files merged, 1 files removed, 0 files unresolved
				saved backup bundle to $TESTTMP/narrow-local-changes/.hg/strip-backup/*-narrow.hg (glob)
				deleting data/d3/f.i
				+ deleting meta/d3/00manifest.i
				$ hg id
				000000000000
				$ cd ..
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)
				searching for changes
				looking for local changes to affected paths
				deleting data/d0/f.i
				+ deleting meta/d0/00manifest.i
				$ hg tracked
				$ hg files
				[1]
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)
				searching for changes
				looking for local changes to affected paths
				deleting data/d6/f.i
				+ deleting meta/d6/00manifest.i
				$ hg tracked
				I path:d0
				I path:d3
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)
				searching for changes
				looking for local changes to affected paths
				deleting data/d0/f.i
				+ deleting meta/d0/00manifest.i
				$ hg tracked
				I path:d3
				I path:d9

				ERROR: test-narrow-narrow.t output changed
				!
				Failed test-narrow-narrow.t: output changed
				# Ran 1 tests, 0 skipped, 1 failed.
				python hash seed: * (glob)

tests/test-narrow-update.t

This file was added.


				$ . "$TESTDIR/narrow-library.sh"

				create full repo

				$ hg init master
				$ cd master
				$ echo init > init
				$ hg ci -Aqm 'initial'

				$ mkdir inside
				$ echo inside > inside/f1
				$ mkdir outside
				$ echo outside > outside/f1
				$ hg ci -Aqm 'add inside and outside'

				$ echo modified > inside/f1
				$ hg ci -qm 'modify inside'

				$ echo modified > outside/f1
				$ hg ci -qm 'modify outside'

				$ cd ..

				$ hg clone --narrow ssh://user@dummy/master narrow --include inside
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 4 changesets with 2 changes to 1 files
				new changesets : (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow
				$ hg debugindex -c
				rev offset length base linkrev nodeid p1 p2
				0 0 64 0 0 9958b1af2add 000000000000 000000000000
				1 64 81 1 1 2db4ce2a3bfe 9958b1af2add 000000000000
				2 145 75 2 2 0980ee31a742 2db4ce2a3bfe 000000000000
				3 220 (76\|77) 3 3 4410145019b7 0980ee31a742 000000000000 (re)

				$ hg update -q 0

				Can update to revision with changes inside

				$ hg update -q 'desc("add inside and outside")'
				$ hg update -q 'desc("modify inside")'
				$ find *
				inside
				inside/f1 (glob)
				$ cat inside/f1
				modified

				Can update to revision with changes outside

				$ hg update -q 'desc("modify outside")'
				$ find *
				inside
				inside/f1 (glob)
				$ cat inside/f1
				modified

				Can update with a deleted file inside

				$ hg rm inside/f1
				$ hg update -q 'desc("modify inside")'
				$ hg update -q 'desc("modify outside")'
				$ hg update -q 'desc("initial")'
				$ hg update -q 'desc("modify inside")'

				Can update with a moved file inside

				$ hg mv inside/f1 inside/f2
				$ hg update -q 'desc("modify outside")'
				$ hg update -q 'desc("initial")'
				$ hg update -q 'desc("modify inside")'

tests/test-narrow-widen-tree.t

This file was added.

				$ cd $TESTDIR && python $RUNTESTDIR/run-tests.py \
				> --extra-config-opt experimental.treemanifest=1 test-narrow-widen.t 2>&1 \| \
				> grep -v 'unexpected mercurial lib' \| egrep -v '\(expected'

				--- */test-narrow-widen.t (glob)
				+++ */test-narrow-widen.t.err (glob)
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)
				$ hg verify
				checking changesets
				checking manifests
				+ checking directory manifests
				crosschecking files in changesets and manifests
				checking files
				4 files, 8 changesets, 4 total revisions
				@@ -\d+,\d+ \+\d+,\d+ @@ (re)
				$ hg verify
				checking changesets
				checking manifests
				+ checking directory manifests
				crosschecking files in changesets and manifests
				checking files
				5 files, 9 changesets, 5 total revisions

				ERROR: test-narrow-widen.t output changed
				!
				Failed test-narrow-widen.t: output changed
				# Ran 1 tests, 0 skipped, 1 failed.
				python hash seed: * (glob)

tests/test-narrow-widen.t

This file was added.

				$ . "$TESTDIR/narrow-library.sh"

				$ hg init master
				$ cd master
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF

				$ mkdir inside
				$ echo 'inside' > inside/f
				$ hg add inside/f
				$ hg commit -m 'add inside'

				$ mkdir widest
				$ echo 'widest' > widest/f
				$ hg add widest/f
				$ hg commit -m 'add widest'

				$ mkdir outside
				$ echo 'outside' > outside/f
				$ hg add outside/f
				$ hg commit -m 'add outside'

				$ cd ..

				narrow clone the inside file

				$ hg clone --narrow ssh://user@dummy/master narrow --include inside
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 2 changesets with 1 changes to 1 files
				new changesets : (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow
				$ hg tracked
				I path:inside
				$ ls
				inside
				$ cat inside/f
				inside
				$ cd ..

				add more upstream files which we will include in a wider narrow spec

				$ cd master

				$ mkdir wider
				$ echo 'wider' > wider/f
				$ hg add wider/f
				$ echo 'widest v2' > widest/f
				$ hg commit -m 'add wider, update widest'

				$ echo 'widest v3' > widest/f
				$ hg commit -m 'update widest v3'

				$ echo 'inside v2' > inside/f
				$ hg commit -m 'update inside'

				$ mkdir outside2
				$ echo 'outside2' > outside2/f
				$ hg add outside2/f
				$ hg commit -m 'add outside2'

				$ echo 'widest v4' > widest/f
				$ hg commit -m 'update widest v4'

				$ hg log -T "{if(ellipsis, '...')}{node\|short}: {desc}\n"
				*: update widest v4 (glob)
				*: add outside2 (glob)
				*: update inside (glob)
				*: update widest v3 (glob)
				*: add wider, update widest (glob)
				*: add outside (glob)
				*: add widest (glob)
				*: add inside (glob)

				$ cd ..

				Widen the narrow spec to see the wider file. This should not get the newly
				added upstream revisions.

				$ cd narrow
				$ hg tracked --addinclude wider/f
				comparing with ssh://user@dummy/master
				searching for changes
				no changes found
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
				adding changesets
				adding manifests
				adding file changes
				added 2 changesets with 1 changes to 1 files
				new changesets : (glob)
				$ hg tracked
				I path:inside
				I path:wider/f

				Pull down the newly added upstream revision.

				$ hg pull
				pulling from ssh://user@dummy/master
				searching for changes
				adding changesets
				adding manifests
				adding file changes
				added 4 changesets with 2 changes to 2 files
				new changesets : (glob)
				(run 'hg update' to get a working copy)
				$ hg update -r 'desc("add wider")'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cat wider/f
				wider

				$ hg update -r 'desc("update inside")'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cat wider/f
				wider
				$ cat inside/f
				inside v2

				$ hg log -T "{if(ellipsis, '...')}{node\|short}: {desc}\n"
				...*: update widest v4 (glob)
				*: update inside (glob)
				...*: update widest v3 (glob)
				*: add wider, update widest (glob)
				...*: add outside (glob)
				*: add inside (glob)

				Check that widening with a newline fails

				$ hg tracked --addinclude 'widest
				> '
				abort: newlines are not allowed in narrowspec paths
				[255]

				widen the narrow spec to include the widest file

				$ hg tracked --addinclude widest
				comparing with ssh://user@dummy/master
				searching for changes
				no changes found
				saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
				adding changesets
				adding manifests
				adding file changes
				added 8 changesets with 7 changes to 3 files
				new changesets : (glob)
				$ hg tracked
				I path:inside
				I path:wider/f
				I path:widest
				$ hg update 'desc("add widest")'
				2 files updated, 0 files merged, 1 files removed, 0 files unresolved
				$ cat widest/f
				widest
				$ hg update 'desc("add wider, update widest")'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cat wider/f
				wider
				$ cat widest/f
				widest v2
				$ hg update 'desc("update widest v3")'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cat widest/f
				widest v3
				$ hg update 'desc("update widest v4")'
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cat widest/f
				widest v4

				$ hg log -T "{if(ellipsis, '...')}{node\|short}: {desc}\n"
				*: update widest v4 (glob)
				...*: add outside2 (glob)
				*: update inside (glob)
				*: update widest v3 (glob)
				*: add wider, update widest (glob)
				...*: add outside (glob)
				*: add widest (glob)
				*: add inside (glob)

				separate suite of tests: files from 0-10 modified in changes 0-10. This allows
				more obvious precise tests tickling particular corner cases.

				$ cd ..
				$ hg init upstream
				$ cd upstream
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF
				$ for x in `$TESTDIR/seq.py 0 10`
				> do
				> mkdir d$x
				> echo $x > d$x/f
				> hg add d$x/f
				> hg commit -m "add d$x/f"
				> done
				$ hg log -T "{node\|short}: {desc}\n"
				*: add d10/f (glob)
				*: add d9/f (glob)
				*: add d8/f (glob)
				*: add d7/f (glob)
				*: add d6/f (glob)
				*: add d5/f (glob)
				*: add d4/f (glob)
				*: add d3/f (glob)
				*: add d2/f (glob)
				*: add d1/f (glob)
				*: add d0/f (glob)

				make narrow clone with every third node.

				$ cd ..
				$ hg clone --narrow ssh://user@dummy/upstream narrow2 --include d0 --include d3 --include d6 --include d9
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 8 changesets with 4 changes to 4 files
				new changesets : (glob)
				updating to branch default
				4 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow2
				$ hg tracked
				I path:d0
				I path:d3
				I path:d6
				I path:d9
				$ hg verify
				checking changesets
				checking manifests
				crosschecking files in changesets and manifests
				checking files
				4 files, 8 changesets, 4 total revisions
				$ hg log -T "{if(ellipsis, '...')}{node\|short}: {desc}\n"
				...*: add d10/f (glob)
				*: add d9/f (glob)
				...*: add d8/f (glob)
				*: add d6/f (glob)
				...*: add d5/f (glob)
				*: add d3/f (glob)
				...*: add d2/f (glob)
				*: add d0/f (glob)
				$ hg tracked --addinclude d1
				comparing with ssh://user@dummy/upstream
				searching for changes
				no changes found
				saved backup bundle to $TESTTMP/narrow2/.hg/strip-backup/*-widen.hg (glob)
				adding changesets
				adding manifests
				adding file changes
				added 9 changesets with 5 changes to 5 files
				new changesets : (glob)
				$ hg tracked
				I path:d0
				I path:d1
				I path:d3
				I path:d6
				I path:d9
				$ hg log -T "{if(ellipsis, '...')}{node\|short}: {desc}\n"
				...*: add d10/f (glob)
				*: add d9/f (glob)
				...*: add d8/f (glob)
				*: add d6/f (glob)
				...*: add d5/f (glob)
				*: add d3/f (glob)
				...*: add d2/f (glob)
				*: add d1/f (glob)
				*: add d0/f (glob)

				Verify shouldn't claim the repo is corrupt after a widen.

				$ hg verify
				checking changesets
				checking manifests
				crosschecking files in changesets and manifests
				checking files
				5 files, 9 changesets, 5 total revisions

				Widening preserves parent of local commit

				$ cd ..
				$ hg clone -q --narrow ssh://user@dummy/upstream narrow3 --include d2 -r 2
				$ cd narrow3
				$ hg log -T "{if(ellipsis, '...')}{node\|short}: {desc}\n"
				*: add d2/f (glob)
				...*: add d1/f (glob)
				$ hg pull -q -r 3
				$ hg co -q tip
				$ hg pull -q -r 4
				$ echo local > d2/f
				$ hg ci -m local
				created new head
				$ hg tracked -q --addinclude d0 --addinclude d9

				Widening preserves bookmarks

				$ cd ..
				$ hg clone -q --narrow ssh://user@dummy/upstream narrow-bookmarks --include d4
				$ cd narrow-bookmarks
				$ echo local > d4/f
				$ hg ci -m local
				$ hg bookmarks bookmark
				$ hg bookmarks
				* bookmark 3:* (glob)
				$ hg -q tracked --addinclude d2
				$ hg bookmarks
				* bookmark 5:* (glob)
				$ hg log -r bookmark -T '{desc}\n'
				local

				Widening that fails can be recovered from

				$ cd ..
				$ hg clone -q --narrow ssh://user@dummy/upstream interrupted --include d0
				$ cd interrupted
				$ echo local > d0/f
				$ hg ci -m local
				$ hg log -T "{if(ellipsis, '...')}{rev}: {desc}\n"
				2: local
				...1: add d10/f
				0: add d0/f
				$ hg bookmarks bookmark
				$ hg --config hooks.pretxnchangegroup.bad=false tracked --addinclude d1
				comparing with ssh://user@dummy/upstream
				searching for changes
				no changes found
				saved backup bundle to $TESTTMP/interrupted/.hg/strip-backup/*-widen.hg (glob)
				adding changesets
				adding manifests
				adding file changes
				added 3 changesets with 2 changes to 2 files
				transaction abort!
				rollback completed
				abort: pretxnchangegroup.bad hook exited with status 1
				[255]
				$ hg log -T "{if(ellipsis, '...')}{rev}: {desc}\n"
				$ hg bookmarks
				no bookmarks set
				$ hg unbundle .hg/strip-backup/*-widen.hg
				adding changesets
				adding manifests
				adding file changes
				added 3 changesets with 2 changes to 1 files
				new changesets : (glob)
				(run 'hg update' to get a working copy)
				$ hg log -T "{if(ellipsis, '...')}{rev}: {desc}\n"
				2: local
				...1: add d10/f
				0: add d0/f
				$ hg bookmarks
				* bookmark 2:* (glob)

tests/test-narrow.t

This file was added.

				$ . "$TESTDIR/narrow-library.sh"

				$ hg init master
				$ cd master
				$ cat >> .hg/hgrc <<EOF
				> [narrow]
				> serveellipses=True
				> EOF
				$ for x in `$TESTDIR/seq.py 0 10`
				> do
				> mkdir d$x
				> echo $x > d$x/f
				> hg add d$x/f
				> hg commit -m "add d$x/f"
				> done
				$ hg log -T "{node\|short}: {desc}\n"
				*: add d10/f (glob)
				*: add d9/f (glob)
				*: add d8/f (glob)
				*: add d7/f (glob)
				*: add d6/f (glob)
				*: add d5/f (glob)
				*: add d4/f (glob)
				*: add d3/f (glob)
				*: add d2/f (glob)
				*: add d1/f (glob)
				*: add d0/f (glob)
				$ cd ..

				Error if '.' or '..' are in the directory to track.
				$ hg clone --narrow ssh://user@dummy/master foo --include ./asdf
				requesting all changes
				abort: "." and ".." are not allowed in narrowspec paths
				[255]
				$ hg clone --narrow ssh://user@dummy/master foo --include asdf/..
				requesting all changes
				abort: "." and ".." are not allowed in narrowspec paths
				[255]
				$ hg clone --narrow ssh://user@dummy/master foo --include a/./c
				requesting all changes
				abort: "." and ".." are not allowed in narrowspec paths
				[255]

				Names with '.' in them are OK.
				$ hg clone --narrow ssh://user@dummy/master $RANDOM --include a/.b/c
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 1 changesets with 0 changes to 0 files
				new changesets * (glob)
				updating to branch default
				0 files updated, 0 files merged, 0 files removed, 0 files unresolved

				Test repo with local changes
				$ hg clone --narrow ssh://user@dummy/master narrow-local-changes --include d0 --include d3 --include d6
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 6 changesets with 3 changes to 3 files
				new changesets : (glob)
				updating to branch default
				3 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow-local-changes
				$ cat >> $HGRCPATH << EOF
				> [experimental]
				> evolution=createmarkers
				> EOF
				$ echo local change >> d0/f
				$ hg ci -m 'local change to d0'
				$ hg co '.^'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ echo local change >> d3/f
				$ hg ci -m 'local hidden change to d3'
				created new head
				$ hg ci --amend -m 'local change to d3'
				$ hg tracked --removeinclude d0
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				The following changeset(s) or their ancestors have local changes not on the remote:
				* (glob)
				abort: local changes found
				(use --force-delete-local-changes to ignore)
				[255]
				Check that nothing was removed by the failed attempts
				$ hg tracked
				I path:d0
				I path:d3
				I path:d6
				$ hg files
				d0/f
				d3/f
				d6/f
				$ find *
				d0
				d0/f
				d3
				d3/f
				d6
				d6/f
				$ hg verify -q
				Force deletion of local changes
				$ hg log -T "{node\|short}: {desc} {outsidenarrow}\n"
				*: local change to d3 (glob)
				*: local change to d0 (glob)
				*: add d10/f outsidenarrow (glob)
				*: add d6/f (glob)
				*: add d5/f outsidenarrow (glob)
				*: add d3/f (glob)
				*: add d2/f outsidenarrow (glob)
				*: add d0/f (glob)
				$ hg tracked --removeinclude d0 --force-delete-local-changes
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				The following changeset(s) or their ancestors have local changes not on the remote:
				* (glob)
				saved backup bundle to $TESTTMP/narrow-local-changes/.hg/strip-backup/*-narrow.hg (glob)
				deleting data/d0/f.i
				$ hg log -T "{node\|short}: {desc} {outsidenarrow}\n"
				*: local change to d3 (glob)
				*: add d10/f outsidenarrow (glob)
				*: add d6/f (glob)
				*: add d5/f outsidenarrow (glob)
				*: add d3/f (glob)
				*: add d2/f outsidenarrow (glob)
				*: add d0/f outsidenarrow (glob)
				Can restore stripped local changes after widening
				$ hg tracked --addinclude d0 -q
				$ hg unbundle .hg/strip-backup/*-narrow.hg -q
				$ hg --hidden co -r 'desc("local change to d0")' -q
				$ cat d0/f
				0
				local change
				Pruned commits affecting removed paths should not prevent narrowing
				$ hg co '.^'
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ hg debugobsolete `hg log -T '{node}' -r 'desc("local change to d0")'`
				obsoleted 1 changesets
				$ hg tracked --removeinclude d0
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				saved backup bundle to $TESTTMP/narrow-local-changes/.hg/strip-backup/*-narrow.hg (glob)
				deleting data/d0/f.i
				Updates off of stripped commit if necessary
				$ hg co -r 'desc("local change to d3")' -q
				$ echo local change >> d6/f
				$ hg ci -m 'local change to d6'
				$ hg tracked --removeinclude d3 --force-delete-local-changes
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				The following changeset(s) or their ancestors have local changes not on the remote:
				* (glob)
				* (glob)
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved
				saved backup bundle to $TESTTMP/narrow-local-changes/.hg/strip-backup/*-narrow.hg (glob)
				deleting data/d3/f.i
				$ hg log -T '{desc}\n' -r .
				add d10/f
				Updates to nullid if necessary
				$ hg tracked --addinclude d3 -q
				$ hg co null -q
				$ mkdir d3
				$ echo local change > d3/f
				$ hg add d3/f
				$ hg ci -m 'local change to d3'
				created new head
				$ hg tracked --removeinclude d3 --force-delete-local-changes
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				The following changeset(s) or their ancestors have local changes not on the remote:
				* (glob)
				0 files updated, 0 files merged, 1 files removed, 0 files unresolved
				saved backup bundle to $TESTTMP/narrow-local-changes/.hg/strip-backup/*-narrow.hg (glob)
				deleting data/d3/f.i
				$ hg id
				000000000000
				$ cd ..

				Can remove last include, making repo empty
				$ hg clone --narrow ssh://user@dummy/master narrow-empty --include d0 -r 5
				adding changesets
				adding manifests
				adding file changes
				added 2 changesets with 1 changes to 1 files
				new changesets : (glob)
				updating to branch default
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow-empty
				$ hg tracked --removeinclude d0
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				deleting data/d0/f.i
				$ hg tracked
				$ hg files
				[1]
				$ test -d d0
				[1]
				Do some work in the empty clone
				$ hg diff --change .
				$ hg branch foo
				marked working directory as branch foo
				(branches are permanent and global, did you want a bookmark?)
				$ hg ci -m empty
				$ hg pull -q
				Can widen the empty clone
				$ hg tracked --addinclude d0
				comparing with ssh://user@dummy/master
				searching for changes
				no changes found
				saved backup bundle to $TESTTMP/narrow-empty/.hg/strip-backup/*-widen.hg (glob)
				adding changesets
				adding manifests
				adding file changes
				added 3 changesets with 1 changes to 1 files
				new changesets : (glob)
				$ hg tracked
				I path:d0
				$ hg files
				d0/f
				$ find *
				d0
				d0/f
				$ cd ..

				TODO(martinvonz): test including e.g. d3/g and then removing it once
				https://bitbucket.org/Google/narrowhg/issues/6 is fixed

				$ hg clone --narrow ssh://user@dummy/master narrow --include d0 --include d3 --include d6 --include d9
				requesting all changes
				adding changesets
				adding manifests
				adding file changes
				added 8 changesets with 4 changes to 4 files
				new changesets : (glob)
				updating to branch default
				4 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ cd narrow
				$ hg tracked
				I path:d0
				I path:d3
				I path:d6
				I path:d9
				$ hg tracked --removeinclude d6
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				deleting data/d6/f.i
				$ hg tracked
				I path:d0
				I path:d3
				I path:d9
				$ hg debugrebuildfncache
				fncache already up to date
				$ find *
				d0
				d0/f
				d3
				d3/f
				d9
				d9/f
				$ hg verify -q
				$ hg tracked --addexclude d3/f
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				deleting data/d3/f.i
				$ hg tracked
				I path:d0
				I path:d3
				I path:d9
				X path:d3/f
				$ hg debugrebuildfncache
				fncache already up to date
				$ find *
				d0
				d0/f
				d9
				d9/f
				$ hg verify -q
				$ hg tracked --addexclude d0
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				deleting data/d0/f.i
				$ hg tracked
				I path:d3
				I path:d9
				X path:d0
				X path:d3/f
				$ hg debugrebuildfncache
				fncache already up to date
				$ find *
				d9
				d9/f

				Make a 15 of changes to d9 to test the path without --verbose
				(Note: using regexes instead of "* (glob)" because if the test fails, it
				produces more sensible diffs)
				$ hg tracked
				I path:d3
				I path:d9
				X path:d0
				X path:d3/f
				$ for x in `$TESTDIR/seq.py 1 15`
				> do
				> echo local change >> d9/f
				> hg commit -m "change $x to d9/f"
				> done
				$ hg tracked --removeinclude d9
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				The following changeset(s) or their ancestors have local changes not on the remote:
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				...and 5 more, use --verbose to list all
				abort: local changes found
				(use --force-delete-local-changes to ignore)
				[255]
				Now test it with verbose.
				$ hg tracked --removeinclude d9 --verbose
				comparing with ssh://user@dummy/master
				searching for changes
				looking for local changes to affected paths
				The following changeset(s) or their ancestors have local changes not on the remote:
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				^[0-9a-f]{12}$ (re)
				abort: local changes found
				(use --force-delete-local-changes to ignore)
				[255]

This is an archive of the discontinued Mercurial Phabricator instance.

narrow: import experimental extension from narrowhg revision cb51d673e9c5ClosedPublic

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 5541

hgext/narrow/__init__.py

hgext/narrow/narrowbundle2.py

hgext/narrow/narrowchangegroup.py

hgext/narrow/narrowcommands.py

hgext/narrow/narrowcopies.py

hgext/narrow/narrowdirstate.py

hgext/narrow/narrowmerge.py

hgext/narrow/narrowpatch.py

hgext/narrow/narrowrepo.py

hgext/narrow/narrowrevlog.py

hgext/narrow/narrowspec.py

hgext/narrow/narrowtemplates.py

hgext/narrow/narrowwirepeer.py

setup.py

tests/narrow-library.sh

tests/test-help.t

tests/test-narrow-acl.t

tests/test-narrow-archive.t

tests/test-narrow-clone-no-ellipsis.t

tests/test-narrow-clone-non-narrow-server.t

tests/test-narrow-clone-nonlinear.t

tests/test-narrow-clone.t

tests/test-narrow-commit-tree.t

tests/test-narrow-commit.t

tests/test-narrow-copies.t

tests/test-narrow-debugcommands.t

tests/test-narrow-debugrebuilddirstate.t

tests/test-narrow-exchange-merges.t

tests/test-narrow-exchange.t

tests/test-narrow-expanddirstate.t

tests/test-narrow-merge-tree.t

tests/test-narrow-merge.t

tests/test-narrow-patch-tree.t

tests/test-narrow-patch.t

tests/test-narrow-patterns.t

tests/test-narrow-pull.t

tests/test-narrow-rebase.t

tests/test-narrow-shallow-merges.t

tests/test-narrow-shallow.t

tests/test-narrow-strip-tree.t

tests/test-narrow-strip.t

tests/test-narrow-tree.t

tests/test-narrow-update.t

tests/test-narrow-widen-tree.t

tests/test-narrow-widen.t

tests/test-narrow.t

narrow: import experimental extension from narrowhg revision cb51d673e9c5
ClosedPublic

Revision Contents
Changeset List

hgext/narrow/init.py