When the client supports v2 responses, the fallback to the legacy
response is undesirable. If the zlib is acceptable for the client,
it should have said so in first place.
We're not using the Accept* HTTP headers, so the use of HTTP 406 is not appropriate. See https://tools.ietf.org/html/rfc7231#section-6.5.6. I believe if you scour the mailing list or even the commit messages, you'll find text explaining why we explicitly chose to not use the Accept headers for protocol negotiation.
I do think I'm OK with aborting the request if the client request is "malformed." Although it's a BC break and needs to be called out as such. Since we already shipped forgiving code, others may insist we keep the existing behavior. They have a point.
We use 400 for parts of hgweb. But not the wire protocol parts. And the use of 400 should arguably be avoided because it is supposed to mean the HTTP request message itself was malformed. A lot of people (including our uses in hgweb) extend 400 to mean things like the query string parameters are invalid. RFC 7231 is still vague in its wording and does seem to allow liberal use of this status code. That's probably because of common use of 400 in the wild.
In the wire protocol today, it looks like we use 200 and a Content-Type: application/hg-error to indicate error. I think this is what we should use here (assuming existing client codes handles the error sanely). It should, since httppeer.py always checks for this content-type as part of processing every HTTP response.
For the next submission, please add a test showing that an hg operation hitting this error in the context of hg pull or hg clone does something reasonable.