( )⚙ D4231 mail: always fall back to iso-8859-1 if us-ascii won't work (BC)

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4231

mail: always fall back to iso-8859-1 if us-ascii won't work (BC)
ClosedPublic

Authored by durin42 on Aug 9 2018, 10:07 PM.

Details

Reviewers
indygreg
Group Reviewers
hg-reviewers
Commits
rHGede768cfe83e: mail: always fall back to iso-8859-1 if us-ascii won't work (BC)
Summary

It looks like this was a well-intentioned backwards compat hack for
previewing the output of hg email in a stable way. Unfortunately I
think this hack's time has come, because Python 3 does a much better
job of ensuring it actually emits *valid* email messages. In
particular, Python 2 would blindly trust us that the bytes we handed
it were valid for the encoding we claimed, but Python 3 has some more
sniff-tests that we end up failing.

As a result, if we're going to print an email to the terminal, try
us-ascii first, but if that fails go straight to iso-8859-1 which
should be reasonably readable for ascii-compatible patch bodies. This
*will* be a breaking change for ascii-incompatible textual patch
content, but I don't think that's avoidable if we want to continue
using the email library from the stdlib.

.. bc::

Emails from the patchbomb extension will always be printed as though
they are iso-8859-1 if they're not valid us-ascii. Previously,
previewed emails were always claimed to be us-ascii and might
contain invalid byte sequences.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

durin42 created this revision.Aug 9 2018, 10:07 PM
indygreg accepted this revision.Aug 9 2018, 11:29 PM
indygreg added a subscriber: indygreg.

This seems reasonable to me. It's only changing the email that is displayed. So I don't think we need to scrutinize this too much.

This revision is now accepted and ready to land.Aug 9 2018, 11:29 PM
Closed by commit rHGede768cfe83e: mail: always fall back to iso-8859-1 if us-ascii won't work (BC) (authored by durin42). · Explain WhyAug 9 2018, 11:29 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/mail.py (2 lines)
Mtests/test-patchbomb.t (18 lines)

Diff 10263

mercurial/mail.py

'''Return MIME message suitable for a patch. '''Return MIME message suitable for a patch.
Charset will be detected by first trying to decode as us-ascii, then utf-8, Charset will be detected by first trying to decode as us-ascii, then utf-8,
and finally the global encodings. If all those fail, fall back to and finally the global encodings. If all those fail, fall back to
ISO-8859-1, an encoding with that allows all byte sequences. ISO-8859-1, an encoding with that allows all byte sequences.
Transfer encodings will be used if necessary.''' Transfer encodings will be used if necessary.'''
cs = ['us-ascii', 'utf-8', encoding.encoding, encoding.fallbackencoding] cs = ['us-ascii', 'utf-8', encoding.encoding, encoding.fallbackencoding]
if display: if display:
return mimetextqp(s, subtype, 'us-ascii') cs = ['us-ascii']
for charset in cs: for charset in cs:
try: try:
s.decode(pycompat.sysstr(charset)) s.decode(pycompat.sysstr(charset))
return mimetextqp(s, subtype, codec2iana(charset)) return mimetextqp(s, subtype, codec2iana(charset))
except UnicodeDecodeError: except UnicodeDecodeError:
pass pass
return mimetextqp(s, subtype, "iso-8859-1") return mimetextqp(s, subtype, "iso-8859-1")

tests/test-patchbomb.t

no mime encoding for email --test: no mime encoding for email --test:
$ hg email --date '1970-1-1 0:4' -f quux -t foo -c bar -r tip -n $ hg email --date '1970-1-1 0:4' -f quux -t foo -c bar -r tip -n
this patch series consists of 1 patches. this patch series consists of 1 patches.
displaying [PATCH] utf-8 content ... displaying [PATCH] utf-8 content ...
MIME-Version: 1.0 MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii" Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: quoted-printable
Subject: [PATCH] utf-8 content Subject: [PATCH] utf-8 content
X-Mercurial-Node: 909a00e13e9d78b575aeee23dddbada46d5a143f X-Mercurial-Node: 909a00e13e9d78b575aeee23dddbada46d5a143f
X-Mercurial-Series-Index: 1 X-Mercurial-Series-Index: 1
X-Mercurial-Series-Total: 1 X-Mercurial-Series-Total: 1
Message-Id: <909a00e13e9d78b575ae.240@*> (glob) Message-Id: <909a00e13e9d78b575ae.240@*> (glob)
X-Mercurial-Series-Id: <909a00e13e9d78b575ae.240@*> (glob) X-Mercurial-Series-Id: <909a00e13e9d78b575ae.240@*> (glob)
User-Agent: Mercurial-patchbomb/* (glob) User-Agent: Mercurial-patchbomb/* (glob)
Date: Thu, 01 Jan 1970 00:04:00 +0000 Date: Thu, 01 Jan 1970 00:04:00 +0000
@@ -0,0 +1,3 @@ @@ -0,0 +1,3 @@
+a multiline +a multiline
+ +
+description +description
diff -r ff2c9fa2018b -r 909a00e13e9d utf diff -r ff2c9fa2018b -r 909a00e13e9d utf
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/utf Thu Jan 01 00:00:04 1970 +0000 +++ b/utf Thu Jan 01 00:00:04 1970 +0000
@@ -0,0 +1,1 @@ @@ -0,0 +1,1 @@
+h\xc3\xb6mma! (esc) +h=C3=B6mma!
mime encoded mbox (base64): mime encoded mbox (base64):
$ hg email --date '1970-1-1 0:4' -f 'Q <quux>' -t foo -c bar -r tip -m mbox $ hg email --date '1970-1-1 0:4' -f 'Q <quux>' -t foo -c bar -r tip -m mbox
this patch series consists of 1 patches. this patch series consists of 1 patches.
sending [PATCH] utf-8 content ... sending [PATCH] utf-8 content ...
diff -r 97d72e5f12c7 -r ff2c9fa2018b c diff -r 97d72e5f12c7 -r ff2c9fa2018b c
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/c Thu Jan 01 00:00:03 1970 +0000 +++ b/c Thu Jan 01 00:00:03 1970 +0000
@@ -0,0 +1,1 @@ @@ -0,0 +1,1 @@
+c +c
displaying [PATCH 2 of 6] utf-8 content ... displaying [PATCH 2 of 6] utf-8 content ...
MIME-Version: 1.0 MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii" Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: quoted-printable
Subject: [PATCH 2 of 6] utf-8 content Subject: [PATCH 2 of 6] utf-8 content
X-Mercurial-Node: 909a00e13e9d78b575aeee23dddbada46d5a143f X-Mercurial-Node: 909a00e13e9d78b575aeee23dddbada46d5a143f
X-Mercurial-Series-Index: 2 X-Mercurial-Series-Index: 2
X-Mercurial-Series-Total: 6 X-Mercurial-Series-Total: 6
Message-Id: <909a00e13e9d78b575ae.315532862@*> (glob) Message-Id: <909a00e13e9d78b575ae.315532862@*> (glob)
X-Mercurial-Series-Id: <ff2c9fa2018b15fa74b3.315532861@*> (glob) X-Mercurial-Series-Id: <ff2c9fa2018b15fa74b3.315532861@*> (glob)
In-Reply-To: <patchbomb.315532860@*> (glob) In-Reply-To: <patchbomb.315532860@*> (glob)
References: <patchbomb.315532860@*> (glob) References: <patchbomb.315532860@*> (glob)
@@ -0,0 +1,3 @@ @@ -0,0 +1,3 @@
+a multiline +a multiline
+ +
+description +description
diff -r ff2c9fa2018b -r 909a00e13e9d utf diff -r ff2c9fa2018b -r 909a00e13e9d utf
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/utf Thu Jan 01 00:00:04 1970 +0000 +++ b/utf Thu Jan 01 00:00:04 1970 +0000
@@ -0,0 +1,1 @@ @@ -0,0 +1,1 @@
+h\xc3\xb6mma! (esc) +h=C3=B6mma!
displaying [PATCH 3 of 6] long line ... displaying [PATCH 3 of 6] long line ...
MIME-Version: 1.0 MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii" Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable
Subject: [PATCH 3 of 6] long line Subject: [PATCH 3 of 6] long line
X-Mercurial-Node: a2ea8fc83dd8b93cfd86ac97b28287204ab806e1 X-Mercurial-Node: a2ea8fc83dd8b93cfd86ac97b28287204ab806e1
X-Mercurial-Series-Index: 3 X-Mercurial-Series-Index: 3
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx= xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+foo +foo
+ +
+bar +bar
displaying [PATCH 4 of 6] isolatin 8-bit encoding ... displaying [PATCH 4 of 6] isolatin 8-bit encoding ...
MIME-Version: 1.0 MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii" Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: quoted-printable
Subject: [PATCH 4 of 6] isolatin 8-bit encoding Subject: [PATCH 4 of 6] isolatin 8-bit encoding
X-Mercurial-Node: 240fb913fc1b7ff15ddb9f33e73d82bf5277c720 X-Mercurial-Node: 240fb913fc1b7ff15ddb9f33e73d82bf5277c720
X-Mercurial-Series-Index: 4 X-Mercurial-Series-Index: 4
X-Mercurial-Series-Total: 6 X-Mercurial-Series-Total: 6
Message-Id: <240fb913fc1b7ff15ddb.315532864@*> (glob) Message-Id: <240fb913fc1b7ff15ddb.315532864@*> (glob)
X-Mercurial-Series-Id: <ff2c9fa2018b15fa74b3.315532861@*> (glob) X-Mercurial-Series-Id: <ff2c9fa2018b15fa74b3.315532861@*> (glob)
In-Reply-To: <patchbomb.315532860@*> (glob) In-Reply-To: <patchbomb.315532860@*> (glob)
References: <patchbomb.315532860@*> (glob) References: <patchbomb.315532860@*> (glob)
# Node ID 240fb913fc1b7ff15ddb9f33e73d82bf5277c720 # Node ID 240fb913fc1b7ff15ddb9f33e73d82bf5277c720
# Parent a2ea8fc83dd8b93cfd86ac97b28287204ab806e1 # Parent a2ea8fc83dd8b93cfd86ac97b28287204ab806e1
isolatin 8-bit encoding isolatin 8-bit encoding
diff -r a2ea8fc83dd8 -r 240fb913fc1b isolatin diff -r a2ea8fc83dd8 -r 240fb913fc1b isolatin
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/isolatin Thu Jan 01 00:00:05 1970 +0000 +++ b/isolatin Thu Jan 01 00:00:05 1970 +0000
@@ -0,0 +1,1 @@ @@ -0,0 +1,1 @@
+h\xf6mma! (esc) +h=F6mma!
displaying [PATCH 5 of 6] Added tag zero, zero.foo for changeset 8580ff50825a ... displaying [PATCH 5 of 6] Added tag zero, zero.foo for changeset 8580ff50825a ...
MIME-Version: 1.0 MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii" Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit
Subject: [PATCH 5 of 6] Added tag zero, zero.foo for changeset 8580ff50825a Subject: [PATCH 5 of 6] Added tag zero, zero.foo for changeset 8580ff50825a
X-Mercurial-Node: 5d5ef15dfe5e7bd3a4ee154b5fff76c7945ec433 X-Mercurial-Node: 5d5ef15dfe5e7bd3a4ee154b5fff76c7945ec433
X-Mercurial-Series-Index: 5 X-Mercurial-Series-Index: 5