This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3953

mail: fix _encode to be more correct on Python 3
ClosedPublic

Authored by durin42 on Jul 16 2018, 7:16 PM.

Details

Reviewers
indygreg
Group Reviewers
hg-reviewers
Commits
rHG858fe9625dab: mail: fix _encode to be more correct on Python 3
Summary

This code appears to be on the wrong side of the law in Python 2, at
least some of the time. In Python 3, it's definitely wrong in places,
but fortunately that's easy to fix.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

durin42 created this revision.Jul 16 2018, 7:16 PM
indygreg accepted this revision.Aug 9 2018, 10:58 PM
indygreg added a subscriber: indygreg.
indygreg added inline comments.
mercurial/mail.py
249–272

Maybe this should be renamed _encodelossy to avoid surprises?

This revision is now accepted and ready to land.Aug 9 2018, 10:58 PM
Closed by commit rHG858fe9625dab: mail: fix _encode to be more correct on Python 3 (authored by durin42). · Explain WhyAug 9 2018, 10:59 PM
This revision was automatically updated to reflect the committed changes.
yuja added a subscriber: yuja.Aug 11 2018, 10:17 PM

def _encode(ui, s, charsets):

'''Returns (converted) string, charset tuple.
Finds out best charset by cycling through sendcharsets in descending
order. Tries both encoding and fallbackencoding for input. Only as
last resort send as is in fake ascii.
Caveat: Do not use for mail parts containing patches!'''

Maybe this should be renamed _encodelossy to avoid surprises?

It isn't lossy in that any input bytes will never be dropped. If no reasonable
charset found, it falls back to 'us-ascii' to send bytes transparently.

Revision Contents
Changeset List

PathPackages
Mmercurial/mail.py (21 lines)

Diff 10245

mercurial/mail.py

charsets = [cs.lower() for cs in ui.configlist('email', 'charsets')] charsets = [cs.lower() for cs in ui.configlist('email', 'charsets')]
fallbacks = [encoding.fallbackencoding.lower(), fallbacks = [encoding.fallbackencoding.lower(),
encoding.encoding.lower(), 'utf-8'] encoding.encoding.lower(), 'utf-8']
for cs in fallbacks: # find unique charsets while keeping order for cs in fallbacks: # find unique charsets while keeping order
if cs not in charsets: if cs not in charsets:
charsets.append(cs) charsets.append(cs)
return [cs for cs in charsets if not cs.endswith('ascii')] return [cs for cs in charsets if not cs.endswith('ascii')]
def _encode(ui, s, charsets): def _encode(ui, s, charsets):
'''Returns (converted) string, charset tuple. '''Returns (converted) string, charset tuple.
Finds out best charset by cycling through sendcharsets in descending Finds out best charset by cycling through sendcharsets in descending
order. Tries both encoding and fallbackencoding for input. Only as order. Tries both encoding and fallbackencoding for input. Only as
last resort send as is in fake ascii. last resort send as is in fake ascii.
Caveat: Do not use for mail parts containing patches!''' Caveat: Do not use for mail parts containing patches!'''
sendcharsets = charsets or _charsets(ui)
if not isinstance(s, bytes):
# We have unicode data, which we need to try and encode to
# some reasonable-ish encoding. Try the encodings the user
# wants, and fall back to garbage-in-ascii.
for ocs in sendcharsets:
try:
return s.encode(pycompat.sysstr(ocs)), ocs
except UnicodeEncodeError:
pass
except LookupError:
ui.warn(_('ignoring invalid sendcharset: %s\n') % ocs)
else:
# Everything failed, ascii-armor what we've got and send it.
return s.encode('ascii', 'backslashreplace')
# We have a bytes of unknown encoding. We'll try and guess a valid
# encoding, falling back to pretending we had ascii even though we
# know that's wrong.
indygregUnsubmitted
Not Done

Maybe this should be renamed _encodelossy to avoid surprises?

indygreg: Maybe this should be renamed `_encodelossy` to avoid surprises?
try: try:
s.decode('ascii') s.decode('ascii')
except UnicodeDecodeError: except UnicodeDecodeError:
sendcharsets = charsets or _charsets(ui)
for ics in (encoding.encoding, encoding.fallbackencoding): for ics in (encoding.encoding, encoding.fallbackencoding):
try: try:
u = s.decode(ics) u = s.decode(ics)
except UnicodeDecodeError: except UnicodeDecodeError:
continue continue
for ocs in sendcharsets: for ocs in sendcharsets:
try: try:
return u.encode(ocs), ocs return u.encode(pycompat.sysstr(ocs)), ocs
except UnicodeEncodeError: except UnicodeEncodeError:
pass pass
except LookupError: except LookupError:
ui.warn(_('ignoring invalid sendcharset: %s\n') % ocs) ui.warn(_('ignoring invalid sendcharset: %s\n') % ocs)
# if ascii, or all conversion attempts fail, send (broken) ascii # if ascii, or all conversion attempts fail, send (broken) ascii
return s, 'us-ascii' return s, 'us-ascii'
def headencode(ui, s, charsets=None, display=False): def headencode(ui, s, charsets=None, display=False):