OSS-Fuzz caught this in its ubsan mode[0]. I'm not worried about a
security issue here because in practice this should work out the way
we naively expected, we're just making things explicit to the
compiler with the casts.
0: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=8876