This is an archive of the discontinued Mercurial Phabricator instance.

Differential D7770

zstandard: vendor python-zstandard 0.13.0
ClosedPublic

Authored by indygreg on Dec 28 2019, 1:13 PM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHGde7838053207: zstandard: vendor python-zstandard 0.13.0
Summary

Version 0.13.0 of the package was just released. It contains
an upgraded zstd C library which can result in some performance
wins, official support for Python 3.8, and a blackened code base.

There were no meaningful code or functionality changes in this
release of python-zstandard: just reformatting and an upgraded
zstd library version. So the diff seems much larger than what it
is.

Files were added without modifications.

The clang-format-ignorelist file was updated to reflect a new
header file in the zstd distribution.

  1. no-check-commit because 3rd party code has different style guidelines

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

indygreg created this revision.Dec 28 2019, 1:13 PM
pulkit accepted this revision.Dec 30 2019, 8:03 AM
This revision is now accepted and ready to land.Dec 30 2019, 8:03 AM
indygreg added a commit: rHGde7838053207: zstandard: vendor python-zstandard 0.13.0.Dec 30 2019, 8:33 AM
Closed by commit rHGde7838053207: zstandard: vendor python-zstandard 0.13.0 (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mcontrib/clang-format-ignorelist (1 line)
Mcontrib/python-zstandard/NEWS.rst (21 lines)
Mcontrib/python-zstandard/README.rst (2 lines)
Mcontrib/python-zstandard/c-ext/python-zstandard.h (2 lines)
Mcontrib/python-zstandard/make_cffi.py (188 lines)
Mcontrib/python-zstandard/setup.py (82 lines)
Mcontrib/python-zstandard/setup_zstd.py (224 lines)
Mcontrib/python-zstandard/tests/common.py (76 lines)
Mcontrib/python-zstandard/tests/test_buffer_util.py (127 lines)
Mcontrib/python-zstandard/tests/test_compressor.py (811 lines)
Mcontrib/python-zstandard/tests/test_compressor_fuzzing.py (631 lines)
Mcontrib/python-zstandard/tests/test_data_structures.py (121 lines)
Mcontrib/python-zstandard/tests/test_data_structures_fuzzing.py (135 lines)
Mcontrib/python-zstandard/tests/test_decompressor.py (729 lines)
Mcontrib/python-zstandard/tests/test_decompressor_fuzzing.py (335 lines)
Mcontrib/python-zstandard/tests/test_estimate_sizes.py (6 lines)
Mcontrib/python-zstandard/tests/test_module_attributes.py (93 lines)
Mcontrib/python-zstandard/tests/test_train_dictionary.py (47 lines)
Mcontrib/python-zstandard/zstandard/__init__.py (46 lines)
Mcontrib/python-zstandard/zstandard/cffi.py (1196 lines)
Mcontrib/python-zstandard/zstd.c (2 lines)
Mcontrib/python-zstandard/zstd/common/bitstream.h (5 lines)
Mcontrib/python-zstandard/zstd/common/compiler.h (16 lines)
Mcontrib/python-zstandard/zstd/common/fse.h (2 lines)
Mcontrib/python-zstandard/zstd/common/fse_decompress.c (2 lines)
Mcontrib/python-zstandard/zstd/common/mem.h (73 lines)
Mcontrib/python-zstandard/zstd/common/pool.c (10 lines)
Mcontrib/python-zstandard/zstd/common/threading.h (33 lines)
Mcontrib/python-zstandard/zstd/common/threading.c (47 lines)
Mcontrib/python-zstandard/zstd/common/zstd_internal.h (87 lines)
Mcontrib/python-zstandard/zstd/compress/zstd_compress.c (1099 lines)
Mcontrib/python-zstandard/zstd/compress/zstd_compress_internal.h (66 lines)
Mcontrib/python-zstandard/zstd/compress/zstd_compress_literals.h (2 lines)
Mcontrib/python-zstandard/zstd/compress/zstd_compress_literals.c (15 lines)
Mcontrib/python-zstandard/zstd/compress/zstd_compress_sequences.h (2 lines)
Mcontrib/python-zstandard/zstd/compress/zstd_compress_sequences.c (6 lines)
AMcontrib/python-zstandard/zstd/compress/zstd_cwksp.h (535 lines)
Mcontrib/python-zstandard/zstd/compress/zstd_double_fast.c (18 lines)
Mcontrib/python-zstandard/zstd/compress/zstd_fast.c (69 lines)
Mcontrib/python-zstandard/zstd/compress/zstd_lazy.c (10 lines)
Mcontrib/python-zstandard/zstd/compress/zstd_ldm.c (8 lines)
Mcontrib/python-zstandard/zstd/compress/zstd_opt.c (2 lines)
Mcontrib/python-zstandard/zstd/compress/zstdmt_compress.c (58 lines)
Mcontrib/python-zstandard/zstd/decompress/huf_decompress.c (2 lines)
Mcontrib/python-zstandard/zstd/decompress/zstd_decompress.c (30 lines)
Mcontrib/python-zstandard/zstd/decompress/zstd_decompress_block.c (288 lines)
Mcontrib/python-zstandard/zstd/deprecated/zbuff.h (11 lines)
Mcontrib/python-zstandard/zstd/dictBuilder/cover.c (4 lines)
Mcontrib/python-zstandard/zstd/dictBuilder/zdict.c (2 lines)
Mcontrib/python-zstandard/zstd/zstd.h (220 lines)

Diff 19001

contrib/clang-format-ignorelist

contrib/python-zstandard/zstd/compress/hist.h contrib/python-zstandard/zstd/compress/hist.h
contrib/python-zstandard/zstd/compress/huf_compress.c contrib/python-zstandard/zstd/compress/huf_compress.c
contrib/python-zstandard/zstd/compress/zstd_compress.c contrib/python-zstandard/zstd/compress/zstd_compress.c
contrib/python-zstandard/zstd/compress/zstd_compress_internal.h contrib/python-zstandard/zstd/compress/zstd_compress_internal.h
contrib/python-zstandard/zstd/compress/zstd_compress_literals.c contrib/python-zstandard/zstd/compress/zstd_compress_literals.c
contrib/python-zstandard/zstd/compress/zstd_compress_literals.h contrib/python-zstandard/zstd/compress/zstd_compress_literals.h
contrib/python-zstandard/zstd/compress/zstd_compress_sequences.c contrib/python-zstandard/zstd/compress/zstd_compress_sequences.c
contrib/python-zstandard/zstd/compress/zstd_compress_sequences.h contrib/python-zstandard/zstd/compress/zstd_compress_sequences.h
contrib/python-zstandard/zstd/compress/zstd_cwksp.h
contrib/python-zstandard/zstd/compress/zstd_double_fast.c contrib/python-zstandard/zstd/compress/zstd_double_fast.c
contrib/python-zstandard/zstd/compress/zstd_double_fast.h contrib/python-zstandard/zstd/compress/zstd_double_fast.h
contrib/python-zstandard/zstd/compress/zstd_fast.c contrib/python-zstandard/zstd/compress/zstd_fast.c
contrib/python-zstandard/zstd/compress/zstd_fast.h contrib/python-zstandard/zstd/compress/zstd_fast.h
contrib/python-zstandard/zstd/compress/zstd_lazy.c contrib/python-zstandard/zstd/compress/zstd_lazy.c
contrib/python-zstandard/zstd/compress/zstd_lazy.h contrib/python-zstandard/zstd/compress/zstd_lazy.h
contrib/python-zstandard/zstd/compress/zstd_ldm.c contrib/python-zstandard/zstd/compress/zstd_ldm.c
contrib/python-zstandard/zstd/compress/zstd_ldm.h contrib/python-zstandard/zstd/compress/zstd_ldm.h

contrib/python-zstandard/NEWS.rst

* Audit for ability to control read/write sizes on all APIs. * Audit for ability to control read/write sizes on all APIs.
* Detect memory leaks via bench.py. * Detect memory leaks via bench.py.
* Remove low-level compression parameters from ``ZstdCompressor.__init__`` and * Remove low-level compression parameters from ``ZstdCompressor.__init__`` and
require use of ``CompressionParameters``. require use of ``CompressionParameters``.
* Expose ``ZSTD_getFrameProgression()`` from more compressor types. * Expose ``ZSTD_getFrameProgression()`` from more compressor types.
* Support modifying compression parameters mid operation when supported by * Support modifying compression parameters mid operation when supported by
zstd API. zstd API.
* Expose ``ZSTD_CLEVEL_DEFAULT`` constant. * Expose ``ZSTD_CLEVEL_DEFAULT`` constant.
* Expose ``ZSTD_SRCSIZEHINT_{MIN,MAX}`` constants.
* Support ``ZSTD_p_forceAttachDict`` compression parameter. * Support ``ZSTD_p_forceAttachDict`` compression parameter.
* Support ``ZSTD_dictForceLoad`` dictionary compression parameter.
* Support ``ZSTD_c_targetCBlockSize`` compression parameter.
* Support ``ZSTD_c_literalCompressionMode `` compression parameter. * Support ``ZSTD_c_literalCompressionMode`` compression parameter.
* Support ``ZSTD_c_srcSizeHint`` compression parameter.
* Use ``ZSTD_CCtx_getParameter()``/``ZSTD_CCtxParam_getParameter()`` for retrieving * Use ``ZSTD_CCtx_getParameter()``/``ZSTD_CCtxParam_getParameter()`` for retrieving
compression parameters. compression parameters.
* Consider exposing ``ZSTDMT_toFlushNow()``. * Consider exposing ``ZSTDMT_toFlushNow()``.
* Expose ``ZDICT_trainFromBuffer_fastCover()``, * Expose ``ZDICT_trainFromBuffer_fastCover()``,
``ZDICT_optimizeTrainFromBuffer_fastCover``. ``ZDICT_optimizeTrainFromBuffer_fastCover``.
* Expose ``ZSTD_Sequence`` struct and related ``ZSTD_getSequences()`` API.
* Expose and enforce ``ZSTD_minCLevel()`` for minimum compression level. * Expose and enforce ``ZSTD_minCLevel()`` for minimum compression level.
* Consider a ``chunker()`` API for decompression. * Consider a ``chunker()`` API for decompression.
* Consider stats for ``chunker()`` API, including finding the last consumed * Consider stats for ``chunker()`` API, including finding the last consumed
offset of input data. offset of input data.
* Consider exposing ``ZSTD_cParam_getBounds()`` and * Consider exposing ``ZSTD_cParam_getBounds()`` and
``ZSTD_dParam_getBounds()`` APIs. ``ZSTD_dParam_getBounds()`` APIs.
* Consider controls over resetting compression contexts (session only, parameters, * Consider controls over resetting compression contexts (session only, parameters,
or session and parameters). or session and parameters).
* Actually use the CFFI backend in fuzzing tests. * Actually use the CFFI backend in fuzzing tests.
Other Actions Not Blocking Release Other Actions Not Blocking Release
--------------------------------------- ---------------------------------------
* Support for block compression APIs. * Support for block compression APIs.
* API for ensuring max memory ceiling isn't exceeded. * API for ensuring max memory ceiling isn't exceeded.
* Move off nose for testing. * Move off nose for testing.
0.13.0 (released 2019-12-28)
============================
Changes
-------
* ``pytest-xdist`` ``pytest`` extension is now installed so tests can be
run in parallel.
* CI now builds ``manylinux2010`` and ``manylinux2014`` binary wheels
instead of a mix of ``manylinux2010`` and ``manylinux1``.
* Official support for Python 3.8 has been added.
* Bundled zstandard library upgraded from 1.4.3 to 1.4.4.
* Python code has been reformatted with black.
0.12.0 (released 2019-09-15) 0.12.0 (released 2019-09-15)
============================ ============================
Backwards Compatibility Notes Backwards Compatibility Notes
----------------------------- -----------------------------
* Support for Python 3.4 has been dropped since Python 3.4 is no longer * Support for Python 3.4 has been dropped since Python 3.4 is no longer
a supported Python version upstream. (But it will likely continue to a supported Python version upstream. (But it will likely continue to

contrib/python-zstandard/README.rst

the author. For convenience, that repository is frequently synchronized to the author. For convenience, that repository is frequently synchronized to
https://github.com/indygreg/python-zstandard. https://github.com/indygreg/python-zstandard.
| |ci-status| | |ci-status|
Requirements Requirements
============ ============
This extension is designed to run with Python 2.7, 3.4, 3.5, 3.6, and 3.7 This extension is designed to run with Python 2.7, 3.5, 3.6, 3.7, and 3.8
on common platforms (Linux, Windows, and OS X). On PyPy (both PyPy2 and PyPy3) we support version 6.0.0 and above. on common platforms (Linux, Windows, and OS X). On PyPy (both PyPy2 and PyPy3) we support version 6.0.0 and above.
x86 and x86_64 are well-tested on Windows. Only x86_64 is well-tested on Linux and macOS. x86 and x86_64 are well-tested on Windows. Only x86_64 is well-tested on Linux and macOS.
Installing Installing
========== ==========
This package is uploaded to PyPI at https://pypi.python.org/pypi/zstandard. This package is uploaded to PyPI at https://pypi.python.org/pypi/zstandard.
So, to install this package:: So, to install this package::

contrib/python-zstandard/c-ext/python-zstandard.h

#include "structmember.h" #include "structmember.h"
#define ZSTD_STATIC_LINKING_ONLY #define ZSTD_STATIC_LINKING_ONLY
#define ZDICT_STATIC_LINKING_ONLY #define ZDICT_STATIC_LINKING_ONLY
#include <zstd.h> #include <zstd.h>
#include <zdict.h> #include <zdict.h>
/* Remember to change the string in zstandard/__init__ as well */ /* Remember to change the string in zstandard/__init__ as well */
#define PYTHON_ZSTANDARD_VERSION "0.12.0" #define PYTHON_ZSTANDARD_VERSION "0.13.0"
typedef enum { typedef enum {
compressorobj_flush_finish, compressorobj_flush_finish,
compressorobj_flush_block, compressorobj_flush_block,
} CompressorObj_Flush; } CompressorObj_Flush;
/* /*
Represents a ZstdCompressionParameters type. Represents a ZstdCompressionParameters type.

contrib/python-zstandard/make_cffi.py

import os import os
import re import re
import subprocess import subprocess
import tempfile import tempfile
HERE = os.path.abspath(os.path.dirname(__file__)) HERE = os.path.abspath(os.path.dirname(__file__))
SOURCES = ['zstd/%s' % p for p in ( SOURCES = [
'common/debug.c', "zstd/%s" % p
'common/entropy_common.c', for p in (
'common/error_private.c', "common/debug.c",
'common/fse_decompress.c', "common/entropy_common.c",
'common/pool.c', "common/error_private.c",
'common/threading.c', "common/fse_decompress.c",
'common/xxhash.c', "common/pool.c",
'common/zstd_common.c', "common/threading.c",
'compress/fse_compress.c', "common/xxhash.c",
'compress/hist.c', "common/zstd_common.c",
'compress/huf_compress.c', "compress/fse_compress.c",
'compress/zstd_compress.c', "compress/hist.c",
'compress/zstd_compress_literals.c', "compress/huf_compress.c",
'compress/zstd_compress_sequences.c', "compress/zstd_compress.c",
'compress/zstd_double_fast.c', "compress/zstd_compress_literals.c",
'compress/zstd_fast.c', "compress/zstd_compress_sequences.c",
'compress/zstd_lazy.c', "compress/zstd_double_fast.c",
'compress/zstd_ldm.c', "compress/zstd_fast.c",
'compress/zstd_opt.c', "compress/zstd_lazy.c",
'compress/zstdmt_compress.c', "compress/zstd_ldm.c",
'decompress/huf_decompress.c', "compress/zstd_opt.c",
'decompress/zstd_ddict.c', "compress/zstdmt_compress.c",
'decompress/zstd_decompress.c', "decompress/huf_decompress.c",
'decompress/zstd_decompress_block.c', "decompress/zstd_ddict.c",
'dictBuilder/cover.c', "decompress/zstd_decompress.c",
'dictBuilder/fastcover.c', "decompress/zstd_decompress_block.c",
'dictBuilder/divsufsort.c', "dictBuilder/cover.c",
'dictBuilder/zdict.c', "dictBuilder/fastcover.c",
)] "dictBuilder/divsufsort.c",
"dictBuilder/zdict.c",
)
]
# Headers whose preprocessed output will be fed into cdef(). # Headers whose preprocessed output will be fed into cdef().
HEADERS = [os.path.join(HERE, 'zstd', *p) for p in ( HEADERS = [
('zstd.h',), os.path.join(HERE, "zstd", *p) for p in (("zstd.h",), ("dictBuilder", "zdict.h"),)
('dictBuilder', 'zdict.h'), ]
)]
INCLUDE_DIRS = [
INCLUDE_DIRS = [os.path.join(HERE, d) for d in ( os.path.join(HERE, d)
'zstd', for d in (
'zstd/common', "zstd",
'zstd/compress', "zstd/common",
'zstd/decompress', "zstd/compress",
'zstd/dictBuilder', "zstd/decompress",
)] "zstd/dictBuilder",
)
]
# cffi can't parse some of the primitives in zstd.h. So we invoke the # cffi can't parse some of the primitives in zstd.h. So we invoke the
# preprocessor and feed its output into cffi. # preprocessor and feed its output into cffi.
compiler = distutils.ccompiler.new_compiler() compiler = distutils.ccompiler.new_compiler()
# Needed for MSVC. # Needed for MSVC.
if hasattr(compiler, 'initialize'): if hasattr(compiler, "initialize"):
compiler.initialize() compiler.initialize()
# Distutils doesn't set compiler.preprocessor, so invoke the preprocessor # Distutils doesn't set compiler.preprocessor, so invoke the preprocessor
# manually. # manually.
if compiler.compiler_type == 'unix': if compiler.compiler_type == "unix":
args = list(compiler.executables['compiler']) args = list(compiler.executables["compiler"])
args.extend([ args.extend(
'-E', ["-E", "-DZSTD_STATIC_LINKING_ONLY", "-DZDICT_STATIC_LINKING_ONLY",]
'-DZSTD_STATIC_LINKING_ONLY', )
'-DZDICT_STATIC_LINKING_ONLY', elif compiler.compiler_type == "msvc":
])
elif compiler.compiler_type == 'msvc':
args = [compiler.cc] args = [compiler.cc]
args.extend([ args.extend(
'/EP', ["/EP", "/DZSTD_STATIC_LINKING_ONLY", "/DZDICT_STATIC_LINKING_ONLY",]
'/DZSTD_STATIC_LINKING_ONLY', )
'/DZDICT_STATIC_LINKING_ONLY',
])
else: else:
raise Exception('unsupported compiler type: %s' % compiler.compiler_type) raise Exception("unsupported compiler type: %s" % compiler.compiler_type)
def preprocess(path): def preprocess(path):
with open(path, 'rb') as fh: with open(path, "rb") as fh:
lines = [] lines = []
it = iter(fh) it = iter(fh)
for l in it: for l in it:
# zstd.h includes <stddef.h>, which is also included by cffi's # zstd.h includes <stddef.h>, which is also included by cffi's
# boilerplate. This can lead to duplicate declarations. So we strip # boilerplate. This can lead to duplicate declarations. So we strip
# this include from the preprocessor invocation. # this include from the preprocessor invocation.
# #
# The same things happens for including zstd.h, so give it the same # The same things happens for including zstd.h, so give it the same
# treatment. # treatment.
# #
# We define ZSTD_STATIC_LINKING_ONLY, which is redundant with the inline # We define ZSTD_STATIC_LINKING_ONLY, which is redundant with the inline
# #define in zstdmt_compress.h and results in a compiler warning. So drop # #define in zstdmt_compress.h and results in a compiler warning. So drop
# the inline #define. # the inline #define.
if l.startswith((b'#include <stddef.h>', if l.startswith(
(
b"#include <stddef.h>",
b'#include "zstd.h"', b'#include "zstd.h"',
b'#define ZSTD_STATIC_LINKING_ONLY')): b"#define ZSTD_STATIC_LINKING_ONLY",
)
):
continue continue
# The preprocessor environment on Windows doesn't define include
# paths, so the #include of limits.h fails. We work around this
# by removing that import and defining INT_MAX ourselves. This is
# a bit hacky. But it gets the job done.
# TODO make limits.h work on Windows so we ensure INT_MAX is
# correct.
if l.startswith(b"#include <limits.h>"):
l = b"#define INT_MAX 2147483647\n"
# ZSTDLIB_API may not be defined if we dropped zstd.h. It isn't # ZSTDLIB_API may not be defined if we dropped zstd.h. It isn't
# important so just filter it out. # important so just filter it out.
if l.startswith(b'ZSTDLIB_API'): if l.startswith(b"ZSTDLIB_API"):
l = l[len(b'ZSTDLIB_API '):] l = l[len(b"ZSTDLIB_API ") :]
lines.append(l) lines.append(l)
fd, input_file = tempfile.mkstemp(suffix='.h') fd, input_file = tempfile.mkstemp(suffix=".h")
os.write(fd, b''.join(lines)) os.write(fd, b"".join(lines))
os.close(fd) os.close(fd)
try: try:
env = dict(os.environ) env = dict(os.environ)
if getattr(compiler, '_paths', None): if getattr(compiler, "_paths", None):
env['PATH'] = compiler._paths env["PATH"] = compiler._paths
process = subprocess.Popen(args + [input_file], stdout=subprocess.PIPE, process = subprocess.Popen(args + [input_file], stdout=subprocess.PIPE, env=env)
env=env)
output = process.communicate()[0] output = process.communicate()[0]
ret = process.poll() ret = process.poll()
if ret: if ret:
raise Exception('preprocessor exited with error') raise Exception("preprocessor exited with error")
return output return output
finally: finally:
os.unlink(input_file) os.unlink(input_file)
def normalize_output(output): def normalize_output(output):
lines = [] lines = []
for line in output.splitlines(): for line in output.splitlines():
# CFFI's parser doesn't like __attribute__ on UNIX compilers. # CFFI's parser doesn't like __attribute__ on UNIX compilers.
if line.startswith(b'__attribute__ ((visibility ("default"))) '): if line.startswith(b'__attribute__ ((visibility ("default"))) '):
line = line[len(b'__attribute__ ((visibility ("default"))) '):] line = line[len(b'__attribute__ ((visibility ("default"))) ') :]
if line.startswith(b'__attribute__((deprecated('): if line.startswith(b"__attribute__((deprecated("):
continue continue
elif b'__declspec(deprecated(' in line: elif b"__declspec(deprecated(" in line:
continue continue
lines.append(line) lines.append(line)
return b'\n'.join(lines) return b"\n".join(lines)
ffi = cffi.FFI() ffi = cffi.FFI()
# zstd.h uses a possible undefined MIN(). Define it until # zstd.h uses a possible undefined MIN(). Define it until
# https://github.com/facebook/zstd/issues/976 is fixed. # https://github.com/facebook/zstd/issues/976 is fixed.
# *_DISABLE_DEPRECATE_WARNINGS prevents the compiler from emitting a warning # *_DISABLE_DEPRECATE_WARNINGS prevents the compiler from emitting a warning
# when cffi uses the function. Since we statically link against zstd, even # when cffi uses the function. Since we statically link against zstd, even
# if we use the deprecated functions it shouldn't be a huge problem. # if we use the deprecated functions it shouldn't be a huge problem.
ffi.set_source('_zstd_cffi', ''' ffi.set_source(
"_zstd_cffi",
"""
#define MIN(a,b) ((a)<(b) ? (a) : (b)) #define MIN(a,b) ((a)<(b) ? (a) : (b))
#define ZSTD_STATIC_LINKING_ONLY #define ZSTD_STATIC_LINKING_ONLY
#include <zstd.h> #include <zstd.h>
#define ZDICT_STATIC_LINKING_ONLY #define ZDICT_STATIC_LINKING_ONLY
#define ZDICT_DISABLE_DEPRECATE_WARNINGS #define ZDICT_DISABLE_DEPRECATE_WARNINGS
#include <zdict.h> #include <zdict.h>
''', sources=SOURCES, """,
sources=SOURCES,
include_dirs=INCLUDE_DIRS, include_dirs=INCLUDE_DIRS,
extra_compile_args=['-DZSTD_MULTITHREAD']) extra_compile_args=["-DZSTD_MULTITHREAD"],
)
DEFINE = re.compile(b'^\\#define ([a-zA-Z0-9_]+) ') DEFINE = re.compile(b"^\\#define ([a-zA-Z0-9_]+) ")
sources = [] sources = []
# Feed normalized preprocessor output for headers into the cdef parser. # Feed normalized preprocessor output for headers into the cdef parser.
for header in HEADERS: for header in HEADERS:
preprocessed = preprocess(header) preprocessed = preprocess(header)
sources.append(normalize_output(preprocessed)) sources.append(normalize_output(preprocessed))
# #define's are effectively erased as part of going through preprocessor. # #define's are effectively erased as part of going through preprocessor.
# So perform a manual pass to re-add those to the cdef source. # So perform a manual pass to re-add those to the cdef source.
with open(header, 'rb') as fh: with open(header, "rb") as fh:
for line in fh: for line in fh:
line = line.strip() line = line.strip()
m = DEFINE.match(line) m = DEFINE.match(line)
if not m: if not m:
continue continue
if m.group(1) == b'ZSTD_STATIC_LINKING_ONLY': if m.group(1) == b"ZSTD_STATIC_LINKING_ONLY":
continue continue
# The parser doesn't like some constants with complex values. # The parser doesn't like some constants with complex values.
if m.group(1) in (b'ZSTD_LIB_VERSION', b'ZSTD_VERSION_STRING'): if m.group(1) in (b"ZSTD_LIB_VERSION", b"ZSTD_VERSION_STRING"):
continue continue
# The ... is magic syntax by the cdef parser to resolve the # The ... is magic syntax by the cdef parser to resolve the
# value at compile time. # value at compile time.
sources.append(m.group(0) + b' ...') sources.append(m.group(0) + b" ...")
cdeflines = b'\n'.join(sources).splitlines() cdeflines = b"\n".join(sources).splitlines()
cdeflines = [l for l in cdeflines if l.strip()] cdeflines = [l for l in cdeflines if l.strip()]
ffi.cdef(b'\n'.join(cdeflines).decode('latin1')) ffi.cdef(b"\n".join(cdeflines).decode("latin1"))
if __name__ == '__main__': if __name__ == "__main__":
ffi.compile() ffi.compile()

contrib/python-zstandard/setup.py

import os import os
import sys import sys
from setuptools import setup from setuptools import setup
# Need change in 1.10 for ffi.from_buffer() to handle all buffer types # Need change in 1.10 for ffi.from_buffer() to handle all buffer types
# (like memoryview). # (like memoryview).
# Need feature in 1.11 for ffi.gc() to declare size of objects so we avoid # Need feature in 1.11 for ffi.gc() to declare size of objects so we avoid
# garbage collection pitfalls. # garbage collection pitfalls.
MINIMUM_CFFI_VERSION = '1.11' MINIMUM_CFFI_VERSION = "1.11"
try: try:
import cffi import cffi
# PyPy (and possibly other distros) have CFFI distributed as part of # PyPy (and possibly other distros) have CFFI distributed as part of
# them. The install_requires for CFFI below won't work. We need to sniff # them. The install_requires for CFFI below won't work. We need to sniff
# out the CFFI version here and reject CFFI if it is too old. # out the CFFI version here and reject CFFI if it is too old.
cffi_version = LooseVersion(cffi.__version__) cffi_version = LooseVersion(cffi.__version__)
if cffi_version < LooseVersion(MINIMUM_CFFI_VERSION): if cffi_version < LooseVersion(MINIMUM_CFFI_VERSION):
print('CFFI 1.11 or newer required (%s found); ' print(
'not building CFFI backend' % cffi_version, "CFFI 1.11 or newer required (%s found); "
file=sys.stderr) "not building CFFI backend" % cffi_version,
file=sys.stderr,
)
cffi = None cffi = None
except ImportError: except ImportError:
cffi = None cffi = None
import setup_zstd import setup_zstd
SUPPORT_LEGACY = False SUPPORT_LEGACY = False
SYSTEM_ZSTD = False SYSTEM_ZSTD = False
WARNINGS_AS_ERRORS = False WARNINGS_AS_ERRORS = False
if os.environ.get('ZSTD_WARNINGS_AS_ERRORS', ''): if os.environ.get("ZSTD_WARNINGS_AS_ERRORS", ""):
WARNINGS_AS_ERRORS = True WARNINGS_AS_ERRORS = True
if '--legacy' in sys.argv: if "--legacy" in sys.argv:
SUPPORT_LEGACY = True SUPPORT_LEGACY = True
sys.argv.remove('--legacy') sys.argv.remove("--legacy")
if '--system-zstd' in sys.argv: if "--system-zstd" in sys.argv:
SYSTEM_ZSTD = True SYSTEM_ZSTD = True
sys.argv.remove('--system-zstd') sys.argv.remove("--system-zstd")
if '--warnings-as-errors' in sys.argv: if "--warnings-as-errors" in sys.argv:
WARNINGS_AS_ERRORS = True WARNINGS_AS_ERRORS = True
sys.argv.remove('--warning-as-errors') sys.argv.remove("--warning-as-errors")
# Code for obtaining the Extension instance is in its own module to # Code for obtaining the Extension instance is in its own module to
# facilitate reuse in other projects. # facilitate reuse in other projects.
extensions = [ extensions = [
setup_zstd.get_c_extension(name='zstd', setup_zstd.get_c_extension(
name="zstd",
support_legacy=SUPPORT_LEGACY, support_legacy=SUPPORT_LEGACY,
system_zstd=SYSTEM_ZSTD, system_zstd=SYSTEM_ZSTD,
warnings_as_errors=WARNINGS_AS_ERRORS), warnings_as_errors=WARNINGS_AS_ERRORS,
),
] ]
install_requires = [] install_requires = []
if cffi: if cffi:
import make_cffi import make_cffi
extensions.append(make_cffi.ffi.distutils_extension()) extensions.append(make_cffi.ffi.distutils_extension())
install_requires.append('cffi>=%s' % MINIMUM_CFFI_VERSION) install_requires.append("cffi>=%s" % MINIMUM_CFFI_VERSION)
version = None version = None
with open('c-ext/python-zstandard.h', 'r') as fh: with open("c-ext/python-zstandard.h", "r") as fh:
for line in fh: for line in fh:
if not line.startswith('#define PYTHON_ZSTANDARD_VERSION'): if not line.startswith("#define PYTHON_ZSTANDARD_VERSION"):
continue continue
version = line.split()[2][1:-1] version = line.split()[2][1:-1]
break break
if not version: if not version:
raise Exception('could not resolve package version; ' raise Exception("could not resolve package version; " "this should never happen")
'this should never happen')
setup( setup(
name='zstandard', name="zstandard",
version=version, version=version,
description='Zstandard bindings for Python', description="Zstandard bindings for Python",
long_description=open('README.rst', 'r').read(), long_description=open("README.rst", "r").read(),
url='https://github.com/indygreg/python-zstandard', url="https://github.com/indygreg/python-zstandard",
author='Gregory Szorc', author="Gregory Szorc",
author_email='gregory.szorc@gmail.com', author_email="gregory.szorc@gmail.com",
license='BSD', license="BSD",
classifiers=[ classifiers=[
'Development Status :: 4 - Beta', "Development Status :: 4 - Beta",
'Intended Audience :: Developers', "Intended Audience :: Developers",
'License :: OSI Approved :: BSD License', "License :: OSI Approved :: BSD License",
'Programming Language :: C', "Programming Language :: C",
'Programming Language :: Python :: 2.7', "Programming Language :: Python :: 2.7",
'Programming Language :: Python :: 3.5', "Programming Language :: Python :: 3.5",
'Programming Language :: Python :: 3.6', "Programming Language :: Python :: 3.6",
'Programming Language :: Python :: 3.7', "Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
], ],
keywords='zstandard zstd compression', keywords="zstandard zstd compression",
packages=['zstandard'], packages=["zstandard"],
ext_modules=extensions, ext_modules=extensions,
test_suite='tests', test_suite="tests",
install_requires=install_requires, install_requires=install_requires,
tests_require=["hypothesis"],
) )

contrib/python-zstandard/setup_zstd.py

# Copyright (c) 2016-present, Gregory Szorc # Copyright (c) 2016-present, Gregory Szorc
# All rights reserved. # All rights reserved.
# #
# This software may be modified and distributed under the terms # This software may be modified and distributed under the terms
# of the BSD license. See the LICENSE file for details. # of the BSD license. See the LICENSE file for details.
import distutils.ccompiler import distutils.ccompiler
import os import os
from distutils.extension import Extension from distutils.extension import Extension
zstd_sources = ['zstd/%s' % p for p in ( zstd_sources = [
'common/debug.c', "zstd/%s" % p
'common/entropy_common.c', for p in (
'common/error_private.c', "common/debug.c",
'common/fse_decompress.c', "common/entropy_common.c",
'common/pool.c', "common/error_private.c",
'common/threading.c', "common/fse_decompress.c",
'common/xxhash.c', "common/pool.c",
'common/zstd_common.c', "common/threading.c",
'compress/fse_compress.c', "common/xxhash.c",
'compress/hist.c', "common/zstd_common.c",
'compress/huf_compress.c', "compress/fse_compress.c",
'compress/zstd_compress_literals.c', "compress/hist.c",
'compress/zstd_compress_sequences.c', "compress/huf_compress.c",
'compress/zstd_compress.c', "compress/zstd_compress_literals.c",
'compress/zstd_double_fast.c', "compress/zstd_compress_sequences.c",
'compress/zstd_fast.c', "compress/zstd_compress.c",
'compress/zstd_lazy.c', "compress/zstd_double_fast.c",
'compress/zstd_ldm.c', "compress/zstd_fast.c",
'compress/zstd_opt.c', "compress/zstd_lazy.c",
'compress/zstdmt_compress.c', "compress/zstd_ldm.c",
'decompress/huf_decompress.c', "compress/zstd_opt.c",
'decompress/zstd_ddict.c', "compress/zstdmt_compress.c",
'decompress/zstd_decompress.c', "decompress/huf_decompress.c",
'decompress/zstd_decompress_block.c', "decompress/zstd_ddict.c",
'dictBuilder/cover.c', "decompress/zstd_decompress.c",
'dictBuilder/divsufsort.c', "decompress/zstd_decompress_block.c",
'dictBuilder/fastcover.c', "dictBuilder/cover.c",
'dictBuilder/zdict.c', "dictBuilder/divsufsort.c",
)] "dictBuilder/fastcover.c",
"dictBuilder/zdict.c",
zstd_sources_legacy = ['zstd/%s' % p for p in ( )
'deprecated/zbuff_common.c', ]
'deprecated/zbuff_compress.c',
'deprecated/zbuff_decompress.c', zstd_sources_legacy = [
'legacy/zstd_v01.c', "zstd/%s" % p
'legacy/zstd_v02.c', for p in (
'legacy/zstd_v03.c', "deprecated/zbuff_common.c",
'legacy/zstd_v04.c', "deprecated/zbuff_compress.c",
'legacy/zstd_v05.c', "deprecated/zbuff_decompress.c",
'legacy/zstd_v06.c', "legacy/zstd_v01.c",
'legacy/zstd_v07.c' "legacy/zstd_v02.c",
)] "legacy/zstd_v03.c",
"legacy/zstd_v04.c",
"legacy/zstd_v05.c",
"legacy/zstd_v06.c",
"legacy/zstd_v07.c",
)
]
zstd_includes = [ zstd_includes = [
'zstd', "zstd",
'zstd/common', "zstd/common",
'zstd/compress', "zstd/compress",
'zstd/decompress', "zstd/decompress",
'zstd/dictBuilder', "zstd/dictBuilder",
] ]
zstd_includes_legacy = [ zstd_includes_legacy = [
'zstd/deprecated', "zstd/deprecated",
'zstd/legacy', "zstd/legacy",
] ]
ext_includes = [ ext_includes = [
'c-ext', "c-ext",
'zstd/common', "zstd/common",
] ]
ext_sources = [ ext_sources = [
'zstd/common/pool.c', "zstd/common/error_private.c",
'zstd/common/threading.c', "zstd/common/pool.c",
'zstd.c', "zstd/common/threading.c",
'c-ext/bufferutil.c', "zstd/common/zstd_common.c",
'c-ext/compressiondict.c', "zstd.c",
'c-ext/compressobj.c', "c-ext/bufferutil.c",
'c-ext/compressor.c', "c-ext/compressiondict.c",
'c-ext/compressoriterator.c', "c-ext/compressobj.c",
'c-ext/compressionchunker.c', "c-ext/compressor.c",
'c-ext/compressionparams.c', "c-ext/compressoriterator.c",
'c-ext/compressionreader.c', "c-ext/compressionchunker.c",
'c-ext/compressionwriter.c', "c-ext/compressionparams.c",
'c-ext/constants.c', "c-ext/compressionreader.c",
'c-ext/decompressobj.c', "c-ext/compressionwriter.c",
'c-ext/decompressor.c', "c-ext/constants.c",
'c-ext/decompressoriterator.c', "c-ext/decompressobj.c",
'c-ext/decompressionreader.c', "c-ext/decompressor.c",
'c-ext/decompressionwriter.c', "c-ext/decompressoriterator.c",
'c-ext/frameparams.c', "c-ext/decompressionreader.c",
"c-ext/decompressionwriter.c",
"c-ext/frameparams.c",
] ]
zstd_depends = [ zstd_depends = [
'c-ext/python-zstandard.h', "c-ext/python-zstandard.h",
] ]
def get_c_extension(support_legacy=False, system_zstd=False, name='zstd', def get_c_extension(
warnings_as_errors=False, root=None): support_legacy=False,
system_zstd=False,
name="zstd",
warnings_as_errors=False,
root=None,
):
"""Obtain a distutils.extension.Extension for the C extension. """Obtain a distutils.extension.Extension for the C extension.
``support_legacy`` controls whether to compile in legacy zstd format support. ``support_legacy`` controls whether to compile in legacy zstd format support.
``system_zstd`` controls whether to compile against the system zstd library. ``system_zstd`` controls whether to compile against the system zstd library.
For this to work, the system zstd library and headers must match what For this to work, the system zstd library and headers must match what
python-zstandard is coded against exactly. python-zstandard is coded against exactly.
``name`` is the module name of the C extension to produce. ``name`` is the module name of the C extension to produce.
``warnings_as_errors`` controls whether compiler warnings are turned into ``warnings_as_errors`` controls whether compiler warnings are turned into
compiler errors. compiler errors.
``root`` defines a root path that source should be computed as relative ``root`` defines a root path that source should be computed as relative
to. This should be the directory with the main ``setup.py`` that is to. This should be the directory with the main ``setup.py`` that is
being invoked. If not defined, paths will be relative to this file. being invoked. If not defined, paths will be relative to this file.
""" """
actual_root = os.path.abspath(os.path.dirname(__file__)) actual_root = os.path.abspath(os.path.dirname(__file__))
root = root or actual_root root = root or actual_root
sources = set([os.path.join(actual_root, p) for p in ext_sources]) sources = set([os.path.join(actual_root, p) for p in ext_sources])
if not system_zstd: if not system_zstd:
sources.update([os.path.join(actual_root, p) for p in zstd_sources]) sources.update([os.path.join(actual_root, p) for p in zstd_sources])
if support_legacy: if support_legacy:
sources.update([os.path.join(actual_root, p) sources.update([os.path.join(actual_root, p) for p in zstd_sources_legacy])
for p in zstd_sources_legacy])
sources = list(sources) sources = list(sources)
include_dirs = set([os.path.join(actual_root, d) for d in ext_includes]) include_dirs = set([os.path.join(actual_root, d) for d in ext_includes])
if not system_zstd: if not system_zstd:
include_dirs.update([os.path.join(actual_root, d) include_dirs.update([os.path.join(actual_root, d) for d in zstd_includes])
for d in zstd_includes])
if support_legacy: if support_legacy:
include_dirs.update([os.path.join(actual_root, d) include_dirs.update(
for d in zstd_includes_legacy]) [os.path.join(actual_root, d) for d in zstd_includes_legacy]
)
include_dirs = list(include_dirs) include_dirs = list(include_dirs)
depends = [os.path.join(actual_root, p) for p in zstd_depends] depends = [os.path.join(actual_root, p) for p in zstd_depends]
compiler = distutils.ccompiler.new_compiler() compiler = distutils.ccompiler.new_compiler()
# Needed for MSVC. # Needed for MSVC.
if hasattr(compiler, 'initialize'): if hasattr(compiler, "initialize"):
compiler.initialize() compiler.initialize()
if compiler.compiler_type == 'unix': if compiler.compiler_type == "unix":
compiler_type = 'unix' compiler_type = "unix"
elif compiler.compiler_type == 'msvc': elif compiler.compiler_type == "msvc":
compiler_type = 'msvc' compiler_type = "msvc"
elif compiler.compiler_type == 'mingw32': elif compiler.compiler_type == "mingw32":
compiler_type = 'mingw32' compiler_type = "mingw32"
else: else:
raise Exception('unhandled compiler type: %s' % raise Exception("unhandled compiler type: %s" % compiler.compiler_type)
compiler.compiler_type)
extra_args = ['-DZSTD_MULTITHREAD'] extra_args = ["-DZSTD_MULTITHREAD"]
if not system_zstd: if not system_zstd:
extra_args.append('-DZSTDLIB_VISIBILITY=') extra_args.append("-DZSTDLIB_VISIBILITY=")
extra_args.append('-DZDICTLIB_VISIBILITY=') extra_args.append("-DZDICTLIB_VISIBILITY=")
extra_args.append('-DZSTDERRORLIB_VISIBILITY=') extra_args.append("-DZSTDERRORLIB_VISIBILITY=")
if compiler_type == 'unix': if compiler_type == "unix":
extra_args.append('-fvisibility=hidden') extra_args.append("-fvisibility=hidden")
if not system_zstd and support_legacy: if not system_zstd and support_legacy:
extra_args.append('-DZSTD_LEGACY_SUPPORT=1') extra_args.append("-DZSTD_LEGACY_SUPPORT=1")
if warnings_as_errors: if warnings_as_errors:
if compiler_type in ('unix', 'mingw32'): if compiler_type in ("unix", "mingw32"):
extra_args.append('-Werror') extra_args.append("-Werror")
elif compiler_type == 'msvc': elif compiler_type == "msvc":
extra_args.append('/WX') extra_args.append("/WX")
else: else:
assert False assert False
libraries = ['zstd'] if system_zstd else [] libraries = ["zstd"] if system_zstd else []
# Python 3.7 doesn't like absolute paths. So normalize to relative. # Python 3.7 doesn't like absolute paths. So normalize to relative.
sources = [os.path.relpath(p, root) for p in sources] sources = [os.path.relpath(p, root) for p in sources]
include_dirs = [os.path.relpath(p, root) for p in include_dirs] include_dirs = [os.path.relpath(p, root) for p in include_dirs]
depends = [os.path.relpath(p, root) for p in depends] depends = [os.path.relpath(p, root) for p in depends]
# TODO compile with optimizations. # TODO compile with optimizations.
return Extension(name, sources, return Extension(
name,
sources,
include_dirs=include_dirs, include_dirs=include_dirs,
depends=depends, depends=depends,
extra_compile_args=extra_args, extra_compile_args=extra_args,
libraries=libraries) libraries=libraries,
)

contrib/python-zstandard/tests/common.py

import imp import imp
import inspect import inspect
import io import io
import os import os
import types import types
import unittest
try: try:
import hypothesis import hypothesis
except ImportError: except ImportError:
hypothesis = None hypothesis = None
class TestCase(unittest.TestCase):
if not getattr(unittest.TestCase, "assertRaisesRegex", False):
assertRaisesRegex = unittest.TestCase.assertRaisesRegexp
def make_cffi(cls): def make_cffi(cls):
"""Decorator to add CFFI versions of each test method.""" """Decorator to add CFFI versions of each test method."""
# The module containing this class definition should # The module containing this class definition should
# `import zstandard as zstd`. Otherwise things may blow up. # `import zstandard as zstd`. Otherwise things may blow up.
mod = inspect.getmodule(cls) mod = inspect.getmodule(cls)
if not hasattr(mod, 'zstd'): if not hasattr(mod, "zstd"):
raise Exception('test module does not contain "zstd" symbol') raise Exception('test module does not contain "zstd" symbol')
if not hasattr(mod.zstd, 'backend'): if not hasattr(mod.zstd, "backend"):
raise Exception('zstd symbol does not have "backend" attribute; did ' raise Exception(
'you `import zstandard as zstd`?') 'zstd symbol does not have "backend" attribute; did '
"you `import zstandard as zstd`?"
)
# If `import zstandard` already chose the cffi backend, there is nothing # If `import zstandard` already chose the cffi backend, there is nothing
# for us to do: we only add the cffi variation if the default backend # for us to do: we only add the cffi variation if the default backend
# is the C extension. # is the C extension.
if mod.zstd.backend == 'cffi': if mod.zstd.backend == "cffi":
return cls return cls
old_env = dict(os.environ) old_env = dict(os.environ)
os.environ['PYTHON_ZSTANDARD_IMPORT_POLICY'] = 'cffi' os.environ["PYTHON_ZSTANDARD_IMPORT_POLICY"] = "cffi"
try: try:
try: try:
mod_info = imp.find_module('zstandard') mod_info = imp.find_module("zstandard")
mod = imp.load_module('zstandard_cffi', *mod_info) mod = imp.load_module("zstandard_cffi", *mod_info)
except ImportError: except ImportError:
return cls return cls
finally: finally:
os.environ.clear() os.environ.clear()
os.environ.update(old_env) os.environ.update(old_env)
if mod.backend != 'cffi': if mod.backend != "cffi":
raise Exception('got the zstandard %s backend instead of cffi' % mod.backend) raise Exception("got the zstandard %s backend instead of cffi" % mod.backend)
# If CFFI version is available, dynamically construct test methods # If CFFI version is available, dynamically construct test methods
# that use it. # that use it.
for attr in dir(cls): for attr in dir(cls):
fn = getattr(cls, attr) fn = getattr(cls, attr)
if not inspect.ismethod(fn) and not inspect.isfunction(fn): if not inspect.ismethod(fn) and not inspect.isfunction(fn):
continue continue
if not fn.__name__.startswith('test_'): if not fn.__name__.startswith("test_"):
continue continue
name = '%s_cffi' % fn.__name__ name = "%s_cffi" % fn.__name__
# Replace the "zstd" symbol with the CFFI module instance. Then copy # Replace the "zstd" symbol with the CFFI module instance. Then copy
# the function object and install it in a new attribute. # the function object and install it in a new attribute.
if isinstance(fn, types.FunctionType): if isinstance(fn, types.FunctionType):
globs = dict(fn.__globals__) globs = dict(fn.__globals__)
globs['zstd'] = mod globs["zstd"] = mod
new_fn = types.FunctionType(fn.__code__, globs, name, new_fn = types.FunctionType(
fn.__defaults__, fn.__closure__) fn.__code__, globs, name, fn.__defaults__, fn.__closure__
)
new_method = new_fn new_method = new_fn
else: else:
globs = dict(fn.__func__.func_globals) globs = dict(fn.__func__.func_globals)
globs['zstd'] = mod globs["zstd"] = mod
new_fn = types.FunctionType(fn.__func__.func_code, globs, name, new_fn = types.FunctionType(
fn.__func__.func_code,
globs,
name,
fn.__func__.func_defaults, fn.__func__.func_defaults,
fn.__func__.func_closure) fn.__func__.func_closure,
new_method = types.UnboundMethodType(new_fn, fn.im_self, )
fn.im_class) new_method = types.UnboundMethodType(new_fn, fn.im_self, fn.im_class)
setattr(cls, name, new_method) setattr(cls, name, new_method)
return cls return cls
class NonClosingBytesIO(io.BytesIO): class NonClosingBytesIO(io.BytesIO):
"""BytesIO that saves the underlying buffer on close(). """BytesIO that saves the underlying buffer on close().
This allows us to access written data after close(). This allows us to access written data after close().
""" """
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
super(NonClosingBytesIO, self).__init__(*args, **kwargs) super(NonClosingBytesIO, self).__init__(*args, **kwargs)
self._saved_buffer = None self._saved_buffer = None
def close(self): def close(self):
self._saved_buffer = self.getvalue() self._saved_buffer = self.getvalue()
return super(NonClosingBytesIO, self).close() return super(NonClosingBytesIO, self).close()
""" """
if _source_files: if _source_files:
return _source_files return _source_files
for root, dirs, files in os.walk(os.path.dirname(__file__)): for root, dirs, files in os.walk(os.path.dirname(__file__)):
dirs[:] = list(sorted(dirs)) dirs[:] = list(sorted(dirs))
for f in sorted(files): for f in sorted(files):
try: try:
with open(os.path.join(root, f), 'rb') as fh: with open(os.path.join(root, f), "rb") as fh:
data = fh.read() data = fh.read()
if data: if data:
_source_files.append(data) _source_files.append(data)
except OSError: except OSError:
pass pass
# Also add some actual random data. # Also add some actual random data.
_source_files.append(os.urandom(100)) _source_files.append(os.urandom(100))
_source_files.append(os.urandom(1000)) _source_files.append(os.urandom(1000))
_source_files.append(os.urandom(10000)) _source_files.append(os.urandom(10000))
_source_files.append(os.urandom(100000)) _source_files.append(os.urandom(100000))
_source_files.append(os.urandom(1000000)) _source_files.append(os.urandom(1000000))
return _source_files return _source_files
def generate_samples(): def generate_samples():
inputs = [ inputs = [
b'foo', b"foo",
b'bar', b"bar",
b'abcdef', b"abcdef",
b'sometext', b"sometext",
b'baz', b"baz",
] ]
samples = [] samples = []
for i in range(128): for i in range(128):
samples.append(inputs[i % 5]) samples.append(inputs[i % 5])
samples.append(inputs[i % 5] * (i + 3)) samples.append(inputs[i % 5] * (i + 3))
samples.append(inputs[-(i % 5)] * (i + 2)) samples.append(inputs[-(i % 5)] * (i + 2))
return samples return samples
if hypothesis: if hypothesis:
default_settings = hypothesis.settings(deadline=10000) default_settings = hypothesis.settings(deadline=10000)
hypothesis.settings.register_profile('default', default_settings) hypothesis.settings.register_profile("default", default_settings)
ci_settings = hypothesis.settings(deadline=20000, max_examples=1000) ci_settings = hypothesis.settings(deadline=20000, max_examples=1000)
hypothesis.settings.register_profile('ci', ci_settings) hypothesis.settings.register_profile("ci", ci_settings)
expensive_settings = hypothesis.settings(deadline=None, max_examples=10000) expensive_settings = hypothesis.settings(deadline=None, max_examples=10000)
hypothesis.settings.register_profile('expensive', expensive_settings) hypothesis.settings.register_profile("expensive", expensive_settings)
hypothesis.settings.load_profile( hypothesis.settings.load_profile(os.environ.get("HYPOTHESIS_PROFILE", "default"))
os.environ.get('HYPOTHESIS_PROFILE', 'default'))

contrib/python-zstandard/tests/test_buffer_util.py

import struct import struct
import unittest import unittest
import zstandard as zstd import zstandard as zstd
ss = struct.Struct('=QQ') from .common import TestCase
ss = struct.Struct("=QQ")
class TestBufferWithSegments(unittest.TestCase):
class TestBufferWithSegments(TestCase):
def test_arguments(self): def test_arguments(self):
if not hasattr(zstd, 'BufferWithSegments'): if not hasattr(zstd, "BufferWithSegments"):
self.skipTest('BufferWithSegments not available') self.skipTest("BufferWithSegments not available")
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
zstd.BufferWithSegments() zstd.BufferWithSegments()
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
zstd.BufferWithSegments(b'foo') zstd.BufferWithSegments(b"foo")
# Segments data should be a multiple of 16. # Segments data should be a multiple of 16.
with self.assertRaisesRegexp(ValueError, 'segments array size is not a multiple of 16'): with self.assertRaisesRegex(
zstd.BufferWithSegments(b'foo', b'\x00\x00') ValueError, "segments array size is not a multiple of 16"
):
zstd.BufferWithSegments(b"foo", b"\x00\x00")
def test_invalid_offset(self): def test_invalid_offset(self):
if not hasattr(zstd, 'BufferWithSegments'): if not hasattr(zstd, "BufferWithSegments"):
self.skipTest('BufferWithSegments not available') self.skipTest("BufferWithSegments not available")
with self.assertRaisesRegexp(ValueError, 'offset within segments array references memory'): with self.assertRaisesRegex(
zstd.BufferWithSegments(b'foo', ss.pack(0, 4)) ValueError, "offset within segments array references memory"
):
zstd.BufferWithSegments(b"foo", ss.pack(0, 4))
def test_invalid_getitem(self): def test_invalid_getitem(self):
if not hasattr(zstd, 'BufferWithSegments'): if not hasattr(zstd, "BufferWithSegments"):
self.skipTest('BufferWithSegments not available') self.skipTest("BufferWithSegments not available")
b = zstd.BufferWithSegments(b'foo', ss.pack(0, 3)) b = zstd.BufferWithSegments(b"foo", ss.pack(0, 3))
with self.assertRaisesRegexp(IndexError, 'offset must be non-negative'): with self.assertRaisesRegex(IndexError, "offset must be non-negative"):
test = b[-10] test = b[-10]
with self.assertRaisesRegexp(IndexError, 'offset must be less than 1'): with self.assertRaisesRegex(IndexError, "offset must be less than 1"):
test = b[1] test = b[1]
with self.assertRaisesRegexp(IndexError, 'offset must be less than 1'): with self.assertRaisesRegex(IndexError, "offset must be less than 1"):
test = b[2] test = b[2]
def test_single(self): def test_single(self):
if not hasattr(zstd, 'BufferWithSegments'): if not hasattr(zstd, "BufferWithSegments"):
self.skipTest('BufferWithSegments not available') self.skipTest("BufferWithSegments not available")
b = zstd.BufferWithSegments(b'foo', ss.pack(0, 3)) b = zstd.BufferWithSegments(b"foo", ss.pack(0, 3))
self.assertEqual(len(b), 1) self.assertEqual(len(b), 1)
self.assertEqual(b.size, 3) self.assertEqual(b.size, 3)
self.assertEqual(b.tobytes(), b'foo') self.assertEqual(b.tobytes(), b"foo")
self.assertEqual(len(b[0]), 3) self.assertEqual(len(b[0]), 3)
self.assertEqual(b[0].offset, 0) self.assertEqual(b[0].offset, 0)
self.assertEqual(b[0].tobytes(), b'foo') self.assertEqual(b[0].tobytes(), b"foo")
def test_multiple(self): def test_multiple(self):
if not hasattr(zstd, 'BufferWithSegments'): if not hasattr(zstd, "BufferWithSegments"):
self.skipTest('BufferWithSegments not available') self.skipTest("BufferWithSegments not available")
b = zstd.BufferWithSegments(b'foofooxfooxy', b''.join([ss.pack(0, 3), b = zstd.BufferWithSegments(
ss.pack(3, 4), b"foofooxfooxy", b"".join([ss.pack(0, 3), ss.pack(3, 4), ss.pack(7, 5)])
ss.pack(7, 5)])) )
self.assertEqual(len(b), 3) self.assertEqual(len(b), 3)
self.assertEqual(b.size, 12) self.assertEqual(b.size, 12)
self.assertEqual(b.tobytes(), b'foofooxfooxy') self.assertEqual(b.tobytes(), b"foofooxfooxy")
self.assertEqual(b[0].tobytes(), b'foo') self.assertEqual(b[0].tobytes(), b"foo")
self.assertEqual(b[1].tobytes(), b'foox') self.assertEqual(b[1].tobytes(), b"foox")
self.assertEqual(b[2].tobytes(), b'fooxy') self.assertEqual(b[2].tobytes(), b"fooxy")
class TestBufferWithSegmentsCollection(unittest.TestCase): class TestBufferWithSegmentsCollection(TestCase):
def test_empty_constructor(self): def test_empty_constructor(self):
if not hasattr(zstd, 'BufferWithSegmentsCollection'): if not hasattr(zstd, "BufferWithSegmentsCollection"):
self.skipTest('BufferWithSegmentsCollection not available') self.skipTest("BufferWithSegmentsCollection not available")
with self.assertRaisesRegexp(ValueError, 'must pass at least 1 argument'): with self.assertRaisesRegex(ValueError, "must pass at least 1 argument"):
zstd.BufferWithSegmentsCollection() zstd.BufferWithSegmentsCollection()
def test_argument_validation(self): def test_argument_validation(self):
if not hasattr(zstd, 'BufferWithSegmentsCollection'): if not hasattr(zstd, "BufferWithSegmentsCollection"):
self.skipTest('BufferWithSegmentsCollection not available') self.skipTest("BufferWithSegmentsCollection not available")
with self.assertRaisesRegexp(TypeError, 'arguments must be BufferWithSegments'): with self.assertRaisesRegex(TypeError, "arguments must be BufferWithSegments"):
zstd.BufferWithSegmentsCollection(None) zstd.BufferWithSegmentsCollection(None)
with self.assertRaisesRegexp(TypeError, 'arguments must be BufferWithSegments'): with self.assertRaisesRegex(TypeError, "arguments must be BufferWithSegments"):
zstd.BufferWithSegmentsCollection(zstd.BufferWithSegments(b'foo', ss.pack(0, 3)), zstd.BufferWithSegmentsCollection(
None) zstd.BufferWithSegments(b"foo", ss.pack(0, 3)), None
)
with self.assertRaisesRegexp(ValueError, 'ZstdBufferWithSegments cannot be empty'):
zstd.BufferWithSegmentsCollection(zstd.BufferWithSegments(b'', b'')) with self.assertRaisesRegex(
ValueError, "ZstdBufferWithSegments cannot be empty"
):
zstd.BufferWithSegmentsCollection(zstd.BufferWithSegments(b"", b""))
def test_length(self): def test_length(self):
if not hasattr(zstd, 'BufferWithSegmentsCollection'): if not hasattr(zstd, "BufferWithSegmentsCollection"):
self.skipTest('BufferWithSegmentsCollection not available') self.skipTest("BufferWithSegmentsCollection not available")
b1 = zstd.BufferWithSegments(b'foo', ss.pack(0, 3)) b1 = zstd.BufferWithSegments(b"foo", ss.pack(0, 3))
b2 = zstd.BufferWithSegments(b'barbaz', b''.join([ss.pack(0, 3), b2 = zstd.BufferWithSegments(
ss.pack(3, 3)])) b"barbaz", b"".join([ss.pack(0, 3), ss.pack(3, 3)])
)
c = zstd.BufferWithSegmentsCollection(b1) c = zstd.BufferWithSegmentsCollection(b1)
self.assertEqual(len(c), 1) self.assertEqual(len(c), 1)
self.assertEqual(c.size(), 3) self.assertEqual(c.size(), 3)
c = zstd.BufferWithSegmentsCollection(b2) c = zstd.BufferWithSegmentsCollection(b2)
self.assertEqual(len(c), 2) self.assertEqual(len(c), 2)
self.assertEqual(c.size(), 6) self.assertEqual(c.size(), 6)
c = zstd.BufferWithSegmentsCollection(b1, b2) c = zstd.BufferWithSegmentsCollection(b1, b2)
self.assertEqual(len(c), 3) self.assertEqual(len(c), 3)
self.assertEqual(c.size(), 9) self.assertEqual(c.size(), 9)
def test_getitem(self): def test_getitem(self):
if not hasattr(zstd, 'BufferWithSegmentsCollection'): if not hasattr(zstd, "BufferWithSegmentsCollection"):
self.skipTest('BufferWithSegmentsCollection not available') self.skipTest("BufferWithSegmentsCollection not available")
b1 = zstd.BufferWithSegments(b'foo', ss.pack(0, 3)) b1 = zstd.BufferWithSegments(b"foo", ss.pack(0, 3))
b2 = zstd.BufferWithSegments(b'barbaz', b''.join([ss.pack(0, 3), b2 = zstd.BufferWithSegments(
ss.pack(3, 3)])) b"barbaz", b"".join([ss.pack(0, 3), ss.pack(3, 3)])
)
c = zstd.BufferWithSegmentsCollection(b1, b2) c = zstd.BufferWithSegmentsCollection(b1, b2)
with self.assertRaisesRegexp(IndexError, 'offset must be less than 3'): with self.assertRaisesRegex(IndexError, "offset must be less than 3"):
c[3] c[3]
with self.assertRaisesRegexp(IndexError, 'offset must be less than 3'): with self.assertRaisesRegex(IndexError, "offset must be less than 3"):
c[4] c[4]
self.assertEqual(c[0].tobytes(), b'foo') self.assertEqual(c[0].tobytes(), b"foo")
self.assertEqual(c[1].tobytes(), b'bar') self.assertEqual(c[1].tobytes(), b"bar")
self.assertEqual(c[2].tobytes(), b'baz') self.assertEqual(c[2].tobytes(), b"baz")

contrib/python-zstandard/tests/test_compressor.py

import hashlib import hashlib
import io import io
import os import os
import struct import struct
import sys import sys
import tarfile import tarfile
import tempfile import tempfile
import unittest import unittest
import zstandard as zstd import zstandard as zstd
from .common import ( from .common import (
make_cffi, make_cffi,
NonClosingBytesIO, NonClosingBytesIO,
OpCountingBytesIO, OpCountingBytesIO,
TestCase,
) )
if sys.version_info[0] >= 3: if sys.version_info[0] >= 3:
next = lambda it: it.__next__() next = lambda it: it.__next__()
else: else:
next = lambda it: it.next() next = lambda it: it.next()
def multithreaded_chunk_size(level, source_size=0): def multithreaded_chunk_size(level, source_size=0):
params = zstd.ZstdCompressionParameters.from_level(level, params = zstd.ZstdCompressionParameters.from_level(level, source_size=source_size)
source_size=source_size)
return 1 << (params.window_log + 2) return 1 << (params.window_log + 2)
@make_cffi @make_cffi
class TestCompressor(unittest.TestCase): class TestCompressor(TestCase):
def test_level_bounds(self): def test_level_bounds(self):
with self.assertRaises(ValueError): with self.assertRaises(ValueError):
zstd.ZstdCompressor(level=23) zstd.ZstdCompressor(level=23)
def test_memory_size(self): def test_memory_size(self):
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
self.assertGreater(cctx.memory_size(), 100) self.assertGreater(cctx.memory_size(), 100)
@make_cffi @make_cffi
class TestCompressor_compress(unittest.TestCase): class TestCompressor_compress(TestCase):
def test_compress_empty(self): def test_compress_empty(self):
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
result = cctx.compress(b'') result = cctx.compress(b"")
self.assertEqual(result, b'\x28\xb5\x2f\xfd\x00\x48\x01\x00\x00') self.assertEqual(result, b"\x28\xb5\x2f\xfd\x00\x48\x01\x00\x00")
params = zstd.get_frame_parameters(result) params = zstd.get_frame_parameters(result)
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 524288) self.assertEqual(params.window_size, 524288)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum, 0) self.assertFalse(params.has_checksum, 0)
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
result = cctx.compress(b'') result = cctx.compress(b"")
self.assertEqual(result, b'\x28\xb5\x2f\xfd\x20\x00\x01\x00\x00') self.assertEqual(result, b"\x28\xb5\x2f\xfd\x20\x00\x01\x00\x00")
params = zstd.get_frame_parameters(result) params = zstd.get_frame_parameters(result)
self.assertEqual(params.content_size, 0) self.assertEqual(params.content_size, 0)
def test_input_types(self): def test_input_types(self):
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
expected = b'\x28\xb5\x2f\xfd\x00\x00\x19\x00\x00\x66\x6f\x6f' expected = b"\x28\xb5\x2f\xfd\x00\x00\x19\x00\x00\x66\x6f\x6f"
mutable_array = bytearray(3) mutable_array = bytearray(3)
mutable_array[:] = b'foo' mutable_array[:] = b"foo"
sources = [ sources = [
memoryview(b'foo'), memoryview(b"foo"),
bytearray(b'foo'), bytearray(b"foo"),
mutable_array, mutable_array,
] ]
for source in sources: for source in sources:
self.assertEqual(cctx.compress(source), expected) self.assertEqual(cctx.compress(source), expected)
def test_compress_large(self): def test_compress_large(self):
chunks = [] chunks = []
for i in range(255): for i in range(255):
chunks.append(struct.Struct('>B').pack(i) * 16384) chunks.append(struct.Struct(">B").pack(i) * 16384)
cctx = zstd.ZstdCompressor(level=3, write_content_size=False) cctx = zstd.ZstdCompressor(level=3, write_content_size=False)
result = cctx.compress(b''.join(chunks)) result = cctx.compress(b"".join(chunks))
self.assertEqual(len(result), 999) self.assertEqual(len(result), 999)
self.assertEqual(result[0:4], b'\x28\xb5\x2f\xfd') self.assertEqual(result[0:4], b"\x28\xb5\x2f\xfd")
# This matches the test for read_to_iter() below. # This matches the test for read_to_iter() below.
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
result = cctx.compress(b'f' * zstd.COMPRESSION_RECOMMENDED_INPUT_SIZE + b'o') result = cctx.compress(b"f" * zstd.COMPRESSION_RECOMMENDED_INPUT_SIZE + b"o")
self.assertEqual(result, b'\x28\xb5\x2f\xfd\x00\x40\x54\x00\x00' self.assertEqual(
b'\x10\x66\x66\x01\x00\xfb\xff\x39\xc0' result,
b'\x02\x09\x00\x00\x6f') b"\x28\xb5\x2f\xfd\x00\x40\x54\x00\x00"
b"\x10\x66\x66\x01\x00\xfb\xff\x39\xc0"
b"\x02\x09\x00\x00\x6f",
)
def test_negative_level(self): def test_negative_level(self):
cctx = zstd.ZstdCompressor(level=-4) cctx = zstd.ZstdCompressor(level=-4)
result = cctx.compress(b'foo' * 256) result = cctx.compress(b"foo" * 256)
def test_no_magic(self): def test_no_magic(self):
params = zstd.ZstdCompressionParameters.from_level( params = zstd.ZstdCompressionParameters.from_level(1, format=zstd.FORMAT_ZSTD1)
1, format=zstd.FORMAT_ZSTD1)
cctx = zstd.ZstdCompressor(compression_params=params) cctx = zstd.ZstdCompressor(compression_params=params)
magic = cctx.compress(b'foobar') magic = cctx.compress(b"foobar")
params = zstd.ZstdCompressionParameters.from_level( params = zstd.ZstdCompressionParameters.from_level(
1, format=zstd.FORMAT_ZSTD1_MAGICLESS) 1, format=zstd.FORMAT_ZSTD1_MAGICLESS
)
cctx = zstd.ZstdCompressor(compression_params=params) cctx = zstd.ZstdCompressor(compression_params=params)
no_magic = cctx.compress(b'foobar') no_magic = cctx.compress(b"foobar")
self.assertEqual(magic[0:4], b'\x28\xb5\x2f\xfd') self.assertEqual(magic[0:4], b"\x28\xb5\x2f\xfd")
self.assertEqual(magic[4:], no_magic) self.assertEqual(magic[4:], no_magic)
def test_write_checksum(self): def test_write_checksum(self):
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
no_checksum = cctx.compress(b'foobar') no_checksum = cctx.compress(b"foobar")
cctx = zstd.ZstdCompressor(level=1, write_checksum=True) cctx = zstd.ZstdCompressor(level=1, write_checksum=True)
with_checksum = cctx.compress(b'foobar') with_checksum = cctx.compress(b"foobar")
self.assertEqual(len(with_checksum), len(no_checksum) + 4) self.assertEqual(len(with_checksum), len(no_checksum) + 4)
no_params = zstd.get_frame_parameters(no_checksum) no_params = zstd.get_frame_parameters(no_checksum)
with_params = zstd.get_frame_parameters(with_checksum) with_params = zstd.get_frame_parameters(with_checksum)
self.assertFalse(no_params.has_checksum) self.assertFalse(no_params.has_checksum)
self.assertTrue(with_params.has_checksum) self.assertTrue(with_params.has_checksum)
def test_write_content_size(self): def test_write_content_size(self):
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
with_size = cctx.compress(b'foobar' * 256) with_size = cctx.compress(b"foobar" * 256)
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
no_size = cctx.compress(b'foobar' * 256) no_size = cctx.compress(b"foobar" * 256)
self.assertEqual(len(with_size), len(no_size) + 1) self.assertEqual(len(with_size), len(no_size) + 1)
no_params = zstd.get_frame_parameters(no_size) no_params = zstd.get_frame_parameters(no_size)
with_params = zstd.get_frame_parameters(with_size) with_params = zstd.get_frame_parameters(with_size)
self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(with_params.content_size, 1536) self.assertEqual(with_params.content_size, 1536)
def test_no_dict_id(self): def test_no_dict_id(self):
samples = [] samples = []
for i in range(128): for i in range(128):
samples.append(b'foo' * 64) samples.append(b"foo" * 64)
samples.append(b'bar' * 64) samples.append(b"bar" * 64)
samples.append(b'foobar' * 64) samples.append(b"foobar" * 64)
d = zstd.train_dictionary(1024, samples) d = zstd.train_dictionary(1024, samples)
cctx = zstd.ZstdCompressor(level=1, dict_data=d) cctx = zstd.ZstdCompressor(level=1, dict_data=d)
with_dict_id = cctx.compress(b'foobarfoobar') with_dict_id = cctx.compress(b"foobarfoobar")
cctx = zstd.ZstdCompressor(level=1, dict_data=d, write_dict_id=False) cctx = zstd.ZstdCompressor(level=1, dict_data=d, write_dict_id=False)
no_dict_id = cctx.compress(b'foobarfoobar') no_dict_id = cctx.compress(b"foobarfoobar")
self.assertEqual(len(with_dict_id), len(no_dict_id) + 4) self.assertEqual(len(with_dict_id), len(no_dict_id) + 4)
no_params = zstd.get_frame_parameters(no_dict_id) no_params = zstd.get_frame_parameters(no_dict_id)
with_params = zstd.get_frame_parameters(with_dict_id) with_params = zstd.get_frame_parameters(with_dict_id)
self.assertEqual(no_params.dict_id, 0) self.assertEqual(no_params.dict_id, 0)
self.assertEqual(with_params.dict_id, 1880053135) self.assertEqual(with_params.dict_id, 1880053135)
def test_compress_dict_multiple(self): def test_compress_dict_multiple(self):
samples = [] samples = []
for i in range(128): for i in range(128):
samples.append(b'foo' * 64) samples.append(b"foo" * 64)
samples.append(b'bar' * 64) samples.append(b"bar" * 64)
samples.append(b'foobar' * 64) samples.append(b"foobar" * 64)
d = zstd.train_dictionary(8192, samples) d = zstd.train_dictionary(8192, samples)
cctx = zstd.ZstdCompressor(level=1, dict_data=d) cctx = zstd.ZstdCompressor(level=1, dict_data=d)
for i in range(32): for i in range(32):
cctx.compress(b'foo bar foobar foo bar foobar') cctx.compress(b"foo bar foobar foo bar foobar")
def test_dict_precompute(self): def test_dict_precompute(self):
samples = [] samples = []
for i in range(128): for i in range(128):
samples.append(b'foo' * 64) samples.append(b"foo" * 64)
samples.append(b'bar' * 64) samples.append(b"bar" * 64)
samples.append(b'foobar' * 64) samples.append(b"foobar" * 64)
d = zstd.train_dictionary(8192, samples) d = zstd.train_dictionary(8192, samples)
d.precompute_compress(level=1) d.precompute_compress(level=1)
cctx = zstd.ZstdCompressor(level=1, dict_data=d) cctx = zstd.ZstdCompressor(level=1, dict_data=d)
for i in range(32): for i in range(32):
cctx.compress(b'foo bar foobar foo bar foobar') cctx.compress(b"foo bar foobar foo bar foobar")
def test_multithreaded(self): def test_multithreaded(self):
chunk_size = multithreaded_chunk_size(1) chunk_size = multithreaded_chunk_size(1)
source = b''.join([b'x' * chunk_size, b'y' * chunk_size]) source = b"".join([b"x" * chunk_size, b"y" * chunk_size])
cctx = zstd.ZstdCompressor(level=1, threads=2) cctx = zstd.ZstdCompressor(level=1, threads=2)
compressed = cctx.compress(source) compressed = cctx.compress(source)
params = zstd.get_frame_parameters(compressed) params = zstd.get_frame_parameters(compressed)
self.assertEqual(params.content_size, chunk_size * 2) self.assertEqual(params.content_size, chunk_size * 2)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
self.assertEqual(dctx.decompress(compressed), source) self.assertEqual(dctx.decompress(compressed), source)
def test_multithreaded_dict(self): def test_multithreaded_dict(self):
samples = [] samples = []
for i in range(128): for i in range(128):
samples.append(b'foo' * 64) samples.append(b"foo" * 64)
samples.append(b'bar' * 64) samples.append(b"bar" * 64)
samples.append(b'foobar' * 64) samples.append(b"foobar" * 64)
d = zstd.train_dictionary(1024, samples) d = zstd.train_dictionary(1024, samples)
cctx = zstd.ZstdCompressor(dict_data=d, threads=2) cctx = zstd.ZstdCompressor(dict_data=d, threads=2)
result = cctx.compress(b'foo') result = cctx.compress(b"foo")
params = zstd.get_frame_parameters(result); params = zstd.get_frame_parameters(result)
self.assertEqual(params.content_size, 3); self.assertEqual(params.content_size, 3)
self.assertEqual(params.dict_id, d.dict_id()) self.assertEqual(params.dict_id, d.dict_id())
self.assertEqual(result, self.assertEqual(
b'\x28\xb5\x2f\xfd\x23\x8f\x55\x0f\x70\x03\x19\x00\x00' result,
b'\x66\x6f\x6f') b"\x28\xb5\x2f\xfd\x23\x8f\x55\x0f\x70\x03\x19\x00\x00" b"\x66\x6f\x6f",
)
def test_multithreaded_compression_params(self): def test_multithreaded_compression_params(self):
params = zstd.ZstdCompressionParameters.from_level(0, threads=2) params = zstd.ZstdCompressionParameters.from_level(0, threads=2)
cctx = zstd.ZstdCompressor(compression_params=params) cctx = zstd.ZstdCompressor(compression_params=params)
result = cctx.compress(b'foo') result = cctx.compress(b"foo")
params = zstd.get_frame_parameters(result); params = zstd.get_frame_parameters(result)
self.assertEqual(params.content_size, 3); self.assertEqual(params.content_size, 3)
self.assertEqual(result, self.assertEqual(result, b"\x28\xb5\x2f\xfd\x20\x03\x19\x00\x00\x66\x6f\x6f")
b'\x28\xb5\x2f\xfd\x20\x03\x19\x00\x00\x66\x6f\x6f')
@make_cffi @make_cffi
class TestCompressor_compressobj(unittest.TestCase): class TestCompressor_compressobj(TestCase):
def test_compressobj_empty(self): def test_compressobj_empty(self):
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
cobj = cctx.compressobj() cobj = cctx.compressobj()
self.assertEqual(cobj.compress(b''), b'') self.assertEqual(cobj.compress(b""), b"")
self.assertEqual(cobj.flush(), self.assertEqual(cobj.flush(), b"\x28\xb5\x2f\xfd\x00\x48\x01\x00\x00")
b'\x28\xb5\x2f\xfd\x00\x48\x01\x00\x00')
def test_input_types(self): def test_input_types(self):
expected = b'\x28\xb5\x2f\xfd\x00\x48\x19\x00\x00\x66\x6f\x6f' expected = b"\x28\xb5\x2f\xfd\x00\x48\x19\x00\x00\x66\x6f\x6f"
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
mutable_array = bytearray(3) mutable_array = bytearray(3)
mutable_array[:] = b'foo' mutable_array[:] = b"foo"
sources = [ sources = [
memoryview(b'foo'), memoryview(b"foo"),
bytearray(b'foo'), bytearray(b"foo"),
mutable_array, mutable_array,
] ]
for source in sources: for source in sources:
cobj = cctx.compressobj() cobj = cctx.compressobj()
self.assertEqual(cobj.compress(source), b'') self.assertEqual(cobj.compress(source), b"")
self.assertEqual(cobj.flush(), expected) self.assertEqual(cobj.flush(), expected)
def test_compressobj_large(self): def test_compressobj_large(self):
chunks = [] chunks = []
for i in range(255): for i in range(255):
chunks.append(struct.Struct('>B').pack(i) * 16384) chunks.append(struct.Struct(">B").pack(i) * 16384)
cctx = zstd.ZstdCompressor(level=3) cctx = zstd.ZstdCompressor(level=3)
cobj = cctx.compressobj() cobj = cctx.compressobj()
result = cobj.compress(b''.join(chunks)) + cobj.flush() result = cobj.compress(b"".join(chunks)) + cobj.flush()
self.assertEqual(len(result), 999) self.assertEqual(len(result), 999)
self.assertEqual(result[0:4], b'\x28\xb5\x2f\xfd') self.assertEqual(result[0:4], b"\x28\xb5\x2f\xfd")
params = zstd.get_frame_parameters(result) params = zstd.get_frame_parameters(result)
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 2097152) self.assertEqual(params.window_size, 2097152)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
def test_write_checksum(self): def test_write_checksum(self):
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
cobj = cctx.compressobj() cobj = cctx.compressobj()
no_checksum = cobj.compress(b'foobar') + cobj.flush() no_checksum = cobj.compress(b"foobar") + cobj.flush()
cctx = zstd.ZstdCompressor(level=1, write_checksum=True) cctx = zstd.ZstdCompressor(level=1, write_checksum=True)
cobj = cctx.compressobj() cobj = cctx.compressobj()
with_checksum = cobj.compress(b'foobar') + cobj.flush() with_checksum = cobj.compress(b"foobar") + cobj.flush()
no_params = zstd.get_frame_parameters(no_checksum) no_params = zstd.get_frame_parameters(no_checksum)
with_params = zstd.get_frame_parameters(with_checksum) with_params = zstd.get_frame_parameters(with_checksum)
self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(with_params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(with_params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(no_params.dict_id, 0) self.assertEqual(no_params.dict_id, 0)
self.assertEqual(with_params.dict_id, 0) self.assertEqual(with_params.dict_id, 0)
self.assertFalse(no_params.has_checksum) self.assertFalse(no_params.has_checksum)
self.assertTrue(with_params.has_checksum) self.assertTrue(with_params.has_checksum)
self.assertEqual(len(with_checksum), len(no_checksum) + 4) self.assertEqual(len(with_checksum), len(no_checksum) + 4)
def test_write_content_size(self): def test_write_content_size(self):
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
cobj = cctx.compressobj(size=len(b'foobar' * 256)) cobj = cctx.compressobj(size=len(b"foobar" * 256))
with_size = cobj.compress(b'foobar' * 256) + cobj.flush() with_size = cobj.compress(b"foobar" * 256) + cobj.flush()
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
cobj = cctx.compressobj(size=len(b'foobar' * 256)) cobj = cctx.compressobj(size=len(b"foobar" * 256))
no_size = cobj.compress(b'foobar' * 256) + cobj.flush() no_size = cobj.compress(b"foobar" * 256) + cobj.flush()
no_params = zstd.get_frame_parameters(no_size) no_params = zstd.get_frame_parameters(no_size)
with_params = zstd.get_frame_parameters(with_size) with_params = zstd.get_frame_parameters(with_size)
self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(with_params.content_size, 1536) self.assertEqual(with_params.content_size, 1536)
self.assertEqual(no_params.dict_id, 0) self.assertEqual(no_params.dict_id, 0)
self.assertEqual(with_params.dict_id, 0) self.assertEqual(with_params.dict_id, 0)
self.assertFalse(no_params.has_checksum) self.assertFalse(no_params.has_checksum)
self.assertFalse(with_params.has_checksum) self.assertFalse(with_params.has_checksum)
self.assertEqual(len(with_size), len(no_size) + 1) self.assertEqual(len(with_size), len(no_size) + 1)
def test_compress_after_finished(self): def test_compress_after_finished(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
cobj = cctx.compressobj() cobj = cctx.compressobj()
cobj.compress(b'foo') cobj.compress(b"foo")
cobj.flush() cobj.flush()
with self.assertRaisesRegexp(zstd.ZstdError, r'cannot call compress\(\) after compressor'): with self.assertRaisesRegex(
cobj.compress(b'foo') zstd.ZstdError, r"cannot call compress\(\) after compressor"
):
with self.assertRaisesRegexp(zstd.ZstdError, 'compressor object already finished'): cobj.compress(b"foo")
with self.assertRaisesRegex(
zstd.ZstdError, "compressor object already finished"
):
cobj.flush() cobj.flush()
def test_flush_block_repeated(self): def test_flush_block_repeated(self):
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
cobj = cctx.compressobj() cobj = cctx.compressobj()
self.assertEqual(cobj.compress(b'foo'), b'') self.assertEqual(cobj.compress(b"foo"), b"")
self.assertEqual(cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK), self.assertEqual(
b'\x28\xb5\x2f\xfd\x00\x48\x18\x00\x00foo') cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK),
self.assertEqual(cobj.compress(b'bar'), b'') b"\x28\xb5\x2f\xfd\x00\x48\x18\x00\x00foo",
)
self.assertEqual(cobj.compress(b"bar"), b"")
# 3 byte header plus content. # 3 byte header plus content.
self.assertEqual(cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK), self.assertEqual(cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK), b"\x18\x00\x00bar")
b'\x18\x00\x00bar') self.assertEqual(cobj.flush(), b"\x01\x00\x00")
self.assertEqual(cobj.flush(), b'\x01\x00\x00')
def test_flush_empty_block(self): def test_flush_empty_block(self):
cctx = zstd.ZstdCompressor(write_checksum=True) cctx = zstd.ZstdCompressor(write_checksum=True)
cobj = cctx.compressobj() cobj = cctx.compressobj()
cobj.compress(b'foobar') cobj.compress(b"foobar")
cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK) cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK)
# No-op if no block is active (this is internal to zstd). # No-op if no block is active (this is internal to zstd).
self.assertEqual(cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK), b'') self.assertEqual(cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK), b"")
trailing = cobj.flush() trailing = cobj.flush()
# 3 bytes block header + 4 bytes frame checksum # 3 bytes block header + 4 bytes frame checksum
self.assertEqual(len(trailing), 7) self.assertEqual(len(trailing), 7)
header = trailing[0:3] header = trailing[0:3]
self.assertEqual(header, b'\x01\x00\x00') self.assertEqual(header, b"\x01\x00\x00")
def test_multithreaded(self): def test_multithreaded(self):
source = io.BytesIO() source = io.BytesIO()
source.write(b'a' * 1048576) source.write(b"a" * 1048576)
source.write(b'b' * 1048576) source.write(b"b" * 1048576)
source.write(b'c' * 1048576) source.write(b"c" * 1048576)
source.seek(0) source.seek(0)
cctx = zstd.ZstdCompressor(level=1, threads=2) cctx = zstd.ZstdCompressor(level=1, threads=2)
cobj = cctx.compressobj() cobj = cctx.compressobj()
chunks = [] chunks = []
while True: while True:
d = source.read(8192) d = source.read(8192)
if not d: if not d:
break break
chunks.append(cobj.compress(d)) chunks.append(cobj.compress(d))
chunks.append(cobj.flush()) chunks.append(cobj.flush())
compressed = b''.join(chunks) compressed = b"".join(chunks)
self.assertEqual(len(compressed), 295) self.assertEqual(len(compressed), 119)
def test_frame_progression(self): def test_frame_progression(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
self.assertEqual(cctx.frame_progression(), (0, 0, 0)) self.assertEqual(cctx.frame_progression(), (0, 0, 0))
cobj = cctx.compressobj() cobj = cctx.compressobj()
cobj.compress(b'foobar') cobj.compress(b"foobar")
self.assertEqual(cctx.frame_progression(), (6, 0, 0)) self.assertEqual(cctx.frame_progression(), (6, 0, 0))
cobj.flush() cobj.flush()
self.assertEqual(cctx.frame_progression(), (6, 6, 15)) self.assertEqual(cctx.frame_progression(), (6, 6, 15))
def test_bad_size(self): def test_bad_size(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
cobj = cctx.compressobj(size=2) cobj = cctx.compressobj(size=2)
with self.assertRaisesRegexp(zstd.ZstdError, 'Src size is incorrect'): with self.assertRaisesRegex(zstd.ZstdError, "Src size is incorrect"):
cobj.compress(b'foo') cobj.compress(b"foo")
# Try another operation on this instance. # Try another operation on this instance.
with self.assertRaisesRegexp(zstd.ZstdError, 'Src size is incorrect'): with self.assertRaisesRegex(zstd.ZstdError, "Src size is incorrect"):
cobj.compress(b'aa') cobj.compress(b"aa")
# Try another operation on the compressor. # Try another operation on the compressor.
cctx.compressobj(size=4) cctx.compressobj(size=4)
cctx.compress(b'foobar') cctx.compress(b"foobar")
@make_cffi @make_cffi
class TestCompressor_copy_stream(unittest.TestCase): class TestCompressor_copy_stream(TestCase):
def test_no_read(self): def test_no_read(self):
source = object() source = object()
dest = io.BytesIO() dest = io.BytesIO()
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
with self.assertRaises(ValueError): with self.assertRaises(ValueError):
cctx.copy_stream(source, dest) cctx.copy_stream(source, dest)
source = io.BytesIO() source = io.BytesIO()
dest = io.BytesIO() dest = io.BytesIO()
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
r, w = cctx.copy_stream(source, dest) r, w = cctx.copy_stream(source, dest)
self.assertEqual(int(r), 0) self.assertEqual(int(r), 0)
self.assertEqual(w, 9) self.assertEqual(w, 9)
self.assertEqual(dest.getvalue(), self.assertEqual(dest.getvalue(), b"\x28\xb5\x2f\xfd\x00\x48\x01\x00\x00")
b'\x28\xb5\x2f\xfd\x00\x48\x01\x00\x00')
def test_large_data(self): def test_large_data(self):
source = io.BytesIO() source = io.BytesIO()
for i in range(255): for i in range(255):
source.write(struct.Struct('>B').pack(i) * 16384) source.write(struct.Struct(">B").pack(i) * 16384)
source.seek(0) source.seek(0)
dest = io.BytesIO() dest = io.BytesIO()
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
r, w = cctx.copy_stream(source, dest) r, w = cctx.copy_stream(source, dest)
self.assertEqual(r, 255 * 16384) self.assertEqual(r, 255 * 16384)
self.assertEqual(w, 999) self.assertEqual(w, 999)
params = zstd.get_frame_parameters(dest.getvalue()) params = zstd.get_frame_parameters(dest.getvalue())
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 2097152) self.assertEqual(params.window_size, 2097152)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
def test_write_checksum(self): def test_write_checksum(self):
source = io.BytesIO(b'foobar') source = io.BytesIO(b"foobar")
no_checksum = io.BytesIO() no_checksum = io.BytesIO()
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
cctx.copy_stream(source, no_checksum) cctx.copy_stream(source, no_checksum)
source.seek(0) source.seek(0)
with_checksum = io.BytesIO() with_checksum = io.BytesIO()
cctx = zstd.ZstdCompressor(level=1, write_checksum=True) cctx = zstd.ZstdCompressor(level=1, write_checksum=True)
cctx.copy_stream(source, with_checksum) cctx.copy_stream(source, with_checksum)
self.assertEqual(len(with_checksum.getvalue()), self.assertEqual(len(with_checksum.getvalue()), len(no_checksum.getvalue()) + 4)
len(no_checksum.getvalue()) + 4)
no_params = zstd.get_frame_parameters(no_checksum.getvalue()) no_params = zstd.get_frame_parameters(no_checksum.getvalue())
with_params = zstd.get_frame_parameters(with_checksum.getvalue()) with_params = zstd.get_frame_parameters(with_checksum.getvalue())
self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(with_params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(with_params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(no_params.dict_id, 0) self.assertEqual(no_params.dict_id, 0)
self.assertEqual(with_params.dict_id, 0) self.assertEqual(with_params.dict_id, 0)
self.assertFalse(no_params.has_checksum) self.assertFalse(no_params.has_checksum)
self.assertTrue(with_params.has_checksum) self.assertTrue(with_params.has_checksum)
def test_write_content_size(self): def test_write_content_size(self):
source = io.BytesIO(b'foobar' * 256) source = io.BytesIO(b"foobar" * 256)
no_size = io.BytesIO() no_size = io.BytesIO()
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
cctx.copy_stream(source, no_size) cctx.copy_stream(source, no_size)
source.seek(0) source.seek(0)
with_size = io.BytesIO() with_size = io.BytesIO()
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
cctx.copy_stream(source, with_size) cctx.copy_stream(source, with_size)
# Source content size is unknown, so no content size written. # Source content size is unknown, so no content size written.
self.assertEqual(len(with_size.getvalue()), self.assertEqual(len(with_size.getvalue()), len(no_size.getvalue()))
len(no_size.getvalue()))
source.seek(0) source.seek(0)
with_size = io.BytesIO() with_size = io.BytesIO()
cctx.copy_stream(source, with_size, size=len(source.getvalue())) cctx.copy_stream(source, with_size, size=len(source.getvalue()))
# We specified source size, so content size header is present. # We specified source size, so content size header is present.
self.assertEqual(len(with_size.getvalue()), self.assertEqual(len(with_size.getvalue()), len(no_size.getvalue()) + 1)
len(no_size.getvalue()) + 1)
no_params = zstd.get_frame_parameters(no_size.getvalue()) no_params = zstd.get_frame_parameters(no_size.getvalue())
with_params = zstd.get_frame_parameters(with_size.getvalue()) with_params = zstd.get_frame_parameters(with_size.getvalue())
self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(with_params.content_size, 1536) self.assertEqual(with_params.content_size, 1536)
self.assertEqual(no_params.dict_id, 0) self.assertEqual(no_params.dict_id, 0)
self.assertEqual(with_params.dict_id, 0) self.assertEqual(with_params.dict_id, 0)
self.assertFalse(no_params.has_checksum) self.assertFalse(no_params.has_checksum)
self.assertFalse(with_params.has_checksum) self.assertFalse(with_params.has_checksum)
def test_read_write_size(self): def test_read_write_size(self):
source = OpCountingBytesIO(b'foobarfoobar') source = OpCountingBytesIO(b"foobarfoobar")
dest = OpCountingBytesIO() dest = OpCountingBytesIO()
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
r, w = cctx.copy_stream(source, dest, read_size=1, write_size=1) r, w = cctx.copy_stream(source, dest, read_size=1, write_size=1)
self.assertEqual(r, len(source.getvalue())) self.assertEqual(r, len(source.getvalue()))
self.assertEqual(w, 21) self.assertEqual(w, 21)
self.assertEqual(source._read_count, len(source.getvalue()) + 1) self.assertEqual(source._read_count, len(source.getvalue()) + 1)
self.assertEqual(dest._write_count, len(dest.getvalue())) self.assertEqual(dest._write_count, len(dest.getvalue()))
def test_multithreaded(self): def test_multithreaded(self):
source = io.BytesIO() source = io.BytesIO()
source.write(b'a' * 1048576) source.write(b"a" * 1048576)
source.write(b'b' * 1048576) source.write(b"b" * 1048576)
source.write(b'c' * 1048576) source.write(b"c" * 1048576)
source.seek(0) source.seek(0)
dest = io.BytesIO() dest = io.BytesIO()
cctx = zstd.ZstdCompressor(threads=2, write_content_size=False) cctx = zstd.ZstdCompressor(threads=2, write_content_size=False)
r, w = cctx.copy_stream(source, dest) r, w = cctx.copy_stream(source, dest)
self.assertEqual(r, 3145728) self.assertEqual(r, 3145728)
self.assertEqual(w, 295) self.assertEqual(w, 111)
params = zstd.get_frame_parameters(dest.getvalue()) params = zstd.get_frame_parameters(dest.getvalue())
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
# Writing content size and checksum works. # Writing content size and checksum works.
cctx = zstd.ZstdCompressor(threads=2, write_checksum=True) cctx = zstd.ZstdCompressor(threads=2, write_checksum=True)
dest = io.BytesIO() dest = io.BytesIO()
source.seek(0) source.seek(0)
cctx.copy_stream(source, dest, size=len(source.getvalue())) cctx.copy_stream(source, dest, size=len(source.getvalue()))
params = zstd.get_frame_parameters(dest.getvalue()) params = zstd.get_frame_parameters(dest.getvalue())
self.assertEqual(params.content_size, 3145728) self.assertEqual(params.content_size, 3145728)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertTrue(params.has_checksum) self.assertTrue(params.has_checksum)
def test_bad_size(self): def test_bad_size(self):
source = io.BytesIO() source = io.BytesIO()
source.write(b'a' * 32768) source.write(b"a" * 32768)
source.write(b'b' * 32768) source.write(b"b" * 32768)
source.seek(0) source.seek(0)
dest = io.BytesIO() dest = io.BytesIO()
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
with self.assertRaisesRegexp(zstd.ZstdError, 'Src size is incorrect'): with self.assertRaisesRegex(zstd.ZstdError, "Src size is incorrect"):
cctx.copy_stream(source, dest, size=42) cctx.copy_stream(source, dest, size=42)
# Try another operation on this compressor. # Try another operation on this compressor.
source.seek(0) source.seek(0)
dest = io.BytesIO() dest = io.BytesIO()
cctx.copy_stream(source, dest) cctx.copy_stream(source, dest)
@make_cffi @make_cffi
class TestCompressor_stream_reader(unittest.TestCase): class TestCompressor_stream_reader(TestCase):
def test_context_manager(self): def test_context_manager(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
with cctx.stream_reader(b'foo') as reader: with cctx.stream_reader(b"foo") as reader:
with self.assertRaisesRegexp(ValueError, 'cannot __enter__ multiple times'): with self.assertRaisesRegex(ValueError, "cannot __enter__ multiple times"):
with reader as reader2: with reader as reader2:
pass pass
def test_no_context_manager(self): def test_no_context_manager(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
reader = cctx.stream_reader(b'foo') reader = cctx.stream_reader(b"foo")
reader.read(4) reader.read(4)
self.assertFalse(reader.closed) self.assertFalse(reader.closed)
reader.close() reader.close()
self.assertTrue(reader.closed) self.assertTrue(reader.closed)
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
reader.read(1) reader.read(1)
def test_not_implemented(self): def test_not_implemented(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
with cctx.stream_reader(b'foo' * 60) as reader: with cctx.stream_reader(b"foo" * 60) as reader:
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
reader.readline() reader.readline()
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
reader.readlines() reader.readlines()
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
iter(reader) iter(reader)
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
next(reader) next(reader)
with self.assertRaises(OSError): with self.assertRaises(OSError):
reader.writelines([]) reader.writelines([])
with self.assertRaises(OSError): with self.assertRaises(OSError):
reader.write(b'foo') reader.write(b"foo")
def test_constant_methods(self): def test_constant_methods(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
with cctx.stream_reader(b'boo') as reader: with cctx.stream_reader(b"boo") as reader:
self.assertTrue(reader.readable()) self.assertTrue(reader.readable())
self.assertFalse(reader.writable()) self.assertFalse(reader.writable())
self.assertFalse(reader.seekable()) self.assertFalse(reader.seekable())
self.assertFalse(reader.isatty()) self.assertFalse(reader.isatty())
self.assertFalse(reader.closed) self.assertFalse(reader.closed)
self.assertIsNone(reader.flush()) self.assertIsNone(reader.flush())
self.assertFalse(reader.closed) self.assertFalse(reader.closed)
self.assertTrue(reader.closed) self.assertTrue(reader.closed)
def test_read_closed(self): def test_read_closed(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
with cctx.stream_reader(b'foo' * 60) as reader: with cctx.stream_reader(b"foo" * 60) as reader:
reader.close() reader.close()
self.assertTrue(reader.closed) self.assertTrue(reader.closed)
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
reader.read(10) reader.read(10)
def test_read_sizes(self): def test_read_sizes(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
foo = cctx.compress(b'foo') foo = cctx.compress(b"foo")
with cctx.stream_reader(b'foo') as reader: with cctx.stream_reader(b"foo") as reader:
with self.assertRaisesRegexp(ValueError, 'cannot read negative amounts less than -1'): with self.assertRaisesRegex(
ValueError, "cannot read negative amounts less than -1"
):
reader.read(-2) reader.read(-2)
self.assertEqual(reader.read(0), b'') self.assertEqual(reader.read(0), b"")
self.assertEqual(reader.read(), foo) self.assertEqual(reader.read(), foo)
def test_read_buffer(self): def test_read_buffer(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
source = b''.join([b'foo' * 60, b'bar' * 60, b'baz' * 60]) source = b"".join([b"foo" * 60, b"bar" * 60, b"baz" * 60])
frame = cctx.compress(source) frame = cctx.compress(source)
with cctx.stream_reader(source) as reader: with cctx.stream_reader(source) as reader:
self.assertEqual(reader.tell(), 0) self.assertEqual(reader.tell(), 0)
# We should get entire frame in one read. # We should get entire frame in one read.
result = reader.read(8192) result = reader.read(8192)
self.assertEqual(result, frame) self.assertEqual(result, frame)
self.assertEqual(reader.tell(), len(result)) self.assertEqual(reader.tell(), len(result))
self.assertEqual(reader.read(), b'') self.assertEqual(reader.read(), b"")
self.assertEqual(reader.tell(), len(result)) self.assertEqual(reader.tell(), len(result))
def test_read_buffer_small_chunks(self): def test_read_buffer_small_chunks(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
source = b'foo' * 60 source = b"foo" * 60
chunks = [] chunks = []
with cctx.stream_reader(source) as reader: with cctx.stream_reader(source) as reader:
self.assertEqual(reader.tell(), 0) self.assertEqual(reader.tell(), 0)
while True: while True:
chunk = reader.read(1) chunk = reader.read(1)
if not chunk: if not chunk:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(reader.tell(), sum(map(len, chunks))) self.assertEqual(reader.tell(), sum(map(len, chunks)))
self.assertEqual(b''.join(chunks), cctx.compress(source)) self.assertEqual(b"".join(chunks), cctx.compress(source))
def test_read_stream(self): def test_read_stream(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
source = b''.join([b'foo' * 60, b'bar' * 60, b'baz' * 60]) source = b"".join([b"foo" * 60, b"bar" * 60, b"baz" * 60])
frame = cctx.compress(source) frame = cctx.compress(source)
with cctx.stream_reader(io.BytesIO(source), size=len(source)) as reader: with cctx.stream_reader(io.BytesIO(source), size=len(source)) as reader:
self.assertEqual(reader.tell(), 0) self.assertEqual(reader.tell(), 0)
chunk = reader.read(8192) chunk = reader.read(8192)
self.assertEqual(chunk, frame) self.assertEqual(chunk, frame)
self.assertEqual(reader.tell(), len(chunk)) self.assertEqual(reader.tell(), len(chunk))
self.assertEqual(reader.read(), b'') self.assertEqual(reader.read(), b"")
self.assertEqual(reader.tell(), len(chunk)) self.assertEqual(reader.tell(), len(chunk))
def test_read_stream_small_chunks(self): def test_read_stream_small_chunks(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
source = b'foo' * 60 source = b"foo" * 60
chunks = [] chunks = []
with cctx.stream_reader(io.BytesIO(source), size=len(source)) as reader: with cctx.stream_reader(io.BytesIO(source), size=len(source)) as reader:
self.assertEqual(reader.tell(), 0) self.assertEqual(reader.tell(), 0)
while True: while True:
chunk = reader.read(1) chunk = reader.read(1)
if not chunk: if not chunk:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(reader.tell(), sum(map(len, chunks))) self.assertEqual(reader.tell(), sum(map(len, chunks)))
self.assertEqual(b''.join(chunks), cctx.compress(source)) self.assertEqual(b"".join(chunks), cctx.compress(source))
def test_read_after_exit(self): def test_read_after_exit(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
with cctx.stream_reader(b'foo' * 60) as reader: with cctx.stream_reader(b"foo" * 60) as reader:
while reader.read(8192): while reader.read(8192):
pass pass
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
reader.read(10) reader.read(10)
def test_bad_size(self): def test_bad_size(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
source = io.BytesIO(b'foobar') source = io.BytesIO(b"foobar")
with cctx.stream_reader(source, size=2) as reader: with cctx.stream_reader(source, size=2) as reader:
with self.assertRaisesRegexp(zstd.ZstdError, 'Src size is incorrect'): with self.assertRaisesRegex(zstd.ZstdError, "Src size is incorrect"):
reader.read(10) reader.read(10)
# Try another compression operation. # Try another compression operation.
with cctx.stream_reader(source, size=42): with cctx.stream_reader(source, size=42):
pass pass
def test_readall(self): def test_readall(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
frame = cctx.compress(b'foo' * 1024) frame = cctx.compress(b"foo" * 1024)
reader = cctx.stream_reader(b'foo' * 1024) reader = cctx.stream_reader(b"foo" * 1024)
self.assertEqual(reader.readall(), frame) self.assertEqual(reader.readall(), frame)
def test_readinto(self): def test_readinto(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
foo = cctx.compress(b'foo') foo = cctx.compress(b"foo")
reader = cctx.stream_reader(b'foo') reader = cctx.stream_reader(b"foo")
with self.assertRaises(Exception): with self.assertRaises(Exception):
reader.readinto(b'foobar') reader.readinto(b"foobar")
# readinto() with sufficiently large destination. # readinto() with sufficiently large destination.
b = bytearray(1024) b = bytearray(1024)
reader = cctx.stream_reader(b'foo') reader = cctx.stream_reader(b"foo")
self.assertEqual(reader.readinto(b), len(foo)) self.assertEqual(reader.readinto(b), len(foo))
self.assertEqual(b[0:len(foo)], foo) self.assertEqual(b[0 : len(foo)], foo)
self.assertEqual(reader.readinto(b), 0) self.assertEqual(reader.readinto(b), 0)
self.assertEqual(b[0:len(foo)], foo) self.assertEqual(b[0 : len(foo)], foo)
# readinto() with small reads. # readinto() with small reads.
b = bytearray(1024) b = bytearray(1024)
reader = cctx.stream_reader(b'foo', read_size=1) reader = cctx.stream_reader(b"foo", read_size=1)
self.assertEqual(reader.readinto(b), len(foo)) self.assertEqual(reader.readinto(b), len(foo))
self.assertEqual(b[0:len(foo)], foo) self.assertEqual(b[0 : len(foo)], foo)
# Too small destination buffer. # Too small destination buffer.
b = bytearray(2) b = bytearray(2)
reader = cctx.stream_reader(b'foo') reader = cctx.stream_reader(b"foo")
self.assertEqual(reader.readinto(b), 2) self.assertEqual(reader.readinto(b), 2)
self.assertEqual(b[:], foo[0:2]) self.assertEqual(b[:], foo[0:2])
self.assertEqual(reader.readinto(b), 2) self.assertEqual(reader.readinto(b), 2)
self.assertEqual(b[:], foo[2:4]) self.assertEqual(b[:], foo[2:4])
self.assertEqual(reader.readinto(b), 2) self.assertEqual(reader.readinto(b), 2)
self.assertEqual(b[:], foo[4:6]) self.assertEqual(b[:], foo[4:6])
def test_readinto1(self): def test_readinto1(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
foo = b''.join(cctx.read_to_iter(io.BytesIO(b'foo'))) foo = b"".join(cctx.read_to_iter(io.BytesIO(b"foo")))
reader = cctx.stream_reader(b'foo') reader = cctx.stream_reader(b"foo")
with self.assertRaises(Exception): with self.assertRaises(Exception):
reader.readinto1(b'foobar') reader.readinto1(b"foobar")
b = bytearray(1024) b = bytearray(1024)
source = OpCountingBytesIO(b'foo') source = OpCountingBytesIO(b"foo")
reader = cctx.stream_reader(source) reader = cctx.stream_reader(source)
self.assertEqual(reader.readinto1(b), len(foo)) self.assertEqual(reader.readinto1(b), len(foo))
self.assertEqual(b[0:len(foo)], foo) self.assertEqual(b[0 : len(foo)], foo)
self.assertEqual(source._read_count, 2) self.assertEqual(source._read_count, 2)
# readinto1() with small reads. # readinto1() with small reads.
b = bytearray(1024) b = bytearray(1024)
source = OpCountingBytesIO(b'foo') source = OpCountingBytesIO(b"foo")
reader = cctx.stream_reader(source, read_size=1) reader = cctx.stream_reader(source, read_size=1)
self.assertEqual(reader.readinto1(b), len(foo)) self.assertEqual(reader.readinto1(b), len(foo))
self.assertEqual(b[0:len(foo)], foo) self.assertEqual(b[0 : len(foo)], foo)
self.assertEqual(source._read_count, 4) self.assertEqual(source._read_count, 4)
def test_read1(self): def test_read1(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
foo = b''.join(cctx.read_to_iter(io.BytesIO(b'foo'))) foo = b"".join(cctx.read_to_iter(io.BytesIO(b"foo")))
b = OpCountingBytesIO(b'foo') b = OpCountingBytesIO(b"foo")
reader = cctx.stream_reader(b) reader = cctx.stream_reader(b)
self.assertEqual(reader.read1(), foo) self.assertEqual(reader.read1(), foo)
self.assertEqual(b._read_count, 2) self.assertEqual(b._read_count, 2)
b = OpCountingBytesIO(b'foo') b = OpCountingBytesIO(b"foo")
reader = cctx.stream_reader(b) reader = cctx.stream_reader(b)
self.assertEqual(reader.read1(0), b'') self.assertEqual(reader.read1(0), b"")
self.assertEqual(reader.read1(2), foo[0:2]) self.assertEqual(reader.read1(2), foo[0:2])
self.assertEqual(b._read_count, 2) self.assertEqual(b._read_count, 2)
self.assertEqual(reader.read1(2), foo[2:4]) self.assertEqual(reader.read1(2), foo[2:4])
self.assertEqual(reader.read1(1024), foo[4:]) self.assertEqual(reader.read1(1024), foo[4:])
@make_cffi @make_cffi
class TestCompressor_stream_writer(unittest.TestCase): class TestCompressor_stream_writer(TestCase):
def test_io_api(self): def test_io_api(self):
buffer = io.BytesIO() buffer = io.BytesIO()
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
writer = cctx.stream_writer(buffer) writer = cctx.stream_writer(buffer)
self.assertFalse(writer.isatty()) self.assertFalse(writer.isatty())
self.assertFalse(writer.readable()) self.assertFalse(writer.readable())
writer.readinto(None) writer.readinto(None)
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
writer.fileno() writer.fileno()
self.assertFalse(writer.closed) self.assertFalse(writer.closed)
def test_fileno_file(self): def test_fileno_file(self):
with tempfile.TemporaryFile('wb') as tf: with tempfile.TemporaryFile("wb") as tf:
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
writer = cctx.stream_writer(tf) writer = cctx.stream_writer(tf)
self.assertEqual(writer.fileno(), tf.fileno()) self.assertEqual(writer.fileno(), tf.fileno())
def test_close(self): def test_close(self):
buffer = NonClosingBytesIO() buffer = NonClosingBytesIO()
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
writer = cctx.stream_writer(buffer) writer = cctx.stream_writer(buffer)
writer.write(b'foo' * 1024) writer.write(b"foo" * 1024)
self.assertFalse(writer.closed) self.assertFalse(writer.closed)
self.assertFalse(buffer.closed) self.assertFalse(buffer.closed)
writer.close() writer.close()
self.assertTrue(writer.closed) self.assertTrue(writer.closed)
self.assertTrue(buffer.closed) self.assertTrue(buffer.closed)
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
writer.write(b'foo') writer.write(b"foo")
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
writer.flush() writer.flush()
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
with writer: with writer:
pass pass
self.assertEqual(buffer.getvalue(), self.assertEqual(
b'\x28\xb5\x2f\xfd\x00\x48\x55\x00\x00\x18\x66\x6f' buffer.getvalue(),
b'\x6f\x01\x00\xfa\xd3\x77\x43') b"\x28\xb5\x2f\xfd\x00\x48\x55\x00\x00\x18\x66\x6f"
b"\x6f\x01\x00\xfa\xd3\x77\x43",
)
# Context manager exit should close stream. # Context manager exit should close stream.
buffer = io.BytesIO() buffer = io.BytesIO()
writer = cctx.stream_writer(buffer) writer = cctx.stream_writer(buffer)
with writer: with writer:
writer.write(b'foo') writer.write(b"foo")
self.assertTrue(writer.closed) self.assertTrue(writer.closed)
def test_empty(self): def test_empty(self):
buffer = NonClosingBytesIO() buffer = NonClosingBytesIO()
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
with cctx.stream_writer(buffer) as compressor: with cctx.stream_writer(buffer) as compressor:
compressor.write(b'') compressor.write(b"")
result = buffer.getvalue() result = buffer.getvalue()
self.assertEqual(result, b'\x28\xb5\x2f\xfd\x00\x48\x01\x00\x00') self.assertEqual(result, b"\x28\xb5\x2f\xfd\x00\x48\x01\x00\x00")
params = zstd.get_frame_parameters(result) params = zstd.get_frame_parameters(result)
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 524288) self.assertEqual(params.window_size, 524288)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
# Test without context manager. # Test without context manager.
buffer = io.BytesIO() buffer = io.BytesIO()
compressor = cctx.stream_writer(buffer) compressor = cctx.stream_writer(buffer)
self.assertEqual(compressor.write(b''), 0) self.assertEqual(compressor.write(b""), 0)
self.assertEqual(buffer.getvalue(), b'') self.assertEqual(buffer.getvalue(), b"")
self.assertEqual(compressor.flush(zstd.FLUSH_FRAME), 9) self.assertEqual(compressor.flush(zstd.FLUSH_FRAME), 9)
result = buffer.getvalue() result = buffer.getvalue()
self.assertEqual(result, b'\x28\xb5\x2f\xfd\x00\x48\x01\x00\x00') self.assertEqual(result, b"\x28\xb5\x2f\xfd\x00\x48\x01\x00\x00")
params = zstd.get_frame_parameters(result) params = zstd.get_frame_parameters(result)
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 524288) self.assertEqual(params.window_size, 524288)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
# Test write_return_read=True # Test write_return_read=True
compressor = cctx.stream_writer(buffer, write_return_read=True) compressor = cctx.stream_writer(buffer, write_return_read=True)
self.assertEqual(compressor.write(b''), 0) self.assertEqual(compressor.write(b""), 0)
def test_input_types(self): def test_input_types(self):
expected = b'\x28\xb5\x2f\xfd\x00\x48\x19\x00\x00\x66\x6f\x6f' expected = b"\x28\xb5\x2f\xfd\x00\x48\x19\x00\x00\x66\x6f\x6f"
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
mutable_array = bytearray(3) mutable_array = bytearray(3)
mutable_array[:] = b'foo' mutable_array[:] = b"foo"
sources = [ sources = [
memoryview(b'foo'), memoryview(b"foo"),
bytearray(b'foo'), bytearray(b"foo"),
mutable_array, mutable_array,
] ]
for source in sources: for source in sources:
buffer = NonClosingBytesIO() buffer = NonClosingBytesIO()
with cctx.stream_writer(buffer) as compressor: with cctx.stream_writer(buffer) as compressor:
compressor.write(source) compressor.write(source)
self.assertEqual(buffer.getvalue(), expected) self.assertEqual(buffer.getvalue(), expected)
compressor = cctx.stream_writer(buffer, write_return_read=True) compressor = cctx.stream_writer(buffer, write_return_read=True)
self.assertEqual(compressor.write(source), len(source)) self.assertEqual(compressor.write(source), len(source))
def test_multiple_compress(self): def test_multiple_compress(self):
buffer = NonClosingBytesIO() buffer = NonClosingBytesIO()
cctx = zstd.ZstdCompressor(level=5) cctx = zstd.ZstdCompressor(level=5)
with cctx.stream_writer(buffer) as compressor: with cctx.stream_writer(buffer) as compressor:
self.assertEqual(compressor.write(b'foo'), 0) self.assertEqual(compressor.write(b"foo"), 0)
self.assertEqual(compressor.write(b'bar'), 0) self.assertEqual(compressor.write(b"bar"), 0)
self.assertEqual(compressor.write(b'x' * 8192), 0) self.assertEqual(compressor.write(b"x" * 8192), 0)
result = buffer.getvalue() result = buffer.getvalue()
self.assertEqual(result, self.assertEqual(
b'\x28\xb5\x2f\xfd\x00\x58\x75\x00\x00\x38\x66\x6f' result,
b'\x6f\x62\x61\x72\x78\x01\x00\xfc\xdf\x03\x23') b"\x28\xb5\x2f\xfd\x00\x58\x75\x00\x00\x38\x66\x6f"
b"\x6f\x62\x61\x72\x78\x01\x00\xfc\xdf\x03\x23",
)
# Test without context manager. # Test without context manager.
buffer = io.BytesIO() buffer = io.BytesIO()
compressor = cctx.stream_writer(buffer) compressor = cctx.stream_writer(buffer)
self.assertEqual(compressor.write(b'foo'), 0) self.assertEqual(compressor.write(b"foo"), 0)
self.assertEqual(compressor.write(b'bar'), 0) self.assertEqual(compressor.write(b"bar"), 0)
self.assertEqual(compressor.write(b'x' * 8192), 0) self.assertEqual(compressor.write(b"x" * 8192), 0)
self.assertEqual(compressor.flush(zstd.FLUSH_FRAME), 23) self.assertEqual(compressor.flush(zstd.FLUSH_FRAME), 23)
result = buffer.getvalue() result = buffer.getvalue()
self.assertEqual(result, self.assertEqual(
b'\x28\xb5\x2f\xfd\x00\x58\x75\x00\x00\x38\x66\x6f' result,
b'\x6f\x62\x61\x72\x78\x01\x00\xfc\xdf\x03\x23') b"\x28\xb5\x2f\xfd\x00\x58\x75\x00\x00\x38\x66\x6f"
b"\x6f\x62\x61\x72\x78\x01\x00\xfc\xdf\x03\x23",
)
# Test with write_return_read=True. # Test with write_return_read=True.
compressor = cctx.stream_writer(buffer, write_return_read=True) compressor = cctx.stream_writer(buffer, write_return_read=True)
self.assertEqual(compressor.write(b'foo'), 3) self.assertEqual(compressor.write(b"foo"), 3)
self.assertEqual(compressor.write(b'barbiz'), 6) self.assertEqual(compressor.write(b"barbiz"), 6)
self.assertEqual(compressor.write(b'x' * 8192), 8192) self.assertEqual(compressor.write(b"x" * 8192), 8192)
def test_dictionary(self): def test_dictionary(self):
samples = [] samples = []
for i in range(128): for i in range(128):
samples.append(b'foo' * 64) samples.append(b"foo" * 64)
samples.append(b'bar' * 64) samples.append(b"bar" * 64)
samples.append(b'foobar' * 64) samples.append(b"foobar" * 64)
d = zstd.train_dictionary(8192, samples) d = zstd.train_dictionary(8192, samples)
h = hashlib.sha1(d.as_bytes()).hexdigest() h = hashlib.sha1(d.as_bytes()).hexdigest()
self.assertEqual(h, '7a2e59a876db958f74257141045af8f912e00d4e') self.assertEqual(h, "7a2e59a876db958f74257141045af8f912e00d4e")
buffer = NonClosingBytesIO() buffer = NonClosingBytesIO()
cctx = zstd.ZstdCompressor(level=9, dict_data=d) cctx = zstd.ZstdCompressor(level=9, dict_data=d)
with cctx.stream_writer(buffer) as compressor: with cctx.stream_writer(buffer) as compressor:
self.assertEqual(compressor.write(b'foo'), 0) self.assertEqual(compressor.write(b"foo"), 0)
self.assertEqual(compressor.write(b'bar'), 0) self.assertEqual(compressor.write(b"bar"), 0)
self.assertEqual(compressor.write(b'foo' * 16384), 0) self.assertEqual(compressor.write(b"foo" * 16384), 0)
compressed = buffer.getvalue() compressed = buffer.getvalue()
params = zstd.get_frame_parameters(compressed) params = zstd.get_frame_parameters(compressed)
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 2097152) self.assertEqual(params.window_size, 2097152)
self.assertEqual(params.dict_id, d.dict_id()) self.assertEqual(params.dict_id, d.dict_id())
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
h = hashlib.sha1(compressed).hexdigest() h = hashlib.sha1(compressed).hexdigest()
self.assertEqual(h, '0a7c05635061f58039727cdbe76388c6f4cfef06') self.assertEqual(h, "0a7c05635061f58039727cdbe76388c6f4cfef06")
source = b'foo' + b'bar' + (b'foo' * 16384) source = b"foo" + b"bar" + (b"foo" * 16384)
dctx = zstd.ZstdDecompressor(dict_data=d) dctx = zstd.ZstdDecompressor(dict_data=d)
self.assertEqual(dctx.decompress(compressed, max_output_size=len(source)), self.assertEqual(
source) dctx.decompress(compressed, max_output_size=len(source)), source
)
def test_compression_params(self): def test_compression_params(self):
params = zstd.ZstdCompressionParameters( params = zstd.ZstdCompressionParameters(
window_log=20, window_log=20,
chain_log=6, chain_log=6,
hash_log=12, hash_log=12,
min_match=5, min_match=5,
search_log=4, search_log=4,
target_length=10, target_length=10,
strategy=zstd.STRATEGY_FAST) strategy=zstd.STRATEGY_FAST,
)
buffer = NonClosingBytesIO() buffer = NonClosingBytesIO()
cctx = zstd.ZstdCompressor(compression_params=params) cctx = zstd.ZstdCompressor(compression_params=params)
with cctx.stream_writer(buffer) as compressor: with cctx.stream_writer(buffer) as compressor:
self.assertEqual(compressor.write(b'foo'), 0) self.assertEqual(compressor.write(b"foo"), 0)
self.assertEqual(compressor.write(b'bar'), 0) self.assertEqual(compressor.write(b"bar"), 0)
self.assertEqual(compressor.write(b'foobar' * 16384), 0) self.assertEqual(compressor.write(b"foobar" * 16384), 0)
compressed = buffer.getvalue() compressed = buffer.getvalue()
params = zstd.get_frame_parameters(compressed) params = zstd.get_frame_parameters(compressed)
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 1048576) self.assertEqual(params.window_size, 1048576)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
h = hashlib.sha1(compressed).hexdigest() h = hashlib.sha1(compressed).hexdigest()
self.assertEqual(h, 'dd4bb7d37c1a0235b38a2f6b462814376843ef0b') self.assertEqual(h, "dd4bb7d37c1a0235b38a2f6b462814376843ef0b")
def test_write_checksum(self): def test_write_checksum(self):
no_checksum = NonClosingBytesIO() no_checksum = NonClosingBytesIO()
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
with cctx.stream_writer(no_checksum) as compressor: with cctx.stream_writer(no_checksum) as compressor:
self.assertEqual(compressor.write(b'foobar'), 0) self.assertEqual(compressor.write(b"foobar"), 0)
with_checksum = NonClosingBytesIO() with_checksum = NonClosingBytesIO()
cctx = zstd.ZstdCompressor(level=1, write_checksum=True) cctx = zstd.ZstdCompressor(level=1, write_checksum=True)
with cctx.stream_writer(with_checksum) as compressor: with cctx.stream_writer(with_checksum) as compressor:
self.assertEqual(compressor.write(b'foobar'), 0) self.assertEqual(compressor.write(b"foobar"), 0)
no_params = zstd.get_frame_parameters(no_checksum.getvalue()) no_params = zstd.get_frame_parameters(no_checksum.getvalue())
with_params = zstd.get_frame_parameters(with_checksum.getvalue()) with_params = zstd.get_frame_parameters(with_checksum.getvalue())
self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(with_params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(with_params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(no_params.dict_id, 0) self.assertEqual(no_params.dict_id, 0)
self.assertEqual(with_params.dict_id, 0) self.assertEqual(with_params.dict_id, 0)
self.assertFalse(no_params.has_checksum) self.assertFalse(no_params.has_checksum)
self.assertTrue(with_params.has_checksum) self.assertTrue(with_params.has_checksum)
self.assertEqual(len(with_checksum.getvalue()), self.assertEqual(len(with_checksum.getvalue()), len(no_checksum.getvalue()) + 4)
len(no_checksum.getvalue()) + 4)
def test_write_content_size(self): def test_write_content_size(self):
no_size = NonClosingBytesIO() no_size = NonClosingBytesIO()
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
with cctx.stream_writer(no_size) as compressor: with cctx.stream_writer(no_size) as compressor:
self.assertEqual(compressor.write(b'foobar' * 256), 0) self.assertEqual(compressor.write(b"foobar" * 256), 0)
with_size = NonClosingBytesIO() with_size = NonClosingBytesIO()
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
with cctx.stream_writer(with_size) as compressor: with cctx.stream_writer(with_size) as compressor:
self.assertEqual(compressor.write(b'foobar' * 256), 0) self.assertEqual(compressor.write(b"foobar" * 256), 0)
# Source size is not known in streaming mode, so header not # Source size is not known in streaming mode, so header not
# written. # written.
self.assertEqual(len(with_size.getvalue()), self.assertEqual(len(with_size.getvalue()), len(no_size.getvalue()))
len(no_size.getvalue()))
# Declaring size will write the header. # Declaring size will write the header.
with_size = NonClosingBytesIO() with_size = NonClosingBytesIO()
with cctx.stream_writer(with_size, size=len(b'foobar' * 256)) as compressor: with cctx.stream_writer(with_size, size=len(b"foobar" * 256)) as compressor:
self.assertEqual(compressor.write(b'foobar' * 256), 0) self.assertEqual(compressor.write(b"foobar" * 256), 0)
no_params = zstd.get_frame_parameters(no_size.getvalue()) no_params = zstd.get_frame_parameters(no_size.getvalue())
with_params = zstd.get_frame_parameters(with_size.getvalue()) with_params = zstd.get_frame_parameters(with_size.getvalue())
self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(with_params.content_size, 1536) self.assertEqual(with_params.content_size, 1536)
self.assertEqual(no_params.dict_id, 0) self.assertEqual(no_params.dict_id, 0)
self.assertEqual(with_params.dict_id, 0) self.assertEqual(with_params.dict_id, 0)
self.assertFalse(no_params.has_checksum) self.assertFalse(no_params.has_checksum)
self.assertFalse(with_params.has_checksum) self.assertFalse(with_params.has_checksum)
self.assertEqual(len(with_size.getvalue()), self.assertEqual(len(with_size.getvalue()), len(no_size.getvalue()) + 1)
len(no_size.getvalue()) + 1)
def test_no_dict_id(self): def test_no_dict_id(self):
samples = [] samples = []
for i in range(128): for i in range(128):
samples.append(b'foo' * 64) samples.append(b"foo" * 64)
samples.append(b'bar' * 64) samples.append(b"bar" * 64)
samples.append(b'foobar' * 64) samples.append(b"foobar" * 64)
d = zstd.train_dictionary(1024, samples) d = zstd.train_dictionary(1024, samples)
with_dict_id = NonClosingBytesIO() with_dict_id = NonClosingBytesIO()
cctx = zstd.ZstdCompressor(level=1, dict_data=d) cctx = zstd.ZstdCompressor(level=1, dict_data=d)
with cctx.stream_writer(with_dict_id) as compressor: with cctx.stream_writer(with_dict_id) as compressor:
self.assertEqual(compressor.write(b'foobarfoobar'), 0) self.assertEqual(compressor.write(b"foobarfoobar"), 0)
self.assertEqual(with_dict_id.getvalue()[4:5], b'\x03') self.assertEqual(with_dict_id.getvalue()[4:5], b"\x03")
cctx = zstd.ZstdCompressor(level=1, dict_data=d, write_dict_id=False) cctx = zstd.ZstdCompressor(level=1, dict_data=d, write_dict_id=False)
no_dict_id = NonClosingBytesIO() no_dict_id = NonClosingBytesIO()
with cctx.stream_writer(no_dict_id) as compressor: with cctx.stream_writer(no_dict_id) as compressor:
self.assertEqual(compressor.write(b'foobarfoobar'), 0) self.assertEqual(compressor.write(b"foobarfoobar"), 0)
self.assertEqual(no_dict_id.getvalue()[4:5], b'\x00') self.assertEqual(no_dict_id.getvalue()[4:5], b"\x00")
no_params = zstd.get_frame_parameters(no_dict_id.getvalue()) no_params = zstd.get_frame_parameters(no_dict_id.getvalue())
with_params = zstd.get_frame_parameters(with_dict_id.getvalue()) with_params = zstd.get_frame_parameters(with_dict_id.getvalue())
self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(no_params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(with_params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(with_params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(no_params.dict_id, 0) self.assertEqual(no_params.dict_id, 0)
self.assertEqual(with_params.dict_id, d.dict_id()) self.assertEqual(with_params.dict_id, d.dict_id())
self.assertFalse(no_params.has_checksum) self.assertFalse(no_params.has_checksum)
self.assertFalse(with_params.has_checksum) self.assertFalse(with_params.has_checksum)
self.assertEqual(len(with_dict_id.getvalue()), self.assertEqual(len(with_dict_id.getvalue()), len(no_dict_id.getvalue()) + 4)
len(no_dict_id.getvalue()) + 4)
def test_memory_size(self): def test_memory_size(self):
cctx = zstd.ZstdCompressor(level=3) cctx = zstd.ZstdCompressor(level=3)
buffer = io.BytesIO() buffer = io.BytesIO()
with cctx.stream_writer(buffer) as compressor: with cctx.stream_writer(buffer) as compressor:
compressor.write(b'foo') compressor.write(b"foo")
size = compressor.memory_size() size = compressor.memory_size()
self.assertGreater(size, 100000) self.assertGreater(size, 100000)
def test_write_size(self): def test_write_size(self):
cctx = zstd.ZstdCompressor(level=3) cctx = zstd.ZstdCompressor(level=3)
dest = OpCountingBytesIO() dest = OpCountingBytesIO()
with cctx.stream_writer(dest, write_size=1) as compressor: with cctx.stream_writer(dest, write_size=1) as compressor:
self.assertEqual(compressor.write(b'foo'), 0) self.assertEqual(compressor.write(b"foo"), 0)
self.assertEqual(compressor.write(b'bar'), 0) self.assertEqual(compressor.write(b"bar"), 0)
self.assertEqual(compressor.write(b'foobar'), 0) self.assertEqual(compressor.write(b"foobar"), 0)
self.assertEqual(len(dest.getvalue()), dest._write_count) self.assertEqual(len(dest.getvalue()), dest._write_count)
def test_flush_repeated(self): def test_flush_repeated(self):
cctx = zstd.ZstdCompressor(level=3) cctx = zstd.ZstdCompressor(level=3)
dest = OpCountingBytesIO() dest = OpCountingBytesIO()
with cctx.stream_writer(dest) as compressor: with cctx.stream_writer(dest) as compressor:
self.assertEqual(compressor.write(b'foo'), 0) self.assertEqual(compressor.write(b"foo"), 0)
self.assertEqual(dest._write_count, 0) self.assertEqual(dest._write_count, 0)
self.assertEqual(compressor.flush(), 12) self.assertEqual(compressor.flush(), 12)
self.assertEqual(dest._write_count, 1) self.assertEqual(dest._write_count, 1)
self.assertEqual(compressor.write(b'bar'), 0) self.assertEqual(compressor.write(b"bar"), 0)
self.assertEqual(dest._write_count, 1) self.assertEqual(dest._write_count, 1)
self.assertEqual(compressor.flush(), 6) self.assertEqual(compressor.flush(), 6)
self.assertEqual(dest._write_count, 2) self.assertEqual(dest._write_count, 2)
self.assertEqual(compressor.write(b'baz'), 0) self.assertEqual(compressor.write(b"baz"), 0)
self.assertEqual(dest._write_count, 3) self.assertEqual(dest._write_count, 3)
def test_flush_empty_block(self): def test_flush_empty_block(self):
cctx = zstd.ZstdCompressor(level=3, write_checksum=True) cctx = zstd.ZstdCompressor(level=3, write_checksum=True)
dest = OpCountingBytesIO() dest = OpCountingBytesIO()
with cctx.stream_writer(dest) as compressor: with cctx.stream_writer(dest) as compressor:
self.assertEqual(compressor.write(b'foobar' * 8192), 0) self.assertEqual(compressor.write(b"foobar" * 8192), 0)
count = dest._write_count count = dest._write_count
offset = dest.tell() offset = dest.tell()
self.assertEqual(compressor.flush(), 23) self.assertEqual(compressor.flush(), 23)
self.assertGreater(dest._write_count, count) self.assertGreater(dest._write_count, count)
self.assertGreater(dest.tell(), offset) self.assertGreater(dest.tell(), offset)
offset = dest.tell() offset = dest.tell()
# Ending the write here should cause an empty block to be written # Ending the write here should cause an empty block to be written
# to denote end of frame. # to denote end of frame.
trailing = dest.getvalue()[offset:] trailing = dest.getvalue()[offset:]
# 3 bytes block header + 4 bytes frame checksum # 3 bytes block header + 4 bytes frame checksum
self.assertEqual(len(trailing), 7) self.assertEqual(len(trailing), 7)
header = trailing[0:3] header = trailing[0:3]
self.assertEqual(header, b'\x01\x00\x00') self.assertEqual(header, b"\x01\x00\x00")
def test_flush_frame(self): def test_flush_frame(self):
cctx = zstd.ZstdCompressor(level=3) cctx = zstd.ZstdCompressor(level=3)
dest = OpCountingBytesIO() dest = OpCountingBytesIO()
with cctx.stream_writer(dest) as compressor: with cctx.stream_writer(dest) as compressor:
self.assertEqual(compressor.write(b'foobar' * 8192), 0) self.assertEqual(compressor.write(b"foobar" * 8192), 0)
self.assertEqual(compressor.flush(zstd.FLUSH_FRAME), 23) self.assertEqual(compressor.flush(zstd.FLUSH_FRAME), 23)
compressor.write(b'biz' * 16384) compressor.write(b"biz" * 16384)
self.assertEqual(dest.getvalue(), self.assertEqual(
dest.getvalue(),
# Frame 1. # Frame 1.
b'\x28\xb5\x2f\xfd\x00\x58\x75\x00\x00\x30\x66\x6f\x6f' b"\x28\xb5\x2f\xfd\x00\x58\x75\x00\x00\x30\x66\x6f\x6f"
b'\x62\x61\x72\x01\x00\xf7\xbf\xe8\xa5\x08' b"\x62\x61\x72\x01\x00\xf7\xbf\xe8\xa5\x08"
# Frame 2. # Frame 2.
b'\x28\xb5\x2f\xfd\x00\x58\x5d\x00\x00\x18\x62\x69\x7a' b"\x28\xb5\x2f\xfd\x00\x58\x5d\x00\x00\x18\x62\x69\x7a"
b'\x01\x00\xfa\x3f\x75\x37\x04') b"\x01\x00\xfa\x3f\x75\x37\x04",
)
def test_bad_flush_mode(self): def test_bad_flush_mode(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
dest = io.BytesIO() dest = io.BytesIO()
with cctx.stream_writer(dest) as compressor: with cctx.stream_writer(dest) as compressor:
with self.assertRaisesRegexp(ValueError, 'unknown flush_mode: 42'): with self.assertRaisesRegex(ValueError, "unknown flush_mode: 42"):
compressor.flush(flush_mode=42) compressor.flush(flush_mode=42)
def test_multithreaded(self): def test_multithreaded(self):
dest = NonClosingBytesIO() dest = NonClosingBytesIO()
cctx = zstd.ZstdCompressor(threads=2) cctx = zstd.ZstdCompressor(threads=2)
with cctx.stream_writer(dest) as compressor: with cctx.stream_writer(dest) as compressor:
compressor.write(b'a' * 1048576) compressor.write(b"a" * 1048576)
compressor.write(b'b' * 1048576) compressor.write(b"b" * 1048576)
compressor.write(b'c' * 1048576) compressor.write(b"c" * 1048576)
self.assertEqual(len(dest.getvalue()), 295) self.assertEqual(len(dest.getvalue()), 111)
def test_tell(self): def test_tell(self):
dest = io.BytesIO() dest = io.BytesIO()
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
with cctx.stream_writer(dest) as compressor: with cctx.stream_writer(dest) as compressor:
self.assertEqual(compressor.tell(), 0) self.assertEqual(compressor.tell(), 0)
for i in range(256): for i in range(256):
compressor.write(b'foo' * (i + 1)) compressor.write(b"foo" * (i + 1))
self.assertEqual(compressor.tell(), dest.tell()) self.assertEqual(compressor.tell(), dest.tell())
def test_bad_size(self): def test_bad_size(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
dest = io.BytesIO() dest = io.BytesIO()
with self.assertRaisesRegexp(zstd.ZstdError, 'Src size is incorrect'): with self.assertRaisesRegex(zstd.ZstdError, "Src size is incorrect"):
with cctx.stream_writer(dest, size=2) as compressor: with cctx.stream_writer(dest, size=2) as compressor:
compressor.write(b'foo') compressor.write(b"foo")
# Test another operation. # Test another operation.
with cctx.stream_writer(dest, size=42): with cctx.stream_writer(dest, size=42):
pass pass
def test_tarfile_compat(self): def test_tarfile_compat(self):
dest = NonClosingBytesIO() dest = NonClosingBytesIO()
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
with cctx.stream_writer(dest) as compressor: with cctx.stream_writer(dest) as compressor:
with tarfile.open('tf', mode='w|', fileobj=compressor) as tf: with tarfile.open("tf", mode="w|", fileobj=compressor) as tf:
tf.add(__file__, 'test_compressor.py') tf.add(__file__, "test_compressor.py")
dest = io.BytesIO(dest.getvalue()) dest = io.BytesIO(dest.getvalue())
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(dest) as reader: with dctx.stream_reader(dest) as reader:
with tarfile.open(mode='r|', fileobj=reader) as tf: with tarfile.open(mode="r|", fileobj=reader) as tf:
for member in tf: for member in tf:
self.assertEqual(member.name, 'test_compressor.py') self.assertEqual(member.name, "test_compressor.py")
@make_cffi @make_cffi
class TestCompressor_read_to_iter(unittest.TestCase): class TestCompressor_read_to_iter(TestCase):
def test_type_validation(self): def test_type_validation(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
# Object with read() works. # Object with read() works.
for chunk in cctx.read_to_iter(io.BytesIO()): for chunk in cctx.read_to_iter(io.BytesIO()):
pass pass
# Buffer protocol works. # Buffer protocol works.
for chunk in cctx.read_to_iter(b'foobar'): for chunk in cctx.read_to_iter(b"foobar"):
pass pass
with self.assertRaisesRegexp(ValueError, 'must pass an object with a read'): with self.assertRaisesRegex(ValueError, "must pass an object with a read"):
for chunk in cctx.read_to_iter(True): for chunk in cctx.read_to_iter(True):
pass pass
def test_read_empty(self): def test_read_empty(self):
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
source = io.BytesIO() source = io.BytesIO()
it = cctx.read_to_iter(source) it = cctx.read_to_iter(source)
chunks = list(it) chunks = list(it)
self.assertEqual(len(chunks), 1) self.assertEqual(len(chunks), 1)
compressed = b''.join(chunks) compressed = b"".join(chunks)
self.assertEqual(compressed, b'\x28\xb5\x2f\xfd\x00\x48\x01\x00\x00') self.assertEqual(compressed, b"\x28\xb5\x2f\xfd\x00\x48\x01\x00\x00")
# And again with the buffer protocol. # And again with the buffer protocol.
it = cctx.read_to_iter(b'') it = cctx.read_to_iter(b"")
chunks = list(it) chunks = list(it)
self.assertEqual(len(chunks), 1) self.assertEqual(len(chunks), 1)
compressed2 = b''.join(chunks) compressed2 = b"".join(chunks)
self.assertEqual(compressed2, compressed) self.assertEqual(compressed2, compressed)
def test_read_large(self): def test_read_large(self):
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
source = io.BytesIO() source = io.BytesIO()
source.write(b'f' * zstd.COMPRESSION_RECOMMENDED_INPUT_SIZE) source.write(b"f" * zstd.COMPRESSION_RECOMMENDED_INPUT_SIZE)
source.write(b'o') source.write(b"o")
source.seek(0) source.seek(0)
# Creating an iterator should not perform any compression until # Creating an iterator should not perform any compression until
# first read. # first read.
it = cctx.read_to_iter(source, size=len(source.getvalue())) it = cctx.read_to_iter(source, size=len(source.getvalue()))
self.assertEqual(source.tell(), 0) self.assertEqual(source.tell(), 0)
# We should have exactly 2 output chunks. # We should have exactly 2 output chunks.
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
# And again for good measure. # And again for good measure.
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
# We should get the same output as the one-shot compression mechanism. # We should get the same output as the one-shot compression mechanism.
self.assertEqual(b''.join(chunks), cctx.compress(source.getvalue())) self.assertEqual(b"".join(chunks), cctx.compress(source.getvalue()))
params = zstd.get_frame_parameters(b''.join(chunks)) params = zstd.get_frame_parameters(b"".join(chunks))
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 262144) self.assertEqual(params.window_size, 262144)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
# Now check the buffer protocol. # Now check the buffer protocol.
it = cctx.read_to_iter(source.getvalue()) it = cctx.read_to_iter(source.getvalue())
chunks = list(it) chunks = list(it)
self.assertEqual(len(chunks), 2) self.assertEqual(len(chunks), 2)
params = zstd.get_frame_parameters(b''.join(chunks)) params = zstd.get_frame_parameters(b"".join(chunks))
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
#self.assertEqual(params.window_size, 262144) # self.assertEqual(params.window_size, 262144)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
self.assertEqual(b''.join(chunks), cctx.compress(source.getvalue())) self.assertEqual(b"".join(chunks), cctx.compress(source.getvalue()))
def test_read_write_size(self): def test_read_write_size(self):
source = OpCountingBytesIO(b'foobarfoobar') source = OpCountingBytesIO(b"foobarfoobar")
cctx = zstd.ZstdCompressor(level=3) cctx = zstd.ZstdCompressor(level=3)
for chunk in cctx.read_to_iter(source, read_size=1, write_size=1): for chunk in cctx.read_to_iter(source, read_size=1, write_size=1):
self.assertEqual(len(chunk), 1) self.assertEqual(len(chunk), 1)
self.assertEqual(source._read_count, len(source.getvalue()) + 1) self.assertEqual(source._read_count, len(source.getvalue()) + 1)
def test_multithreaded(self): def test_multithreaded(self):
source = io.BytesIO() source = io.BytesIO()
source.write(b'a' * 1048576) source.write(b"a" * 1048576)
source.write(b'b' * 1048576) source.write(b"b" * 1048576)
source.write(b'c' * 1048576) source.write(b"c" * 1048576)
source.seek(0) source.seek(0)
cctx = zstd.ZstdCompressor(threads=2) cctx = zstd.ZstdCompressor(threads=2)
compressed = b''.join(cctx.read_to_iter(source)) compressed = b"".join(cctx.read_to_iter(source))
self.assertEqual(len(compressed), 295) self.assertEqual(len(compressed), 111)
def test_bad_size(self): def test_bad_size(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
source = io.BytesIO(b'a' * 42) source = io.BytesIO(b"a" * 42)
with self.assertRaisesRegexp(zstd.ZstdError, 'Src size is incorrect'): with self.assertRaisesRegex(zstd.ZstdError, "Src size is incorrect"):
b''.join(cctx.read_to_iter(source, size=2)) b"".join(cctx.read_to_iter(source, size=2))
# Test another operation on errored compressor. # Test another operation on errored compressor.
b''.join(cctx.read_to_iter(source)) b"".join(cctx.read_to_iter(source))
@make_cffi @make_cffi
class TestCompressor_chunker(unittest.TestCase): class TestCompressor_chunker(TestCase):
def test_empty(self): def test_empty(self):
cctx = zstd.ZstdCompressor(write_content_size=False) cctx = zstd.ZstdCompressor(write_content_size=False)
chunker = cctx.chunker() chunker = cctx.chunker()
it = chunker.compress(b'') it = chunker.compress(b"")
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
it = chunker.finish() it = chunker.finish()
self.assertEqual(next(it), b'\x28\xb5\x2f\xfd\x00\x58\x01\x00\x00') self.assertEqual(next(it), b"\x28\xb5\x2f\xfd\x00\x58\x01\x00\x00")
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
def test_simple_input(self): def test_simple_input(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
chunker = cctx.chunker() chunker = cctx.chunker()
it = chunker.compress(b'foobar') it = chunker.compress(b"foobar")
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
it = chunker.compress(b'baz' * 30) it = chunker.compress(b"baz" * 30)
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
it = chunker.finish() it = chunker.finish()
self.assertEqual(next(it), self.assertEqual(
b'\x28\xb5\x2f\xfd\x00\x58\x7d\x00\x00\x48\x66\x6f' next(it),
b'\x6f\x62\x61\x72\x62\x61\x7a\x01\x00\xe4\xe4\x8e') b"\x28\xb5\x2f\xfd\x00\x58\x7d\x00\x00\x48\x66\x6f"
b"\x6f\x62\x61\x72\x62\x61\x7a\x01\x00\xe4\xe4\x8e",
)
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
def test_input_size(self): def test_input_size(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
chunker = cctx.chunker(size=1024) chunker = cctx.chunker(size=1024)
it = chunker.compress(b'x' * 1000) it = chunker.compress(b"x" * 1000)
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
it = chunker.compress(b'y' * 24) it = chunker.compress(b"y" * 24)
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
chunks = list(chunker.finish()) chunks = list(chunker.finish())
self.assertEqual(chunks, [ self.assertEqual(
b'\x28\xb5\x2f\xfd\x60\x00\x03\x65\x00\x00\x18\x78\x78\x79\x02\x00' chunks,
b'\xa0\x16\xe3\x2b\x80\x05' [
]) b"\x28\xb5\x2f\xfd\x60\x00\x03\x65\x00\x00\x18\x78\x78\x79\x02\x00"
b"\xa0\x16\xe3\x2b\x80\x05"
],
)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
self.assertEqual(dctx.decompress(b''.join(chunks)), self.assertEqual(dctx.decompress(b"".join(chunks)), (b"x" * 1000) + (b"y" * 24))
(b'x' * 1000) + (b'y' * 24))
def test_small_chunk_size(self): def test_small_chunk_size(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
chunker = cctx.chunker(chunk_size=1) chunker = cctx.chunker(chunk_size=1)
chunks = list(chunker.compress(b'foo' * 1024)) chunks = list(chunker.compress(b"foo" * 1024))
self.assertEqual(chunks, []) self.assertEqual(chunks, [])
chunks = list(chunker.finish()) chunks = list(chunker.finish())
self.assertTrue(all(len(chunk) == 1 for chunk in chunks)) self.assertTrue(all(len(chunk) == 1 for chunk in chunks))
self.assertEqual( self.assertEqual(
b''.join(chunks), b"".join(chunks),
b'\x28\xb5\x2f\xfd\x00\x58\x55\x00\x00\x18\x66\x6f\x6f\x01\x00' b"\x28\xb5\x2f\xfd\x00\x58\x55\x00\x00\x18\x66\x6f\x6f\x01\x00"
b'\xfa\xd3\x77\x43') b"\xfa\xd3\x77\x43",
)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
self.assertEqual(dctx.decompress(b''.join(chunks), self.assertEqual(
max_output_size=10000), dctx.decompress(b"".join(chunks), max_output_size=10000), b"foo" * 1024
b'foo' * 1024) )
def test_input_types(self): def test_input_types(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
mutable_array = bytearray(3) mutable_array = bytearray(3)
mutable_array[:] = b'foo' mutable_array[:] = b"foo"
sources = [ sources = [
memoryview(b'foo'), memoryview(b"foo"),
bytearray(b'foo'), bytearray(b"foo"),
mutable_array, mutable_array,
] ]
for source in sources: for source in sources:
chunker = cctx.chunker() chunker = cctx.chunker()
self.assertEqual(list(chunker.compress(source)), []) self.assertEqual(list(chunker.compress(source)), [])
self.assertEqual(list(chunker.finish()), [ self.assertEqual(
b'\x28\xb5\x2f\xfd\x00\x58\x19\x00\x00\x66\x6f\x6f' list(chunker.finish()),
]) [b"\x28\xb5\x2f\xfd\x00\x58\x19\x00\x00\x66\x6f\x6f"],
)
def test_flush(self): def test_flush(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
chunker = cctx.chunker() chunker = cctx.chunker()
self.assertEqual(list(chunker.compress(b'foo' * 1024)), []) self.assertEqual(list(chunker.compress(b"foo" * 1024)), [])
self.assertEqual(list(chunker.compress(b'bar' * 1024)), []) self.assertEqual(list(chunker.compress(b"bar" * 1024)), [])
chunks1 = list(chunker.flush()) chunks1 = list(chunker.flush())
self.assertEqual(chunks1, [ self.assertEqual(
b'\x28\xb5\x2f\xfd\x00\x58\x8c\x00\x00\x30\x66\x6f\x6f\x62\x61\x72' chunks1,
b'\x02\x00\xfa\x03\xfe\xd0\x9f\xbe\x1b\x02' [
]) b"\x28\xb5\x2f\xfd\x00\x58\x8c\x00\x00\x30\x66\x6f\x6f\x62\x61\x72"
b"\x02\x00\xfa\x03\xfe\xd0\x9f\xbe\x1b\x02"
],
)
self.assertEqual(list(chunker.flush()), []) self.assertEqual(list(chunker.flush()), [])
self.assertEqual(list(chunker.flush()), []) self.assertEqual(list(chunker.flush()), [])
self.assertEqual(list(chunker.compress(b'baz' * 1024)), []) self.assertEqual(list(chunker.compress(b"baz" * 1024)), [])
chunks2 = list(chunker.flush()) chunks2 = list(chunker.flush())
self.assertEqual(len(chunks2), 1) self.assertEqual(len(chunks2), 1)
chunks3 = list(chunker.finish()) chunks3 = list(chunker.finish())
self.assertEqual(len(chunks2), 1) self.assertEqual(len(chunks2), 1)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
self.assertEqual(dctx.decompress(b''.join(chunks1 + chunks2 + chunks3), self.assertEqual(
max_output_size=10000), dctx.decompress(
(b'foo' * 1024) + (b'bar' * 1024) + (b'baz' * 1024)) b"".join(chunks1 + chunks2 + chunks3), max_output_size=10000
),
(b"foo" * 1024) + (b"bar" * 1024) + (b"baz" * 1024),
)
def test_compress_after_finish(self): def test_compress_after_finish(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
chunker = cctx.chunker() chunker = cctx.chunker()
list(chunker.compress(b'foo')) list(chunker.compress(b"foo"))
list(chunker.finish()) list(chunker.finish())
with self.assertRaisesRegexp( with self.assertRaisesRegex(
zstd.ZstdError, zstd.ZstdError, r"cannot call compress\(\) after compression finished"
r'cannot call compress\(\) after compression finished'): ):
list(chunker.compress(b'foo')) list(chunker.compress(b"foo"))
def test_flush_after_finish(self): def test_flush_after_finish(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
chunker = cctx.chunker() chunker = cctx.chunker()
list(chunker.compress(b'foo')) list(chunker.compress(b"foo"))
list(chunker.finish()) list(chunker.finish())
with self.assertRaisesRegexp( with self.assertRaisesRegex(
zstd.ZstdError, zstd.ZstdError, r"cannot call flush\(\) after compression finished"
r'cannot call flush\(\) after compression finished'): ):
list(chunker.flush()) list(chunker.flush())
def test_finish_after_finish(self): def test_finish_after_finish(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
chunker = cctx.chunker() chunker = cctx.chunker()
list(chunker.compress(b'foo')) list(chunker.compress(b"foo"))
list(chunker.finish()) list(chunker.finish())
with self.assertRaisesRegexp( with self.assertRaisesRegex(
zstd.ZstdError, zstd.ZstdError, r"cannot call finish\(\) after compression finished"
r'cannot call finish\(\) after compression finished'): ):
list(chunker.finish()) list(chunker.finish())
class TestCompressor_multi_compress_to_buffer(unittest.TestCase): class TestCompressor_multi_compress_to_buffer(TestCase):
def test_invalid_inputs(self): def test_invalid_inputs(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
if not hasattr(cctx, 'multi_compress_to_buffer'): if not hasattr(cctx, "multi_compress_to_buffer"):
self.skipTest('multi_compress_to_buffer not available') self.skipTest("multi_compress_to_buffer not available")
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
cctx.multi_compress_to_buffer(True) cctx.multi_compress_to_buffer(True)
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
cctx.multi_compress_to_buffer((1, 2)) cctx.multi_compress_to_buffer((1, 2))
with self.assertRaisesRegexp(TypeError, 'item 0 not a bytes like object'): with self.assertRaisesRegex(TypeError, "item 0 not a bytes like object"):
cctx.multi_compress_to_buffer([u'foo']) cctx.multi_compress_to_buffer([u"foo"])
def test_empty_input(self): def test_empty_input(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
if not hasattr(cctx, 'multi_compress_to_buffer'): if not hasattr(cctx, "multi_compress_to_buffer"):
self.skipTest('multi_compress_to_buffer not available') self.skipTest("multi_compress_to_buffer not available")
with self.assertRaisesRegexp(ValueError, 'no source elements found'): with self.assertRaisesRegex(ValueError, "no source elements found"):
cctx.multi_compress_to_buffer([]) cctx.multi_compress_to_buffer([])
with self.assertRaisesRegexp(ValueError, 'source elements are empty'): with self.assertRaisesRegex(ValueError, "source elements are empty"):
cctx.multi_compress_to_buffer([b'', b'', b'']) cctx.multi_compress_to_buffer([b"", b"", b""])
def test_list_input(self): def test_list_input(self):
cctx = zstd.ZstdCompressor(write_checksum=True) cctx = zstd.ZstdCompressor(write_checksum=True)
if not hasattr(cctx, 'multi_compress_to_buffer'): if not hasattr(cctx, "multi_compress_to_buffer"):
self.skipTest('multi_compress_to_buffer not available') self.skipTest("multi_compress_to_buffer not available")
original = [b'foo' * 12, b'bar' * 6] original = [b"foo" * 12, b"bar" * 6]
frames = [cctx.compress(c) for c in original] frames = [cctx.compress(c) for c in original]
b = cctx.multi_compress_to_buffer(original) b = cctx.multi_compress_to_buffer(original)
self.assertIsInstance(b, zstd.BufferWithSegmentsCollection) self.assertIsInstance(b, zstd.BufferWithSegmentsCollection)
self.assertEqual(len(b), 2) self.assertEqual(len(b), 2)
self.assertEqual(b.size(), 44) self.assertEqual(b.size(), 44)
self.assertEqual(b[0].tobytes(), frames[0]) self.assertEqual(b[0].tobytes(), frames[0])
self.assertEqual(b[1].tobytes(), frames[1]) self.assertEqual(b[1].tobytes(), frames[1])
def test_buffer_with_segments_input(self): def test_buffer_with_segments_input(self):
cctx = zstd.ZstdCompressor(write_checksum=True) cctx = zstd.ZstdCompressor(write_checksum=True)
if not hasattr(cctx, 'multi_compress_to_buffer'): if not hasattr(cctx, "multi_compress_to_buffer"):
self.skipTest('multi_compress_to_buffer not available') self.skipTest("multi_compress_to_buffer not available")
original = [b'foo' * 4, b'bar' * 6] original = [b"foo" * 4, b"bar" * 6]
frames = [cctx.compress(c) for c in original] frames = [cctx.compress(c) for c in original]
offsets = struct.pack('=QQQQ', 0, len(original[0]), offsets = struct.pack(
len(original[0]), len(original[1])) "=QQQQ", 0, len(original[0]), len(original[0]), len(original[1])
segments = zstd.BufferWithSegments(b''.join(original), offsets) )
segments = zstd.BufferWithSegments(b"".join(original), offsets)
result = cctx.multi_compress_to_buffer(segments) result = cctx.multi_compress_to_buffer(segments)
self.assertEqual(len(result), 2) self.assertEqual(len(result), 2)
self.assertEqual(result.size(), 47) self.assertEqual(result.size(), 47)
self.assertEqual(result[0].tobytes(), frames[0]) self.assertEqual(result[0].tobytes(), frames[0])
self.assertEqual(result[1].tobytes(), frames[1]) self.assertEqual(result[1].tobytes(), frames[1])
def test_buffer_with_segments_collection_input(self): def test_buffer_with_segments_collection_input(self):
cctx = zstd.ZstdCompressor(write_checksum=True) cctx = zstd.ZstdCompressor(write_checksum=True)
if not hasattr(cctx, 'multi_compress_to_buffer'): if not hasattr(cctx, "multi_compress_to_buffer"):
self.skipTest('multi_compress_to_buffer not available') self.skipTest("multi_compress_to_buffer not available")
original = [ original = [
b'foo1', b"foo1",
b'foo2' * 2, b"foo2" * 2,
b'foo3' * 3, b"foo3" * 3,
b'foo4' * 4, b"foo4" * 4,
b'foo5' * 5, b"foo5" * 5,
] ]
frames = [cctx.compress(c) for c in original] frames = [cctx.compress(c) for c in original]
b = b''.join([original[0], original[1]]) b = b"".join([original[0], original[1]])
b1 = zstd.BufferWithSegments(b, struct.pack('=QQQQ', b1 = zstd.BufferWithSegments(
0, len(original[0]), b,
len(original[0]), len(original[1]))) struct.pack(
b = b''.join([original[2], original[3], original[4]]) "=QQQQ", 0, len(original[0]), len(original[0]), len(original[1])
b2 = zstd.BufferWithSegments(b, struct.pack('=QQQQQQ', ),
0, len(original[2]), )
len(original[2]), len(original[3]), b = b"".join([original[2], original[3], original[4]])
len(original[2]) + len(original[3]), len(original[4]))) b2 = zstd.BufferWithSegments(
b,
struct.pack(
"=QQQQQQ",
0,
len(original[2]),
len(original[2]),
len(original[3]),
len(original[2]) + len(original[3]),
len(original[4]),
),
)
c = zstd.BufferWithSegmentsCollection(b1, b2) c = zstd.BufferWithSegmentsCollection(b1, b2)
result = cctx.multi_compress_to_buffer(c) result = cctx.multi_compress_to_buffer(c)
self.assertEqual(len(result), len(frames)) self.assertEqual(len(result), len(frames))
for i, frame in enumerate(frames): for i, frame in enumerate(frames):
self.assertEqual(result[i].tobytes(), frame) self.assertEqual(result[i].tobytes(), frame)
def test_multiple_threads(self): def test_multiple_threads(self):
# threads argument will cause multi-threaded ZSTD APIs to be used, which will # threads argument will cause multi-threaded ZSTD APIs to be used, which will
# make output different. # make output different.
refcctx = zstd.ZstdCompressor(write_checksum=True) refcctx = zstd.ZstdCompressor(write_checksum=True)
reference = [refcctx.compress(b'x' * 64), refcctx.compress(b'y' * 64)] reference = [refcctx.compress(b"x" * 64), refcctx.compress(b"y" * 64)]
cctx = zstd.ZstdCompressor(write_checksum=True) cctx = zstd.ZstdCompressor(write_checksum=True)
if not hasattr(cctx, 'multi_compress_to_buffer'): if not hasattr(cctx, "multi_compress_to_buffer"):
self.skipTest('multi_compress_to_buffer not available') self.skipTest("multi_compress_to_buffer not available")
frames = [] frames = []
frames.extend(b'x' * 64 for i in range(256)) frames.extend(b"x" * 64 for i in range(256))
frames.extend(b'y' * 64 for i in range(256)) frames.extend(b"y" * 64 for i in range(256))
result = cctx.multi_compress_to_buffer(frames, threads=-1) result = cctx.multi_compress_to_buffer(frames, threads=-1)
self.assertEqual(len(result), 512) self.assertEqual(len(result), 512)
for i in range(512): for i in range(512):
if i < 256: if i < 256:
self.assertEqual(result[i].tobytes(), reference[0]) self.assertEqual(result[i].tobytes(), reference[0])
else: else:
self.assertEqual(result[i].tobytes(), reference[1]) self.assertEqual(result[i].tobytes(), reference[1])

contrib/python-zstandard/tests/test_compressor_fuzzing.py

import io import io
import os import os
import unittest import unittest
try: try:
import hypothesis import hypothesis
import hypothesis.strategies as strategies import hypothesis.strategies as strategies
except ImportError: except ImportError:
raise unittest.SkipTest('hypothesis not available') raise unittest.SkipTest("hypothesis not available")
import zstandard as zstd import zstandard as zstd
from . common import ( from .common import (
make_cffi, make_cffi,
NonClosingBytesIO, NonClosingBytesIO,
random_input_data, random_input_data,
TestCase,
) )
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
@make_cffi @make_cffi
class TestCompressor_stream_reader_fuzzing(unittest.TestCase): class TestCompressor_stream_reader_fuzzing(TestCase):
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[hypothesis.HealthCheck.large_base_example]
@hypothesis.given(original=strategies.sampled_from(random_input_data()), )
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_size=strategies.integers(-1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE)) read_size=strategies.integers(-1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE),
def test_stream_source_read(self, original, level, source_read_size, )
read_size): def test_stream_source_read(self, original, level, source_read_size, read_size):
if read_size == 0: if read_size == 0:
read_size = -1 read_size = -1
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(io.BytesIO(original), size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: io.BytesIO(original), size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
chunk = reader.read(read_size) chunk = reader.read(read_size)
if not chunk: if not chunk:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[hypothesis.HealthCheck.large_base_example]
@hypothesis.given(original=strategies.sampled_from(random_input_data()), )
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_size=strategies.integers(-1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE)) read_size=strategies.integers(-1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE),
def test_buffer_source_read(self, original, level, source_read_size, )
read_size): def test_buffer_source_read(self, original, level, source_read_size, read_size):
if read_size == 0: if read_size == 0:
read_size = -1 read_size = -1
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(original, size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: original, size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
chunk = reader.read(read_size) chunk = reader.read(read_size)
if not chunk: if not chunk:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_stream_source_read_variance(self, original, level, source_read_size, )
read_sizes): def test_stream_source_read_variance(
self, original, level, source_read_size, read_sizes
):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(io.BytesIO(original), size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: io.BytesIO(original), size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
read_size = read_sizes.draw(strategies.integers(-1, 16384)) read_size = read_sizes.draw(strategies.integers(-1, 16384))
chunk = reader.read(read_size) chunk = reader.read(read_size)
if not chunk and read_size: if not chunk and read_size:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_buffer_source_read_variance(self, original, level, source_read_size, )
read_sizes): def test_buffer_source_read_variance(
self, original, level, source_read_size, read_sizes
):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(original, size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: original, size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
read_size = read_sizes.draw(strategies.integers(-1, 16384)) read_size = read_sizes.draw(strategies.integers(-1, 16384))
chunk = reader.read(read_size) chunk = reader.read(read_size)
if not chunk and read_size: if not chunk and read_size:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[hypothesis.HealthCheck.large_base_example]
@hypothesis.given(original=strategies.sampled_from(random_input_data()), )
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_size=strategies.integers(1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE)) read_size=strategies.integers(1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE),
def test_stream_source_readinto(self, original, level, )
source_read_size, read_size): def test_stream_source_readinto(self, original, level, source_read_size, read_size):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(io.BytesIO(original), size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: io.BytesIO(original), size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
b = bytearray(read_size) b = bytearray(read_size)
count = reader.readinto(b) count = reader.readinto(b)
if not count: if not count:
break break
chunks.append(bytes(b[0:count])) chunks.append(bytes(b[0:count]))
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[hypothesis.HealthCheck.large_base_example]
@hypothesis.given(original=strategies.sampled_from(random_input_data()), )
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_size=strategies.integers(1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE)) read_size=strategies.integers(1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE),
def test_buffer_source_readinto(self, original, level, )
source_read_size, read_size): def test_buffer_source_readinto(self, original, level, source_read_size, read_size):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(original, size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: original, size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
b = bytearray(read_size) b = bytearray(read_size)
count = reader.readinto(b) count = reader.readinto(b)
if not count: if not count:
break break
chunks.append(bytes(b[0:count])) chunks.append(bytes(b[0:count]))
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_stream_source_readinto_variance(self, original, level, )
source_read_size, read_sizes): def test_stream_source_readinto_variance(
self, original, level, source_read_size, read_sizes
):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(io.BytesIO(original), size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: io.BytesIO(original), size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
read_size = read_sizes.draw(strategies.integers(1, 16384)) read_size = read_sizes.draw(strategies.integers(1, 16384))
b = bytearray(read_size) b = bytearray(read_size)
count = reader.readinto(b) count = reader.readinto(b)
if not count: if not count:
break break
chunks.append(bytes(b[0:count])) chunks.append(bytes(b[0:count]))
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_buffer_source_readinto_variance(self, original, level, )
source_read_size, read_sizes): def test_buffer_source_readinto_variance(
self, original, level, source_read_size, read_sizes
):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(original, size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: original, size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
read_size = read_sizes.draw(strategies.integers(1, 16384)) read_size = read_sizes.draw(strategies.integers(1, 16384))
b = bytearray(read_size) b = bytearray(read_size)
count = reader.readinto(b) count = reader.readinto(b)
if not count: if not count:
break break
chunks.append(bytes(b[0:count])) chunks.append(bytes(b[0:count]))
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[hypothesis.HealthCheck.large_base_example]
@hypothesis.given(original=strategies.sampled_from(random_input_data()), )
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_size=strategies.integers(-1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE)) read_size=strategies.integers(-1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE),
def test_stream_source_read1(self, original, level, source_read_size, )
read_size): def test_stream_source_read1(self, original, level, source_read_size, read_size):
if read_size == 0: if read_size == 0:
read_size = -1 read_size = -1
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(io.BytesIO(original), size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: io.BytesIO(original), size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
chunk = reader.read1(read_size) chunk = reader.read1(read_size)
if not chunk: if not chunk:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[hypothesis.HealthCheck.large_base_example]
@hypothesis.given(original=strategies.sampled_from(random_input_data()), )
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_size=strategies.integers(-1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE)) read_size=strategies.integers(-1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE),
def test_buffer_source_read1(self, original, level, source_read_size, )
read_size): def test_buffer_source_read1(self, original, level, source_read_size, read_size):
if read_size == 0: if read_size == 0:
read_size = -1 read_size = -1
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(original, size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: original, size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
chunk = reader.read1(read_size) chunk = reader.read1(read_size)
if not chunk: if not chunk:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_stream_source_read1_variance(self, original, level, source_read_size, )
read_sizes): def test_stream_source_read1_variance(
self, original, level, source_read_size, read_sizes
):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(io.BytesIO(original), size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: io.BytesIO(original), size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
read_size = read_sizes.draw(strategies.integers(-1, 16384)) read_size = read_sizes.draw(strategies.integers(-1, 16384))
chunk = reader.read1(read_size) chunk = reader.read1(read_size)
if not chunk and read_size: if not chunk and read_size:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_buffer_source_read1_variance(self, original, level, source_read_size, )
read_sizes): def test_buffer_source_read1_variance(
self, original, level, source_read_size, read_sizes
):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(original, size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: original, size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
read_size = read_sizes.draw(strategies.integers(-1, 16384)) read_size = read_sizes.draw(strategies.integers(-1, 16384))
chunk = reader.read1(read_size) chunk = reader.read1(read_size)
if not chunk and read_size: if not chunk and read_size:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[hypothesis.HealthCheck.large_base_example]
@hypothesis.given(original=strategies.sampled_from(random_input_data()), )
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_size=strategies.integers(1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE)) read_size=strategies.integers(1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE),
def test_stream_source_readinto1(self, original, level, source_read_size, )
read_size): def test_stream_source_readinto1(
self, original, level, source_read_size, read_size
):
if read_size == 0: if read_size == 0:
read_size = -1 read_size = -1
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(io.BytesIO(original), size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: io.BytesIO(original), size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
b = bytearray(read_size) b = bytearray(read_size)
count = reader.readinto1(b) count = reader.readinto1(b)
if not count: if not count:
break break
chunks.append(bytes(b[0:count])) chunks.append(bytes(b[0:count]))
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[hypothesis.HealthCheck.large_base_example]
@hypothesis.given(original=strategies.sampled_from(random_input_data()), )
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_size=strategies.integers(1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE)) read_size=strategies.integers(1, zstd.COMPRESSION_RECOMMENDED_OUTPUT_SIZE),
def test_buffer_source_readinto1(self, original, level, source_read_size, )
read_size): def test_buffer_source_readinto1(
self, original, level, source_read_size, read_size
):
if read_size == 0: if read_size == 0:
read_size = -1 read_size = -1
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(original, size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: original, size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
b = bytearray(read_size) b = bytearray(read_size)
count = reader.readinto1(b) count = reader.readinto1(b)
if not count: if not count:
break break
chunks.append(bytes(b[0:count])) chunks.append(bytes(b[0:count]))
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_stream_source_readinto1_variance(self, original, level, source_read_size, )
read_sizes): def test_stream_source_readinto1_variance(
self, original, level, source_read_size, read_sizes
):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(io.BytesIO(original), size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: io.BytesIO(original), size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
read_size = read_sizes.draw(strategies.integers(1, 16384)) read_size = read_sizes.draw(strategies.integers(1, 16384))
b = bytearray(read_size) b = bytearray(read_size)
count = reader.readinto1(b) count = reader.readinto1(b)
if not count: if not count:
break break
chunks.append(bytes(b[0:count])) chunks.append(bytes(b[0:count]))
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 16384), source_read_size=strategies.integers(1, 16384),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_buffer_source_readinto1_variance(self, original, level, source_read_size, )
read_sizes): def test_buffer_source_readinto1_variance(
self, original, level, source_read_size, read_sizes
):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
with cctx.stream_reader(original, size=len(original), with cctx.stream_reader(
read_size=source_read_size) as reader: original, size=len(original), read_size=source_read_size
) as reader:
chunks = [] chunks = []
while True: while True:
read_size = read_sizes.draw(strategies.integers(1, 16384)) read_size = read_sizes.draw(strategies.integers(1, 16384))
b = bytearray(read_size) b = bytearray(read_size)
count = reader.readinto1(b) count = reader.readinto1(b)
if not count: if not count:
break break
chunks.append(bytes(b[0:count])) chunks.append(bytes(b[0:count]))
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
@make_cffi @make_cffi
class TestCompressor_stream_writer_fuzzing(unittest.TestCase): class TestCompressor_stream_writer_fuzzing(TestCase):
@hypothesis.given(original=strategies.sampled_from(random_input_data()), @hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
write_size=strategies.integers(min_value=1, max_value=1048576)) write_size=strategies.integers(min_value=1, max_value=1048576),
)
def test_write_size_variance(self, original, level, write_size): def test_write_size_variance(self, original, level, write_size):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
b = NonClosingBytesIO() b = NonClosingBytesIO()
with cctx.stream_writer(b, size=len(original), write_size=write_size) as compressor: with cctx.stream_writer(
b, size=len(original), write_size=write_size
) as compressor:
compressor.write(original) compressor.write(original)
self.assertEqual(b.getvalue(), ref_frame) self.assertEqual(b.getvalue(), ref_frame)
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
@make_cffi @make_cffi
class TestCompressor_copy_stream_fuzzing(unittest.TestCase): class TestCompressor_copy_stream_fuzzing(TestCase):
@hypothesis.given(original=strategies.sampled_from(random_input_data()), @hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
read_size=strategies.integers(min_value=1, max_value=1048576), read_size=strategies.integers(min_value=1, max_value=1048576),
write_size=strategies.integers(min_value=1, max_value=1048576)) write_size=strategies.integers(min_value=1, max_value=1048576),
)
def test_read_write_size_variance(self, original, level, read_size, write_size): def test_read_write_size_variance(self, original, level, read_size, write_size):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
source = io.BytesIO(original) source = io.BytesIO(original)
dest = io.BytesIO() dest = io.BytesIO()
cctx.copy_stream(source, dest, size=len(original), read_size=read_size, cctx.copy_stream(
write_size=write_size) source, dest, size=len(original), read_size=read_size, write_size=write_size
)
self.assertEqual(dest.getvalue(), ref_frame) self.assertEqual(dest.getvalue(), ref_frame)
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
@make_cffi @make_cffi
class TestCompressor_compressobj_fuzzing(unittest.TestCase): class TestCompressor_compressobj_fuzzing(TestCase):
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
chunk_sizes=strategies.data()) chunk_sizes=strategies.data(),
)
def test_random_input_sizes(self, original, level, chunk_sizes): def test_random_input_sizes(self, original, level, chunk_sizes):
refctx = zstd.ZstdCompressor(level=level) refctx = zstd.ZstdCompressor(level=level)
ref_frame = refctx.compress(original) ref_frame = refctx.compress(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
cobj = cctx.compressobj(size=len(original)) cobj = cctx.compressobj(size=len(original))
chunks = [] chunks = []
i = 0 i = 0
while True: while True:
chunk_size = chunk_sizes.draw(strategies.integers(1, 4096)) chunk_size = chunk_sizes.draw(strategies.integers(1, 4096))
source = original[i:i + chunk_size] source = original[i : i + chunk_size]
if not source: if not source:
break break
chunks.append(cobj.compress(source)) chunks.append(cobj.compress(source))
i += chunk_size i += chunk_size
chunks.append(cobj.flush()) chunks.append(cobj.flush())
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
chunk_sizes=strategies.data(), chunk_sizes=strategies.data(),
flushes=strategies.data()) flushes=strategies.data(),
)
def test_flush_block(self, original, level, chunk_sizes, flushes): def test_flush_block(self, original, level, chunk_sizes, flushes):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
cobj = cctx.compressobj() cobj = cctx.compressobj()
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
dobj = dctx.decompressobj() dobj = dctx.decompressobj()
compressed_chunks = [] compressed_chunks = []
decompressed_chunks = [] decompressed_chunks = []
i = 0 i = 0
while True: while True:
input_size = chunk_sizes.draw(strategies.integers(1, 4096)) input_size = chunk_sizes.draw(strategies.integers(1, 4096))
source = original[i:i + input_size] source = original[i : i + input_size]
if not source: if not source:
break break
i += input_size i += input_size
chunk = cobj.compress(source) chunk = cobj.compress(source)
compressed_chunks.append(chunk) compressed_chunks.append(chunk)
decompressed_chunks.append(dobj.decompress(chunk)) decompressed_chunks.append(dobj.decompress(chunk))
if not flushes.draw(strategies.booleans()): if not flushes.draw(strategies.booleans()):
continue continue
chunk = cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK) chunk = cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK)
compressed_chunks.append(chunk) compressed_chunks.append(chunk)
decompressed_chunks.append(dobj.decompress(chunk)) decompressed_chunks.append(dobj.decompress(chunk))
self.assertEqual(b''.join(decompressed_chunks), original[0:i]) self.assertEqual(b"".join(decompressed_chunks), original[0:i])
chunk = cobj.flush(zstd.COMPRESSOBJ_FLUSH_FINISH) chunk = cobj.flush(zstd.COMPRESSOBJ_FLUSH_FINISH)
compressed_chunks.append(chunk) compressed_chunks.append(chunk)
decompressed_chunks.append(dobj.decompress(chunk)) decompressed_chunks.append(dobj.decompress(chunk))
self.assertEqual(dctx.decompress(b''.join(compressed_chunks), self.assertEqual(
max_output_size=len(original)), dctx.decompress(b"".join(compressed_chunks), max_output_size=len(original)),
original) original,
self.assertEqual(b''.join(decompressed_chunks), original) )
self.assertEqual(b"".join(decompressed_chunks), original)
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
@make_cffi @make_cffi
class TestCompressor_read_to_iter_fuzzing(unittest.TestCase): class TestCompressor_read_to_iter_fuzzing(TestCase):
@hypothesis.given(original=strategies.sampled_from(random_input_data()), @hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
read_size=strategies.integers(min_value=1, max_value=4096), read_size=strategies.integers(min_value=1, max_value=4096),
write_size=strategies.integers(min_value=1, max_value=4096)) write_size=strategies.integers(min_value=1, max_value=4096),
)
def test_read_write_size_variance(self, original, level, read_size, write_size): def test_read_write_size_variance(self, original, level, read_size, write_size):
refcctx = zstd.ZstdCompressor(level=level) refcctx = zstd.ZstdCompressor(level=level)
ref_frame = refcctx.compress(original) ref_frame = refcctx.compress(original)
source = io.BytesIO(original) source = io.BytesIO(original)
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
chunks = list(cctx.read_to_iter(source, size=len(original), chunks = list(
read_size=read_size, cctx.read_to_iter(
write_size=write_size)) source, size=len(original), read_size=read_size, write_size=write_size
)
)
self.assertEqual(b''.join(chunks), ref_frame) self.assertEqual(b"".join(chunks), ref_frame)
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
class TestCompressor_multi_compress_to_buffer_fuzzing(unittest.TestCase): class TestCompressor_multi_compress_to_buffer_fuzzing(TestCase):
@hypothesis.given(original=strategies.lists(strategies.sampled_from(random_input_data()), @hypothesis.given(
min_size=1, max_size=1024), original=strategies.lists(
strategies.sampled_from(random_input_data()), min_size=1, max_size=1024
),
threads=strategies.integers(min_value=1, max_value=8), threads=strategies.integers(min_value=1, max_value=8),
use_dict=strategies.booleans()) use_dict=strategies.booleans(),
)
def test_data_equivalence(self, original, threads, use_dict): def test_data_equivalence(self, original, threads, use_dict):
kwargs = {} kwargs = {}
# Use a content dictionary because it is cheap to create. # Use a content dictionary because it is cheap to create.
if use_dict: if use_dict:
kwargs['dict_data'] = zstd.ZstdCompressionDict(original[0]) kwargs["dict_data"] = zstd.ZstdCompressionDict(original[0])
cctx = zstd.ZstdCompressor(level=1, cctx = zstd.ZstdCompressor(level=1, write_checksum=True, **kwargs)
write_checksum=True,
**kwargs)
if not hasattr(cctx, 'multi_compress_to_buffer'): if not hasattr(cctx, "multi_compress_to_buffer"):
self.skipTest('multi_compress_to_buffer not available') self.skipTest("multi_compress_to_buffer not available")
result = cctx.multi_compress_to_buffer(original, threads=-1) result = cctx.multi_compress_to_buffer(original, threads=-1)
self.assertEqual(len(result), len(original)) self.assertEqual(len(result), len(original))
# The frame produced via the batch APIs may not be bit identical to that # The frame produced via the batch APIs may not be bit identical to that
# produced by compress() because compression parameters are adjusted # produced by compress() because compression parameters are adjusted
# from the first input in batch mode. So the only thing we can do is # from the first input in batch mode. So the only thing we can do is
# verify the decompressed data matches the input. # verify the decompressed data matches the input.
dctx = zstd.ZstdDecompressor(**kwargs) dctx = zstd.ZstdDecompressor(**kwargs)
for i, frame in enumerate(result): for i, frame in enumerate(result):
self.assertEqual(dctx.decompress(frame), original[i]) self.assertEqual(dctx.decompress(frame), original[i])
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
@make_cffi @make_cffi
class TestCompressor_chunker_fuzzing(unittest.TestCase): class TestCompressor_chunker_fuzzing(TestCase):
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
chunk_size=strategies.integers( chunk_size=strategies.integers(min_value=1, max_value=32 * 1048576),
min_value=1, input_sizes=strategies.data(),
max_value=32 * 1048576), )
input_sizes=strategies.data())
def test_random_input_sizes(self, original, level, chunk_size, input_sizes): def test_random_input_sizes(self, original, level, chunk_size, input_sizes):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
chunker = cctx.chunker(chunk_size=chunk_size) chunker = cctx.chunker(chunk_size=chunk_size)
chunks = [] chunks = []
i = 0 i = 0
while True: while True:
input_size = input_sizes.draw(strategies.integers(1, 4096)) input_size = input_sizes.draw(strategies.integers(1, 4096))
source = original[i:i + input_size] source = original[i : i + input_size]
if not source: if not source:
break break
chunks.extend(chunker.compress(source)) chunks.extend(chunker.compress(source))
i += input_size i += input_size
chunks.extend(chunker.finish()) chunks.extend(chunker.finish())
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
self.assertEqual(dctx.decompress(b''.join(chunks), self.assertEqual(
max_output_size=len(original)), dctx.decompress(b"".join(chunks), max_output_size=len(original)), original
original) )
self.assertTrue(all(len(chunk) == chunk_size for chunk in chunks[:-1])) self.assertTrue(all(len(chunk) == chunk_size for chunk in chunks[:-1]))
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
chunk_size=strategies.integers( chunk_size=strategies.integers(min_value=1, max_value=32 * 1048576),
min_value=1,
max_value=32 * 1048576),
input_sizes=strategies.data(), input_sizes=strategies.data(),
flushes=strategies.data()) flushes=strategies.data(),
def test_flush_block(self, original, level, chunk_size, input_sizes, )
flushes): def test_flush_block(self, original, level, chunk_size, input_sizes, flushes):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
chunker = cctx.chunker(chunk_size=chunk_size) chunker = cctx.chunker(chunk_size=chunk_size)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
dobj = dctx.decompressobj() dobj = dctx.decompressobj()
compressed_chunks = [] compressed_chunks = []
decompressed_chunks = [] decompressed_chunks = []
i = 0 i = 0
while True: while True:
input_size = input_sizes.draw(strategies.integers(1, 4096)) input_size = input_sizes.draw(strategies.integers(1, 4096))
source = original[i:i + input_size] source = original[i : i + input_size]
if not source: if not source:
break break
i += input_size i += input_size
chunks = list(chunker.compress(source)) chunks = list(chunker.compress(source))
compressed_chunks.extend(chunks) compressed_chunks.extend(chunks)
decompressed_chunks.append(dobj.decompress(b''.join(chunks))) decompressed_chunks.append(dobj.decompress(b"".join(chunks)))
if not flushes.draw(strategies.booleans()): if not flushes.draw(strategies.booleans()):
continue continue
chunks = list(chunker.flush()) chunks = list(chunker.flush())
compressed_chunks.extend(chunks) compressed_chunks.extend(chunks)
decompressed_chunks.append(dobj.decompress(b''.join(chunks))) decompressed_chunks.append(dobj.decompress(b"".join(chunks)))
self.assertEqual(b''.join(decompressed_chunks), original[0:i]) self.assertEqual(b"".join(decompressed_chunks), original[0:i])
chunks = list(chunker.finish()) chunks = list(chunker.finish())
compressed_chunks.extend(chunks) compressed_chunks.extend(chunks)
decompressed_chunks.append(dobj.decompress(b''.join(chunks))) decompressed_chunks.append(dobj.decompress(b"".join(chunks)))
self.assertEqual(dctx.decompress(b''.join(compressed_chunks), self.assertEqual(
max_output_size=len(original)), dctx.decompress(b"".join(compressed_chunks), max_output_size=len(original)),
original) original,
self.assertEqual(b''.join(decompressed_chunks), original) )
No newline at end of file self.assertEqual(b"".join(decompressed_chunks), original)

contrib/python-zstandard/tests/test_data_structures.py

import sys import sys
import unittest import unittest
import zstandard as zstd import zstandard as zstd
from . common import ( from .common import (
make_cffi, make_cffi,
TestCase,
) )
@make_cffi @make_cffi
class TestCompressionParameters(unittest.TestCase): class TestCompressionParameters(TestCase):
def test_bounds(self): def test_bounds(self):
zstd.ZstdCompressionParameters(window_log=zstd.WINDOWLOG_MIN, zstd.ZstdCompressionParameters(
window_log=zstd.WINDOWLOG_MIN,
chain_log=zstd.CHAINLOG_MIN, chain_log=zstd.CHAINLOG_MIN,
hash_log=zstd.HASHLOG_MIN, hash_log=zstd.HASHLOG_MIN,
search_log=zstd.SEARCHLOG_MIN, search_log=zstd.SEARCHLOG_MIN,
min_match=zstd.MINMATCH_MIN + 1, min_match=zstd.MINMATCH_MIN + 1,
target_length=zstd.TARGETLENGTH_MIN, target_length=zstd.TARGETLENGTH_MIN,
strategy=zstd.STRATEGY_FAST) strategy=zstd.STRATEGY_FAST,
)
zstd.ZstdCompressionParameters(window_log=zstd.WINDOWLOG_MAX, zstd.ZstdCompressionParameters(
window_log=zstd.WINDOWLOG_MAX,
chain_log=zstd.CHAINLOG_MAX, chain_log=zstd.CHAINLOG_MAX,
hash_log=zstd.HASHLOG_MAX, hash_log=zstd.HASHLOG_MAX,
search_log=zstd.SEARCHLOG_MAX, search_log=zstd.SEARCHLOG_MAX,
min_match=zstd.MINMATCH_MAX - 1, min_match=zstd.MINMATCH_MAX - 1,
target_length=zstd.TARGETLENGTH_MAX, target_length=zstd.TARGETLENGTH_MAX,
strategy=zstd.STRATEGY_BTULTRA2) strategy=zstd.STRATEGY_BTULTRA2,
)
def test_from_level(self): def test_from_level(self):
p = zstd.ZstdCompressionParameters.from_level(1) p = zstd.ZstdCompressionParameters.from_level(1)
self.assertIsInstance(p, zstd.CompressionParameters) self.assertIsInstance(p, zstd.CompressionParameters)
self.assertEqual(p.window_log, 19) self.assertEqual(p.window_log, 19)
p = zstd.ZstdCompressionParameters.from_level(-4) p = zstd.ZstdCompressionParameters.from_level(-4)
self.assertEqual(p.window_log, 19) self.assertEqual(p.window_log, 19)
def test_members(self): def test_members(self):
p = zstd.ZstdCompressionParameters(window_log=10, p = zstd.ZstdCompressionParameters(
window_log=10,
chain_log=6, chain_log=6,
hash_log=7, hash_log=7,
search_log=4, search_log=4,
min_match=5, min_match=5,
target_length=8, target_length=8,
strategy=1) strategy=1,
)
self.assertEqual(p.window_log, 10) self.assertEqual(p.window_log, 10)
self.assertEqual(p.chain_log, 6) self.assertEqual(p.chain_log, 6)
self.assertEqual(p.hash_log, 7) self.assertEqual(p.hash_log, 7)
self.assertEqual(p.search_log, 4) self.assertEqual(p.search_log, 4)
self.assertEqual(p.min_match, 5) self.assertEqual(p.min_match, 5)
self.assertEqual(p.target_length, 8) self.assertEqual(p.target_length, 8)
self.assertEqual(p.compression_strategy, 1) self.assertEqual(p.compression_strategy, 1)
p = zstd.ZstdCompressionParameters(compression_level=2) p = zstd.ZstdCompressionParameters(compression_level=2)
self.assertEqual(p.compression_level, 2) self.assertEqual(p.compression_level, 2)
p = zstd.ZstdCompressionParameters(threads=4) p = zstd.ZstdCompressionParameters(threads=4)
self.assertEqual(p.threads, 4) self.assertEqual(p.threads, 4)
p = zstd.ZstdCompressionParameters(threads=2, job_size=1048576, p = zstd.ZstdCompressionParameters(threads=2, job_size=1048576, overlap_log=6)
overlap_log=6)
self.assertEqual(p.threads, 2) self.assertEqual(p.threads, 2)
self.assertEqual(p.job_size, 1048576) self.assertEqual(p.job_size, 1048576)
self.assertEqual(p.overlap_log, 6) self.assertEqual(p.overlap_log, 6)
self.assertEqual(p.overlap_size_log, 6) self.assertEqual(p.overlap_size_log, 6)
p = zstd.ZstdCompressionParameters(compression_level=-1) p = zstd.ZstdCompressionParameters(compression_level=-1)
self.assertEqual(p.compression_level, -1) self.assertEqual(p.compression_level, -1)
p = zstd.ZstdCompressionParameters(ldm_bucket_size_log=7) p = zstd.ZstdCompressionParameters(ldm_bucket_size_log=7)
self.assertEqual(p.ldm_bucket_size_log, 7) self.assertEqual(p.ldm_bucket_size_log, 7)
p = zstd.ZstdCompressionParameters(ldm_hash_rate_log=8) p = zstd.ZstdCompressionParameters(ldm_hash_rate_log=8)
self.assertEqual(p.ldm_hash_every_log, 8) self.assertEqual(p.ldm_hash_every_log, 8)
self.assertEqual(p.ldm_hash_rate_log, 8) self.assertEqual(p.ldm_hash_rate_log, 8)
def test_estimated_compression_context_size(self): def test_estimated_compression_context_size(self):
p = zstd.ZstdCompressionParameters(window_log=20, p = zstd.ZstdCompressionParameters(
window_log=20,
chain_log=16, chain_log=16,
hash_log=17, hash_log=17,
search_log=1, search_log=1,
min_match=5, min_match=5,
target_length=16, target_length=16,
strategy=zstd.STRATEGY_DFAST) strategy=zstd.STRATEGY_DFAST,
)
# 32-bit has slightly different values from 64-bit. # 32-bit has slightly different values from 64-bit.
self.assertAlmostEqual(p.estimated_compression_context_size(), 1294144, self.assertAlmostEqual(
delta=250) p.estimated_compression_context_size(), 1294464, delta=400
)
def test_strategy(self): def test_strategy(self):
with self.assertRaisesRegexp(ValueError, 'cannot specify both compression_strategy'): with self.assertRaisesRegex(
ValueError, "cannot specify both compression_strategy"
):
zstd.ZstdCompressionParameters(strategy=0, compression_strategy=0) zstd.ZstdCompressionParameters(strategy=0, compression_strategy=0)
p = zstd.ZstdCompressionParameters(strategy=2) p = zstd.ZstdCompressionParameters(strategy=2)
self.assertEqual(p.compression_strategy, 2) self.assertEqual(p.compression_strategy, 2)
p = zstd.ZstdCompressionParameters(strategy=3) p = zstd.ZstdCompressionParameters(strategy=3)
self.assertEqual(p.compression_strategy, 3) self.assertEqual(p.compression_strategy, 3)
def test_ldm_hash_rate_log(self): def test_ldm_hash_rate_log(self):
with self.assertRaisesRegexp(ValueError, 'cannot specify both ldm_hash_rate_log'): with self.assertRaisesRegex(
ValueError, "cannot specify both ldm_hash_rate_log"
):
zstd.ZstdCompressionParameters(ldm_hash_rate_log=8, ldm_hash_every_log=4) zstd.ZstdCompressionParameters(ldm_hash_rate_log=8, ldm_hash_every_log=4)
p = zstd.ZstdCompressionParameters(ldm_hash_rate_log=8) p = zstd.ZstdCompressionParameters(ldm_hash_rate_log=8)
self.assertEqual(p.ldm_hash_every_log, 8) self.assertEqual(p.ldm_hash_every_log, 8)
p = zstd.ZstdCompressionParameters(ldm_hash_every_log=16) p = zstd.ZstdCompressionParameters(ldm_hash_every_log=16)
self.assertEqual(p.ldm_hash_every_log, 16) self.assertEqual(p.ldm_hash_every_log, 16)
def test_overlap_log(self): def test_overlap_log(self):
with self.assertRaisesRegexp(ValueError, 'cannot specify both overlap_log'): with self.assertRaisesRegex(ValueError, "cannot specify both overlap_log"):
zstd.ZstdCompressionParameters(overlap_log=1, overlap_size_log=9) zstd.ZstdCompressionParameters(overlap_log=1, overlap_size_log=9)
p = zstd.ZstdCompressionParameters(overlap_log=2) p = zstd.ZstdCompressionParameters(overlap_log=2)
self.assertEqual(p.overlap_log, 2) self.assertEqual(p.overlap_log, 2)
self.assertEqual(p.overlap_size_log, 2) self.assertEqual(p.overlap_size_log, 2)
p = zstd.ZstdCompressionParameters(overlap_size_log=4) p = zstd.ZstdCompressionParameters(overlap_size_log=4)
self.assertEqual(p.overlap_log, 4) self.assertEqual(p.overlap_log, 4)
self.assertEqual(p.overlap_size_log, 4) self.assertEqual(p.overlap_size_log, 4)
@make_cffi @make_cffi
class TestFrameParameters(unittest.TestCase): class TestFrameParameters(TestCase):
def test_invalid_type(self): def test_invalid_type(self):
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
zstd.get_frame_parameters(None) zstd.get_frame_parameters(None)
# Python 3 doesn't appear to convert unicode to Py_buffer. # Python 3 doesn't appear to convert unicode to Py_buffer.
if sys.version_info[0] >= 3: if sys.version_info[0] >= 3:
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
zstd.get_frame_parameters(u'foobarbaz') zstd.get_frame_parameters(u"foobarbaz")
else: else:
# CPython will convert unicode to Py_buffer. But CFFI won't. # CPython will convert unicode to Py_buffer. But CFFI won't.
if zstd.backend == 'cffi': if zstd.backend == "cffi":
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
zstd.get_frame_parameters(u'foobarbaz') zstd.get_frame_parameters(u"foobarbaz")
else: else:
with self.assertRaises(zstd.ZstdError): with self.assertRaises(zstd.ZstdError):
zstd.get_frame_parameters(u'foobarbaz') zstd.get_frame_parameters(u"foobarbaz")
def test_invalid_input_sizes(self): def test_invalid_input_sizes(self):
with self.assertRaisesRegexp(zstd.ZstdError, 'not enough data for frame'): with self.assertRaisesRegex(zstd.ZstdError, "not enough data for frame"):
zstd.get_frame_parameters(b'') zstd.get_frame_parameters(b"")
with self.assertRaisesRegexp(zstd.ZstdError, 'not enough data for frame'): with self.assertRaisesRegex(zstd.ZstdError, "not enough data for frame"):
zstd.get_frame_parameters(zstd.FRAME_HEADER) zstd.get_frame_parameters(zstd.FRAME_HEADER)
def test_invalid_frame(self): def test_invalid_frame(self):
with self.assertRaisesRegexp(zstd.ZstdError, 'Unknown frame descriptor'): with self.assertRaisesRegex(zstd.ZstdError, "Unknown frame descriptor"):
zstd.get_frame_parameters(b'foobarbaz') zstd.get_frame_parameters(b"foobarbaz")
def test_attributes(self): def test_attributes(self):
params = zstd.get_frame_parameters(zstd.FRAME_HEADER + b'\x00\x00') params = zstd.get_frame_parameters(zstd.FRAME_HEADER + b"\x00\x00")
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 1024) self.assertEqual(params.window_size, 1024)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
# Lowest 2 bits indicate a dictionary and length. Here, the dict id is 1 byte. # Lowest 2 bits indicate a dictionary and length. Here, the dict id is 1 byte.
params = zstd.get_frame_parameters(zstd.FRAME_HEADER + b'\x01\x00\xff') params = zstd.get_frame_parameters(zstd.FRAME_HEADER + b"\x01\x00\xff")
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 1024) self.assertEqual(params.window_size, 1024)
self.assertEqual(params.dict_id, 255) self.assertEqual(params.dict_id, 255)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
# Lowest 3rd bit indicates if checksum is present. # Lowest 3rd bit indicates if checksum is present.
params = zstd.get_frame_parameters(zstd.FRAME_HEADER + b'\x04\x00') params = zstd.get_frame_parameters(zstd.FRAME_HEADER + b"\x04\x00")
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 1024) self.assertEqual(params.window_size, 1024)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertTrue(params.has_checksum) self.assertTrue(params.has_checksum)
# Upper 2 bits indicate content size. # Upper 2 bits indicate content size.
params = zstd.get_frame_parameters(zstd.FRAME_HEADER + b'\x40\x00\xff\x00') params = zstd.get_frame_parameters(zstd.FRAME_HEADER + b"\x40\x00\xff\x00")
self.assertEqual(params.content_size, 511) self.assertEqual(params.content_size, 511)
self.assertEqual(params.window_size, 1024) self.assertEqual(params.window_size, 1024)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
# Window descriptor is 2nd byte after frame header. # Window descriptor is 2nd byte after frame header.
params = zstd.get_frame_parameters(zstd.FRAME_HEADER + b'\x00\x40') params = zstd.get_frame_parameters(zstd.FRAME_HEADER + b"\x00\x40")
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 262144) self.assertEqual(params.window_size, 262144)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)
# Set multiple things. # Set multiple things.
params = zstd.get_frame_parameters(zstd.FRAME_HEADER + b'\x45\x40\x0f\x10\x00') params = zstd.get_frame_parameters(zstd.FRAME_HEADER + b"\x45\x40\x0f\x10\x00")
self.assertEqual(params.content_size, 272) self.assertEqual(params.content_size, 272)
self.assertEqual(params.window_size, 262144) self.assertEqual(params.window_size, 262144)
self.assertEqual(params.dict_id, 15) self.assertEqual(params.dict_id, 15)
self.assertTrue(params.has_checksum) self.assertTrue(params.has_checksum)
def test_input_types(self): def test_input_types(self):
v = zstd.FRAME_HEADER + b'\x00\x00' v = zstd.FRAME_HEADER + b"\x00\x00"
mutable_array = bytearray(len(v)) mutable_array = bytearray(len(v))
mutable_array[:] = v mutable_array[:] = v
sources = [ sources = [
memoryview(v), memoryview(v),
bytearray(v), bytearray(v),
mutable_array, mutable_array,
] ]
for source in sources: for source in sources:
params = zstd.get_frame_parameters(source) params = zstd.get_frame_parameters(source)
self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN) self.assertEqual(params.content_size, zstd.CONTENTSIZE_UNKNOWN)
self.assertEqual(params.window_size, 1024) self.assertEqual(params.window_size, 1024)
self.assertEqual(params.dict_id, 0) self.assertEqual(params.dict_id, 0)
self.assertFalse(params.has_checksum) self.assertFalse(params.has_checksum)

contrib/python-zstandard/tests/test_data_structures_fuzzing.py

import io import io
import os import os
import sys import sys
import unittest import unittest
try: try:
import hypothesis import hypothesis
import hypothesis.strategies as strategies import hypothesis.strategies as strategies
except ImportError: except ImportError:
raise unittest.SkipTest('hypothesis not available') raise unittest.SkipTest("hypothesis not available")
import zstandard as zstd import zstandard as zstd
from .common import ( from .common import (
make_cffi, make_cffi,
TestCase,
) )
s_windowlog = strategies.integers(min_value=zstd.WINDOWLOG_MIN, s_windowlog = strategies.integers(
max_value=zstd.WINDOWLOG_MAX) min_value=zstd.WINDOWLOG_MIN, max_value=zstd.WINDOWLOG_MAX
s_chainlog = strategies.integers(min_value=zstd.CHAINLOG_MIN, )
max_value=zstd.CHAINLOG_MAX) s_chainlog = strategies.integers(
s_hashlog = strategies.integers(min_value=zstd.HASHLOG_MIN, min_value=zstd.CHAINLOG_MIN, max_value=zstd.CHAINLOG_MAX
max_value=zstd.HASHLOG_MAX) )
s_searchlog = strategies.integers(min_value=zstd.SEARCHLOG_MIN, s_hashlog = strategies.integers(min_value=zstd.HASHLOG_MIN, max_value=zstd.HASHLOG_MAX)
max_value=zstd.SEARCHLOG_MAX) s_searchlog = strategies.integers(
s_minmatch = strategies.integers(min_value=zstd.MINMATCH_MIN, min_value=zstd.SEARCHLOG_MIN, max_value=zstd.SEARCHLOG_MAX
max_value=zstd.MINMATCH_MAX) )
s_targetlength = strategies.integers(min_value=zstd.TARGETLENGTH_MIN, s_minmatch = strategies.integers(
max_value=zstd.TARGETLENGTH_MAX) min_value=zstd.MINMATCH_MIN, max_value=zstd.MINMATCH_MAX
s_strategy = strategies.sampled_from((zstd.STRATEGY_FAST, )
s_targetlength = strategies.integers(
min_value=zstd.TARGETLENGTH_MIN, max_value=zstd.TARGETLENGTH_MAX
)
s_strategy = strategies.sampled_from(
(
zstd.STRATEGY_FAST,
zstd.STRATEGY_DFAST, zstd.STRATEGY_DFAST,
zstd.STRATEGY_GREEDY, zstd.STRATEGY_GREEDY,
zstd.STRATEGY_LAZY, zstd.STRATEGY_LAZY,
zstd.STRATEGY_LAZY2, zstd.STRATEGY_LAZY2,
zstd.STRATEGY_BTLAZY2, zstd.STRATEGY_BTLAZY2,
zstd.STRATEGY_BTOPT, zstd.STRATEGY_BTOPT,
zstd.STRATEGY_BTULTRA, zstd.STRATEGY_BTULTRA,
zstd.STRATEGY_BTULTRA2)) zstd.STRATEGY_BTULTRA2,
)
)
@make_cffi @make_cffi
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
class TestCompressionParametersHypothesis(unittest.TestCase): class TestCompressionParametersHypothesis(TestCase):
@hypothesis.given(s_windowlog, s_chainlog, s_hashlog, s_searchlog, @hypothesis.given(
s_minmatch, s_targetlength, s_strategy) s_windowlog,
def test_valid_init(self, windowlog, chainlog, hashlog, searchlog, s_chainlog,
minmatch, targetlength, strategy): s_hashlog,
zstd.ZstdCompressionParameters(window_log=windowlog, s_searchlog,
s_minmatch,
s_targetlength,
s_strategy,
)
def test_valid_init(
self, windowlog, chainlog, hashlog, searchlog, minmatch, targetlength, strategy
):
zstd.ZstdCompressionParameters(
window_log=windowlog,
chain_log=chainlog, chain_log=chainlog,
hash_log=hashlog, hash_log=hashlog,
search_log=searchlog, search_log=searchlog,
min_match=minmatch, min_match=minmatch,
target_length=targetlength, target_length=targetlength,
strategy=strategy) strategy=strategy,
)
@hypothesis.given(s_windowlog, s_chainlog, s_hashlog, s_searchlog, @hypothesis.given(
s_minmatch, s_targetlength, s_strategy) s_windowlog,
def test_estimated_compression_context_size(self, windowlog, chainlog, s_chainlog,
hashlog, searchlog, s_hashlog,
minmatch, targetlength, s_searchlog,
strategy): s_minmatch,
if minmatch == zstd.MINMATCH_MIN and strategy in (zstd.STRATEGY_FAST, zstd.STRATEGY_GREEDY): s_targetlength,
s_strategy,
)
def test_estimated_compression_context_size(
self, windowlog, chainlog, hashlog, searchlog, minmatch, targetlength, strategy
):
if minmatch == zstd.MINMATCH_MIN and strategy in (
zstd.STRATEGY_FAST,
zstd.STRATEGY_GREEDY,
):
minmatch += 1 minmatch += 1
elif minmatch == zstd.MINMATCH_MAX and strategy != zstd.STRATEGY_FAST: elif minmatch == zstd.MINMATCH_MAX and strategy != zstd.STRATEGY_FAST:
minmatch -= 1 minmatch -= 1
p = zstd.ZstdCompressionParameters(window_log=windowlog, p = zstd.ZstdCompressionParameters(
window_log=windowlog,
chain_log=chainlog, chain_log=chainlog,
hash_log=hashlog, hash_log=hashlog,
search_log=searchlog, search_log=searchlog,
min_match=minmatch, min_match=minmatch,
target_length=targetlength, target_length=targetlength,
strategy=strategy) strategy=strategy,
)
size = p.estimated_compression_context_size() size = p.estimated_compression_context_size()

contrib/python-zstandard/tests/test_decompressor.py

import io import io
import os import os
import random import random
import struct import struct
import sys import sys
import tempfile import tempfile
import unittest import unittest
import zstandard as zstd import zstandard as zstd
from .common import ( from .common import (
generate_samples, generate_samples,
make_cffi, make_cffi,
NonClosingBytesIO, NonClosingBytesIO,
OpCountingBytesIO, OpCountingBytesIO,
TestCase,
) )
if sys.version_info[0] >= 3: if sys.version_info[0] >= 3:
next = lambda it: it.__next__() next = lambda it: it.__next__()
else: else:
next = lambda it: it.next() next = lambda it: it.next()
@make_cffi @make_cffi
class TestFrameHeaderSize(unittest.TestCase): class TestFrameHeaderSize(TestCase):
def test_empty(self): def test_empty(self):
with self.assertRaisesRegexp( with self.assertRaisesRegex(
zstd.ZstdError, 'could not determine frame header size: Src size ' zstd.ZstdError,
'is incorrect'): "could not determine frame header size: Src size " "is incorrect",
zstd.frame_header_size(b'') ):
zstd.frame_header_size(b"")
def test_too_small(self): def test_too_small(self):
with self.assertRaisesRegexp( with self.assertRaisesRegex(
zstd.ZstdError, 'could not determine frame header size: Src size ' zstd.ZstdError,
'is incorrect'): "could not determine frame header size: Src size " "is incorrect",
zstd.frame_header_size(b'foob') ):
zstd.frame_header_size(b"foob")
def test_basic(self): def test_basic(self):
# It doesn't matter that it isn't a valid frame. # It doesn't matter that it isn't a valid frame.
self.assertEqual(zstd.frame_header_size(b'long enough but no magic'), 6) self.assertEqual(zstd.frame_header_size(b"long enough but no magic"), 6)
@make_cffi @make_cffi
class TestFrameContentSize(unittest.TestCase): class TestFrameContentSize(TestCase):
def test_empty(self): def test_empty(self):
with self.assertRaisesRegexp(zstd.ZstdError, with self.assertRaisesRegex(
'error when determining content size'): zstd.ZstdError, "error when determining content size"
zstd.frame_content_size(b'') ):
zstd.frame_content_size(b"")
def test_too_small(self): def test_too_small(self):
with self.assertRaisesRegexp(zstd.ZstdError, with self.assertRaisesRegex(
'error when determining content size'): zstd.ZstdError, "error when determining content size"
zstd.frame_content_size(b'foob') ):
zstd.frame_content_size(b"foob")
def test_bad_frame(self): def test_bad_frame(self):
with self.assertRaisesRegexp(zstd.ZstdError, with self.assertRaisesRegex(
'error when determining content size'): zstd.ZstdError, "error when determining content size"
zstd.frame_content_size(b'invalid frame header') ):
zstd.frame_content_size(b"invalid frame header")
def test_unknown(self): def test_unknown(self):
cctx = zstd.ZstdCompressor(write_content_size=False) cctx = zstd.ZstdCompressor(write_content_size=False)
frame = cctx.compress(b'foobar') frame = cctx.compress(b"foobar")
self.assertEqual(zstd.frame_content_size(frame), -1) self.assertEqual(zstd.frame_content_size(frame), -1)
def test_empty(self): def test_empty(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
frame = cctx.compress(b'') frame = cctx.compress(b"")
self.assertEqual(zstd.frame_content_size(frame), 0) self.assertEqual(zstd.frame_content_size(frame), 0)
def test_basic(self): def test_basic(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
frame = cctx.compress(b'foobar') frame = cctx.compress(b"foobar")
self.assertEqual(zstd.frame_content_size(frame), 6) self.assertEqual(zstd.frame_content_size(frame), 6)
@make_cffi @make_cffi
class TestDecompressor(unittest.TestCase): class TestDecompressor(TestCase):
def test_memory_size(self): def test_memory_size(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
self.assertGreater(dctx.memory_size(), 100) self.assertGreater(dctx.memory_size(), 100)
@make_cffi @make_cffi
class TestDecompressor_decompress(unittest.TestCase): class TestDecompressor_decompress(TestCase):
def test_empty_input(self): def test_empty_input(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with self.assertRaisesRegexp(zstd.ZstdError, 'error determining content size from frame header'): with self.assertRaisesRegex(
dctx.decompress(b'') zstd.ZstdError, "error determining content size from frame header"
):
dctx.decompress(b"")
def test_invalid_input(self): def test_invalid_input(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with self.assertRaisesRegexp(zstd.ZstdError, 'error determining content size from frame header'): with self.assertRaisesRegex(
dctx.decompress(b'foobar') zstd.ZstdError, "error determining content size from frame header"
):
dctx.decompress(b"foobar")
def test_input_types(self): def test_input_types(self):
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
compressed = cctx.compress(b'foo') compressed = cctx.compress(b"foo")
mutable_array = bytearray(len(compressed)) mutable_array = bytearray(len(compressed))
mutable_array[:] = compressed mutable_array[:] = compressed
sources = [ sources = [
memoryview(compressed), memoryview(compressed),
bytearray(compressed), bytearray(compressed),
mutable_array, mutable_array,
] ]
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
for source in sources: for source in sources:
self.assertEqual(dctx.decompress(source), b'foo') self.assertEqual(dctx.decompress(source), b"foo")
def test_no_content_size_in_frame(self): def test_no_content_size_in_frame(self):
cctx = zstd.ZstdCompressor(write_content_size=False) cctx = zstd.ZstdCompressor(write_content_size=False)
compressed = cctx.compress(b'foobar') compressed = cctx.compress(b"foobar")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with self.assertRaisesRegexp(zstd.ZstdError, 'could not determine content size in frame header'): with self.assertRaisesRegex(
zstd.ZstdError, "could not determine content size in frame header"
):
dctx.decompress(compressed) dctx.decompress(compressed)
def test_content_size_present(self): def test_content_size_present(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
compressed = cctx.compress(b'foobar') compressed = cctx.compress(b"foobar")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
decompressed = dctx.decompress(compressed) decompressed = dctx.decompress(compressed)
self.assertEqual(decompressed, b'foobar') self.assertEqual(decompressed, b"foobar")
def test_empty_roundtrip(self): def test_empty_roundtrip(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
compressed = cctx.compress(b'') compressed = cctx.compress(b"")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
decompressed = dctx.decompress(compressed) decompressed = dctx.decompress(compressed)
self.assertEqual(decompressed, b'') self.assertEqual(decompressed, b"")
def test_max_output_size(self): def test_max_output_size(self):
cctx = zstd.ZstdCompressor(write_content_size=False) cctx = zstd.ZstdCompressor(write_content_size=False)
source = b'foobar' * 256 source = b"foobar" * 256
compressed = cctx.compress(source) compressed = cctx.compress(source)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
# Will fit into buffer exactly the size of input. # Will fit into buffer exactly the size of input.
decompressed = dctx.decompress(compressed, max_output_size=len(source)) decompressed = dctx.decompress(compressed, max_output_size=len(source))
self.assertEqual(decompressed, source) self.assertEqual(decompressed, source)
# Input size - 1 fails # Input size - 1 fails
with self.assertRaisesRegexp(zstd.ZstdError, with self.assertRaisesRegex(
'decompression error: did not decompress full frame'): zstd.ZstdError, "decompression error: did not decompress full frame"
):
dctx.decompress(compressed, max_output_size=len(source) - 1) dctx.decompress(compressed, max_output_size=len(source) - 1)
# Input size + 1 works # Input size + 1 works
decompressed = dctx.decompress(compressed, max_output_size=len(source) + 1) decompressed = dctx.decompress(compressed, max_output_size=len(source) + 1)
self.assertEqual(decompressed, source) self.assertEqual(decompressed, source)
# A much larger buffer works. # A much larger buffer works.
decompressed = dctx.decompress(compressed, max_output_size=len(source) * 64) decompressed = dctx.decompress(compressed, max_output_size=len(source) * 64)
self.assertEqual(decompressed, source) self.assertEqual(decompressed, source)
def test_stupidly_large_output_buffer(self): def test_stupidly_large_output_buffer(self):
cctx = zstd.ZstdCompressor(write_content_size=False) cctx = zstd.ZstdCompressor(write_content_size=False)
compressed = cctx.compress(b'foobar' * 256) compressed = cctx.compress(b"foobar" * 256)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
# Will get OverflowError on some Python distributions that can't # Will get OverflowError on some Python distributions that can't
# handle really large integers. # handle really large integers.
with self.assertRaises((MemoryError, OverflowError)): with self.assertRaises((MemoryError, OverflowError)):
dctx.decompress(compressed, max_output_size=2**62) dctx.decompress(compressed, max_output_size=2 ** 62)
def test_dictionary(self): def test_dictionary(self):
samples = [] samples = []
for i in range(128): for i in range(128):
samples.append(b'foo' * 64) samples.append(b"foo" * 64)
samples.append(b'bar' * 64) samples.append(b"bar" * 64)
samples.append(b'foobar' * 64) samples.append(b"foobar" * 64)
d = zstd.train_dictionary(8192, samples) d = zstd.train_dictionary(8192, samples)
orig = b'foobar' * 16384 orig = b"foobar" * 16384
cctx = zstd.ZstdCompressor(level=1, dict_data=d) cctx = zstd.ZstdCompressor(level=1, dict_data=d)
compressed = cctx.compress(orig) compressed = cctx.compress(orig)
dctx = zstd.ZstdDecompressor(dict_data=d) dctx = zstd.ZstdDecompressor(dict_data=d)
decompressed = dctx.decompress(compressed) decompressed = dctx.decompress(compressed)
self.assertEqual(decompressed, orig) self.assertEqual(decompressed, orig)
def test_dictionary_multiple(self): def test_dictionary_multiple(self):
samples = [] samples = []
for i in range(128): for i in range(128):
samples.append(b'foo' * 64) samples.append(b"foo" * 64)
samples.append(b'bar' * 64) samples.append(b"bar" * 64)
samples.append(b'foobar' * 64) samples.append(b"foobar" * 64)
d = zstd.train_dictionary(8192, samples) d = zstd.train_dictionary(8192, samples)
sources = (b'foobar' * 8192, b'foo' * 8192, b'bar' * 8192) sources = (b"foobar" * 8192, b"foo" * 8192, b"bar" * 8192)
compressed = [] compressed = []
cctx = zstd.ZstdCompressor(level=1, dict_data=d) cctx = zstd.ZstdCompressor(level=1, dict_data=d)
for source in sources: for source in sources:
compressed.append(cctx.compress(source)) compressed.append(cctx.compress(source))
dctx = zstd.ZstdDecompressor(dict_data=d) dctx = zstd.ZstdDecompressor(dict_data=d)
for i in range(len(sources)): for i in range(len(sources)):
decompressed = dctx.decompress(compressed[i]) decompressed = dctx.decompress(compressed[i])
self.assertEqual(decompressed, sources[i]) self.assertEqual(decompressed, sources[i])
def test_max_window_size(self): def test_max_window_size(self):
with open(__file__, 'rb') as fh: with open(__file__, "rb") as fh:
source = fh.read() source = fh.read()
# If we write a content size, the decompressor engages single pass # If we write a content size, the decompressor engages single pass
# mode and the window size doesn't come into play. # mode and the window size doesn't come into play.
cctx = zstd.ZstdCompressor(write_content_size=False) cctx = zstd.ZstdCompressor(write_content_size=False)
frame = cctx.compress(source) frame = cctx.compress(source)
dctx = zstd.ZstdDecompressor(max_window_size=2**zstd.WINDOWLOG_MIN) dctx = zstd.ZstdDecompressor(max_window_size=2 ** zstd.WINDOWLOG_MIN)
with self.assertRaisesRegexp( with self.assertRaisesRegex(
zstd.ZstdError, 'decompression error: Frame requires too much memory'): zstd.ZstdError, "decompression error: Frame requires too much memory"
):
dctx.decompress(frame, max_output_size=len(source)) dctx.decompress(frame, max_output_size=len(source))
@make_cffi @make_cffi
class TestDecompressor_copy_stream(unittest.TestCase): class TestDecompressor_copy_stream(TestCase):
def test_no_read(self): def test_no_read(self):
source = object() source = object()
dest = io.BytesIO() dest = io.BytesIO()
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with self.assertRaises(ValueError): with self.assertRaises(ValueError):
dctx.copy_stream(source, dest) dctx.copy_stream(source, dest)
dest = io.BytesIO() dest = io.BytesIO()
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
# TODO should this raise an error? # TODO should this raise an error?
r, w = dctx.copy_stream(source, dest) r, w = dctx.copy_stream(source, dest)
self.assertEqual(r, 0) self.assertEqual(r, 0)
self.assertEqual(w, 0) self.assertEqual(w, 0)
self.assertEqual(dest.getvalue(), b'') self.assertEqual(dest.getvalue(), b"")
def test_large_data(self): def test_large_data(self):
source = io.BytesIO() source = io.BytesIO()
for i in range(255): for i in range(255):
source.write(struct.Struct('>B').pack(i) * 16384) source.write(struct.Struct(">B").pack(i) * 16384)
source.seek(0) source.seek(0)
compressed = io.BytesIO() compressed = io.BytesIO()
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
cctx.copy_stream(source, compressed) cctx.copy_stream(source, compressed)
compressed.seek(0) compressed.seek(0)
dest = io.BytesIO() dest = io.BytesIO()
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
r, w = dctx.copy_stream(compressed, dest) r, w = dctx.copy_stream(compressed, dest)
self.assertEqual(r, len(compressed.getvalue())) self.assertEqual(r, len(compressed.getvalue()))
self.assertEqual(w, len(source.getvalue())) self.assertEqual(w, len(source.getvalue()))
def test_read_write_size(self): def test_read_write_size(self):
source = OpCountingBytesIO(zstd.ZstdCompressor().compress( source = OpCountingBytesIO(zstd.ZstdCompressor().compress(b"foobarfoobar"))
b'foobarfoobar'))
dest = OpCountingBytesIO() dest = OpCountingBytesIO()
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
r, w = dctx.copy_stream(source, dest, read_size=1, write_size=1) r, w = dctx.copy_stream(source, dest, read_size=1, write_size=1)
self.assertEqual(r, len(source.getvalue())) self.assertEqual(r, len(source.getvalue()))
self.assertEqual(w, len(b'foobarfoobar')) self.assertEqual(w, len(b"foobarfoobar"))
self.assertEqual(source._read_count, len(source.getvalue()) + 1) self.assertEqual(source._read_count, len(source.getvalue()) + 1)
self.assertEqual(dest._write_count, len(dest.getvalue())) self.assertEqual(dest._write_count, len(dest.getvalue()))
@make_cffi @make_cffi
class TestDecompressor_stream_reader(unittest.TestCase): class TestDecompressor_stream_reader(TestCase):
def test_context_manager(self): def test_context_manager(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(b'foo') as reader: with dctx.stream_reader(b"foo") as reader:
with self.assertRaisesRegexp(ValueError, 'cannot __enter__ multiple times'): with self.assertRaisesRegex(ValueError, "cannot __enter__ multiple times"):
with reader as reader2: with reader as reader2:
pass pass
def test_not_implemented(self): def test_not_implemented(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(b'foo') as reader: with dctx.stream_reader(b"foo") as reader:
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
reader.readline() reader.readline()
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
reader.readlines() reader.readlines()
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
iter(reader) iter(reader)
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
next(reader) next(reader)
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
reader.write(b'foo') reader.write(b"foo")
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
reader.writelines([]) reader.writelines([])
def test_constant_methods(self): def test_constant_methods(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(b'foo') as reader: with dctx.stream_reader(b"foo") as reader:
self.assertFalse(reader.closed) self.assertFalse(reader.closed)
self.assertTrue(reader.readable()) self.assertTrue(reader.readable())
self.assertFalse(reader.writable()) self.assertFalse(reader.writable())
self.assertTrue(reader.seekable()) self.assertTrue(reader.seekable())
self.assertFalse(reader.isatty()) self.assertFalse(reader.isatty())
self.assertFalse(reader.closed) self.assertFalse(reader.closed)
self.assertIsNone(reader.flush()) self.assertIsNone(reader.flush())
self.assertFalse(reader.closed) self.assertFalse(reader.closed)
self.assertTrue(reader.closed) self.assertTrue(reader.closed)
def test_read_closed(self): def test_read_closed(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(b'foo') as reader: with dctx.stream_reader(b"foo") as reader:
reader.close() reader.close()
self.assertTrue(reader.closed) self.assertTrue(reader.closed)
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
reader.read(1) reader.read(1)
def test_read_sizes(self): def test_read_sizes(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
foo = cctx.compress(b'foo') foo = cctx.compress(b"foo")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(foo) as reader: with dctx.stream_reader(foo) as reader:
with self.assertRaisesRegexp(ValueError, 'cannot read negative amounts less than -1'): with self.assertRaisesRegex(
ValueError, "cannot read negative amounts less than -1"
):
reader.read(-2) reader.read(-2)
self.assertEqual(reader.read(0), b'') self.assertEqual(reader.read(0), b"")
self.assertEqual(reader.read(), b'foo') self.assertEqual(reader.read(), b"foo")
def test_read_buffer(self): def test_read_buffer(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
source = b''.join([b'foo' * 60, b'bar' * 60, b'baz' * 60]) source = b"".join([b"foo" * 60, b"bar" * 60, b"baz" * 60])
frame = cctx.compress(source) frame = cctx.compress(source)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(frame) as reader: with dctx.stream_reader(frame) as reader:
self.assertEqual(reader.tell(), 0) self.assertEqual(reader.tell(), 0)
# We should get entire frame in one read. # We should get entire frame in one read.
result = reader.read(8192) result = reader.read(8192)
self.assertEqual(result, source) self.assertEqual(result, source)
self.assertEqual(reader.tell(), len(source)) self.assertEqual(reader.tell(), len(source))
# Read after EOF should return empty bytes. # Read after EOF should return empty bytes.
self.assertEqual(reader.read(1), b'') self.assertEqual(reader.read(1), b"")
self.assertEqual(reader.tell(), len(result)) self.assertEqual(reader.tell(), len(result))
self.assertTrue(reader.closed) self.assertTrue(reader.closed)
def test_read_buffer_small_chunks(self): def test_read_buffer_small_chunks(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
source = b''.join([b'foo' * 60, b'bar' * 60, b'baz' * 60]) source = b"".join([b"foo" * 60, b"bar" * 60, b"baz" * 60])
frame = cctx.compress(source) frame = cctx.compress(source)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
chunks = [] chunks = []
with dctx.stream_reader(frame, read_size=1) as reader: with dctx.stream_reader(frame, read_size=1) as reader:
while True: while True:
chunk = reader.read(1) chunk = reader.read(1)
if not chunk: if not chunk:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(reader.tell(), sum(map(len, chunks))) self.assertEqual(reader.tell(), sum(map(len, chunks)))
self.assertEqual(b''.join(chunks), source) self.assertEqual(b"".join(chunks), source)
def test_read_stream(self): def test_read_stream(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
source = b''.join([b'foo' * 60, b'bar' * 60, b'baz' * 60]) source = b"".join([b"foo" * 60, b"bar" * 60, b"baz" * 60])
frame = cctx.compress(source) frame = cctx.compress(source)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(io.BytesIO(frame)) as reader: with dctx.stream_reader(io.BytesIO(frame)) as reader:
self.assertEqual(reader.tell(), 0) self.assertEqual(reader.tell(), 0)
chunk = reader.read(8192) chunk = reader.read(8192)
self.assertEqual(chunk, source) self.assertEqual(chunk, source)
self.assertEqual(reader.tell(), len(source)) self.assertEqual(reader.tell(), len(source))
self.assertEqual(reader.read(1), b'') self.assertEqual(reader.read(1), b"")
self.assertEqual(reader.tell(), len(source)) self.assertEqual(reader.tell(), len(source))
self.assertFalse(reader.closed) self.assertFalse(reader.closed)
self.assertTrue(reader.closed) self.assertTrue(reader.closed)
def test_read_stream_small_chunks(self): def test_read_stream_small_chunks(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
source = b''.join([b'foo' * 60, b'bar' * 60, b'baz' * 60]) source = b"".join([b"foo" * 60, b"bar" * 60, b"baz" * 60])
frame = cctx.compress(source) frame = cctx.compress(source)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
chunks = [] chunks = []
with dctx.stream_reader(io.BytesIO(frame), read_size=1) as reader: with dctx.stream_reader(io.BytesIO(frame), read_size=1) as reader:
while True: while True:
chunk = reader.read(1) chunk = reader.read(1)
if not chunk: if not chunk:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(reader.tell(), sum(map(len, chunks))) self.assertEqual(reader.tell(), sum(map(len, chunks)))
self.assertEqual(b''.join(chunks), source) self.assertEqual(b"".join(chunks), source)
def test_read_after_exit(self): def test_read_after_exit(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
frame = cctx.compress(b'foo' * 60) frame = cctx.compress(b"foo" * 60)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(frame) as reader: with dctx.stream_reader(frame) as reader:
while reader.read(16): while reader.read(16):
pass pass
self.assertTrue(reader.closed) self.assertTrue(reader.closed)
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
reader.read(10) reader.read(10)
def test_illegal_seeks(self): def test_illegal_seeks(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
frame = cctx.compress(b'foo' * 60) frame = cctx.compress(b"foo" * 60)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(frame) as reader: with dctx.stream_reader(frame) as reader:
with self.assertRaisesRegexp(ValueError, with self.assertRaisesRegex(ValueError, "cannot seek to negative position"):
'cannot seek to negative position'):
reader.seek(-1, os.SEEK_SET) reader.seek(-1, os.SEEK_SET)
reader.read(1) reader.read(1)
with self.assertRaisesRegexp( with self.assertRaisesRegex(
ValueError, 'cannot seek zstd decompression stream backwards'): ValueError, "cannot seek zstd decompression stream backwards"
):
reader.seek(0, os.SEEK_SET) reader.seek(0, os.SEEK_SET)
with self.assertRaisesRegexp( with self.assertRaisesRegex(
ValueError, 'cannot seek zstd decompression stream backwards'): ValueError, "cannot seek zstd decompression stream backwards"
):
reader.seek(-1, os.SEEK_CUR) reader.seek(-1, os.SEEK_CUR)
with self.assertRaisesRegexp( with self.assertRaisesRegex(
ValueError, ValueError, "zstd decompression streams cannot be seeked with SEEK_END"
'zstd decompression streams cannot be seeked with SEEK_END'): ):
reader.seek(0, os.SEEK_END) reader.seek(0, os.SEEK_END)
reader.close() reader.close()
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
reader.seek(4, os.SEEK_SET) reader.seek(4, os.SEEK_SET)
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
reader.seek(0) reader.seek(0)
def test_seek(self): def test_seek(self):
source = b'foobar' * 60 source = b"foobar" * 60
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
frame = cctx.compress(source) frame = cctx.compress(source)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(frame) as reader: with dctx.stream_reader(frame) as reader:
reader.seek(3) reader.seek(3)
self.assertEqual(reader.read(3), b'bar') self.assertEqual(reader.read(3), b"bar")
reader.seek(4, os.SEEK_CUR) reader.seek(4, os.SEEK_CUR)
self.assertEqual(reader.read(2), b'ar') self.assertEqual(reader.read(2), b"ar")
def test_no_context_manager(self): def test_no_context_manager(self):
source = b'foobar' * 60 source = b"foobar" * 60
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
frame = cctx.compress(source) frame = cctx.compress(source)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
reader = dctx.stream_reader(frame) reader = dctx.stream_reader(frame)
self.assertEqual(reader.read(6), b'foobar') self.assertEqual(reader.read(6), b"foobar")
self.assertEqual(reader.read(18), b'foobar' * 3) self.assertEqual(reader.read(18), b"foobar" * 3)
self.assertFalse(reader.closed) self.assertFalse(reader.closed)
# Calling close prevents subsequent use. # Calling close prevents subsequent use.
reader.close() reader.close()
self.assertTrue(reader.closed) self.assertTrue(reader.closed)
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
reader.read(6) reader.read(6)
def test_read_after_error(self): def test_read_after_error(self):
source = io.BytesIO(b'') source = io.BytesIO(b"")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
reader = dctx.stream_reader(source) reader = dctx.stream_reader(source)
with reader: with reader:
reader.read(0) reader.read(0)
with reader: with reader:
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
reader.read(100) reader.read(100)
def test_partial_read(self): def test_partial_read(self):
# Inspired by https://github.com/indygreg/python-zstandard/issues/71. # Inspired by https://github.com/indygreg/python-zstandard/issues/71.
buffer = io.BytesIO() buffer = io.BytesIO()
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
writer = cctx.stream_writer(buffer) writer = cctx.stream_writer(buffer)
writer.write(bytearray(os.urandom(1000000))) writer.write(bytearray(os.urandom(1000000)))
writer.flush(zstd.FLUSH_FRAME) writer.flush(zstd.FLUSH_FRAME)
buffer.seek(0) buffer.seek(0)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
reader = dctx.stream_reader(buffer) reader = dctx.stream_reader(buffer)
while True: while True:
chunk = reader.read(8192) chunk = reader.read(8192)
if not chunk: if not chunk:
break break
def test_read_multiple_frames(self): def test_read_multiple_frames(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
source = io.BytesIO() source = io.BytesIO()
writer = cctx.stream_writer(source) writer = cctx.stream_writer(source)
writer.write(b'foo') writer.write(b"foo")
writer.flush(zstd.FLUSH_FRAME) writer.flush(zstd.FLUSH_FRAME)
writer.write(b'bar') writer.write(b"bar")
writer.flush(zstd.FLUSH_FRAME) writer.flush(zstd.FLUSH_FRAME)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
reader = dctx.stream_reader(source.getvalue()) reader = dctx.stream_reader(source.getvalue())
self.assertEqual(reader.read(2), b'fo') self.assertEqual(reader.read(2), b"fo")
self.assertEqual(reader.read(2), b'o') self.assertEqual(reader.read(2), b"o")
self.assertEqual(reader.read(2), b'ba') self.assertEqual(reader.read(2), b"ba")
self.assertEqual(reader.read(2), b'r') self.assertEqual(reader.read(2), b"r")
source.seek(0) source.seek(0)
reader = dctx.stream_reader(source) reader = dctx.stream_reader(source)
self.assertEqual(reader.read(2), b'fo') self.assertEqual(reader.read(2), b"fo")
self.assertEqual(reader.read(2), b'o') self.assertEqual(reader.read(2), b"o")
self.assertEqual(reader.read(2), b'ba') self.assertEqual(reader.read(2), b"ba")
self.assertEqual(reader.read(2), b'r') self.assertEqual(reader.read(2), b"r")
reader = dctx.stream_reader(source.getvalue()) reader = dctx.stream_reader(source.getvalue())
self.assertEqual(reader.read(3), b'foo') self.assertEqual(reader.read(3), b"foo")
self.assertEqual(reader.read(3), b'bar') self.assertEqual(reader.read(3), b"bar")
source.seek(0) source.seek(0)
reader = dctx.stream_reader(source) reader = dctx.stream_reader(source)
self.assertEqual(reader.read(3), b'foo') self.assertEqual(reader.read(3), b"foo")
self.assertEqual(reader.read(3), b'bar') self.assertEqual(reader.read(3), b"bar")
reader = dctx.stream_reader(source.getvalue()) reader = dctx.stream_reader(source.getvalue())
self.assertEqual(reader.read(4), b'foo') self.assertEqual(reader.read(4), b"foo")
self.assertEqual(reader.read(4), b'bar') self.assertEqual(reader.read(4), b"bar")
source.seek(0) source.seek(0)
reader = dctx.stream_reader(source) reader = dctx.stream_reader(source)
self.assertEqual(reader.read(4), b'foo') self.assertEqual(reader.read(4), b"foo")
self.assertEqual(reader.read(4), b'bar') self.assertEqual(reader.read(4), b"bar")
reader = dctx.stream_reader(source.getvalue()) reader = dctx.stream_reader(source.getvalue())
self.assertEqual(reader.read(128), b'foo') self.assertEqual(reader.read(128), b"foo")
self.assertEqual(reader.read(128), b'bar') self.assertEqual(reader.read(128), b"bar")
source.seek(0) source.seek(0)
reader = dctx.stream_reader(source) reader = dctx.stream_reader(source)
self.assertEqual(reader.read(128), b'foo') self.assertEqual(reader.read(128), b"foo")
self.assertEqual(reader.read(128), b'bar') self.assertEqual(reader.read(128), b"bar")
# Now tests for reads spanning frames. # Now tests for reads spanning frames.
reader = dctx.stream_reader(source.getvalue(), read_across_frames=True) reader = dctx.stream_reader(source.getvalue(), read_across_frames=True)
self.assertEqual(reader.read(3), b'foo') self.assertEqual(reader.read(3), b"foo")
self.assertEqual(reader.read(3), b'bar') self.assertEqual(reader.read(3), b"bar")
source.seek(0) source.seek(0)
reader = dctx.stream_reader(source, read_across_frames=True) reader = dctx.stream_reader(source, read_across_frames=True)
self.assertEqual(reader.read(3), b'foo') self.assertEqual(reader.read(3), b"foo")
self.assertEqual(reader.read(3), b'bar') self.assertEqual(reader.read(3), b"bar")
reader = dctx.stream_reader(source.getvalue(), read_across_frames=True) reader = dctx.stream_reader(source.getvalue(), read_across_frames=True)
self.assertEqual(reader.read(6), b'foobar') self.assertEqual(reader.read(6), b"foobar")
source.seek(0) source.seek(0)
reader = dctx.stream_reader(source, read_across_frames=True) reader = dctx.stream_reader(source, read_across_frames=True)
self.assertEqual(reader.read(6), b'foobar') self.assertEqual(reader.read(6), b"foobar")
reader = dctx.stream_reader(source.getvalue(), read_across_frames=True) reader = dctx.stream_reader(source.getvalue(), read_across_frames=True)
self.assertEqual(reader.read(7), b'foobar') self.assertEqual(reader.read(7), b"foobar")
source.seek(0) source.seek(0)
reader = dctx.stream_reader(source, read_across_frames=True) reader = dctx.stream_reader(source, read_across_frames=True)
self.assertEqual(reader.read(7), b'foobar') self.assertEqual(reader.read(7), b"foobar")
reader = dctx.stream_reader(source.getvalue(), read_across_frames=True) reader = dctx.stream_reader(source.getvalue(), read_across_frames=True)
self.assertEqual(reader.read(128), b'foobar') self.assertEqual(reader.read(128), b"foobar")
source.seek(0) source.seek(0)
reader = dctx.stream_reader(source, read_across_frames=True) reader = dctx.stream_reader(source, read_across_frames=True)
self.assertEqual(reader.read(128), b'foobar') self.assertEqual(reader.read(128), b"foobar")
def test_readinto(self): def test_readinto(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
foo = cctx.compress(b'foo') foo = cctx.compress(b"foo")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
# Attempting to readinto() a non-writable buffer fails. # Attempting to readinto() a non-writable buffer fails.
# The exact exception varies based on the backend. # The exact exception varies based on the backend.
reader = dctx.stream_reader(foo) reader = dctx.stream_reader(foo)
with self.assertRaises(Exception): with self.assertRaises(Exception):
reader.readinto(b'foobar') reader.readinto(b"foobar")
# readinto() with sufficiently large destination. # readinto() with sufficiently large destination.
b = bytearray(1024) b = bytearray(1024)
reader = dctx.stream_reader(foo) reader = dctx.stream_reader(foo)
self.assertEqual(reader.readinto(b), 3) self.assertEqual(reader.readinto(b), 3)
self.assertEqual(b[0:3], b'foo') self.assertEqual(b[0:3], b"foo")
self.assertEqual(reader.readinto(b), 0) self.assertEqual(reader.readinto(b), 0)
self.assertEqual(b[0:3], b'foo') self.assertEqual(b[0:3], b"foo")
# readinto() with small reads. # readinto() with small reads.
b = bytearray(1024) b = bytearray(1024)
reader = dctx.stream_reader(foo, read_size=1) reader = dctx.stream_reader(foo, read_size=1)
self.assertEqual(reader.readinto(b), 3) self.assertEqual(reader.readinto(b), 3)
self.assertEqual(b[0:3], b'foo') self.assertEqual(b[0:3], b"foo")
# Too small destination buffer. # Too small destination buffer.
b = bytearray(2) b = bytearray(2)
reader = dctx.stream_reader(foo) reader = dctx.stream_reader(foo)
self.assertEqual(reader.readinto(b), 2) self.assertEqual(reader.readinto(b), 2)
self.assertEqual(b[:], b'fo') self.assertEqual(b[:], b"fo")
def test_readinto1(self): def test_readinto1(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
foo = cctx.compress(b'foo') foo = cctx.compress(b"foo")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
reader = dctx.stream_reader(foo) reader = dctx.stream_reader(foo)
with self.assertRaises(Exception): with self.assertRaises(Exception):
reader.readinto1(b'foobar') reader.readinto1(b"foobar")
# Sufficiently large destination. # Sufficiently large destination.
b = bytearray(1024) b = bytearray(1024)
reader = dctx.stream_reader(foo) reader = dctx.stream_reader(foo)
self.assertEqual(reader.readinto1(b), 3) self.assertEqual(reader.readinto1(b), 3)
self.assertEqual(b[0:3], b'foo') self.assertEqual(b[0:3], b"foo")
self.assertEqual(reader.readinto1(b), 0) self.assertEqual(reader.readinto1(b), 0)
self.assertEqual(b[0:3], b'foo') self.assertEqual(b[0:3], b"foo")
# readinto() with small reads. # readinto() with small reads.
b = bytearray(1024) b = bytearray(1024)
reader = dctx.stream_reader(foo, read_size=1) reader = dctx.stream_reader(foo, read_size=1)
self.assertEqual(reader.readinto1(b), 3) self.assertEqual(reader.readinto1(b), 3)
self.assertEqual(b[0:3], b'foo') self.assertEqual(b[0:3], b"foo")
# Too small destination buffer. # Too small destination buffer.
b = bytearray(2) b = bytearray(2)
reader = dctx.stream_reader(foo) reader = dctx.stream_reader(foo)
self.assertEqual(reader.readinto1(b), 2) self.assertEqual(reader.readinto1(b), 2)
self.assertEqual(b[:], b'fo') self.assertEqual(b[:], b"fo")
def test_readall(self): def test_readall(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
foo = cctx.compress(b'foo') foo = cctx.compress(b"foo")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
reader = dctx.stream_reader(foo) reader = dctx.stream_reader(foo)
self.assertEqual(reader.readall(), b'foo') self.assertEqual(reader.readall(), b"foo")
def test_read1(self): def test_read1(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
foo = cctx.compress(b'foo') foo = cctx.compress(b"foo")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
b = OpCountingBytesIO(foo) b = OpCountingBytesIO(foo)
reader = dctx.stream_reader(b) reader = dctx.stream_reader(b)
self.assertEqual(reader.read1(), b'foo') self.assertEqual(reader.read1(), b"foo")
self.assertEqual(b._read_count, 1) self.assertEqual(b._read_count, 1)
b = OpCountingBytesIO(foo) b = OpCountingBytesIO(foo)
reader = dctx.stream_reader(b) reader = dctx.stream_reader(b)
self.assertEqual(reader.read1(0), b'') self.assertEqual(reader.read1(0), b"")
self.assertEqual(reader.read1(2), b'fo') self.assertEqual(reader.read1(2), b"fo")
self.assertEqual(b._read_count, 1) self.assertEqual(b._read_count, 1)
self.assertEqual(reader.read1(1), b'o') self.assertEqual(reader.read1(1), b"o")
self.assertEqual(b._read_count, 1) self.assertEqual(b._read_count, 1)
self.assertEqual(reader.read1(1), b'') self.assertEqual(reader.read1(1), b"")
self.assertEqual(b._read_count, 2) self.assertEqual(b._read_count, 2)
def test_read_lines(self): def test_read_lines(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
source = b'\n'.join(('line %d' % i).encode('ascii') for i in range(1024)) source = b"\n".join(("line %d" % i).encode("ascii") for i in range(1024))
frame = cctx.compress(source) frame = cctx.compress(source)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
reader = dctx.stream_reader(frame) reader = dctx.stream_reader(frame)
tr = io.TextIOWrapper(reader, encoding='utf-8') tr = io.TextIOWrapper(reader, encoding="utf-8")
lines = [] lines = []
for line in tr: for line in tr:
lines.append(line.encode('utf-8')) lines.append(line.encode("utf-8"))
self.assertEqual(len(lines), 1024) self.assertEqual(len(lines), 1024)
self.assertEqual(b''.join(lines), source) self.assertEqual(b"".join(lines), source)
reader = dctx.stream_reader(frame) reader = dctx.stream_reader(frame)
tr = io.TextIOWrapper(reader, encoding='utf-8') tr = io.TextIOWrapper(reader, encoding="utf-8")
lines = tr.readlines() lines = tr.readlines()
self.assertEqual(len(lines), 1024) self.assertEqual(len(lines), 1024)
self.assertEqual(''.join(lines).encode('utf-8'), source) self.assertEqual("".join(lines).encode("utf-8"), source)
reader = dctx.stream_reader(frame) reader = dctx.stream_reader(frame)
tr = io.TextIOWrapper(reader, encoding='utf-8') tr = io.TextIOWrapper(reader, encoding="utf-8")
lines = [] lines = []
while True: while True:
line = tr.readline() line = tr.readline()
if not line: if not line:
break break
lines.append(line.encode('utf-8')) lines.append(line.encode("utf-8"))
self.assertEqual(len(lines), 1024) self.assertEqual(len(lines), 1024)
self.assertEqual(b''.join(lines), source) self.assertEqual(b"".join(lines), source)
@make_cffi @make_cffi
class TestDecompressor_decompressobj(unittest.TestCase): class TestDecompressor_decompressobj(TestCase):
def test_simple(self): def test_simple(self):
data = zstd.ZstdCompressor(level=1).compress(b'foobar') data = zstd.ZstdCompressor(level=1).compress(b"foobar")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
dobj = dctx.decompressobj() dobj = dctx.decompressobj()
self.assertEqual(dobj.decompress(data), b'foobar') self.assertEqual(dobj.decompress(data), b"foobar")
self.assertIsNone(dobj.flush()) self.assertIsNone(dobj.flush())
self.assertIsNone(dobj.flush(10)) self.assertIsNone(dobj.flush(10))
self.assertIsNone(dobj.flush(length=100)) self.assertIsNone(dobj.flush(length=100))
def test_input_types(self): def test_input_types(self):
compressed = zstd.ZstdCompressor(level=1).compress(b'foo') compressed = zstd.ZstdCompressor(level=1).compress(b"foo")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
mutable_array = bytearray(len(compressed)) mutable_array = bytearray(len(compressed))
mutable_array[:] = compressed mutable_array[:] = compressed
sources = [ sources = [
memoryview(compressed), memoryview(compressed),
bytearray(compressed), bytearray(compressed),
mutable_array, mutable_array,
] ]
for source in sources: for source in sources:
dobj = dctx.decompressobj() dobj = dctx.decompressobj()
self.assertIsNone(dobj.flush()) self.assertIsNone(dobj.flush())
self.assertIsNone(dobj.flush(10)) self.assertIsNone(dobj.flush(10))
self.assertIsNone(dobj.flush(length=100)) self.assertIsNone(dobj.flush(length=100))
self.assertEqual(dobj.decompress(source), b'foo') self.assertEqual(dobj.decompress(source), b"foo")
self.assertIsNone(dobj.flush()) self.assertIsNone(dobj.flush())
def test_reuse(self): def test_reuse(self):
data = zstd.ZstdCompressor(level=1).compress(b'foobar') data = zstd.ZstdCompressor(level=1).compress(b"foobar")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
dobj = dctx.decompressobj() dobj = dctx.decompressobj()
dobj.decompress(data) dobj.decompress(data)
with self.assertRaisesRegexp(zstd.ZstdError, 'cannot use a decompressobj'): with self.assertRaisesRegex(zstd.ZstdError, "cannot use a decompressobj"):
dobj.decompress(data) dobj.decompress(data)
self.assertIsNone(dobj.flush()) self.assertIsNone(dobj.flush())
def test_bad_write_size(self): def test_bad_write_size(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with self.assertRaisesRegexp(ValueError, 'write_size must be positive'): with self.assertRaisesRegex(ValueError, "write_size must be positive"):
dctx.decompressobj(write_size=0) dctx.decompressobj(write_size=0)
def test_write_size(self): def test_write_size(self):
source = b'foo' * 64 + b'bar' * 128 source = b"foo" * 64 + b"bar" * 128
data = zstd.ZstdCompressor(level=1).compress(source) data = zstd.ZstdCompressor(level=1).compress(source)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
for i in range(128): for i in range(128):
dobj = dctx.decompressobj(write_size=i + 1) dobj = dctx.decompressobj(write_size=i + 1)
self.assertEqual(dobj.decompress(data), source) self.assertEqual(dobj.decompress(data), source)
def decompress_via_writer(data): def decompress_via_writer(data):
buffer = io.BytesIO() buffer = io.BytesIO()
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
decompressor = dctx.stream_writer(buffer) decompressor = dctx.stream_writer(buffer)
decompressor.write(data) decompressor.write(data)
return buffer.getvalue() return buffer.getvalue()
@make_cffi @make_cffi
class TestDecompressor_stream_writer(unittest.TestCase): class TestDecompressor_stream_writer(TestCase):
def test_io_api(self): def test_io_api(self):
buffer = io.BytesIO() buffer = io.BytesIO()
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
writer = dctx.stream_writer(buffer) writer = dctx.stream_writer(buffer)
self.assertFalse(writer.closed) self.assertFalse(writer.closed)
self.assertFalse(writer.isatty()) self.assertFalse(writer.isatty())
self.assertFalse(writer.readable()) self.assertFalse(writer.readable())
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
writer.readinto(None) writer.readinto(None)
with self.assertRaises(io.UnsupportedOperation): with self.assertRaises(io.UnsupportedOperation):
writer.fileno() writer.fileno()
def test_fileno_file(self): def test_fileno_file(self):
with tempfile.TemporaryFile('wb') as tf: with tempfile.TemporaryFile("wb") as tf:
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
writer = dctx.stream_writer(tf) writer = dctx.stream_writer(tf)
self.assertEqual(writer.fileno(), tf.fileno()) self.assertEqual(writer.fileno(), tf.fileno())
def test_close(self): def test_close(self):
foo = zstd.ZstdCompressor().compress(b'foo') foo = zstd.ZstdCompressor().compress(b"foo")
buffer = NonClosingBytesIO() buffer = NonClosingBytesIO()
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
writer = dctx.stream_writer(buffer) writer = dctx.stream_writer(buffer)
writer.write(foo) writer.write(foo)
self.assertFalse(writer.closed) self.assertFalse(writer.closed)
self.assertFalse(buffer.closed) self.assertFalse(buffer.closed)
writer.close() writer.close()
self.assertTrue(writer.closed) self.assertTrue(writer.closed)
self.assertTrue(buffer.closed) self.assertTrue(buffer.closed)
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
writer.write(b'') writer.write(b"")
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
writer.flush() writer.flush()
with self.assertRaisesRegexp(ValueError, 'stream is closed'): with self.assertRaisesRegex(ValueError, "stream is closed"):
with writer: with writer:
pass pass
self.assertEqual(buffer.getvalue(), b'foo') self.assertEqual(buffer.getvalue(), b"foo")
# Context manager exit should close stream. # Context manager exit should close stream.
buffer = NonClosingBytesIO() buffer = NonClosingBytesIO()
writer = dctx.stream_writer(buffer) writer = dctx.stream_writer(buffer)
with writer: with writer:
writer.write(foo) writer.write(foo)
self.assertTrue(writer.closed) self.assertTrue(writer.closed)
self.assertEqual(buffer.getvalue(), b'foo') self.assertEqual(buffer.getvalue(), b"foo")
def test_flush(self): def test_flush(self):
buffer = OpCountingBytesIO() buffer = OpCountingBytesIO()
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
writer = dctx.stream_writer(buffer) writer = dctx.stream_writer(buffer)
writer.flush() writer.flush()
self.assertEqual(buffer._flush_count, 1) self.assertEqual(buffer._flush_count, 1)
writer.flush() writer.flush()
self.assertEqual(buffer._flush_count, 2) self.assertEqual(buffer._flush_count, 2)
def test_empty_roundtrip(self): def test_empty_roundtrip(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
empty = cctx.compress(b'') empty = cctx.compress(b"")
self.assertEqual(decompress_via_writer(empty), b'') self.assertEqual(decompress_via_writer(empty), b"")
def test_input_types(self): def test_input_types(self):
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
compressed = cctx.compress(b'foo') compressed = cctx.compress(b"foo")
mutable_array = bytearray(len(compressed)) mutable_array = bytearray(len(compressed))
mutable_array[:] = compressed mutable_array[:] = compressed
sources = [ sources = [
memoryview(compressed), memoryview(compressed),
bytearray(compressed), bytearray(compressed),
mutable_array, mutable_array,
] ]
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
for source in sources: for source in sources:
buffer = io.BytesIO() buffer = io.BytesIO()
decompressor = dctx.stream_writer(buffer) decompressor = dctx.stream_writer(buffer)
decompressor.write(source) decompressor.write(source)
self.assertEqual(buffer.getvalue(), b'foo') self.assertEqual(buffer.getvalue(), b"foo")
buffer = NonClosingBytesIO() buffer = NonClosingBytesIO()
with dctx.stream_writer(buffer) as decompressor: with dctx.stream_writer(buffer) as decompressor:
self.assertEqual(decompressor.write(source), 3) self.assertEqual(decompressor.write(source), 3)
self.assertEqual(buffer.getvalue(), b'foo') self.assertEqual(buffer.getvalue(), b"foo")
buffer = io.BytesIO() buffer = io.BytesIO()
writer = dctx.stream_writer(buffer, write_return_read=True) writer = dctx.stream_writer(buffer, write_return_read=True)
self.assertEqual(writer.write(source), len(source)) self.assertEqual(writer.write(source), len(source))
self.assertEqual(buffer.getvalue(), b'foo') self.assertEqual(buffer.getvalue(), b"foo")
def test_large_roundtrip(self): def test_large_roundtrip(self):
chunks = [] chunks = []
for i in range(255): for i in range(255):
chunks.append(struct.Struct('>B').pack(i) * 16384) chunks.append(struct.Struct(">B").pack(i) * 16384)
orig = b''.join(chunks) orig = b"".join(chunks)
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
compressed = cctx.compress(orig) compressed = cctx.compress(orig)
self.assertEqual(decompress_via_writer(compressed), orig) self.assertEqual(decompress_via_writer(compressed), orig)
def test_multiple_calls(self): def test_multiple_calls(self):
chunks = [] chunks = []
for i in range(255): for i in range(255):
for j in range(255): for j in range(255):
chunks.append(struct.Struct('>B').pack(j) * i) chunks.append(struct.Struct(">B").pack(j) * i)
orig = b''.join(chunks) orig = b"".join(chunks)
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
compressed = cctx.compress(orig) compressed = cctx.compress(orig)
buffer = NonClosingBytesIO() buffer = NonClosingBytesIO()
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_writer(buffer) as decompressor: with dctx.stream_writer(buffer) as decompressor:
pos = 0 pos = 0
while pos < len(compressed): while pos < len(compressed):
chunk = compressed[pos:pos2] chunk = compressed[pos:pos2]
self.assertEqual(writer.write(chunk), len(chunk)) self.assertEqual(writer.write(chunk), len(chunk))
pos += 8192 pos += 8192
self.assertEqual(buffer.getvalue(), orig) self.assertEqual(buffer.getvalue(), orig)
def test_dictionary(self): def test_dictionary(self):
samples = [] samples = []
for i in range(128): for i in range(128):
samples.append(b'foo' * 64) samples.append(b"foo" * 64)
samples.append(b'bar' * 64) samples.append(b"bar" * 64)
samples.append(b'foobar' * 64) samples.append(b"foobar" * 64)
d = zstd.train_dictionary(8192, samples) d = zstd.train_dictionary(8192, samples)
orig = b'foobar' * 16384 orig = b"foobar" * 16384
buffer = NonClosingBytesIO() buffer = NonClosingBytesIO()
cctx = zstd.ZstdCompressor(dict_data=d) cctx = zstd.ZstdCompressor(dict_data=d)
with cctx.stream_writer(buffer) as compressor: with cctx.stream_writer(buffer) as compressor:
self.assertEqual(compressor.write(orig), 0) self.assertEqual(compressor.write(orig), 0)
compressed = buffer.getvalue() compressed = buffer.getvalue()
buffer = io.BytesIO() buffer = io.BytesIO()
self.assertGreater(size, 100000) self.assertGreater(size, 100000)
with dctx.stream_writer(buffer) as decompressor: with dctx.stream_writer(buffer) as decompressor:
size = decompressor.memory_size() size = decompressor.memory_size()
self.assertGreater(size, 100000) self.assertGreater(size, 100000)
def test_write_size(self): def test_write_size(self):
source = zstd.ZstdCompressor().compress(b'foobarfoobar') source = zstd.ZstdCompressor().compress(b"foobarfoobar")
dest = OpCountingBytesIO() dest = OpCountingBytesIO()
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_writer(dest, write_size=1) as decompressor: with dctx.stream_writer(dest, write_size=1) as decompressor:
s = struct.Struct('>B') s = struct.Struct(">B")
for c in source: for c in source:
if not isinstance(c, str): if not isinstance(c, str):
c = s.pack(c) c = s.pack(c)
decompressor.write(c) decompressor.write(c)
self.assertEqual(dest.getvalue(), b'foobarfoobar') self.assertEqual(dest.getvalue(), b"foobarfoobar")
self.assertEqual(dest._write_count, len(dest.getvalue())) self.assertEqual(dest._write_count, len(dest.getvalue()))
@make_cffi @make_cffi
class TestDecompressor_read_to_iter(unittest.TestCase): class TestDecompressor_read_to_iter(TestCase):
def test_type_validation(self): def test_type_validation(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
# Object with read() works. # Object with read() works.
dctx.read_to_iter(io.BytesIO()) dctx.read_to_iter(io.BytesIO())
# Buffer protocol works. # Buffer protocol works.
dctx.read_to_iter(b'foobar') dctx.read_to_iter(b"foobar")
with self.assertRaisesRegexp(ValueError, 'must pass an object with a read'): with self.assertRaisesRegex(ValueError, "must pass an object with a read"):
b''.join(dctx.read_to_iter(True)) b"".join(dctx.read_to_iter(True))
def test_empty_input(self): def test_empty_input(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
source = io.BytesIO() source = io.BytesIO()
it = dctx.read_to_iter(source) it = dctx.read_to_iter(source)
# TODO this is arguably wrong. Should get an error about missing frame foo. # TODO this is arguably wrong. Should get an error about missing frame foo.
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
it = dctx.read_to_iter(b'') it = dctx.read_to_iter(b"")
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
def test_invalid_input(self): def test_invalid_input(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
source = io.BytesIO(b'foobar') source = io.BytesIO(b"foobar")
it = dctx.read_to_iter(source) it = dctx.read_to_iter(source)
with self.assertRaisesRegexp(zstd.ZstdError, 'Unknown frame descriptor'): with self.assertRaisesRegex(zstd.ZstdError, "Unknown frame descriptor"):
next(it) next(it)
it = dctx.read_to_iter(b'foobar') it = dctx.read_to_iter(b"foobar")
with self.assertRaisesRegexp(zstd.ZstdError, 'Unknown frame descriptor'): with self.assertRaisesRegex(zstd.ZstdError, "Unknown frame descriptor"):
next(it) next(it)
def test_empty_roundtrip(self): def test_empty_roundtrip(self):
cctx = zstd.ZstdCompressor(level=1, write_content_size=False) cctx = zstd.ZstdCompressor(level=1, write_content_size=False)
empty = cctx.compress(b'') empty = cctx.compress(b"")
source = io.BytesIO(empty) source = io.BytesIO(empty)
source.seek(0) source.seek(0)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
it = dctx.read_to_iter(source) it = dctx.read_to_iter(source)
# No chunks should be emitted since there is no data. # No chunks should be emitted since there is no data.
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
# Again for good measure. # Again for good measure.
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
def test_skip_bytes_too_large(self): def test_skip_bytes_too_large(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with self.assertRaisesRegexp(ValueError, 'skip_bytes must be smaller than read_size'): with self.assertRaisesRegex(
b''.join(dctx.read_to_iter(b'', skip_bytes=1, read_size=1)) ValueError, "skip_bytes must be smaller than read_size"
):
with self.assertRaisesRegexp(ValueError, 'skip_bytes larger than first input chunk'): b"".join(dctx.read_to_iter(b"", skip_bytes=1, read_size=1))
b''.join(dctx.read_to_iter(b'foobar', skip_bytes=10))
with self.assertRaisesRegex(
ValueError, "skip_bytes larger than first input chunk"
):
b"".join(dctx.read_to_iter(b"foobar", skip_bytes=10))
def test_skip_bytes(self): def test_skip_bytes(self):
cctx = zstd.ZstdCompressor(write_content_size=False) cctx = zstd.ZstdCompressor(write_content_size=False)
compressed = cctx.compress(b'foobar') compressed = cctx.compress(b"foobar")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
output = b''.join(dctx.read_to_iter(b'hdr' + compressed, skip_bytes=3)) output = b"".join(dctx.read_to_iter(b"hdr" + compressed, skip_bytes=3))
self.assertEqual(output, b'foobar') self.assertEqual(output, b"foobar")
def test_large_output(self): def test_large_output(self):
source = io.BytesIO() source = io.BytesIO()
source.write(b'f' * zstd.DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE) source.write(b"f" * zstd.DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE)
source.write(b'o') source.write(b"o")
source.seek(0) source.seek(0)
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
compressed = io.BytesIO(cctx.compress(source.getvalue())) compressed = io.BytesIO(cctx.compress(source.getvalue()))
compressed.seek(0) compressed.seek(0)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
it = dctx.read_to_iter(compressed) it = dctx.read_to_iter(compressed)
chunks = [] chunks = []
chunks.append(next(it)) chunks.append(next(it))
chunks.append(next(it)) chunks.append(next(it))
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
decompressed = b''.join(chunks) decompressed = b"".join(chunks)
self.assertEqual(decompressed, source.getvalue()) self.assertEqual(decompressed, source.getvalue())
# And again with buffer protocol. # And again with buffer protocol.
it = dctx.read_to_iter(compressed.getvalue()) it = dctx.read_to_iter(compressed.getvalue())
chunks = [] chunks = []
chunks.append(next(it)) chunks.append(next(it))
chunks.append(next(it)) chunks.append(next(it))
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
decompressed = b''.join(chunks) decompressed = b"".join(chunks)
self.assertEqual(decompressed, source.getvalue()) self.assertEqual(decompressed, source.getvalue())
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
def test_large_input(self): def test_large_input(self):
bytes = list(struct.Struct('>B').pack(i) for i in range(256)) bytes = list(struct.Struct(">B").pack(i) for i in range(256))
compressed = NonClosingBytesIO() compressed = NonClosingBytesIO()
input_size = 0 input_size = 0
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
with cctx.stream_writer(compressed) as compressor: with cctx.stream_writer(compressed) as compressor:
while True: while True:
compressor.write(random.choice(bytes)) compressor.write(random.choice(bytes))
input_size += 1 input_size += 1
have_compressed = len(compressed.getvalue()) > zstd.DECOMPRESSION_RECOMMENDED_INPUT_SIZE have_compressed = (
len(compressed.getvalue())
> zstd.DECOMPRESSION_RECOMMENDED_INPUT_SIZE
)
have_raw = input_size > zstd.DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE * 2 have_raw = input_size > zstd.DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE * 2
if have_compressed and have_raw: if have_compressed and have_raw:
break break
compressed = io.BytesIO(compressed.getvalue()) compressed = io.BytesIO(compressed.getvalue())
self.assertGreater(len(compressed.getvalue()), self.assertGreater(
zstd.DECOMPRESSION_RECOMMENDED_INPUT_SIZE) len(compressed.getvalue()), zstd.DECOMPRESSION_RECOMMENDED_INPUT_SIZE
)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
it = dctx.read_to_iter(compressed) it = dctx.read_to_iter(compressed)
chunks = [] chunks = []
chunks.append(next(it)) chunks.append(next(it))
chunks.append(next(it)) chunks.append(next(it))
chunks.append(next(it)) chunks.append(next(it))
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
decompressed = b''.join(chunks) decompressed = b"".join(chunks)
self.assertEqual(len(decompressed), input_size) self.assertEqual(len(decompressed), input_size)
# And again with buffer protocol. # And again with buffer protocol.
it = dctx.read_to_iter(compressed.getvalue()) it = dctx.read_to_iter(compressed.getvalue())
chunks = [] chunks = []
chunks.append(next(it)) chunks.append(next(it))
chunks.append(next(it)) chunks.append(next(it))
chunks.append(next(it)) chunks.append(next(it))
with self.assertRaises(StopIteration): with self.assertRaises(StopIteration):
next(it) next(it)
decompressed = b''.join(chunks) decompressed = b"".join(chunks)
self.assertEqual(len(decompressed), input_size) self.assertEqual(len(decompressed), input_size)
def test_interesting(self): def test_interesting(self):
# Found this edge case via fuzzing. # Found this edge case via fuzzing.
cctx = zstd.ZstdCompressor(level=1) cctx = zstd.ZstdCompressor(level=1)
source = io.BytesIO() source = io.BytesIO()
compressed = NonClosingBytesIO() compressed = NonClosingBytesIO()
with cctx.stream_writer(compressed) as compressor: with cctx.stream_writer(compressed) as compressor:
for i in range(256): for i in range(256):
chunk = b'\0' * 1024 chunk = b"\0" * 1024
compressor.write(chunk) compressor.write(chunk)
source.write(chunk) source.write(chunk)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
simple = dctx.decompress(compressed.getvalue(), simple = dctx.decompress(
max_output_size=len(source.getvalue())) compressed.getvalue(), max_output_size=len(source.getvalue())
)
self.assertEqual(simple, source.getvalue()) self.assertEqual(simple, source.getvalue())
compressed = io.BytesIO(compressed.getvalue()) compressed = io.BytesIO(compressed.getvalue())
streamed = b''.join(dctx.read_to_iter(compressed)) streamed = b"".join(dctx.read_to_iter(compressed))
self.assertEqual(streamed, source.getvalue()) self.assertEqual(streamed, source.getvalue())
def test_read_write_size(self): def test_read_write_size(self):
source = OpCountingBytesIO(zstd.ZstdCompressor().compress(b'foobarfoobar')) source = OpCountingBytesIO(zstd.ZstdCompressor().compress(b"foobarfoobar"))
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
for chunk in dctx.read_to_iter(source, read_size=1, write_size=1): for chunk in dctx.read_to_iter(source, read_size=1, write_size=1):
self.assertEqual(len(chunk), 1) self.assertEqual(len(chunk), 1)
self.assertEqual(source._read_count, len(source.getvalue())) self.assertEqual(source._read_count, len(source.getvalue()))
def test_magic_less(self): def test_magic_less(self):
params = zstd.CompressionParameters.from_level( params = zstd.CompressionParameters.from_level(
1, format=zstd.FORMAT_ZSTD1_MAGICLESS) 1, format=zstd.FORMAT_ZSTD1_MAGICLESS
)
cctx = zstd.ZstdCompressor(compression_params=params) cctx = zstd.ZstdCompressor(compression_params=params)
frame = cctx.compress(b'foobar') frame = cctx.compress(b"foobar")
self.assertNotEqual(frame[0:4], b'\x28\xb5\x2f\xfd') self.assertNotEqual(frame[0:4], b"\x28\xb5\x2f\xfd")
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with self.assertRaisesRegexp( with self.assertRaisesRegex(
zstd.ZstdError, 'error determining content size from frame header'): zstd.ZstdError, "error determining content size from frame header"
):
dctx.decompress(frame) dctx.decompress(frame)
dctx = zstd.ZstdDecompressor(format=zstd.FORMAT_ZSTD1_MAGICLESS) dctx = zstd.ZstdDecompressor(format=zstd.FORMAT_ZSTD1_MAGICLESS)
res = b''.join(dctx.read_to_iter(frame)) res = b"".join(dctx.read_to_iter(frame))
self.assertEqual(res, b'foobar') self.assertEqual(res, b"foobar")
@make_cffi @make_cffi
class TestDecompressor_content_dict_chain(unittest.TestCase): class TestDecompressor_content_dict_chain(TestCase):
def test_bad_inputs_simple(self): def test_bad_inputs_simple(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
dctx.decompress_content_dict_chain(b'foo') dctx.decompress_content_dict_chain(b"foo")
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
dctx.decompress_content_dict_chain((b'foo', b'bar')) dctx.decompress_content_dict_chain((b"foo", b"bar"))
with self.assertRaisesRegexp(ValueError, 'empty input chain'): with self.assertRaisesRegex(ValueError, "empty input chain"):
dctx.decompress_content_dict_chain([]) dctx.decompress_content_dict_chain([])
with self.assertRaisesRegexp(ValueError, 'chunk 0 must be bytes'): with self.assertRaisesRegex(ValueError, "chunk 0 must be bytes"):
dctx.decompress_content_dict_chain([u'foo']) dctx.decompress_content_dict_chain([u"foo"])
with self.assertRaisesRegexp(ValueError, 'chunk 0 must be bytes'): with self.assertRaisesRegex(ValueError, "chunk 0 must be bytes"):
dctx.decompress_content_dict_chain([True]) dctx.decompress_content_dict_chain([True])
with self.assertRaisesRegexp(ValueError, 'chunk 0 is too small to contain a zstd frame'): with self.assertRaisesRegex(
ValueError, "chunk 0 is too small to contain a zstd frame"
):
dctx.decompress_content_dict_chain([zstd.FRAME_HEADER]) dctx.decompress_content_dict_chain([zstd.FRAME_HEADER])
with self.assertRaisesRegexp(ValueError, 'chunk 0 is not a valid zstd frame'): with self.assertRaisesRegex(ValueError, "chunk 0 is not a valid zstd frame"):
dctx.decompress_content_dict_chain([b'foo' * 8]) dctx.decompress_content_dict_chain([b"foo" * 8])
no_size = zstd.ZstdCompressor(write_content_size=False).compress(b'foo' * 64) no_size = zstd.ZstdCompressor(write_content_size=False).compress(b"foo" * 64)
with self.assertRaisesRegexp(ValueError, 'chunk 0 missing content size in frame'): with self.assertRaisesRegex(
ValueError, "chunk 0 missing content size in frame"
):
dctx.decompress_content_dict_chain([no_size]) dctx.decompress_content_dict_chain([no_size])
# Corrupt first frame. # Corrupt first frame.
frame = zstd.ZstdCompressor().compress(b'foo' * 64) frame = zstd.ZstdCompressor().compress(b"foo" * 64)
frame = frame[0:12] + frame[15:] frame = frame[0:12] + frame[15:]
with self.assertRaisesRegexp(zstd.ZstdError, with self.assertRaisesRegex(
'chunk 0 did not decompress full frame'): zstd.ZstdError, "chunk 0 did not decompress full frame"
):
dctx.decompress_content_dict_chain([frame]) dctx.decompress_content_dict_chain([frame])
def test_bad_subsequent_input(self): def test_bad_subsequent_input(self):
initial = zstd.ZstdCompressor().compress(b'foo' * 64) initial = zstd.ZstdCompressor().compress(b"foo" * 64)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with self.assertRaisesRegexp(ValueError, 'chunk 1 must be bytes'): with self.assertRaisesRegex(ValueError, "chunk 1 must be bytes"):
dctx.decompress_content_dict_chain([initial, u'foo']) dctx.decompress_content_dict_chain([initial, u"foo"])
with self.assertRaisesRegexp(ValueError, 'chunk 1 must be bytes'): with self.assertRaisesRegex(ValueError, "chunk 1 must be bytes"):
dctx.decompress_content_dict_chain([initial, None]) dctx.decompress_content_dict_chain([initial, None])
with self.assertRaisesRegexp(ValueError, 'chunk 1 is too small to contain a zstd frame'): with self.assertRaisesRegex(
ValueError, "chunk 1 is too small to contain a zstd frame"
):
dctx.decompress_content_dict_chain([initial, zstd.FRAME_HEADER]) dctx.decompress_content_dict_chain([initial, zstd.FRAME_HEADER])
with self.assertRaisesRegexp(ValueError, 'chunk 1 is not a valid zstd frame'): with self.assertRaisesRegex(ValueError, "chunk 1 is not a valid zstd frame"):
dctx.decompress_content_dict_chain([initial, b'foo' * 8]) dctx.decompress_content_dict_chain([initial, b"foo" * 8])
no_size = zstd.ZstdCompressor(write_content_size=False).compress(b'foo' * 64) no_size = zstd.ZstdCompressor(write_content_size=False).compress(b"foo" * 64)
with self.assertRaisesRegexp(ValueError, 'chunk 1 missing content size in frame'): with self.assertRaisesRegex(
ValueError, "chunk 1 missing content size in frame"
):
dctx.decompress_content_dict_chain([initial, no_size]) dctx.decompress_content_dict_chain([initial, no_size])
# Corrupt second frame. # Corrupt second frame.
cctx = zstd.ZstdCompressor(dict_data=zstd.ZstdCompressionDict(b'foo' * 64)) cctx = zstd.ZstdCompressor(dict_data=zstd.ZstdCompressionDict(b"foo" * 64))
frame = cctx.compress(b'bar' * 64) frame = cctx.compress(b"bar" * 64)
frame = frame[0:12] + frame[15:] frame = frame[0:12] + frame[15:]
with self.assertRaisesRegexp(zstd.ZstdError, 'chunk 1 did not decompress full frame'): with self.assertRaisesRegex(
zstd.ZstdError, "chunk 1 did not decompress full frame"
):
dctx.decompress_content_dict_chain([initial, frame]) dctx.decompress_content_dict_chain([initial, frame])
def test_simple(self): def test_simple(self):
original = [ original = [
b'foo' * 64, b"foo" * 64,
b'foobar' * 64, b"foobar" * 64,
b'baz' * 64, b"baz" * 64,
b'foobaz' * 64, b"foobaz" * 64,
b'foobarbaz' * 64, b"foobarbaz" * 64,
] ]
chunks = [] chunks = []
chunks.append(zstd.ZstdCompressor().compress(original[0])) chunks.append(zstd.ZstdCompressor().compress(original[0]))
for i, chunk in enumerate(original[1:]): for i, chunk in enumerate(original[1:]):
d = zstd.ZstdCompressionDict(original[i]) d = zstd.ZstdCompressionDict(original[i])
cctx = zstd.ZstdCompressor(dict_data=d) cctx = zstd.ZstdCompressor(dict_data=d)
chunks.append(cctx.compress(chunk)) chunks.append(cctx.compress(chunk))
for i in range(1, len(original)): for i in range(1, len(original)):
chain = chunks[0:i] chain = chunks[0:i]
expected = original[i - 1] expected = original[i - 1]
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
decompressed = dctx.decompress_content_dict_chain(chain) decompressed = dctx.decompress_content_dict_chain(chain)
self.assertEqual(decompressed, expected) self.assertEqual(decompressed, expected)
# TODO enable for CFFI # TODO enable for CFFI
class TestDecompressor_multi_decompress_to_buffer(unittest.TestCase): class TestDecompressor_multi_decompress_to_buffer(TestCase):
def test_invalid_inputs(self): def test_invalid_inputs(self):
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
if not hasattr(dctx, 'multi_decompress_to_buffer'): if not hasattr(dctx, "multi_decompress_to_buffer"):
self.skipTest('multi_decompress_to_buffer not available') self.skipTest("multi_decompress_to_buffer not available")
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
dctx.multi_decompress_to_buffer(True) dctx.multi_decompress_to_buffer(True)
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
dctx.multi_decompress_to_buffer((1, 2)) dctx.multi_decompress_to_buffer((1, 2))
with self.assertRaisesRegexp(TypeError, 'item 0 not a bytes like object'): with self.assertRaisesRegex(TypeError, "item 0 not a bytes like object"):
dctx.multi_decompress_to_buffer([u'foo']) dctx.multi_decompress_to_buffer([u"foo"])
with self.assertRaisesRegexp(ValueError, 'could not determine decompressed size of item 0'): with self.assertRaisesRegex(
dctx.multi_decompress_to_buffer([b'foobarbaz']) ValueError, "could not determine decompressed size of item 0"
):
dctx.multi_decompress_to_buffer([b"foobarbaz"])
def test_list_input(self): def test_list_input(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
original = [b'foo' * 4, b'bar' * 6] original = [b"foo" * 4, b"bar" * 6]
frames = [cctx.compress(d) for d in original] frames = [cctx.compress(d) for d in original]
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
if not hasattr(dctx, 'multi_decompress_to_buffer'): if not hasattr(dctx, "multi_decompress_to_buffer"):
self.skipTest('multi_decompress_to_buffer not available') self.skipTest("multi_decompress_to_buffer not available")
result = dctx.multi_decompress_to_buffer(frames) result = dctx.multi_decompress_to_buffer(frames)
self.assertEqual(len(result), len(frames)) self.assertEqual(len(result), len(frames))
self.assertEqual(result.size(), sum(map(len, original))) self.assertEqual(result.size(), sum(map(len, original)))
for i, data in enumerate(original): for i, data in enumerate(original):
self.assertEqual(result[i].tobytes(), data) self.assertEqual(result[i].tobytes(), data)
self.assertEqual(result[0].offset, 0) self.assertEqual(result[0].offset, 0)
self.assertEqual(len(result[0]), 12) self.assertEqual(len(result[0]), 12)
self.assertEqual(result[1].offset, 12) self.assertEqual(result[1].offset, 12)
self.assertEqual(len(result[1]), 18) self.assertEqual(len(result[1]), 18)
def test_list_input_frame_sizes(self): def test_list_input_frame_sizes(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
original = [b'foo' * 4, b'bar' * 6, b'baz' * 8] original = [b"foo" * 4, b"bar" * 6, b"baz" * 8]
frames = [cctx.compress(d) for d in original] frames = [cctx.compress(d) for d in original]
sizes = struct.pack('=' + 'Q' * len(original), *map(len, original)) sizes = struct.pack("=" + "Q" * len(original), *map(len, original))
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
if not hasattr(dctx, 'multi_decompress_to_buffer'): if not hasattr(dctx, "multi_decompress_to_buffer"):
self.skipTest('multi_decompress_to_buffer not available') self.skipTest("multi_decompress_to_buffer not available")
result = dctx.multi_decompress_to_buffer(frames, decompressed_sizes=sizes) result = dctx.multi_decompress_to_buffer(frames, decompressed_sizes=sizes)
self.assertEqual(len(result), len(frames)) self.assertEqual(len(result), len(frames))
self.assertEqual(result.size(), sum(map(len, original))) self.assertEqual(result.size(), sum(map(len, original)))
for i, data in enumerate(original): for i, data in enumerate(original):
self.assertEqual(result[i].tobytes(), data) self.assertEqual(result[i].tobytes(), data)
def test_buffer_with_segments_input(self): def test_buffer_with_segments_input(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
original = [b'foo' * 4, b'bar' * 6] original = [b"foo" * 4, b"bar" * 6]
frames = [cctx.compress(d) for d in original] frames = [cctx.compress(d) for d in original]
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
if not hasattr(dctx, 'multi_decompress_to_buffer'): if not hasattr(dctx, "multi_decompress_to_buffer"):
self.skipTest('multi_decompress_to_buffer not available') self.skipTest("multi_decompress_to_buffer not available")
segments = struct.pack('=QQQQ', 0, len(frames[0]), len(frames[0]), len(frames[1])) segments = struct.pack(
b = zstd.BufferWithSegments(b''.join(frames), segments) "=QQQQ", 0, len(frames[0]), len(frames[0]), len(frames[1])
)
b = zstd.BufferWithSegments(b"".join(frames), segments)
result = dctx.multi_decompress_to_buffer(b) result = dctx.multi_decompress_to_buffer(b)
self.assertEqual(len(result), len(frames)) self.assertEqual(len(result), len(frames))
self.assertEqual(result[0].offset, 0) self.assertEqual(result[0].offset, 0)
self.assertEqual(len(result[0]), 12) self.assertEqual(len(result[0]), 12)
self.assertEqual(result[1].offset, 12) self.assertEqual(result[1].offset, 12)
self.assertEqual(len(result[1]), 18) self.assertEqual(len(result[1]), 18)
def test_buffer_with_segments_sizes(self): def test_buffer_with_segments_sizes(self):
cctx = zstd.ZstdCompressor(write_content_size=False) cctx = zstd.ZstdCompressor(write_content_size=False)
original = [b'foo' * 4, b'bar' * 6, b'baz' * 8] original = [b"foo" * 4, b"bar" * 6, b"baz" * 8]
frames = [cctx.compress(d) for d in original] frames = [cctx.compress(d) for d in original]
sizes = struct.pack('=' + 'Q' * len(original), *map(len, original)) sizes = struct.pack("=" + "Q" * len(original), *map(len, original))
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
if not hasattr(dctx, 'multi_decompress_to_buffer'): if not hasattr(dctx, "multi_decompress_to_buffer"):
self.skipTest('multi_decompress_to_buffer not available') self.skipTest("multi_decompress_to_buffer not available")
segments = struct.pack('=QQQQQQ', 0, len(frames[0]), segments = struct.pack(
len(frames[0]), len(frames[1]), "=QQQQQQ",
len(frames[0]) + len(frames[1]), len(frames[2])) 0,
b = zstd.BufferWithSegments(b''.join(frames), segments) len(frames[0]),
len(frames[0]),
len(frames[1]),
len(frames[0]) + len(frames[1]),
len(frames[2]),
)
b = zstd.BufferWithSegments(b"".join(frames), segments)
result = dctx.multi_decompress_to_buffer(b, decompressed_sizes=sizes) result = dctx.multi_decompress_to_buffer(b, decompressed_sizes=sizes)
self.assertEqual(len(result), len(frames)) self.assertEqual(len(result), len(frames))
self.assertEqual(result.size(), sum(map(len, original))) self.assertEqual(result.size(), sum(map(len, original)))
for i, data in enumerate(original): for i, data in enumerate(original):
self.assertEqual(result[i].tobytes(), data) self.assertEqual(result[i].tobytes(), data)
def test_buffer_with_segments_collection_input(self): def test_buffer_with_segments_collection_input(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
original = [ original = [
b'foo0' * 2, b"foo0" * 2,
b'foo1' * 3, b"foo1" * 3,
b'foo2' * 4, b"foo2" * 4,
b'foo3' * 5, b"foo3" * 5,
b'foo4' * 6, b"foo4" * 6,
] ]
if not hasattr(cctx, 'multi_compress_to_buffer'): if not hasattr(cctx, "multi_compress_to_buffer"):
self.skipTest('multi_compress_to_buffer not available') self.skipTest("multi_compress_to_buffer not available")
frames = cctx.multi_compress_to_buffer(original) frames = cctx.multi_compress_to_buffer(original)
# Check round trip. # Check round trip.
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
decompressed = dctx.multi_decompress_to_buffer(frames, threads=3) decompressed = dctx.multi_decompress_to_buffer(frames, threads=3)
self.assertEqual(len(decompressed), len(original)) self.assertEqual(len(decompressed), len(original))
for i, data in enumerate(original): for i, data in enumerate(original):
self.assertEqual(data, decompressed[i].tobytes()) self.assertEqual(data, decompressed[i].tobytes())
# And a manual mode. # And a manual mode.
b = b''.join([frames[0].tobytes(), frames[1].tobytes()]) b = b"".join([frames[0].tobytes(), frames[1].tobytes()])
b1 = zstd.BufferWithSegments(b, struct.pack('=QQQQ', b1 = zstd.BufferWithSegments(
0, len(frames[0]), b, struct.pack("=QQQQ", 0, len(frames[0]), len(frames[0]), len(frames[1]))
len(frames[0]), len(frames[1]))) )
b = b''.join([frames[2].tobytes(), frames[3].tobytes(), frames[4].tobytes()]) b = b"".join([frames[2].tobytes(), frames[3].tobytes(), frames[4].tobytes()])
b2 = zstd.BufferWithSegments(b, struct.pack('=QQQQQQ', b2 = zstd.BufferWithSegments(
0, len(frames[2]), b,
len(frames[2]), len(frames[3]), struct.pack(
len(frames[2]) + len(frames[3]), len(frames[4]))) "=QQQQQQ",
0,
len(frames[2]),
len(frames[2]),
len(frames[3]),
len(frames[2]) + len(frames[3]),
len(frames[4]),
),
)
c = zstd.BufferWithSegmentsCollection(b1, b2) c = zstd.BufferWithSegmentsCollection(b1, b2)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
decompressed = dctx.multi_decompress_to_buffer(c) decompressed = dctx.multi_decompress_to_buffer(c)
self.assertEqual(len(decompressed), 5) self.assertEqual(len(decompressed), 5)
for i in range(5): for i in range(5):
self.assertEqual(decompressed[i].tobytes(), original[i]) self.assertEqual(decompressed[i].tobytes(), original[i])
def test_dict(self): def test_dict(self):
d = zstd.train_dictionary(16384, generate_samples(), k=64, d=16) d = zstd.train_dictionary(16384, generate_samples(), k=64, d=16)
cctx = zstd.ZstdCompressor(dict_data=d, level=1) cctx = zstd.ZstdCompressor(dict_data=d, level=1)
frames = [cctx.compress(s) for s in generate_samples()] frames = [cctx.compress(s) for s in generate_samples()]
dctx = zstd.ZstdDecompressor(dict_data=d) dctx = zstd.ZstdDecompressor(dict_data=d)
if not hasattr(dctx, 'multi_decompress_to_buffer'): if not hasattr(dctx, "multi_decompress_to_buffer"):
self.skipTest('multi_decompress_to_buffer not available') self.skipTest("multi_decompress_to_buffer not available")
result = dctx.multi_decompress_to_buffer(frames) result = dctx.multi_decompress_to_buffer(frames)
self.assertEqual([o.tobytes() for o in result], generate_samples()) self.assertEqual([o.tobytes() for o in result], generate_samples())
def test_multiple_threads(self): def test_multiple_threads(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
frames = [] frames = []
frames.extend(cctx.compress(b'x' * 64) for i in range(256)) frames.extend(cctx.compress(b"x" * 64) for i in range(256))
frames.extend(cctx.compress(b'y' * 64) for i in range(256)) frames.extend(cctx.compress(b"y" * 64) for i in range(256))
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
if not hasattr(dctx, 'multi_decompress_to_buffer'): if not hasattr(dctx, "multi_decompress_to_buffer"):
self.skipTest('multi_decompress_to_buffer not available') self.skipTest("multi_decompress_to_buffer not available")
result = dctx.multi_decompress_to_buffer(frames, threads=-1) result = dctx.multi_decompress_to_buffer(frames, threads=-1)
self.assertEqual(len(result), len(frames)) self.assertEqual(len(result), len(frames))
self.assertEqual(result.size(), 2 * 64 * 256) self.assertEqual(result.size(), 2 * 64 * 256)
self.assertEqual(result[0].tobytes(), b'x' * 64) self.assertEqual(result[0].tobytes(), b"x" * 64)
self.assertEqual(result[256].tobytes(), b'y' * 64) self.assertEqual(result[256].tobytes(), b"y" * 64)
def test_item_failure(self): def test_item_failure(self):
cctx = zstd.ZstdCompressor() cctx = zstd.ZstdCompressor()
frames = [cctx.compress(b'x' * 128), cctx.compress(b'y' * 128)] frames = [cctx.compress(b"x" * 128), cctx.compress(b"y" * 128)]
frames[1] = frames[1][0:15] + b'extra' + frames[1][15:] frames[1] = frames[1][0:15] + b"extra" + frames[1][15:]
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
if not hasattr(dctx, 'multi_decompress_to_buffer'): if not hasattr(dctx, "multi_decompress_to_buffer"):
self.skipTest('multi_decompress_to_buffer not available') self.skipTest("multi_decompress_to_buffer not available")
with self.assertRaisesRegexp(zstd.ZstdError, with self.assertRaisesRegex(
'error decompressing item 1: (' zstd.ZstdError,
'Corrupted block|' "error decompressing item 1: ("
'Destination buffer is too small)'): "Corrupted block|"
"Destination buffer is too small)",
):
dctx.multi_decompress_to_buffer(frames) dctx.multi_decompress_to_buffer(frames)
with self.assertRaisesRegexp(zstd.ZstdError, with self.assertRaisesRegex(
'error decompressing item 1: (' zstd.ZstdError,
'Corrupted block|' "error decompressing item 1: ("
'Destination buffer is too small)'): "Corrupted block|"
"Destination buffer is too small)",
):
dctx.multi_decompress_to_buffer(frames, threads=2) dctx.multi_decompress_to_buffer(frames, threads=2)

contrib/python-zstandard/tests/test_decompressor_fuzzing.py

import io import io
import os import os
import unittest import unittest
try: try:
import hypothesis import hypothesis
import hypothesis.strategies as strategies import hypothesis.strategies as strategies
except ImportError: except ImportError:
raise unittest.SkipTest('hypothesis not available') raise unittest.SkipTest("hypothesis not available")
import zstandard as zstd import zstandard as zstd
from . common import ( from .common import (
make_cffi, make_cffi,
NonClosingBytesIO, NonClosingBytesIO,
random_input_data, random_input_data,
TestCase,
) )
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
@make_cffi @make_cffi
class TestDecompressor_stream_reader_fuzzing(unittest.TestCase): class TestDecompressor_stream_reader_fuzzing(TestCase):
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
streaming=strategies.booleans(), streaming=strategies.booleans(),
source_read_size=strategies.integers(1, 1048576), source_read_size=strategies.integers(1, 1048576),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_stream_source_read_variance(self, original, level, streaming, )
source_read_size, read_sizes): def test_stream_source_read_variance(
self, original, level, streaming, source_read_size, read_sizes
):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
if streaming: if streaming:
source = io.BytesIO() source = io.BytesIO()
writer = cctx.stream_writer(source) writer = cctx.stream_writer(source)
writer.write(original) writer.write(original)
writer.flush(zstd.FLUSH_FRAME) writer.flush(zstd.FLUSH_FRAME)
source.seek(0) source.seek(0)
else: else:
frame = cctx.compress(original) frame = cctx.compress(original)
source = io.BytesIO(frame) source = io.BytesIO(frame)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
chunks = [] chunks = []
with dctx.stream_reader(source, read_size=source_read_size) as reader: with dctx.stream_reader(source, read_size=source_read_size) as reader:
while True: while True:
read_size = read_sizes.draw(strategies.integers(-1, 131072)) read_size = read_sizes.draw(strategies.integers(-1, 131072))
chunk = reader.read(read_size) chunk = reader.read(read_size)
if not chunk and read_size: if not chunk and read_size:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), original) self.assertEqual(b"".join(chunks), original)
# Similar to above except we have a constant read() size. # Similar to above except we have a constant read() size.
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[hypothesis.HealthCheck.large_base_example]
@hypothesis.given(original=strategies.sampled_from(random_input_data()), )
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
streaming=strategies.booleans(), streaming=strategies.booleans(),
source_read_size=strategies.integers(1, 1048576), source_read_size=strategies.integers(1, 1048576),
read_size=strategies.integers(-1, 131072)) read_size=strategies.integers(-1, 131072),
def test_stream_source_read_size(self, original, level, streaming, )
source_read_size, read_size): def test_stream_source_read_size(
self, original, level, streaming, source_read_size, read_size
):
if read_size == 0: if read_size == 0:
read_size = 1 read_size = 1
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
if streaming: if streaming:
source = io.BytesIO() source = io.BytesIO()
writer = cctx.stream_writer(source) writer = cctx.stream_writer(source)
reader = dctx.stream_reader(source, read_size=source_read_size) reader = dctx.stream_reader(source, read_size=source_read_size)
while True: while True:
chunk = reader.read(read_size) chunk = reader.read(read_size)
if not chunk and read_size: if not chunk and read_size:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), original) self.assertEqual(b"".join(chunks), original)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
streaming=strategies.booleans(), streaming=strategies.booleans(),
source_read_size=strategies.integers(1, 1048576), source_read_size=strategies.integers(1, 1048576),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_buffer_source_read_variance(self, original, level, streaming, )
source_read_size, read_sizes): def test_buffer_source_read_variance(
self, original, level, streaming, source_read_size, read_sizes
):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
if streaming: if streaming:
source = io.BytesIO() source = io.BytesIO()
writer = cctx.stream_writer(source) writer = cctx.stream_writer(source)
writer.write(original) writer.write(original)
writer.flush(zstd.FLUSH_FRAME) writer.flush(zstd.FLUSH_FRAME)
frame = source.getvalue() frame = source.getvalue()
else: else:
frame = cctx.compress(original) frame = cctx.compress(original)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
chunks = [] chunks = []
with dctx.stream_reader(frame, read_size=source_read_size) as reader: with dctx.stream_reader(frame, read_size=source_read_size) as reader:
while True: while True:
read_size = read_sizes.draw(strategies.integers(-1, 131072)) read_size = read_sizes.draw(strategies.integers(-1, 131072))
chunk = reader.read(read_size) chunk = reader.read(read_size)
if not chunk and read_size: if not chunk and read_size:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), original) self.assertEqual(b"".join(chunks), original)
# Similar to above except we have a constant read() size. # Similar to above except we have a constant read() size.
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[hypothesis.HealthCheck.large_base_example]
@hypothesis.given(original=strategies.sampled_from(random_input_data()), )
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
streaming=strategies.booleans(), streaming=strategies.booleans(),
source_read_size=strategies.integers(1, 1048576), source_read_size=strategies.integers(1, 1048576),
read_size=strategies.integers(-1, 131072)) read_size=strategies.integers(-1, 131072),
def test_buffer_source_constant_read_size(self, original, level, streaming, )
source_read_size, read_size): def test_buffer_source_constant_read_size(
self, original, level, streaming, source_read_size, read_size
):
if read_size == 0: if read_size == 0:
read_size = -1 read_size = -1
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
if streaming: if streaming:
source = io.BytesIO() source = io.BytesIO()
writer = cctx.stream_writer(source) writer = cctx.stream_writer(source)
reader = dctx.stream_reader(frame, read_size=source_read_size) reader = dctx.stream_reader(frame, read_size=source_read_size)
while True: while True:
chunk = reader.read(read_size) chunk = reader.read(read_size)
if not chunk and read_size: if not chunk and read_size:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), original) self.assertEqual(b"".join(chunks), original)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[hypothesis.HealthCheck.large_base_example]
@hypothesis.given(original=strategies.sampled_from(random_input_data()), )
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
streaming=strategies.booleans(), streaming=strategies.booleans(),
source_read_size=strategies.integers(1, 1048576)) source_read_size=strategies.integers(1, 1048576),
def test_stream_source_readall(self, original, level, streaming, )
source_read_size): def test_stream_source_readall(self, original, level, streaming, source_read_size):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
if streaming: if streaming:
source = io.BytesIO() source = io.BytesIO()
writer = cctx.stream_writer(source) writer = cctx.stream_writer(source)
writer.write(original) writer.write(original)
writer.flush(zstd.FLUSH_FRAME) writer.flush(zstd.FLUSH_FRAME)
source.seek(0) source.seek(0)
else: else:
frame = cctx.compress(original) frame = cctx.compress(original)
source = io.BytesIO(frame) source = io.BytesIO(frame)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
data = dctx.stream_reader(source, read_size=source_read_size).readall() data = dctx.stream_reader(source, read_size=source_read_size).readall()
self.assertEqual(data, original) self.assertEqual(data, original)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
streaming=strategies.booleans(), streaming=strategies.booleans(),
source_read_size=strategies.integers(1, 1048576), source_read_size=strategies.integers(1, 1048576),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_stream_source_read1_variance(self, original, level, streaming, )
source_read_size, read_sizes): def test_stream_source_read1_variance(
self, original, level, streaming, source_read_size, read_sizes
):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
if streaming: if streaming:
source = io.BytesIO() source = io.BytesIO()
writer = cctx.stream_writer(source) writer = cctx.stream_writer(source)
writer.write(original) writer.write(original)
writer.flush(zstd.FLUSH_FRAME) writer.flush(zstd.FLUSH_FRAME)
source.seek(0) source.seek(0)
else: else:
frame = cctx.compress(original) frame = cctx.compress(original)
source = io.BytesIO(frame) source = io.BytesIO(frame)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
chunks = [] chunks = []
with dctx.stream_reader(source, read_size=source_read_size) as reader: with dctx.stream_reader(source, read_size=source_read_size) as reader:
while True: while True:
read_size = read_sizes.draw(strategies.integers(-1, 131072)) read_size = read_sizes.draw(strategies.integers(-1, 131072))
chunk = reader.read1(read_size) chunk = reader.read1(read_size)
if not chunk and read_size: if not chunk and read_size:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(b''.join(chunks), original) self.assertEqual(b"".join(chunks), original)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
@hypothesis.given(original=strategies.sampled_from(random_input_data()), hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
streaming=strategies.booleans(), streaming=strategies.booleans(),
source_read_size=strategies.integers(1, 1048576), source_read_size=strategies.integers(1, 1048576),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_stream_source_readinto1_variance(self, original, level, streaming, )
source_read_size, read_sizes): def test_stream_source_readinto1_variance(
self, original, level, streaming, source_read_size, read_sizes
):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
if streaming: if streaming:
source = io.BytesIO() source = io.BytesIO()
writer = cctx.stream_writer(source) writer = cctx.stream_writer(source)
writer.write(original) writer.write(original)
writer.flush(zstd.FLUSH_FRAME) writer.flush(zstd.FLUSH_FRAME)
source.seek(0) source.seek(0)
b = bytearray(read_size) b = bytearray(read_size)
count = reader.readinto1(b) count = reader.readinto1(b)
if not count: if not count:
break break
chunks.append(bytes(b[0:count])) chunks.append(bytes(b[0:count]))
self.assertEqual(b''.join(chunks), original) self.assertEqual(b"".join(chunks), original)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given( @hypothesis.given(
original=strategies.sampled_from(random_input_data()), original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 1048576), source_read_size=strategies.integers(1, 1048576),
seek_amounts=strategies.data(), seek_amounts=strategies.data(),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_relative_seeks(self, original, level, source_read_size, seek_amounts, )
read_sizes): def test_relative_seeks(
self, original, level, source_read_size, seek_amounts, read_sizes
):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
frame = cctx.compress(original) frame = cctx.compress(original)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
with dctx.stream_reader(frame, read_size=source_read_size) as reader: with dctx.stream_reader(frame, read_size=source_read_size) as reader:
while True: while True:
amount = seek_amounts.draw(strategies.integers(0, 16384)) amount = seek_amounts.draw(strategies.integers(0, 16384))
reader.seek(amount, os.SEEK_CUR) reader.seek(amount, os.SEEK_CUR)
offset = reader.tell() offset = reader.tell()
read_amount = read_sizes.draw(strategies.integers(1, 16384)) read_amount = read_sizes.draw(strategies.integers(1, 16384))
chunk = reader.read(read_amount) chunk = reader.read(read_amount)
if not chunk: if not chunk:
break break
self.assertEqual(original[offset:offset + len(chunk)], chunk) self.assertEqual(original[offset : offset + len(chunk)], chunk)
@hypothesis.settings( @hypothesis.settings(
suppress_health_check=[hypothesis.HealthCheck.large_base_example]) suppress_health_check=[
hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given( @hypothesis.given(
originals=strategies.data(), originals=strategies.data(),
frame_count=strategies.integers(min_value=2, max_value=10), frame_count=strategies.integers(min_value=2, max_value=10),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
source_read_size=strategies.integers(1, 1048576), source_read_size=strategies.integers(1, 1048576),
read_sizes=strategies.data()) read_sizes=strategies.data(),
def test_multiple_frames(self, originals, frame_count, level, )
source_read_size, read_sizes): def test_multiple_frames(
self, originals, frame_count, level, source_read_size, read_sizes
):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
source = io.BytesIO() source = io.BytesIO()
buffer = io.BytesIO() buffer = io.BytesIO()
writer = cctx.stream_writer(buffer) writer = cctx.stream_writer(buffer)
for i in range(frame_count): for i in range(frame_count):
data = originals.draw(strategies.sampled_from(random_input_data())) data = originals.draw(strategies.sampled_from(random_input_data()))
source.write(data) source.write(data)
writer.write(data) writer.write(data)
writer.flush(zstd.FLUSH_FRAME) writer.flush(zstd.FLUSH_FRAME)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
buffer.seek(0) buffer.seek(0)
reader = dctx.stream_reader(buffer, read_size=source_read_size, reader = dctx.stream_reader(
read_across_frames=True) buffer, read_size=source_read_size, read_across_frames=True
)
chunks = [] chunks = []
while True: while True:
read_amount = read_sizes.draw(strategies.integers(-1, 16384)) read_amount = read_sizes.draw(strategies.integers(-1, 16384))
chunk = reader.read(read_amount) chunk = reader.read(read_amount)
if not chunk and read_amount: if not chunk and read_amount:
break break
chunks.append(chunk) chunks.append(chunk)
self.assertEqual(source.getvalue(), b''.join(chunks)) self.assertEqual(source.getvalue(), b"".join(chunks))
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
@make_cffi @make_cffi
class TestDecompressor_stream_writer_fuzzing(unittest.TestCase): class TestDecompressor_stream_writer_fuzzing(TestCase):
@hypothesis.given(original=strategies.sampled_from(random_input_data()), @hypothesis.settings(
suppress_health_check=[
hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
write_size=strategies.integers(min_value=1, max_value=8192), write_size=strategies.integers(min_value=1, max_value=8192),
input_sizes=strategies.data()) input_sizes=strategies.data(),
)
def test_write_size_variance(self, original, level, write_size, input_sizes): def test_write_size_variance(self, original, level, write_size, input_sizes):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
frame = cctx.compress(original) frame = cctx.compress(original)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
source = io.BytesIO(frame) source = io.BytesIO(frame)
dest = NonClosingBytesIO() dest = NonClosingBytesIO()
with dctx.stream_writer(dest, write_size=write_size) as decompressor: with dctx.stream_writer(dest, write_size=write_size) as decompressor:
while True: while True:
input_size = input_sizes.draw(strategies.integers(1, 4096)) input_size = input_sizes.draw(strategies.integers(1, 4096))
chunk = source.read(input_size) chunk = source.read(input_size)
if not chunk: if not chunk:
break break
decompressor.write(chunk) decompressor.write(chunk)
self.assertEqual(dest.getvalue(), original) self.assertEqual(dest.getvalue(), original)
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
@make_cffi @make_cffi
class TestDecompressor_copy_stream_fuzzing(unittest.TestCase): class TestDecompressor_copy_stream_fuzzing(TestCase):
@hypothesis.given(original=strategies.sampled_from(random_input_data()), @hypothesis.settings(
suppress_health_check=[
hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
read_size=strategies.integers(min_value=1, max_value=8192), read_size=strategies.integers(min_value=1, max_value=8192),
write_size=strategies.integers(min_value=1, max_value=8192)) write_size=strategies.integers(min_value=1, max_value=8192),
)
def test_read_write_size_variance(self, original, level, read_size, write_size): def test_read_write_size_variance(self, original, level, read_size, write_size):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
frame = cctx.compress(original) frame = cctx.compress(original)
source = io.BytesIO(frame) source = io.BytesIO(frame)
dest = io.BytesIO() dest = io.BytesIO()
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
dctx.copy_stream(source, dest, read_size=read_size, write_size=write_size) dctx.copy_stream(source, dest, read_size=read_size, write_size=write_size)
self.assertEqual(dest.getvalue(), original) self.assertEqual(dest.getvalue(), original)
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
@make_cffi @make_cffi
class TestDecompressor_decompressobj_fuzzing(unittest.TestCase): class TestDecompressor_decompressobj_fuzzing(TestCase):
@hypothesis.given(original=strategies.sampled_from(random_input_data()), @hypothesis.settings(
suppress_health_check=[
hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
chunk_sizes=strategies.data()) chunk_sizes=strategies.data(),
)
def test_random_input_sizes(self, original, level, chunk_sizes): def test_random_input_sizes(self, original, level, chunk_sizes):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
frame = cctx.compress(original) frame = cctx.compress(original)
source = io.BytesIO(frame) source = io.BytesIO(frame)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
dobj = dctx.decompressobj() dobj = dctx.decompressobj()
chunks = [] chunks = []
while True: while True:
chunk_size = chunk_sizes.draw(strategies.integers(1, 4096)) chunk_size = chunk_sizes.draw(strategies.integers(1, 4096))
chunk = source.read(chunk_size) chunk = source.read(chunk_size)
if not chunk: if not chunk:
break break
chunks.append(dobj.decompress(chunk)) chunks.append(dobj.decompress(chunk))
self.assertEqual(b''.join(chunks), original) self.assertEqual(b"".join(chunks), original)
@hypothesis.given(original=strategies.sampled_from(random_input_data()), @hypothesis.settings(
suppress_health_check=[
hypothesis.HealthCheck.large_base_example,
hypothesis.HealthCheck.too_slow,
]
)
@hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
write_size=strategies.integers(min_value=1, write_size=strategies.integers(
max_value=4 * zstd.DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE), min_value=1, max_value=4 * zstd.DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE
chunk_sizes=strategies.data()) ),
chunk_sizes=strategies.data(),
)
def test_random_output_sizes(self, original, level, write_size, chunk_sizes): def test_random_output_sizes(self, original, level, write_size, chunk_sizes):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
frame = cctx.compress(original) frame = cctx.compress(original)
source = io.BytesIO(frame) source = io.BytesIO(frame)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
dobj = dctx.decompressobj(write_size=write_size) dobj = dctx.decompressobj(write_size=write_size)
chunks = [] chunks = []
while True: while True:
chunk_size = chunk_sizes.draw(strategies.integers(1, 4096)) chunk_size = chunk_sizes.draw(strategies.integers(1, 4096))
chunk = source.read(chunk_size) chunk = source.read(chunk_size)
if not chunk: if not chunk:
break break
chunks.append(dobj.decompress(chunk)) chunks.append(dobj.decompress(chunk))
self.assertEqual(b''.join(chunks), original) self.assertEqual(b"".join(chunks), original)
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
@make_cffi @make_cffi
class TestDecompressor_read_to_iter_fuzzing(unittest.TestCase): class TestDecompressor_read_to_iter_fuzzing(TestCase):
@hypothesis.given(original=strategies.sampled_from(random_input_data()), @hypothesis.given(
original=strategies.sampled_from(random_input_data()),
level=strategies.integers(min_value=1, max_value=5), level=strategies.integers(min_value=1, max_value=5),
read_size=strategies.integers(min_value=1, max_value=4096), read_size=strategies.integers(min_value=1, max_value=4096),
write_size=strategies.integers(min_value=1, max_value=4096)) write_size=strategies.integers(min_value=1, max_value=4096),
)
def test_read_write_size_variance(self, original, level, read_size, write_size): def test_read_write_size_variance(self, original, level, read_size, write_size):
cctx = zstd.ZstdCompressor(level=level) cctx = zstd.ZstdCompressor(level=level)
frame = cctx.compress(original) frame = cctx.compress(original)
source = io.BytesIO(frame) source = io.BytesIO(frame)
dctx = zstd.ZstdDecompressor() dctx = zstd.ZstdDecompressor()
chunks = list(dctx.read_to_iter(source, read_size=read_size, write_size=write_size)) chunks = list(
dctx.read_to_iter(source, read_size=read_size, write_size=write_size)
)
self.assertEqual(b''.join(chunks), original) self.assertEqual(b"".join(chunks), original)
@unittest.skipUnless('ZSTD_SLOW_TESTS' in os.environ, 'ZSTD_SLOW_TESTS not set') @unittest.skipUnless("ZSTD_SLOW_TESTS" in os.environ, "ZSTD_SLOW_TESTS not set")
class TestDecompressor_multi_decompress_to_buffer_fuzzing(unittest.TestCase): class TestDecompressor_multi_decompress_to_buffer_fuzzing(TestCase):
@hypothesis.given(original=strategies.lists(strategies.sampled_from(random_input_data()), @hypothesis.given(
min_size=1, max_size=1024), original=strategies.lists(
strategies.sampled_from(random_input_data()), min_size=1, max_size=1024
),
threads=strategies.integers(min_value=1, max_value=8), threads=strategies.integers(min_value=1, max_value=8),
use_dict=strategies.booleans()) use_dict=strategies.booleans(),
)
def test_data_equivalence(self, original, threads, use_dict): def test_data_equivalence(self, original, threads, use_dict):
kwargs = {} kwargs = {}
if use_dict: if use_dict:
kwargs['dict_data'] = zstd.ZstdCompressionDict(original[0]) kwargs["dict_data"] = zstd.ZstdCompressionDict(original[0])
cctx = zstd.ZstdCompressor(level=1, cctx = zstd.ZstdCompressor(
write_content_size=True, level=1, write_content_size=True, write_checksum=True, **kwargs
write_checksum=True, )
**kwargs)
if not hasattr(cctx, 'multi_compress_to_buffer'): if not hasattr(cctx, "multi_compress_to_buffer"):
self.skipTest('multi_compress_to_buffer not available') self.skipTest("multi_compress_to_buffer not available")
frames_buffer = cctx.multi_compress_to_buffer(original, threads=-1) frames_buffer = cctx.multi_compress_to_buffer(original, threads=-1)
dctx = zstd.ZstdDecompressor(**kwargs) dctx = zstd.ZstdDecompressor(**kwargs)
result = dctx.multi_decompress_to_buffer(frames_buffer) result = dctx.multi_decompress_to_buffer(frames_buffer)
self.assertEqual(len(result), len(original)) self.assertEqual(len(result), len(original))
for i, frame in enumerate(result): for i, frame in enumerate(result):
self.assertEqual(frame.tobytes(), original[i]) self.assertEqual(frame.tobytes(), original[i])
frames_list = [f.tobytes() for f in frames_buffer] frames_list = [f.tobytes() for f in frames_buffer]
result = dctx.multi_decompress_to_buffer(frames_list) result = dctx.multi_decompress_to_buffer(frames_list)
self.assertEqual(len(result), len(original)) self.assertEqual(len(result), len(original))
for i, frame in enumerate(result): for i, frame in enumerate(result):
self.assertEqual(frame.tobytes(), original[i]) self.assertEqual(frame.tobytes(), original[i])

contrib/python-zstandard/tests/test_estimate_sizes.py

import unittest import unittest
import zstandard as zstd import zstandard as zstd
from . common import ( from .common import (
make_cffi, make_cffi,
TestCase,
) )
@make_cffi @make_cffi
class TestSizes(unittest.TestCase): class TestSizes(TestCase):
def test_decompression_size(self): def test_decompression_size(self):
size = zstd.estimate_decompression_context_size() size = zstd.estimate_decompression_context_size()
self.assertGreater(size, 100000) self.assertGreater(size, 100000)

contrib/python-zstandard/tests/test_module_attributes.py

from __future__ import unicode_literals from __future__ import unicode_literals
import unittest import unittest
import zstandard as zstd import zstandard as zstd
from . common import ( from .common import (
make_cffi, make_cffi,
TestCase,
) )
@make_cffi @make_cffi
class TestModuleAttributes(unittest.TestCase): class TestModuleAttributes(TestCase):
def test_version(self): def test_version(self):
self.assertEqual(zstd.ZSTD_VERSION, (1, 4, 3)) self.assertEqual(zstd.ZSTD_VERSION, (1, 4, 4))
self.assertEqual(zstd.__version__, '0.12.0') self.assertEqual(zstd.__version__, "0.13.0")
def test_constants(self): def test_constants(self):
self.assertEqual(zstd.MAX_COMPRESSION_LEVEL, 22) self.assertEqual(zstd.MAX_COMPRESSION_LEVEL, 22)
self.assertEqual(zstd.FRAME_HEADER, b'\x28\xb5\x2f\xfd') self.assertEqual(zstd.FRAME_HEADER, b"\x28\xb5\x2f\xfd")
def test_hasattr(self): def test_hasattr(self):
attrs = ( attrs = (
'CONTENTSIZE_UNKNOWN', "CONTENTSIZE_UNKNOWN",
'CONTENTSIZE_ERROR', "CONTENTSIZE_ERROR",
'COMPRESSION_RECOMMENDED_INPUT_SIZE', "COMPRESSION_RECOMMENDED_INPUT_SIZE",
'COMPRESSION_RECOMMENDED_OUTPUT_SIZE', "COMPRESSION_RECOMMENDED_OUTPUT_SIZE",
'DECOMPRESSION_RECOMMENDED_INPUT_SIZE', "DECOMPRESSION_RECOMMENDED_INPUT_SIZE",
'DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE', "DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE",
'MAGIC_NUMBER', "MAGIC_NUMBER",
'FLUSH_BLOCK', "FLUSH_BLOCK",
'FLUSH_FRAME', "FLUSH_FRAME",
'BLOCKSIZELOG_MAX', "BLOCKSIZELOG_MAX",
'BLOCKSIZE_MAX', "BLOCKSIZE_MAX",
'WINDOWLOG_MIN', "WINDOWLOG_MIN",
'WINDOWLOG_MAX', "WINDOWLOG_MAX",
'CHAINLOG_MIN', "CHAINLOG_MIN",
'CHAINLOG_MAX', "CHAINLOG_MAX",
'HASHLOG_MIN', "HASHLOG_MIN",
'HASHLOG_MAX', "HASHLOG_MAX",
'HASHLOG3_MAX', "HASHLOG3_MAX",
'MINMATCH_MIN', "MINMATCH_MIN",
'MINMATCH_MAX', "MINMATCH_MAX",
'SEARCHLOG_MIN', "SEARCHLOG_MIN",
'SEARCHLOG_MAX', "SEARCHLOG_MAX",
'SEARCHLENGTH_MIN', "SEARCHLENGTH_MIN",
'SEARCHLENGTH_MAX', "SEARCHLENGTH_MAX",
'TARGETLENGTH_MIN', "TARGETLENGTH_MIN",
'TARGETLENGTH_MAX', "TARGETLENGTH_MAX",
'LDM_MINMATCH_MIN', "LDM_MINMATCH_MIN",
'LDM_MINMATCH_MAX', "LDM_MINMATCH_MAX",
'LDM_BUCKETSIZELOG_MAX', "LDM_BUCKETSIZELOG_MAX",
'STRATEGY_FAST', "STRATEGY_FAST",
'STRATEGY_DFAST', "STRATEGY_DFAST",
'STRATEGY_GREEDY', "STRATEGY_GREEDY",
'STRATEGY_LAZY', "STRATEGY_LAZY",
'STRATEGY_LAZY2', "STRATEGY_LAZY2",
'STRATEGY_BTLAZY2', "STRATEGY_BTLAZY2",
'STRATEGY_BTOPT', "STRATEGY_BTOPT",
'STRATEGY_BTULTRA', "STRATEGY_BTULTRA",
'STRATEGY_BTULTRA2', "STRATEGY_BTULTRA2",
'DICT_TYPE_AUTO', "DICT_TYPE_AUTO",
'DICT_TYPE_RAWCONTENT', "DICT_TYPE_RAWCONTENT",
'DICT_TYPE_FULLDICT', "DICT_TYPE_FULLDICT",
) )
for a in attrs: for a in attrs:
self.assertTrue(hasattr(zstd, a), a) self.assertTrue(hasattr(zstd, a), a)

contrib/python-zstandard/tests/test_train_dictionary.py

import struct import struct
import sys import sys
import unittest import unittest
import zstandard as zstd import zstandard as zstd
from . common import ( from .common import (
generate_samples, generate_samples,
make_cffi, make_cffi,
random_input_data, random_input_data,
TestCase,
) )
if sys.version_info[0] >= 3: if sys.version_info[0] >= 3:
int_type = int int_type = int
else: else:
int_type = long int_type = long
@make_cffi @make_cffi
class TestTrainDictionary(unittest.TestCase): class TestTrainDictionary(TestCase):
def test_no_args(self): def test_no_args(self):
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
zstd.train_dictionary() zstd.train_dictionary()
def test_bad_args(self): def test_bad_args(self):
with self.assertRaises(TypeError): with self.assertRaises(TypeError):
zstd.train_dictionary(8192, u'foo') zstd.train_dictionary(8192, u"foo")
with self.assertRaises(ValueError): with self.assertRaises(ValueError):
zstd.train_dictionary(8192, [u'foo']) zstd.train_dictionary(8192, [u"foo"])
def test_no_params(self): def test_no_params(self):
d = zstd.train_dictionary(8192, random_input_data()) d = zstd.train_dictionary(8192, random_input_data())
self.assertIsInstance(d.dict_id(), int_type) self.assertIsInstance(d.dict_id(), int_type)
# The dictionary ID may be different across platforms. # The dictionary ID may be different across platforms.
expected = b'\x37\xa4\x30\xec' + struct.pack('<I', d.dict_id()) expected = b"\x37\xa4\x30\xec" + struct.pack("<I", d.dict_id())
data = d.as_bytes() data = d.as_bytes()
self.assertEqual(data[0:8], expected) self.assertEqual(data[0:8], expected)
def test_basic(self): def test_basic(self):
d = zstd.train_dictionary(8192, generate_samples(), k=64, d=16) d = zstd.train_dictionary(8192, generate_samples(), k=64, d=16)
self.assertIsInstance(d.dict_id(), int_type) self.assertIsInstance(d.dict_id(), int_type)
data = d.as_bytes() data = d.as_bytes()
self.assertEqual(data[0:4], b'\x37\xa4\x30\xec') self.assertEqual(data[0:4], b"\x37\xa4\x30\xec")
self.assertEqual(d.k, 64) self.assertEqual(d.k, 64)
self.assertEqual(d.d, 16) self.assertEqual(d.d, 16)
def test_set_dict_id(self): def test_set_dict_id(self):
d = zstd.train_dictionary(8192, generate_samples(), k=64, d=16, d = zstd.train_dictionary(8192, generate_samples(), k=64, d=16, dict_id=42)
dict_id=42)
self.assertEqual(d.dict_id(), 42) self.assertEqual(d.dict_id(), 42)
def test_optimize(self): def test_optimize(self):
d = zstd.train_dictionary(8192, generate_samples(), threads=-1, steps=1, d = zstd.train_dictionary(8192, generate_samples(), threads=-1, steps=1, d=16)
d=16)
# This varies by platform. # This varies by platform.
self.assertIn(d.k, (50, 2000)) self.assertIn(d.k, (50, 2000))
self.assertEqual(d.d, 16) self.assertEqual(d.d, 16)
@make_cffi @make_cffi
class TestCompressionDict(unittest.TestCase): class TestCompressionDict(TestCase):
def test_bad_mode(self): def test_bad_mode(self):
with self.assertRaisesRegexp(ValueError, 'invalid dictionary load mode'): with self.assertRaisesRegex(ValueError, "invalid dictionary load mode"):
zstd.ZstdCompressionDict(b'foo', dict_type=42) zstd.ZstdCompressionDict(b"foo", dict_type=42)
def test_bad_precompute_compress(self): def test_bad_precompute_compress(self):
d = zstd.train_dictionary(8192, generate_samples(), k=64, d=16) d = zstd.train_dictionary(8192, generate_samples(), k=64, d=16)
with self.assertRaisesRegexp(ValueError, 'must specify one of level or '): with self.assertRaisesRegex(ValueError, "must specify one of level or "):
d.precompute_compress() d.precompute_compress()
with self.assertRaisesRegexp(ValueError, 'must only specify one of level or '): with self.assertRaisesRegex(ValueError, "must only specify one of level or "):
d.precompute_compress(level=3, d.precompute_compress(
compression_params=zstd.CompressionParameters()) level=3, compression_params=zstd.CompressionParameters()
)
def test_precompute_compress_rawcontent(self): def test_precompute_compress_rawcontent(self):
d = zstd.ZstdCompressionDict(b'dictcontent' * 64, d = zstd.ZstdCompressionDict(
dict_type=zstd.DICT_TYPE_RAWCONTENT) b"dictcontent" * 64, dict_type=zstd.DICT_TYPE_RAWCONTENT
)
d.precompute_compress(level=1) d.precompute_compress(level=1)
d = zstd.ZstdCompressionDict(b'dictcontent' * 64, d = zstd.ZstdCompressionDict(
dict_type=zstd.DICT_TYPE_FULLDICT) b"dictcontent" * 64, dict_type=zstd.DICT_TYPE_FULLDICT
with self.assertRaisesRegexp(zstd.ZstdError, 'unable to precompute dictionary'): )
with self.assertRaisesRegex(zstd.ZstdError, "unable to precompute dictionary"):
d.precompute_compress(level=1) d.precompute_compress(level=1)

contrib/python-zstandard/zstandard/__init__.py

# imported. But for performance reasons, we only do this on unknown Python # imported. But for performance reasons, we only do this on unknown Python
# implementation. Notably, for CPython we require the C extension by default. # implementation. Notably, for CPython we require the C extension by default.
# Because someone will inevitably want special behavior, the behavior is # Because someone will inevitably want special behavior, the behavior is
# configurable via an environment variable. A potentially better way to handle # configurable via an environment variable. A potentially better way to handle
# this is to import a special ``__importpolicy__`` module or something # this is to import a special ``__importpolicy__`` module or something
# defining a variable and `setup.py` could write the file with whatever # defining a variable and `setup.py` could write the file with whatever
# policy was specified at build time. Until someone needs it, we go with # policy was specified at build time. Until someone needs it, we go with
# the hacky but simple environment variable approach. # the hacky but simple environment variable approach.
_module_policy = os.environ.get('PYTHON_ZSTANDARD_IMPORT_POLICY', 'default') _module_policy = os.environ.get("PYTHON_ZSTANDARD_IMPORT_POLICY", "default")
if _module_policy == 'default': if _module_policy == "default":
if platform.python_implementation() in ('CPython',): if platform.python_implementation() in ("CPython",):
from zstd import * from zstd import *
backend = 'cext'
elif platform.python_implementation() in ('PyPy',): backend = "cext"
elif platform.python_implementation() in ("PyPy",):
from .cffi import * from .cffi import *
backend = 'cffi'
backend = "cffi"
else: else:
try: try:
from zstd import * from zstd import *
backend = 'cext'
backend = "cext"
except ImportError: except ImportError:
from .cffi import * from .cffi import *
backend = 'cffi'
elif _module_policy == 'cffi_fallback': backend = "cffi"
elif _module_policy == "cffi_fallback":
try: try:
from zstd import * from zstd import *
backend = 'cext'
backend = "cext"
except ImportError: except ImportError:
from .cffi import * from .cffi import *
backend = 'cffi'
elif _module_policy == 'cext': backend = "cffi"
elif _module_policy == "cext":
from zstd import * from zstd import *
backend = 'cext'
elif _module_policy == 'cffi': backend = "cext"
elif _module_policy == "cffi":
from .cffi import * from .cffi import *
backend = 'cffi'
backend = "cffi"
else: else:
raise ImportError('unknown module import policy: %s; use default, cffi_fallback, ' raise ImportError(
'cext, or cffi' % _module_policy) "unknown module import policy: %s; use default, cffi_fallback, "
"cext, or cffi" % _module_policy
)
# Keep this in sync with python-zstandard.h. # Keep this in sync with python-zstandard.h.
__version__ = '0.12.0' __version__ = "0.13.0"

contrib/python-zstandard/zstandard/cffi.py

# Copyright (c) 2016-present, Gregory Szorc # Copyright (c) 2016-present, Gregory Szorc
# All rights reserved. # All rights reserved.
# #
# This software may be modified and distributed under the terms # This software may be modified and distributed under the terms
# of the BSD license. See the LICENSE file for details. # of the BSD license. See the LICENSE file for details.
"""Python interface to the Zstandard (zstd) compression library.""" """Python interface to the Zstandard (zstd) compression library."""
from __future__ import absolute_import, unicode_literals from __future__ import absolute_import, unicode_literals
# This should match what the C extension exports. # This should match what the C extension exports.
__all__ = [ __all__ = [
#'BufferSegment', #'BufferSegment',
#'BufferSegments', #'BufferSegments',
#'BufferWithSegments', #'BufferWithSegments',
#'BufferWithSegmentsCollection', #'BufferWithSegmentsCollection',
'CompressionParameters', "CompressionParameters",
'ZstdCompressionDict', "ZstdCompressionDict",
'ZstdCompressionParameters', "ZstdCompressionParameters",
'ZstdCompressor', "ZstdCompressor",
'ZstdError', "ZstdError",
'ZstdDecompressor', "ZstdDecompressor",
'FrameParameters', "FrameParameters",
'estimate_decompression_context_size', "estimate_decompression_context_size",
'frame_content_size', "frame_content_size",
'frame_header_size', "frame_header_size",
'get_frame_parameters', "get_frame_parameters",
'train_dictionary', "train_dictionary",
# Constants. # Constants.
'FLUSH_BLOCK', "FLUSH_BLOCK",
'FLUSH_FRAME', "FLUSH_FRAME",
'COMPRESSOBJ_FLUSH_FINISH', "COMPRESSOBJ_FLUSH_FINISH",
'COMPRESSOBJ_FLUSH_BLOCK', "COMPRESSOBJ_FLUSH_BLOCK",
'ZSTD_VERSION', "ZSTD_VERSION",
'FRAME_HEADER', "FRAME_HEADER",
'CONTENTSIZE_UNKNOWN', "CONTENTSIZE_UNKNOWN",
'CONTENTSIZE_ERROR', "CONTENTSIZE_ERROR",
'MAX_COMPRESSION_LEVEL', "MAX_COMPRESSION_LEVEL",
'COMPRESSION_RECOMMENDED_INPUT_SIZE', "COMPRESSION_RECOMMENDED_INPUT_SIZE",
'COMPRESSION_RECOMMENDED_OUTPUT_SIZE', "COMPRESSION_RECOMMENDED_OUTPUT_SIZE",
'DECOMPRESSION_RECOMMENDED_INPUT_SIZE', "DECOMPRESSION_RECOMMENDED_INPUT_SIZE",
'DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE', "DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE",
'MAGIC_NUMBER', "MAGIC_NUMBER",
'BLOCKSIZELOG_MAX', "BLOCKSIZELOG_MAX",
'BLOCKSIZE_MAX', "BLOCKSIZE_MAX",
'WINDOWLOG_MIN', "WINDOWLOG_MIN",
'WINDOWLOG_MAX', "WINDOWLOG_MAX",
'CHAINLOG_MIN', "CHAINLOG_MIN",
'CHAINLOG_MAX', "CHAINLOG_MAX",
'HASHLOG_MIN', "HASHLOG_MIN",
'HASHLOG_MAX', "HASHLOG_MAX",
'HASHLOG3_MAX', "HASHLOG3_MAX",
'MINMATCH_MIN', "MINMATCH_MIN",
'MINMATCH_MAX', "MINMATCH_MAX",
'SEARCHLOG_MIN', "SEARCHLOG_MIN",
'SEARCHLOG_MAX', "SEARCHLOG_MAX",
'SEARCHLENGTH_MIN', "SEARCHLENGTH_MIN",
'SEARCHLENGTH_MAX', "SEARCHLENGTH_MAX",
'TARGETLENGTH_MIN', "TARGETLENGTH_MIN",
'TARGETLENGTH_MAX', "TARGETLENGTH_MAX",
'LDM_MINMATCH_MIN', "LDM_MINMATCH_MIN",
'LDM_MINMATCH_MAX', "LDM_MINMATCH_MAX",
'LDM_BUCKETSIZELOG_MAX', "LDM_BUCKETSIZELOG_MAX",
'STRATEGY_FAST', "STRATEGY_FAST",
'STRATEGY_DFAST', "STRATEGY_DFAST",
'STRATEGY_GREEDY', "STRATEGY_GREEDY",
'STRATEGY_LAZY', "STRATEGY_LAZY",
'STRATEGY_LAZY2', "STRATEGY_LAZY2",
'STRATEGY_BTLAZY2', "STRATEGY_BTLAZY2",
'STRATEGY_BTOPT', "STRATEGY_BTOPT",
'STRATEGY_BTULTRA', "STRATEGY_BTULTRA",
'STRATEGY_BTULTRA2', "STRATEGY_BTULTRA2",
'DICT_TYPE_AUTO', "DICT_TYPE_AUTO",
'DICT_TYPE_RAWCONTENT', "DICT_TYPE_RAWCONTENT",
'DICT_TYPE_FULLDICT', "DICT_TYPE_FULLDICT",
'FORMAT_ZSTD1', "FORMAT_ZSTD1",
'FORMAT_ZSTD1_MAGICLESS', "FORMAT_ZSTD1_MAGICLESS",
] ]
import io import io
import os import os
import sys import sys
from _zstd_cffi import ( from _zstd_cffi import (
ffi, ffi,
DECOMPRESSION_RECOMMENDED_INPUT_SIZE = lib.ZSTD_DStreamInSize() DECOMPRESSION_RECOMMENDED_INPUT_SIZE = lib.ZSTD_DStreamInSize()
DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE = lib.ZSTD_DStreamOutSize() DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE = lib.ZSTD_DStreamOutSize()
new_nonzero = ffi.new_allocator(should_clear_after_alloc=False) new_nonzero = ffi.new_allocator(should_clear_after_alloc=False)
MAX_COMPRESSION_LEVEL = lib.ZSTD_maxCLevel() MAX_COMPRESSION_LEVEL = lib.ZSTD_maxCLevel()
MAGIC_NUMBER = lib.ZSTD_MAGICNUMBER MAGIC_NUMBER = lib.ZSTD_MAGICNUMBER
FRAME_HEADER = b'\x28\xb5\x2f\xfd' FRAME_HEADER = b"\x28\xb5\x2f\xfd"
CONTENTSIZE_UNKNOWN = lib.ZSTD_CONTENTSIZE_UNKNOWN CONTENTSIZE_UNKNOWN = lib.ZSTD_CONTENTSIZE_UNKNOWN
CONTENTSIZE_ERROR = lib.ZSTD_CONTENTSIZE_ERROR CONTENTSIZE_ERROR = lib.ZSTD_CONTENTSIZE_ERROR
ZSTD_VERSION = (lib.ZSTD_VERSION_MAJOR, lib.ZSTD_VERSION_MINOR, lib.ZSTD_VERSION_RELEASE) ZSTD_VERSION = (
lib.ZSTD_VERSION_MAJOR,
lib.ZSTD_VERSION_MINOR,
lib.ZSTD_VERSION_RELEASE,
)
BLOCKSIZELOG_MAX = lib.ZSTD_BLOCKSIZELOG_MAX BLOCKSIZELOG_MAX = lib.ZSTD_BLOCKSIZELOG_MAX
BLOCKSIZE_MAX = lib.ZSTD_BLOCKSIZE_MAX BLOCKSIZE_MAX = lib.ZSTD_BLOCKSIZE_MAX
WINDOWLOG_MIN = lib.ZSTD_WINDOWLOG_MIN WINDOWLOG_MIN = lib.ZSTD_WINDOWLOG_MIN
WINDOWLOG_MAX = lib.ZSTD_WINDOWLOG_MAX WINDOWLOG_MAX = lib.ZSTD_WINDOWLOG_MAX
CHAINLOG_MIN = lib.ZSTD_CHAINLOG_MIN CHAINLOG_MIN = lib.ZSTD_CHAINLOG_MIN
CHAINLOG_MAX = lib.ZSTD_CHAINLOG_MAX CHAINLOG_MAX = lib.ZSTD_CHAINLOG_MAX
HASHLOG_MIN = lib.ZSTD_HASHLOG_MIN HASHLOG_MIN = lib.ZSTD_HASHLOG_MIN
try: try:
return os.cpu_count() or 0 return os.cpu_count() or 0
except AttributeError: except AttributeError:
pass pass
# Linux. # Linux.
try: try:
if sys.version_info[0] == 2: if sys.version_info[0] == 2:
return os.sysconf(b'SC_NPROCESSORS_ONLN') return os.sysconf(b"SC_NPROCESSORS_ONLN")
else: else:
return os.sysconf(u'SC_NPROCESSORS_ONLN') return os.sysconf("SC_NPROCESSORS_ONLN")
except (AttributeError, ValueError): except (AttributeError, ValueError):
pass pass
# TODO implement on other platforms. # TODO implement on other platforms.
return 0 return 0
class ZstdError(Exception): class ZstdError(Exception):
pass pass
def _zstd_error(zresult): def _zstd_error(zresult):
# Resolves to bytes on Python 2 and 3. We use the string for formatting # Resolves to bytes on Python 2 and 3. We use the string for formatting
# into error messages, which will be literal unicode. So convert it to # into error messages, which will be literal unicode. So convert it to
# unicode. # unicode.
return ffi.string(lib.ZSTD_getErrorName(zresult)).decode('utf-8') return ffi.string(lib.ZSTD_getErrorName(zresult)).decode("utf-8")
def _make_cctx_params(params): def _make_cctx_params(params):
res = lib.ZSTD_createCCtxParams() res = lib.ZSTD_createCCtxParams()
if res == ffi.NULL: if res == ffi.NULL:
raise MemoryError() raise MemoryError()
res = ffi.gc(res, lib.ZSTD_freeCCtxParams) res = ffi.gc(res, lib.ZSTD_freeCCtxParams)
(lib.ZSTD_c_ldmHashRateLog, params.ldm_hash_rate_log), (lib.ZSTD_c_ldmHashRateLog, params.ldm_hash_rate_log),
] ]
for param, value in attrs: for param, value in attrs:
_set_compression_parameter(res, param, value) _set_compression_parameter(res, param, value)
return res return res
class ZstdCompressionParameters(object): class ZstdCompressionParameters(object):
@staticmethod @staticmethod
def from_level(level, source_size=0, dict_size=0, **kwargs): def from_level(level, source_size=0, dict_size=0, **kwargs):
params = lib.ZSTD_getCParams(level, source_size, dict_size) params = lib.ZSTD_getCParams(level, source_size, dict_size)
args = { args = {
'window_log': 'windowLog', "window_log": "windowLog",
'chain_log': 'chainLog', "chain_log": "chainLog",
'hash_log': 'hashLog', "hash_log": "hashLog",
'search_log': 'searchLog', "search_log": "searchLog",
'min_match': 'minMatch', "min_match": "minMatch",
'target_length': 'targetLength', "target_length": "targetLength",
'compression_strategy': 'strategy', "compression_strategy": "strategy",
} }
for arg, attr in args.items(): for arg, attr in args.items():
if arg not in kwargs: if arg not in kwargs:
kwargs[arg] = getattr(params, attr) kwargs[arg] = getattr(params, attr)
return ZstdCompressionParameters(**kwargs) return ZstdCompressionParameters(**kwargs)
def __init__(self, format=0, compression_level=0, window_log=0, hash_log=0, def __init__(
chain_log=0, search_log=0, min_match=0, target_length=0, self,
strategy=-1, compression_strategy=-1, format=0,
write_content_size=1, write_checksum=0, compression_level=0,
write_dict_id=0, job_size=0, overlap_log=-1, window_log=0,
overlap_size_log=-1, force_max_window=0, enable_ldm=0, hash_log=0,
ldm_hash_log=0, ldm_min_match=0, ldm_bucket_size_log=0, chain_log=0,
ldm_hash_rate_log=-1, ldm_hash_every_log=-1, threads=0): search_log=0,
min_match=0,
target_length=0,
strategy=-1,
compression_strategy=-1,
write_content_size=1,
write_checksum=0,
write_dict_id=0,
job_size=0,
overlap_log=-1,
overlap_size_log=-1,
force_max_window=0,
enable_ldm=0,
ldm_hash_log=0,
ldm_min_match=0,
ldm_bucket_size_log=0,
ldm_hash_rate_log=-1,
ldm_hash_every_log=-1,
threads=0,
):
params = lib.ZSTD_createCCtxParams() params = lib.ZSTD_createCCtxParams()
if params == ffi.NULL: if params == ffi.NULL:
raise MemoryError() raise MemoryError()
params = ffi.gc(params, lib.ZSTD_freeCCtxParams) params = ffi.gc(params, lib.ZSTD_freeCCtxParams)
self._params = params self._params = params
if threads < 0: if threads < 0:
threads = _cpu_count() threads = _cpu_count()
# We need to set ZSTD_c_nbWorkers before ZSTD_c_jobSize and ZSTD_c_overlapLog # We need to set ZSTD_c_nbWorkers before ZSTD_c_jobSize and ZSTD_c_overlapLog
# because setting ZSTD_c_nbWorkers resets the other parameters. # because setting ZSTD_c_nbWorkers resets the other parameters.
_set_compression_parameter(params, lib.ZSTD_c_nbWorkers, threads) _set_compression_parameter(params, lib.ZSTD_c_nbWorkers, threads)
_set_compression_parameter(params, lib.ZSTD_c_format, format) _set_compression_parameter(params, lib.ZSTD_c_format, format)
_set_compression_parameter(params, lib.ZSTD_c_compressionLevel, compression_level) _set_compression_parameter(
params, lib.ZSTD_c_compressionLevel, compression_level
)
_set_compression_parameter(params, lib.ZSTD_c_windowLog, window_log) _set_compression_parameter(params, lib.ZSTD_c_windowLog, window_log)
_set_compression_parameter(params, lib.ZSTD_c_hashLog, hash_log) _set_compression_parameter(params, lib.ZSTD_c_hashLog, hash_log)
_set_compression_parameter(params, lib.ZSTD_c_chainLog, chain_log) _set_compression_parameter(params, lib.ZSTD_c_chainLog, chain_log)
_set_compression_parameter(params, lib.ZSTD_c_searchLog, search_log) _set_compression_parameter(params, lib.ZSTD_c_searchLog, search_log)
_set_compression_parameter(params, lib.ZSTD_c_minMatch, min_match) _set_compression_parameter(params, lib.ZSTD_c_minMatch, min_match)
_set_compression_parameter(params, lib.ZSTD_c_targetLength, target_length) _set_compression_parameter(params, lib.ZSTD_c_targetLength, target_length)
if strategy != -1 and compression_strategy != -1: if strategy != -1 and compression_strategy != -1:
raise ValueError('cannot specify both compression_strategy and strategy') raise ValueError("cannot specify both compression_strategy and strategy")
if compression_strategy != -1: if compression_strategy != -1:
strategy = compression_strategy strategy = compression_strategy
elif strategy == -1: elif strategy == -1:
strategy = 0 strategy = 0
_set_compression_parameter(params, lib.ZSTD_c_strategy, strategy) _set_compression_parameter(params, lib.ZSTD_c_strategy, strategy)
_set_compression_parameter(params, lib.ZSTD_c_contentSizeFlag, write_content_size) _set_compression_parameter(
params, lib.ZSTD_c_contentSizeFlag, write_content_size
)
_set_compression_parameter(params, lib.ZSTD_c_checksumFlag, write_checksum) _set_compression_parameter(params, lib.ZSTD_c_checksumFlag, write_checksum)
_set_compression_parameter(params, lib.ZSTD_c_dictIDFlag, write_dict_id) _set_compression_parameter(params, lib.ZSTD_c_dictIDFlag, write_dict_id)
_set_compression_parameter(params, lib.ZSTD_c_jobSize, job_size) _set_compression_parameter(params, lib.ZSTD_c_jobSize, job_size)
if overlap_log != -1 and overlap_size_log != -1: if overlap_log != -1 and overlap_size_log != -1:
raise ValueError('cannot specify both overlap_log and overlap_size_log') raise ValueError("cannot specify both overlap_log and overlap_size_log")
if overlap_size_log != -1: if overlap_size_log != -1:
overlap_log = overlap_size_log overlap_log = overlap_size_log
elif overlap_log == -1: elif overlap_log == -1:
overlap_log = 0 overlap_log = 0
_set_compression_parameter(params, lib.ZSTD_c_overlapLog, overlap_log) _set_compression_parameter(params, lib.ZSTD_c_overlapLog, overlap_log)
_set_compression_parameter(params, lib.ZSTD_c_forceMaxWindow, force_max_window) _set_compression_parameter(params, lib.ZSTD_c_forceMaxWindow, force_max_window)
_set_compression_parameter(params, lib.ZSTD_c_enableLongDistanceMatching, enable_ldm) _set_compression_parameter(
params, lib.ZSTD_c_enableLongDistanceMatching, enable_ldm
)
_set_compression_parameter(params, lib.ZSTD_c_ldmHashLog, ldm_hash_log) _set_compression_parameter(params, lib.ZSTD_c_ldmHashLog, ldm_hash_log)
_set_compression_parameter(params, lib.ZSTD_c_ldmMinMatch, ldm_min_match) _set_compression_parameter(params, lib.ZSTD_c_ldmMinMatch, ldm_min_match)
_set_compression_parameter(params, lib.ZSTD_c_ldmBucketSizeLog, ldm_bucket_size_log) _set_compression_parameter(
params, lib.ZSTD_c_ldmBucketSizeLog, ldm_bucket_size_log
)
if ldm_hash_rate_log != -1 and ldm_hash_every_log != -1: if ldm_hash_rate_log != -1 and ldm_hash_every_log != -1:
raise ValueError('cannot specify both ldm_hash_rate_log and ldm_hash_every_log') raise ValueError(
"cannot specify both ldm_hash_rate_log and ldm_hash_every_log"
)
if ldm_hash_every_log != -1: if ldm_hash_every_log != -1:
ldm_hash_rate_log = ldm_hash_every_log ldm_hash_rate_log = ldm_hash_every_log
elif ldm_hash_rate_log == -1: elif ldm_hash_rate_log == -1:
ldm_hash_rate_log = 0 ldm_hash_rate_log = 0
_set_compression_parameter(params, lib.ZSTD_c_ldmHashRateLog, ldm_hash_rate_log) _set_compression_parameter(params, lib.ZSTD_c_ldmHashRateLog, ldm_hash_rate_log)
return self.overlap_log return self.overlap_log
@property @property
def force_max_window(self): def force_max_window(self):
return _get_compression_parameter(self._params, lib.ZSTD_c_forceMaxWindow) return _get_compression_parameter(self._params, lib.ZSTD_c_forceMaxWindow)
@property @property
def enable_ldm(self): def enable_ldm(self):
return _get_compression_parameter(self._params, lib.ZSTD_c_enableLongDistanceMatching) return _get_compression_parameter(
self._params, lib.ZSTD_c_enableLongDistanceMatching
)
@property @property
def ldm_hash_log(self): def ldm_hash_log(self):
return _get_compression_parameter(self._params, lib.ZSTD_c_ldmHashLog) return _get_compression_parameter(self._params, lib.ZSTD_c_ldmHashLog)
@property @property
def ldm_min_match(self): def ldm_min_match(self):
return _get_compression_parameter(self._params, lib.ZSTD_c_ldmMinMatch) return _get_compression_parameter(self._params, lib.ZSTD_c_ldmMinMatch)
@property @property
def threads(self): def threads(self):
return _get_compression_parameter(self._params, lib.ZSTD_c_nbWorkers) return _get_compression_parameter(self._params, lib.ZSTD_c_nbWorkers)
def estimated_compression_context_size(self): def estimated_compression_context_size(self):
return lib.ZSTD_estimateCCtxSize_usingCCtxParams(self._params) return lib.ZSTD_estimateCCtxSize_usingCCtxParams(self._params)
CompressionParameters = ZstdCompressionParameters CompressionParameters = ZstdCompressionParameters
def estimate_decompression_context_size(): def estimate_decompression_context_size():
return lib.ZSTD_estimateDCtxSize() return lib.ZSTD_estimateDCtxSize()
def _set_compression_parameter(params, param, value): def _set_compression_parameter(params, param, value):
zresult = lib.ZSTD_CCtxParams_setParameter(params, param, value) zresult = lib.ZSTD_CCtxParams_setParameter(params, param, value)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('unable to set compression context parameter: %s' % raise ZstdError(
_zstd_error(zresult)) "unable to set compression context parameter: %s" % _zstd_error(zresult)
)
def _get_compression_parameter(params, param): def _get_compression_parameter(params, param):
result = ffi.new('int *') result = ffi.new("int *")
zresult = lib.ZSTD_CCtxParams_getParameter(params, param, result) zresult = lib.ZSTD_CCtxParams_getParameter(params, param, result)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('unable to get compression context parameter: %s' % raise ZstdError(
_zstd_error(zresult)) "unable to get compression context parameter: %s" % _zstd_error(zresult)
)
return result[0] return result[0]
class ZstdCompressionWriter(object): class ZstdCompressionWriter(object):
def __init__(self, compressor, writer, source_size, write_size, def __init__(self, compressor, writer, source_size, write_size, write_return_read):
write_return_read):
self._compressor = compressor self._compressor = compressor
self._writer = writer self._writer = writer
self._write_size = write_size self._write_size = write_size
self._write_return_read = bool(write_return_read) self._write_return_read = bool(write_return_read)
self._entered = False self._entered = False
self._closed = False self._closed = False
self._bytes_compressed = 0 self._bytes_compressed = 0
self._dst_buffer = ffi.new('char[]', write_size) self._dst_buffer = ffi.new("char[]", write_size)
self._out_buffer = ffi.new('ZSTD_outBuffer *') self._out_buffer = ffi.new("ZSTD_outBuffer *")
self._out_buffer.dst = self._dst_buffer self._out_buffer.dst = self._dst_buffer
self._out_buffer.size = len(self._dst_buffer) self._out_buffer.size = len(self._dst_buffer)
self._out_buffer.pos = 0 self._out_buffer.pos = 0
zresult = lib.ZSTD_CCtx_setPledgedSrcSize(compressor._cctx, zresult = lib.ZSTD_CCtx_setPledgedSrcSize(compressor._cctx, source_size)
source_size)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error setting source size: %s' % raise ZstdError("error setting source size: %s" % _zstd_error(zresult))
_zstd_error(zresult))
def __enter__(self): def __enter__(self):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
if self._entered: if self._entered:
raise ZstdError('cannot __enter__ multiple times') raise ZstdError("cannot __enter__ multiple times")
self._entered = True self._entered = True
return self return self
def __exit__(self, exc_type, exc_value, exc_tb): def __exit__(self, exc_type, exc_value, exc_tb):
self._entered = False self._entered = False
if not exc_type and not exc_value and not exc_tb: if not exc_type and not exc_value and not exc_tb:
self.close() self.close()
self._compressor = None self._compressor = None
return False return False
def memory_size(self): def memory_size(self):
return lib.ZSTD_sizeof_CCtx(self._compressor._cctx) return lib.ZSTD_sizeof_CCtx(self._compressor._cctx)
def fileno(self): def fileno(self):
f = getattr(self._writer, 'fileno', None) f = getattr(self._writer, "fileno", None)
if f: if f:
return f() return f()
else: else:
raise OSError('fileno not available on underlying writer') raise OSError("fileno not available on underlying writer")
def close(self): def close(self):
if self._closed: if self._closed:
return return
try: try:
self.flush(FLUSH_FRAME) self.flush(FLUSH_FRAME)
finally: finally:
self._closed = True self._closed = True
# Call close() on underlying stream as well. # Call close() on underlying stream as well.
f = getattr(self._writer, 'close', None) f = getattr(self._writer, "close", None)
if f: if f:
f() f()
@property @property
def closed(self): def closed(self):
return self._closed return self._closed
def isatty(self): def isatty(self):
def truncate(self, size=None): def truncate(self, size=None):
raise io.UnsupportedOperation() raise io.UnsupportedOperation()
def writable(self): def writable(self):
return True return True
def writelines(self, lines): def writelines(self, lines):
raise NotImplementedError('writelines() is not yet implemented') raise NotImplementedError("writelines() is not yet implemented")
def read(self, size=-1): def read(self, size=-1):
raise io.UnsupportedOperation() raise io.UnsupportedOperation()
def readall(self): def readall(self):
raise io.UnsupportedOperation() raise io.UnsupportedOperation()
def readinto(self, b): def readinto(self, b):
raise io.UnsupportedOperation() raise io.UnsupportedOperation()
def write(self, data): def write(self, data):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
total_write = 0 total_write = 0
data_buffer = ffi.from_buffer(data) data_buffer = ffi.from_buffer(data)
in_buffer = ffi.new('ZSTD_inBuffer *') in_buffer = ffi.new("ZSTD_inBuffer *")
in_buffer.src = data_buffer in_buffer.src = data_buffer
in_buffer.size = len(data_buffer) in_buffer.size = len(data_buffer)
in_buffer.pos = 0 in_buffer.pos = 0
out_buffer = self._out_buffer out_buffer = self._out_buffer
out_buffer.pos = 0 out_buffer.pos = 0
while in_buffer.pos < in_buffer.size: while in_buffer.pos < in_buffer.size:
zresult = lib.ZSTD_compressStream2(self._compressor._cctx, zresult = lib.ZSTD_compressStream2(
out_buffer, in_buffer, self._compressor._cctx, out_buffer, in_buffer, lib.ZSTD_e_continue
lib.ZSTD_e_continue) )
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd compress error: %s' % raise ZstdError("zstd compress error: %s" % _zstd_error(zresult))
_zstd_error(zresult))
if out_buffer.pos: if out_buffer.pos:
self._writer.write(ffi.buffer(out_buffer.dst, out_buffer.pos)[:]) self._writer.write(ffi.buffer(out_buffer.dst, out_buffer.pos)[:])
total_write += out_buffer.pos total_write += out_buffer.pos
self._bytes_compressed += out_buffer.pos self._bytes_compressed += out_buffer.pos
out_buffer.pos = 0 out_buffer.pos = 0
if self._write_return_read: if self._write_return_read:
return in_buffer.pos return in_buffer.pos
else: else:
return total_write return total_write
def flush(self, flush_mode=FLUSH_BLOCK): def flush(self, flush_mode=FLUSH_BLOCK):
if flush_mode == FLUSH_BLOCK: if flush_mode == FLUSH_BLOCK:
flush = lib.ZSTD_e_flush flush = lib.ZSTD_e_flush
elif flush_mode == FLUSH_FRAME: elif flush_mode == FLUSH_FRAME:
flush = lib.ZSTD_e_end flush = lib.ZSTD_e_end
else: else:
raise ValueError('unknown flush_mode: %r' % flush_mode) raise ValueError("unknown flush_mode: %r" % flush_mode)
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
total_write = 0 total_write = 0
out_buffer = self._out_buffer out_buffer = self._out_buffer
out_buffer.pos = 0 out_buffer.pos = 0
in_buffer = ffi.new('ZSTD_inBuffer *') in_buffer = ffi.new("ZSTD_inBuffer *")
in_buffer.src = ffi.NULL in_buffer.src = ffi.NULL
in_buffer.size = 0 in_buffer.size = 0
in_buffer.pos = 0 in_buffer.pos = 0
while True: while True:
zresult = lib.ZSTD_compressStream2(self._compressor._cctx, zresult = lib.ZSTD_compressStream2(
out_buffer, in_buffer, self._compressor._cctx, out_buffer, in_buffer, flush
flush) )
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd compress error: %s' % raise ZstdError("zstd compress error: %s" % _zstd_error(zresult))
_zstd_error(zresult))
if out_buffer.pos: if out_buffer.pos:
self._writer.write(ffi.buffer(out_buffer.dst, out_buffer.pos)[:]) self._writer.write(ffi.buffer(out_buffer.dst, out_buffer.pos)[:])
total_write += out_buffer.pos total_write += out_buffer.pos
self._bytes_compressed += out_buffer.pos self._bytes_compressed += out_buffer.pos
out_buffer.pos = 0 out_buffer.pos = 0
if not zresult: if not zresult:
break break
return total_write return total_write
def tell(self): def tell(self):
return self._bytes_compressed return self._bytes_compressed
class ZstdCompressionObj(object): class ZstdCompressionObj(object):
def compress(self, data): def compress(self, data):
if self._finished: if self._finished:
raise ZstdError('cannot call compress() after compressor finished') raise ZstdError("cannot call compress() after compressor finished")
data_buffer = ffi.from_buffer(data) data_buffer = ffi.from_buffer(data)
source = ffi.new('ZSTD_inBuffer *') source = ffi.new("ZSTD_inBuffer *")
source.src = data_buffer source.src = data_buffer
source.size = len(data_buffer) source.size = len(data_buffer)
source.pos = 0 source.pos = 0
chunks = [] chunks = []
while source.pos < len(data): while source.pos < len(data):
zresult = lib.ZSTD_compressStream2(self._compressor._cctx, zresult = lib.ZSTD_compressStream2(
self._out, self._compressor._cctx, self._out, source, lib.ZSTD_e_continue
source, )
lib.ZSTD_e_continue)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd compress error: %s' % raise ZstdError("zstd compress error: %s" % _zstd_error(zresult))
_zstd_error(zresult))
if self._out.pos: if self._out.pos:
chunks.append(ffi.buffer(self._out.dst, self._out.pos)[:]) chunks.append(ffi.buffer(self._out.dst, self._out.pos)[:])
self._out.pos = 0 self._out.pos = 0
return b''.join(chunks) return b"".join(chunks)
def flush(self, flush_mode=COMPRESSOBJ_FLUSH_FINISH): def flush(self, flush_mode=COMPRESSOBJ_FLUSH_FINISH):
if flush_mode not in (COMPRESSOBJ_FLUSH_FINISH, COMPRESSOBJ_FLUSH_BLOCK): if flush_mode not in (COMPRESSOBJ_FLUSH_FINISH, COMPRESSOBJ_FLUSH_BLOCK):
raise ValueError('flush mode not recognized') raise ValueError("flush mode not recognized")
if self._finished: if self._finished:
raise ZstdError('compressor object already finished') raise ZstdError("compressor object already finished")
if flush_mode == COMPRESSOBJ_FLUSH_BLOCK: if flush_mode == COMPRESSOBJ_FLUSH_BLOCK:
z_flush_mode = lib.ZSTD_e_flush z_flush_mode = lib.ZSTD_e_flush
elif flush_mode == COMPRESSOBJ_FLUSH_FINISH: elif flush_mode == COMPRESSOBJ_FLUSH_FINISH:
z_flush_mode = lib.ZSTD_e_end z_flush_mode = lib.ZSTD_e_end
self._finished = True self._finished = True
else: else:
raise ZstdError('unhandled flush mode') raise ZstdError("unhandled flush mode")
assert self._out.pos == 0 assert self._out.pos == 0
in_buffer = ffi.new('ZSTD_inBuffer *') in_buffer = ffi.new("ZSTD_inBuffer *")
in_buffer.src = ffi.NULL in_buffer.src = ffi.NULL
in_buffer.size = 0 in_buffer.size = 0
in_buffer.pos = 0 in_buffer.pos = 0
chunks = [] chunks = []
while True: while True:
zresult = lib.ZSTD_compressStream2(self._compressor._cctx, zresult = lib.ZSTD_compressStream2(
self._out, self._compressor._cctx, self._out, in_buffer, z_flush_mode
in_buffer, )
z_flush_mode)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error ending compression stream: %s' % raise ZstdError(
_zstd_error(zresult)) "error ending compression stream: %s" % _zstd_error(zresult)
)
if self._out.pos: if self._out.pos:
chunks.append(ffi.buffer(self._out.dst, self._out.pos)[:]) chunks.append(ffi.buffer(self._out.dst, self._out.pos)[:])
self._out.pos = 0 self._out.pos = 0
if not zresult: if not zresult:
break break
return b''.join(chunks) return b"".join(chunks)
class ZstdCompressionChunker(object): class ZstdCompressionChunker(object):
def __init__(self, compressor, chunk_size): def __init__(self, compressor, chunk_size):
self._compressor = compressor self._compressor = compressor
self._out = ffi.new('ZSTD_outBuffer *') self._out = ffi.new("ZSTD_outBuffer *")
self._dst_buffer = ffi.new('char[]', chunk_size) self._dst_buffer = ffi.new("char[]", chunk_size)
self._out.dst = self._dst_buffer self._out.dst = self._dst_buffer
self._out.size = chunk_size self._out.size = chunk_size
self._out.pos = 0 self._out.pos = 0
self._in = ffi.new('ZSTD_inBuffer *') self._in = ffi.new("ZSTD_inBuffer *")
self._in.src = ffi.NULL self._in.src = ffi.NULL
self._in.size = 0 self._in.size = 0
self._in.pos = 0 self._in.pos = 0
self._finished = False self._finished = False
def compress(self, data): def compress(self, data):
if self._finished: if self._finished:
raise ZstdError('cannot call compress() after compression finished') raise ZstdError("cannot call compress() after compression finished")
if self._in.src != ffi.NULL: if self._in.src != ffi.NULL:
raise ZstdError('cannot perform operation before consuming output ' raise ZstdError(
'from previous operation') "cannot perform operation before consuming output "
"from previous operation"
)
data_buffer = ffi.from_buffer(data) data_buffer = ffi.from_buffer(data)
if not len(data_buffer): if not len(data_buffer):
return return
self._in.src = data_buffer self._in.src = data_buffer
self._in.size = len(data_buffer) self._in.size = len(data_buffer)
self._in.pos = 0 self._in.pos = 0
while self._in.pos < self._in.size: while self._in.pos < self._in.size:
zresult = lib.ZSTD_compressStream2(self._compressor._cctx, zresult = lib.ZSTD_compressStream2(
self._out, self._compressor._cctx, self._out, self._in, lib.ZSTD_e_continue
self._in, )
lib.ZSTD_e_continue)
if self._in.pos == self._in.size: if self._in.pos == self._in.size:
self._in.src = ffi.NULL self._in.src = ffi.NULL
self._in.size = 0 self._in.size = 0
self._in.pos = 0 self._in.pos = 0
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd compress error: %s' % raise ZstdError("zstd compress error: %s" % _zstd_error(zresult))
_zstd_error(zresult))
if self._out.pos == self._out.size: if self._out.pos == self._out.size:
yield ffi.buffer(self._out.dst, self._out.pos)[:] yield ffi.buffer(self._out.dst, self._out.pos)[:]
self._out.pos = 0 self._out.pos = 0
def flush(self): def flush(self):
if self._finished: if self._finished:
raise ZstdError('cannot call flush() after compression finished') raise ZstdError("cannot call flush() after compression finished")
if self._in.src != ffi.NULL: if self._in.src != ffi.NULL:
raise ZstdError('cannot call flush() before consuming output from ' raise ZstdError(
'previous operation') "cannot call flush() before consuming output from " "previous operation"
)
while True: while True:
zresult = lib.ZSTD_compressStream2(self._compressor._cctx, zresult = lib.ZSTD_compressStream2(
self._out, self._in, self._compressor._cctx, self._out, self._in, lib.ZSTD_e_flush
lib.ZSTD_e_flush) )
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd compress error: %s' % _zstd_error(zresult)) raise ZstdError("zstd compress error: %s" % _zstd_error(zresult))
if self._out.pos: if self._out.pos:
yield ffi.buffer(self._out.dst, self._out.pos)[:] yield ffi.buffer(self._out.dst, self._out.pos)[:]
self._out.pos = 0 self._out.pos = 0
if not zresult: if not zresult:
return return
def finish(self): def finish(self):
if self._finished: if self._finished:
raise ZstdError('cannot call finish() after compression finished') raise ZstdError("cannot call finish() after compression finished")
if self._in.src != ffi.NULL: if self._in.src != ffi.NULL:
raise ZstdError('cannot call finish() before consuming output from ' raise ZstdError(
'previous operation') "cannot call finish() before consuming output from "
"previous operation"
)
while True: while True:
zresult = lib.ZSTD_compressStream2(self._compressor._cctx, zresult = lib.ZSTD_compressStream2(
self._out, self._in, self._compressor._cctx, self._out, self._in, lib.ZSTD_e_end
lib.ZSTD_e_end) )
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd compress error: %s' % _zstd_error(zresult)) raise ZstdError("zstd compress error: %s" % _zstd_error(zresult))
if self._out.pos: if self._out.pos:
yield ffi.buffer(self._out.dst, self._out.pos)[:] yield ffi.buffer(self._out.dst, self._out.pos)[:]
self._out.pos = 0 self._out.pos = 0
if not zresult: if not zresult:
self._finished = True self._finished = True
return return
class ZstdCompressionReader(object): class ZstdCompressionReader(object):
def __init__(self, compressor, source, read_size): def __init__(self, compressor, source, read_size):
self._compressor = compressor self._compressor = compressor
self._source = source self._source = source
self._read_size = read_size self._read_size = read_size
self._entered = False self._entered = False
self._closed = False self._closed = False
self._bytes_compressed = 0 self._bytes_compressed = 0
self._finished_input = False self._finished_input = False
self._finished_output = False self._finished_output = False
self._in_buffer = ffi.new('ZSTD_inBuffer *') self._in_buffer = ffi.new("ZSTD_inBuffer *")
# Holds a ref so backing bytes in self._in_buffer stay alive. # Holds a ref so backing bytes in self._in_buffer stay alive.
self._source_buffer = None self._source_buffer = None
def __enter__(self): def __enter__(self):
if self._entered: if self._entered:
raise ValueError('cannot __enter__ multiple times') raise ValueError("cannot __enter__ multiple times")
self._entered = True self._entered = True
return self return self
def __exit__(self, exc_type, exc_value, exc_tb): def __exit__(self, exc_type, exc_value, exc_tb):
self._entered = False self._entered = False
self._closed = True self._closed = True
self._source = None self._source = None
def readline(self): def readline(self):
raise io.UnsupportedOperation() raise io.UnsupportedOperation()
def readlines(self): def readlines(self):
raise io.UnsupportedOperation() raise io.UnsupportedOperation()
def write(self, data): def write(self, data):
raise OSError('stream is not writable') raise OSError("stream is not writable")
def writelines(self, ignored): def writelines(self, ignored):
raise OSError('stream is not writable') raise OSError("stream is not writable")
def isatty(self): def isatty(self):
return False return False
def flush(self): def flush(self):
return None return None
def close(self): def close(self):
while True: while True:
chunk = self.read(1048576) chunk = self.read(1048576)
if not chunk: if not chunk:
break break
chunks.append(chunk) chunks.append(chunk)
return b''.join(chunks) return b"".join(chunks)
def __iter__(self): def __iter__(self):
raise io.UnsupportedOperation() raise io.UnsupportedOperation()
def __next__(self): def __next__(self):
raise io.UnsupportedOperation() raise io.UnsupportedOperation()
next = __next__ next = __next__
def _read_input(self): def _read_input(self):
if self._finished_input: if self._finished_input:
return return
if hasattr(self._source, 'read'): if hasattr(self._source, "read"):
data = self._source.read(self._read_size) data = self._source.read(self._read_size)
if not data: if not data:
self._finished_input = True self._finished_input = True
return return
self._source_buffer = ffi.from_buffer(data) self._source_buffer = ffi.from_buffer(data)
self._in_buffer.src = self._source_buffer self._in_buffer.src = self._source_buffer
self._in_buffer.size = len(self._source_buffer) self._in_buffer.size = len(self._source_buffer)
self._in_buffer.pos = 0 self._in_buffer.pos = 0
else: else:
self._source_buffer = ffi.from_buffer(self._source) self._source_buffer = ffi.from_buffer(self._source)
self._in_buffer.src = self._source_buffer self._in_buffer.src = self._source_buffer
self._in_buffer.size = len(self._source_buffer) self._in_buffer.size = len(self._source_buffer)
self._in_buffer.pos = 0 self._in_buffer.pos = 0
def _compress_into_buffer(self, out_buffer): def _compress_into_buffer(self, out_buffer):
if self._in_buffer.pos >= self._in_buffer.size: if self._in_buffer.pos >= self._in_buffer.size:
return return
old_pos = out_buffer.pos old_pos = out_buffer.pos
zresult = lib.ZSTD_compressStream2(self._compressor._cctx, zresult = lib.ZSTD_compressStream2(
out_buffer, self._in_buffer, self._compressor._cctx, out_buffer, self._in_buffer, lib.ZSTD_e_continue
lib.ZSTD_e_continue) )
self._bytes_compressed += out_buffer.pos - old_pos self._bytes_compressed += out_buffer.pos - old_pos
if self._in_buffer.pos == self._in_buffer.size: if self._in_buffer.pos == self._in_buffer.size:
self._in_buffer.src = ffi.NULL self._in_buffer.src = ffi.NULL
self._in_buffer.pos = 0 self._in_buffer.pos = 0
self._in_buffer.size = 0 self._in_buffer.size = 0
self._source_buffer = None self._source_buffer = None
if not hasattr(self._source, 'read'): if not hasattr(self._source, "read"):
self._finished_input = True self._finished_input = True
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd compress error: %s', raise ZstdError("zstd compress error: %s", _zstd_error(zresult))
_zstd_error(zresult))
return out_buffer.pos and out_buffer.pos == out_buffer.size return out_buffer.pos and out_buffer.pos == out_buffer.size
def read(self, size=-1): def read(self, size=-1):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
if size < -1: if size < -1:
raise ValueError('cannot read negative amounts less than -1') raise ValueError("cannot read negative amounts less than -1")
if size == -1: if size == -1:
return self.readall() return self.readall()
if self._finished_output or size == 0: if self._finished_output or size == 0:
return b'' return b""
# Need a dedicated ref to dest buffer otherwise it gets collected. # Need a dedicated ref to dest buffer otherwise it gets collected.
dst_buffer = ffi.new('char[]', size) dst_buffer = ffi.new("char[]", size)
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
out_buffer.dst = dst_buffer out_buffer.dst = dst_buffer
out_buffer.size = size out_buffer.size = size
out_buffer.pos = 0 out_buffer.pos = 0
if self._compress_into_buffer(out_buffer): if self._compress_into_buffer(out_buffer):
return ffi.buffer(out_buffer.dst, out_buffer.pos)[:] return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
while not self._finished_input: while not self._finished_input:
self._read_input() self._read_input()
if self._compress_into_buffer(out_buffer): if self._compress_into_buffer(out_buffer):
return ffi.buffer(out_buffer.dst, out_buffer.pos)[:] return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
# EOF # EOF
old_pos = out_buffer.pos old_pos = out_buffer.pos
zresult = lib.ZSTD_compressStream2(self._compressor._cctx, zresult = lib.ZSTD_compressStream2(
out_buffer, self._in_buffer, self._compressor._cctx, out_buffer, self._in_buffer, lib.ZSTD_e_end
lib.ZSTD_e_end) )
self._bytes_compressed += out_buffer.pos - old_pos self._bytes_compressed += out_buffer.pos - old_pos
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error ending compression stream: %s', raise ZstdError("error ending compression stream: %s", _zstd_error(zresult))
_zstd_error(zresult))
if zresult == 0: if zresult == 0:
self._finished_output = True self._finished_output = True
return ffi.buffer(out_buffer.dst, out_buffer.pos)[:] return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
def read1(self, size=-1): def read1(self, size=-1):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
if size < -1: if size < -1:
raise ValueError('cannot read negative amounts less than -1') raise ValueError("cannot read negative amounts less than -1")
if self._finished_output or size == 0: if self._finished_output or size == 0:
return b'' return b""
# -1 returns arbitrary number of bytes. # -1 returns arbitrary number of bytes.
if size == -1: if size == -1:
size = COMPRESSION_RECOMMENDED_OUTPUT_SIZE size = COMPRESSION_RECOMMENDED_OUTPUT_SIZE
dst_buffer = ffi.new('char[]', size) dst_buffer = ffi.new("char[]", size)
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
out_buffer.dst = dst_buffer out_buffer.dst = dst_buffer
out_buffer.size = size out_buffer.size = size
out_buffer.pos = 0 out_buffer.pos = 0
# read1() dictates that we can perform at most 1 call to the # read1() dictates that we can perform at most 1 call to the
# underlying stream to get input. However, we can't satisfy this # underlying stream to get input. However, we can't satisfy this
# restriction with compression because not all input generates output. # restriction with compression because not all input generates output.
# It is possible to perform a block flush in order to ensure output. # It is possible to perform a block flush in order to ensure output.
return ffi.buffer(out_buffer.dst, out_buffer.pos)[:] return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
# Else if we're at EOS and we have room left in the buffer, # Else if we're at EOS and we have room left in the buffer,
# fall through to below and try to add more data to the output. # fall through to below and try to add more data to the output.
# EOF. # EOF.
old_pos = out_buffer.pos old_pos = out_buffer.pos
zresult = lib.ZSTD_compressStream2(self._compressor._cctx, zresult = lib.ZSTD_compressStream2(
out_buffer, self._in_buffer, self._compressor._cctx, out_buffer, self._in_buffer, lib.ZSTD_e_end
lib.ZSTD_e_end) )
self._bytes_compressed += out_buffer.pos - old_pos self._bytes_compressed += out_buffer.pos - old_pos
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error ending compression stream: %s' % raise ZstdError(
_zstd_error(zresult)) "error ending compression stream: %s" % _zstd_error(zresult)
)
if zresult == 0: if zresult == 0:
self._finished_output = True self._finished_output = True
return ffi.buffer(out_buffer.dst, out_buffer.pos)[:] return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
def readinto(self, b): def readinto(self, b):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
if self._finished_output: if self._finished_output:
return 0 return 0
# TODO use writable=True once we require CFFI >= 1.12. # TODO use writable=True once we require CFFI >= 1.12.
dest_buffer = ffi.from_buffer(b) dest_buffer = ffi.from_buffer(b)
ffi.memmove(b, b'', 0) ffi.memmove(b, b"", 0)
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
out_buffer.dst = dest_buffer out_buffer.dst = dest_buffer
out_buffer.size = len(dest_buffer) out_buffer.size = len(dest_buffer)
out_buffer.pos = 0 out_buffer.pos = 0
if self._compress_into_buffer(out_buffer): if self._compress_into_buffer(out_buffer):
return out_buffer.pos return out_buffer.pos
while not self._finished_input: while not self._finished_input:
self._read_input() self._read_input()
if self._compress_into_buffer(out_buffer): if self._compress_into_buffer(out_buffer):
return out_buffer.pos return out_buffer.pos
# EOF. # EOF.
old_pos = out_buffer.pos old_pos = out_buffer.pos
zresult = lib.ZSTD_compressStream2(self._compressor._cctx, zresult = lib.ZSTD_compressStream2(
out_buffer, self._in_buffer, self._compressor._cctx, out_buffer, self._in_buffer, lib.ZSTD_e_end
lib.ZSTD_e_end) )
self._bytes_compressed += out_buffer.pos - old_pos self._bytes_compressed += out_buffer.pos - old_pos
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error ending compression stream: %s', raise ZstdError("error ending compression stream: %s", _zstd_error(zresult))
_zstd_error(zresult))
if zresult == 0: if zresult == 0:
self._finished_output = True self._finished_output = True
return out_buffer.pos return out_buffer.pos
def readinto1(self, b): def readinto1(self, b):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
if self._finished_output: if self._finished_output:
return 0 return 0
# TODO use writable=True once we require CFFI >= 1.12. # TODO use writable=True once we require CFFI >= 1.12.
dest_buffer = ffi.from_buffer(b) dest_buffer = ffi.from_buffer(b)
ffi.memmove(b, b'', 0) ffi.memmove(b, b"", 0)
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
out_buffer.dst = dest_buffer out_buffer.dst = dest_buffer
out_buffer.size = len(dest_buffer) out_buffer.size = len(dest_buffer)
out_buffer.pos = 0 out_buffer.pos = 0
self._compress_into_buffer(out_buffer) self._compress_into_buffer(out_buffer)
if out_buffer.pos: if out_buffer.pos:
return out_buffer.pos return out_buffer.pos
while not self._finished_input: while not self._finished_input:
self._read_input() self._read_input()
if self._compress_into_buffer(out_buffer): if self._compress_into_buffer(out_buffer):
return out_buffer.pos return out_buffer.pos
if out_buffer.pos and not self._finished_input: if out_buffer.pos and not self._finished_input:
return out_buffer.pos return out_buffer.pos
# EOF. # EOF.
old_pos = out_buffer.pos old_pos = out_buffer.pos
zresult = lib.ZSTD_compressStream2(self._compressor._cctx, zresult = lib.ZSTD_compressStream2(
out_buffer, self._in_buffer, self._compressor._cctx, out_buffer, self._in_buffer, lib.ZSTD_e_end
lib.ZSTD_e_end) )
self._bytes_compressed += out_buffer.pos - old_pos self._bytes_compressed += out_buffer.pos - old_pos
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error ending compression stream: %s' % raise ZstdError(
_zstd_error(zresult)) "error ending compression stream: %s" % _zstd_error(zresult)
)
if zresult == 0: if zresult == 0:
self._finished_output = True self._finished_output = True
return out_buffer.pos return out_buffer.pos
class ZstdCompressor(object): class ZstdCompressor(object):
def __init__(self, level=3, dict_data=None, compression_params=None, def __init__(
write_checksum=None, write_content_size=None, self,
write_dict_id=None, threads=0): level=3,
dict_data=None,
compression_params=None,
write_checksum=None,
write_content_size=None,
write_dict_id=None,
threads=0,
):
if level > lib.ZSTD_maxCLevel(): if level > lib.ZSTD_maxCLevel():
raise ValueError('level must be less than %d' % lib.ZSTD_maxCLevel()) raise ValueError("level must be less than %d" % lib.ZSTD_maxCLevel())
if threads < 0: if threads < 0:
threads = _cpu_count() threads = _cpu_count()
if compression_params and write_checksum is not None: if compression_params and write_checksum is not None:
raise ValueError('cannot define compression_params and ' raise ValueError("cannot define compression_params and " "write_checksum")
'write_checksum')
if compression_params and write_content_size is not None: if compression_params and write_content_size is not None:
raise ValueError('cannot define compression_params and ' raise ValueError(
'write_content_size') "cannot define compression_params and " "write_content_size"
)
if compression_params and write_dict_id is not None: if compression_params and write_dict_id is not None:
raise ValueError('cannot define compression_params and ' raise ValueError("cannot define compression_params and " "write_dict_id")
'write_dict_id')
if compression_params and threads: if compression_params and threads:
raise ValueError('cannot define compression_params and threads') raise ValueError("cannot define compression_params and threads")
if compression_params: if compression_params:
self._params = _make_cctx_params(compression_params) self._params = _make_cctx_params(compression_params)
else: else:
if write_dict_id is None: if write_dict_id is None:
write_dict_id = True write_dict_id = True
params = lib.ZSTD_createCCtxParams() params = lib.ZSTD_createCCtxParams()
if params == ffi.NULL: if params == ffi.NULL:
raise MemoryError() raise MemoryError()
self._params = ffi.gc(params, lib.ZSTD_freeCCtxParams) self._params = ffi.gc(params, lib.ZSTD_freeCCtxParams)
_set_compression_parameter(self._params, _set_compression_parameter(self._params, lib.ZSTD_c_compressionLevel, level)
lib.ZSTD_c_compressionLevel,
level)
_set_compression_parameter( _set_compression_parameter(
self._params, self._params,
lib.ZSTD_c_contentSizeFlag, lib.ZSTD_c_contentSizeFlag,
write_content_size if write_content_size is not None else 1) write_content_size if write_content_size is not None else 1,
)
_set_compression_parameter(self._params, _set_compression_parameter(
lib.ZSTD_c_checksumFlag, self._params, lib.ZSTD_c_checksumFlag, 1 if write_checksum else 0
1 if write_checksum else 0) )
_set_compression_parameter(self._params, _set_compression_parameter(
lib.ZSTD_c_dictIDFlag, self._params, lib.ZSTD_c_dictIDFlag, 1 if write_dict_id else 0
1 if write_dict_id else 0) )
if threads: if threads:
_set_compression_parameter(self._params, _set_compression_parameter(self._params, lib.ZSTD_c_nbWorkers, threads)
lib.ZSTD_c_nbWorkers,
threads)
cctx = lib.ZSTD_createCCtx() cctx = lib.ZSTD_createCCtx()
if cctx == ffi.NULL: if cctx == ffi.NULL:
raise MemoryError() raise MemoryError()
self._cctx = cctx self._cctx = cctx
self._dict_data = dict_data self._dict_data = dict_data
# We defer setting up garbage collection until after calling # We defer setting up garbage collection until after calling
# _setup_cctx() to ensure the memory size estimate is more accurate. # _setup_cctx() to ensure the memory size estimate is more accurate.
try: try:
self._setup_cctx() self._setup_cctx()
finally: finally:
self._cctx = ffi.gc(cctx, lib.ZSTD_freeCCtx, self._cctx = ffi.gc(
size=lib.ZSTD_sizeof_CCtx(cctx)) cctx, lib.ZSTD_freeCCtx, size=lib.ZSTD_sizeof_CCtx(cctx)
)
def _setup_cctx(self): def _setup_cctx(self):
zresult = lib.ZSTD_CCtx_setParametersUsingCCtxParams(self._cctx, zresult = lib.ZSTD_CCtx_setParametersUsingCCtxParams(self._cctx, self._params)
self._params)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('could not set compression parameters: %s' % raise ZstdError(
_zstd_error(zresult)) "could not set compression parameters: %s" % _zstd_error(zresult)
)
dict_data = self._dict_data dict_data = self._dict_data
if dict_data: if dict_data:
if dict_data._cdict: if dict_data._cdict:
zresult = lib.ZSTD_CCtx_refCDict(self._cctx, dict_data._cdict) zresult = lib.ZSTD_CCtx_refCDict(self._cctx, dict_data._cdict)
else: else:
zresult = lib.ZSTD_CCtx_loadDictionary_advanced( zresult = lib.ZSTD_CCtx_loadDictionary_advanced(
self._cctx, dict_data.as_bytes(), len(dict_data), self._cctx,
lib.ZSTD_dlm_byRef, dict_data._dict_type) dict_data.as_bytes(),
len(dict_data),
lib.ZSTD_dlm_byRef,
dict_data._dict_type,
)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('could not load compression dictionary: %s' % raise ZstdError(
_zstd_error(zresult)) "could not load compression dictionary: %s" % _zstd_error(zresult)
)
def memory_size(self): def memory_size(self):
return lib.ZSTD_sizeof_CCtx(self._cctx) return lib.ZSTD_sizeof_CCtx(self._cctx)
def compress(self, data): def compress(self, data):
lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only) lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
data_buffer = ffi.from_buffer(data) data_buffer = ffi.from_buffer(data)
dest_size = lib.ZSTD_compressBound(len(data_buffer)) dest_size = lib.ZSTD_compressBound(len(data_buffer))
out = new_nonzero('char[]', dest_size) out = new_nonzero("char[]", dest_size)
zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, len(data_buffer)) zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, len(data_buffer))
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error setting source size: %s' % raise ZstdError("error setting source size: %s" % _zstd_error(zresult))
_zstd_error(zresult))
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
in_buffer = ffi.new('ZSTD_inBuffer *') in_buffer = ffi.new("ZSTD_inBuffer *")
out_buffer.dst = out out_buffer.dst = out
out_buffer.size = dest_size out_buffer.size = dest_size
out_buffer.pos = 0 out_buffer.pos = 0
in_buffer.src = data_buffer in_buffer.src = data_buffer
in_buffer.size = len(data_buffer) in_buffer.size = len(data_buffer)
in_buffer.pos = 0 in_buffer.pos = 0
zresult = lib.ZSTD_compressStream2(self._cctx, zresult = lib.ZSTD_compressStream2(
out_buffer, self._cctx, out_buffer, in_buffer, lib.ZSTD_e_end
in_buffer, )
lib.ZSTD_e_end)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('cannot compress: %s' % raise ZstdError("cannot compress: %s" % _zstd_error(zresult))
_zstd_error(zresult))
elif zresult: elif zresult:
raise ZstdError('unexpected partial frame flush') raise ZstdError("unexpected partial frame flush")
return ffi.buffer(out, out_buffer.pos)[:] return ffi.buffer(out, out_buffer.pos)[:]
def compressobj(self, size=-1): def compressobj(self, size=-1):
lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only) lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
if size < 0: if size < 0:
size = lib.ZSTD_CONTENTSIZE_UNKNOWN size = lib.ZSTD_CONTENTSIZE_UNKNOWN
zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size) zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error setting source size: %s' % raise ZstdError("error setting source size: %s" % _zstd_error(zresult))
_zstd_error(zresult))
cobj = ZstdCompressionObj() cobj = ZstdCompressionObj()
cobj._out = ffi.new('ZSTD_outBuffer *') cobj._out = ffi.new("ZSTD_outBuffer *")
cobj._dst_buffer = ffi.new('char[]', COMPRESSION_RECOMMENDED_OUTPUT_SIZE) cobj._dst_buffer = ffi.new("char[]", COMPRESSION_RECOMMENDED_OUTPUT_SIZE)
cobj._out.dst = cobj._dst_buffer cobj._out.dst = cobj._dst_buffer
cobj._out.size = COMPRESSION_RECOMMENDED_OUTPUT_SIZE cobj._out.size = COMPRESSION_RECOMMENDED_OUTPUT_SIZE
cobj._out.pos = 0 cobj._out.pos = 0
cobj._compressor = self cobj._compressor = self
cobj._finished = False cobj._finished = False
return cobj return cobj
def chunker(self, size=-1, chunk_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE): def chunker(self, size=-1, chunk_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE):
lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only) lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
if size < 0: if size < 0:
size = lib.ZSTD_CONTENTSIZE_UNKNOWN size = lib.ZSTD_CONTENTSIZE_UNKNOWN
zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size) zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error setting source size: %s' % raise ZstdError("error setting source size: %s" % _zstd_error(zresult))
_zstd_error(zresult))
return ZstdCompressionChunker(self, chunk_size=chunk_size) return ZstdCompressionChunker(self, chunk_size=chunk_size)
def copy_stream(self, ifh, ofh, size=-1, def copy_stream(
self,
ifh,
ofh,
size=-1,
read_size=COMPRESSION_RECOMMENDED_INPUT_SIZE, read_size=COMPRESSION_RECOMMENDED_INPUT_SIZE,
write_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE): write_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE,
):
if not hasattr(ifh, 'read'): if not hasattr(ifh, "read"):
raise ValueError('first argument must have a read() method') raise ValueError("first argument must have a read() method")
if not hasattr(ofh, 'write'): if not hasattr(ofh, "write"):
raise ValueError('second argument must have a write() method') raise ValueError("second argument must have a write() method")
lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only) lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
if size < 0: if size < 0:
size = lib.ZSTD_CONTENTSIZE_UNKNOWN size = lib.ZSTD_CONTENTSIZE_UNKNOWN
zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size) zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error setting source size: %s' % raise ZstdError("error setting source size: %s" % _zstd_error(zresult))
_zstd_error(zresult))
in_buffer = ffi.new('ZSTD_inBuffer *') in_buffer = ffi.new("ZSTD_inBuffer *")
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
dst_buffer = ffi.new('char[]', write_size) dst_buffer = ffi.new("char[]", write_size)
out_buffer.dst = dst_buffer out_buffer.dst = dst_buffer
out_buffer.size = write_size out_buffer.size = write_size
out_buffer.pos = 0 out_buffer.pos = 0
total_read, total_write = 0, 0 total_read, total_write = 0, 0
while True: while True:
data = ifh.read(read_size) data = ifh.read(read_size)
if not data: if not data:
break break
data_buffer = ffi.from_buffer(data) data_buffer = ffi.from_buffer(data)
total_read += len(data_buffer) total_read += len(data_buffer)
in_buffer.src = data_buffer in_buffer.src = data_buffer
in_buffer.size = len(data_buffer) in_buffer.size = len(data_buffer)
in_buffer.pos = 0 in_buffer.pos = 0
while in_buffer.pos < in_buffer.size: while in_buffer.pos < in_buffer.size:
zresult = lib.ZSTD_compressStream2(self._cctx, zresult = lib.ZSTD_compressStream2(
out_buffer, self._cctx, out_buffer, in_buffer, lib.ZSTD_e_continue
in_buffer, )
lib.ZSTD_e_continue)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd compress error: %s' % raise ZstdError("zstd compress error: %s" % _zstd_error(zresult))
_zstd_error(zresult))
if out_buffer.pos: if out_buffer.pos:
ofh.write(ffi.buffer(out_buffer.dst, out_buffer.pos)) ofh.write(ffi.buffer(out_buffer.dst, out_buffer.pos))
total_write += out_buffer.pos total_write += out_buffer.pos
out_buffer.pos = 0 out_buffer.pos = 0
# We've finished reading. Flush the compressor. # We've finished reading. Flush the compressor.
while True: while True:
zresult = lib.ZSTD_compressStream2(self._cctx, zresult = lib.ZSTD_compressStream2(
out_buffer, self._cctx, out_buffer, in_buffer, lib.ZSTD_e_end
in_buffer, )
lib.ZSTD_e_end)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error ending compression stream: %s' % raise ZstdError(
_zstd_error(zresult)) "error ending compression stream: %s" % _zstd_error(zresult)
)
if out_buffer.pos: if out_buffer.pos:
ofh.write(ffi.buffer(out_buffer.dst, out_buffer.pos)) ofh.write(ffi.buffer(out_buffer.dst, out_buffer.pos))
total_write += out_buffer.pos total_write += out_buffer.pos
out_buffer.pos = 0 out_buffer.pos = 0
if zresult == 0: if zresult == 0:
break break
return total_read, total_write return total_read, total_write
def stream_reader(self, source, size=-1, def stream_reader(
read_size=COMPRESSION_RECOMMENDED_INPUT_SIZE): self, source, size=-1, read_size=COMPRESSION_RECOMMENDED_INPUT_SIZE
):
lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only) lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
try: try:
size = len(source) size = len(source)
except Exception: except Exception:
pass pass
if size < 0: if size < 0:
size = lib.ZSTD_CONTENTSIZE_UNKNOWN size = lib.ZSTD_CONTENTSIZE_UNKNOWN
zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size) zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error setting source size: %s' % raise ZstdError("error setting source size: %s" % _zstd_error(zresult))
_zstd_error(zresult))
return ZstdCompressionReader(self, source, read_size) return ZstdCompressionReader(self, source, read_size)
def stream_writer(self, writer, size=-1, def stream_writer(
self,
writer,
size=-1,
write_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE, write_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE,
write_return_read=False): write_return_read=False,
):
if not hasattr(writer, 'write'): if not hasattr(writer, "write"):
raise ValueError('must pass an object with a write() method') raise ValueError("must pass an object with a write() method")
lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only) lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
if size < 0: if size < 0:
size = lib.ZSTD_CONTENTSIZE_UNKNOWN size = lib.ZSTD_CONTENTSIZE_UNKNOWN
return ZstdCompressionWriter(self, writer, size, write_size, return ZstdCompressionWriter(self, writer, size, write_size, write_return_read)
write_return_read)
write_to = stream_writer write_to = stream_writer
def read_to_iter(self, reader, size=-1, def read_to_iter(
self,
reader,
size=-1,
read_size=COMPRESSION_RECOMMENDED_INPUT_SIZE, read_size=COMPRESSION_RECOMMENDED_INPUT_SIZE,
write_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE): write_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE,
if hasattr(reader, 'read'): ):
if hasattr(reader, "read"):
have_read = True have_read = True
elif hasattr(reader, '__getitem__'): elif hasattr(reader, "__getitem__"):
have_read = False have_read = False
buffer_offset = 0 buffer_offset = 0
size = len(reader) size = len(reader)
else: else:
raise ValueError('must pass an object with a read() method or ' raise ValueError(
'conforms to buffer protocol') "must pass an object with a read() method or "
"conforms to buffer protocol"
)
lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only) lib.ZSTD_CCtx_reset(self._cctx, lib.ZSTD_reset_session_only)
if size < 0: if size < 0:
size = lib.ZSTD_CONTENTSIZE_UNKNOWN size = lib.ZSTD_CONTENTSIZE_UNKNOWN
zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size) zresult = lib.ZSTD_CCtx_setPledgedSrcSize(self._cctx, size)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error setting source size: %s' % raise ZstdError("error setting source size: %s" % _zstd_error(zresult))
_zstd_error(zresult))
in_buffer = ffi.new('ZSTD_inBuffer *') in_buffer = ffi.new("ZSTD_inBuffer *")
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
in_buffer.src = ffi.NULL in_buffer.src = ffi.NULL
in_buffer.size = 0 in_buffer.size = 0
in_buffer.pos = 0 in_buffer.pos = 0
dst_buffer = ffi.new('char[]', write_size) dst_buffer = ffi.new("char[]", write_size)
out_buffer.dst = dst_buffer out_buffer.dst = dst_buffer
out_buffer.size = write_size out_buffer.size = write_size
out_buffer.pos = 0 out_buffer.pos = 0
while True: while True:
# We should never have output data sitting around after a previous # We should never have output data sitting around after a previous
# iteration. # iteration.
assert out_buffer.pos == 0 assert out_buffer.pos == 0
# Collect input data. # Collect input data.
if have_read: if have_read:
read_result = reader.read(read_size) read_result = reader.read(read_size)
else: else:
remaining = len(reader) - buffer_offset remaining = len(reader) - buffer_offset
slice_size = min(remaining, read_size) slice_size = min(remaining, read_size)
read_result = reader[buffer_offset:buffer_offset + slice_size] read_result = reader[buffer_offset : buffer_offset + slice_size]
buffer_offset += slice_size buffer_offset += slice_size
# No new input data. Break out of the read loop. # No new input data. Break out of the read loop.
if not read_result: if not read_result:
break break
# Feed all read data into the compressor and emit output until # Feed all read data into the compressor and emit output until
# exhausted. # exhausted.
read_buffer = ffi.from_buffer(read_result) read_buffer = ffi.from_buffer(read_result)
in_buffer.src = read_buffer in_buffer.src = read_buffer
in_buffer.size = len(read_buffer) in_buffer.size = len(read_buffer)
in_buffer.pos = 0 in_buffer.pos = 0
while in_buffer.pos < in_buffer.size: while in_buffer.pos < in_buffer.size:
zresult = lib.ZSTD_compressStream2(self._cctx, out_buffer, in_buffer, zresult = lib.ZSTD_compressStream2(
lib.ZSTD_e_continue) self._cctx, out_buffer, in_buffer, lib.ZSTD_e_continue
)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd compress error: %s' % raise ZstdError("zstd compress error: %s" % _zstd_error(zresult))
_zstd_error(zresult))
if out_buffer.pos: if out_buffer.pos:
data = ffi.buffer(out_buffer.dst, out_buffer.pos)[:] data = ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
out_buffer.pos = 0 out_buffer.pos = 0
yield data yield data
assert out_buffer.pos == 0 assert out_buffer.pos == 0
# And repeat the loop to collect more data. # And repeat the loop to collect more data.
continue continue
# If we get here, input is exhausted. End the stream and emit what # If we get here, input is exhausted. End the stream and emit what
# remains. # remains.
while True: while True:
assert out_buffer.pos == 0 assert out_buffer.pos == 0
zresult = lib.ZSTD_compressStream2(self._cctx, zresult = lib.ZSTD_compressStream2(
out_buffer, self._cctx, out_buffer, in_buffer, lib.ZSTD_e_end
in_buffer, )
lib.ZSTD_e_end)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('error ending compression stream: %s' % raise ZstdError(
_zstd_error(zresult)) "error ending compression stream: %s" % _zstd_error(zresult)
)
if out_buffer.pos: if out_buffer.pos:
data = ffi.buffer(out_buffer.dst, out_buffer.pos)[:] data = ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
out_buffer.pos = 0 out_buffer.pos = 0
yield data yield data
if zresult == 0: if zresult == 0:
break break
def frame_content_size(data): def frame_content_size(data):
data_buffer = ffi.from_buffer(data) data_buffer = ffi.from_buffer(data)
size = lib.ZSTD_getFrameContentSize(data_buffer, len(data_buffer)) size = lib.ZSTD_getFrameContentSize(data_buffer, len(data_buffer))
if size == lib.ZSTD_CONTENTSIZE_ERROR: if size == lib.ZSTD_CONTENTSIZE_ERROR:
raise ZstdError('error when determining content size') raise ZstdError("error when determining content size")
elif size == lib.ZSTD_CONTENTSIZE_UNKNOWN: elif size == lib.ZSTD_CONTENTSIZE_UNKNOWN:
return -1 return -1
else: else:
return size return size
def frame_header_size(data): def frame_header_size(data):
data_buffer = ffi.from_buffer(data) data_buffer = ffi.from_buffer(data)
zresult = lib.ZSTD_frameHeaderSize(data_buffer, len(data_buffer)) zresult = lib.ZSTD_frameHeaderSize(data_buffer, len(data_buffer))
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('could not determine frame header size: %s' % raise ZstdError(
_zstd_error(zresult)) "could not determine frame header size: %s" % _zstd_error(zresult)
)
return zresult return zresult
def get_frame_parameters(data): def get_frame_parameters(data):
params = ffi.new('ZSTD_frameHeader *') params = ffi.new("ZSTD_frameHeader *")
data_buffer = ffi.from_buffer(data) data_buffer = ffi.from_buffer(data)
zresult = lib.ZSTD_getFrameHeader(params, data_buffer, len(data_buffer)) zresult = lib.ZSTD_getFrameHeader(params, data_buffer, len(data_buffer))
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('cannot get frame parameters: %s' % raise ZstdError("cannot get frame parameters: %s" % _zstd_error(zresult))
_zstd_error(zresult))
if zresult: if zresult:
raise ZstdError('not enough data for frame parameters; need %d bytes' % raise ZstdError("not enough data for frame parameters; need %d bytes" % zresult)
zresult)
return FrameParameters(params[0]) return FrameParameters(params[0])
class ZstdCompressionDict(object): class ZstdCompressionDict(object):
def __init__(self, data, dict_type=DICT_TYPE_AUTO, k=0, d=0): def __init__(self, data, dict_type=DICT_TYPE_AUTO, k=0, d=0):
assert isinstance(data, bytes_type) assert isinstance(data, bytes_type)
self._data = data self._data = data
self.k = k self.k = k
self.d = d self.d = d
if dict_type not in (DICT_TYPE_AUTO, DICT_TYPE_RAWCONTENT, if dict_type not in (DICT_TYPE_AUTO, DICT_TYPE_RAWCONTENT, DICT_TYPE_FULLDICT):
DICT_TYPE_FULLDICT): raise ValueError(
raise ValueError('invalid dictionary load mode: %d; must use ' "invalid dictionary load mode: %d; must use " "DICT_TYPE_* constants"
'DICT_TYPE_* constants') )
self._dict_type = dict_type self._dict_type = dict_type
self._cdict = None self._cdict = None
def __len__(self): def __len__(self):
return len(self._data) return len(self._data)
def dict_id(self): def dict_id(self):
return int_type(lib.ZDICT_getDictID(self._data, len(self._data))) return int_type(lib.ZDICT_getDictID(self._data, len(self._data)))
def as_bytes(self): def as_bytes(self):
return self._data return self._data
def precompute_compress(self, level=0, compression_params=None): def precompute_compress(self, level=0, compression_params=None):
if level and compression_params: if level and compression_params:
raise ValueError('must only specify one of level or ' raise ValueError("must only specify one of level or " "compression_params")
'compression_params')
if not level and not compression_params: if not level and not compression_params:
raise ValueError('must specify one of level or compression_params') raise ValueError("must specify one of level or compression_params")
if level: if level:
cparams = lib.ZSTD_getCParams(level, 0, len(self._data)) cparams = lib.ZSTD_getCParams(level, 0, len(self._data))
else: else:
cparams = ffi.new('ZSTD_compressionParameters') cparams = ffi.new("ZSTD_compressionParameters")
cparams.chainLog = compression_params.chain_log cparams.chainLog = compression_params.chain_log
cparams.hashLog = compression_params.hash_log cparams.hashLog = compression_params.hash_log
cparams.minMatch = compression_params.min_match cparams.minMatch = compression_params.min_match
cparams.searchLog = compression_params.search_log cparams.searchLog = compression_params.search_log
cparams.strategy = compression_params.compression_strategy cparams.strategy = compression_params.compression_strategy
cparams.targetLength = compression_params.target_length cparams.targetLength = compression_params.target_length
cparams.windowLog = compression_params.window_log cparams.windowLog = compression_params.window_log
cdict = lib.ZSTD_createCDict_advanced(self._data, len(self._data), cdict = lib.ZSTD_createCDict_advanced(
self._data,
len(self._data),
lib.ZSTD_dlm_byRef, lib.ZSTD_dlm_byRef,
self._dict_type, self._dict_type,
cparams, cparams,
lib.ZSTD_defaultCMem) lib.ZSTD_defaultCMem,
)
if cdict == ffi.NULL: if cdict == ffi.NULL:
raise ZstdError('unable to precompute dictionary') raise ZstdError("unable to precompute dictionary")
self._cdict = ffi.gc(cdict, lib.ZSTD_freeCDict, self._cdict = ffi.gc(
size=lib.ZSTD_sizeof_CDict(cdict)) cdict, lib.ZSTD_freeCDict, size=lib.ZSTD_sizeof_CDict(cdict)
)
@property @property
def _ddict(self): def _ddict(self):
ddict = lib.ZSTD_createDDict_advanced(self._data, len(self._data), ddict = lib.ZSTD_createDDict_advanced(
self._data,
len(self._data),
lib.ZSTD_dlm_byRef, lib.ZSTD_dlm_byRef,
self._dict_type, self._dict_type,
lib.ZSTD_defaultCMem) lib.ZSTD_defaultCMem,
)
if ddict == ffi.NULL: if ddict == ffi.NULL:
raise ZstdError('could not create decompression dict') raise ZstdError("could not create decompression dict")
ddict = ffi.gc(ddict, lib.ZSTD_freeDDict, ddict = ffi.gc(ddict, lib.ZSTD_freeDDict, size=lib.ZSTD_sizeof_DDict(ddict))
size=lib.ZSTD_sizeof_DDict(ddict)) self.__dict__["_ddict"] = ddict
self.__dict__['_ddict'] = ddict
return ddict return ddict
def train_dictionary(dict_size, samples, k=0, d=0, notifications=0, dict_id=0,
level=0, steps=0, threads=0): def train_dictionary(
dict_size,
samples,
k=0,
d=0,
notifications=0,
dict_id=0,
level=0,
steps=0,
threads=0,
):
if not isinstance(samples, list): if not isinstance(samples, list):
raise TypeError('samples must be a list') raise TypeError("samples must be a list")
if threads < 0: if threads < 0:
threads = _cpu_count() threads = _cpu_count()
total_size = sum(map(len, samples)) total_size = sum(map(len, samples))
samples_buffer = new_nonzero('char[]', total_size) samples_buffer = new_nonzero("char[]", total_size)
sample_sizes = new_nonzero('size_t[]', len(samples)) sample_sizes = new_nonzero("size_t[]", len(samples))
offset = 0 offset = 0
for i, sample in enumerate(samples): for i, sample in enumerate(samples):
if not isinstance(sample, bytes_type): if not isinstance(sample, bytes_type):
raise ValueError('samples must be bytes') raise ValueError("samples must be bytes")
l = len(sample) l = len(sample)
ffi.memmove(samples_buffer + offset, sample, l) ffi.memmove(samples_buffer + offset, sample, l)
offset += l offset += l
sample_sizes[i] = l sample_sizes[i] = l
dict_data = new_nonzero('char[]', dict_size) dict_data = new_nonzero("char[]", dict_size)
dparams = ffi.new('ZDICT_cover_params_t *')[0] dparams = ffi.new("ZDICT_cover_params_t *")[0]
dparams.k = k dparams.k = k
dparams.d = d dparams.d = d
dparams.steps = steps dparams.steps = steps
dparams.nbThreads = threads dparams.nbThreads = threads
dparams.zParams.notificationLevel = notifications dparams.zParams.notificationLevel = notifications
dparams.zParams.dictID = dict_id dparams.zParams.dictID = dict_id
dparams.zParams.compressionLevel = level dparams.zParams.compressionLevel = level
if (not dparams.k and not dparams.d and not dparams.steps if (
and not dparams.nbThreads and not dparams.zParams.notificationLevel not dparams.k
and not dparams.d
and not dparams.steps
and not dparams.nbThreads
and not dparams.zParams.notificationLevel
and not dparams.zParams.dictID and not dparams.zParams.dictID
and not dparams.zParams.compressionLevel): and not dparams.zParams.compressionLevel
):
zresult = lib.ZDICT_trainFromBuffer( zresult = lib.ZDICT_trainFromBuffer(
ffi.addressof(dict_data), dict_size, ffi.addressof(dict_data),
dict_size,
ffi.addressof(samples_buffer), ffi.addressof(samples_buffer),
ffi.addressof(sample_sizes, 0), len(samples)) ffi.addressof(sample_sizes, 0),
len(samples),
)
elif dparams.steps or dparams.nbThreads: elif dparams.steps or dparams.nbThreads:
zresult = lib.ZDICT_optimizeTrainFromBuffer_cover( zresult = lib.ZDICT_optimizeTrainFromBuffer_cover(
ffi.addressof(dict_data), dict_size, ffi.addressof(dict_data),
dict_size,
ffi.addressof(samples_buffer), ffi.addressof(samples_buffer),
ffi.addressof(sample_sizes, 0), len(samples), ffi.addressof(sample_sizes, 0),
ffi.addressof(dparams)) len(samples),
ffi.addressof(dparams),
)
else: else:
zresult = lib.ZDICT_trainFromBuffer_cover( zresult = lib.ZDICT_trainFromBuffer_cover(
ffi.addressof(dict_data), dict_size, ffi.addressof(dict_data),
dict_size,
ffi.addressof(samples_buffer), ffi.addressof(samples_buffer),
ffi.addressof(sample_sizes, 0), len(samples), ffi.addressof(sample_sizes, 0),
dparams) len(samples),
dparams,
)
if lib.ZDICT_isError(zresult): if lib.ZDICT_isError(zresult):
msg = ffi.string(lib.ZDICT_getErrorName(zresult)).decode('utf-8') msg = ffi.string(lib.ZDICT_getErrorName(zresult)).decode("utf-8")
raise ZstdError('cannot train dict: %s' % msg) raise ZstdError("cannot train dict: %s" % msg)
return ZstdCompressionDict(ffi.buffer(dict_data, zresult)[:], return ZstdCompressionDict(
ffi.buffer(dict_data, zresult)[:],
dict_type=DICT_TYPE_FULLDICT, dict_type=DICT_TYPE_FULLDICT,
k=dparams.k, d=dparams.d) k=dparams.k,
d=dparams.d,
)
class ZstdDecompressionObj(object): class ZstdDecompressionObj(object):
def __init__(self, decompressor, write_size): def __init__(self, decompressor, write_size):
self._decompressor = decompressor self._decompressor = decompressor
self._write_size = write_size self._write_size = write_size
self._finished = False self._finished = False
def decompress(self, data): def decompress(self, data):
if self._finished: if self._finished:
raise ZstdError('cannot use a decompressobj multiple times') raise ZstdError("cannot use a decompressobj multiple times")
in_buffer = ffi.new('ZSTD_inBuffer *') in_buffer = ffi.new("ZSTD_inBuffer *")
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
data_buffer = ffi.from_buffer(data) data_buffer = ffi.from_buffer(data)
if len(data_buffer) == 0: if len(data_buffer) == 0:
return b'' return b""
in_buffer.src = data_buffer in_buffer.src = data_buffer
in_buffer.size = len(data_buffer) in_buffer.size = len(data_buffer)
in_buffer.pos = 0 in_buffer.pos = 0
dst_buffer = ffi.new('char[]', self._write_size) dst_buffer = ffi.new("char[]", self._write_size)
out_buffer.dst = dst_buffer out_buffer.dst = dst_buffer
out_buffer.size = len(dst_buffer) out_buffer.size = len(dst_buffer)
out_buffer.pos = 0 out_buffer.pos = 0
chunks = [] chunks = []
while True: while True:
zresult = lib.ZSTD_decompressStream(self._decompressor._dctx, zresult = lib.ZSTD_decompressStream(
out_buffer, in_buffer) self._decompressor._dctx, out_buffer, in_buffer
)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd decompressor error: %s' % raise ZstdError("zstd decompressor error: %s" % _zstd_error(zresult))
_zstd_error(zresult))
if zresult == 0: if zresult == 0:
self._finished = True self._finished = True
self._decompressor = None self._decompressor = None
if out_buffer.pos: if out_buffer.pos:
chunks.append(ffi.buffer(out_buffer.dst, out_buffer.pos)[:]) chunks.append(ffi.buffer(out_buffer.dst, out_buffer.pos)[:])
if (zresult == 0 or if zresult == 0 or (
(in_buffer.pos == in_buffer.size and out_buffer.pos == 0)): in_buffer.pos == in_buffer.size and out_buffer.pos == 0
):
break break
out_buffer.pos = 0 out_buffer.pos = 0
return b''.join(chunks) return b"".join(chunks)
def flush(self, length=0): def flush(self, length=0):
pass pass
class ZstdDecompressionReader(object): class ZstdDecompressionReader(object):
def __init__(self, decompressor, source, read_size, read_across_frames): def __init__(self, decompressor, source, read_size, read_across_frames):
self._decompressor = decompressor self._decompressor = decompressor
self._source = source self._source = source
self._read_size = read_size self._read_size = read_size
self._read_across_frames = bool(read_across_frames) self._read_across_frames = bool(read_across_frames)
self._entered = False self._entered = False
self._closed = False self._closed = False
self._bytes_decompressed = 0 self._bytes_decompressed = 0
self._finished_input = False self._finished_input = False
self._finished_output = False self._finished_output = False
self._in_buffer = ffi.new('ZSTD_inBuffer *') self._in_buffer = ffi.new("ZSTD_inBuffer *")
# Holds a ref to self._in_buffer.src. # Holds a ref to self._in_buffer.src.
self._source_buffer = None self._source_buffer = None
def __enter__(self): def __enter__(self):
if self._entered: if self._entered:
raise ValueError('cannot __enter__ multiple times') raise ValueError("cannot __enter__ multiple times")
self._entered = True self._entered = True
return self return self
def __exit__(self, exc_type, exc_value, exc_tb): def __exit__(self, exc_type, exc_value, exc_tb):
self._entered = False self._entered = False
self._closed = True self._closed = True
self._source = None self._source = None
while True: while True:
chunk = self.read(1048576) chunk = self.read(1048576)
if not chunk: if not chunk:
break break
chunks.append(chunk) chunks.append(chunk)
return b''.join(chunks) return b"".join(chunks)
def __iter__(self): def __iter__(self):
raise io.UnsupportedOperation() raise io.UnsupportedOperation()
def __next__(self): def __next__(self):
raise io.UnsupportedOperation() raise io.UnsupportedOperation()
next = __next__ next = __next__
def _read_input(self): def _read_input(self):
# We have data left over in the input buffer. Use it. # We have data left over in the input buffer. Use it.
if self._in_buffer.pos < self._in_buffer.size: if self._in_buffer.pos < self._in_buffer.size:
return return
# All input data exhausted. Nothing to do. # All input data exhausted. Nothing to do.
if self._finished_input: if self._finished_input:
return return
# Else populate the input buffer from our source. # Else populate the input buffer from our source.
if hasattr(self._source, 'read'): if hasattr(self._source, "read"):
data = self._source.read(self._read_size) data = self._source.read(self._read_size)
if not data: if not data:
self._finished_input = True self._finished_input = True
return return
self._source_buffer = ffi.from_buffer(data) self._source_buffer = ffi.from_buffer(data)
self._in_buffer.src = self._source_buffer self._in_buffer.src = self._source_buffer
self._in_buffer.size = len(self._source_buffer) self._in_buffer.size = len(self._source_buffer)
self._in_buffer.pos = 0 self._in_buffer.pos = 0
else: else:
self._source_buffer = ffi.from_buffer(self._source) self._source_buffer = ffi.from_buffer(self._source)
self._in_buffer.src = self._source_buffer self._in_buffer.src = self._source_buffer
self._in_buffer.size = len(self._source_buffer) self._in_buffer.size = len(self._source_buffer)
self._in_buffer.pos = 0 self._in_buffer.pos = 0
def _decompress_into_buffer(self, out_buffer): def _decompress_into_buffer(self, out_buffer):
"""Decompress available input into an output buffer. """Decompress available input into an output buffer.
Returns True if data in output buffer should be emitted. Returns True if data in output buffer should be emitted.
""" """
zresult = lib.ZSTD_decompressStream(self._decompressor._dctx, zresult = lib.ZSTD_decompressStream(
out_buffer, self._in_buffer) self._decompressor._dctx, out_buffer, self._in_buffer
)
if self._in_buffer.pos == self._in_buffer.size: if self._in_buffer.pos == self._in_buffer.size:
self._in_buffer.src = ffi.NULL self._in_buffer.src = ffi.NULL
self._in_buffer.pos = 0 self._in_buffer.pos = 0
self._in_buffer.size = 0 self._in_buffer.size = 0
self._source_buffer = None self._source_buffer = None
if not hasattr(self._source, 'read'): if not hasattr(self._source, "read"):
self._finished_input = True self._finished_input = True
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd decompress error: %s' % raise ZstdError("zstd decompress error: %s" % _zstd_error(zresult))
_zstd_error(zresult))
# Emit data if there is data AND either: # Emit data if there is data AND either:
# a) output buffer is full (read amount is satisfied) # a) output buffer is full (read amount is satisfied)
# b) we're at end of a frame and not in frame spanning mode # b) we're at end of a frame and not in frame spanning mode
return (out_buffer.pos and return out_buffer.pos and (
(out_buffer.pos == out_buffer.size or out_buffer.pos == out_buffer.size
zresult == 0 and not self._read_across_frames)) or zresult == 0
and not self._read_across_frames
)
def read(self, size=-1): def read(self, size=-1):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
if size < -1: if size < -1:
raise ValueError('cannot read negative amounts less than -1') raise ValueError("cannot read negative amounts less than -1")
if size == -1: if size == -1:
# This is recursive. But it gets the job done. # This is recursive. But it gets the job done.
return self.readall() return self.readall()
if self._finished_output or size == 0: if self._finished_output or size == 0:
return b'' return b""
# We /could/ call into readinto() here. But that introduces more # We /could/ call into readinto() here. But that introduces more
# overhead. # overhead.
dst_buffer = ffi.new('char[]', size) dst_buffer = ffi.new("char[]", size)
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
out_buffer.dst = dst_buffer out_buffer.dst = dst_buffer
out_buffer.size = size out_buffer.size = size
out_buffer.pos = 0 out_buffer.pos = 0
self._read_input() self._read_input()
if self._decompress_into_buffer(out_buffer): if self._decompress_into_buffer(out_buffer):
self._bytes_decompressed += out_buffer.pos self._bytes_decompressed += out_buffer.pos
return ffi.buffer(out_buffer.dst, out_buffer.pos)[:] return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
while not self._finished_input: while not self._finished_input:
self._read_input() self._read_input()
if self._decompress_into_buffer(out_buffer): if self._decompress_into_buffer(out_buffer):
self._bytes_decompressed += out_buffer.pos self._bytes_decompressed += out_buffer.pos
return ffi.buffer(out_buffer.dst, out_buffer.pos)[:] return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
self._bytes_decompressed += out_buffer.pos self._bytes_decompressed += out_buffer.pos
return ffi.buffer(out_buffer.dst, out_buffer.pos)[:] return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
def readinto(self, b): def readinto(self, b):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
if self._finished_output: if self._finished_output:
return 0 return 0
# TODO use writable=True once we require CFFI >= 1.12. # TODO use writable=True once we require CFFI >= 1.12.
dest_buffer = ffi.from_buffer(b) dest_buffer = ffi.from_buffer(b)
ffi.memmove(b, b'', 0) ffi.memmove(b, b"", 0)
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
out_buffer.dst = dest_buffer out_buffer.dst = dest_buffer
out_buffer.size = len(dest_buffer) out_buffer.size = len(dest_buffer)
out_buffer.pos = 0 out_buffer.pos = 0
self._read_input() self._read_input()
if self._decompress_into_buffer(out_buffer): if self._decompress_into_buffer(out_buffer):
self._bytes_decompressed += out_buffer.pos self._bytes_decompressed += out_buffer.pos
return out_buffer.pos return out_buffer.pos
while not self._finished_input: while not self._finished_input:
self._read_input() self._read_input()
if self._decompress_into_buffer(out_buffer): if self._decompress_into_buffer(out_buffer):
self._bytes_decompressed += out_buffer.pos self._bytes_decompressed += out_buffer.pos
return out_buffer.pos return out_buffer.pos
self._bytes_decompressed += out_buffer.pos self._bytes_decompressed += out_buffer.pos
return out_buffer.pos return out_buffer.pos
def read1(self, size=-1): def read1(self, size=-1):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
if size < -1: if size < -1:
raise ValueError('cannot read negative amounts less than -1') raise ValueError("cannot read negative amounts less than -1")
if self._finished_output or size == 0: if self._finished_output or size == 0:
return b'' return b""
# -1 returns arbitrary number of bytes. # -1 returns arbitrary number of bytes.
if size == -1: if size == -1:
size = DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE size = DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE
dst_buffer = ffi.new('char[]', size) dst_buffer = ffi.new("char[]", size)
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
out_buffer.dst = dst_buffer out_buffer.dst = dst_buffer
out_buffer.size = size out_buffer.size = size
out_buffer.pos = 0 out_buffer.pos = 0
# read1() dictates that we can perform at most 1 call to underlying # read1() dictates that we can perform at most 1 call to underlying
# stream to get input. However, we can't satisfy this restriction with # stream to get input. However, we can't satisfy this restriction with
# decompression because not all input generates output. So we allow # decompression because not all input generates output. So we allow
# multiple read(). But unlike read(), we stop once we have any output. # multiple read(). But unlike read(), we stop once we have any output.
while not self._finished_input: while not self._finished_input:
self._read_input() self._read_input()
self._decompress_into_buffer(out_buffer) self._decompress_into_buffer(out_buffer)
if out_buffer.pos: if out_buffer.pos:
break break
self._bytes_decompressed += out_buffer.pos self._bytes_decompressed += out_buffer.pos
return ffi.buffer(out_buffer.dst, out_buffer.pos)[:] return ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
def readinto1(self, b): def readinto1(self, b):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
if self._finished_output: if self._finished_output:
return 0 return 0
# TODO use writable=True once we require CFFI >= 1.12. # TODO use writable=True once we require CFFI >= 1.12.
dest_buffer = ffi.from_buffer(b) dest_buffer = ffi.from_buffer(b)
ffi.memmove(b, b'', 0) ffi.memmove(b, b"", 0)
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
out_buffer.dst = dest_buffer out_buffer.dst = dest_buffer
out_buffer.size = len(dest_buffer) out_buffer.size = len(dest_buffer)
out_buffer.pos = 0 out_buffer.pos = 0
while not self._finished_input and not self._finished_output: while not self._finished_input and not self._finished_output:
self._read_input() self._read_input()
self._decompress_into_buffer(out_buffer) self._decompress_into_buffer(out_buffer)
if out_buffer.pos: if out_buffer.pos:
break break
self._bytes_decompressed += out_buffer.pos self._bytes_decompressed += out_buffer.pos
return out_buffer.pos return out_buffer.pos
def seek(self, pos, whence=os.SEEK_SET): def seek(self, pos, whence=os.SEEK_SET):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
read_amount = 0 read_amount = 0
if whence == os.SEEK_SET: if whence == os.SEEK_SET:
if pos < 0: if pos < 0:
raise ValueError('cannot seek to negative position with SEEK_SET') raise ValueError("cannot seek to negative position with SEEK_SET")
if pos < self._bytes_decompressed: if pos < self._bytes_decompressed:
raise ValueError('cannot seek zstd decompression stream ' raise ValueError("cannot seek zstd decompression stream " "backwards")
'backwards')
read_amount = pos - self._bytes_decompressed read_amount = pos - self._bytes_decompressed
elif whence == os.SEEK_CUR: elif whence == os.SEEK_CUR:
if pos < 0: if pos < 0:
raise ValueError('cannot seek zstd decompression stream ' raise ValueError("cannot seek zstd decompression stream " "backwards")
'backwards')
read_amount = pos read_amount = pos
elif whence == os.SEEK_END: elif whence == os.SEEK_END:
raise ValueError('zstd decompression streams cannot be seeked ' raise ValueError(
'with SEEK_END') "zstd decompression streams cannot be seeked " "with SEEK_END"
)
while read_amount: while read_amount:
result = self.read(min(read_amount, result = self.read(min(read_amount, DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE))
DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE))
if not result: if not result:
break break
read_amount -= len(result) read_amount -= len(result)
return self._bytes_decompressed return self._bytes_decompressed
class ZstdDecompressionWriter(object): class ZstdDecompressionWriter(object):
def __init__(self, decompressor, writer, write_size, write_return_read): def __init__(self, decompressor, writer, write_size, write_return_read):
decompressor._ensure_dctx() decompressor._ensure_dctx()
self._decompressor = decompressor self._decompressor = decompressor
self._writer = writer self._writer = writer
self._write_size = write_size self._write_size = write_size
self._write_return_read = bool(write_return_read) self._write_return_read = bool(write_return_read)
self._entered = False self._entered = False
self._closed = False self._closed = False
def __enter__(self): def __enter__(self):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
if self._entered: if self._entered:
raise ZstdError('cannot __enter__ multiple times') raise ZstdError("cannot __enter__ multiple times")
self._entered = True self._entered = True
return self return self
def __exit__(self, exc_type, exc_value, exc_tb): def __exit__(self, exc_type, exc_value, exc_tb):
self._entered = False self._entered = False
self.close() self.close()
def memory_size(self): def memory_size(self):
return lib.ZSTD_sizeof_DCtx(self._decompressor._dctx) return lib.ZSTD_sizeof_DCtx(self._decompressor._dctx)
def close(self): def close(self):
if self._closed: if self._closed:
return return
try: try:
self.flush() self.flush()
finally: finally:
self._closed = True self._closed = True
f = getattr(self._writer, 'close', None) f = getattr(self._writer, "close", None)
if f: if f:
f() f()
@property @property
def closed(self): def closed(self):
return self._closed return self._closed
def fileno(self): def fileno(self):
f = getattr(self._writer, 'fileno', None) f = getattr(self._writer, "fileno", None)
if f: if f:
return f() return f()
else: else:
raise OSError('fileno not available on underlying writer') raise OSError("fileno not available on underlying writer")
def flush(self): def flush(self):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
f = getattr(self._writer, 'flush', None) f = getattr(self._writer, "flush", None)
if f: if f:
return f() return f()
def isatty(self): def isatty(self):
return False return False
def readable(self): def readable(self):
return False return False
def readall(self): def readall(self):
raise io.UnsupportedOperation() raise io.UnsupportedOperation()
def readinto(self, b): def readinto(self, b):
raise io.UnsupportedOperation() raise io.UnsupportedOperation()
def write(self, data): def write(self, data):
if self._closed: if self._closed:
raise ValueError('stream is closed') raise ValueError("stream is closed")
total_write = 0 total_write = 0
in_buffer = ffi.new('ZSTD_inBuffer *') in_buffer = ffi.new("ZSTD_inBuffer *")
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
data_buffer = ffi.from_buffer(data) data_buffer = ffi.from_buffer(data)
in_buffer.src = data_buffer in_buffer.src = data_buffer
in_buffer.size = len(data_buffer) in_buffer.size = len(data_buffer)
in_buffer.pos = 0 in_buffer.pos = 0
dst_buffer = ffi.new('char[]', self._write_size) dst_buffer = ffi.new("char[]", self._write_size)
out_buffer.dst = dst_buffer out_buffer.dst = dst_buffer
out_buffer.size = len(dst_buffer) out_buffer.size = len(dst_buffer)
out_buffer.pos = 0 out_buffer.pos = 0
dctx = self._decompressor._dctx dctx = self._decompressor._dctx
while in_buffer.pos < in_buffer.size: while in_buffer.pos < in_buffer.size:
zresult = lib.ZSTD_decompressStream(dctx, out_buffer, in_buffer) zresult = lib.ZSTD_decompressStream(dctx, out_buffer, in_buffer)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd decompress error: %s' % raise ZstdError("zstd decompress error: %s" % _zstd_error(zresult))
_zstd_error(zresult))
if out_buffer.pos: if out_buffer.pos:
self._writer.write(ffi.buffer(out_buffer.dst, out_buffer.pos)[:]) self._writer.write(ffi.buffer(out_buffer.dst, out_buffer.pos)[:])
total_write += out_buffer.pos total_write += out_buffer.pos
out_buffer.pos = 0 out_buffer.pos = 0
if self._write_return_read: if self._write_return_read:
return in_buffer.pos return in_buffer.pos
self._dctx = dctx self._dctx = dctx
# Defer setting up garbage collection until full state is loaded so # Defer setting up garbage collection until full state is loaded so
# the memory size is more accurate. # the memory size is more accurate.
try: try:
self._ensure_dctx() self._ensure_dctx()
finally: finally:
self._dctx = ffi.gc(dctx, lib.ZSTD_freeDCtx, self._dctx = ffi.gc(
size=lib.ZSTD_sizeof_DCtx(dctx)) dctx, lib.ZSTD_freeDCtx, size=lib.ZSTD_sizeof_DCtx(dctx)
)
def memory_size(self): def memory_size(self):
return lib.ZSTD_sizeof_DCtx(self._dctx) return lib.ZSTD_sizeof_DCtx(self._dctx)
def decompress(self, data, max_output_size=0): def decompress(self, data, max_output_size=0):
self._ensure_dctx() self._ensure_dctx()
data_buffer = ffi.from_buffer(data) data_buffer = ffi.from_buffer(data)
output_size = lib.ZSTD_getFrameContentSize(data_buffer, len(data_buffer)) output_size = lib.ZSTD_getFrameContentSize(data_buffer, len(data_buffer))
if output_size == lib.ZSTD_CONTENTSIZE_ERROR: if output_size == lib.ZSTD_CONTENTSIZE_ERROR:
raise ZstdError('error determining content size from frame header') raise ZstdError("error determining content size from frame header")
elif output_size == 0: elif output_size == 0:
return b'' return b""
elif output_size == lib.ZSTD_CONTENTSIZE_UNKNOWN: elif output_size == lib.ZSTD_CONTENTSIZE_UNKNOWN:
if not max_output_size: if not max_output_size:
raise ZstdError('could not determine content size in frame header') raise ZstdError("could not determine content size in frame header")
result_buffer = ffi.new('char[]', max_output_size) result_buffer = ffi.new("char[]", max_output_size)
result_size = max_output_size result_size = max_output_size
output_size = 0 output_size = 0
else: else:
result_buffer = ffi.new('char[]', output_size) result_buffer = ffi.new("char[]", output_size)
result_size = output_size result_size = output_size
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
out_buffer.dst = result_buffer out_buffer.dst = result_buffer
out_buffer.size = result_size out_buffer.size = result_size
out_buffer.pos = 0 out_buffer.pos = 0
in_buffer = ffi.new('ZSTD_inBuffer *') in_buffer = ffi.new("ZSTD_inBuffer *")
in_buffer.src = data_buffer in_buffer.src = data_buffer
in_buffer.size = len(data_buffer) in_buffer.size = len(data_buffer)
in_buffer.pos = 0 in_buffer.pos = 0
zresult = lib.ZSTD_decompressStream(self._dctx, out_buffer, in_buffer) zresult = lib.ZSTD_decompressStream(self._dctx, out_buffer, in_buffer)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('decompression error: %s' % raise ZstdError("decompression error: %s" % _zstd_error(zresult))
_zstd_error(zresult))
elif zresult: elif zresult:
raise ZstdError('decompression error: did not decompress full frame') raise ZstdError("decompression error: did not decompress full frame")
elif output_size and out_buffer.pos != output_size: elif output_size and out_buffer.pos != output_size:
raise ZstdError('decompression error: decompressed %d bytes; expected %d' % raise ZstdError(
(zresult, output_size)) "decompression error: decompressed %d bytes; expected %d"
% (zresult, output_size)
)
return ffi.buffer(result_buffer, out_buffer.pos)[:] return ffi.buffer(result_buffer, out_buffer.pos)[:]
def stream_reader(self, source, read_size=DECOMPRESSION_RECOMMENDED_INPUT_SIZE, def stream_reader(
read_across_frames=False): self,
source,
read_size=DECOMPRESSION_RECOMMENDED_INPUT_SIZE,
read_across_frames=False,
):
self._ensure_dctx() self._ensure_dctx()
return ZstdDecompressionReader(self, source, read_size, read_across_frames) return ZstdDecompressionReader(self, source, read_size, read_across_frames)
def decompressobj(self, write_size=DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE): def decompressobj(self, write_size=DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE):
if write_size < 1: if write_size < 1:
raise ValueError('write_size must be positive') raise ValueError("write_size must be positive")
self._ensure_dctx() self._ensure_dctx()
return ZstdDecompressionObj(self, write_size=write_size) return ZstdDecompressionObj(self, write_size=write_size)
def read_to_iter(self, reader, read_size=DECOMPRESSION_RECOMMENDED_INPUT_SIZE, def read_to_iter(
self,
reader,
read_size=DECOMPRESSION_RECOMMENDED_INPUT_SIZE,
write_size=DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE, write_size=DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE,
skip_bytes=0): skip_bytes=0,
):
if skip_bytes >= read_size: if skip_bytes >= read_size:
raise ValueError('skip_bytes must be smaller than read_size') raise ValueError("skip_bytes must be smaller than read_size")
if hasattr(reader, 'read'): if hasattr(reader, "read"):
have_read = True have_read = True
elif hasattr(reader, '__getitem__'): elif hasattr(reader, "__getitem__"):
have_read = False have_read = False
buffer_offset = 0 buffer_offset = 0
size = len(reader) size = len(reader)
else: else:
raise ValueError('must pass an object with a read() method or ' raise ValueError(
'conforms to buffer protocol') "must pass an object with a read() method or "
"conforms to buffer protocol"
)
if skip_bytes: if skip_bytes:
if have_read: if have_read:
reader.read(skip_bytes) reader.read(skip_bytes)
else: else:
if skip_bytes > size: if skip_bytes > size:
raise ValueError('skip_bytes larger than first input chunk') raise ValueError("skip_bytes larger than first input chunk")
buffer_offset = skip_bytes buffer_offset = skip_bytes
self._ensure_dctx() self._ensure_dctx()
in_buffer = ffi.new('ZSTD_inBuffer *') in_buffer = ffi.new("ZSTD_inBuffer *")
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
dst_buffer = ffi.new('char[]', write_size) dst_buffer = ffi.new("char[]", write_size)
out_buffer.dst = dst_buffer out_buffer.dst = dst_buffer
out_buffer.size = len(dst_buffer) out_buffer.size = len(dst_buffer)
out_buffer.pos = 0 out_buffer.pos = 0
while True: while True:
assert out_buffer.pos == 0 assert out_buffer.pos == 0
if have_read: if have_read:
read_result = reader.read(read_size) read_result = reader.read(read_size)
else: else:
remaining = size - buffer_offset remaining = size - buffer_offset
slice_size = min(remaining, read_size) slice_size = min(remaining, read_size)
read_result = reader[buffer_offset:buffer_offset + slice_size] read_result = reader[buffer_offset : buffer_offset + slice_size]
buffer_offset += slice_size buffer_offset += slice_size
# No new input. Break out of read loop. # No new input. Break out of read loop.
if not read_result: if not read_result:
break break
# Feed all read data into decompressor and emit output until # Feed all read data into decompressor and emit output until
# exhausted. # exhausted.
read_buffer = ffi.from_buffer(read_result) read_buffer = ffi.from_buffer(read_result)
in_buffer.src = read_buffer in_buffer.src = read_buffer
in_buffer.size = len(read_buffer) in_buffer.size = len(read_buffer)
in_buffer.pos = 0 in_buffer.pos = 0
while in_buffer.pos < in_buffer.size: while in_buffer.pos < in_buffer.size:
assert out_buffer.pos == 0 assert out_buffer.pos == 0
zresult = lib.ZSTD_decompressStream(self._dctx, out_buffer, in_buffer) zresult = lib.ZSTD_decompressStream(self._dctx, out_buffer, in_buffer)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd decompress error: %s' % raise ZstdError("zstd decompress error: %s" % _zstd_error(zresult))
_zstd_error(zresult))
if out_buffer.pos: if out_buffer.pos:
data = ffi.buffer(out_buffer.dst, out_buffer.pos)[:] data = ffi.buffer(out_buffer.dst, out_buffer.pos)[:]
out_buffer.pos = 0 out_buffer.pos = 0
yield data yield data
if zresult == 0: if zresult == 0:
return return
# Repeat loop to collect more input data. # Repeat loop to collect more input data.
continue continue
# If we get here, input is exhausted. # If we get here, input is exhausted.
read_from = read_to_iter read_from = read_to_iter
def stream_writer(self, writer, write_size=DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE, def stream_writer(
write_return_read=False): self,
if not hasattr(writer, 'write'): writer,
raise ValueError('must pass an object with a write() method') write_size=DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE,
write_return_read=False,
):
if not hasattr(writer, "write"):
raise ValueError("must pass an object with a write() method")
return ZstdDecompressionWriter(self, writer, write_size, return ZstdDecompressionWriter(self, writer, write_size, write_return_read)
write_return_read)
write_to = stream_writer write_to = stream_writer
def copy_stream(self, ifh, ofh, def copy_stream(
self,
ifh,
ofh,
read_size=DECOMPRESSION_RECOMMENDED_INPUT_SIZE, read_size=DECOMPRESSION_RECOMMENDED_INPUT_SIZE,
write_size=DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE): write_size=DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE,
if not hasattr(ifh, 'read'): ):
raise ValueError('first argument must have a read() method') if not hasattr(ifh, "read"):
if not hasattr(ofh, 'write'): raise ValueError("first argument must have a read() method")
raise ValueError('second argument must have a write() method') if not hasattr(ofh, "write"):
raise ValueError("second argument must have a write() method")
self._ensure_dctx() self._ensure_dctx()
in_buffer = ffi.new('ZSTD_inBuffer *') in_buffer = ffi.new("ZSTD_inBuffer *")
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
dst_buffer = ffi.new('char[]', write_size) dst_buffer = ffi.new("char[]", write_size)
out_buffer.dst = dst_buffer out_buffer.dst = dst_buffer
out_buffer.size = write_size out_buffer.size = write_size
out_buffer.pos = 0 out_buffer.pos = 0
total_read, total_write = 0, 0 total_read, total_write = 0, 0
# Read all available input. # Read all available input.
while True: while True:
data = ifh.read(read_size) data = ifh.read(read_size)
if not data: if not data:
break break
data_buffer = ffi.from_buffer(data) data_buffer = ffi.from_buffer(data)
total_read += len(data_buffer) total_read += len(data_buffer)
in_buffer.src = data_buffer in_buffer.src = data_buffer
in_buffer.size = len(data_buffer) in_buffer.size = len(data_buffer)
in_buffer.pos = 0 in_buffer.pos = 0
# Flush all read data to output. # Flush all read data to output.
while in_buffer.pos < in_buffer.size: while in_buffer.pos < in_buffer.size:
zresult = lib.ZSTD_decompressStream(self._dctx, out_buffer, in_buffer) zresult = lib.ZSTD_decompressStream(self._dctx, out_buffer, in_buffer)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('zstd decompressor error: %s' % raise ZstdError(
_zstd_error(zresult)) "zstd decompressor error: %s" % _zstd_error(zresult)
)
if out_buffer.pos: if out_buffer.pos:
ofh.write(ffi.buffer(out_buffer.dst, out_buffer.pos)) ofh.write(ffi.buffer(out_buffer.dst, out_buffer.pos))
total_write += out_buffer.pos total_write += out_buffer.pos
out_buffer.pos = 0 out_buffer.pos = 0
# Continue loop to keep reading. # Continue loop to keep reading.
return total_read, total_write return total_read, total_write
def decompress_content_dict_chain(self, frames): def decompress_content_dict_chain(self, frames):
if not isinstance(frames, list): if not isinstance(frames, list):
raise TypeError('argument must be a list') raise TypeError("argument must be a list")
if not frames: if not frames:
raise ValueError('empty input chain') raise ValueError("empty input chain")
# First chunk should not be using a dictionary. We handle it specially. # First chunk should not be using a dictionary. We handle it specially.
chunk = frames[0] chunk = frames[0]
if not isinstance(chunk, bytes_type): if not isinstance(chunk, bytes_type):
raise ValueError('chunk 0 must be bytes') raise ValueError("chunk 0 must be bytes")
# All chunks should be zstd frames and should have content size set. # All chunks should be zstd frames and should have content size set.
chunk_buffer = ffi.from_buffer(chunk) chunk_buffer = ffi.from_buffer(chunk)
params = ffi.new('ZSTD_frameHeader *') params = ffi.new("ZSTD_frameHeader *")
zresult = lib.ZSTD_getFrameHeader(params, chunk_buffer, len(chunk_buffer)) zresult = lib.ZSTD_getFrameHeader(params, chunk_buffer, len(chunk_buffer))
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ValueError('chunk 0 is not a valid zstd frame') raise ValueError("chunk 0 is not a valid zstd frame")
elif zresult: elif zresult:
raise ValueError('chunk 0 is too small to contain a zstd frame') raise ValueError("chunk 0 is too small to contain a zstd frame")
if params.frameContentSize == lib.ZSTD_CONTENTSIZE_UNKNOWN: if params.frameContentSize == lib.ZSTD_CONTENTSIZE_UNKNOWN:
raise ValueError('chunk 0 missing content size in frame') raise ValueError("chunk 0 missing content size in frame")
self._ensure_dctx(load_dict=False) self._ensure_dctx(load_dict=False)
last_buffer = ffi.new('char[]', params.frameContentSize) last_buffer = ffi.new("char[]", params.frameContentSize)
out_buffer = ffi.new('ZSTD_outBuffer *') out_buffer = ffi.new("ZSTD_outBuffer *")
out_buffer.dst = last_buffer out_buffer.dst = last_buffer
out_buffer.size = len(last_buffer) out_buffer.size = len(last_buffer)
out_buffer.pos = 0 out_buffer.pos = 0
in_buffer = ffi.new('ZSTD_inBuffer *') in_buffer = ffi.new("ZSTD_inBuffer *")
in_buffer.src = chunk_buffer in_buffer.src = chunk_buffer
in_buffer.size = len(chunk_buffer) in_buffer.size = len(chunk_buffer)
in_buffer.pos = 0 in_buffer.pos = 0
zresult = lib.ZSTD_decompressStream(self._dctx, out_buffer, in_buffer) zresult = lib.ZSTD_decompressStream(self._dctx, out_buffer, in_buffer)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('could not decompress chunk 0: %s' % raise ZstdError("could not decompress chunk 0: %s" % _zstd_error(zresult))
_zstd_error(zresult))
elif zresult: elif zresult:
raise ZstdError('chunk 0 did not decompress full frame') raise ZstdError("chunk 0 did not decompress full frame")
# Special case of chain length of 1 # Special case of chain length of 1
if len(frames) == 1: if len(frames) == 1:
return ffi.buffer(last_buffer, len(last_buffer))[:] return ffi.buffer(last_buffer, len(last_buffer))[:]
i = 1 i = 1
while i < len(frames): while i < len(frames):
chunk = frames[i] chunk = frames[i]
if not isinstance(chunk, bytes_type): if not isinstance(chunk, bytes_type):
raise ValueError('chunk %d must be bytes' % i) raise ValueError("chunk %d must be bytes" % i)
chunk_buffer = ffi.from_buffer(chunk) chunk_buffer = ffi.from_buffer(chunk)
zresult = lib.ZSTD_getFrameHeader(params, chunk_buffer, len(chunk_buffer)) zresult = lib.ZSTD_getFrameHeader(params, chunk_buffer, len(chunk_buffer))
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ValueError('chunk %d is not a valid zstd frame' % i) raise ValueError("chunk %d is not a valid zstd frame" % i)
elif zresult: elif zresult:
raise ValueError('chunk %d is too small to contain a zstd frame' % i) raise ValueError("chunk %d is too small to contain a zstd frame" % i)
if params.frameContentSize == lib.ZSTD_CONTENTSIZE_UNKNOWN: if params.frameContentSize == lib.ZSTD_CONTENTSIZE_UNKNOWN:
raise ValueError('chunk %d missing content size in frame' % i) raise ValueError("chunk %d missing content size in frame" % i)
dest_buffer = ffi.new('char[]', params.frameContentSize) dest_buffer = ffi.new("char[]", params.frameContentSize)
out_buffer.dst = dest_buffer out_buffer.dst = dest_buffer
out_buffer.size = len(dest_buffer) out_buffer.size = len(dest_buffer)
out_buffer.pos = 0 out_buffer.pos = 0
in_buffer.src = chunk_buffer in_buffer.src = chunk_buffer
in_buffer.size = len(chunk_buffer) in_buffer.size = len(chunk_buffer)
in_buffer.pos = 0 in_buffer.pos = 0
zresult = lib.ZSTD_decompressStream(self._dctx, out_buffer, in_buffer) zresult = lib.ZSTD_decompressStream(self._dctx, out_buffer, in_buffer)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('could not decompress chunk %d: %s' % raise ZstdError(
_zstd_error(zresult)) "could not decompress chunk %d: %s" % _zstd_error(zresult)
)
elif zresult: elif zresult:
raise ZstdError('chunk %d did not decompress full frame' % i) raise ZstdError("chunk %d did not decompress full frame" % i)
last_buffer = dest_buffer last_buffer = dest_buffer
i += 1 i += 1
return ffi.buffer(last_buffer, len(last_buffer))[:] return ffi.buffer(last_buffer, len(last_buffer))[:]
def _ensure_dctx(self, load_dict=True): def _ensure_dctx(self, load_dict=True):
lib.ZSTD_DCtx_reset(self._dctx, lib.ZSTD_reset_session_only) lib.ZSTD_DCtx_reset(self._dctx, lib.ZSTD_reset_session_only)
if self._max_window_size: if self._max_window_size:
zresult = lib.ZSTD_DCtx_setMaxWindowSize(self._dctx, zresult = lib.ZSTD_DCtx_setMaxWindowSize(self._dctx, self._max_window_size)
self._max_window_size)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('unable to set max window size: %s' % raise ZstdError(
_zstd_error(zresult)) "unable to set max window size: %s" % _zstd_error(zresult)
)
zresult = lib.ZSTD_DCtx_setFormat(self._dctx, self._format) zresult = lib.ZSTD_DCtx_setFormat(self._dctx, self._format)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('unable to set decoding format: %s' % raise ZstdError("unable to set decoding format: %s" % _zstd_error(zresult))
_zstd_error(zresult))
if self._dict_data and load_dict: if self._dict_data and load_dict:
zresult = lib.ZSTD_DCtx_refDDict(self._dctx, self._dict_data._ddict) zresult = lib.ZSTD_DCtx_refDDict(self._dctx, self._dict_data._ddict)
if lib.ZSTD_isError(zresult): if lib.ZSTD_isError(zresult):
raise ZstdError('unable to reference prepared dictionary: %s' % raise ZstdError(
_zstd_error(zresult)) "unable to reference prepared dictionary: %s" % _zstd_error(zresult)
)

contrib/python-zstandard/zstd.c

However, downstream redistributors or packagers may unbundle zstd However, downstream redistributors or packagers may unbundle zstd
from python-zstandard. This can result in a mismatch between zstd from python-zstandard. This can result in a mismatch between zstd
versions and API semantics. This essentially "voids the warranty" versions and API semantics. This essentially "voids the warranty"
of python-zstandard and may cause undefined behavior. of python-zstandard and may cause undefined behavior.
We detect this mismatch here and refuse to load the module if this We detect this mismatch here and refuse to load the module if this
scenario is detected. scenario is detected.
*/ */
if (ZSTD_VERSION_NUMBER != 10403 || ZSTD_versionNumber() != 10403) { if (ZSTD_VERSION_NUMBER != 10404 || ZSTD_versionNumber() != 10404) {
PyErr_SetString(PyExc_ImportError, "zstd C API mismatch; Python bindings not compiled against expected zstd version"); PyErr_SetString(PyExc_ImportError, "zstd C API mismatch; Python bindings not compiled against expected zstd version");
return; return;
} }
bufferutil_module_init(m); bufferutil_module_init(m);
compressionparams_module_init(m); compressionparams_module_init(m);
compressiondict_module_init(m); compressiondict_module_init(m);
compressobj_module_init(m); compressobj_module_init(m);
memcpy(PyBytes_AS_STRING(tmp), PyBytes_AS_STRING(*obj), memcpy(PyBytes_AS_STRING(tmp), PyBytes_AS_STRING(*obj),
PyBytes_GET_SIZE(*obj)); PyBytes_GET_SIZE(*obj));
Py_DECREF(*obj); Py_DECREF(*obj);
*obj = tmp; *obj = tmp;
return 0; return 0;
} }
No newline at end of file No newline at end of file

contrib/python-zstandard/zstd/common/bitstream.h

{ {
assert(val != 0); assert(val != 0);
{ {
# if defined(_MSC_VER) /* Visual */ # if defined(_MSC_VER) /* Visual */
unsigned long r=0; unsigned long r=0;
_BitScanReverse ( &r, val ); _BitScanReverse ( &r, val );
return (unsigned) r; return (unsigned) r;
# elif defined(__GNUC__) && (__GNUC__ >= 3) /* Use GCC Intrinsic */ # elif defined(__GNUC__) && (__GNUC__ >= 3) /* Use GCC Intrinsic */
return 31 - __builtin_clz (val); return __builtin_clz (val) ^ 31;
# elif defined(__ICCARM__) /* IAR Intrinsic */ # elif defined(__ICCARM__) /* IAR Intrinsic */
return 31 - __CLZ(val); return 31 - __CLZ(val);
# else /* Software version */ # else /* Software version */
static const unsigned DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, static const unsigned DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29,
11, 14, 16, 18, 22, 25, 3, 30, 11, 14, 16, 18, 22, 25, 3, 30,
8, 12, 20, 28, 15, 17, 24, 7, 8, 12, 20, 28, 15, 17, 24, 7,
19, 27, 23, 6, 26, 5, 4, 31 }; 19, 27, 23, 6, 26, 5, 4, 31 };
U32 v = val; U32 v = val;
/*! BIT_flushBitsFast() : /*! BIT_flushBitsFast() :
* assumption : bitContainer has not overflowed * assumption : bitContainer has not overflowed
* unsafe version; does not check buffer overflow */ * unsafe version; does not check buffer overflow */
MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC) MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC)
{ {
size_t const nbBytes = bitC->bitPos >> 3; size_t const nbBytes = bitC->bitPos >> 3;
assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8); assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
assert(bitC->ptr <= bitC->endPtr);
MEM_writeLEST(bitC->ptr, bitC->bitContainer); MEM_writeLEST(bitC->ptr, bitC->bitContainer);
bitC->ptr += nbBytes; bitC->ptr += nbBytes;
assert(bitC->ptr <= bitC->endPtr);
bitC->bitPos &= 7; bitC->bitPos &= 7;
bitC->bitContainer >>= nbBytes*8; bitC->bitContainer >>= nbBytes*8;
} }
/*! BIT_flushBits() : /*! BIT_flushBits() :
* assumption : bitContainer has not overflowed * assumption : bitContainer has not overflowed
* safe version; check for buffer overflow, and prevents it. * safe version; check for buffer overflow, and prevents it.
* note : does not signal buffer overflow. * note : does not signal buffer overflow.
* overflow will be revealed later on using BIT_closeCStream() */ * overflow will be revealed later on using BIT_closeCStream() */
MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC) MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC)
{ {
size_t const nbBytes = bitC->bitPos >> 3; size_t const nbBytes = bitC->bitPos >> 3;
assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8); assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
assert(bitC->ptr <= bitC->endPtr);
MEM_writeLEST(bitC->ptr, bitC->bitContainer); MEM_writeLEST(bitC->ptr, bitC->bitContainer);
bitC->ptr += nbBytes; bitC->ptr += nbBytes;
if (bitC->ptr > bitC->endPtr) bitC->ptr = bitC->endPtr; if (bitC->ptr > bitC->endPtr) bitC->ptr = bitC->endPtr;
bitC->bitPos &= 7; bitC->bitPos &= 7;
bitC->bitContainer >>= nbBytes*8; bitC->bitContainer >>= nbBytes*8;
} }
/*! BIT_closeCStream() : /*! BIT_closeCStream() :

contrib/python-zstandard/zstd/common/compiler.h

* attribute. * attribute.
*/ */
#if !defined(__clang__) && defined(__GNUC__) && __GNUC__ >= 4 && __GNUC_MINOR__ >= 8 && __GNUC__ < 5 #if !defined(__clang__) && defined(__GNUC__) && __GNUC__ >= 4 && __GNUC_MINOR__ >= 8 && __GNUC__ < 5
# define HINT_INLINE static INLINE_KEYWORD # define HINT_INLINE static INLINE_KEYWORD
#else #else
# define HINT_INLINE static INLINE_KEYWORD FORCE_INLINE_ATTR # define HINT_INLINE static INLINE_KEYWORD FORCE_INLINE_ATTR
#endif #endif
/* UNUSED_ATTR tells the compiler it is okay if the function is unused. */
#if defined(__GNUC__)
# define UNUSED_ATTR __attribute__((unused))
#else
# define UNUSED_ATTR
#endif
/* force no inlining */ /* force no inlining */
#ifdef _MSC_VER #ifdef _MSC_VER
# define FORCE_NOINLINE static __declspec(noinline) # define FORCE_NOINLINE static __declspec(noinline)
#else #else
# if defined(__GNUC__) || defined(__ICCARM__) # if defined(__GNUC__) || defined(__ICCARM__)
# define FORCE_NOINLINE static __attribute__((__noinline__)) # define FORCE_NOINLINE static __attribute__((__noinline__))
# else # else
# define FORCE_NOINLINE static # define FORCE_NOINLINE static
const char* const _ptr = (const char*)(p); \ const char* const _ptr = (const char*)(p); \
size_t const _size = (size_t)(s); \ size_t const _size = (size_t)(s); \
size_t _pos; \ size_t _pos; \
for (_pos=0; _pos<_size; _pos+=CACHELINE_SIZE) { \ for (_pos=0; _pos<_size; _pos+=CACHELINE_SIZE) { \
PREFETCH_L2(_ptr + _pos); \ PREFETCH_L2(_ptr + _pos); \
} \ } \
} }
/* vectorization */ /* vectorization
* older GCC (pre gcc-4.3 picked as the cutoff) uses a different syntax */
#if !defined(__clang__) && defined(__GNUC__) #if !defined(__clang__) && defined(__GNUC__)
# if (__GNUC__ == 4 && __GNUC_MINOR__ > 3) || (__GNUC__ >= 5)
# define DONT_VECTORIZE __attribute__((optimize("no-tree-vectorize"))) # define DONT_VECTORIZE __attribute__((optimize("no-tree-vectorize")))
#else # else
# define DONT_VECTORIZE _Pragma("GCC optimize(\"no-tree-vectorize\")")
# endif
#else
# define DONT_VECTORIZE # define DONT_VECTORIZE
#endif #endif
/* disable warnings */ /* disable warnings */
#ifdef _MSC_VER /* Visual Studio */ #ifdef _MSC_VER /* Visual Studio */
# include <intrin.h> /* For Visual 2005 */ # include <intrin.h> /* For Visual 2005 */
# pragma warning(disable : 4100) /* disable: C4100: unreferenced formal parameter */ # pragma warning(disable : 4100) /* disable: C4100: unreferenced formal parameter */
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ # pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
# pragma warning(disable : 4204) /* disable: C4204: non-constant aggregate initializer */ # pragma warning(disable : 4204) /* disable: C4204: non-constant aggregate initializer */
# pragma warning(disable : 4214) /* disable: C4214: non-int bitfields */ # pragma warning(disable : 4214) /* disable: C4214: non-int bitfields */
# pragma warning(disable : 4324) /* disable: C4324: padded structure */ # pragma warning(disable : 4324) /* disable: C4324: padded structure */
#endif #endif
#endif /* ZSTD_COMPILER_H */ #endif /* ZSTD_COMPILER_H */

contrib/python-zstandard/zstd/common/fse.h

#include "bitstream.h" #include "bitstream.h"
/* ***************************************** /* *****************************************
* Static allocation * Static allocation
*******************************************/ *******************************************/
/* FSE buffer bounds */ /* FSE buffer bounds */
#define FSE_NCOUNTBOUND 512 #define FSE_NCOUNTBOUND 512
#define FSE_BLOCKBOUND(size) (size + (size>>7)) #define FSE_BLOCKBOUND(size) (size + (size>>7) + 4 /* fse states */ + sizeof(size_t) /* bitContainer */)
#define FSE_COMPRESSBOUND(size) (FSE_NCOUNTBOUND + FSE_BLOCKBOUND(size)) /* Macro version, useful for static allocation */ #define FSE_COMPRESSBOUND(size) (FSE_NCOUNTBOUND + FSE_BLOCKBOUND(size)) /* Macro version, useful for static allocation */
/* It is possible to statically allocate FSE CTable/DTable as a table of FSE_CTable/FSE_DTable using below macros */ /* It is possible to statically allocate FSE CTable/DTable as a table of FSE_CTable/FSE_DTable using below macros */
#define FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) (1 + (1<<(maxTableLog-1)) + ((maxSymbolValue+1)*2)) #define FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) (1 + (1<<(maxTableLog-1)) + ((maxSymbolValue+1)*2))
#define FSE_DTABLE_SIZE_U32(maxTableLog) (1 + (1<<maxTableLog)) #define FSE_DTABLE_SIZE_U32(maxTableLog) (1 + (1<<maxTableLog))
/* or use the size to malloc() space directly. Pay attention to alignment restrictions though */ /* or use the size to malloc() space directly. Pay attention to alignment restrictions though */
#define FSE_CTABLE_SIZE(maxTableLog, maxSymbolValue) (FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) * sizeof(FSE_CTable)) #define FSE_CTABLE_SIZE(maxTableLog, maxSymbolValue) (FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) * sizeof(FSE_CTable))

contrib/python-zstandard/zstd/common/fse_decompress.c

/* ************************************************************** /* **************************************************************
* Error Management * Error Management
****************************************************************/ ****************************************************************/
#define FSE_isError ERR_isError #define FSE_isError ERR_isError
#define FSE_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c) /* use only *after* variable declarations */ #define FSE_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c) /* use only *after* variable declarations */
/* check and forward error code */ /* check and forward error code */
#ifndef CHECK_F
#define CHECK_F(f) { size_t const e = f; if (FSE_isError(e)) return e; } #define CHECK_F(f) { size_t const e = f; if (FSE_isError(e)) return e; }
#endif
/* ************************************************************** /* **************************************************************
* Templates * Templates
****************************************************************/ ****************************************************************/
/* /*
designed to be included designed to be included
for type-specific functions (template emulation in C) for type-specific functions (template emulation in C)

contrib/python-zstandard/zstd/common/mem.h

#ifndef __has_builtin #ifndef __has_builtin
# define __has_builtin(x) 0 /* compat. with non-clang compilers */ # define __has_builtin(x) 0 /* compat. with non-clang compilers */
#endif #endif
/* code only tested on 32 and 64 bits systems */ /* code only tested on 32 and 64 bits systems */
#define MEM_STATIC_ASSERT(c) { enum { MEM_static_assert = 1/(int)(!!(c)) }; } #define MEM_STATIC_ASSERT(c) { enum { MEM_static_assert = 1/(int)(!!(c)) }; }
MEM_STATIC void MEM_check(void) { MEM_STATIC_ASSERT((sizeof(size_t)==4) || (sizeof(size_t)==8)); } MEM_STATIC void MEM_check(void) { MEM_STATIC_ASSERT((sizeof(size_t)==4) || (sizeof(size_t)==8)); }
/* detects whether we are being compiled under msan */
#if defined (__has_feature)
# if __has_feature(memory_sanitizer)
# define MEMORY_SANITIZER 1
# endif
#endif
#if defined (MEMORY_SANITIZER)
/* Not all platforms that support msan provide sanitizers/msan_interface.h.
* We therefore declare the functions we need ourselves, rather than trying to
* include the header file... */
#include <stdint.h> /* intptr_t */
/* Make memory region fully initialized (without changing its contents). */
void __msan_unpoison(const volatile void *a, size_t size);
/* Make memory region fully uninitialized (without changing its contents).
This is a legacy interface that does not update origin information. Use
__msan_allocated_memory() instead. */
void __msan_poison(const volatile void *a, size_t size);
/* Returns the offset of the first (at least partially) poisoned byte in the
memory range, or -1 if the whole range is good. */
intptr_t __msan_test_shadow(const volatile void *x, size_t size);
#endif
/* detects whether we are being compiled under asan */
#if defined (__has_feature)
# if __has_feature(address_sanitizer)
# define ADDRESS_SANITIZER 1
# endif
#elif defined(__SANITIZE_ADDRESS__)
# define ADDRESS_SANITIZER 1
#endif
#if defined (ADDRESS_SANITIZER)
/* Not all platforms that support asan provide sanitizers/asan_interface.h.
* We therefore declare the functions we need ourselves, rather than trying to
* include the header file... */
/**
* Marks a memory region (<c>[addr, addr+size)</c>) as unaddressable.
*
* This memory must be previously allocated by your program. Instrumented
* code is forbidden from accessing addresses in this region until it is
* unpoisoned. This function is not guaranteed to poison the entire region -
* it could poison only a subregion of <c>[addr, addr+size)</c> due to ASan
* alignment restrictions.
*
* \note This function is not thread-safe because no two threads can poison or
* unpoison memory in the same memory region simultaneously.
*
* \param addr Start of memory region.
* \param size Size of memory region. */
void __asan_poison_memory_region(void const volatile *addr, size_t size);
/**
* Marks a memory region (<c>[addr, addr+size)</c>) as addressable.
*
* This memory must be previously allocated by your program. Accessing
* addresses in this region is allowed until this region is poisoned again.
* This function could unpoison a super-region of <c>[addr, addr+size)</c> due
* to ASan alignment restrictions.
*
* \note This function is not thread-safe because no two threads can
* poison or unpoison memory in the same memory region simultaneously.
*
* \param addr Start of memory region.
* \param size Size of memory region. */
void __asan_unpoison_memory_region(void const volatile *addr, size_t size);
#endif
/*-************************************************************** /*-**************************************************************
* Basic Types * Basic Types
*****************************************************************/ *****************************************************************/
#if !defined (__VMS) && (defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) #if !defined (__VMS) && (defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
# include <stdint.h> # include <stdint.h>
typedef uint8_t BYTE; typedef uint8_t BYTE;
typedef uint16_t U16; typedef uint16_t U16;

contrib/python-zstandard/zstd/common/pool.c

* empty and full queues. * empty and full queues.
*/ */
ctx->queueSize = queueSize + 1; ctx->queueSize = queueSize + 1;
ctx->queue = (POOL_job*)ZSTD_malloc(ctx->queueSize * sizeof(POOL_job), customMem); ctx->queue = (POOL_job*)ZSTD_malloc(ctx->queueSize * sizeof(POOL_job), customMem);
ctx->queueHead = 0; ctx->queueHead = 0;
ctx->queueTail = 0; ctx->queueTail = 0;
ctx->numThreadsBusy = 0; ctx->numThreadsBusy = 0;
ctx->queueEmpty = 1; ctx->queueEmpty = 1;
(void)ZSTD_pthread_mutex_init(&ctx->queueMutex, NULL); {
(void)ZSTD_pthread_cond_init(&ctx->queuePushCond, NULL); int error = 0;
(void)ZSTD_pthread_cond_init(&ctx->queuePopCond, NULL); error |= ZSTD_pthread_mutex_init(&ctx->queueMutex, NULL);
error |= ZSTD_pthread_cond_init(&ctx->queuePushCond, NULL);
error |= ZSTD_pthread_cond_init(&ctx->queuePopCond, NULL);
if (error) { POOL_free(ctx); return NULL; }
}
ctx->shutdown = 0; ctx->shutdown = 0;
/* Allocate space for the thread handles */ /* Allocate space for the thread handles */
ctx->threads = (ZSTD_pthread_t*)ZSTD_malloc(numThreads * sizeof(ZSTD_pthread_t), customMem); ctx->threads = (ZSTD_pthread_t*)ZSTD_malloc(numThreads * sizeof(ZSTD_pthread_t), customMem);
ctx->threadCapacity = 0; ctx->threadCapacity = 0;
ctx->customMem = customMem; ctx->customMem = customMem;
/* Check for errors */ /* Check for errors */
if (!ctx->threads || !ctx->queue) { POOL_free(ctx); return NULL; } if (!ctx->threads || !ctx->queue) { POOL_free(ctx); return NULL; }
/* Initialize the threads */ /* Initialize the threads */

contrib/python-zstandard/zstd/common/threading.h

/** /**
* Copyright (c) 2016 Tino Reichardt * Copyright (c) 2016 Tino Reichardt
* All rights reserved. * All rights reserved.
* *
* This source code is licensed under both the BSD-style license (found in the * This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found * LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree). * in the COPYING file in the root directory of this source tree).
* *
* You can contact the author at: * You can contact the author at:
* - zstdmt source repository: https://github.com/mcmilk/zstdmt * - zstdmt source repository: https://github.com/mcmilk/zstdmt
*/ */
#ifndef THREADING_H_938743 #ifndef THREADING_H_938743
#define THREADING_H_938743 #define THREADING_H_938743
#include "debug.h"
#if defined (__cplusplus) #if defined (__cplusplus)
extern "C" { extern "C" {
#endif #endif
#if defined(ZSTD_MULTITHREAD) && defined(_WIN32) #if defined(ZSTD_MULTITHREAD) && defined(_WIN32)
/** /**
* Windows minimalist Pthread Wrapper, based on : * Windows minimalist Pthread Wrapper, based on :
int ZSTD_pthread_join(ZSTD_pthread_t thread, void** value_ptr); int ZSTD_pthread_join(ZSTD_pthread_t thread, void** value_ptr);
/** /**
* add here more wrappers as required * add here more wrappers as required
*/ */
#elif defined(ZSTD_MULTITHREAD) /* posix assumed ; need a better detection method */ #elif defined(ZSTD_MULTITHREAD) /* posix assumed ; need a better detection method */
/* === POSIX Systems === */ /* === POSIX Systems === */
# include <pthread.h> # include <pthread.h>
#if DEBUGLEVEL < 1
#define ZSTD_pthread_mutex_t pthread_mutex_t #define ZSTD_pthread_mutex_t pthread_mutex_t
#define ZSTD_pthread_mutex_init(a, b) pthread_mutex_init((a), (b)) #define ZSTD_pthread_mutex_init(a, b) pthread_mutex_init((a), (b))
#define ZSTD_pthread_mutex_destroy(a) pthread_mutex_destroy((a)) #define ZSTD_pthread_mutex_destroy(a) pthread_mutex_destroy((a))
#define ZSTD_pthread_mutex_lock(a) pthread_mutex_lock((a)) #define ZSTD_pthread_mutex_lock(a) pthread_mutex_lock((a))
#define ZSTD_pthread_mutex_unlock(a) pthread_mutex_unlock((a)) #define ZSTD_pthread_mutex_unlock(a) pthread_mutex_unlock((a))
#define ZSTD_pthread_cond_t pthread_cond_t #define ZSTD_pthread_cond_t pthread_cond_t
#define ZSTD_pthread_cond_init(a, b) pthread_cond_init((a), (b)) #define ZSTD_pthread_cond_init(a, b) pthread_cond_init((a), (b))
#define ZSTD_pthread_cond_destroy(a) pthread_cond_destroy((a)) #define ZSTD_pthread_cond_destroy(a) pthread_cond_destroy((a))
#define ZSTD_pthread_cond_wait(a, b) pthread_cond_wait((a), (b)) #define ZSTD_pthread_cond_wait(a, b) pthread_cond_wait((a), (b))
#define ZSTD_pthread_cond_signal(a) pthread_cond_signal((a)) #define ZSTD_pthread_cond_signal(a) pthread_cond_signal((a))
#define ZSTD_pthread_cond_broadcast(a) pthread_cond_broadcast((a)) #define ZSTD_pthread_cond_broadcast(a) pthread_cond_broadcast((a))
#define ZSTD_pthread_t pthread_t #define ZSTD_pthread_t pthread_t
#define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d)) #define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d))
#define ZSTD_pthread_join(a, b) pthread_join((a),(b)) #define ZSTD_pthread_join(a, b) pthread_join((a),(b))
#else /* DEBUGLEVEL >= 1 */
/* Debug implementation of threading.
* In this implementation we use pointers for mutexes and condition variables.
* This way, if we forget to init/destroy them the program will crash or ASAN
* will report leaks.
*/
#define ZSTD_pthread_mutex_t pthread_mutex_t*
int ZSTD_pthread_mutex_init(ZSTD_pthread_mutex_t* mutex, pthread_mutexattr_t const* attr);
int ZSTD_pthread_mutex_destroy(ZSTD_pthread_mutex_t* mutex);
#define ZSTD_pthread_mutex_lock(a) pthread_mutex_lock(*(a))
#define ZSTD_pthread_mutex_unlock(a) pthread_mutex_unlock(*(a))
#define ZSTD_pthread_cond_t pthread_cond_t*
int ZSTD_pthread_cond_init(ZSTD_pthread_cond_t* cond, pthread_condattr_t const* attr);
int ZSTD_pthread_cond_destroy(ZSTD_pthread_cond_t* cond);
#define ZSTD_pthread_cond_wait(a, b) pthread_cond_wait(*(a), *(b))
#define ZSTD_pthread_cond_signal(a) pthread_cond_signal(*(a))
#define ZSTD_pthread_cond_broadcast(a) pthread_cond_broadcast(*(a))
#define ZSTD_pthread_t pthread_t
#define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d))
#define ZSTD_pthread_join(a, b) pthread_join((a),(b))
#endif
#else /* ZSTD_MULTITHREAD not defined */ #else /* ZSTD_MULTITHREAD not defined */
/* No multithreading support */ /* No multithreading support */
typedef int ZSTD_pthread_mutex_t; typedef int ZSTD_pthread_mutex_t;
#define ZSTD_pthread_mutex_init(a, b) ((void)(a), (void)(b), 0) #define ZSTD_pthread_mutex_init(a, b) ((void)(a), (void)(b), 0)
#define ZSTD_pthread_mutex_destroy(a) ((void)(a)) #define ZSTD_pthread_mutex_destroy(a) ((void)(a))
#define ZSTD_pthread_mutex_lock(a) ((void)(a)) #define ZSTD_pthread_mutex_lock(a) ((void)(a))
#define ZSTD_pthread_mutex_unlock(a) ((void)(a)) #define ZSTD_pthread_mutex_unlock(a) ((void)(a))

contrib/python-zstandard/zstd/common/threading.c

/** /**
* Copyright (c) 2016 Tino Reichardt * Copyright (c) 2016 Tino Reichardt
* All rights reserved. * All rights reserved.
* *
* This source code is licensed under both the BSD-style license (found in the * This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found * LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree). * in the COPYING file in the root directory of this source tree).
* *
* You can contact the author at: * You can contact the author at:
* - zstdmt source repository: https://github.com/mcmilk/zstdmt * - zstdmt source repository: https://github.com/mcmilk/zstdmt
*/ */
/** /**
* This file will hold wrapper for systems, which do not support pthreads * This file will hold wrapper for systems, which do not support pthreads
*/ */
#include "threading.h"
/* create fake symbol to avoid empty translation unit warning */ /* create fake symbol to avoid empty translation unit warning */
int g_ZSTD_threading_useless_symbol; int g_ZSTD_threading_useless_symbol;
#if defined(ZSTD_MULTITHREAD) && defined(_WIN32) #if defined(ZSTD_MULTITHREAD) && defined(_WIN32)
/** /**
* Windows minimalist Pthread Wrapper, based on : * Windows minimalist Pthread Wrapper, based on :
* http://www.cse.wustl.edu/~schmidt/win32-cv-1.html * http://www.cse.wustl.edu/~schmidt/win32-cv-1.html
*/ */
/* === Dependencies === */ /* === Dependencies === */
#include <process.h> #include <process.h>
#include <errno.h> #include <errno.h>
#include "threading.h"
/* === Implementation === */ /* === Implementation === */
static unsigned __stdcall worker(void *arg) static unsigned __stdcall worker(void *arg)
{ {
ZSTD_pthread_t* const thread = (ZSTD_pthread_t*) arg; ZSTD_pthread_t* const thread = (ZSTD_pthread_t*) arg;
thread->arg = thread->start_routine(thread->arg); thread->arg = thread->start_routine(thread->arg);
case WAIT_ABANDONED: case WAIT_ABANDONED:
return EINVAL; return EINVAL;
default: default:
return GetLastError(); return GetLastError();
} }
} }
#endif /* ZSTD_MULTITHREAD */ #endif /* ZSTD_MULTITHREAD */
#if defined(ZSTD_MULTITHREAD) && DEBUGLEVEL >= 1 && !defined(_WIN32)
#include <stdlib.h>
int ZSTD_pthread_mutex_init(ZSTD_pthread_mutex_t* mutex, pthread_mutexattr_t const* attr)
{
*mutex = (pthread_mutex_t*)malloc(sizeof(pthread_mutex_t));
if (!*mutex)
return 1;
return pthread_mutex_init(*mutex, attr);
}
int ZSTD_pthread_mutex_destroy(ZSTD_pthread_mutex_t* mutex)
{
if (!*mutex)
return 0;
{
int const ret = pthread_mutex_destroy(*mutex);
free(*mutex);
return ret;
}
}
int ZSTD_pthread_cond_init(ZSTD_pthread_cond_t* cond, pthread_condattr_t const* attr)
{
*cond = (pthread_cond_t*)malloc(sizeof(pthread_cond_t));
if (!*cond)
return 1;
return pthread_cond_init(*cond, attr);
}
int ZSTD_pthread_cond_destroy(ZSTD_pthread_cond_t* cond)
{
if (!*cond)
return 0;
{
int const ret = pthread_cond_destroy(*cond);
free(*cond);
return ret;
}
}
#endif

contrib/python-zstandard/zstd/common/zstd_internal.h

* Shared functions to include for inlining * Shared functions to include for inlining
*********************************************/ *********************************************/
static void ZSTD_copy8(void* dst, const void* src) { memcpy(dst, src, 8); } static void ZSTD_copy8(void* dst, const void* src) { memcpy(dst, src, 8); }
#define COPY8(d,s) { ZSTD_copy8(d,s); d+=8; s+=8; } #define COPY8(d,s) { ZSTD_copy8(d,s); d+=8; s+=8; }
static void ZSTD_copy16(void* dst, const void* src) { memcpy(dst, src, 16); } static void ZSTD_copy16(void* dst, const void* src) { memcpy(dst, src, 16); }
#define COPY16(d,s) { ZSTD_copy16(d,s); d+=16; s+=16; } #define COPY16(d,s) { ZSTD_copy16(d,s); d+=16; s+=16; }
#define WILDCOPY_OVERLENGTH 8 #define WILDCOPY_OVERLENGTH 32
#define VECLEN 16 #define WILDCOPY_VECLEN 16
typedef enum { typedef enum {
ZSTD_no_overlap, ZSTD_no_overlap,
ZSTD_overlap_src_before_dst, ZSTD_overlap_src_before_dst
/* ZSTD_overlap_dst_before_src, */ /* ZSTD_overlap_dst_before_src, */
} ZSTD_overlap_e; } ZSTD_overlap_e;
/*! ZSTD_wildcopy() : /*! ZSTD_wildcopy() :
* custom version of memcpy(), can overwrite up to WILDCOPY_OVERLENGTH bytes (if length==0) */ * Custom version of memcpy(), can over read/write up to WILDCOPY_OVERLENGTH bytes (if length==0)
MEM_STATIC FORCE_INLINE_ATTR DONT_VECTORIZE * @param ovtype controls the overlap detection
void ZSTD_wildcopy(void* dst, const void* src, ptrdiff_t length, ZSTD_overlap_e ovtype) * - ZSTD_no_overlap: The source and destination are guaranteed to be at least WILDCOPY_VECLEN bytes apart.
{ * - ZSTD_overlap_src_before_dst: The src and dst may overlap, but they MUST be at least 8 bytes apart.
ptrdiff_t diff = (BYTE*)dst - (const BYTE*)src; * The src buffer must be before the dst buffer.
const BYTE* ip = (const BYTE*)src; */
BYTE* op = (BYTE*)dst;
BYTE* const oend = op + length;
assert(diff >= 8 || (ovtype == ZSTD_no_overlap && diff < -8));
if (length < VECLEN || (ovtype == ZSTD_overlap_src_before_dst && diff < VECLEN)) {
do
COPY8(op, ip)
while (op < oend);
}
else {
if ((length & 8) == 0)
COPY8(op, ip);
do {
COPY16(op, ip);
}
while (op < oend);
}
}
/*! ZSTD_wildcopy_16min() :
* same semantics as ZSTD_wilcopy() except guaranteed to be able to copy 16 bytes at the start */
MEM_STATIC FORCE_INLINE_ATTR DONT_VECTORIZE MEM_STATIC FORCE_INLINE_ATTR DONT_VECTORIZE
void ZSTD_wildcopy_16min(void* dst, const void* src, ptrdiff_t length, ZSTD_overlap_e ovtype) void ZSTD_wildcopy(void* dst, const void* src, ptrdiff_t length, ZSTD_overlap_e const ovtype)
{ {
ptrdiff_t diff = (BYTE*)dst - (const BYTE*)src; ptrdiff_t diff = (BYTE*)dst - (const BYTE*)src;
const BYTE* ip = (const BYTE*)src; const BYTE* ip = (const BYTE*)src;
BYTE* op = (BYTE*)dst; BYTE* op = (BYTE*)dst;
BYTE* const oend = op + length; BYTE* const oend = op + length;
assert(length >= 8); assert(diff >= 8 || (ovtype == ZSTD_no_overlap && diff <= -WILDCOPY_VECLEN));
assert(diff >= 8 || (ovtype == ZSTD_no_overlap && diff < -8));
if (ovtype == ZSTD_overlap_src_before_dst && diff < VECLEN) { if (ovtype == ZSTD_overlap_src_before_dst && diff < WILDCOPY_VECLEN) {
do /* Handle short offset copies. */
do {
COPY8(op, ip) COPY8(op, ip)
while (op < oend); } while (op < oend);
} } else {
else { assert(diff >= WILDCOPY_VECLEN || diff <= -WILDCOPY_VECLEN);
if ((length & 8) == 0) /* Separate out the first two COPY16() calls because the copy length is
COPY8(op, ip); * almost certain to be short, so the branches have different
* probabilities.
* On gcc-9 unrolling once is +1.6%, twice is +2%, thrice is +1.8%.
* On clang-8 unrolling once is +1.4%, twice is +3.3%, thrice is +3%.
*/
COPY16(op, ip);
COPY16(op, ip);
if (op >= oend) return;
do { do {
COPY16(op, ip); COPY16(op, ip);
COPY16(op, ip);
} }
while (op < oend); while (op < oend);
} }
} }
MEM_STATIC void ZSTD_wildcopy_e(void* dst, const void* src, void* dstEnd) /* should be faster for decoding, but strangely, not verified on all platform */
{
const BYTE* ip = (const BYTE*)src;
BYTE* op = (BYTE*)dst;
BYTE* const oend = (BYTE*)dstEnd;
do
COPY8(op, ip)
while (op < oend);
}
/*-******************************************* /*-*******************************************
* Private declarations * Private declarations
*********************************************/ *********************************************/
typedef struct seqDef_s { typedef struct seqDef_s {
U32 offset; U32 offset;
U16 litLength; U16 litLength;
U16 matchLength; U16 matchLength;
{ {
assert(val != 0); assert(val != 0);
{ {
# if defined(_MSC_VER) /* Visual */ # if defined(_MSC_VER) /* Visual */
unsigned long r=0; unsigned long r=0;
_BitScanReverse(&r, val); _BitScanReverse(&r, val);
return (unsigned)r; return (unsigned)r;
# elif defined(__GNUC__) && (__GNUC__ >= 3) /* GCC Intrinsic */ # elif defined(__GNUC__) && (__GNUC__ >= 3) /* GCC Intrinsic */
return 31 - __builtin_clz(val); return __builtin_clz (val) ^ 31;
# elif defined(__ICCARM__) /* IAR Intrinsic */ # elif defined(__ICCARM__) /* IAR Intrinsic */
return 31 - __CLZ(val); return 31 - __CLZ(val);
# else /* Software version */ # else /* Software version */
static const U32 DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, 8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31 }; static const U32 DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, 8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31 };
U32 v = val; U32 v = val;
v |= v >> 1; v |= v >> 1;
v |= v >> 2; v |= v >> 2;
v |= v >> 4; v |= v >> 4;

contrib/python-zstandard/zstd/compress/zstd_compress.c

return ZSTD_COMPRESSBOUND(srcSize); return ZSTD_COMPRESSBOUND(srcSize);
} }
/*-************************************* /*-*************************************
* Context memory management * Context memory management
***************************************/ ***************************************/
struct ZSTD_CDict_s { struct ZSTD_CDict_s {
void* dictBuffer;
const void* dictContent; const void* dictContent;
size_t dictContentSize; size_t dictContentSize;
void* workspace; U32* entropyWorkspace; /* entropy workspace of HUF_WORKSPACE_SIZE bytes */
size_t workspaceSize; ZSTD_cwksp workspace;
ZSTD_matchState_t matchState; ZSTD_matchState_t matchState;
ZSTD_compressedBlockState_t cBlockState; ZSTD_compressedBlockState_t cBlockState;
ZSTD_customMem customMem; ZSTD_customMem customMem;
U32 dictID; U32 dictID;
int compressionLevel; /* 0 indicates that advanced API was used to select CDict params */
}; /* typedef'd to ZSTD_CDict within "zstd.h" */ }; /* typedef'd to ZSTD_CDict within "zstd.h" */
ZSTD_CCtx* ZSTD_createCCtx(void) ZSTD_CCtx* ZSTD_createCCtx(void)
{ {
return ZSTD_createCCtx_advanced(ZSTD_defaultCMem); return ZSTD_createCCtx_advanced(ZSTD_defaultCMem);
} }
static void ZSTD_initCCtx(ZSTD_CCtx* cctx, ZSTD_customMem memManager) static void ZSTD_initCCtx(ZSTD_CCtx* cctx, ZSTD_customMem memManager)
if (!cctx) return NULL; if (!cctx) return NULL;
ZSTD_initCCtx(cctx, customMem); ZSTD_initCCtx(cctx, customMem);
return cctx; return cctx;
} }
} }
ZSTD_CCtx* ZSTD_initStaticCCtx(void *workspace, size_t workspaceSize) ZSTD_CCtx* ZSTD_initStaticCCtx(void *workspace, size_t workspaceSize)
{ {
ZSTD_CCtx* const cctx = (ZSTD_CCtx*) workspace; ZSTD_cwksp ws;
ZSTD_CCtx* cctx;
if (workspaceSize <= sizeof(ZSTD_CCtx)) return NULL; /* minimum size */ if (workspaceSize <= sizeof(ZSTD_CCtx)) return NULL; /* minimum size */
if ((size_t)workspace & 7) return NULL; /* must be 8-aligned */ if ((size_t)workspace & 7) return NULL; /* must be 8-aligned */
memset(workspace, 0, workspaceSize); /* may be a bit generous, could memset be smaller ? */ ZSTD_cwksp_init(&ws, workspace, workspaceSize);
cctx = (ZSTD_CCtx*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CCtx));
if (cctx == NULL) {
return NULL;
}
memset(cctx, 0, sizeof(ZSTD_CCtx));
ZSTD_cwksp_move(&cctx->workspace, &ws);
cctx->staticSize = workspaceSize; cctx->staticSize = workspaceSize;
cctx->workSpace = (void*)(cctx+1);
cctx->workSpaceSize = workspaceSize - sizeof(ZSTD_CCtx);
/* statically sized space. entropyWorkspace never moves (but prev/next block swap places) */ /* statically sized space. entropyWorkspace never moves (but prev/next block swap places) */
if (cctx->workSpaceSize < HUF_WORKSPACE_SIZE + 2 * sizeof(ZSTD_compressedBlockState_t)) return NULL; if (!ZSTD_cwksp_check_available(&cctx->workspace, HUF_WORKSPACE_SIZE + 2 * sizeof(ZSTD_compressedBlockState_t))) return NULL;
assert(((size_t)cctx->workSpace & (sizeof(void*)-1)) == 0); /* ensure correct alignment */ cctx->blockState.prevCBlock = (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object(&cctx->workspace, sizeof(ZSTD_compressedBlockState_t));
cctx->blockState.prevCBlock = (ZSTD_compressedBlockState_t*)cctx->workSpace; cctx->blockState.nextCBlock = (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object(&cctx->workspace, sizeof(ZSTD_compressedBlockState_t));
cctx->blockState.nextCBlock = cctx->blockState.prevCBlock + 1; cctx->entropyWorkspace = (U32*)ZSTD_cwksp_reserve_object(
{ &cctx->workspace, HUF_WORKSPACE_SIZE);
void* const ptr = cctx->blockState.nextCBlock + 1;
cctx->entropyWorkspace = (U32*)ptr;
}
cctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid()); cctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid());
return cctx; return cctx;
} }
/** /**
* Clears and frees all of the dictionaries in the CCtx. * Clears and frees all of the dictionaries in the CCtx.
*/ */
static void ZSTD_clearAllDicts(ZSTD_CCtx* cctx) static void ZSTD_clearAllDicts(ZSTD_CCtx* cctx)
size_t const cdictSize = ZSTD_sizeof_CDict(dict.cdict); size_t const cdictSize = ZSTD_sizeof_CDict(dict.cdict);
return bufferSize + cdictSize; return bufferSize + cdictSize;
} }
static void ZSTD_freeCCtxContent(ZSTD_CCtx* cctx) static void ZSTD_freeCCtxContent(ZSTD_CCtx* cctx)
{ {
assert(cctx != NULL); assert(cctx != NULL);
assert(cctx->staticSize == 0); assert(cctx->staticSize == 0);
ZSTD_free(cctx->workSpace, cctx->customMem); cctx->workSpace = NULL;
ZSTD_clearAllDicts(cctx); ZSTD_clearAllDicts(cctx);
#ifdef ZSTD_MULTITHREAD #ifdef ZSTD_MULTITHREAD
ZSTDMT_freeCCtx(cctx->mtctx); cctx->mtctx = NULL; ZSTDMT_freeCCtx(cctx->mtctx); cctx->mtctx = NULL;
#endif #endif
ZSTD_cwksp_free(&cctx->workspace, cctx->customMem);
} }
size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx) size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx)
{ {
if (cctx==NULL) return 0; /* support free on NULL */ if (cctx==NULL) return 0; /* support free on NULL */
RETURN_ERROR_IF(cctx->staticSize, memory_allocation, RETURN_ERROR_IF(cctx->staticSize, memory_allocation,
"not compatible with static CCtx"); "not compatible with static CCtx");
{
int cctxInWorkspace = ZSTD_cwksp_owns_buffer(&cctx->workspace, cctx);
ZSTD_freeCCtxContent(cctx); ZSTD_freeCCtxContent(cctx);
if (!cctxInWorkspace) {
ZSTD_free(cctx, cctx->customMem); ZSTD_free(cctx, cctx->customMem);
}
}
return 0; return 0;
} }
static size_t ZSTD_sizeof_mtctx(const ZSTD_CCtx* cctx) static size_t ZSTD_sizeof_mtctx(const ZSTD_CCtx* cctx)
{ {
#ifdef ZSTD_MULTITHREAD #ifdef ZSTD_MULTITHREAD
return ZSTDMT_sizeof_CCtx(cctx->mtctx); return ZSTDMT_sizeof_CCtx(cctx->mtctx);
#else #else
(void)cctx; (void)cctx;
return 0; return 0;
#endif #endif
} }
size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx) size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx)
{ {
if (cctx==NULL) return 0; /* support sizeof on NULL */ if (cctx==NULL) return 0; /* support sizeof on NULL */
return sizeof(*cctx) + cctx->workSpaceSize /* cctx may be in the workspace */
return (cctx->workspace.workspace == cctx ? 0 : sizeof(*cctx))
+ ZSTD_cwksp_sizeof(&cctx->workspace)
+ ZSTD_sizeof_localDict(cctx->localDict) + ZSTD_sizeof_localDict(cctx->localDict)
+ ZSTD_sizeof_mtctx(cctx); + ZSTD_sizeof_mtctx(cctx);
} }
size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs) size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs)
{ {
return ZSTD_sizeof_CCtx(zcs); /* same object */ return ZSTD_sizeof_CCtx(zcs); /* same object */
} }
return 0; return 0;
} }
size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params) size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params)
{ {
RETURN_ERROR_IF(!cctxParams, GENERIC); RETURN_ERROR_IF(!cctxParams, GENERIC);
FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) ); FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) );
memset(cctxParams, 0, sizeof(*cctxParams)); memset(cctxParams, 0, sizeof(*cctxParams));
assert(!ZSTD_checkCParams(params.cParams));
cctxParams->cParams = params.cParams; cctxParams->cParams = params.cParams;
cctxParams->fParams = params.fParams; cctxParams->fParams = params.fParams;
cctxParams->compressionLevel = ZSTD_CLEVEL_DEFAULT; /* should not matter, as all cParams are presumed properly defined */ cctxParams->compressionLevel = ZSTD_CLEVEL_DEFAULT; /* should not matter, as all cParams are presumed properly defined */
assert(!ZSTD_checkCParams(params.cParams));
return 0; return 0;
} }
/* ZSTD_assignParamsToCCtxParams() : /* ZSTD_assignParamsToCCtxParams() :
* params is presumed valid at this stage */ * params is presumed valid at this stage */
static ZSTD_CCtx_params ZSTD_assignParamsToCCtxParams( static ZSTD_CCtx_params ZSTD_assignParamsToCCtxParams(
ZSTD_CCtx_params cctxParams, ZSTD_parameters params) const ZSTD_CCtx_params* cctxParams, ZSTD_parameters params)
{ {
ZSTD_CCtx_params ret = cctxParams; ZSTD_CCtx_params ret = *cctxParams;
assert(!ZSTD_checkCParams(params.cParams));
ret.cParams = params.cParams; ret.cParams = params.cParams;
ret.fParams = params.fParams; ret.fParams = params.fParams;
ret.compressionLevel = ZSTD_CLEVEL_DEFAULT; /* should not matter, as all cParams are presumed properly defined */ ret.compressionLevel = ZSTD_CLEVEL_DEFAULT; /* should not matter, as all cParams are presumed properly defined */
assert(!ZSTD_checkCParams(params.cParams));
return ret; return ret;
} }
ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter param) ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter param)
{ {
ZSTD_bounds bounds = { 0, 0, 0 }; ZSTD_bounds bounds = { 0, 0, 0 };
switch(param) switch(param)
ZSTD_STATIC_ASSERT(ZSTD_f_zstd1 < ZSTD_f_zstd1_magicless); ZSTD_STATIC_ASSERT(ZSTD_f_zstd1 < ZSTD_f_zstd1_magicless);
bounds.lowerBound = ZSTD_f_zstd1; bounds.lowerBound = ZSTD_f_zstd1;
bounds.upperBound = ZSTD_f_zstd1_magicless; /* note : how to ensure at compile time that this is the highest value enum ? */ bounds.upperBound = ZSTD_f_zstd1_magicless; /* note : how to ensure at compile time that this is the highest value enum ? */
return bounds; return bounds;
case ZSTD_c_forceAttachDict: case ZSTD_c_forceAttachDict:
ZSTD_STATIC_ASSERT(ZSTD_dictDefaultAttach < ZSTD_dictForceCopy); ZSTD_STATIC_ASSERT(ZSTD_dictDefaultAttach < ZSTD_dictForceCopy);
bounds.lowerBound = ZSTD_dictDefaultAttach; bounds.lowerBound = ZSTD_dictDefaultAttach;
bounds.upperBound = ZSTD_dictForceCopy; /* note : how to ensure at compile time that this is the highest value enum ? */ bounds.upperBound = ZSTD_dictForceLoad; /* note : how to ensure at compile time that this is the highest value enum ? */
return bounds; return bounds;
case ZSTD_c_literalCompressionMode: case ZSTD_c_literalCompressionMode:
ZSTD_STATIC_ASSERT(ZSTD_lcm_auto < ZSTD_lcm_huffman && ZSTD_lcm_huffman < ZSTD_lcm_uncompressed); ZSTD_STATIC_ASSERT(ZSTD_lcm_auto < ZSTD_lcm_huffman && ZSTD_lcm_huffman < ZSTD_lcm_uncompressed);
bounds.lowerBound = ZSTD_lcm_auto; bounds.lowerBound = ZSTD_lcm_auto;
bounds.upperBound = ZSTD_lcm_uncompressed; bounds.upperBound = ZSTD_lcm_uncompressed;
return bounds; return bounds;
case ZSTD_c_targetCBlockSize: case ZSTD_c_targetCBlockSize:
bounds.lowerBound = ZSTD_TARGETCBLOCKSIZE_MIN; bounds.lowerBound = ZSTD_TARGETCBLOCKSIZE_MIN;
bounds.upperBound = ZSTD_TARGETCBLOCKSIZE_MAX; bounds.upperBound = ZSTD_TARGETCBLOCKSIZE_MAX;
return bounds; return bounds;
case ZSTD_c_srcSizeHint:
bounds.lowerBound = ZSTD_SRCSIZEHINT_MIN;
bounds.upperBound = ZSTD_SRCSIZEHINT_MAX;
return bounds;
default: default:
{ ZSTD_bounds const boundError = { ERROR(parameter_unsupported), 0, 0 }; { ZSTD_bounds const boundError = { ERROR(parameter_unsupported), 0, 0 };
return boundError; return boundError;
} }
} }
} }
/* ZSTD_cParam_clampBounds: /* ZSTD_cParam_clampBounds:
case ZSTD_c_enableLongDistanceMatching: case ZSTD_c_enableLongDistanceMatching:
case ZSTD_c_ldmHashLog: case ZSTD_c_ldmHashLog:
case ZSTD_c_ldmMinMatch: case ZSTD_c_ldmMinMatch:
case ZSTD_c_ldmBucketSizeLog: case ZSTD_c_ldmBucketSizeLog:
case ZSTD_c_ldmHashRateLog: case ZSTD_c_ldmHashRateLog:
case ZSTD_c_forceAttachDict: case ZSTD_c_forceAttachDict:
case ZSTD_c_literalCompressionMode: case ZSTD_c_literalCompressionMode:
case ZSTD_c_targetCBlockSize: case ZSTD_c_targetCBlockSize:
case ZSTD_c_srcSizeHint:
default: default:
return 0; return 0;
} }
} }
size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value) size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value)
{ {
DEBUGLOG(4, "ZSTD_CCtx_setParameter (%i, %i)", (int)param, value); DEBUGLOG(4, "ZSTD_CCtx_setParameter (%i, %i)", (int)param, value);
case ZSTD_c_jobSize: case ZSTD_c_jobSize:
case ZSTD_c_overlapLog: case ZSTD_c_overlapLog:
case ZSTD_c_rsyncable: case ZSTD_c_rsyncable:
case ZSTD_c_enableLongDistanceMatching: case ZSTD_c_enableLongDistanceMatching:
case ZSTD_c_ldmHashLog: case ZSTD_c_ldmHashLog:
case ZSTD_c_ldmMinMatch: case ZSTD_c_ldmMinMatch:
case ZSTD_c_ldmBucketSizeLog: case ZSTD_c_ldmBucketSizeLog:
case ZSTD_c_targetCBlockSize: case ZSTD_c_targetCBlockSize:
case ZSTD_c_srcSizeHint:
break; break;
default: RETURN_ERROR(parameter_unsupported); default: RETURN_ERROR(parameter_unsupported);
} }
return ZSTD_CCtxParams_setParameter(&cctx->requestedParams, param, value); return ZSTD_CCtxParams_setParameter(&cctx->requestedParams, param, value);
} }
size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams, size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams,
ZSTD_cParameter param, int value) ZSTD_cParameter param, int value)
{ {
DEBUGLOG(4, "ZSTD_CCtxParams_setParameter (%i, %i)", (int)param, value); DEBUGLOG(4, "ZSTD_CCtxParams_setParameter (%i, %i)", (int)param, value);
switch(param) switch(param)
{ {
case ZSTD_c_format : case ZSTD_c_format :
BOUNDCHECK(ZSTD_c_format, value); BOUNDCHECK(ZSTD_c_format, value);
CCtxParams->format = (ZSTD_format_e)value; CCtxParams->format = (ZSTD_format_e)value;
return (size_t)CCtxParams->format; return (size_t)CCtxParams->format;
case ZSTD_c_compressionLevel : { case ZSTD_c_compressionLevel : {
FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value)); FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value));
if (value) { /* 0 : does not change current level */ if (value) { /* 0 : does not change current level */
CCtxParams->compressionLevel = value; CCtxParams->compressionLevel = value;
} }
if (CCtxParams->compressionLevel >= 0) return CCtxParams->compressionLevel; if (CCtxParams->compressionLevel >= 0) return (size_t)CCtxParams->compressionLevel;
return 0; /* return type (size_t) cannot represent negative values */ return 0; /* return type (size_t) cannot represent negative values */
} }
case ZSTD_c_windowLog : case ZSTD_c_windowLog :
if (value!=0) /* 0 => use default */ if (value!=0) /* 0 => use default */
BOUNDCHECK(ZSTD_c_windowLog, value); BOUNDCHECK(ZSTD_c_windowLog, value);
CCtxParams->cParams.windowLog = value; CCtxParams->cParams.windowLog = (U32)value;
return CCtxParams->cParams.windowLog; return CCtxParams->cParams.windowLog;
case ZSTD_c_hashLog : case ZSTD_c_hashLog :
if (value!=0) /* 0 => use default */ if (value!=0) /* 0 => use default */
BOUNDCHECK(ZSTD_c_hashLog, value); BOUNDCHECK(ZSTD_c_hashLog, value);
CCtxParams->cParams.hashLog = value; CCtxParams->cParams.hashLog = (U32)value;
return CCtxParams->cParams.hashLog; return CCtxParams->cParams.hashLog;
case ZSTD_c_chainLog : case ZSTD_c_chainLog :
if (value!=0) /* 0 => use default */ if (value!=0) /* 0 => use default */
BOUNDCHECK(ZSTD_c_chainLog, value); BOUNDCHECK(ZSTD_c_chainLog, value);
CCtxParams->cParams.chainLog = value; CCtxParams->cParams.chainLog = (U32)value;
return CCtxParams->cParams.chainLog; return CCtxParams->cParams.chainLog;
case ZSTD_c_searchLog : case ZSTD_c_searchLog :
if (value!=0) /* 0 => use default */ if (value!=0) /* 0 => use default */
BOUNDCHECK(ZSTD_c_searchLog, value); BOUNDCHECK(ZSTD_c_searchLog, value);
CCtxParams->cParams.searchLog = value; CCtxParams->cParams.searchLog = (U32)value;
return value; return (size_t)value;
case ZSTD_c_minMatch : case ZSTD_c_minMatch :
if (value!=0) /* 0 => use default */ if (value!=0) /* 0 => use default */
BOUNDCHECK(ZSTD_c_minMatch, value); BOUNDCHECK(ZSTD_c_minMatch, value);
CCtxParams->cParams.minMatch = value; CCtxParams->cParams.minMatch = value;
return CCtxParams->cParams.minMatch; return CCtxParams->cParams.minMatch;
case ZSTD_c_targetLength : case ZSTD_c_targetLength :
return CCtxParams->ldmParams.hashRateLog; return CCtxParams->ldmParams.hashRateLog;
case ZSTD_c_targetCBlockSize : case ZSTD_c_targetCBlockSize :
if (value!=0) /* 0 ==> default */ if (value!=0) /* 0 ==> default */
BOUNDCHECK(ZSTD_c_targetCBlockSize, value); BOUNDCHECK(ZSTD_c_targetCBlockSize, value);
CCtxParams->targetCBlockSize = value; CCtxParams->targetCBlockSize = value;
return CCtxParams->targetCBlockSize; return CCtxParams->targetCBlockSize;
case ZSTD_c_srcSizeHint :
if (value!=0) /* 0 ==> default */
BOUNDCHECK(ZSTD_c_srcSizeHint, value);
CCtxParams->srcSizeHint = value;
return CCtxParams->srcSizeHint;
default: RETURN_ERROR(parameter_unsupported, "unknown parameter"); default: RETURN_ERROR(parameter_unsupported, "unknown parameter");
} }
} }
size_t ZSTD_CCtx_getParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int* value) size_t ZSTD_CCtx_getParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int* value)
{ {
return ZSTD_CCtxParams_getParameter(&cctx->requestedParams, param, value); return ZSTD_CCtxParams_getParameter(&cctx->requestedParams, param, value);
} }
*value = CCtxParams->ldmParams.bucketSizeLog; *value = CCtxParams->ldmParams.bucketSizeLog;
break; break;
case ZSTD_c_ldmHashRateLog : case ZSTD_c_ldmHashRateLog :
*value = CCtxParams->ldmParams.hashRateLog; *value = CCtxParams->ldmParams.hashRateLog;
break; break;
case ZSTD_c_targetCBlockSize : case ZSTD_c_targetCBlockSize :
*value = (int)CCtxParams->targetCBlockSize; *value = (int)CCtxParams->targetCBlockSize;
break; break;
case ZSTD_c_srcSizeHint :
*value = (int)CCtxParams->srcSizeHint;
break;
default: RETURN_ERROR(parameter_unsupported, "unknown parameter"); default: RETURN_ERROR(parameter_unsupported, "unknown parameter");
} }
return 0; return 0;
} }
/** ZSTD_CCtx_setParametersUsingCCtxParams() : /** ZSTD_CCtx_setParametersUsingCCtxParams() :
* just applies `params` into `cctx` * just applies `params` into `cctx`
* no action is performed, parameters are merely stored. * no action is performed, parameters are merely stored.
{ {
cPar = ZSTD_clampCParams(cPar); /* resulting cPar is necessarily valid (all parameters within range) */ cPar = ZSTD_clampCParams(cPar); /* resulting cPar is necessarily valid (all parameters within range) */
return ZSTD_adjustCParams_internal(cPar, srcSize, dictSize); return ZSTD_adjustCParams_internal(cPar, srcSize, dictSize);
} }
ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams( ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams(
const ZSTD_CCtx_params* CCtxParams, U64 srcSizeHint, size_t dictSize) const ZSTD_CCtx_params* CCtxParams, U64 srcSizeHint, size_t dictSize)
{ {
ZSTD_compressionParameters cParams = ZSTD_getCParams(CCtxParams->compressionLevel, srcSizeHint, dictSize); ZSTD_compressionParameters cParams;
if (srcSizeHint == ZSTD_CONTENTSIZE_UNKNOWN && CCtxParams->srcSizeHint > 0) {
srcSizeHint = CCtxParams->srcSizeHint;
}
cParams = ZSTD_getCParams(CCtxParams->compressionLevel, srcSizeHint, dictSize);
if (CCtxParams->ldmParams.enableLdm) cParams.windowLog = ZSTD_LDM_DEFAULT_WINDOW_LOG; if (CCtxParams->ldmParams.enableLdm) cParams.windowLog = ZSTD_LDM_DEFAULT_WINDOW_LOG;
if (CCtxParams->cParams.windowLog) cParams.windowLog = CCtxParams->cParams.windowLog; if (CCtxParams->cParams.windowLog) cParams.windowLog = CCtxParams->cParams.windowLog;
if (CCtxParams->cParams.hashLog) cParams.hashLog = CCtxParams->cParams.hashLog; if (CCtxParams->cParams.hashLog) cParams.hashLog = CCtxParams->cParams.hashLog;
if (CCtxParams->cParams.chainLog) cParams.chainLog = CCtxParams->cParams.chainLog; if (CCtxParams->cParams.chainLog) cParams.chainLog = CCtxParams->cParams.chainLog;
if (CCtxParams->cParams.searchLog) cParams.searchLog = CCtxParams->cParams.searchLog; if (CCtxParams->cParams.searchLog) cParams.searchLog = CCtxParams->cParams.searchLog;
if (CCtxParams->cParams.minMatch) cParams.minMatch = CCtxParams->cParams.minMatch; if (CCtxParams->cParams.minMatch) cParams.minMatch = CCtxParams->cParams.minMatch;
if (CCtxParams->cParams.targetLength) cParams.targetLength = CCtxParams->cParams.targetLength; if (CCtxParams->cParams.targetLength) cParams.targetLength = CCtxParams->cParams.targetLength;
if (CCtxParams->cParams.strategy) cParams.strategy = CCtxParams->cParams.strategy; if (CCtxParams->cParams.strategy) cParams.strategy = CCtxParams->cParams.strategy;
assert(!ZSTD_checkCParams(cParams)); assert(!ZSTD_checkCParams(cParams));
return ZSTD_adjustCParams_internal(cParams, srcSizeHint, dictSize); return ZSTD_adjustCParams_internal(cParams, srcSizeHint, dictSize);
} }
static size_t static size_t
ZSTD_sizeof_matchState(const ZSTD_compressionParameters* const cParams, ZSTD_sizeof_matchState(const ZSTD_compressionParameters* const cParams,
const U32 forCCtx) const U32 forCCtx)
{ {
size_t const chainSize = (cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cParams->chainLog); size_t const chainSize = (cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cParams->chainLog);
size_t const hSize = ((size_t)1) << cParams->hashLog; size_t const hSize = ((size_t)1) << cParams->hashLog;
U32 const hashLog3 = (forCCtx && cParams->minMatch==3) ? MIN(ZSTD_HASHLOG3_MAX, cParams->windowLog) : 0; U32 const hashLog3 = (forCCtx && cParams->minMatch==3) ? MIN(ZSTD_HASHLOG3_MAX, cParams->windowLog) : 0;
size_t const h3Size = ((size_t)1) << hashLog3; size_t const h3Size = hashLog3 ? ((size_t)1) << hashLog3 : 0;
size_t const tableSpace = (chainSize + hSize + h3Size) * sizeof(U32); /* We don't use ZSTD_cwksp_alloc_size() here because the tables aren't
size_t const optPotentialSpace = ((MaxML+1) + (MaxLL+1) + (MaxOff+1) + (1<<Litbits)) * sizeof(U32) * surrounded by redzones in ASAN. */
+ (ZSTD_OPT_NUM+1) * (sizeof(ZSTD_match_t)+sizeof(ZSTD_optimal_t)); size_t const tableSpace = chainSize * sizeof(U32)
+ hSize * sizeof(U32)
+ h3Size * sizeof(U32);
size_t const optPotentialSpace =
ZSTD_cwksp_alloc_size((MaxML+1) * sizeof(U32))
+ ZSTD_cwksp_alloc_size((MaxLL+1) * sizeof(U32))
+ ZSTD_cwksp_alloc_size((MaxOff+1) * sizeof(U32))
+ ZSTD_cwksp_alloc_size((1<<Litbits) * sizeof(U32))
+ ZSTD_cwksp_alloc_size((ZSTD_OPT_NUM+1) * sizeof(ZSTD_match_t))
+ ZSTD_cwksp_alloc_size((ZSTD_OPT_NUM+1) * sizeof(ZSTD_optimal_t));
size_t const optSpace = (forCCtx && (cParams->strategy >= ZSTD_btopt)) size_t const optSpace = (forCCtx && (cParams->strategy >= ZSTD_btopt))
? optPotentialSpace ? optPotentialSpace
: 0; : 0;
DEBUGLOG(4, "chainSize: %u - hSize: %u - h3Size: %u", DEBUGLOG(4, "chainSize: %u - hSize: %u - h3Size: %u",
(U32)chainSize, (U32)hSize, (U32)h3Size); (U32)chainSize, (U32)hSize, (U32)h3Size);
return tableSpace + optSpace; return tableSpace + optSpace;
} }
size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params) size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params)
{ {
RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only."); RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
{ ZSTD_compressionParameters const cParams = { ZSTD_compressionParameters const cParams =
ZSTD_getCParamsFromCCtxParams(params, 0, 0); ZSTD_getCParamsFromCCtxParams(params, 0, 0);
size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << cParams.windowLog); size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << cParams.windowLog);
U32 const divider = (cParams.minMatch==3) ? 3 : 4; U32 const divider = (cParams.minMatch==3) ? 3 : 4;
size_t const maxNbSeq = blockSize / divider; size_t const maxNbSeq = blockSize / divider;
size_t const tokenSpace = WILDCOPY_OVERLENGTH + blockSize + 11*maxNbSeq; size_t const tokenSpace = ZSTD_cwksp_alloc_size(WILDCOPY_OVERLENGTH + blockSize)
size_t const entropySpace = HUF_WORKSPACE_SIZE; + ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(seqDef))
size_t const blockStateSpace = 2 * sizeof(ZSTD_compressedBlockState_t); + 3 * ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(BYTE));
size_t const entropySpace = ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE);
size_t const blockStateSpace = 2 * ZSTD_cwksp_alloc_size(sizeof(ZSTD_compressedBlockState_t));
size_t const matchStateSize = ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 1); size_t const matchStateSize = ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 1);
size_t const ldmSpace = ZSTD_ldm_getTableSize(params->ldmParams); size_t const ldmSpace = ZSTD_ldm_getTableSize(params->ldmParams);
size_t const ldmSeqSpace = ZSTD_ldm_getMaxNbSeq(params->ldmParams, blockSize) * sizeof(rawSeq); size_t const ldmSeqSpace = ZSTD_cwksp_alloc_size(ZSTD_ldm_getMaxNbSeq(params->ldmParams, blockSize) * sizeof(rawSeq));
size_t const neededSpace = entropySpace + blockStateSpace + tokenSpace + size_t const neededSpace = entropySpace + blockStateSpace + tokenSpace +
matchStateSize + ldmSpace + ldmSeqSpace; matchStateSize + ldmSpace + ldmSeqSpace;
size_t const cctxSpace = ZSTD_cwksp_alloc_size(sizeof(ZSTD_CCtx));
DEBUGLOG(5, "sizeof(ZSTD_CCtx) : %u", (U32)sizeof(ZSTD_CCtx)); DEBUGLOG(5, "sizeof(ZSTD_CCtx) : %u", (U32)cctxSpace);
DEBUGLOG(5, "estimate workSpace : %u", (U32)neededSpace); DEBUGLOG(5, "estimate workspace : %u", (U32)neededSpace);
return sizeof(ZSTD_CCtx) + neededSpace; return cctxSpace + neededSpace;
} }
} }
size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams) size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams)
{ {
ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams); ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams);
return ZSTD_estimateCCtxSize_usingCCtxParams(&params); return ZSTD_estimateCCtxSize_usingCCtxParams(&params);
} }
{ {
RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only."); RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
{ ZSTD_compressionParameters const cParams = { ZSTD_compressionParameters const cParams =
ZSTD_getCParamsFromCCtxParams(params, 0, 0); ZSTD_getCParamsFromCCtxParams(params, 0, 0);
size_t const CCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(params); size_t const CCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(params);
size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << cParams.windowLog); size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << cParams.windowLog);
size_t const inBuffSize = ((size_t)1 << cParams.windowLog) + blockSize; size_t const inBuffSize = ((size_t)1 << cParams.windowLog) + blockSize;
size_t const outBuffSize = ZSTD_compressBound(blockSize) + 1; size_t const outBuffSize = ZSTD_compressBound(blockSize) + 1;
size_t const streamingSize = inBuffSize + outBuffSize; size_t const streamingSize = ZSTD_cwksp_alloc_size(inBuffSize)
+ ZSTD_cwksp_alloc_size(outBuffSize);
return CCtxSize + streamingSize; return CCtxSize + streamingSize;
} }
} }
size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams) size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams)
{ {
ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams); ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams);
if (cctx->appliedParams.nbWorkers > 0) { if (cctx->appliedParams.nbWorkers > 0) {
return ZSTDMT_toFlushNow(cctx->mtctx); return ZSTDMT_toFlushNow(cctx->mtctx);
} }
#endif #endif
(void)cctx; (void)cctx;
return 0; /* over-simplification; could also check if context is currently running in streaming mode, and in which case, report how many bytes are left to be flushed within output buffer */ return 0; /* over-simplification; could also check if context is currently running in streaming mode, and in which case, report how many bytes are left to be flushed within output buffer */
} }
static U32 ZSTD_equivalentCParams(ZSTD_compressionParameters cParams1,
ZSTD_compressionParameters cParams2)
{
return (cParams1.hashLog == cParams2.hashLog)
& (cParams1.chainLog == cParams2.chainLog)
& (cParams1.strategy == cParams2.strategy) /* opt parser space */
& ((cParams1.minMatch==3) == (cParams2.minMatch==3)); /* hashlog3 space */
}
static void ZSTD_assertEqualCParams(ZSTD_compressionParameters cParams1, static void ZSTD_assertEqualCParams(ZSTD_compressionParameters cParams1,
ZSTD_compressionParameters cParams2) ZSTD_compressionParameters cParams2)
{ {
(void)cParams1; (void)cParams1;
(void)cParams2; (void)cParams2;
assert(cParams1.windowLog == cParams2.windowLog); assert(cParams1.windowLog == cParams2.windowLog);
assert(cParams1.chainLog == cParams2.chainLog); assert(cParams1.chainLog == cParams2.chainLog);
assert(cParams1.hashLog == cParams2.hashLog); assert(cParams1.hashLog == cParams2.hashLog);
assert(cParams1.searchLog == cParams2.searchLog); assert(cParams1.searchLog == cParams2.searchLog);
assert(cParams1.minMatch == cParams2.minMatch); assert(cParams1.minMatch == cParams2.minMatch);
assert(cParams1.targetLength == cParams2.targetLength); assert(cParams1.targetLength == cParams2.targetLength);
assert(cParams1.strategy == cParams2.strategy); assert(cParams1.strategy == cParams2.strategy);
} }
/** The parameters are equivalent if ldm is not enabled in both sets or
* all the parameters are equivalent. */
static U32 ZSTD_equivalentLdmParams(ldmParams_t ldmParams1,
ldmParams_t ldmParams2)
{
return (!ldmParams1.enableLdm && !ldmParams2.enableLdm) ||
(ldmParams1.enableLdm == ldmParams2.enableLdm &&
ldmParams1.hashLog == ldmParams2.hashLog &&
ldmParams1.bucketSizeLog == ldmParams2.bucketSizeLog &&
ldmParams1.minMatchLength == ldmParams2.minMatchLength &&
ldmParams1.hashRateLog == ldmParams2.hashRateLog);
}
typedef enum { ZSTDb_not_buffered, ZSTDb_buffered } ZSTD_buffered_policy_e;
/* ZSTD_sufficientBuff() :
* check internal buffers exist for streaming if buffPol == ZSTDb_buffered .
* Note : they are assumed to be correctly sized if ZSTD_equivalentCParams()==1 */
static U32 ZSTD_sufficientBuff(size_t bufferSize1, size_t maxNbSeq1,
size_t maxNbLit1,
ZSTD_buffered_policy_e buffPol2,
ZSTD_compressionParameters cParams2,
U64 pledgedSrcSize)
{
size_t const windowSize2 = MAX(1, (size_t)MIN(((U64)1 << cParams2.windowLog), pledgedSrcSize));
size_t const blockSize2 = MIN(ZSTD_BLOCKSIZE_MAX, windowSize2);
size_t const maxNbSeq2 = blockSize2 / ((cParams2.minMatch == 3) ? 3 : 4);
size_t const maxNbLit2 = blockSize2;
size_t const neededBufferSize2 = (buffPol2==ZSTDb_buffered) ? windowSize2 + blockSize2 : 0;
DEBUGLOG(4, "ZSTD_sufficientBuff: is neededBufferSize2=%u <= bufferSize1=%u",
(U32)neededBufferSize2, (U32)bufferSize1);
DEBUGLOG(4, "ZSTD_sufficientBuff: is maxNbSeq2=%u <= maxNbSeq1=%u",
(U32)maxNbSeq2, (U32)maxNbSeq1);
DEBUGLOG(4, "ZSTD_sufficientBuff: is maxNbLit2=%u <= maxNbLit1=%u",
(U32)maxNbLit2, (U32)maxNbLit1);
return (maxNbLit2 <= maxNbLit1)
& (maxNbSeq2 <= maxNbSeq1)
& (neededBufferSize2 <= bufferSize1);
}
/** Equivalence for resetCCtx purposes */
static U32 ZSTD_equivalentParams(ZSTD_CCtx_params params1,
ZSTD_CCtx_params params2,
size_t buffSize1,
size_t maxNbSeq1, size_t maxNbLit1,
ZSTD_buffered_policy_e buffPol2,
U64 pledgedSrcSize)
{
DEBUGLOG(4, "ZSTD_equivalentParams: pledgedSrcSize=%u", (U32)pledgedSrcSize);
if (!ZSTD_equivalentCParams(params1.cParams, params2.cParams)) {
DEBUGLOG(4, "ZSTD_equivalentCParams() == 0");
return 0;
}
if (!ZSTD_equivalentLdmParams(params1.ldmParams, params2.ldmParams)) {
DEBUGLOG(4, "ZSTD_equivalentLdmParams() == 0");
return 0;
}
if (!ZSTD_sufficientBuff(buffSize1, maxNbSeq1, maxNbLit1, buffPol2,
params2.cParams, pledgedSrcSize)) {
DEBUGLOG(4, "ZSTD_sufficientBuff() == 0");
return 0;
}
return 1;
}
static void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs) static void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs)
{ {
int i; int i;
for (i = 0; i < ZSTD_REP_NUM; ++i) for (i = 0; i < ZSTD_REP_NUM; ++i)
bs->rep[i] = repStartValue[i]; bs->rep[i] = repStartValue[i];
bs->entropy.huf.repeatMode = HUF_repeat_none; bs->entropy.huf.repeatMode = HUF_repeat_none;
bs->entropy.fse.offcode_repeatMode = FSE_repeat_none; bs->entropy.fse.offcode_repeatMode = FSE_repeat_none;
bs->entropy.fse.matchlength_repeatMode = FSE_repeat_none; bs->entropy.fse.matchlength_repeatMode = FSE_repeat_none;
ZSTD_window_clear(&ms->window); ZSTD_window_clear(&ms->window);
ms->nextToUpdate = ms->window.dictLimit; ms->nextToUpdate = ms->window.dictLimit;
ms->loadedDictEnd = 0; ms->loadedDictEnd = 0;
ms->opt.litLengthSum = 0; /* force reset of btopt stats */ ms->opt.litLengthSum = 0; /* force reset of btopt stats */
ms->dictMatchState = NULL; ms->dictMatchState = NULL;
} }
/*! ZSTD_continueCCtx() : /**
* reuse CCtx without reset (note : requires no dictionary) */ * Indicates whether this compression proceeds directly from user-provided
static size_t ZSTD_continueCCtx(ZSTD_CCtx* cctx, ZSTD_CCtx_params params, U64 pledgedSrcSize) * source buffer to user-provided destination buffer (ZSTDb_not_buffered), or
{ * whether the context needs to buffer the input/output (ZSTDb_buffered).
size_t const windowSize = MAX(1, (size_t)MIN(((U64)1 << params.cParams.windowLog), pledgedSrcSize)); */
size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, windowSize); typedef enum {
DEBUGLOG(4, "ZSTD_continueCCtx: re-use context in place"); ZSTDb_not_buffered,
ZSTDb_buffered
cctx->blockSize = blockSize; /* previous block size could be different even for same windowLog, due to pledgedSrcSize */ } ZSTD_buffered_policy_e;
cctx->appliedParams = params;
cctx->blockState.matchState.cParams = params.cParams;
cctx->pledgedSrcSizePlusOne = pledgedSrcSize+1;
cctx->consumedSrcSize = 0;
cctx->producedCSize = 0;
if (pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN)
cctx->appliedParams.fParams.contentSizeFlag = 0;
DEBUGLOG(4, "pledged content size : %u ; flag : %u",
(U32)pledgedSrcSize, cctx->appliedParams.fParams.contentSizeFlag);
cctx->stage = ZSTDcs_init;
cctx->dictID = 0;
if (params.ldmParams.enableLdm)
ZSTD_window_clear(&cctx->ldmState.window);
ZSTD_referenceExternalSequences(cctx, NULL, 0);
ZSTD_invalidateMatchState(&cctx->blockState.matchState);
ZSTD_reset_compressedBlockState(cctx->blockState.prevCBlock);
XXH64_reset(&cctx->xxhState, 0);
return 0;
}
typedef enum { ZSTDcrp_continue, ZSTDcrp_noMemset } ZSTD_compResetPolicy_e; /**
* Controls, for this matchState reset, whether the tables need to be cleared /
* prepared for the coming compression (ZSTDcrp_makeClean), or whether the
* tables can be left unclean (ZSTDcrp_leaveDirty), because we know that a
* subsequent operation will overwrite the table space anyways (e.g., copying
* the matchState contents in from a CDict).
*/
typedef enum {
ZSTDcrp_makeClean,
ZSTDcrp_leaveDirty
} ZSTD_compResetPolicy_e;
typedef enum { ZSTD_resetTarget_CDict, ZSTD_resetTarget_CCtx } ZSTD_resetTarget_e; /**
* Controls, for this matchState reset, whether indexing can continue where it
* left off (ZSTDirp_continue), or whether it needs to be restarted from zero
* (ZSTDirp_reset).
*/
typedef enum {
ZSTDirp_continue,
ZSTDirp_reset
} ZSTD_indexResetPolicy_e;
typedef enum {
ZSTD_resetTarget_CDict,
ZSTD_resetTarget_CCtx
} ZSTD_resetTarget_e;
static void* static size_t
ZSTD_reset_matchState(ZSTD_matchState_t* ms, ZSTD_reset_matchState(ZSTD_matchState_t* ms,
void* ptr, ZSTD_cwksp* ws,
const ZSTD_compressionParameters* cParams, const ZSTD_compressionParameters* cParams,
ZSTD_compResetPolicy_e const crp, ZSTD_resetTarget_e const forWho) const ZSTD_compResetPolicy_e crp,
const ZSTD_indexResetPolicy_e forceResetIndex,
const ZSTD_resetTarget_e forWho)
{ {
size_t const chainSize = (cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cParams->chainLog); size_t const chainSize = (cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cParams->chainLog);
size_t const hSize = ((size_t)1) << cParams->hashLog; size_t const hSize = ((size_t)1) << cParams->hashLog;
U32 const hashLog3 = ((forWho == ZSTD_resetTarget_CCtx) && cParams->minMatch==3) ? MIN(ZSTD_HASHLOG3_MAX, cParams->windowLog) : 0; U32 const hashLog3 = ((forWho == ZSTD_resetTarget_CCtx) && cParams->minMatch==3) ? MIN(ZSTD_HASHLOG3_MAX, cParams->windowLog) : 0;
size_t const h3Size = ((size_t)1) << hashLog3; size_t const h3Size = hashLog3 ? ((size_t)1) << hashLog3 : 0;
size_t const tableSpace = (chainSize + hSize + h3Size) * sizeof(U32);
assert(((size_t)ptr & 3) == 0);
ms->hashLog3 = hashLog3; DEBUGLOG(4, "reset indices : %u", forceResetIndex == ZSTDirp_reset);
if (forceResetIndex == ZSTDirp_reset) {
memset(&ms->window, 0, sizeof(ms->window)); memset(&ms->window, 0, sizeof(ms->window));
ms->window.dictLimit = 1; /* start from 1, so that 1st position is valid */ ms->window.dictLimit = 1; /* start from 1, so that 1st position is valid */
ms->window.lowLimit = 1; /* it ensures first and later CCtx usages compress the same */ ms->window.lowLimit = 1; /* it ensures first and later CCtx usages compress the same */
ms->window.nextSrc = ms->window.base + 1; /* see issue #1241 */ ms->window.nextSrc = ms->window.base + 1; /* see issue #1241 */
ZSTD_cwksp_mark_tables_dirty(ws);
}
ms->hashLog3 = hashLog3;
ZSTD_invalidateMatchState(ms); ZSTD_invalidateMatchState(ms);
assert(!ZSTD_cwksp_reserve_failed(ws)); /* check that allocation hasn't already failed */
ZSTD_cwksp_clear_tables(ws);
DEBUGLOG(5, "reserving table space");
/* table Space */
ms->hashTable = (U32*)ZSTD_cwksp_reserve_table(ws, hSize * sizeof(U32));
ms->chainTable = (U32*)ZSTD_cwksp_reserve_table(ws, chainSize * sizeof(U32));
ms->hashTable3 = (U32*)ZSTD_cwksp_reserve_table(ws, h3Size * sizeof(U32));
RETURN_ERROR_IF(ZSTD_cwksp_reserve_failed(ws), memory_allocation,
"failed a workspace allocation in ZSTD_reset_matchState");
DEBUGLOG(4, "reset table : %u", crp!=ZSTDcrp_leaveDirty);
if (crp!=ZSTDcrp_leaveDirty) {
/* reset tables only */
ZSTD_cwksp_clean_tables(ws);
}
/* opt parser space */ /* opt parser space */
if ((forWho == ZSTD_resetTarget_CCtx) && (cParams->strategy >= ZSTD_btopt)) { if ((forWho == ZSTD_resetTarget_CCtx) && (cParams->strategy >= ZSTD_btopt)) {
DEBUGLOG(4, "reserving optimal parser space"); DEBUGLOG(4, "reserving optimal parser space");
ms->opt.litFreq = (unsigned*)ptr; ms->opt.litFreq = (unsigned*)ZSTD_cwksp_reserve_aligned(ws, (1<<Litbits) * sizeof(unsigned));
ms->opt.litLengthFreq = ms->opt.litFreq + (1<<Litbits); ms->opt.litLengthFreq = (unsigned*)ZSTD_cwksp_reserve_aligned(ws, (MaxLL+1) * sizeof(unsigned));
ms->opt.matchLengthFreq = ms->opt.litLengthFreq + (MaxLL+1); ms->opt.matchLengthFreq = (unsigned*)ZSTD_cwksp_reserve_aligned(ws, (MaxML+1) * sizeof(unsigned));
ms->opt.offCodeFreq = ms->opt.matchLengthFreq + (MaxML+1); ms->opt.offCodeFreq = (unsigned*)ZSTD_cwksp_reserve_aligned(ws, (MaxOff+1) * sizeof(unsigned));
ptr = ms->opt.offCodeFreq + (MaxOff+1); ms->opt.matchTable = (ZSTD_match_t*)ZSTD_cwksp_reserve_aligned(ws, (ZSTD_OPT_NUM+1) * sizeof(ZSTD_match_t));
ms->opt.matchTable = (ZSTD_match_t*)ptr; ms->opt.priceTable = (ZSTD_optimal_t*)ZSTD_cwksp_reserve_aligned(ws, (ZSTD_OPT_NUM+1) * sizeof(ZSTD_optimal_t));
ptr = ms->opt.matchTable + ZSTD_OPT_NUM+1;
ms->opt.priceTable = (ZSTD_optimal_t*)ptr;
ptr = ms->opt.priceTable + ZSTD_OPT_NUM+1;
} }
/* table Space */
DEBUGLOG(4, "reset table : %u", crp!=ZSTDcrp_noMemset);
assert(((size_t)ptr & 3) == 0); /* ensure ptr is properly aligned */
if (crp!=ZSTDcrp_noMemset) memset(ptr, 0, tableSpace); /* reset tables only */
ms->hashTable = (U32*)(ptr);
ms->chainTable = ms->hashTable + hSize;
ms->hashTable3 = ms->chainTable + chainSize;
ptr = ms->hashTable3 + h3Size;
ms->cParams = *cParams; ms->cParams = *cParams;
assert(((size_t)ptr & 3) == 0); RETURN_ERROR_IF(ZSTD_cwksp_reserve_failed(ws), memory_allocation,
return ptr; "failed a workspace allocation in ZSTD_reset_matchState");
return 0;
} }
/* ZSTD_indexTooCloseToMax() : /* ZSTD_indexTooCloseToMax() :
* minor optimization : prefer memset() rather than reduceIndex() * minor optimization : prefer memset() rather than reduceIndex()
* which is measurably slow in some circumstances (reported for Visual Studio). * which is measurably slow in some circumstances (reported for Visual Studio).
* Works when re-using a context for a lot of smallish inputs : * Works when re-using a context for a lot of smallish inputs :
* if all inputs are smaller than ZSTD_INDEXOVERFLOW_MARGIN, * if all inputs are smaller than ZSTD_INDEXOVERFLOW_MARGIN,
* memset() will be triggered before reduceIndex(). * memset() will be triggered before reduceIndex().
*/ */
#define ZSTD_INDEXOVERFLOW_MARGIN (16 MB) #define ZSTD_INDEXOVERFLOW_MARGIN (16 MB)
static int ZSTD_indexTooCloseToMax(ZSTD_window_t w) static int ZSTD_indexTooCloseToMax(ZSTD_window_t w)
{ {
return (size_t)(w.nextSrc - w.base) > (ZSTD_CURRENT_MAX - ZSTD_INDEXOVERFLOW_MARGIN); return (size_t)(w.nextSrc - w.base) > (ZSTD_CURRENT_MAX - ZSTD_INDEXOVERFLOW_MARGIN);
} }
#define ZSTD_WORKSPACETOOLARGE_FACTOR 3 /* define "workspace is too large" as this number of times larger than needed */
#define ZSTD_WORKSPACETOOLARGE_MAXDURATION 128 /* when workspace is continuously too large
* during at least this number of times,
* context's memory usage is considered wasteful,
* because it's sized to handle a worst case scenario which rarely happens.
* In which case, resize it down to free some memory */
/*! ZSTD_resetCCtx_internal() : /*! ZSTD_resetCCtx_internal() :
note : `params` are assumed fully validated at this stage */ note : `params` are assumed fully validated at this stage */
static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc, static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
ZSTD_CCtx_params params, ZSTD_CCtx_params params,
U64 const pledgedSrcSize, U64 const pledgedSrcSize,
ZSTD_compResetPolicy_e const crp, ZSTD_compResetPolicy_e const crp,
ZSTD_buffered_policy_e const zbuff) ZSTD_buffered_policy_e const zbuff)
{ {
ZSTD_cwksp* const ws = &zc->workspace;
DEBUGLOG(4, "ZSTD_resetCCtx_internal: pledgedSrcSize=%u, wlog=%u", DEBUGLOG(4, "ZSTD_resetCCtx_internal: pledgedSrcSize=%u, wlog=%u",
(U32)pledgedSrcSize, params.cParams.windowLog); (U32)pledgedSrcSize, params.cParams.windowLog);
assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams))); assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams)));
if (crp == ZSTDcrp_continue) { zc->isFirstBlock = 1;
if (ZSTD_equivalentParams(zc->appliedParams, params,
zc->inBuffSize,
zc->seqStore.maxNbSeq, zc->seqStore.maxNbLit,
zbuff, pledgedSrcSize) ) {
DEBUGLOG(4, "ZSTD_equivalentParams()==1 -> consider continue mode");
zc->workSpaceOversizedDuration += (zc->workSpaceOversizedDuration > 0); /* if it was too large, it still is */
if (zc->workSpaceOversizedDuration <= ZSTD_WORKSPACETOOLARGE_MAXDURATION) {
DEBUGLOG(4, "continue mode confirmed (wLog1=%u, blockSize1=%zu)",
zc->appliedParams.cParams.windowLog, zc->blockSize);
if (ZSTD_indexTooCloseToMax(zc->blockState.matchState.window)) {
/* prefer a reset, faster than a rescale */
ZSTD_reset_matchState(&zc->blockState.matchState,
zc->entropyWorkspace + HUF_WORKSPACE_SIZE_U32,
&params.cParams,
crp, ZSTD_resetTarget_CCtx);
}
return ZSTD_continueCCtx(zc, params, pledgedSrcSize);
} } }
DEBUGLOG(4, "ZSTD_equivalentParams()==0 -> reset CCtx");
if (params.ldmParams.enableLdm) { if (params.ldmParams.enableLdm) {
/* Adjust long distance matching parameters */ /* Adjust long distance matching parameters */
ZSTD_ldm_adjustParameters(&params.ldmParams, &params.cParams); ZSTD_ldm_adjustParameters(&params.ldmParams, &params.cParams);
assert(params.ldmParams.hashLog >= params.ldmParams.bucketSizeLog); assert(params.ldmParams.hashLog >= params.ldmParams.bucketSizeLog);
assert(params.ldmParams.hashRateLog < 32); assert(params.ldmParams.hashRateLog < 32);
zc->ldmState.hashPower = ZSTD_rollingHash_primePower(params.ldmParams.minMatchLength); zc->ldmState.hashPower = ZSTD_rollingHash_primePower(params.ldmParams.minMatchLength);
} }
{ size_t const windowSize = MAX(1, (size_t)MIN(((U64)1 << params.cParams.windowLog), pledgedSrcSize)); { size_t const windowSize = MAX(1, (size_t)MIN(((U64)1 << params.cParams.windowLog), pledgedSrcSize));
size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, windowSize); size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, windowSize);
U32 const divider = (params.cParams.minMatch==3) ? 3 : 4; U32 const divider = (params.cParams.minMatch==3) ? 3 : 4;
size_t const maxNbSeq = blockSize / divider; size_t const maxNbSeq = blockSize / divider;
size_t const tokenSpace = WILDCOPY_OVERLENGTH + blockSize + 11*maxNbSeq; size_t const tokenSpace = ZSTD_cwksp_alloc_size(WILDCOPY_OVERLENGTH + blockSize)
+ ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(seqDef))
+ 3 * ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(BYTE));
size_t const buffOutSize = (zbuff==ZSTDb_buffered) ? ZSTD_compressBound(blockSize)+1 : 0; size_t const buffOutSize = (zbuff==ZSTDb_buffered) ? ZSTD_compressBound(blockSize)+1 : 0;
size_t const buffInSize = (zbuff==ZSTDb_buffered) ? windowSize + blockSize : 0; size_t const buffInSize = (zbuff==ZSTDb_buffered) ? windowSize + blockSize : 0;
size_t const matchStateSize = ZSTD_sizeof_matchState(&params.cParams, /* forCCtx */ 1); size_t const matchStateSize = ZSTD_sizeof_matchState(&params.cParams, /* forCCtx */ 1);
size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(params.ldmParams, blockSize); size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(params.ldmParams, blockSize);
void* ptr; /* used to partition workSpace */
/* Check if workSpace is large enough, alloc a new one if needed */ ZSTD_indexResetPolicy_e needsIndexReset = ZSTDirp_continue;
{ size_t const entropySpace = HUF_WORKSPACE_SIZE;
size_t const blockStateSpace = 2 * sizeof(ZSTD_compressedBlockState_t); if (ZSTD_indexTooCloseToMax(zc->blockState.matchState.window)) {
size_t const bufferSpace = buffInSize + buffOutSize; needsIndexReset = ZSTDirp_reset;
}
ZSTD_cwksp_bump_oversized_duration(ws, 0);
/* Check if workspace is large enough, alloc a new one if needed */
{ size_t const cctxSpace = zc->staticSize ? ZSTD_cwksp_alloc_size(sizeof(ZSTD_CCtx)) : 0;
size_t const entropySpace = ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE);
size_t const blockStateSpace = 2 * ZSTD_cwksp_alloc_size(sizeof(ZSTD_compressedBlockState_t));
size_t const bufferSpace = ZSTD_cwksp_alloc_size(buffInSize) + ZSTD_cwksp_alloc_size(buffOutSize);
size_t const ldmSpace = ZSTD_ldm_getTableSize(params.ldmParams); size_t const ldmSpace = ZSTD_ldm_getTableSize(params.ldmParams);
size_t const ldmSeqSpace = maxNbLdmSeq * sizeof(rawSeq); size_t const ldmSeqSpace = ZSTD_cwksp_alloc_size(maxNbLdmSeq * sizeof(rawSeq));
size_t const neededSpace = entropySpace + blockStateSpace + ldmSpace + size_t const neededSpace =
ldmSeqSpace + matchStateSize + tokenSpace + cctxSpace +
entropySpace +
blockStateSpace +
ldmSpace +
ldmSeqSpace +
matchStateSize +
tokenSpace +
bufferSpace; bufferSpace;
int const workSpaceTooSmall = zc->workSpaceSize < neededSpace; int const workspaceTooSmall = ZSTD_cwksp_sizeof(ws) < neededSpace;
int const workSpaceTooLarge = zc->workSpaceSize > ZSTD_WORKSPACETOOLARGE_FACTOR * neededSpace; int const workspaceWasteful = ZSTD_cwksp_check_wasteful(ws, neededSpace);
int const workSpaceWasteful = workSpaceTooLarge && (zc->workSpaceOversizedDuration > ZSTD_WORKSPACETOOLARGE_MAXDURATION);
zc->workSpaceOversizedDuration = workSpaceTooLarge ? zc->workSpaceOversizedDuration+1 : 0;
DEBUGLOG(4, "Need %zuKB workspace, including %zuKB for match state, and %zuKB for buffers", DEBUGLOG(4, "Need %zuKB workspace, including %zuKB for match state, and %zuKB for buffers",
neededSpace>>10, matchStateSize>>10, bufferSpace>>10); neededSpace>>10, matchStateSize>>10, bufferSpace>>10);
DEBUGLOG(4, "windowSize: %zu - blockSize: %zu", windowSize, blockSize); DEBUGLOG(4, "windowSize: %zu - blockSize: %zu", windowSize, blockSize);
if (workSpaceTooSmall || workSpaceWasteful) { if (workspaceTooSmall || workspaceWasteful) {
DEBUGLOG(4, "Resize workSpaceSize from %zuKB to %zuKB", DEBUGLOG(4, "Resize workspaceSize from %zuKB to %zuKB",
zc->workSpaceSize >> 10, ZSTD_cwksp_sizeof(ws) >> 10,
neededSpace >> 10); neededSpace >> 10);
RETURN_ERROR_IF(zc->staticSize, memory_allocation, "static cctx : no resize"); RETURN_ERROR_IF(zc->staticSize, memory_allocation, "static cctx : no resize");
zc->workSpaceSize = 0; needsIndexReset = ZSTDirp_reset;
ZSTD_free(zc->workSpace, zc->customMem);
zc->workSpace = ZSTD_malloc(neededSpace, zc->customMem); ZSTD_cwksp_free(ws, zc->customMem);
RETURN_ERROR_IF(zc->workSpace == NULL, memory_allocation); FORWARD_IF_ERROR(ZSTD_cwksp_create(ws, neededSpace, zc->customMem));
zc->workSpaceSize = neededSpace;
zc->workSpaceOversizedDuration = 0;
DEBUGLOG(5, "reserving object space");
/* Statically sized space. /* Statically sized space.
* entropyWorkspace never moves, * entropyWorkspace never moves,
* though prev/next block swap places */ * though prev/next block swap places */
assert(((size_t)zc->workSpace & 3) == 0); /* ensure correct alignment */ assert(ZSTD_cwksp_check_available(ws, 2 * sizeof(ZSTD_compressedBlockState_t)));
assert(zc->workSpaceSize >= 2 * sizeof(ZSTD_compressedBlockState_t)); zc->blockState.prevCBlock = (ZSTD_compressedBlockState_t*) ZSTD_cwksp_reserve_object(ws, sizeof(ZSTD_compressedBlockState_t));
zc->blockState.prevCBlock = (ZSTD_compressedBlockState_t*)zc->workSpace; RETURN_ERROR_IF(zc->blockState.prevCBlock == NULL, memory_allocation, "couldn't allocate prevCBlock");
zc->blockState.nextCBlock = zc->blockState.prevCBlock + 1; zc->blockState.nextCBlock = (ZSTD_compressedBlockState_t*) ZSTD_cwksp_reserve_object(ws, sizeof(ZSTD_compressedBlockState_t));
ptr = zc->blockState.nextCBlock + 1; RETURN_ERROR_IF(zc->blockState.nextCBlock == NULL, memory_allocation, "couldn't allocate nextCBlock");
zc->entropyWorkspace = (U32*)ptr; zc->entropyWorkspace = (U32*) ZSTD_cwksp_reserve_object(ws, HUF_WORKSPACE_SIZE);
RETURN_ERROR_IF(zc->blockState.nextCBlock == NULL, memory_allocation, "couldn't allocate entropyWorkspace");
} } } }
ZSTD_cwksp_clear(ws);
/* init params */ /* init params */
zc->appliedParams = params; zc->appliedParams = params;
zc->blockState.matchState.cParams = params.cParams; zc->blockState.matchState.cParams = params.cParams;
zc->pledgedSrcSizePlusOne = pledgedSrcSize+1; zc->pledgedSrcSizePlusOne = pledgedSrcSize+1;
zc->consumedSrcSize = 0; zc->consumedSrcSize = 0;
zc->producedCSize = 0; zc->producedCSize = 0;
if (pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN) if (pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN)
zc->appliedParams.fParams.contentSizeFlag = 0; zc->appliedParams.fParams.contentSizeFlag = 0;
DEBUGLOG(4, "pledged content size : %u ; flag : %u", DEBUGLOG(4, "pledged content size : %u ; flag : %u",
(unsigned)pledgedSrcSize, zc->appliedParams.fParams.contentSizeFlag); (unsigned)pledgedSrcSize, zc->appliedParams.fParams.contentSizeFlag);
zc->blockSize = blockSize; zc->blockSize = blockSize;
XXH64_reset(&zc->xxhState, 0); XXH64_reset(&zc->xxhState, 0);
zc->stage = ZSTDcs_init; zc->stage = ZSTDcs_init;
zc->dictID = 0; zc->dictID = 0;
ZSTD_reset_compressedBlockState(zc->blockState.prevCBlock); ZSTD_reset_compressedBlockState(zc->blockState.prevCBlock);
ptr = ZSTD_reset_matchState(&zc->blockState.matchState,
zc->entropyWorkspace + HUF_WORKSPACE_SIZE_U32,
&params.cParams,
crp, ZSTD_resetTarget_CCtx);
/* ldm hash table */
/* initialize bucketOffsets table later for pointer alignment */
if (params.ldmParams.enableLdm) {
size_t const ldmHSize = ((size_t)1) << params.ldmParams.hashLog;
memset(ptr, 0, ldmHSize * sizeof(ldmEntry_t));
assert(((size_t)ptr & 3) == 0); /* ensure ptr is properly aligned */
zc->ldmState.hashTable = (ldmEntry_t*)ptr;
ptr = zc->ldmState.hashTable + ldmHSize;
zc->ldmSequences = (rawSeq*)ptr;
ptr = zc->ldmSequences + maxNbLdmSeq;
zc->maxNbLdmSequences = maxNbLdmSeq;
memset(&zc->ldmState.window, 0, sizeof(zc->ldmState.window));
}
assert(((size_t)ptr & 3) == 0); /* ensure ptr is properly aligned */
/* sequences storage */
zc->seqStore.maxNbSeq = maxNbSeq;
zc->seqStore.sequencesStart = (seqDef*)ptr;
ptr = zc->seqStore.sequencesStart + maxNbSeq;
zc->seqStore.llCode = (BYTE*) ptr;
zc->seqStore.mlCode = zc->seqStore.llCode + maxNbSeq;
zc->seqStore.ofCode = zc->seqStore.mlCode + maxNbSeq;
zc->seqStore.litStart = zc->seqStore.ofCode + maxNbSeq;
/* ZSTD_wildcopy() is used to copy into the literals buffer, /* ZSTD_wildcopy() is used to copy into the literals buffer,
* so we have to oversize the buffer by WILDCOPY_OVERLENGTH bytes. * so we have to oversize the buffer by WILDCOPY_OVERLENGTH bytes.
*/ */
zc->seqStore.litStart = ZSTD_cwksp_reserve_buffer(ws, blockSize + WILDCOPY_OVERLENGTH);
zc->seqStore.maxNbLit = blockSize; zc->seqStore.maxNbLit = blockSize;
ptr = zc->seqStore.litStart + blockSize + WILDCOPY_OVERLENGTH;
/* buffers */
zc->inBuffSize = buffInSize;
zc->inBuff = (char*)ZSTD_cwksp_reserve_buffer(ws, buffInSize);
zc->outBuffSize = buffOutSize;
zc->outBuff = (char*)ZSTD_cwksp_reserve_buffer(ws, buffOutSize);
/* ldm bucketOffsets table */ /* ldm bucketOffsets table */
if (params.ldmParams.enableLdm) { if (params.ldmParams.enableLdm) {
/* TODO: avoid memset? */
size_t const ldmBucketSize = size_t const ldmBucketSize =
((size_t)1) << (params.ldmParams.hashLog - ((size_t)1) << (params.ldmParams.hashLog -
params.ldmParams.bucketSizeLog); params.ldmParams.bucketSizeLog);
memset(ptr, 0, ldmBucketSize); zc->ldmState.bucketOffsets = ZSTD_cwksp_reserve_buffer(ws, ldmBucketSize);
zc->ldmState.bucketOffsets = (BYTE*)ptr; memset(zc->ldmState.bucketOffsets, 0, ldmBucketSize);
ptr = zc->ldmState.bucketOffsets + ldmBucketSize;
ZSTD_window_clear(&zc->ldmState.window);
} }
/* sequences storage */
ZSTD_referenceExternalSequences(zc, NULL, 0); ZSTD_referenceExternalSequences(zc, NULL, 0);
zc->seqStore.maxNbSeq = maxNbSeq;
zc->seqStore.llCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(BYTE));
zc->seqStore.mlCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(BYTE));
zc->seqStore.ofCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(BYTE));
zc->seqStore.sequencesStart = (seqDef*)ZSTD_cwksp_reserve_aligned(ws, maxNbSeq * sizeof(seqDef));
FORWARD_IF_ERROR(ZSTD_reset_matchState(
&zc->blockState.matchState,
ws,
&params.cParams,
crp,
needsIndexReset,
ZSTD_resetTarget_CCtx));
/* buffers */ /* ldm hash table */
zc->inBuffSize = buffInSize; if (params.ldmParams.enableLdm) {
zc->inBuff = (char*)ptr; /* TODO: avoid memset? */
zc->outBuffSize = buffOutSize; size_t const ldmHSize = ((size_t)1) << params.ldmParams.hashLog;
zc->outBuff = zc->inBuff + buffInSize; zc->ldmState.hashTable = (ldmEntry_t*)ZSTD_cwksp_reserve_aligned(ws, ldmHSize * sizeof(ldmEntry_t));
memset(zc->ldmState.hashTable, 0, ldmHSize * sizeof(ldmEntry_t));
zc->ldmSequences = (rawSeq*)ZSTD_cwksp_reserve_aligned(ws, maxNbLdmSeq * sizeof(rawSeq));
zc->maxNbLdmSequences = maxNbLdmSeq;
memset(&zc->ldmState.window, 0, sizeof(zc->ldmState.window));
ZSTD_window_clear(&zc->ldmState.window);
}
DEBUGLOG(3, "wksp: finished allocating, %zd bytes remain available", ZSTD_cwksp_available_space(ws));
return 0; return 0;
} }
} }
/* ZSTD_invalidateRepCodes() : /* ZSTD_invalidateRepCodes() :
* ensures next compression will not use repcodes from previous block. * ensures next compression will not use repcodes from previous block.
* Note : only works with regular variant; * Note : only works with regular variant;
32 KB, /* ZSTD_lazy2 */ 32 KB, /* ZSTD_lazy2 */
32 KB, /* ZSTD_btlazy2 */ 32 KB, /* ZSTD_btlazy2 */
32 KB, /* ZSTD_btopt */ 32 KB, /* ZSTD_btopt */
8 KB, /* ZSTD_btultra */ 8 KB, /* ZSTD_btultra */
8 KB /* ZSTD_btultra2 */ 8 KB /* ZSTD_btultra2 */
}; };
static int ZSTD_shouldAttachDict(const ZSTD_CDict* cdict, static int ZSTD_shouldAttachDict(const ZSTD_CDict* cdict,
ZSTD_CCtx_params params, const ZSTD_CCtx_params* params,
U64 pledgedSrcSize) U64 pledgedSrcSize)
{ {
size_t cutoff = attachDictSizeCutoffs[cdict->matchState.cParams.strategy]; size_t cutoff = attachDictSizeCutoffs[cdict->matchState.cParams.strategy];
return ( pledgedSrcSize <= cutoff return ( pledgedSrcSize <= cutoff
|| pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN || pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN
|| params.attachDictPref == ZSTD_dictForceAttach ) || params->attachDictPref == ZSTD_dictForceAttach )
&& params.attachDictPref != ZSTD_dictForceCopy && params->attachDictPref != ZSTD_dictForceCopy
&& !params.forceWindow; /* dictMatchState isn't correctly && !params->forceWindow; /* dictMatchState isn't correctly
* handled in _enforceMaxDist */ * handled in _enforceMaxDist */
} }
static size_t static size_t
ZSTD_resetCCtx_byAttachingCDict(ZSTD_CCtx* cctx, ZSTD_resetCCtx_byAttachingCDict(ZSTD_CCtx* cctx,
const ZSTD_CDict* cdict, const ZSTD_CDict* cdict,
ZSTD_CCtx_params params, ZSTD_CCtx_params params,
U64 pledgedSrcSize, U64 pledgedSrcSize,
ZSTD_buffered_policy_e zbuff) ZSTD_buffered_policy_e zbuff)
{ {
{ const ZSTD_compressionParameters* const cdict_cParams = &cdict->matchState.cParams; { const ZSTD_compressionParameters* const cdict_cParams = &cdict->matchState.cParams;
unsigned const windowLog = params.cParams.windowLog; unsigned const windowLog = params.cParams.windowLog;
assert(windowLog != 0); assert(windowLog != 0);
/* Resize working context table params for input only, since the dict /* Resize working context table params for input only, since the dict
* has its own tables. */ * has its own tables. */
params.cParams = ZSTD_adjustCParams_internal(*cdict_cParams, pledgedSrcSize, 0); params.cParams = ZSTD_adjustCParams_internal(*cdict_cParams, pledgedSrcSize, 0);
params.cParams.windowLog = windowLog; params.cParams.windowLog = windowLog;
ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize, FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize,
ZSTDcrp_continue, zbuff); ZSTDcrp_makeClean, zbuff));
assert(cctx->appliedParams.cParams.strategy == cdict_cParams->strategy); assert(cctx->appliedParams.cParams.strategy == cdict_cParams->strategy);
} }
{ const U32 cdictEnd = (U32)( cdict->matchState.window.nextSrc { const U32 cdictEnd = (U32)( cdict->matchState.window.nextSrc
- cdict->matchState.window.base); - cdict->matchState.window.base);
const U32 cdictLen = cdictEnd - cdict->matchState.window.dictLimit; const U32 cdictLen = cdictEnd - cdict->matchState.window.dictLimit;
if (cdictLen == 0) { if (cdictLen == 0) {
/* don't even attach dictionaries with no contents */ /* don't even attach dictionaries with no contents */
DEBUGLOG(4, "copying dictionary into context"); DEBUGLOG(4, "copying dictionary into context");
{ unsigned const windowLog = params.cParams.windowLog; { unsigned const windowLog = params.cParams.windowLog;
assert(windowLog != 0); assert(windowLog != 0);
/* Copy only compression parameters related to tables. */ /* Copy only compression parameters related to tables. */
params.cParams = *cdict_cParams; params.cParams = *cdict_cParams;
params.cParams.windowLog = windowLog; params.cParams.windowLog = windowLog;
ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize, FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize,
ZSTDcrp_noMemset, zbuff); ZSTDcrp_leaveDirty, zbuff));
assert(cctx->appliedParams.cParams.strategy == cdict_cParams->strategy); assert(cctx->appliedParams.cParams.strategy == cdict_cParams->strategy);
assert(cctx->appliedParams.cParams.hashLog == cdict_cParams->hashLog); assert(cctx->appliedParams.cParams.hashLog == cdict_cParams->hashLog);
assert(cctx->appliedParams.cParams.chainLog == cdict_cParams->chainLog); assert(cctx->appliedParams.cParams.chainLog == cdict_cParams->chainLog);
} }
ZSTD_cwksp_mark_tables_dirty(&cctx->workspace);
/* copy tables */ /* copy tables */
{ size_t const chainSize = (cdict_cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cdict_cParams->chainLog); { size_t const chainSize = (cdict_cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cdict_cParams->chainLog);
size_t const hSize = (size_t)1 << cdict_cParams->hashLog; size_t const hSize = (size_t)1 << cdict_cParams->hashLog;
size_t const tableSpace = (chainSize + hSize) * sizeof(U32);
assert((U32*)cctx->blockState.matchState.chainTable == (U32*)cctx->blockState.matchState.hashTable + hSize); /* chainTable must follow hashTable */ memcpy(cctx->blockState.matchState.hashTable,
assert((U32*)cctx->blockState.matchState.hashTable3 == (U32*)cctx->blockState.matchState.chainTable + chainSize); cdict->matchState.hashTable,
assert((U32*)cdict->matchState.chainTable == (U32*)cdict->matchState.hashTable + hSize); /* chainTable must follow hashTable */ hSize * sizeof(U32));
assert((U32*)cdict->matchState.hashTable3 == (U32*)cdict->matchState.chainTable + chainSize); memcpy(cctx->blockState.matchState.chainTable,
memcpy(cctx->blockState.matchState.hashTable, cdict->matchState.hashTable, tableSpace); /* presumes all tables follow each other */ cdict->matchState.chainTable,
chainSize * sizeof(U32));
} }
/* Zero the hashTable3, since the cdict never fills it */ /* Zero the hashTable3, since the cdict never fills it */
{ size_t const h3Size = (size_t)1 << cctx->blockState.matchState.hashLog3; { int const h3log = cctx->blockState.matchState.hashLog3;
size_t const h3Size = h3log ? ((size_t)1 << h3log) : 0;
assert(cdict->matchState.hashLog3 == 0); assert(cdict->matchState.hashLog3 == 0);
memset(cctx->blockState.matchState.hashTable3, 0, h3Size * sizeof(U32)); memset(cctx->blockState.matchState.hashTable3, 0, h3Size * sizeof(U32));
} }
ZSTD_cwksp_mark_tables_clean(&cctx->workspace);
/* copy dictionary offsets */ /* copy dictionary offsets */
{ ZSTD_matchState_t const* srcMatchState = &cdict->matchState; { ZSTD_matchState_t const* srcMatchState = &cdict->matchState;
ZSTD_matchState_t* dstMatchState = &cctx->blockState.matchState; ZSTD_matchState_t* dstMatchState = &cctx->blockState.matchState;
dstMatchState->window = srcMatchState->window; dstMatchState->window = srcMatchState->window;
dstMatchState->nextToUpdate = srcMatchState->nextToUpdate; dstMatchState->nextToUpdate = srcMatchState->nextToUpdate;
dstMatchState->loadedDictEnd= srcMatchState->loadedDictEnd; dstMatchState->loadedDictEnd= srcMatchState->loadedDictEnd;
} }
cctx->dictID = cdict->dictID; cctx->dictID = cdict->dictID;
/* copy block state */ /* copy block state */
memcpy(cctx->blockState.prevCBlock, &cdict->cBlockState, sizeof(cdict->cBlockState)); memcpy(cctx->blockState.prevCBlock, &cdict->cBlockState, sizeof(cdict->cBlockState));
return 0; return 0;
} }
/* We have a choice between copying the dictionary context into the working /* We have a choice between copying the dictionary context into the working
* context, or referencing the dictionary context from the working context * context, or referencing the dictionary context from the working context
* in-place. We decide here which strategy to use. */ * in-place. We decide here which strategy to use. */
static size_t ZSTD_resetCCtx_usingCDict(ZSTD_CCtx* cctx, static size_t ZSTD_resetCCtx_usingCDict(ZSTD_CCtx* cctx,
const ZSTD_CDict* cdict, const ZSTD_CDict* cdict,
ZSTD_CCtx_params params, const ZSTD_CCtx_params* params,
U64 pledgedSrcSize, U64 pledgedSrcSize,
ZSTD_buffered_policy_e zbuff) ZSTD_buffered_policy_e zbuff)
{ {
DEBUGLOG(4, "ZSTD_resetCCtx_usingCDict (pledgedSrcSize=%u)", DEBUGLOG(4, "ZSTD_resetCCtx_usingCDict (pledgedSrcSize=%u)",
(unsigned)pledgedSrcSize); (unsigned)pledgedSrcSize);
if (ZSTD_shouldAttachDict(cdict, params, pledgedSrcSize)) { if (ZSTD_shouldAttachDict(cdict, params, pledgedSrcSize)) {
return ZSTD_resetCCtx_byAttachingCDict( return ZSTD_resetCCtx_byAttachingCDict(
cctx, cdict, params, pledgedSrcSize, zbuff); cctx, cdict, *params, pledgedSrcSize, zbuff);
} else { } else {
return ZSTD_resetCCtx_byCopyingCDict( return ZSTD_resetCCtx_byCopyingCDict(
cctx, cdict, params, pledgedSrcSize, zbuff); cctx, cdict, *params, pledgedSrcSize, zbuff);
} }
} }
/*! ZSTD_copyCCtx_internal() : /*! ZSTD_copyCCtx_internal() :
* Duplicate an existing context `srcCCtx` into another one `dstCCtx`. * Duplicate an existing context `srcCCtx` into another one `dstCCtx`.
* Only works during stage ZSTDcs_init (i.e. after creation, but before first call to ZSTD_compressContinue()). * Only works during stage ZSTDcs_init (i.e. after creation, but before first call to ZSTD_compressContinue()).
* The "context", in this case, refers to the hash and chain tables, * The "context", in this case, refers to the hash and chain tables,
* entropy tables, and dictionary references. * entropy tables, and dictionary references.
RETURN_ERROR_IF(srcCCtx->stage!=ZSTDcs_init, stage_wrong); RETURN_ERROR_IF(srcCCtx->stage!=ZSTDcs_init, stage_wrong);
memcpy(&dstCCtx->customMem, &srcCCtx->customMem, sizeof(ZSTD_customMem)); memcpy(&dstCCtx->customMem, &srcCCtx->customMem, sizeof(ZSTD_customMem));
{ ZSTD_CCtx_params params = dstCCtx->requestedParams; { ZSTD_CCtx_params params = dstCCtx->requestedParams;
/* Copy only compression parameters related to tables. */ /* Copy only compression parameters related to tables. */
params.cParams = srcCCtx->appliedParams.cParams; params.cParams = srcCCtx->appliedParams.cParams;
params.fParams = fParams; params.fParams = fParams;
ZSTD_resetCCtx_internal(dstCCtx, params, pledgedSrcSize, ZSTD_resetCCtx_internal(dstCCtx, params, pledgedSrcSize,
ZSTDcrp_noMemset, zbuff); ZSTDcrp_leaveDirty, zbuff);
assert(dstCCtx->appliedParams.cParams.windowLog == srcCCtx->appliedParams.cParams.windowLog); assert(dstCCtx->appliedParams.cParams.windowLog == srcCCtx->appliedParams.cParams.windowLog);
assert(dstCCtx->appliedParams.cParams.strategy == srcCCtx->appliedParams.cParams.strategy); assert(dstCCtx->appliedParams.cParams.strategy == srcCCtx->appliedParams.cParams.strategy);
assert(dstCCtx->appliedParams.cParams.hashLog == srcCCtx->appliedParams.cParams.hashLog); assert(dstCCtx->appliedParams.cParams.hashLog == srcCCtx->appliedParams.cParams.hashLog);
assert(dstCCtx->appliedParams.cParams.chainLog == srcCCtx->appliedParams.cParams.chainLog); assert(dstCCtx->appliedParams.cParams.chainLog == srcCCtx->appliedParams.cParams.chainLog);
assert(dstCCtx->blockState.matchState.hashLog3 == srcCCtx->blockState.matchState.hashLog3); assert(dstCCtx->blockState.matchState.hashLog3 == srcCCtx->blockState.matchState.hashLog3);
} }
ZSTD_cwksp_mark_tables_dirty(&dstCCtx->workspace);
/* copy tables */ /* copy tables */
{ size_t const chainSize = (srcCCtx->appliedParams.cParams.strategy == ZSTD_fast) ? 0 : ((size_t)1 << srcCCtx->appliedParams.cParams.chainLog); { size_t const chainSize = (srcCCtx->appliedParams.cParams.strategy == ZSTD_fast) ? 0 : ((size_t)1 << srcCCtx->appliedParams.cParams.chainLog);
size_t const hSize = (size_t)1 << srcCCtx->appliedParams.cParams.hashLog; size_t const hSize = (size_t)1 << srcCCtx->appliedParams.cParams.hashLog;
size_t const h3Size = (size_t)1 << srcCCtx->blockState.matchState.hashLog3; int const h3log = srcCCtx->blockState.matchState.hashLog3;
size_t const tableSpace = (chainSize + hSize + h3Size) * sizeof(U32); size_t const h3Size = h3log ? ((size_t)1 << h3log) : 0;
assert((U32*)dstCCtx->blockState.matchState.chainTable == (U32*)dstCCtx->blockState.matchState.hashTable + hSize); /* chainTable must follow hashTable */
assert((U32*)dstCCtx->blockState.matchState.hashTable3 == (U32*)dstCCtx->blockState.matchState.chainTable + chainSize); memcpy(dstCCtx->blockState.matchState.hashTable,
memcpy(dstCCtx->blockState.matchState.hashTable, srcCCtx->blockState.matchState.hashTable, tableSpace); /* presumes all tables follow each other */ srcCCtx->blockState.matchState.hashTable,
hSize * sizeof(U32));
memcpy(dstCCtx->blockState.matchState.chainTable,
srcCCtx->blockState.matchState.chainTable,
chainSize * sizeof(U32));
memcpy(dstCCtx->blockState.matchState.hashTable3,
srcCCtx->blockState.matchState.hashTable3,
h3Size * sizeof(U32));
} }
ZSTD_cwksp_mark_tables_clean(&dstCCtx->workspace);
/* copy dictionary offsets */ /* copy dictionary offsets */
{ {
const ZSTD_matchState_t* srcMatchState = &srcCCtx->blockState.matchState; const ZSTD_matchState_t* srcMatchState = &srcCCtx->blockState.matchState;
ZSTD_matchState_t* dstMatchState = &dstCCtx->blockState.matchState; ZSTD_matchState_t* dstMatchState = &dstCCtx->blockState.matchState;
dstMatchState->window = srcMatchState->window; dstMatchState->window = srcMatchState->window;
dstMatchState->nextToUpdate = srcMatchState->nextToUpdate; dstMatchState->nextToUpdate = srcMatchState->nextToUpdate;
dstMatchState->loadedDictEnd= srcMatchState->loadedDictEnd; dstMatchState->loadedDictEnd= srcMatchState->loadedDictEnd;
} }
FORCE_INLINE_TEMPLATE void FORCE_INLINE_TEMPLATE void
ZSTD_reduceTable_internal (U32* const table, U32 const size, U32 const reducerValue, int const preserveMark) ZSTD_reduceTable_internal (U32* const table, U32 const size, U32 const reducerValue, int const preserveMark)
{ {
int const nbRows = (int)size / ZSTD_ROWSIZE; int const nbRows = (int)size / ZSTD_ROWSIZE;
int cellNb = 0; int cellNb = 0;
int rowNb; int rowNb;
assert((size & (ZSTD_ROWSIZE-1)) == 0); /* multiple of ZSTD_ROWSIZE */ assert((size & (ZSTD_ROWSIZE-1)) == 0); /* multiple of ZSTD_ROWSIZE */
assert(size < (1U<<31)); /* can be casted to int */ assert(size < (1U<<31)); /* can be casted to int */
#if defined (MEMORY_SANITIZER) && !defined (ZSTD_MSAN_DONT_POISON_WORKSPACE)
/* To validate that the table re-use logic is sound, and that we don't
* access table space that we haven't cleaned, we re-"poison" the table
* space every time we mark it dirty.
*
* This function however is intended to operate on those dirty tables and
* re-clean them. So when this function is used correctly, we can unpoison
* the memory it operated on. This introduces a blind spot though, since
* if we now try to operate on __actually__ poisoned memory, we will not
* detect that. */
__msan_unpoison(table, size * sizeof(U32));
#endif
for (rowNb=0 ; rowNb < nbRows ; rowNb++) { for (rowNb=0 ; rowNb < nbRows ; rowNb++) {
int column; int column;
for (column=0; column<ZSTD_ROWSIZE; column++) { for (column=0; column<ZSTD_ROWSIZE; column++) {
if (preserveMark) { if (preserveMark) {
U32 const adder = (table[cellNb] == ZSTD_DUBT_UNSORTED_MARK) ? reducerValue : 0; U32 const adder = (table[cellNb] == ZSTD_DUBT_UNSORTED_MARK) ? reducerValue : 0;
table[cellNb] += adder; table[cellNb] += adder;
} }
if (table[cellNb] < reducerValue) table[cellNb] = 0; if (table[cellNb] < reducerValue) table[cellNb] = 0;
/* ZSTD_compressSequences_internal(): /* ZSTD_compressSequences_internal():
* actually compresses both literals and sequences */ * actually compresses both literals and sequences */
MEM_STATIC size_t MEM_STATIC size_t
ZSTD_compressSequences_internal(seqStore_t* seqStorePtr, ZSTD_compressSequences_internal(seqStore_t* seqStorePtr,
const ZSTD_entropyCTables_t* prevEntropy, const ZSTD_entropyCTables_t* prevEntropy,
ZSTD_entropyCTables_t* nextEntropy, ZSTD_entropyCTables_t* nextEntropy,
const ZSTD_CCtx_params* cctxParams, const ZSTD_CCtx_params* cctxParams,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
void* workspace, size_t wkspSize, void* entropyWorkspace, size_t entropyWkspSize,
const int bmi2) const int bmi2)
{ {
const int longOffsets = cctxParams->cParams.windowLog > STREAM_ACCUMULATOR_MIN; const int longOffsets = cctxParams->cParams.windowLog > STREAM_ACCUMULATOR_MIN;
ZSTD_strategy const strategy = cctxParams->cParams.strategy; ZSTD_strategy const strategy = cctxParams->cParams.strategy;
unsigned count[MaxSeq+1]; unsigned count[MaxSeq+1];
FSE_CTable* CTable_LitLength = nextEntropy->fse.litlengthCTable; FSE_CTable* CTable_LitLength = nextEntropy->fse.litlengthCTable;
FSE_CTable* CTable_OffsetBits = nextEntropy->fse.offcodeCTable; FSE_CTable* CTable_OffsetBits = nextEntropy->fse.offcodeCTable;
FSE_CTable* CTable_MatchLength = nextEntropy->fse.matchlengthCTable; FSE_CTable* CTable_MatchLength = nextEntropy->fse.matchlengthCTable;
{ const BYTE* const literals = seqStorePtr->litStart; { const BYTE* const literals = seqStorePtr->litStart;
size_t const litSize = (size_t)(seqStorePtr->lit - literals); size_t const litSize = (size_t)(seqStorePtr->lit - literals);
size_t const cSize = ZSTD_compressLiterals( size_t const cSize = ZSTD_compressLiterals(
&prevEntropy->huf, &nextEntropy->huf, &prevEntropy->huf, &nextEntropy->huf,
cctxParams->cParams.strategy, cctxParams->cParams.strategy,
ZSTD_disableLiteralsCompression(cctxParams), ZSTD_disableLiteralsCompression(cctxParams),
op, dstCapacity, op, dstCapacity,
literals, litSize, literals, litSize,
workspace, wkspSize, entropyWorkspace, entropyWkspSize,
bmi2); bmi2);
FORWARD_IF_ERROR(cSize); FORWARD_IF_ERROR(cSize);
assert(cSize <= dstCapacity); assert(cSize <= dstCapacity);
op += cSize; op += cSize;
} }
/* Sequences Header */ /* Sequences Header */
RETURN_ERROR_IF((oend-op) < 3 /*max nbSeq Size*/ + 1 /*seqHead*/, RETURN_ERROR_IF((oend-op) < 3 /*max nbSeq Size*/ + 1 /*seqHead*/,
dstSize_tooSmall); dstSize_tooSmall);
if (nbSeq < 0x7F) if (nbSeq < 128) {
*op++ = (BYTE)nbSeq; *op++ = (BYTE)nbSeq;
else if (nbSeq < LONGNBSEQ) } else if (nbSeq < LONGNBSEQ) {
op[0] = (BYTE)((nbSeq>>8) + 0x80), op[1] = (BYTE)nbSeq, op+=2; op[0] = (BYTE)((nbSeq>>8) + 0x80);
else op[1] = (BYTE)nbSeq;
op[0]=0xFF, MEM_writeLE16(op+1, (U16)(nbSeq - LONGNBSEQ)), op+=3; op+=2;
} else {
op[0]=0xFF;
MEM_writeLE16(op+1, (U16)(nbSeq - LONGNBSEQ));
op+=3;
}
assert(op <= oend); assert(op <= oend);
if (nbSeq==0) { if (nbSeq==0) {
/* Copy the old tables over as if we repeated them */ /* Copy the old tables over as if we repeated them */
memcpy(&nextEntropy->fse, &prevEntropy->fse, sizeof(prevEntropy->fse)); memcpy(&nextEntropy->fse, &prevEntropy->fse, sizeof(prevEntropy->fse));
return (size_t)(op - ostart); return (size_t)(op - ostart);
} }
/* seqHead : flags for FSE encoding type */ /* seqHead : flags for FSE encoding type */
seqHead = op++; seqHead = op++;
assert(op <= oend); assert(op <= oend);
/* convert length/distances into codes */ /* convert length/distances into codes */
ZSTD_seqToCodes(seqStorePtr); ZSTD_seqToCodes(seqStorePtr);
/* build CTable for Literal Lengths */ /* build CTable for Literal Lengths */
{ unsigned max = MaxLL; { unsigned max = MaxLL;
size_t const mostFrequent = HIST_countFast_wksp(count, &max, llCodeTable, nbSeq, workspace, wkspSize); /* can't fail */ size_t const mostFrequent = HIST_countFast_wksp(count, &max, llCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
DEBUGLOG(5, "Building LL table"); DEBUGLOG(5, "Building LL table");
nextEntropy->fse.litlength_repeatMode = prevEntropy->fse.litlength_repeatMode; nextEntropy->fse.litlength_repeatMode = prevEntropy->fse.litlength_repeatMode;
LLtype = ZSTD_selectEncodingType(&nextEntropy->fse.litlength_repeatMode, LLtype = ZSTD_selectEncodingType(&nextEntropy->fse.litlength_repeatMode,
count, max, mostFrequent, nbSeq, count, max, mostFrequent, nbSeq,
LLFSELog, prevEntropy->fse.litlengthCTable, LLFSELog, prevEntropy->fse.litlengthCTable,
LL_defaultNorm, LL_defaultNormLog, LL_defaultNorm, LL_defaultNormLog,
ZSTD_defaultAllowed, strategy); ZSTD_defaultAllowed, strategy);
assert(set_basic < set_compressed && set_rle < set_compressed); assert(set_basic < set_compressed && set_rle < set_compressed);
assert(!(LLtype < set_compressed && nextEntropy->fse.litlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */ assert(!(LLtype < set_compressed && nextEntropy->fse.litlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
{ size_t const countSize = ZSTD_buildCTable(op, (size_t)(oend - op), CTable_LitLength, LLFSELog, (symbolEncodingType_e)LLtype, { size_t const countSize = ZSTD_buildCTable(
count, max, llCodeTable, nbSeq, LL_defaultNorm, LL_defaultNormLog, MaxLL, op, (size_t)(oend - op),
prevEntropy->fse.litlengthCTable, sizeof(prevEntropy->fse.litlengthCTable), CTable_LitLength, LLFSELog, (symbolEncodingType_e)LLtype,
workspace, wkspSize); count, max, llCodeTable, nbSeq,
LL_defaultNorm, LL_defaultNormLog, MaxLL,
prevEntropy->fse.litlengthCTable,
sizeof(prevEntropy->fse.litlengthCTable),
entropyWorkspace, entropyWkspSize);
FORWARD_IF_ERROR(countSize); FORWARD_IF_ERROR(countSize);
if (LLtype == set_compressed) if (LLtype == set_compressed)
lastNCount = op; lastNCount = op;
op += countSize; op += countSize;
assert(op <= oend); assert(op <= oend);
} } } }
/* build CTable for Offsets */ /* build CTable for Offsets */
{ unsigned max = MaxOff; { unsigned max = MaxOff;
size_t const mostFrequent = HIST_countFast_wksp(count, &max, ofCodeTable, nbSeq, workspace, wkspSize); /* can't fail */ size_t const mostFrequent = HIST_countFast_wksp(
count, &max, ofCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
/* We can only use the basic table if max <= DefaultMaxOff, otherwise the offsets are too large */ /* We can only use the basic table if max <= DefaultMaxOff, otherwise the offsets are too large */
ZSTD_defaultPolicy_e const defaultPolicy = (max <= DefaultMaxOff) ? ZSTD_defaultAllowed : ZSTD_defaultDisallowed; ZSTD_defaultPolicy_e const defaultPolicy = (max <= DefaultMaxOff) ? ZSTD_defaultAllowed : ZSTD_defaultDisallowed;
DEBUGLOG(5, "Building OF table"); DEBUGLOG(5, "Building OF table");
nextEntropy->fse.offcode_repeatMode = prevEntropy->fse.offcode_repeatMode; nextEntropy->fse.offcode_repeatMode = prevEntropy->fse.offcode_repeatMode;
Offtype = ZSTD_selectEncodingType(&nextEntropy->fse.offcode_repeatMode, Offtype = ZSTD_selectEncodingType(&nextEntropy->fse.offcode_repeatMode,
count, max, mostFrequent, nbSeq, count, max, mostFrequent, nbSeq,
OffFSELog, prevEntropy->fse.offcodeCTable, OffFSELog, prevEntropy->fse.offcodeCTable,
OF_defaultNorm, OF_defaultNormLog, OF_defaultNorm, OF_defaultNormLog,
defaultPolicy, strategy); defaultPolicy, strategy);
assert(!(Offtype < set_compressed && nextEntropy->fse.offcode_repeatMode != FSE_repeat_none)); /* We don't copy tables */ assert(!(Offtype < set_compressed && nextEntropy->fse.offcode_repeatMode != FSE_repeat_none)); /* We don't copy tables */
{ size_t const countSize = ZSTD_buildCTable(op, (size_t)(oend - op), CTable_OffsetBits, OffFSELog, (symbolEncodingType_e)Offtype, { size_t const countSize = ZSTD_buildCTable(
count, max, ofCodeTable, nbSeq, OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff, op, (size_t)(oend - op),
prevEntropy->fse.offcodeCTable, sizeof(prevEntropy->fse.offcodeCTable), CTable_OffsetBits, OffFSELog, (symbolEncodingType_e)Offtype,
workspace, wkspSize); count, max, ofCodeTable, nbSeq,
OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
prevEntropy->fse.offcodeCTable,
sizeof(prevEntropy->fse.offcodeCTable),
entropyWorkspace, entropyWkspSize);
FORWARD_IF_ERROR(countSize); FORWARD_IF_ERROR(countSize);
if (Offtype == set_compressed) if (Offtype == set_compressed)
lastNCount = op; lastNCount = op;
op += countSize; op += countSize;
assert(op <= oend); assert(op <= oend);
} } } }
/* build CTable for MatchLengths */ /* build CTable for MatchLengths */
{ unsigned max = MaxML; { unsigned max = MaxML;
size_t const mostFrequent = HIST_countFast_wksp(count, &max, mlCodeTable, nbSeq, workspace, wkspSize); /* can't fail */ size_t const mostFrequent = HIST_countFast_wksp(
count, &max, mlCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
DEBUGLOG(5, "Building ML table (remaining space : %i)", (int)(oend-op)); DEBUGLOG(5, "Building ML table (remaining space : %i)", (int)(oend-op));
nextEntropy->fse.matchlength_repeatMode = prevEntropy->fse.matchlength_repeatMode; nextEntropy->fse.matchlength_repeatMode = prevEntropy->fse.matchlength_repeatMode;
MLtype = ZSTD_selectEncodingType(&nextEntropy->fse.matchlength_repeatMode, MLtype = ZSTD_selectEncodingType(&nextEntropy->fse.matchlength_repeatMode,
count, max, mostFrequent, nbSeq, count, max, mostFrequent, nbSeq,
MLFSELog, prevEntropy->fse.matchlengthCTable, MLFSELog, prevEntropy->fse.matchlengthCTable,
ML_defaultNorm, ML_defaultNormLog, ML_defaultNorm, ML_defaultNormLog,
ZSTD_defaultAllowed, strategy); ZSTD_defaultAllowed, strategy);
assert(!(MLtype < set_compressed && nextEntropy->fse.matchlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */ assert(!(MLtype < set_compressed && nextEntropy->fse.matchlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
{ size_t const countSize = ZSTD_buildCTable(op, (size_t)(oend - op), CTable_MatchLength, MLFSELog, (symbolEncodingType_e)MLtype, { size_t const countSize = ZSTD_buildCTable(
count, max, mlCodeTable, nbSeq, ML_defaultNorm, ML_defaultNormLog, MaxML, op, (size_t)(oend - op),
prevEntropy->fse.matchlengthCTable, sizeof(prevEntropy->fse.matchlengthCTable), CTable_MatchLength, MLFSELog, (symbolEncodingType_e)MLtype,
workspace, wkspSize); count, max, mlCodeTable, nbSeq,
ML_defaultNorm, ML_defaultNormLog, MaxML,
prevEntropy->fse.matchlengthCTable,
sizeof(prevEntropy->fse.matchlengthCTable),
entropyWorkspace, entropyWkspSize);
FORWARD_IF_ERROR(countSize); FORWARD_IF_ERROR(countSize);
if (MLtype == set_compressed) if (MLtype == set_compressed)
lastNCount = op; lastNCount = op;
op += countSize; op += countSize;
assert(op <= oend); assert(op <= oend);
} } } }
*seqHead = (BYTE)((LLtype<<6) + (Offtype<<4) + (MLtype<<2)); *seqHead = (BYTE)((LLtype<<6) + (Offtype<<4) + (MLtype<<2));
MEM_STATIC size_t MEM_STATIC size_t
ZSTD_compressSequences(seqStore_t* seqStorePtr, ZSTD_compressSequences(seqStore_t* seqStorePtr,
const ZSTD_entropyCTables_t* prevEntropy, const ZSTD_entropyCTables_t* prevEntropy,
ZSTD_entropyCTables_t* nextEntropy, ZSTD_entropyCTables_t* nextEntropy,
const ZSTD_CCtx_params* cctxParams, const ZSTD_CCtx_params* cctxParams,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
size_t srcSize, size_t srcSize,
void* workspace, size_t wkspSize, void* entropyWorkspace, size_t entropyWkspSize,
int bmi2) int bmi2)
{ {
size_t const cSize = ZSTD_compressSequences_internal( size_t const cSize = ZSTD_compressSequences_internal(
seqStorePtr, prevEntropy, nextEntropy, cctxParams, seqStorePtr, prevEntropy, nextEntropy, cctxParams,
dst, dstCapacity, dst, dstCapacity,
workspace, wkspSize, bmi2); entropyWorkspace, entropyWkspSize, bmi2);
if (cSize == 0) return 0; if (cSize == 0) return 0;
/* When srcSize <= dstCapacity, there is enough space to write a raw uncompressed block. /* When srcSize <= dstCapacity, there is enough space to write a raw uncompressed block.
* Since we ran out of space, block must be not compressible, so fall back to raw uncompressed block. * Since we ran out of space, block must be not compressible, so fall back to raw uncompressed block.
*/ */
if ((cSize == ERROR(dstSize_tooSmall)) & (srcSize <= dstCapacity)) if ((cSize == ERROR(dstSize_tooSmall)) & (srcSize <= dstCapacity))
return 0; /* block not compressed */ return 0; /* block not compressed */
FORWARD_IF_ERROR(cSize); FORWARD_IF_ERROR(cSize);
lastLLSize = blockCompressor(ms, &zc->seqStore, zc->blockState.nextCBlock->rep, src, srcSize); lastLLSize = blockCompressor(ms, &zc->seqStore, zc->blockState.nextCBlock->rep, src, srcSize);
} }
{ const BYTE* const lastLiterals = (const BYTE*)src + srcSize - lastLLSize; { const BYTE* const lastLiterals = (const BYTE*)src + srcSize - lastLLSize;
ZSTD_storeLastLiterals(&zc->seqStore, lastLiterals, lastLLSize); ZSTD_storeLastLiterals(&zc->seqStore, lastLiterals, lastLLSize);
} } } }
return ZSTDbss_compress; return ZSTDbss_compress;
} }
static void ZSTD_copyBlockSequences(ZSTD_CCtx* zc)
{
const seqStore_t* seqStore = ZSTD_getSeqStore(zc);
const seqDef* seqs = seqStore->sequencesStart;
size_t seqsSize = seqStore->sequences - seqs;
ZSTD_Sequence* outSeqs = &zc->seqCollector.seqStart[zc->seqCollector.seqIndex];
size_t i; size_t position; int repIdx;
assert(zc->seqCollector.seqIndex + 1 < zc->seqCollector.maxSequences);
for (i = 0, position = 0; i < seqsSize; ++i) {
outSeqs[i].offset = seqs[i].offset;
outSeqs[i].litLength = seqs[i].litLength;
outSeqs[i].matchLength = seqs[i].matchLength + MINMATCH;
if (i == seqStore->longLengthPos) {
if (seqStore->longLengthID == 1) {
outSeqs[i].litLength += 0x10000;
} else if (seqStore->longLengthID == 2) {
outSeqs[i].matchLength += 0x10000;
}
}
if (outSeqs[i].offset <= ZSTD_REP_NUM) {
outSeqs[i].rep = outSeqs[i].offset;
repIdx = (unsigned int)i - outSeqs[i].offset;
if (outSeqs[i].litLength == 0) {
if (outSeqs[i].offset < 3) {
--repIdx;
} else {
repIdx = (unsigned int)i - 1;
}
++outSeqs[i].rep;
}
assert(repIdx >= -3);
outSeqs[i].offset = repIdx >= 0 ? outSeqs[repIdx].offset : repStartValue[-repIdx - 1];
if (outSeqs[i].rep == 4) {
--outSeqs[i].offset;
}
} else {
outSeqs[i].offset -= ZSTD_REP_NUM;
}
position += outSeqs[i].litLength;
outSeqs[i].matchPos = (unsigned int)position;
position += outSeqs[i].matchLength;
}
zc->seqCollector.seqIndex += seqsSize;
}
size_t ZSTD_getSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
size_t outSeqsSize, const void* src, size_t srcSize)
{
const size_t dstCapacity = ZSTD_compressBound(srcSize);
void* dst = ZSTD_malloc(dstCapacity, ZSTD_defaultCMem);
SeqCollector seqCollector;
RETURN_ERROR_IF(dst == NULL, memory_allocation);
seqCollector.collectSequences = 1;
seqCollector.seqStart = outSeqs;
seqCollector.seqIndex = 0;
seqCollector.maxSequences = outSeqsSize;
zc->seqCollector = seqCollector;
ZSTD_compress2(zc, dst, dstCapacity, src, srcSize);
ZSTD_free(dst, ZSTD_defaultCMem);
return zc->seqCollector.seqIndex;
}
/* Returns true if the given block is a RLE block */
static int ZSTD_isRLE(const BYTE *ip, size_t length) {
size_t i;
if (length < 2) return 1;
for (i = 1; i < length; ++i) {
if (ip[0] != ip[i]) return 0;
}
return 1;
}
static size_t ZSTD_compressBlock_internal(ZSTD_CCtx* zc, static size_t ZSTD_compressBlock_internal(ZSTD_CCtx* zc,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize) const void* src, size_t srcSize, U32 frame)
{ {
/* This the upper bound for the length of an rle block.
* This isn't the actual upper bound. Finding the real threshold
* needs further investigation.
*/
const U32 rleMaxLength = 25;
size_t cSize; size_t cSize;
const BYTE* ip = (const BYTE*)src;
BYTE* op = (BYTE*)dst;
DEBUGLOG(5, "ZSTD_compressBlock_internal (dstCapacity=%u, dictLimit=%u, nextToUpdate=%u)", DEBUGLOG(5, "ZSTD_compressBlock_internal (dstCapacity=%u, dictLimit=%u, nextToUpdate=%u)",
(unsigned)dstCapacity, (unsigned)zc->blockState.matchState.window.dictLimit, (unsigned)dstCapacity, (unsigned)zc->blockState.matchState.window.dictLimit,
(unsigned)zc->blockState.matchState.nextToUpdate); (unsigned)zc->blockState.matchState.nextToUpdate);
{ const size_t bss = ZSTD_buildSeqStore(zc, src, srcSize); { const size_t bss = ZSTD_buildSeqStore(zc, src, srcSize);
FORWARD_IF_ERROR(bss); FORWARD_IF_ERROR(bss);
if (bss == ZSTDbss_noCompress) { cSize = 0; goto out; } if (bss == ZSTDbss_noCompress) { cSize = 0; goto out; }
} }
if (zc->seqCollector.collectSequences) {
ZSTD_copyBlockSequences(zc);
return 0;
}
/* encode sequences and literals */ /* encode sequences and literals */
cSize = ZSTD_compressSequences(&zc->seqStore, cSize = ZSTD_compressSequences(&zc->seqStore,
&zc->blockState.prevCBlock->entropy, &zc->blockState.nextCBlock->entropy, &zc->blockState.prevCBlock->entropy, &zc->blockState.nextCBlock->entropy,
&zc->appliedParams, &zc->appliedParams,
dst, dstCapacity, dst, dstCapacity,
srcSize, srcSize,
zc->entropyWorkspace, HUF_WORKSPACE_SIZE /* statically allocated in resetCCtx */, zc->entropyWorkspace, HUF_WORKSPACE_SIZE /* statically allocated in resetCCtx */,
zc->bmi2); zc->bmi2);
if (frame &&
/* We don't want to emit our first block as a RLE even if it qualifies because
* doing so will cause the decoder (cli only) to throw a "should consume all input error."
* This is only an issue for zstd <= v1.4.3
*/
!zc->isFirstBlock &&
cSize < rleMaxLength &&
ZSTD_isRLE(ip, srcSize))
{
cSize = 1;
op[0] = ip[0];
}
out: out:
if (!ZSTD_isError(cSize) && cSize != 0) { if (!ZSTD_isError(cSize) && cSize > 1) {
/* confirm repcodes and entropy tables when emitting a compressed block */ /* confirm repcodes and entropy tables when emitting a compressed block */
ZSTD_compressedBlockState_t* const tmp = zc->blockState.prevCBlock; ZSTD_compressedBlockState_t* const tmp = zc->blockState.prevCBlock;
zc->blockState.prevCBlock = zc->blockState.nextCBlock; zc->blockState.prevCBlock = zc->blockState.nextCBlock;
zc->blockState.nextCBlock = tmp; zc->blockState.nextCBlock = tmp;
} }
/* We check that dictionaries have offset codes available for the first /* We check that dictionaries have offset codes available for the first
* block. After the first block, the offcode table might not have large * block. After the first block, the offcode table might not have large
* enough codes to represent the offsets in the data. * enough codes to represent the offsets in the data.
*/ */
if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid) if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid)
zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check; zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check;
return cSize; return cSize;
} }
static void ZSTD_overflowCorrectIfNeeded(ZSTD_matchState_t* ms, ZSTD_CCtx_params const* params, void const* ip, void const* iend) static void ZSTD_overflowCorrectIfNeeded(ZSTD_matchState_t* ms,
ZSTD_cwksp* ws,
ZSTD_CCtx_params const* params,
void const* ip,
void const* iend)
{ {
if (ZSTD_window_needOverflowCorrection(ms->window, iend)) { if (ZSTD_window_needOverflowCorrection(ms->window, iend)) {
U32 const maxDist = (U32)1 << params->cParams.windowLog; U32 const maxDist = (U32)1 << params->cParams.windowLog;
U32 const cycleLog = ZSTD_cycleLog(params->cParams.chainLog, params->cParams.strategy); U32 const cycleLog = ZSTD_cycleLog(params->cParams.chainLog, params->cParams.strategy);
U32 const correction = ZSTD_window_correctOverflow(&ms->window, cycleLog, maxDist, ip); U32 const correction = ZSTD_window_correctOverflow(&ms->window, cycleLog, maxDist, ip);
ZSTD_STATIC_ASSERT(ZSTD_CHAINLOG_MAX <= 30); ZSTD_STATIC_ASSERT(ZSTD_CHAINLOG_MAX <= 30);
ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX_32 <= 30); ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX_32 <= 30);
ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX <= 31); ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX <= 31);
ZSTD_cwksp_mark_tables_dirty(ws);
ZSTD_reduceIndex(ms, params, correction); ZSTD_reduceIndex(ms, params, correction);
ZSTD_cwksp_mark_tables_clean(ws);
if (ms->nextToUpdate < correction) ms->nextToUpdate = 0; if (ms->nextToUpdate < correction) ms->nextToUpdate = 0;
else ms->nextToUpdate -= correction; else ms->nextToUpdate -= correction;
/* invalidate dictionaries on overflow correction */ /* invalidate dictionaries on overflow correction */
ms->loadedDictEnd = 0; ms->loadedDictEnd = 0;
ms->dictMatchState = NULL; ms->dictMatchState = NULL;
} }
} }
/*! ZSTD_compress_frameChunk() : /*! ZSTD_compress_frameChunk() :
* Compress a chunk of data into one or multiple blocks. * Compress a chunk of data into one or multiple blocks.
* All blocks will be terminated, all input will be consumed. * All blocks will be terminated, all input will be consumed.
* Function will issue an error if there is not enough `dstCapacity` to hold the compressed content. * Function will issue an error if there is not enough `dstCapacity` to hold the compressed content.
* Frame is supposed already started (header already produced) * Frame is supposed already started (header already produced)
* @return : compressed size, or an error code * @return : compressed size, or an error code
*/ */
static size_t ZSTD_compress_frameChunk (ZSTD_CCtx* cctx, static size_t ZSTD_compress_frameChunk (ZSTD_CCtx* cctx,
ZSTD_matchState_t* const ms = &cctx->blockState.matchState; ZSTD_matchState_t* const ms = &cctx->blockState.matchState;
U32 const lastBlock = lastFrameChunk & (blockSize >= remaining); U32 const lastBlock = lastFrameChunk & (blockSize >= remaining);
RETURN_ERROR_IF(dstCapacity < ZSTD_blockHeaderSize + MIN_CBLOCK_SIZE, RETURN_ERROR_IF(dstCapacity < ZSTD_blockHeaderSize + MIN_CBLOCK_SIZE,
dstSize_tooSmall, dstSize_tooSmall,
"not enough space to store compressed block"); "not enough space to store compressed block");
if (remaining < blockSize) blockSize = remaining; if (remaining < blockSize) blockSize = remaining;
ZSTD_overflowCorrectIfNeeded(ms, &cctx->appliedParams, ip, ip + blockSize); ZSTD_overflowCorrectIfNeeded(
ms, &cctx->workspace, &cctx->appliedParams, ip, ip + blockSize);
ZSTD_checkDictValidity(&ms->window, ip + blockSize, maxDist, &ms->loadedDictEnd, &ms->dictMatchState); ZSTD_checkDictValidity(&ms->window, ip + blockSize, maxDist, &ms->loadedDictEnd, &ms->dictMatchState);
/* Ensure hash/chain table insertion resumes no sooner than lowlimit */ /* Ensure hash/chain table insertion resumes no sooner than lowlimit */
if (ms->nextToUpdate < ms->window.lowLimit) ms->nextToUpdate = ms->window.lowLimit; if (ms->nextToUpdate < ms->window.lowLimit) ms->nextToUpdate = ms->window.lowLimit;
{ size_t cSize = ZSTD_compressBlock_internal(cctx, { size_t cSize = ZSTD_compressBlock_internal(cctx,
op+ZSTD_blockHeaderSize, dstCapacity-ZSTD_blockHeaderSize, op+ZSTD_blockHeaderSize, dstCapacity-ZSTD_blockHeaderSize,
ip, blockSize); ip, blockSize, 1 /* frame */);
FORWARD_IF_ERROR(cSize); FORWARD_IF_ERROR(cSize);
if (cSize == 0) { /* block is not compressible */ if (cSize == 0) { /* block is not compressible */
cSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock); cSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock);
FORWARD_IF_ERROR(cSize); FORWARD_IF_ERROR(cSize);
} else { } else {
U32 const cBlockHeader24 = lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3); const U32 cBlockHeader = cSize == 1 ?
MEM_writeLE24(op, cBlockHeader24); lastBlock + (((U32)bt_rle)<<1) + (U32)(blockSize << 3) :
lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3);
MEM_writeLE24(op, cBlockHeader);
cSize += ZSTD_blockHeaderSize; cSize += ZSTD_blockHeaderSize;
} }
ip += blockSize; ip += blockSize;
assert(remaining >= blockSize); assert(remaining >= blockSize);
remaining -= blockSize; remaining -= blockSize;
op += cSize; op += cSize;
assert(dstCapacity >= cSize); assert(dstCapacity >= cSize);
dstCapacity -= cSize; dstCapacity -= cSize;
cctx->isFirstBlock = 0;
DEBUGLOG(5, "ZSTD_compress_frameChunk: adding a block of size %u", DEBUGLOG(5, "ZSTD_compress_frameChunk: adding a block of size %u",
(unsigned)cSize); (unsigned)cSize);
} } } }
if (lastFrameChunk && (op>ostart)) cctx->stage = ZSTDcs_ending; if (lastFrameChunk && (op>ostart)) cctx->stage = ZSTDcs_ending;
return (size_t)(op-ostart); return (size_t)(op-ostart);
} }
static size_t ZSTD_writeFrameHeader(void* dst, size_t dstCapacity, static size_t ZSTD_writeFrameHeader(void* dst, size_t dstCapacity,
ZSTD_CCtx_params params, U64 pledgedSrcSize, U32 dictID) const ZSTD_CCtx_params* params, U64 pledgedSrcSize, U32 dictID)
{ BYTE* const op = (BYTE*)dst; { BYTE* const op = (BYTE*)dst;
U32 const dictIDSizeCodeLength = (dictID>0) + (dictID>=256) + (dictID>=65536); /* 0-3 */ U32 const dictIDSizeCodeLength = (dictID>0) + (dictID>=256) + (dictID>=65536); /* 0-3 */
U32 const dictIDSizeCode = params.fParams.noDictIDFlag ? 0 : dictIDSizeCodeLength; /* 0-3 */ U32 const dictIDSizeCode = params->fParams.noDictIDFlag ? 0 : dictIDSizeCodeLength; /* 0-3 */
U32 const checksumFlag = params.fParams.checksumFlag>0; U32 const checksumFlag = params->fParams.checksumFlag>0;
U32 const windowSize = (U32)1 << params.cParams.windowLog; U32 const windowSize = (U32)1 << params->cParams.windowLog;
U32 const singleSegment = params.fParams.contentSizeFlag && (windowSize >= pledgedSrcSize); U32 const singleSegment = params->fParams.contentSizeFlag && (windowSize >= pledgedSrcSize);
BYTE const windowLogByte = (BYTE)((params.cParams.windowLog - ZSTD_WINDOWLOG_ABSOLUTEMIN) << 3); BYTE const windowLogByte = (BYTE)((params->cParams.windowLog - ZSTD_WINDOWLOG_ABSOLUTEMIN) << 3);
U32 const fcsCode = params.fParams.contentSizeFlag ? U32 const fcsCode = params->fParams.contentSizeFlag ?
(pledgedSrcSize>=256) + (pledgedSrcSize>=65536+256) + (pledgedSrcSize>=0xFFFFFFFFU) : 0; /* 0-3 */ (pledgedSrcSize>=256) + (pledgedSrcSize>=65536+256) + (pledgedSrcSize>=0xFFFFFFFFU) : 0; /* 0-3 */
BYTE const frameHeaderDescriptionByte = (BYTE)(dictIDSizeCode + (checksumFlag<<2) + (singleSegment<<5) + (fcsCode<<6) ); BYTE const frameHeaderDescriptionByte = (BYTE)(dictIDSizeCode + (checksumFlag<<2) + (singleSegment<<5) + (fcsCode<<6) );
size_t pos=0; size_t pos=0;
assert(!(params.fParams.contentSizeFlag && pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN)); assert(!(params->fParams.contentSizeFlag && pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN));
RETURN_ERROR_IF(dstCapacity < ZSTD_FRAMEHEADERSIZE_MAX, dstSize_tooSmall); RETURN_ERROR_IF(dstCapacity < ZSTD_FRAMEHEADERSIZE_MAX, dstSize_tooSmall);
DEBUGLOG(4, "ZSTD_writeFrameHeader : dictIDFlag : %u ; dictID : %u ; dictIDSizeCode : %u", DEBUGLOG(4, "ZSTD_writeFrameHeader : dictIDFlag : %u ; dictID : %u ; dictIDSizeCode : %u",
!params.fParams.noDictIDFlag, (unsigned)dictID, (unsigned)dictIDSizeCode); !params->fParams.noDictIDFlag, (unsigned)dictID, (unsigned)dictIDSizeCode);
if (params.format == ZSTD_f_zstd1) { if (params->format == ZSTD_f_zstd1) {
MEM_writeLE32(dst, ZSTD_MAGICNUMBER); MEM_writeLE32(dst, ZSTD_MAGICNUMBER);
pos = 4; pos = 4;
} }
op[pos++] = frameHeaderDescriptionByte; op[pos++] = frameHeaderDescriptionByte;
if (!singleSegment) op[pos++] = windowLogByte; if (!singleSegment) op[pos++] = windowLogByte;
switch(dictIDSizeCode) switch(dictIDSizeCode)
{ {
default: assert(0); /* impossible */ default: assert(0); /* impossible */
size_t fhSize = 0; size_t fhSize = 0;
DEBUGLOG(5, "ZSTD_compressContinue_internal, stage: %u, srcSize: %u", DEBUGLOG(5, "ZSTD_compressContinue_internal, stage: %u, srcSize: %u",
cctx->stage, (unsigned)srcSize); cctx->stage, (unsigned)srcSize);
RETURN_ERROR_IF(cctx->stage==ZSTDcs_created, stage_wrong, RETURN_ERROR_IF(cctx->stage==ZSTDcs_created, stage_wrong,
"missing init (ZSTD_compressBegin)"); "missing init (ZSTD_compressBegin)");
if (frame && (cctx->stage==ZSTDcs_init)) { if (frame && (cctx->stage==ZSTDcs_init)) {
fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, cctx->appliedParams, fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, &cctx->appliedParams,
cctx->pledgedSrcSizePlusOne-1, cctx->dictID); cctx->pledgedSrcSizePlusOne-1, cctx->dictID);
FORWARD_IF_ERROR(fhSize); FORWARD_IF_ERROR(fhSize);
assert(fhSize <= dstCapacity); assert(fhSize <= dstCapacity);
dstCapacity -= fhSize; dstCapacity -= fhSize;
dst = (char*)dst + fhSize; dst = (char*)dst + fhSize;
cctx->stage = ZSTDcs_ongoing; cctx->stage = ZSTDcs_ongoing;
} }
if (!srcSize) return fhSize; /* do not generate an empty block if no input */ if (!srcSize) return fhSize; /* do not generate an empty block if no input */
if (!ZSTD_window_update(&ms->window, src, srcSize)) { if (!ZSTD_window_update(&ms->window, src, srcSize)) {
ms->nextToUpdate = ms->window.dictLimit; ms->nextToUpdate = ms->window.dictLimit;
} }
if (cctx->appliedParams.ldmParams.enableLdm) { if (cctx->appliedParams.ldmParams.enableLdm) {
ZSTD_window_update(&cctx->ldmState.window, src, srcSize); ZSTD_window_update(&cctx->ldmState.window, src, srcSize);
} }
if (!frame) { if (!frame) {
/* overflow check and correction for block mode */ /* overflow check and correction for block mode */
ZSTD_overflowCorrectIfNeeded(ms, &cctx->appliedParams, src, (BYTE const*)src + srcSize); ZSTD_overflowCorrectIfNeeded(
ms, &cctx->workspace, &cctx->appliedParams,
src, (BYTE const*)src + srcSize);
} }
DEBUGLOG(5, "ZSTD_compressContinue_internal (blockSize=%u)", (unsigned)cctx->blockSize); DEBUGLOG(5, "ZSTD_compressContinue_internal (blockSize=%u)", (unsigned)cctx->blockSize);
{ size_t const cSize = frame ? { size_t const cSize = frame ?
ZSTD_compress_frameChunk (cctx, dst, dstCapacity, src, srcSize, lastFrameChunk) : ZSTD_compress_frameChunk (cctx, dst, dstCapacity, src, srcSize, lastFrameChunk) :
ZSTD_compressBlock_internal (cctx, dst, dstCapacity, src, srcSize); ZSTD_compressBlock_internal (cctx, dst, dstCapacity, src, srcSize, 0 /* frame */);
FORWARD_IF_ERROR(cSize); FORWARD_IF_ERROR(cSize);
cctx->consumedSrcSize += srcSize; cctx->consumedSrcSize += srcSize;
cctx->producedCSize += (cSize + fhSize); cctx->producedCSize += (cSize + fhSize);
assert(!(cctx->appliedParams.fParams.contentSizeFlag && cctx->pledgedSrcSizePlusOne == 0)); assert(!(cctx->appliedParams.fParams.contentSizeFlag && cctx->pledgedSrcSizePlusOne == 0));
if (cctx->pledgedSrcSizePlusOne != 0) { /* control src size */ if (cctx->pledgedSrcSizePlusOne != 0) { /* control src size */
ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_UNKNOWN == (unsigned long long)-1); ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_UNKNOWN == (unsigned long long)-1);
RETURN_ERROR_IF( RETURN_ERROR_IF(
cctx->consumedSrcSize+1 > cctx->pledgedSrcSizePlusOne, cctx->consumedSrcSize+1 > cctx->pledgedSrcSizePlusOne,
return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 0 /* frame mode */, 0 /* last chunk */); return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 0 /* frame mode */, 0 /* last chunk */);
} }
/*! ZSTD_loadDictionaryContent() : /*! ZSTD_loadDictionaryContent() :
* @return : 0, or an error code * @return : 0, or an error code
*/ */
static size_t ZSTD_loadDictionaryContent(ZSTD_matchState_t* ms, static size_t ZSTD_loadDictionaryContent(ZSTD_matchState_t* ms,
ZSTD_cwksp* ws,
ZSTD_CCtx_params const* params, ZSTD_CCtx_params const* params,
const void* src, size_t srcSize, const void* src, size_t srcSize,
ZSTD_dictTableLoadMethod_e dtlm) ZSTD_dictTableLoadMethod_e dtlm)
{ {
const BYTE* ip = (const BYTE*) src; const BYTE* ip = (const BYTE*) src;
const BYTE* const iend = ip + srcSize; const BYTE* const iend = ip + srcSize;
ZSTD_window_update(&ms->window, src, srcSize); ZSTD_window_update(&ms->window, src, srcSize);
ms->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ms->window.base); ms->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ms->window.base);
/* Assert that we the ms params match the params we're being given */ /* Assert that we the ms params match the params we're being given */
ZSTD_assertEqualCParams(params->cParams, ms->cParams); ZSTD_assertEqualCParams(params->cParams, ms->cParams);
if (srcSize <= HASH_READ_SIZE) return 0; if (srcSize <= HASH_READ_SIZE) return 0;
while (iend - ip > HASH_READ_SIZE) { while (iend - ip > HASH_READ_SIZE) {
size_t const remaining = (size_t)(iend - ip); size_t const remaining = (size_t)(iend - ip);
size_t const chunk = MIN(remaining, ZSTD_CHUNKSIZE_MAX); size_t const chunk = MIN(remaining, ZSTD_CHUNKSIZE_MAX);
const BYTE* const ichunk = ip + chunk; const BYTE* const ichunk = ip + chunk;
ZSTD_overflowCorrectIfNeeded(ms, params, ip, ichunk); ZSTD_overflowCorrectIfNeeded(ms, ws, params, ip, ichunk);
switch(params->cParams.strategy) switch(params->cParams.strategy)
{ {
case ZSTD_fast: case ZSTD_fast:
ZSTD_fillHashTable(ms, ichunk, dtlm); ZSTD_fillHashTable(ms, ichunk, dtlm);
break; break;
case ZSTD_dfast: case ZSTD_dfast:
ZSTD_fillDoubleHashTable(ms, ichunk, dtlm); ZSTD_fillDoubleHashTable(ms, ichunk, dtlm);
/* Dictionary format : /* Dictionary format :
* See : * See :
* https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#dictionary-format * https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#dictionary-format
*/ */
/*! ZSTD_loadZstdDictionary() : /*! ZSTD_loadZstdDictionary() :
* @return : dictID, or an error code * @return : dictID, or an error code
* assumptions : magic number supposed already checked * assumptions : magic number supposed already checked
* dictSize supposed > 8 * dictSize supposed >= 8
*/ */
static size_t ZSTD_loadZstdDictionary(ZSTD_compressedBlockState_t* bs, static size_t ZSTD_loadZstdDictionary(ZSTD_compressedBlockState_t* bs,
ZSTD_matchState_t* ms, ZSTD_matchState_t* ms,
ZSTD_cwksp* ws,
ZSTD_CCtx_params const* params, ZSTD_CCtx_params const* params,
const void* dict, size_t dictSize, const void* dict, size_t dictSize,
ZSTD_dictTableLoadMethod_e dtlm, ZSTD_dictTableLoadMethod_e dtlm,
void* workspace) void* workspace)
{ {
const BYTE* dictPtr = (const BYTE*)dict; const BYTE* dictPtr = (const BYTE*)dict;
const BYTE* const dictEnd = dictPtr + dictSize; const BYTE* const dictEnd = dictPtr + dictSize;
short offcodeNCount[MaxOff+1]; short offcodeNCount[MaxOff+1];
unsigned offcodeMaxValue = MaxOff; unsigned offcodeMaxValue = MaxOff;
size_t dictID; size_t dictID;
ZSTD_STATIC_ASSERT(HUF_WORKSPACE_SIZE >= (1<<MAX(MLFSELog,LLFSELog))); ZSTD_STATIC_ASSERT(HUF_WORKSPACE_SIZE >= (1<<MAX(MLFSELog,LLFSELog)));
assert(dictSize > 8); assert(dictSize >= 8);
assert(MEM_readLE32(dictPtr) == ZSTD_MAGIC_DICTIONARY); assert(MEM_readLE32(dictPtr) == ZSTD_MAGIC_DICTIONARY);
dictPtr += 4; /* skip magic number */ dictPtr += 4; /* skip magic number */
dictID = params->fParams.noDictIDFlag ? 0 : MEM_readLE32(dictPtr); dictID = params->fParams.noDictIDFlag ? 0 : MEM_readLE32(dictPtr);
dictPtr += 4; dictPtr += 4;
{ unsigned maxSymbolValue = 255; { unsigned maxSymbolValue = 255;
size_t const hufHeaderSize = HUF_readCTable((HUF_CElt*)bs->entropy.huf.CTable, &maxSymbolValue, dictPtr, dictEnd-dictPtr); size_t const hufHeaderSize = HUF_readCTable((HUF_CElt*)bs->entropy.huf.CTable, &maxSymbolValue, dictPtr, dictEnd-dictPtr);
RETURN_ERROR_IF(bs->rep[u] == 0, dictionary_corrupted); RETURN_ERROR_IF(bs->rep[u] == 0, dictionary_corrupted);
RETURN_ERROR_IF(bs->rep[u] > dictContentSize, dictionary_corrupted); RETURN_ERROR_IF(bs->rep[u] > dictContentSize, dictionary_corrupted);
} } } }
bs->entropy.huf.repeatMode = HUF_repeat_valid; bs->entropy.huf.repeatMode = HUF_repeat_valid;
bs->entropy.fse.offcode_repeatMode = FSE_repeat_valid; bs->entropy.fse.offcode_repeatMode = FSE_repeat_valid;
bs->entropy.fse.matchlength_repeatMode = FSE_repeat_valid; bs->entropy.fse.matchlength_repeatMode = FSE_repeat_valid;
bs->entropy.fse.litlength_repeatMode = FSE_repeat_valid; bs->entropy.fse.litlength_repeatMode = FSE_repeat_valid;
FORWARD_IF_ERROR(ZSTD_loadDictionaryContent(ms, params, dictPtr, dictContentSize, dtlm)); FORWARD_IF_ERROR(ZSTD_loadDictionaryContent(
ms, ws, params, dictPtr, dictContentSize, dtlm));
return dictID; return dictID;
} }
} }
/** ZSTD_compress_insertDictionary() : /** ZSTD_compress_insertDictionary() :
* @return : dictID, or an error code */ * @return : dictID, or an error code */
static size_t static size_t
ZSTD_compress_insertDictionary(ZSTD_compressedBlockState_t* bs, ZSTD_compress_insertDictionary(ZSTD_compressedBlockState_t* bs,
ZSTD_matchState_t* ms, ZSTD_matchState_t* ms,
ZSTD_cwksp* ws,
const ZSTD_CCtx_params* params, const ZSTD_CCtx_params* params,
const void* dict, size_t dictSize, const void* dict, size_t dictSize,
ZSTD_dictContentType_e dictContentType, ZSTD_dictContentType_e dictContentType,
ZSTD_dictTableLoadMethod_e dtlm, ZSTD_dictTableLoadMethod_e dtlm,
void* workspace) void* workspace)
{ {
DEBUGLOG(4, "ZSTD_compress_insertDictionary (dictSize=%u)", (U32)dictSize); DEBUGLOG(4, "ZSTD_compress_insertDictionary (dictSize=%u)", (U32)dictSize);
if ((dict==NULL) || (dictSize<=8)) return 0; if ((dict==NULL) || (dictSize<8)) {
RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong);
return 0;
}
ZSTD_reset_compressedBlockState(bs); ZSTD_reset_compressedBlockState(bs);
/* dict restricted modes */ /* dict restricted modes */
if (dictContentType == ZSTD_dct_rawContent) if (dictContentType == ZSTD_dct_rawContent)
return ZSTD_loadDictionaryContent(ms, params, dict, dictSize, dtlm); return ZSTD_loadDictionaryContent(ms, ws, params, dict, dictSize, dtlm);
if (MEM_readLE32(dict) != ZSTD_MAGIC_DICTIONARY) { if (MEM_readLE32(dict) != ZSTD_MAGIC_DICTIONARY) {
if (dictContentType == ZSTD_dct_auto) { if (dictContentType == ZSTD_dct_auto) {
DEBUGLOG(4, "raw content dictionary detected"); DEBUGLOG(4, "raw content dictionary detected");
return ZSTD_loadDictionaryContent(ms, params, dict, dictSize, dtlm); return ZSTD_loadDictionaryContent(
ms, ws, params, dict, dictSize, dtlm);
} }
RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong); RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong);
assert(0); /* impossible */ assert(0); /* impossible */
} }
/* dict as full zstd dictionary */ /* dict as full zstd dictionary */
return ZSTD_loadZstdDictionary(bs, ms, params, dict, dictSize, dtlm, workspace); return ZSTD_loadZstdDictionary(
bs, ms, ws, params, dict, dictSize, dtlm, workspace);
} }
#define ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF (128 KB)
#define ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER (6)
/*! ZSTD_compressBegin_internal() : /*! ZSTD_compressBegin_internal() :
* @return : 0, or an error code */ * @return : 0, or an error code */
static size_t ZSTD_compressBegin_internal(ZSTD_CCtx* cctx, static size_t ZSTD_compressBegin_internal(ZSTD_CCtx* cctx,
const void* dict, size_t dictSize, const void* dict, size_t dictSize,
ZSTD_dictContentType_e dictContentType, ZSTD_dictContentType_e dictContentType,
ZSTD_dictTableLoadMethod_e dtlm, ZSTD_dictTableLoadMethod_e dtlm,
const ZSTD_CDict* cdict, const ZSTD_CDict* cdict,
ZSTD_CCtx_params params, U64 pledgedSrcSize, const ZSTD_CCtx_params* params, U64 pledgedSrcSize,
ZSTD_buffered_policy_e zbuff) ZSTD_buffered_policy_e zbuff)
{ {
DEBUGLOG(4, "ZSTD_compressBegin_internal: wlog=%u", params.cParams.windowLog); DEBUGLOG(4, "ZSTD_compressBegin_internal: wlog=%u", params->cParams.windowLog);
/* params are supposed to be fully validated at this point */ /* params are supposed to be fully validated at this point */
assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams))); assert(!ZSTD_isError(ZSTD_checkCParams(params->cParams)));
assert(!((dict) && (cdict))); /* either dict or cdict, not both */ assert(!((dict) && (cdict))); /* either dict or cdict, not both */
if ( (cdict)
if (cdict && cdict->dictContentSize>0) { && (cdict->dictContentSize > 0)
&& ( pledgedSrcSize < ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF
|| pledgedSrcSize < cdict->dictContentSize * ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER
|| pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN
|| cdict->compressionLevel == 0)
&& (params->attachDictPref != ZSTD_dictForceLoad) ) {
return ZSTD_resetCCtx_usingCDict(cctx, cdict, params, pledgedSrcSize, zbuff); return ZSTD_resetCCtx_usingCDict(cctx, cdict, params, pledgedSrcSize, zbuff);
} }
FORWARD_IF_ERROR( ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize, FORWARD_IF_ERROR( ZSTD_resetCCtx_internal(cctx, *params, pledgedSrcSize,
ZSTDcrp_continue, zbuff) ); ZSTDcrp_makeClean, zbuff) );
{ size_t const dictID = ZSTD_compress_insertDictionary( { size_t const dictID = cdict ?
ZSTD_compress_insertDictionary(
cctx->blockState.prevCBlock, &cctx->blockState.matchState,
&cctx->workspace, params, cdict->dictContent, cdict->dictContentSize,
dictContentType, dtlm, cctx->entropyWorkspace)
: ZSTD_compress_insertDictionary(
cctx->blockState.prevCBlock, &cctx->blockState.matchState, cctx->blockState.prevCBlock, &cctx->blockState.matchState,
&params, dict, dictSize, dictContentType, dtlm, cctx->entropyWorkspace); &cctx->workspace, params, dict, dictSize,
dictContentType, dtlm, cctx->entropyWorkspace);
FORWARD_IF_ERROR(dictID); FORWARD_IF_ERROR(dictID);
assert(dictID <= UINT_MAX); assert(dictID <= UINT_MAX);
cctx->dictID = (U32)dictID; cctx->dictID = (U32)dictID;
} }
return 0; return 0;
} }
size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx, size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx,
const void* dict, size_t dictSize, const void* dict, size_t dictSize,
ZSTD_dictContentType_e dictContentType, ZSTD_dictContentType_e dictContentType,
ZSTD_dictTableLoadMethod_e dtlm, ZSTD_dictTableLoadMethod_e dtlm,
const ZSTD_CDict* cdict, const ZSTD_CDict* cdict,
ZSTD_CCtx_params params, const ZSTD_CCtx_params* params,
unsigned long long pledgedSrcSize) unsigned long long pledgedSrcSize)
{ {
DEBUGLOG(4, "ZSTD_compressBegin_advanced_internal: wlog=%u", params.cParams.windowLog); DEBUGLOG(4, "ZSTD_compressBegin_advanced_internal: wlog=%u", params->cParams.windowLog);
/* compression parameters verification and optimization */ /* compression parameters verification and optimization */
FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) ); FORWARD_IF_ERROR( ZSTD_checkCParams(params->cParams) );
return ZSTD_compressBegin_internal(cctx, return ZSTD_compressBegin_internal(cctx,
dict, dictSize, dictContentType, dtlm, dict, dictSize, dictContentType, dtlm,
cdict, cdict,
params, pledgedSrcSize, params, pledgedSrcSize,
ZSTDb_not_buffered); ZSTDb_not_buffered);
} }
/*! ZSTD_compressBegin_advanced() : /*! ZSTD_compressBegin_advanced() :
* @return : 0, or an error code */ * @return : 0, or an error code */
size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx,
const void* dict, size_t dictSize, const void* dict, size_t dictSize,
ZSTD_parameters params, unsigned long long pledgedSrcSize) ZSTD_parameters params, unsigned long long pledgedSrcSize)
{ {
ZSTD_CCtx_params const cctxParams = ZSTD_CCtx_params const cctxParams =
ZSTD_assignParamsToCCtxParams(cctx->requestedParams, params); ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, params);
return ZSTD_compressBegin_advanced_internal(cctx, return ZSTD_compressBegin_advanced_internal(cctx,
dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast,
NULL /*cdict*/, NULL /*cdict*/,
cctxParams, pledgedSrcSize); &cctxParams, pledgedSrcSize);
} }
size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel) size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel)
{ {
ZSTD_parameters const params = ZSTD_getParams(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize); ZSTD_parameters const params = ZSTD_getParams(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize);
ZSTD_CCtx_params const cctxParams = ZSTD_CCtx_params const cctxParams =
ZSTD_assignParamsToCCtxParams(cctx->requestedParams, params); ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, params);
DEBUGLOG(4, "ZSTD_compressBegin_usingDict (dictSize=%u)", (unsigned)dictSize); DEBUGLOG(4, "ZSTD_compressBegin_usingDict (dictSize=%u)", (unsigned)dictSize);
return ZSTD_compressBegin_internal(cctx, dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL, return ZSTD_compressBegin_internal(cctx, dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL,
cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, ZSTDb_not_buffered); &cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, ZSTDb_not_buffered);
} }
size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel) size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel)
{ {
return ZSTD_compressBegin_usingDict(cctx, NULL, 0, compressionLevel); return ZSTD_compressBegin_usingDict(cctx, NULL, 0, compressionLevel);
} }
/*! ZSTD_writeEpilogue() : /*! ZSTD_writeEpilogue() :
* Ends a frame. * Ends a frame.
* @return : nb of bytes written into dst (or an error code) */ * @return : nb of bytes written into dst (or an error code) */
static size_t ZSTD_writeEpilogue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity) static size_t ZSTD_writeEpilogue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity)
{ {
BYTE* const ostart = (BYTE*)dst; BYTE* const ostart = (BYTE*)dst;
BYTE* op = ostart; BYTE* op = ostart;
size_t fhSize = 0; size_t fhSize = 0;
DEBUGLOG(4, "ZSTD_writeEpilogue"); DEBUGLOG(4, "ZSTD_writeEpilogue");
RETURN_ERROR_IF(cctx->stage == ZSTDcs_created, stage_wrong, "init missing"); RETURN_ERROR_IF(cctx->stage == ZSTDcs_created, stage_wrong, "init missing");
/* special case : empty frame */ /* special case : empty frame */
if (cctx->stage == ZSTDcs_init) { if (cctx->stage == ZSTDcs_init) {
fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, cctx->appliedParams, 0, 0); fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, &cctx->appliedParams, 0, 0);
FORWARD_IF_ERROR(fhSize); FORWARD_IF_ERROR(fhSize);
dstCapacity -= fhSize; dstCapacity -= fhSize;
op += fhSize; op += fhSize;
cctx->stage = ZSTDcs_ongoing; cctx->stage = ZSTDcs_ongoing;
} }
if (cctx->stage != ZSTDcs_ending) { if (cctx->stage != ZSTDcs_ending) {
/* write one last empty block, make it the "last" block */ /* write one last empty block, make it the "last" block */
static size_t ZSTD_compress_internal (ZSTD_CCtx* cctx, static size_t ZSTD_compress_internal (ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize, const void* src, size_t srcSize,
const void* dict,size_t dictSize, const void* dict,size_t dictSize,
ZSTD_parameters params) ZSTD_parameters params)
{ {
ZSTD_CCtx_params const cctxParams = ZSTD_CCtx_params const cctxParams =
ZSTD_assignParamsToCCtxParams(cctx->requestedParams, params); ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, params);
DEBUGLOG(4, "ZSTD_compress_internal"); DEBUGLOG(4, "ZSTD_compress_internal");
return ZSTD_compress_advanced_internal(cctx, return ZSTD_compress_advanced_internal(cctx,
dst, dstCapacity, dst, dstCapacity,
src, srcSize, src, srcSize,
dict, dictSize, dict, dictSize,
cctxParams); &cctxParams);
} }
size_t ZSTD_compress_advanced (ZSTD_CCtx* cctx, size_t ZSTD_compress_advanced (ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize, const void* src, size_t srcSize,
const void* dict,size_t dictSize, const void* dict,size_t dictSize,
ZSTD_parameters params) ZSTD_parameters params)
{ {
DEBUGLOG(4, "ZSTD_compress_advanced"); DEBUGLOG(4, "ZSTD_compress_advanced");
FORWARD_IF_ERROR(ZSTD_checkCParams(params.cParams)); FORWARD_IF_ERROR(ZSTD_checkCParams(params.cParams));
return ZSTD_compress_internal(cctx, return ZSTD_compress_internal(cctx,
dst, dstCapacity, dst, dstCapacity,
src, srcSize, src, srcSize,
dict, dictSize, dict, dictSize,
params); params);
} }
/* Internal */ /* Internal */
size_t ZSTD_compress_advanced_internal( size_t ZSTD_compress_advanced_internal(
ZSTD_CCtx* cctx, ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize, const void* src, size_t srcSize,
const void* dict,size_t dictSize, const void* dict,size_t dictSize,
ZSTD_CCtx_params params) const ZSTD_CCtx_params* params)
{ {
DEBUGLOG(4, "ZSTD_compress_advanced_internal (srcSize:%u)", (unsigned)srcSize); DEBUGLOG(4, "ZSTD_compress_advanced_internal (srcSize:%u)", (unsigned)srcSize);
FORWARD_IF_ERROR( ZSTD_compressBegin_internal(cctx, FORWARD_IF_ERROR( ZSTD_compressBegin_internal(cctx,
dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL, dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL,
params, srcSize, ZSTDb_not_buffered) ); params, srcSize, ZSTDb_not_buffered) );
return ZSTD_compressEnd(cctx, dst, dstCapacity, src, srcSize); return ZSTD_compressEnd(cctx, dst, dstCapacity, src, srcSize);
} }
size_t ZSTD_compress_usingDict(ZSTD_CCtx* cctx, size_t ZSTD_compress_usingDict(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize, const void* src, size_t srcSize,
const void* dict, size_t dictSize, const void* dict, size_t dictSize,
int compressionLevel) int compressionLevel)
{ {
ZSTD_parameters const params = ZSTD_getParams(compressionLevel, srcSize + (!srcSize), dict ? dictSize : 0); ZSTD_parameters const params = ZSTD_getParams(compressionLevel, srcSize + (!srcSize), dict ? dictSize : 0);
ZSTD_CCtx_params cctxParams = ZSTD_assignParamsToCCtxParams(cctx->requestedParams, params); ZSTD_CCtx_params cctxParams = ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, params);
assert(params.fParams.contentSizeFlag == 1); assert(params.fParams.contentSizeFlag == 1);
return ZSTD_compress_advanced_internal(cctx, dst, dstCapacity, src, srcSize, dict, dictSize, cctxParams); return ZSTD_compress_advanced_internal(cctx, dst, dstCapacity, src, srcSize, dict, dictSize, &cctxParams);
} }
size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx, size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize, const void* src, size_t srcSize,
int compressionLevel) int compressionLevel)
{ {
DEBUGLOG(4, "ZSTD_compressCCtx (srcSize=%u)", (unsigned)srcSize); DEBUGLOG(4, "ZSTD_compressCCtx (srcSize=%u)", (unsigned)srcSize);
/*! ZSTD_estimateCDictSize_advanced() : /*! ZSTD_estimateCDictSize_advanced() :
* Estimate amount of memory that will be needed to create a dictionary with following arguments */ * Estimate amount of memory that will be needed to create a dictionary with following arguments */
size_t ZSTD_estimateCDictSize_advanced( size_t ZSTD_estimateCDictSize_advanced(
size_t dictSize, ZSTD_compressionParameters cParams, size_t dictSize, ZSTD_compressionParameters cParams,
ZSTD_dictLoadMethod_e dictLoadMethod) ZSTD_dictLoadMethod_e dictLoadMethod)
{ {
DEBUGLOG(5, "sizeof(ZSTD_CDict) : %u", (unsigned)sizeof(ZSTD_CDict)); DEBUGLOG(5, "sizeof(ZSTD_CDict) : %u", (unsigned)sizeof(ZSTD_CDict));
return sizeof(ZSTD_CDict) + HUF_WORKSPACE_SIZE + ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0) return ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict))
+ (dictLoadMethod == ZSTD_dlm_byRef ? 0 : dictSize); + ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE)
+ ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0)
+ (dictLoadMethod == ZSTD_dlm_byRef ? 0
: ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void *))));
} }
size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel) size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel)
{ {
ZSTD_compressionParameters const cParams = ZSTD_getCParams(compressionLevel, 0, dictSize); ZSTD_compressionParameters const cParams = ZSTD_getCParams(compressionLevel, 0, dictSize);
return ZSTD_estimateCDictSize_advanced(dictSize, cParams, ZSTD_dlm_byCopy); return ZSTD_estimateCDictSize_advanced(dictSize, cParams, ZSTD_dlm_byCopy);
} }
size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict) size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict)
{ {
if (cdict==NULL) return 0; /* support sizeof on NULL */ if (cdict==NULL) return 0; /* support sizeof on NULL */
DEBUGLOG(5, "sizeof(*cdict) : %u", (unsigned)sizeof(*cdict)); DEBUGLOG(5, "sizeof(*cdict) : %u", (unsigned)sizeof(*cdict));
return cdict->workspaceSize + (cdict->dictBuffer ? cdict->dictContentSize : 0) + sizeof(*cdict); /* cdict may be in the workspace */
return (cdict->workspace.workspace == cdict ? 0 : sizeof(*cdict))
+ ZSTD_cwksp_sizeof(&cdict->workspace);
} }
static size_t ZSTD_initCDict_internal( static size_t ZSTD_initCDict_internal(
ZSTD_CDict* cdict, ZSTD_CDict* cdict,
const void* dictBuffer, size_t dictSize, const void* dictBuffer, size_t dictSize,
ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictLoadMethod_e dictLoadMethod,
ZSTD_dictContentType_e dictContentType, ZSTD_dictContentType_e dictContentType,
ZSTD_compressionParameters cParams) ZSTD_compressionParameters cParams)
{ {
DEBUGLOG(3, "ZSTD_initCDict_internal (dictContentType:%u)", (unsigned)dictContentType); DEBUGLOG(3, "ZSTD_initCDict_internal (dictContentType:%u)", (unsigned)dictContentType);
assert(!ZSTD_checkCParams(cParams)); assert(!ZSTD_checkCParams(cParams));
cdict->matchState.cParams = cParams; cdict->matchState.cParams = cParams;
if ((dictLoadMethod == ZSTD_dlm_byRef) || (!dictBuffer) || (!dictSize)) { if ((dictLoadMethod == ZSTD_dlm_byRef) || (!dictBuffer) || (!dictSize)) {
cdict->dictBuffer = NULL;
cdict->dictContent = dictBuffer; cdict->dictContent = dictBuffer;
} else { } else {
void* const internalBuffer = ZSTD_malloc(dictSize, cdict->customMem); void *internalBuffer = ZSTD_cwksp_reserve_object(&cdict->workspace, ZSTD_cwksp_align(dictSize, sizeof(void*)));
cdict->dictBuffer = internalBuffer;
cdict->dictContent = internalBuffer;
RETURN_ERROR_IF(!internalBuffer, memory_allocation); RETURN_ERROR_IF(!internalBuffer, memory_allocation);
cdict->dictContent = internalBuffer;
memcpy(internalBuffer, dictBuffer, dictSize); memcpy(internalBuffer, dictBuffer, dictSize);
} }
cdict->dictContentSize = dictSize; cdict->dictContentSize = dictSize;
cdict->entropyWorkspace = (U32*)ZSTD_cwksp_reserve_object(&cdict->workspace, HUF_WORKSPACE_SIZE);
/* Reset the state to no dictionary */ /* Reset the state to no dictionary */
ZSTD_reset_compressedBlockState(&cdict->cBlockState); ZSTD_reset_compressedBlockState(&cdict->cBlockState);
{ void* const end = ZSTD_reset_matchState(&cdict->matchState, FORWARD_IF_ERROR(ZSTD_reset_matchState(
(U32*)cdict->workspace + HUF_WORKSPACE_SIZE_U32, &cdict->matchState,
&cdict->workspace,
&cParams, &cParams,
ZSTDcrp_continue, ZSTD_resetTarget_CDict); ZSTDcrp_makeClean,
assert(end == (char*)cdict->workspace + cdict->workspaceSize); ZSTDirp_reset,
(void)end; ZSTD_resetTarget_CDict));
}
/* (Maybe) load the dictionary /* (Maybe) load the dictionary
* Skips loading the dictionary if it is <= 8 bytes. * Skips loading the dictionary if it is < 8 bytes.
*/ */
{ ZSTD_CCtx_params params; { ZSTD_CCtx_params params;
memset(&params, 0, sizeof(params)); memset(&params, 0, sizeof(params));
params.compressionLevel = ZSTD_CLEVEL_DEFAULT; params.compressionLevel = ZSTD_CLEVEL_DEFAULT;
params.fParams.contentSizeFlag = 1; params.fParams.contentSizeFlag = 1;
params.cParams = cParams; params.cParams = cParams;
{ size_t const dictID = ZSTD_compress_insertDictionary( { size_t const dictID = ZSTD_compress_insertDictionary(
&cdict->cBlockState, &cdict->matchState, &params, &cdict->cBlockState, &cdict->matchState, &cdict->workspace,
cdict->dictContent, cdict->dictContentSize, &params, cdict->dictContent, cdict->dictContentSize,
dictContentType, ZSTD_dtlm_full, cdict->workspace); dictContentType, ZSTD_dtlm_full, cdict->entropyWorkspace);
FORWARD_IF_ERROR(dictID); FORWARD_IF_ERROR(dictID);
assert(dictID <= (size_t)(U32)-1); assert(dictID <= (size_t)(U32)-1);
cdict->dictID = (U32)dictID; cdict->dictID = (U32)dictID;
} }
} }
return 0; return 0;
} }
ZSTD_CDict* ZSTD_createCDict_advanced(const void* dictBuffer, size_t dictSize, ZSTD_CDict* ZSTD_createCDict_advanced(const void* dictBuffer, size_t dictSize,
ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictLoadMethod_e dictLoadMethod,
ZSTD_dictContentType_e dictContentType, ZSTD_dictContentType_e dictContentType,
ZSTD_compressionParameters cParams, ZSTD_customMem customMem) ZSTD_compressionParameters cParams, ZSTD_customMem customMem)
{ {
DEBUGLOG(3, "ZSTD_createCDict_advanced, mode %u", (unsigned)dictContentType); DEBUGLOG(3, "ZSTD_createCDict_advanced, mode %u", (unsigned)dictContentType);
if (!customMem.customAlloc ^ !customMem.customFree) return NULL; if (!customMem.customAlloc ^ !customMem.customFree) return NULL;
{ ZSTD_CDict* const cdict = (ZSTD_CDict*)ZSTD_malloc(sizeof(ZSTD_CDict), customMem); { size_t const workspaceSize =
size_t const workspaceSize = HUF_WORKSPACE_SIZE + ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0); ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict)) +
ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE) +
ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0) +
(dictLoadMethod == ZSTD_dlm_byRef ? 0
: ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void*))));
void* const workspace = ZSTD_malloc(workspaceSize, customMem); void* const workspace = ZSTD_malloc(workspaceSize, customMem);
ZSTD_cwksp ws;
ZSTD_CDict* cdict;
if (!cdict || !workspace) { if (!workspace) {
ZSTD_free(cdict, customMem);
ZSTD_free(workspace, customMem); ZSTD_free(workspace, customMem);
return NULL; return NULL;
} }
ZSTD_cwksp_init(&ws, workspace, workspaceSize);
cdict = (ZSTD_CDict*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CDict));
assert(cdict != NULL);
ZSTD_cwksp_move(&cdict->workspace, &ws);
cdict->customMem = customMem; cdict->customMem = customMem;
cdict->workspace = workspace; cdict->compressionLevel = 0; /* signals advanced API usage */
cdict->workspaceSize = workspaceSize;
if (ZSTD_isError( ZSTD_initCDict_internal(cdict, if (ZSTD_isError( ZSTD_initCDict_internal(cdict,
dictBuffer, dictSize, dictBuffer, dictSize,
dictLoadMethod, dictContentType, dictLoadMethod, dictContentType,
cParams) )) { cParams) )) {
ZSTD_freeCDict(cdict); ZSTD_freeCDict(cdict);
return NULL; return NULL;
} }
return cdict; return cdict;
} }
} }
ZSTD_CDict* ZSTD_createCDict(const void* dict, size_t dictSize, int compressionLevel) ZSTD_CDict* ZSTD_createCDict(const void* dict, size_t dictSize, int compressionLevel)
{ {
ZSTD_compressionParameters cParams = ZSTD_getCParams(compressionLevel, 0, dictSize); ZSTD_compressionParameters cParams = ZSTD_getCParams(compressionLevel, 0, dictSize);
return ZSTD_createCDict_advanced(dict, dictSize, ZSTD_CDict* cdict = ZSTD_createCDict_advanced(dict, dictSize,
ZSTD_dlm_byCopy, ZSTD_dct_auto, ZSTD_dlm_byCopy, ZSTD_dct_auto,
cParams, ZSTD_defaultCMem); cParams, ZSTD_defaultCMem);
if (cdict)
cdict->compressionLevel = compressionLevel == 0 ? ZSTD_CLEVEL_DEFAULT : compressionLevel;
return cdict;
} }
ZSTD_CDict* ZSTD_createCDict_byReference(const void* dict, size_t dictSize, int compressionLevel) ZSTD_CDict* ZSTD_createCDict_byReference(const void* dict, size_t dictSize, int compressionLevel)
{ {
ZSTD_compressionParameters cParams = ZSTD_getCParams(compressionLevel, 0, dictSize); ZSTD_compressionParameters cParams = ZSTD_getCParams(compressionLevel, 0, dictSize);
return ZSTD_createCDict_advanced(dict, dictSize, return ZSTD_createCDict_advanced(dict, dictSize,
ZSTD_dlm_byRef, ZSTD_dct_auto, ZSTD_dlm_byRef, ZSTD_dct_auto,
cParams, ZSTD_defaultCMem); cParams, ZSTD_defaultCMem);
} }
size_t ZSTD_freeCDict(ZSTD_CDict* cdict) size_t ZSTD_freeCDict(ZSTD_CDict* cdict)
{ {
if (cdict==NULL) return 0; /* support free on NULL */ if (cdict==NULL) return 0; /* support free on NULL */
{ ZSTD_customMem const cMem = cdict->customMem; { ZSTD_customMem const cMem = cdict->customMem;
ZSTD_free(cdict->workspace, cMem); int cdictInWorkspace = ZSTD_cwksp_owns_buffer(&cdict->workspace, cdict);
ZSTD_free(cdict->dictBuffer, cMem); ZSTD_cwksp_free(&cdict->workspace, cMem);
if (!cdictInWorkspace) {
ZSTD_free(cdict, cMem); ZSTD_free(cdict, cMem);
}
return 0; return 0;
} }
} }
/*! ZSTD_initStaticCDict_advanced() : /*! ZSTD_initStaticCDict_advanced() :
* Generate a digested dictionary in provided memory area. * Generate a digested dictionary in provided memory area.
* workspace: The memory area to emplace the dictionary into. * workspace: The memory area to emplace the dictionary into.
* Provided pointer must 8-bytes aligned. * Provided pointer must 8-bytes aligned.
const ZSTD_CDict* ZSTD_initStaticCDict( const ZSTD_CDict* ZSTD_initStaticCDict(
void* workspace, size_t workspaceSize, void* workspace, size_t workspaceSize,
const void* dict, size_t dictSize, const void* dict, size_t dictSize,
ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictLoadMethod_e dictLoadMethod,
ZSTD_dictContentType_e dictContentType, ZSTD_dictContentType_e dictContentType,
ZSTD_compressionParameters cParams) ZSTD_compressionParameters cParams)
{ {
size_t const matchStateSize = ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0); size_t const matchStateSize = ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0);
size_t const neededSize = sizeof(ZSTD_CDict) + (dictLoadMethod == ZSTD_dlm_byRef ? 0 : dictSize) size_t const neededSize = ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict))
+ HUF_WORKSPACE_SIZE + matchStateSize; + (dictLoadMethod == ZSTD_dlm_byRef ? 0
ZSTD_CDict* const cdict = (ZSTD_CDict*) workspace; : ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void*))))
void* ptr; + ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE)
+ matchStateSize;
ZSTD_CDict* cdict;
if ((size_t)workspace & 7) return NULL; /* 8-aligned */ if ((size_t)workspace & 7) return NULL; /* 8-aligned */
{
ZSTD_cwksp ws;
ZSTD_cwksp_init(&ws, workspace, workspaceSize);
cdict = (ZSTD_CDict*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CDict));
if (cdict == NULL) return NULL;
ZSTD_cwksp_move(&cdict->workspace, &ws);
}
DEBUGLOG(4, "(workspaceSize < neededSize) : (%u < %u) => %u", DEBUGLOG(4, "(workspaceSize < neededSize) : (%u < %u) => %u",
(unsigned)workspaceSize, (unsigned)neededSize, (unsigned)(workspaceSize < neededSize)); (unsigned)workspaceSize, (unsigned)neededSize, (unsigned)(workspaceSize < neededSize));
if (workspaceSize < neededSize) return NULL; if (workspaceSize < neededSize) return NULL;
if (dictLoadMethod == ZSTD_dlm_byCopy) {
memcpy(cdict+1, dict, dictSize);
dict = cdict+1;
ptr = (char*)workspace + sizeof(ZSTD_CDict) + dictSize;
} else {
ptr = cdict+1;
}
cdict->workspace = ptr;
cdict->workspaceSize = HUF_WORKSPACE_SIZE + matchStateSize;
if (ZSTD_isError( ZSTD_initCDict_internal(cdict, if (ZSTD_isError( ZSTD_initCDict_internal(cdict,
dict, dictSize, dict, dictSize,
ZSTD_dlm_byRef, dictContentType, dictLoadMethod, dictContentType,
cParams) )) cParams) ))
return NULL; return NULL;
return cdict; return cdict;
} }
ZSTD_compressionParameters ZSTD_getCParamsFromCDict(const ZSTD_CDict* cdict) ZSTD_compressionParameters ZSTD_getCParamsFromCDict(const ZSTD_CDict* cdict)
{ {
assert(cdict != NULL); assert(cdict != NULL);
return cdict->matchState.cParams; return cdict->matchState.cParams;
} }
/* ZSTD_compressBegin_usingCDict_advanced() : /* ZSTD_compressBegin_usingCDict_advanced() :
* cdict must be != NULL */ * cdict must be != NULL */
size_t ZSTD_compressBegin_usingCDict_advanced( size_t ZSTD_compressBegin_usingCDict_advanced(
ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict,
ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize) ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize)
{ {
DEBUGLOG(4, "ZSTD_compressBegin_usingCDict_advanced"); DEBUGLOG(4, "ZSTD_compressBegin_usingCDict_advanced");
RETURN_ERROR_IF(cdict==NULL, dictionary_wrong); RETURN_ERROR_IF(cdict==NULL, dictionary_wrong);
{ ZSTD_CCtx_params params = cctx->requestedParams; { ZSTD_CCtx_params params = cctx->requestedParams;
params.cParams = ZSTD_getCParamsFromCDict(cdict); params.cParams = ( pledgedSrcSize < ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF
|| pledgedSrcSize < cdict->dictContentSize * ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER
|| pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN
|| cdict->compressionLevel == 0 )
&& (params.attachDictPref != ZSTD_dictForceLoad) ?
ZSTD_getCParamsFromCDict(cdict)
: ZSTD_getCParams(cdict->compressionLevel,
pledgedSrcSize,
cdict->dictContentSize);
/* Increase window log to fit the entire dictionary and source if the /* Increase window log to fit the entire dictionary and source if the
* source size is known. Limit the increase to 19, which is the * source size is known. Limit the increase to 19, which is the
* window log for compression level 1 with the largest source size. * window log for compression level 1 with the largest source size.
*/ */
if (pledgedSrcSize != ZSTD_CONTENTSIZE_UNKNOWN) { if (pledgedSrcSize != ZSTD_CONTENTSIZE_UNKNOWN) {
U32 const limitedSrcSize = (U32)MIN(pledgedSrcSize, 1U << 19); U32 const limitedSrcSize = (U32)MIN(pledgedSrcSize, 1U << 19);
U32 const limitedSrcLog = limitedSrcSize > 1 ? ZSTD_highbit32(limitedSrcSize - 1) + 1 : 1; U32 const limitedSrcLog = limitedSrcSize > 1 ? ZSTD_highbit32(limitedSrcSize - 1) + 1 : 1;
params.cParams.windowLog = MAX(params.cParams.windowLog, limitedSrcLog); params.cParams.windowLog = MAX(params.cParams.windowLog, limitedSrcLog);
} }
params.fParams = fParams; params.fParams = fParams;
return ZSTD_compressBegin_internal(cctx, return ZSTD_compressBegin_internal(cctx,
NULL, 0, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL, 0, ZSTD_dct_auto, ZSTD_dtlm_fast,
cdict, cdict,
params, pledgedSrcSize, &params, pledgedSrcSize,
ZSTDb_not_buffered); ZSTDb_not_buffered);
} }
} }
/* ZSTD_compressBegin_usingCDict() : /* ZSTD_compressBegin_usingCDict() :
* pledgedSrcSize=0 means "unknown" * pledgedSrcSize=0 means "unknown"
* if pledgedSrcSize>0, it will enable contentSizeFlag */ * if pledgedSrcSize>0, it will enable contentSizeFlag */
size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict) size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict)
params.cParams = ZSTD_getCParamsFromCCtxParams(&params, pledgedSrcSize, dictSize); params.cParams = ZSTD_getCParamsFromCCtxParams(&params, pledgedSrcSize, dictSize);
/* params are supposed to be fully validated at this point */ /* params are supposed to be fully validated at this point */
assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams))); assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams)));
assert(!((dict) && (cdict))); /* either dict or cdict, not both */ assert(!((dict) && (cdict))); /* either dict or cdict, not both */
FORWARD_IF_ERROR( ZSTD_compressBegin_internal(cctx, FORWARD_IF_ERROR( ZSTD_compressBegin_internal(cctx,
dict, dictSize, dictContentType, ZSTD_dtlm_fast, dict, dictSize, dictContentType, ZSTD_dtlm_fast,
cdict, cdict,
params, pledgedSrcSize, &params, pledgedSrcSize,
ZSTDb_buffered) ); ZSTDb_buffered) );
cctx->inToCompress = 0; cctx->inToCompress = 0;
cctx->inBuffPos = 0; cctx->inBuffPos = 0;
cctx->inBuffTarget = cctx->blockSize cctx->inBuffTarget = cctx->blockSize
+ (cctx->blockSize == pledgedSrcSize); /* for small input: avoid automatic flush on reaching end of block, since it would require to add a 3-bytes null block to end frame */ + (cctx->blockSize == pledgedSrcSize); /* for small input: avoid automatic flush on reaching end of block, since it would require to add a 3-bytes null block to end frame */
cctx->outBuffContentSize = cctx->outBuffFlushedSize = 0; cctx->outBuffContentSize = cctx->outBuffFlushedSize = 0;
cctx->streamStage = zcss_load; cctx->streamStage = zcss_load;
} }
/*! ZSTD_initCStream_internal() : /*! ZSTD_initCStream_internal() :
* Note : for lib/compress only. Used by zstdmt_compress.c. * Note : for lib/compress only. Used by zstdmt_compress.c.
* Assumption 1 : params are valid * Assumption 1 : params are valid
* Assumption 2 : either dict, or cdict, is defined, not both */ * Assumption 2 : either dict, or cdict, is defined, not both */
size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs, size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs,
const void* dict, size_t dictSize, const ZSTD_CDict* cdict, const void* dict, size_t dictSize, const ZSTD_CDict* cdict,
ZSTD_CCtx_params params, unsigned long long pledgedSrcSize) const ZSTD_CCtx_params* params,
unsigned long long pledgedSrcSize)
{ {
DEBUGLOG(4, "ZSTD_initCStream_internal"); DEBUGLOG(4, "ZSTD_initCStream_internal");
FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) ); FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) );
FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) ); FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) );
assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams))); assert(!ZSTD_isError(ZSTD_checkCParams(params->cParams)));
zcs->requestedParams = params; zcs->requestedParams = *params;
assert(!((dict) && (cdict))); /* either dict or cdict, not both */ assert(!((dict) && (cdict))); /* either dict or cdict, not both */
if (dict) { if (dict) {
FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) ); FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) );
} else { } else {
/* Dictionary is cleared if !cdict */ /* Dictionary is cleared if !cdict */
FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) ); FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) );
} }
return 0; return 0;
FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) ); FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) );
return 0; return 0;
} }
/* ZSTD_initCStream_advanced() : /* ZSTD_initCStream_advanced() :
* pledgedSrcSize must be exact. * pledgedSrcSize must be exact.
* if srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN. * if srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN.
* dict is loaded with default parameters ZSTD_dm_auto and ZSTD_dlm_byCopy. */ * dict is loaded with default parameters ZSTD_dct_auto and ZSTD_dlm_byCopy. */
size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs, size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs,
const void* dict, size_t dictSize, const void* dict, size_t dictSize,
ZSTD_parameters params, unsigned long long pss) ZSTD_parameters params, unsigned long long pss)
{ {
/* for compatibility with older programs relying on this behavior. /* for compatibility with older programs relying on this behavior.
* Users should now specify ZSTD_CONTENTSIZE_UNKNOWN. * Users should now specify ZSTD_CONTENTSIZE_UNKNOWN.
* This line will be removed in the future. * This line will be removed in the future.
*/ */
U64 const pledgedSrcSize = (pss==0 && params.fParams.contentSizeFlag==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss; U64 const pledgedSrcSize = (pss==0 && params.fParams.contentSizeFlag==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss;
DEBUGLOG(4, "ZSTD_initCStream_advanced"); DEBUGLOG(4, "ZSTD_initCStream_advanced");
FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) ); FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) );
FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) ); FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) );
FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) ); FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) );
zcs->requestedParams = ZSTD_assignParamsToCCtxParams(zcs->requestedParams, params); zcs->requestedParams = ZSTD_assignParamsToCCtxParams(&zcs->requestedParams, params);
FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) ); FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) );
return 0; return 0;
} }
size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel) size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel)
{ {
DEBUGLOG(4, "ZSTD_initCStream_usingDict"); DEBUGLOG(4, "ZSTD_initCStream_usingDict");
FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) ); FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) );
if ((cctx->pledgedSrcSizePlusOne-1) <= ZSTDMT_JOBSIZE_MIN) { if ((cctx->pledgedSrcSizePlusOne-1) <= ZSTDMT_JOBSIZE_MIN) {
params.nbWorkers = 0; /* do not invoke multi-threading when src size is too small */ params.nbWorkers = 0; /* do not invoke multi-threading when src size is too small */
} }
if (params.nbWorkers > 0) { if (params.nbWorkers > 0) {
/* mt context creation */ /* mt context creation */
if (cctx->mtctx == NULL) { if (cctx->mtctx == NULL) {
DEBUGLOG(4, "ZSTD_compressStream2: creating new mtctx for nbWorkers=%u", DEBUGLOG(4, "ZSTD_compressStream2: creating new mtctx for nbWorkers=%u",
params.nbWorkers); params.nbWorkers);
cctx->mtctx = ZSTDMT_createCCtx_advanced(params.nbWorkers, cctx->customMem); cctx->mtctx = ZSTDMT_createCCtx_advanced((U32)params.nbWorkers, cctx->customMem);
RETURN_ERROR_IF(cctx->mtctx == NULL, memory_allocation); RETURN_ERROR_IF(cctx->mtctx == NULL, memory_allocation);
} }
/* mt compression */ /* mt compression */
DEBUGLOG(4, "call ZSTDMT_initCStream_internal as nbWorkers=%u", params.nbWorkers); DEBUGLOG(4, "call ZSTDMT_initCStream_internal as nbWorkers=%u", params.nbWorkers);
FORWARD_IF_ERROR( ZSTDMT_initCStream_internal( FORWARD_IF_ERROR( ZSTDMT_initCStream_internal(
cctx->mtctx, cctx->mtctx,
prefixDict.dict, prefixDict.dictSize, ZSTD_dct_rawContent, prefixDict.dict, prefixDict.dictSize, ZSTD_dct_rawContent,
cctx->cdict, params, cctx->pledgedSrcSizePlusOne-1) ); cctx->cdict, params, cctx->pledgedSrcSizePlusOne-1) );
int ZSTD_minCLevel(void) { return (int)-ZSTD_TARGETLENGTH_MAX; } int ZSTD_minCLevel(void) { return (int)-ZSTD_TARGETLENGTH_MAX; }
static const ZSTD_compressionParameters ZSTD_defaultCParameters[4][ZSTD_MAX_CLEVEL+1] = { static const ZSTD_compressionParameters ZSTD_defaultCParameters[4][ZSTD_MAX_CLEVEL+1] = {
{ /* "default" - for any srcSize > 256 KB */ { /* "default" - for any srcSize > 256 KB */
/* W, C, H, S, L, TL, strat */ /* W, C, H, S, L, TL, strat */
{ 19, 12, 13, 1, 6, 1, ZSTD_fast }, /* base for negative levels */ { 19, 12, 13, 1, 6, 1, ZSTD_fast }, /* base for negative levels */
{ 19, 13, 14, 1, 7, 0, ZSTD_fast }, /* level 1 */ { 19, 13, 14, 1, 7, 0, ZSTD_fast }, /* level 1 */
{ 20, 15, 16, 1, 6, 0, ZSTD_fast }, /* level 2 */ { 20, 15, 16, 1, 6, 0, ZSTD_fast }, /* level 2 */
{ 21, 16, 17, 1, 5, 1, ZSTD_dfast }, /* level 3 */ { 21, 16, 17, 1, 5, 0, ZSTD_dfast }, /* level 3 */
{ 21, 18, 18, 1, 5, 1, ZSTD_dfast }, /* level 4 */ { 21, 18, 18, 1, 5, 0, ZSTD_dfast }, /* level 4 */
{ 21, 18, 19, 2, 5, 2, ZSTD_greedy }, /* level 5 */ { 21, 18, 19, 2, 5, 2, ZSTD_greedy }, /* level 5 */
{ 21, 19, 19, 3, 5, 4, ZSTD_greedy }, /* level 6 */ { 21, 19, 19, 3, 5, 4, ZSTD_greedy }, /* level 6 */
{ 21, 19, 19, 3, 5, 8, ZSTD_lazy }, /* level 7 */ { 21, 19, 19, 3, 5, 8, ZSTD_lazy }, /* level 7 */
{ 21, 19, 19, 3, 5, 16, ZSTD_lazy2 }, /* level 8 */ { 21, 19, 19, 3, 5, 16, ZSTD_lazy2 }, /* level 8 */
{ 21, 19, 20, 4, 5, 16, ZSTD_lazy2 }, /* level 9 */ { 21, 19, 20, 4, 5, 16, ZSTD_lazy2 }, /* level 9 */
{ 22, 20, 21, 4, 5, 16, ZSTD_lazy2 }, /* level 10 */ { 22, 20, 21, 4, 5, 16, ZSTD_lazy2 }, /* level 10 */
{ 22, 21, 22, 4, 5, 16, ZSTD_lazy2 }, /* level 11 */ { 22, 21, 22, 4, 5, 16, ZSTD_lazy2 }, /* level 11 */
{ 22, 21, 22, 5, 5, 16, ZSTD_lazy2 }, /* level 12 */ { 22, 21, 22, 5, 5, 16, ZSTD_lazy2 }, /* level 12 */
{ 22, 21, 22, 5, 5, 32, ZSTD_btlazy2 }, /* level 13 */ { 22, 21, 22, 5, 5, 32, ZSTD_btlazy2 }, /* level 13 */
{ 22, 22, 23, 5, 5, 32, ZSTD_btlazy2 }, /* level 14 */ { 22, 22, 23, 5, 5, 32, ZSTD_btlazy2 }, /* level 14 */
{ 22, 23, 23, 6, 5, 32, ZSTD_btlazy2 }, /* level 15 */ { 22, 23, 23, 6, 5, 32, ZSTD_btlazy2 }, /* level 15 */
{ 22, 22, 22, 5, 5, 48, ZSTD_btopt }, /* level 16 */ { 22, 22, 22, 5, 5, 48, ZSTD_btopt }, /* level 16 */
{ 23, 23, 22, 5, 4, 64, ZSTD_btopt }, /* level 17 */ { 23, 23, 22, 5, 4, 64, ZSTD_btopt }, /* level 17 */
{ 23, 23, 22, 6, 3, 64, ZSTD_btultra }, /* level 18 */ { 23, 23, 22, 6, 3, 64, ZSTD_btultra }, /* level 18 */
{ 23, 24, 22, 7, 3,256, ZSTD_btultra2}, /* level 19 */ { 23, 24, 22, 7, 3,256, ZSTD_btultra2}, /* level 19 */
{ 25, 25, 23, 7, 3,256, ZSTD_btultra2}, /* level 20 */ { 25, 25, 23, 7, 3,256, ZSTD_btultra2}, /* level 20 */
{ 26, 26, 24, 7, 3,512, ZSTD_btultra2}, /* level 21 */ { 26, 26, 24, 7, 3,512, ZSTD_btultra2}, /* level 21 */
{ 27, 27, 25, 9, 3,999, ZSTD_btultra2}, /* level 22 */ { 27, 27, 25, 9, 3,999, ZSTD_btultra2}, /* level 22 */
}, },
{ /* for srcSize <= 256 KB */ { /* for srcSize <= 256 KB */
/* W, C, H, S, L, T, strat */ /* W, C, H, S, L, T, strat */
{ 18, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */ { 18, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
{ 18, 13, 14, 1, 6, 0, ZSTD_fast }, /* level 1 */ { 18, 13, 14, 1, 6, 0, ZSTD_fast }, /* level 1 */
{ 18, 14, 14, 1, 5, 1, ZSTD_dfast }, /* level 2 */ { 18, 14, 14, 1, 5, 0, ZSTD_dfast }, /* level 2 */
{ 18, 16, 16, 1, 4, 1, ZSTD_dfast }, /* level 3 */ { 18, 16, 16, 1, 4, 0, ZSTD_dfast }, /* level 3 */
{ 18, 16, 17, 2, 5, 2, ZSTD_greedy }, /* level 4.*/ { 18, 16, 17, 2, 5, 2, ZSTD_greedy }, /* level 4.*/
{ 18, 18, 18, 3, 5, 2, ZSTD_greedy }, /* level 5.*/ { 18, 18, 18, 3, 5, 2, ZSTD_greedy }, /* level 5.*/
{ 18, 18, 19, 3, 5, 4, ZSTD_lazy }, /* level 6.*/ { 18, 18, 19, 3, 5, 4, ZSTD_lazy }, /* level 6.*/
{ 18, 18, 19, 4, 4, 4, ZSTD_lazy }, /* level 7 */ { 18, 18, 19, 4, 4, 4, ZSTD_lazy }, /* level 7 */
{ 18, 18, 19, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */ { 18, 18, 19, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */
{ 18, 18, 19, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */ { 18, 18, 19, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */
{ 18, 18, 19, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */ { 18, 18, 19, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */
{ 18, 18, 19, 5, 4, 12, ZSTD_btlazy2 }, /* level 11.*/ { 18, 18, 19, 5, 4, 12, ZSTD_btlazy2 }, /* level 11.*/
{ 18, 19, 19, 12, 3,512, ZSTD_btultra2}, /* level 21.*/ { 18, 19, 19, 12, 3,512, ZSTD_btultra2}, /* level 21.*/
{ 18, 19, 19, 13, 3,999, ZSTD_btultra2}, /* level 22.*/ { 18, 19, 19, 13, 3,999, ZSTD_btultra2}, /* level 22.*/
}, },
{ /* for srcSize <= 128 KB */ { /* for srcSize <= 128 KB */
/* W, C, H, S, L, T, strat */ /* W, C, H, S, L, T, strat */
{ 17, 12, 12, 1, 5, 1, ZSTD_fast }, /* base for negative levels */ { 17, 12, 12, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
{ 17, 12, 13, 1, 6, 0, ZSTD_fast }, /* level 1 */ { 17, 12, 13, 1, 6, 0, ZSTD_fast }, /* level 1 */
{ 17, 13, 15, 1, 5, 0, ZSTD_fast }, /* level 2 */ { 17, 13, 15, 1, 5, 0, ZSTD_fast }, /* level 2 */
{ 17, 15, 16, 2, 5, 1, ZSTD_dfast }, /* level 3 */ { 17, 15, 16, 2, 5, 0, ZSTD_dfast }, /* level 3 */
{ 17, 17, 17, 2, 4, 1, ZSTD_dfast }, /* level 4 */ { 17, 17, 17, 2, 4, 0, ZSTD_dfast }, /* level 4 */
{ 17, 16, 17, 3, 4, 2, ZSTD_greedy }, /* level 5 */ { 17, 16, 17, 3, 4, 2, ZSTD_greedy }, /* level 5 */
{ 17, 17, 17, 3, 4, 4, ZSTD_lazy }, /* level 6 */ { 17, 17, 17, 3, 4, 4, ZSTD_lazy }, /* level 6 */
{ 17, 17, 17, 3, 4, 8, ZSTD_lazy2 }, /* level 7 */ { 17, 17, 17, 3, 4, 8, ZSTD_lazy2 }, /* level 7 */
{ 17, 17, 17, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */ { 17, 17, 17, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */
{ 17, 17, 17, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */ { 17, 17, 17, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */
{ 17, 17, 17, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */ { 17, 17, 17, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */
{ 17, 17, 17, 5, 4, 8, ZSTD_btlazy2 }, /* level 11 */ { 17, 17, 17, 5, 4, 8, ZSTD_btlazy2 }, /* level 11 */
{ 17, 18, 17, 7, 4, 12, ZSTD_btlazy2 }, /* level 12 */ { 17, 18, 17, 7, 4, 12, ZSTD_btlazy2 }, /* level 12 */
{ 17, 18, 17, 3, 4, 12, ZSTD_btopt }, /* level 13.*/ { 17, 18, 17, 3, 4, 12, ZSTD_btopt }, /* level 13.*/
{ 17, 18, 17, 4, 3, 32, ZSTD_btopt }, /* level 14.*/ { 17, 18, 17, 4, 3, 32, ZSTD_btopt }, /* level 14.*/
{ 17, 18, 17, 6, 3,256, ZSTD_btopt }, /* level 15.*/ { 17, 18, 17, 6, 3,256, ZSTD_btopt }, /* level 15.*/
{ 17, 18, 17, 6, 3,128, ZSTD_btultra }, /* level 16.*/ { 17, 18, 17, 6, 3,128, ZSTD_btultra }, /* level 16.*/
{ 17, 18, 17, 8, 3,256, ZSTD_btultra }, /* level 17.*/ { 17, 18, 17, 8, 3,256, ZSTD_btultra }, /* level 17.*/
{ 17, 18, 17, 10, 3,512, ZSTD_btultra }, /* level 18.*/ { 17, 18, 17, 10, 3,512, ZSTD_btultra }, /* level 18.*/
{ 17, 18, 17, 5, 3,256, ZSTD_btultra2}, /* level 19.*/ { 17, 18, 17, 5, 3,256, ZSTD_btultra2}, /* level 19.*/
{ 17, 18, 17, 7, 3,512, ZSTD_btultra2}, /* level 20.*/ { 17, 18, 17, 7, 3,512, ZSTD_btultra2}, /* level 20.*/
{ 17, 18, 17, 9, 3,512, ZSTD_btultra2}, /* level 21.*/ { 17, 18, 17, 9, 3,512, ZSTD_btultra2}, /* level 21.*/
{ 17, 18, 17, 11, 3,999, ZSTD_btultra2}, /* level 22.*/ { 17, 18, 17, 11, 3,999, ZSTD_btultra2}, /* level 22.*/
}, },
{ /* for srcSize <= 16 KB */ { /* for srcSize <= 16 KB */
/* W, C, H, S, L, T, strat */ /* W, C, H, S, L, T, strat */
{ 14, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */ { 14, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
{ 14, 14, 15, 1, 5, 0, ZSTD_fast }, /* level 1 */ { 14, 14, 15, 1, 5, 0, ZSTD_fast }, /* level 1 */
{ 14, 14, 15, 1, 4, 0, ZSTD_fast }, /* level 2 */ { 14, 14, 15, 1, 4, 0, ZSTD_fast }, /* level 2 */
{ 14, 14, 15, 2, 4, 1, ZSTD_dfast }, /* level 3 */ { 14, 14, 15, 2, 4, 0, ZSTD_dfast }, /* level 3 */
{ 14, 14, 14, 4, 4, 2, ZSTD_greedy }, /* level 4 */ { 14, 14, 14, 4, 4, 2, ZSTD_greedy }, /* level 4 */
{ 14, 14, 14, 3, 4, 4, ZSTD_lazy }, /* level 5.*/ { 14, 14, 14, 3, 4, 4, ZSTD_lazy }, /* level 5.*/
{ 14, 14, 14, 4, 4, 8, ZSTD_lazy2 }, /* level 6 */ { 14, 14, 14, 4, 4, 8, ZSTD_lazy2 }, /* level 6 */
{ 14, 14, 14, 6, 4, 8, ZSTD_lazy2 }, /* level 7 */ { 14, 14, 14, 6, 4, 8, ZSTD_lazy2 }, /* level 7 */
{ 14, 14, 14, 8, 4, 8, ZSTD_lazy2 }, /* level 8.*/ { 14, 14, 14, 8, 4, 8, ZSTD_lazy2 }, /* level 8.*/
{ 14, 15, 14, 5, 4, 8, ZSTD_btlazy2 }, /* level 9.*/ { 14, 15, 14, 5, 4, 8, ZSTD_btlazy2 }, /* level 9.*/
{ 14, 15, 14, 9, 4, 8, ZSTD_btlazy2 }, /* level 10.*/ { 14, 15, 14, 9, 4, 8, ZSTD_btlazy2 }, /* level 10.*/
{ 14, 15, 14, 3, 4, 12, ZSTD_btopt }, /* level 11.*/ { 14, 15, 14, 3, 4, 12, ZSTD_btopt }, /* level 11.*/

contrib/python-zstandard/zstd/compress/zstd_compress_internal.h

#ifndef ZSTD_COMPRESS_H #ifndef ZSTD_COMPRESS_H
#define ZSTD_COMPRESS_H #define ZSTD_COMPRESS_H
/*-************************************* /*-*************************************
* Dependencies * Dependencies
***************************************/ ***************************************/
#include "zstd_internal.h" #include "zstd_internal.h"
#include "zstd_cwksp.h"
#ifdef ZSTD_MULTITHREAD #ifdef ZSTD_MULTITHREAD
# include "zstdmt_compress.h" # include "zstdmt_compress.h"
#endif #endif
#if defined (__cplusplus) #if defined (__cplusplus)
extern "C" { extern "C" {
#endif #endif
typedef struct { typedef struct {
rawSeq* seq; /* The start of the sequences */ rawSeq* seq; /* The start of the sequences */
size_t pos; /* The position where reading stopped. <= size. */ size_t pos; /* The position where reading stopped. <= size. */
size_t size; /* The number of sequences. <= capacity. */ size_t size; /* The number of sequences. <= capacity. */
size_t capacity; /* The capacity starting from `seq` pointer */ size_t capacity; /* The capacity starting from `seq` pointer */
} rawSeqStore_t; } rawSeqStore_t;
typedef struct {
int collectSequences;
ZSTD_Sequence* seqStart;
size_t seqIndex;
size_t maxSequences;
} SeqCollector;
struct ZSTD_CCtx_params_s { struct ZSTD_CCtx_params_s {
ZSTD_format_e format; ZSTD_format_e format;
ZSTD_compressionParameters cParams; ZSTD_compressionParameters cParams;
ZSTD_frameParameters fParams; ZSTD_frameParameters fParams;
int compressionLevel; int compressionLevel;
int forceWindow; /* force back-references to respect limit of int forceWindow; /* force back-references to respect limit of
* 1<<wLog, even for dictionary */ * 1<<wLog, even for dictionary */
size_t targetCBlockSize; /* Tries to fit compressed block size to be around targetCBlockSize. size_t targetCBlockSize; /* Tries to fit compressed block size to be around targetCBlockSize.
* No target when targetCBlockSize == 0. * No target when targetCBlockSize == 0.
* There is no guarantee on compressed block size */ * There is no guarantee on compressed block size */
int srcSizeHint; /* User's best guess of source size.
* Hint is not valid when srcSizeHint == 0.
* There is no guarantee that hint is close to actual source size */
ZSTD_dictAttachPref_e attachDictPref; ZSTD_dictAttachPref_e attachDictPref;
ZSTD_literalCompressionMode_e literalCompressionMode; ZSTD_literalCompressionMode_e literalCompressionMode;
/* Multithreading: used to pass parameters to mtctx */ /* Multithreading: used to pass parameters to mtctx */
int nbWorkers; int nbWorkers;
size_t jobSize; size_t jobSize;
int overlapLog; int overlapLog;
struct ZSTD_CCtx_s { struct ZSTD_CCtx_s {
ZSTD_compressionStage_e stage; ZSTD_compressionStage_e stage;
int cParamsChanged; /* == 1 if cParams(except wlog) or compression level are changed in requestedParams. Triggers transmission of new params to ZSTDMT (if available) then reset to 0. */ int cParamsChanged; /* == 1 if cParams(except wlog) or compression level are changed in requestedParams. Triggers transmission of new params to ZSTDMT (if available) then reset to 0. */
int bmi2; /* == 1 if the CPU supports BMI2 and 0 otherwise. CPU support is determined dynamically once per context lifetime. */ int bmi2; /* == 1 if the CPU supports BMI2 and 0 otherwise. CPU support is determined dynamically once per context lifetime. */
ZSTD_CCtx_params requestedParams; ZSTD_CCtx_params requestedParams;
ZSTD_CCtx_params appliedParams; ZSTD_CCtx_params appliedParams;
U32 dictID; U32 dictID;
int workSpaceOversizedDuration; ZSTD_cwksp workspace; /* manages buffer for dynamic allocations */
void* workSpace;
size_t workSpaceSize;
size_t blockSize; size_t blockSize;
unsigned long long pledgedSrcSizePlusOne; /* this way, 0 (default) == unknown */ unsigned long long pledgedSrcSizePlusOne; /* this way, 0 (default) == unknown */
unsigned long long consumedSrcSize; unsigned long long consumedSrcSize;
unsigned long long producedCSize; unsigned long long producedCSize;
XXH64_state_t xxhState; XXH64_state_t xxhState;
ZSTD_customMem customMem; ZSTD_customMem customMem;
size_t staticSize; size_t staticSize;
SeqCollector seqCollector;
int isFirstBlock;
seqStore_t seqStore; /* sequences storage ptrs */ seqStore_t seqStore; /* sequences storage ptrs */
ldmState_t ldmState; /* long distance matching state */ ldmState_t ldmState; /* long distance matching state */
rawSeq* ldmSequences; /* Storage for the ldm output sequences */ rawSeq* ldmSequences; /* Storage for the ldm output sequences */
size_t maxNbLdmSequences; size_t maxNbLdmSequences;
rawSeqStore_t externSeqStore; /* Mutable reference to external sequences */ rawSeqStore_t externSeqStore; /* Mutable reference to external sequences */
ZSTD_blockState_t blockState; ZSTD_blockState_t blockState;
U32* entropyWorkspace; /* entropy workspace of HUF_WORKSPACE_SIZE bytes */ U32* entropyWorkspace; /* entropy workspace of HUF_WORKSPACE_SIZE bytes */
MEM_STATIC size_t ZSTD_minGain(size_t srcSize, ZSTD_strategy strat) MEM_STATIC size_t ZSTD_minGain(size_t srcSize, ZSTD_strategy strat)
{ {
U32 const minlog = (strat>=ZSTD_btultra) ? (U32)(strat) - 1 : 6; U32 const minlog = (strat>=ZSTD_btultra) ? (U32)(strat) - 1 : 6;
ZSTD_STATIC_ASSERT(ZSTD_btultra == 8); ZSTD_STATIC_ASSERT(ZSTD_btultra == 8);
assert(ZSTD_cParam_withinBounds(ZSTD_c_strategy, strat)); assert(ZSTD_cParam_withinBounds(ZSTD_c_strategy, strat));
return (srcSize >> minlog) + 2; return (srcSize >> minlog) + 2;
} }
/*! ZSTD_safecopyLiterals() :
* memcpy() function that won't read beyond more than WILDCOPY_OVERLENGTH bytes past ilimit_w.
* Only called when the sequence ends past ilimit_w, so it only needs to be optimized for single
* large copies.
*/
static void ZSTD_safecopyLiterals(BYTE* op, BYTE const* ip, BYTE const* const iend, BYTE const* ilimit_w) {
assert(iend > ilimit_w);
if (ip <= ilimit_w) {
ZSTD_wildcopy(op, ip, ilimit_w - ip, ZSTD_no_overlap);
op += ilimit_w - ip;
ip = ilimit_w;
}
while (ip < iend) *op++ = *ip++;
}
/*! ZSTD_storeSeq() : /*! ZSTD_storeSeq() :
* Store a sequence (literal length, literals, offset code and match length code) into seqStore_t. * Store a sequence (litlen, litPtr, offCode and mlBase) into seqStore_t.
* `offsetCode` : distance to match + 3 (values 1-3 are repCodes). * `offCode` : distance to match + ZSTD_REP_MOVE (values <= ZSTD_REP_MOVE are repCodes).
* `mlBase` : matchLength - MINMATCH * `mlBase` : matchLength - MINMATCH
* Allowed to overread literals up to litLimit.
*/ */
MEM_STATIC void ZSTD_storeSeq(seqStore_t* seqStorePtr, size_t litLength, const void* literals, U32 offsetCode, size_t mlBase) HINT_INLINE UNUSED_ATTR
void ZSTD_storeSeq(seqStore_t* seqStorePtr, size_t litLength, const BYTE* literals, const BYTE* litLimit, U32 offCode, size_t mlBase)
{ {
BYTE const* const litLimit_w = litLimit - WILDCOPY_OVERLENGTH;
BYTE const* const litEnd = literals + litLength;
#if defined(DEBUGLEVEL) && (DEBUGLEVEL >= 6) #if defined(DEBUGLEVEL) && (DEBUGLEVEL >= 6)
static const BYTE* g_start = NULL; static const BYTE* g_start = NULL;
if (g_start==NULL) g_start = (const BYTE*)literals; /* note : index only works for compression within a single segment */ if (g_start==NULL) g_start = (const BYTE*)literals; /* note : index only works for compression within a single segment */
{ U32 const pos = (U32)((const BYTE*)literals - g_start); { U32 const pos = (U32)((const BYTE*)literals - g_start);
DEBUGLOG(6, "Cpos%7u :%3u literals, match%4u bytes at offCode%7u", DEBUGLOG(6, "Cpos%7u :%3u literals, match%4u bytes at offCode%7u",
pos, (U32)litLength, (U32)mlBase+MINMATCH, (U32)offsetCode); pos, (U32)litLength, (U32)mlBase+MINMATCH, (U32)offCode);
} }
#endif #endif
assert((size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart) < seqStorePtr->maxNbSeq); assert((size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart) < seqStorePtr->maxNbSeq);
/* copy Literals */ /* copy Literals */
assert(seqStorePtr->maxNbLit <= 128 KB); assert(seqStorePtr->maxNbLit <= 128 KB);
assert(seqStorePtr->lit + litLength <= seqStorePtr->litStart + seqStorePtr->maxNbLit); assert(seqStorePtr->lit + litLength <= seqStorePtr->litStart + seqStorePtr->maxNbLit);
ZSTD_wildcopy(seqStorePtr->lit, literals, (ptrdiff_t)litLength, ZSTD_no_overlap); assert(literals + litLength <= litLimit);
if (litEnd <= litLimit_w) {
/* Common case we can use wildcopy.
* First copy 16 bytes, because literals are likely short.
*/
assert(WILDCOPY_OVERLENGTH >= 16);
ZSTD_copy16(seqStorePtr->lit, literals);
if (litLength > 16) {
ZSTD_wildcopy(seqStorePtr->lit+16, literals+16, (ptrdiff_t)litLength-16, ZSTD_no_overlap);
}
} else {
ZSTD_safecopyLiterals(seqStorePtr->lit, literals, litEnd, litLimit_w);
}
seqStorePtr->lit += litLength; seqStorePtr->lit += litLength;
/* literal Length */ /* literal Length */
if (litLength>0xFFFF) { if (litLength>0xFFFF) {
assert(seqStorePtr->longLengthID == 0); /* there can only be a single long length */ assert(seqStorePtr->longLengthID == 0); /* there can only be a single long length */
seqStorePtr->longLengthID = 1; seqStorePtr->longLengthID = 1;
seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart); seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
} }
seqStorePtr->sequences[0].litLength = (U16)litLength; seqStorePtr->sequences[0].litLength = (U16)litLength;
/* match offset */ /* match offset */
seqStorePtr->sequences[0].offset = offsetCode + 1; seqStorePtr->sequences[0].offset = offCode + 1;
/* match Length */ /* match Length */
if (mlBase>0xFFFF) { if (mlBase>0xFFFF) {
assert(seqStorePtr->longLengthID == 0); /* there can only be a single long length */ assert(seqStorePtr->longLengthID == 0); /* there can only be a single long length */
seqStorePtr->longLengthID = 2; seqStorePtr->longLengthID = 2;
seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart); seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
} }
seqStorePtr->sequences[0].matchLength = (U16)mlBase; seqStorePtr->sequences[0].matchLength = (U16)mlBase;
/*! ZSTD_initCStream_internal() : /*! ZSTD_initCStream_internal() :
* Private use only. Init streaming operation. * Private use only. Init streaming operation.
* expects params to be valid. * expects params to be valid.
* must receive dict, or cdict, or none, but not both. * must receive dict, or cdict, or none, but not both.
* @return : 0, or an error code */ * @return : 0, or an error code */
size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs, size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs,
const void* dict, size_t dictSize, const void* dict, size_t dictSize,
const ZSTD_CDict* cdict, const ZSTD_CDict* cdict,
ZSTD_CCtx_params params, unsigned long long pledgedSrcSize); const ZSTD_CCtx_params* params, unsigned long long pledgedSrcSize);
void ZSTD_resetSeqStore(seqStore_t* ssPtr); void ZSTD_resetSeqStore(seqStore_t* ssPtr);
/*! ZSTD_getCParamsFromCDict() : /*! ZSTD_getCParamsFromCDict() :
* as the name implies */ * as the name implies */
ZSTD_compressionParameters ZSTD_getCParamsFromCDict(const ZSTD_CDict* cdict); ZSTD_compressionParameters ZSTD_getCParamsFromCDict(const ZSTD_CDict* cdict);
/* ZSTD_compressBegin_advanced_internal() : /* ZSTD_compressBegin_advanced_internal() :
* Private use only. To be called from zstdmt_compress.c. */ * Private use only. To be called from zstdmt_compress.c. */
size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx, size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx,
const void* dict, size_t dictSize, const void* dict, size_t dictSize,
ZSTD_dictContentType_e dictContentType, ZSTD_dictContentType_e dictContentType,
ZSTD_dictTableLoadMethod_e dtlm, ZSTD_dictTableLoadMethod_e dtlm,
const ZSTD_CDict* cdict, const ZSTD_CDict* cdict,
ZSTD_CCtx_params params, const ZSTD_CCtx_params* params,
unsigned long long pledgedSrcSize); unsigned long long pledgedSrcSize);
/* ZSTD_compress_advanced_internal() : /* ZSTD_compress_advanced_internal() :
* Private use only. To be called from zstdmt_compress.c. */ * Private use only. To be called from zstdmt_compress.c. */
size_t ZSTD_compress_advanced_internal(ZSTD_CCtx* cctx, size_t ZSTD_compress_advanced_internal(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize, const void* src, size_t srcSize,
const void* dict,size_t dictSize, const void* dict,size_t dictSize,
ZSTD_CCtx_params params); const ZSTD_CCtx_params* params);
/* ZSTD_writeLastEmptyBlock() : /* ZSTD_writeLastEmptyBlock() :
* output an empty Block with end-of-frame mark to complete a frame * output an empty Block with end-of-frame mark to complete a frame
* @return : size of data written into `dst` (== ZSTD_blockHeaderSize (defined in zstd_internal.h)) * @return : size of data written into `dst` (== ZSTD_blockHeaderSize (defined in zstd_internal.h))
* or an error code if `dstCapacity` is too small (<ZSTD_blockHeaderSize) * or an error code if `dstCapacity` is too small (<ZSTD_blockHeaderSize)
*/ */
size_t ZSTD_writeLastEmptyBlock(void* dst, size_t dstCapacity); size_t ZSTD_writeLastEmptyBlock(void* dst, size_t dstCapacity);

contrib/python-zstandard/zstd/compress/zstd_compress_literals.h

size_t ZSTD_compressRleLiteralsBlock (void* dst, size_t dstCapacity, const void* src, size_t srcSize); size_t ZSTD_compressRleLiteralsBlock (void* dst, size_t dstCapacity, const void* src, size_t srcSize);
size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf, size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
ZSTD_hufCTables_t* nextHuf, ZSTD_hufCTables_t* nextHuf,
ZSTD_strategy strategy, int disableLiteralCompression, ZSTD_strategy strategy, int disableLiteralCompression,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize, const void* src, size_t srcSize,
void* workspace, size_t wkspSize, void* entropyWorkspace, size_t entropyWorkspaceSize,
const int bmi2); const int bmi2);
#endif /* ZSTD_COMPRESS_LITERALS_H */ #endif /* ZSTD_COMPRESS_LITERALS_H */

contrib/python-zstandard/zstd/compress/zstd_compress_literals.c

return flSize+1; return flSize+1;
} }
size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf, size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
ZSTD_hufCTables_t* nextHuf, ZSTD_hufCTables_t* nextHuf,
ZSTD_strategy strategy, int disableLiteralCompression, ZSTD_strategy strategy, int disableLiteralCompression,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize, const void* src, size_t srcSize,
void* workspace, size_t wkspSize, void* entropyWorkspace, size_t entropyWorkspaceSize,
const int bmi2) const int bmi2)
{ {
size_t const minGain = ZSTD_minGain(srcSize, strategy); size_t const minGain = ZSTD_minGain(srcSize, strategy);
size_t const lhSize = 3 + (srcSize >= 1 KB) + (srcSize >= 16 KB); size_t const lhSize = 3 + (srcSize >= 1 KB) + (srcSize >= 16 KB);
BYTE* const ostart = (BYTE*)dst; BYTE* const ostart = (BYTE*)dst;
U32 singleStream = srcSize < 256; U32 singleStream = srcSize < 256;
symbolEncodingType_e hType = set_compressed; symbolEncodingType_e hType = set_compressed;
size_t cLitSize; size_t cLitSize;
{ size_t const minLitSize = (prevHuf->repeatMode == HUF_repeat_valid) ? 6 : COMPRESS_LITERALS_SIZE_MIN; { size_t const minLitSize = (prevHuf->repeatMode == HUF_repeat_valid) ? 6 : COMPRESS_LITERALS_SIZE_MIN;
if (srcSize <= minLitSize) return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize); if (srcSize <= minLitSize) return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize);
} }
RETURN_ERROR_IF(dstCapacity < lhSize+1, dstSize_tooSmall, "not enough space for compression"); RETURN_ERROR_IF(dstCapacity < lhSize+1, dstSize_tooSmall, "not enough space for compression");
{ HUF_repeat repeat = prevHuf->repeatMode; { HUF_repeat repeat = prevHuf->repeatMode;
int const preferRepeat = strategy < ZSTD_lazy ? srcSize <= 1024 : 0; int const preferRepeat = strategy < ZSTD_lazy ? srcSize <= 1024 : 0;
if (repeat == HUF_repeat_valid && lhSize == 3) singleStream = 1; if (repeat == HUF_repeat_valid && lhSize == 3) singleStream = 1;
cLitSize = singleStream ? HUF_compress1X_repeat(ostart+lhSize, dstCapacity-lhSize, src, srcSize, 255, 11, cLitSize = singleStream ?
workspace, wkspSize, (HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2) HUF_compress1X_repeat(
: HUF_compress4X_repeat(ostart+lhSize, dstCapacity-lhSize, src, srcSize, 255, 11, ostart+lhSize, dstCapacity-lhSize, src, srcSize,
workspace, wkspSize, (HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2); 255, 11, entropyWorkspace, entropyWorkspaceSize,
(HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2) :
HUF_compress4X_repeat(
ostart+lhSize, dstCapacity-lhSize, src, srcSize,
255, 11, entropyWorkspace, entropyWorkspaceSize,
(HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2);
if (repeat != HUF_repeat_none) { if (repeat != HUF_repeat_none) {
/* reused the existing table */ /* reused the existing table */
hType = set_repeat; hType = set_repeat;
} }
} }
if ((cLitSize==0) | (cLitSize >= srcSize - minGain) | ERR_isError(cLitSize)) { if ((cLitSize==0) | (cLitSize >= srcSize - minGain) | ERR_isError(cLitSize)) {
memcpy(nextHuf, prevHuf, sizeof(*prevHuf)); memcpy(nextHuf, prevHuf, sizeof(*prevHuf));

contrib/python-zstandard/zstd/compress/zstd_compress_sequences.h

size_t size_t
ZSTD_buildCTable(void* dst, size_t dstCapacity, ZSTD_buildCTable(void* dst, size_t dstCapacity,
FSE_CTable* nextCTable, U32 FSELog, symbolEncodingType_e type, FSE_CTable* nextCTable, U32 FSELog, symbolEncodingType_e type,
unsigned* count, U32 max, unsigned* count, U32 max,
const BYTE* codeTable, size_t nbSeq, const BYTE* codeTable, size_t nbSeq,
const S16* defaultNorm, U32 defaultNormLog, U32 defaultMax, const S16* defaultNorm, U32 defaultNormLog, U32 defaultMax,
const FSE_CTable* prevCTable, size_t prevCTableSize, const FSE_CTable* prevCTable, size_t prevCTableSize,
void* workspace, size_t workspaceSize); void* entropyWorkspace, size_t entropyWorkspaceSize);
size_t ZSTD_encodeSequences( size_t ZSTD_encodeSequences(
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable, FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable,
FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable, FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable,
FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable, FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable,
seqDef const* sequences, size_t nbSeq, int longOffsets, int bmi2); seqDef const* sequences, size_t nbSeq, int longOffsets, int bmi2);
#endif /* ZSTD_COMPRESS_SEQUENCES_H */ #endif /* ZSTD_COMPRESS_SEQUENCES_H */

contrib/python-zstandard/zstd/compress/zstd_compress_sequences.c

size_t size_t
ZSTD_buildCTable(void* dst, size_t dstCapacity, ZSTD_buildCTable(void* dst, size_t dstCapacity,
FSE_CTable* nextCTable, U32 FSELog, symbolEncodingType_e type, FSE_CTable* nextCTable, U32 FSELog, symbolEncodingType_e type,
unsigned* count, U32 max, unsigned* count, U32 max,
const BYTE* codeTable, size_t nbSeq, const BYTE* codeTable, size_t nbSeq,
const S16* defaultNorm, U32 defaultNormLog, U32 defaultMax, const S16* defaultNorm, U32 defaultNormLog, U32 defaultMax,
const FSE_CTable* prevCTable, size_t prevCTableSize, const FSE_CTable* prevCTable, size_t prevCTableSize,
void* workspace, size_t workspaceSize) void* entropyWorkspace, size_t entropyWorkspaceSize)
{ {
BYTE* op = (BYTE*)dst; BYTE* op = (BYTE*)dst;
const BYTE* const oend = op + dstCapacity; const BYTE* const oend = op + dstCapacity;
DEBUGLOG(6, "ZSTD_buildCTable (dstCapacity=%u)", (unsigned)dstCapacity); DEBUGLOG(6, "ZSTD_buildCTable (dstCapacity=%u)", (unsigned)dstCapacity);
switch (type) { switch (type) {
case set_rle: case set_rle:
FORWARD_IF_ERROR(FSE_buildCTable_rle(nextCTable, (BYTE)max)); FORWARD_IF_ERROR(FSE_buildCTable_rle(nextCTable, (BYTE)max));
RETURN_ERROR_IF(dstCapacity==0, dstSize_tooSmall); RETURN_ERROR_IF(dstCapacity==0, dstSize_tooSmall);
*op = codeTable[0]; *op = codeTable[0];
return 1; return 1;
case set_repeat: case set_repeat:
memcpy(nextCTable, prevCTable, prevCTableSize); memcpy(nextCTable, prevCTable, prevCTableSize);
return 0; return 0;
case set_basic: case set_basic:
FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, defaultNorm, defaultMax, defaultNormLog, workspace, workspaceSize)); /* note : could be pre-calculated */ FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, defaultNorm, defaultMax, defaultNormLog, entropyWorkspace, entropyWorkspaceSize)); /* note : could be pre-calculated */
return 0; return 0;
case set_compressed: { case set_compressed: {
S16 norm[MaxSeq + 1]; S16 norm[MaxSeq + 1];
size_t nbSeq_1 = nbSeq; size_t nbSeq_1 = nbSeq;
const U32 tableLog = FSE_optimalTableLog(FSELog, nbSeq, max); const U32 tableLog = FSE_optimalTableLog(FSELog, nbSeq, max);
if (count[codeTable[nbSeq-1]] > 1) { if (count[codeTable[nbSeq-1]] > 1) {
count[codeTable[nbSeq-1]]--; count[codeTable[nbSeq-1]]--;
nbSeq_1--; nbSeq_1--;
} }
assert(nbSeq_1 > 1); assert(nbSeq_1 > 1);
FORWARD_IF_ERROR(FSE_normalizeCount(norm, tableLog, count, nbSeq_1, max)); FORWARD_IF_ERROR(FSE_normalizeCount(norm, tableLog, count, nbSeq_1, max));
{ size_t const NCountSize = FSE_writeNCount(op, oend - op, norm, max, tableLog); /* overflow protected */ { size_t const NCountSize = FSE_writeNCount(op, oend - op, norm, max, tableLog); /* overflow protected */
FORWARD_IF_ERROR(NCountSize); FORWARD_IF_ERROR(NCountSize);
FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, norm, max, tableLog, workspace, workspaceSize)); FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, norm, max, tableLog, entropyWorkspace, entropyWorkspaceSize));
return NCountSize; return NCountSize;
} }
} }
default: assert(0); RETURN_ERROR(GENERIC); default: assert(0); RETURN_ERROR(GENERIC);
} }
} }
FORCE_INLINE_TEMPLATE size_t FORCE_INLINE_TEMPLATE size_t

contrib/python-zstandard/zstd/compress/zstd_cwksp.h

  • This file was added.
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
#ifndef ZSTD_CWKSP_H
#define ZSTD_CWKSP_H
/*-*************************************
* Dependencies
***************************************/
#include "zstd_internal.h"
#if defined (__cplusplus)
extern "C" {
#endif
/*-*************************************
* Constants
***************************************/
/* define "workspace is too large" as this number of times larger than needed */
#define ZSTD_WORKSPACETOOLARGE_FACTOR 3
/* when workspace is continuously too large
* during at least this number of times,
* context's memory usage is considered wasteful,
* because it's sized to handle a worst case scenario which rarely happens.
* In which case, resize it down to free some memory */
#define ZSTD_WORKSPACETOOLARGE_MAXDURATION 128
/* Since the workspace is effectively its own little malloc implementation /
* arena, when we run under ASAN, we should similarly insert redzones between
* each internal element of the workspace, so ASAN will catch overruns that
* reach outside an object but that stay inside the workspace.
*
* This defines the size of that redzone.
*/
#ifndef ZSTD_CWKSP_ASAN_REDZONE_SIZE
#define ZSTD_CWKSP_ASAN_REDZONE_SIZE 128
#endif
/*-*************************************
* Structures
***************************************/
typedef enum {
ZSTD_cwksp_alloc_objects,
ZSTD_cwksp_alloc_buffers,
ZSTD_cwksp_alloc_aligned
} ZSTD_cwksp_alloc_phase_e;
/**
* Zstd fits all its internal datastructures into a single continuous buffer,
* so that it only needs to perform a single OS allocation (or so that a buffer
* can be provided to it and it can perform no allocations at all). This buffer
* is called the workspace.
*
* Several optimizations complicate that process of allocating memory ranges
* from this workspace for each internal datastructure:
*
* - These different internal datastructures have different setup requirements:
*
* - The static objects need to be cleared once and can then be trivially
* reused for each compression.
*
* - Various buffers don't need to be initialized at all--they are always
* written into before they're read.
*
* - The matchstate tables have a unique requirement that they don't need
* their memory to be totally cleared, but they do need the memory to have
* some bound, i.e., a guarantee that all values in the memory they've been
* allocated is less than some maximum value (which is the starting value
* for the indices that they will then use for compression). When this
* guarantee is provided to them, they can use the memory without any setup
* work. When it can't, they have to clear the area.
*
* - These buffers also have different alignment requirements.
*
* - We would like to reuse the objects in the workspace for multiple
* compressions without having to perform any expensive reallocation or
* reinitialization work.
*
* - We would like to be able to efficiently reuse the workspace across
* multiple compressions **even when the compression parameters change** and
* we need to resize some of the objects (where possible).
*
* To attempt to manage this buffer, given these constraints, the ZSTD_cwksp
* abstraction was created. It works as follows:
*
* Workspace Layout:
*
* [ ... workspace ... ]
* [objects][tables ... ->] free space [<- ... aligned][<- ... buffers]
*
* The various objects that live in the workspace are divided into the
* following categories, and are allocated separately:
*
* - Static objects: this is optionally the enclosing ZSTD_CCtx or ZSTD_CDict,
* so that literally everything fits in a single buffer. Note: if present,
* this must be the first object in the workspace, since ZSTD_free{CCtx,
* CDict}() rely on a pointer comparison to see whether one or two frees are
* required.
*
* - Fixed size objects: these are fixed-size, fixed-count objects that are
* nonetheless "dynamically" allocated in the workspace so that we can
* control how they're initialized separately from the broader ZSTD_CCtx.
* Examples:
* - Entropy Workspace
* - 2 x ZSTD_compressedBlockState_t
* - CDict dictionary contents
*
* - Tables: these are any of several different datastructures (hash tables,
* chain tables, binary trees) that all respect a common format: they are
* uint32_t arrays, all of whose values are between 0 and (nextSrc - base).
* Their sizes depend on the cparams.
*
* - Aligned: these buffers are used for various purposes that require 4 byte
* alignment, but don't require any initialization before they're used.
*
* - Buffers: these buffers are used for various purposes that don't require
* any alignment or initialization before they're used. This means they can
* be moved around at no cost for a new compression.
*
* Allocating Memory:
*
* The various types of objects must be allocated in order, so they can be
* correctly packed into the workspace buffer. That order is:
*
* 1. Objects
* 2. Buffers
* 3. Aligned
* 4. Tables
*
* Attempts to reserve objects of different types out of order will fail.
*/
typedef struct {
void* workspace;
void* workspaceEnd;
void* objectEnd;
void* tableEnd;
void* tableValidEnd;
void* allocStart;
int allocFailed;
int workspaceOversizedDuration;
ZSTD_cwksp_alloc_phase_e phase;
} ZSTD_cwksp;
/*-*************************************
* Functions
***************************************/
MEM_STATIC size_t ZSTD_cwksp_available_space(ZSTD_cwksp* ws);
MEM_STATIC void ZSTD_cwksp_assert_internal_consistency(ZSTD_cwksp* ws) {
(void)ws;
assert(ws->workspace <= ws->objectEnd);
assert(ws->objectEnd <= ws->tableEnd);
assert(ws->objectEnd <= ws->tableValidEnd);
assert(ws->tableEnd <= ws->allocStart);
assert(ws->tableValidEnd <= ws->allocStart);
assert(ws->allocStart <= ws->workspaceEnd);
}
/**
* Align must be a power of 2.
*/
MEM_STATIC size_t ZSTD_cwksp_align(size_t size, size_t const align) {
size_t const mask = align - 1;
assert((align & mask) == 0);
return (size + mask) & ~mask;
}
/**
* Use this to determine how much space in the workspace we will consume to
* allocate this object. (Normally it should be exactly the size of the object,
* but under special conditions, like ASAN, where we pad each object, it might
* be larger.)
*
* Since tables aren't currently redzoned, you don't need to call through this
* to figure out how much space you need for the matchState tables. Everything
* else is though.
*/
MEM_STATIC size_t ZSTD_cwksp_alloc_size(size_t size) {
#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
return size + 2 * ZSTD_CWKSP_ASAN_REDZONE_SIZE;
#else
return size;
#endif
}
MEM_STATIC void ZSTD_cwksp_internal_advance_phase(
ZSTD_cwksp* ws, ZSTD_cwksp_alloc_phase_e phase) {
assert(phase >= ws->phase);
if (phase > ws->phase) {
if (ws->phase < ZSTD_cwksp_alloc_buffers &&
phase >= ZSTD_cwksp_alloc_buffers) {
ws->tableValidEnd = ws->objectEnd;
}
if (ws->phase < ZSTD_cwksp_alloc_aligned &&
phase >= ZSTD_cwksp_alloc_aligned) {
/* If unaligned allocations down from a too-large top have left us
* unaligned, we need to realign our alloc ptr. Technically, this
* can consume space that is unaccounted for in the neededSpace
* calculation. However, I believe this can only happen when the
* workspace is too large, and specifically when it is too large
* by a larger margin than the space that will be consumed. */
/* TODO: cleaner, compiler warning friendly way to do this??? */
ws->allocStart = (BYTE*)ws->allocStart - ((size_t)ws->allocStart & (sizeof(U32)-1));
if (ws->allocStart < ws->tableValidEnd) {
ws->tableValidEnd = ws->allocStart;
}
}
ws->phase = phase;
}
}
/**
* Returns whether this object/buffer/etc was allocated in this workspace.
*/
MEM_STATIC int ZSTD_cwksp_owns_buffer(const ZSTD_cwksp* ws, const void* ptr) {
return (ptr != NULL) && (ws->workspace <= ptr) && (ptr <= ws->workspaceEnd);
}
/**
* Internal function. Do not use directly.
*/
MEM_STATIC void* ZSTD_cwksp_reserve_internal(
ZSTD_cwksp* ws, size_t bytes, ZSTD_cwksp_alloc_phase_e phase) {
void* alloc;
void* bottom = ws->tableEnd;
ZSTD_cwksp_internal_advance_phase(ws, phase);
alloc = (BYTE *)ws->allocStart - bytes;
#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
/* over-reserve space */
alloc = (BYTE *)alloc - 2 * ZSTD_CWKSP_ASAN_REDZONE_SIZE;
#endif
DEBUGLOG(5, "cwksp: reserving %p %zd bytes, %zd bytes remaining",
alloc, bytes, ZSTD_cwksp_available_space(ws) - bytes);
ZSTD_cwksp_assert_internal_consistency(ws);
assert(alloc >= bottom);
if (alloc < bottom) {
DEBUGLOG(4, "cwksp: alloc failed!");
ws->allocFailed = 1;
return NULL;
}
if (alloc < ws->tableValidEnd) {
ws->tableValidEnd = alloc;
}
ws->allocStart = alloc;
#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
/* Move alloc so there's ZSTD_CWKSP_ASAN_REDZONE_SIZE unused space on
* either size. */
alloc = (BYTE *)alloc + ZSTD_CWKSP_ASAN_REDZONE_SIZE;
__asan_unpoison_memory_region(alloc, bytes);
#endif
return alloc;
}
/**
* Reserves and returns unaligned memory.
*/
MEM_STATIC BYTE* ZSTD_cwksp_reserve_buffer(ZSTD_cwksp* ws, size_t bytes) {
return (BYTE*)ZSTD_cwksp_reserve_internal(ws, bytes, ZSTD_cwksp_alloc_buffers);
}
/**
* Reserves and returns memory sized on and aligned on sizeof(unsigned).
*/
MEM_STATIC void* ZSTD_cwksp_reserve_aligned(ZSTD_cwksp* ws, size_t bytes) {
assert((bytes & (sizeof(U32)-1)) == 0);
return ZSTD_cwksp_reserve_internal(ws, ZSTD_cwksp_align(bytes, sizeof(U32)), ZSTD_cwksp_alloc_aligned);
}
/**
* Aligned on sizeof(unsigned). These buffers have the special property that
* their values remain constrained, allowing us to re-use them without
* memset()-ing them.
*/
MEM_STATIC void* ZSTD_cwksp_reserve_table(ZSTD_cwksp* ws, size_t bytes) {
const ZSTD_cwksp_alloc_phase_e phase = ZSTD_cwksp_alloc_aligned;
void* alloc = ws->tableEnd;
void* end = (BYTE *)alloc + bytes;
void* top = ws->allocStart;
DEBUGLOG(5, "cwksp: reserving %p table %zd bytes, %zd bytes remaining",
alloc, bytes, ZSTD_cwksp_available_space(ws) - bytes);
assert((bytes & (sizeof(U32)-1)) == 0);
ZSTD_cwksp_internal_advance_phase(ws, phase);
ZSTD_cwksp_assert_internal_consistency(ws);
assert(end <= top);
if (end > top) {
DEBUGLOG(4, "cwksp: table alloc failed!");
ws->allocFailed = 1;
return NULL;
}
ws->tableEnd = end;
#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
__asan_unpoison_memory_region(alloc, bytes);
#endif
return alloc;
}
/**
* Aligned on sizeof(void*).
*/
MEM_STATIC void* ZSTD_cwksp_reserve_object(ZSTD_cwksp* ws, size_t bytes) {
size_t roundedBytes = ZSTD_cwksp_align(bytes, sizeof(void*));
void* alloc = ws->objectEnd;
void* end = (BYTE*)alloc + roundedBytes;
#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
/* over-reserve space */
end = (BYTE *)end + 2 * ZSTD_CWKSP_ASAN_REDZONE_SIZE;
#endif
DEBUGLOG(5,
"cwksp: reserving %p object %zd bytes (rounded to %zd), %zd bytes remaining",
alloc, bytes, roundedBytes, ZSTD_cwksp_available_space(ws) - roundedBytes);
assert(((size_t)alloc & (sizeof(void*)-1)) == 0);
assert((bytes & (sizeof(void*)-1)) == 0);
ZSTD_cwksp_assert_internal_consistency(ws);
/* we must be in the first phase, no advance is possible */
if (ws->phase != ZSTD_cwksp_alloc_objects || end > ws->workspaceEnd) {
DEBUGLOG(4, "cwksp: object alloc failed!");
ws->allocFailed = 1;
return NULL;
}
ws->objectEnd = end;
ws->tableEnd = end;
ws->tableValidEnd = end;
#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
/* Move alloc so there's ZSTD_CWKSP_ASAN_REDZONE_SIZE unused space on
* either size. */
alloc = (BYTE *)alloc + ZSTD_CWKSP_ASAN_REDZONE_SIZE;
__asan_unpoison_memory_region(alloc, bytes);
#endif
return alloc;
}
MEM_STATIC void ZSTD_cwksp_mark_tables_dirty(ZSTD_cwksp* ws) {
DEBUGLOG(4, "cwksp: ZSTD_cwksp_mark_tables_dirty");
#if defined (MEMORY_SANITIZER) && !defined (ZSTD_MSAN_DONT_POISON_WORKSPACE)
/* To validate that the table re-use logic is sound, and that we don't
* access table space that we haven't cleaned, we re-"poison" the table
* space every time we mark it dirty. */
{
size_t size = (BYTE*)ws->tableValidEnd - (BYTE*)ws->objectEnd;
assert(__msan_test_shadow(ws->objectEnd, size) == -1);
__msan_poison(ws->objectEnd, size);
}
#endif
assert(ws->tableValidEnd >= ws->objectEnd);
assert(ws->tableValidEnd <= ws->allocStart);
ws->tableValidEnd = ws->objectEnd;
ZSTD_cwksp_assert_internal_consistency(ws);
}
MEM_STATIC void ZSTD_cwksp_mark_tables_clean(ZSTD_cwksp* ws) {
DEBUGLOG(4, "cwksp: ZSTD_cwksp_mark_tables_clean");
assert(ws->tableValidEnd >= ws->objectEnd);
assert(ws->tableValidEnd <= ws->allocStart);
if (ws->tableValidEnd < ws->tableEnd) {
ws->tableValidEnd = ws->tableEnd;
}
ZSTD_cwksp_assert_internal_consistency(ws);
}
/**
* Zero the part of the allocated tables not already marked clean.
*/
MEM_STATIC void ZSTD_cwksp_clean_tables(ZSTD_cwksp* ws) {
DEBUGLOG(4, "cwksp: ZSTD_cwksp_clean_tables");
assert(ws->tableValidEnd >= ws->objectEnd);
assert(ws->tableValidEnd <= ws->allocStart);
if (ws->tableValidEnd < ws->tableEnd) {
memset(ws->tableValidEnd, 0, (BYTE*)ws->tableEnd - (BYTE*)ws->tableValidEnd);
}
ZSTD_cwksp_mark_tables_clean(ws);
}
/**
* Invalidates table allocations.
* All other allocations remain valid.
*/
MEM_STATIC void ZSTD_cwksp_clear_tables(ZSTD_cwksp* ws) {
DEBUGLOG(4, "cwksp: clearing tables!");
#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
{
size_t size = (BYTE*)ws->tableValidEnd - (BYTE*)ws->objectEnd;
__asan_poison_memory_region(ws->objectEnd, size);
}
#endif
ws->tableEnd = ws->objectEnd;
ZSTD_cwksp_assert_internal_consistency(ws);
}
/**
* Invalidates all buffer, aligned, and table allocations.
* Object allocations remain valid.
*/
MEM_STATIC void ZSTD_cwksp_clear(ZSTD_cwksp* ws) {
DEBUGLOG(4, "cwksp: clearing!");
#if defined (MEMORY_SANITIZER) && !defined (ZSTD_MSAN_DONT_POISON_WORKSPACE)
/* To validate that the context re-use logic is sound, and that we don't
* access stuff that this compression hasn't initialized, we re-"poison"
* the workspace (or at least the non-static, non-table parts of it)
* every time we start a new compression. */
{
size_t size = (BYTE*)ws->workspaceEnd - (BYTE*)ws->tableValidEnd;
__msan_poison(ws->tableValidEnd, size);
}
#endif
#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
{
size_t size = (BYTE*)ws->workspaceEnd - (BYTE*)ws->objectEnd;
__asan_poison_memory_region(ws->objectEnd, size);
}
#endif
ws->tableEnd = ws->objectEnd;
ws->allocStart = ws->workspaceEnd;
ws->allocFailed = 0;
if (ws->phase > ZSTD_cwksp_alloc_buffers) {
ws->phase = ZSTD_cwksp_alloc_buffers;
}
ZSTD_cwksp_assert_internal_consistency(ws);
}
/**
* The provided workspace takes ownership of the buffer [start, start+size).
* Any existing values in the workspace are ignored (the previously managed
* buffer, if present, must be separately freed).
*/
MEM_STATIC void ZSTD_cwksp_init(ZSTD_cwksp* ws, void* start, size_t size) {
DEBUGLOG(4, "cwksp: init'ing workspace with %zd bytes", size);
assert(((size_t)start & (sizeof(void*)-1)) == 0); /* ensure correct alignment */
ws->workspace = start;
ws->workspaceEnd = (BYTE*)start + size;
ws->objectEnd = ws->workspace;
ws->tableValidEnd = ws->objectEnd;
ws->phase = ZSTD_cwksp_alloc_objects;
ZSTD_cwksp_clear(ws);
ws->workspaceOversizedDuration = 0;
ZSTD_cwksp_assert_internal_consistency(ws);
}
MEM_STATIC size_t ZSTD_cwksp_create(ZSTD_cwksp* ws, size_t size, ZSTD_customMem customMem) {
void* workspace = ZSTD_malloc(size, customMem);
DEBUGLOG(4, "cwksp: creating new workspace with %zd bytes", size);
RETURN_ERROR_IF(workspace == NULL, memory_allocation);
ZSTD_cwksp_init(ws, workspace, size);
return 0;
}
MEM_STATIC void ZSTD_cwksp_free(ZSTD_cwksp* ws, ZSTD_customMem customMem) {
void *ptr = ws->workspace;
DEBUGLOG(4, "cwksp: freeing workspace");
memset(ws, 0, sizeof(ZSTD_cwksp));
ZSTD_free(ptr, customMem);
}
/**
* Moves the management of a workspace from one cwksp to another. The src cwksp
* is left in an invalid state (src must be re-init()'ed before its used again).
*/
MEM_STATIC void ZSTD_cwksp_move(ZSTD_cwksp* dst, ZSTD_cwksp* src) {
*dst = *src;
memset(src, 0, sizeof(ZSTD_cwksp));
}
MEM_STATIC size_t ZSTD_cwksp_sizeof(const ZSTD_cwksp* ws) {
return (size_t)((BYTE*)ws->workspaceEnd - (BYTE*)ws->workspace);
}
MEM_STATIC int ZSTD_cwksp_reserve_failed(const ZSTD_cwksp* ws) {
return ws->allocFailed;
}
/*-*************************************
* Functions Checking Free Space
***************************************/
MEM_STATIC size_t ZSTD_cwksp_available_space(ZSTD_cwksp* ws) {
return (size_t)((BYTE*)ws->allocStart - (BYTE*)ws->tableEnd);
}
MEM_STATIC int ZSTD_cwksp_check_available(ZSTD_cwksp* ws, size_t additionalNeededSpace) {
return ZSTD_cwksp_available_space(ws) >= additionalNeededSpace;
}
MEM_STATIC int ZSTD_cwksp_check_too_large(ZSTD_cwksp* ws, size_t additionalNeededSpace) {
return ZSTD_cwksp_check_available(
ws, additionalNeededSpace * ZSTD_WORKSPACETOOLARGE_FACTOR);
}
MEM_STATIC int ZSTD_cwksp_check_wasteful(ZSTD_cwksp* ws, size_t additionalNeededSpace) {
return ZSTD_cwksp_check_too_large(ws, additionalNeededSpace)
&& ws->workspaceOversizedDuration > ZSTD_WORKSPACETOOLARGE_MAXDURATION;
}
MEM_STATIC void ZSTD_cwksp_bump_oversized_duration(
ZSTD_cwksp* ws, size_t additionalNeededSpace) {
if (ZSTD_cwksp_check_too_large(ws, additionalNeededSpace)) {
ws->workspaceOversizedDuration++;
} else {
ws->workspaceOversizedDuration = 0;
}
}
#if defined (__cplusplus)
}
#endif
#endif /* ZSTD_CWKSP_H */

contrib/python-zstandard/zstd/compress/zstd_double_fast.c

/* check dictMatchState repcode */ /* check dictMatchState repcode */
if (dictMode == ZSTD_dictMatchState if (dictMode == ZSTD_dictMatchState
&& ((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */) && ((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */)
&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) { && (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend; const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend;
mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4; mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4;
ip++; ip++;
ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, 0, mLength-MINMATCH); ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
goto _match_stored; goto _match_stored;
} }
/* check noDict repcode */ /* check noDict repcode */
if ( dictMode == ZSTD_noDict if ( dictMode == ZSTD_noDict
&& ((offset_1 > 0) & (MEM_read32(ip+1-offset_1) == MEM_read32(ip+1)))) { && ((offset_1 > 0) & (MEM_read32(ip+1-offset_1) == MEM_read32(ip+1)))) {
mLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4; mLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4;
ip++; ip++;
ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, 0, mLength-MINMATCH); ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
goto _match_stored; goto _match_stored;
} }
if (matchIndexL > prefixLowestIndex) { if (matchIndexL > prefixLowestIndex) {
/* check prefix long match */ /* check prefix long match */
if (MEM_read64(matchLong) == MEM_read64(ip)) { if (MEM_read64(matchLong) == MEM_read64(ip)) {
mLength = ZSTD_count(ip+8, matchLong+8, iend) + 8; mLength = ZSTD_count(ip+8, matchLong+8, iend) + 8;
offset = (U32)(ip-matchLong); offset = (U32)(ip-matchLong);
} }
/* fall-through */ /* fall-through */
_match_found: _match_found:
offset_2 = offset_1; offset_2 = offset_1;
offset_1 = offset; offset_1 = offset;
ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, offset + ZSTD_REP_MOVE, mLength-MINMATCH); ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
_match_stored: _match_stored:
/* match found */ /* match found */
ip += mLength; ip += mLength;
anchor = ip; anchor = ip;
if (ip <= ilimit) { if (ip <= ilimit) {
/* Complementary insertion */ /* Complementary insertion */
&& repIndex2 < prefixLowestIndex ? && repIndex2 < prefixLowestIndex ?
dictBase - dictIndexDelta + repIndex2 : dictBase - dictIndexDelta + repIndex2 :
base + repIndex2; base + repIndex2;
if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */) if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */)
&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) { && (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
const BYTE* const repEnd2 = repIndex2 < prefixLowestIndex ? dictEnd : iend; const BYTE* const repEnd2 = repIndex2 < prefixLowestIndex ? dictEnd : iend;
size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixLowest) + 4; size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixLowest) + 4;
U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */ U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
ZSTD_storeSeq(seqStore, 0, anchor, 0, repLength2-MINMATCH); ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, repLength2-MINMATCH);
hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2; hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2;
hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2; hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2;
ip += repLength2; ip += repLength2;
anchor = ip; anchor = ip;
continue; continue;
} }
break; break;
} } } }
if (dictMode == ZSTD_noDict) { if (dictMode == ZSTD_noDict) {
while ( (ip <= ilimit) while ( (ip <= ilimit)
&& ( (offset_2>0) && ( (offset_2>0)
& (MEM_read32(ip) == MEM_read32(ip - offset_2)) )) { & (MEM_read32(ip) == MEM_read32(ip - offset_2)) )) {
/* store sequence */ /* store sequence */
size_t const rLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4; size_t const rLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4;
U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; /* swap offset_2 <=> offset_1 */ U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; /* swap offset_2 <=> offset_1 */
hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = (U32)(ip-base); hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = (U32)(ip-base);
hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = (U32)(ip-base); hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = (U32)(ip-base);
ZSTD_storeSeq(seqStore, 0, anchor, 0, rLength-MINMATCH); ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, rLength-MINMATCH);
ip += rLength; ip += rLength;
anchor = ip; anchor = ip;
continue; /* faster when present ... (?) */ continue; /* faster when present ... (?) */
} } } } } }
} /* while (ip < ilimit) */ } /* while (ip < ilimit) */
/* save reps for next block */ /* save reps for next block */
rep[0] = offset_1 ? offset_1 : offsetSaved; rep[0] = offset_1 ? offset_1 : offsetSaved;
hashSmall[hSmall] = hashLong[hLong] = current; /* update hash table */ hashSmall[hSmall] = hashLong[hLong] = current; /* update hash table */
if ((((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow : ensure repIndex doesn't overlap dict + prefix */ if ((((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow : ensure repIndex doesn't overlap dict + prefix */
& (repIndex > dictStartIndex)) & (repIndex > dictStartIndex))
&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) { && (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
const BYTE* repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend; const BYTE* repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend;
mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixStart) + 4; mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixStart) + 4;
ip++; ip++;
ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, 0, mLength-MINMATCH); ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
} else { } else {
if ((matchLongIndex > dictStartIndex) && (MEM_read64(matchLong) == MEM_read64(ip))) { if ((matchLongIndex > dictStartIndex) && (MEM_read64(matchLong) == MEM_read64(ip))) {
const BYTE* const matchEnd = matchLongIndex < prefixStartIndex ? dictEnd : iend; const BYTE* const matchEnd = matchLongIndex < prefixStartIndex ? dictEnd : iend;
const BYTE* const lowMatchPtr = matchLongIndex < prefixStartIndex ? dictStart : prefixStart; const BYTE* const lowMatchPtr = matchLongIndex < prefixStartIndex ? dictStart : prefixStart;
U32 offset; U32 offset;
mLength = ZSTD_count_2segments(ip+8, matchLong+8, iend, matchEnd, prefixStart) + 8; mLength = ZSTD_count_2segments(ip+8, matchLong+8, iend, matchEnd, prefixStart) + 8;
offset = current - matchLongIndex; offset = current - matchLongIndex;
while (((ip>anchor) & (matchLong>lowMatchPtr)) && (ip[-1] == matchLong[-1])) { ip--; matchLong--; mLength++; } /* catch up */ while (((ip>anchor) & (matchLong>lowMatchPtr)) && (ip[-1] == matchLong[-1])) { ip--; matchLong--; mLength++; } /* catch up */
offset_2 = offset_1; offset_2 = offset_1;
offset_1 = offset; offset_1 = offset;
ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, offset + ZSTD_REP_MOVE, mLength-MINMATCH); ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
} else if ((matchIndex > dictStartIndex) && (MEM_read32(match) == MEM_read32(ip))) { } else if ((matchIndex > dictStartIndex) && (MEM_read32(match) == MEM_read32(ip))) {
size_t const h3 = ZSTD_hashPtr(ip+1, hBitsL, 8); size_t const h3 = ZSTD_hashPtr(ip+1, hBitsL, 8);
U32 const matchIndex3 = hashLong[h3]; U32 const matchIndex3 = hashLong[h3];
const BYTE* const match3Base = matchIndex3 < prefixStartIndex ? dictBase : base; const BYTE* const match3Base = matchIndex3 < prefixStartIndex ? dictBase : base;
const BYTE* match3 = match3Base + matchIndex3; const BYTE* match3 = match3Base + matchIndex3;
U32 offset; U32 offset;
hashLong[h3] = current + 1; hashLong[h3] = current + 1;
if ( (matchIndex3 > dictStartIndex) && (MEM_read64(match3) == MEM_read64(ip+1)) ) { if ( (matchIndex3 > dictStartIndex) && (MEM_read64(match3) == MEM_read64(ip+1)) ) {
const BYTE* const matchEnd = matchIndex3 < prefixStartIndex ? dictEnd : iend; const BYTE* const matchEnd = matchIndex3 < prefixStartIndex ? dictEnd : iend;
const BYTE* const lowMatchPtr = matchIndex3 < prefixStartIndex ? dictStart : prefixStart; const BYTE* const lowMatchPtr = matchIndex3 < prefixStartIndex ? dictStart : prefixStart;
mLength = ZSTD_count_2segments(ip+9, match3+8, iend, matchEnd, prefixStart) + 8; mLength = ZSTD_count_2segments(ip+9, match3+8, iend, matchEnd, prefixStart) + 8;
ip++; ip++;
offset = current+1 - matchIndex3; offset = current+1 - matchIndex3;
while (((ip>anchor) & (match3>lowMatchPtr)) && (ip[-1] == match3[-1])) { ip--; match3--; mLength++; } /* catch up */ while (((ip>anchor) & (match3>lowMatchPtr)) && (ip[-1] == match3[-1])) { ip--; match3--; mLength++; } /* catch up */
} else { } else {
const BYTE* const matchEnd = matchIndex < prefixStartIndex ? dictEnd : iend; const BYTE* const matchEnd = matchIndex < prefixStartIndex ? dictEnd : iend;
const BYTE* const lowMatchPtr = matchIndex < prefixStartIndex ? dictStart : prefixStart; const BYTE* const lowMatchPtr = matchIndex < prefixStartIndex ? dictStart : prefixStart;
mLength = ZSTD_count_2segments(ip+4, match+4, iend, matchEnd, prefixStart) + 4; mLength = ZSTD_count_2segments(ip+4, match+4, iend, matchEnd, prefixStart) + 4;
offset = current - matchIndex; offset = current - matchIndex;
while (((ip>anchor) & (match>lowMatchPtr)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */ while (((ip>anchor) & (match>lowMatchPtr)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
} }
offset_2 = offset_1; offset_2 = offset_1;
offset_1 = offset; offset_1 = offset;
ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, offset + ZSTD_REP_MOVE, mLength-MINMATCH); ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
} else { } else {
ip += ((ip-anchor) >> kSearchStrength) + 1; ip += ((ip-anchor) >> kSearchStrength) + 1;
continue; continue;
} } } }
/* move to next sequence start */ /* move to next sequence start */
ip += mLength; ip += mLength;
U32 const repIndex2 = current2 - offset_2; U32 const repIndex2 = current2 - offset_2;
const BYTE* repMatch2 = repIndex2 < prefixStartIndex ? dictBase + repIndex2 : base + repIndex2; const BYTE* repMatch2 = repIndex2 < prefixStartIndex ? dictBase + repIndex2 : base + repIndex2;
if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) /* intentional overflow : ensure repIndex2 doesn't overlap dict + prefix */ if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) /* intentional overflow : ensure repIndex2 doesn't overlap dict + prefix */
& (repIndex2 > dictStartIndex)) & (repIndex2 > dictStartIndex))
&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) { && (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend; const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend;
size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4; size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4;
U32 const tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */ U32 const tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
ZSTD_storeSeq(seqStore, 0, anchor, 0, repLength2-MINMATCH); ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, repLength2-MINMATCH);
hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2; hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2;
hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2; hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2;
ip += repLength2; ip += repLength2;
anchor = ip; anchor = ip;
continue; continue;
} }
break; break;
} } } } } }

contrib/python-zstandard/zstd/compress/zstd_fast.c

/* /*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc. * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved. * All rights reserved.
* *
* This source code is licensed under both the BSD-style license (found in the * This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found * LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree). * in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses. * You may select, at your option, one of the above-listed licenses.
*/ */
#include "zstd_compress_internal.h" #include "zstd_compress_internal.h" /* ZSTD_hashPtr, ZSTD_count, ZSTD_storeSeq */
#include "zstd_fast.h" #include "zstd_fast.h"
void ZSTD_fillHashTable(ZSTD_matchState_t* ms, void ZSTD_fillHashTable(ZSTD_matchState_t* ms,
const void* const end, const void* const end,
ZSTD_dictTableLoadMethod_e dtlm) ZSTD_dictTableLoadMethod_e dtlm)
{ {
const ZSTD_compressionParameters* const cParams = &ms->cParams; const ZSTD_compressionParameters* const cParams = &ms->cParams;
for (p = 1; p < fastHashFillStep; ++p) { for (p = 1; p < fastHashFillStep; ++p) {
size_t const hash = ZSTD_hashPtr(ip + p, hBits, mls); size_t const hash = ZSTD_hashPtr(ip + p, hBits, mls);
if (hashTable[hash] == 0) { /* not yet filled */ if (hashTable[hash] == 0) { /* not yet filled */
hashTable[hash] = current + p; hashTable[hash] = current + p;
} } } } } } } }
} }
FORCE_INLINE_TEMPLATE FORCE_INLINE_TEMPLATE size_t
size_t ZSTD_compressBlock_fast_generic( ZSTD_compressBlock_fast_generic(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize, void const* src, size_t srcSize,
U32 const mls) U32 const mls)
{ {
const ZSTD_compressionParameters* const cParams = &ms->cParams; const ZSTD_compressionParameters* const cParams = &ms->cParams;
U32* const hashTable = ms->hashTable; U32* const hashTable = ms->hashTable;
U32 const hlog = cParams->hashLog; U32 const hlog = cParams->hashLog;
/* support stepSize of 0 */ /* support stepSize of 0 */
const BYTE* const ilimit = iend - HASH_READ_SIZE; const BYTE* const ilimit = iend - HASH_READ_SIZE;
U32 offset_1=rep[0], offset_2=rep[1]; U32 offset_1=rep[0], offset_2=rep[1];
U32 offsetSaved = 0; U32 offsetSaved = 0;
/* init */ /* init */
DEBUGLOG(5, "ZSTD_compressBlock_fast_generic"); DEBUGLOG(5, "ZSTD_compressBlock_fast_generic");
ip0 += (ip0 == prefixStart); ip0 += (ip0 == prefixStart);
ip1 = ip0 + 1; ip1 = ip0 + 1;
{ { U32 const maxRep = (U32)(ip0 - prefixStart);
U32 const maxRep = (U32)(ip0 - prefixStart);
if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0; if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0;
if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0; if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0;
} }
/* Main Search Loop */ /* Main Search Loop */
while (ip1 < ilimit) { /* < instead of <=, because check at ip0+2 */ while (ip1 < ilimit) { /* < instead of <=, because check at ip0+2 */
size_t mLength; size_t mLength;
BYTE const* ip2 = ip0 + 2; BYTE const* ip2 = ip0 + 2;
goto _offset; goto _offset;
} }
if ((matchIndex1 > prefixStartIndex) && MEM_read32(match1) == val1) { if ((matchIndex1 > prefixStartIndex) && MEM_read32(match1) == val1) {
/* found a regular match after one literal */ /* found a regular match after one literal */
ip0 = ip1; ip0 = ip1;
match0 = match1; match0 = match1;
goto _offset; goto _offset;
} }
{ { size_t const step = ((size_t)(ip0-anchor) >> (kSearchStrength - 1)) + stepSize;
size_t const step = ((ip0-anchor) >> (kSearchStrength - 1)) + stepSize;
assert(step >= 2); assert(step >= 2);
ip0 += step; ip0 += step;
ip1 += step; ip1 += step;
continue; continue;
} }
_offset: /* Requires: ip0, match0 */ _offset: /* Requires: ip0, match0 */
/* Compute the offset code */ /* Compute the offset code */
offset_2 = offset_1; offset_2 = offset_1;
offset_1 = (U32)(ip0-match0); offset_1 = (U32)(ip0-match0);
offcode = offset_1 + ZSTD_REP_MOVE; offcode = offset_1 + ZSTD_REP_MOVE;
mLength = 0; mLength = 0;
/* Count the backwards match length */ /* Count the backwards match length */
while (((ip0>anchor) & (match0>prefixStart)) while (((ip0>anchor) & (match0>prefixStart))
&& (ip0[-1] == match0[-1])) { ip0--; match0--; mLength++; } /* catch up */ && (ip0[-1] == match0[-1])) { ip0--; match0--; mLength++; } /* catch up */
_match: /* Requires: ip0, match0, offcode */ _match: /* Requires: ip0, match0, offcode */
/* Count the forward length */ /* Count the forward length */
mLength += ZSTD_count(ip0+mLength+4, match0+mLength+4, iend) + 4; mLength += ZSTD_count(ip0+mLength+4, match0+mLength+4, iend) + 4;
ZSTD_storeSeq(seqStore, ip0-anchor, anchor, offcode, mLength-MINMATCH); ZSTD_storeSeq(seqStore, (size_t)(ip0-anchor), anchor, iend, offcode, mLength-MINMATCH);
/* match found */ /* match found */
ip0 += mLength; ip0 += mLength;
anchor = ip0; anchor = ip0;
ip1 = ip0 + 1; ip1 = ip0 + 1;
if (ip0 <= ilimit) { if (ip0 <= ilimit) {
/* Fill Table */ /* Fill Table */
assert(base+current0+2 > istart); /* check base overflow */ assert(base+current0+2 > istart); /* check base overflow */
hashTable[ZSTD_hashPtr(base+current0+2, hlog, mls)] = current0+2; /* here because current+2 could be > iend-8 */ hashTable[ZSTD_hashPtr(base+current0+2, hlog, mls)] = current0+2; /* here because current+2 could be > iend-8 */
hashTable[ZSTD_hashPtr(ip0-2, hlog, mls)] = (U32)(ip0-2-base); hashTable[ZSTD_hashPtr(ip0-2, hlog, mls)] = (U32)(ip0-2-base);
while ( (ip0 <= ilimit) while ( ((ip0 <= ilimit) & (offset_2>0)) /* offset_2==0 means offset_2 is invalidated */
&& ( (offset_2>0) && (MEM_read32(ip0) == MEM_read32(ip0 - offset_2)) ) {
& (MEM_read32(ip0) == MEM_read32(ip0 - offset_2)) )) {
/* store sequence */ /* store sequence */
size_t const rLength = ZSTD_count(ip0+4, ip0+4-offset_2, iend) + 4; size_t const rLength = ZSTD_count(ip0+4, ip0+4-offset_2, iend) + 4;
U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; /* swap offset_2 <=> offset_1 */ { U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; } /* swap offset_2 <=> offset_1 */
hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = (U32)(ip0-base); hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = (U32)(ip0-base);
ip0 += rLength; ip0 += rLength;
ip1 = ip0 + 1; ip1 = ip0 + 1;
ZSTD_storeSeq(seqStore, 0, anchor, 0, rLength-MINMATCH); ZSTD_storeSeq(seqStore, 0 /*litLen*/, anchor, iend, 0 /*offCode*/, rLength-MINMATCH);
anchor = ip0; anchor = ip0;
continue; /* faster when present (confirmed on gcc-8) ... (?) */ continue; /* faster when present (confirmed on gcc-8) ... (?) */
} }
} }
} }
/* save reps for next block */ /* save reps for next block */
rep[0] = offset_1 ? offset_1 : offsetSaved; rep[0] = offset_1 ? offset_1 : offsetSaved;
rep[1] = offset_2 ? offset_2 : offsetSaved; rep[1] = offset_2 ? offset_2 : offsetSaved;
/* Return the last literals size */ /* Return the last literals size */
return (size_t)(iend - anchor); return (size_t)(iend - anchor);
} }
size_t ZSTD_compressBlock_fast( size_t ZSTD_compressBlock_fast(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize) void const* src, size_t srcSize)
{ {
ZSTD_compressionParameters const* cParams = &ms->cParams; U32 const mls = ms->cParams.minMatch;
U32 const mls = cParams->minMatch;
assert(ms->dictMatchState == NULL); assert(ms->dictMatchState == NULL);
switch(mls) switch(mls)
{ {
default: /* includes case 3 */ default: /* includes case 3 */
case 4 : case 4 :
return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 4); return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 4);
case 5 : case 5 :
return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 5); return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 5);
base + repIndex; base + repIndex;
hashTable[h] = current; /* update hash table */ hashTable[h] = current; /* update hash table */
if ( ((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow : ensure repIndex isn't overlapping dict + prefix */ if ( ((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow : ensure repIndex isn't overlapping dict + prefix */
&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) { && (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
const BYTE* const repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend; const BYTE* const repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend;
mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixStart) + 4; mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixStart) + 4;
ip++; ip++;
ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, 0, mLength-MINMATCH); ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
} else if ( (matchIndex <= prefixStartIndex) ) { } else if ( (matchIndex <= prefixStartIndex) ) {
size_t const dictHash = ZSTD_hashPtr(ip, dictHLog, mls); size_t const dictHash = ZSTD_hashPtr(ip, dictHLog, mls);
U32 const dictMatchIndex = dictHashTable[dictHash]; U32 const dictMatchIndex = dictHashTable[dictHash];
const BYTE* dictMatch = dictBase + dictMatchIndex; const BYTE* dictMatch = dictBase + dictMatchIndex;
if (dictMatchIndex <= dictStartIndex || if (dictMatchIndex <= dictStartIndex ||
MEM_read32(dictMatch) != MEM_read32(ip)) { MEM_read32(dictMatch) != MEM_read32(ip)) {
assert(stepSize >= 1); assert(stepSize >= 1);
ip += ((ip-anchor) >> kSearchStrength) + stepSize; ip += ((ip-anchor) >> kSearchStrength) + stepSize;
continue; continue;
} else { } else {
/* found a dict match */ /* found a dict match */
U32 const offset = (U32)(current-dictMatchIndex-dictIndexDelta); U32 const offset = (U32)(current-dictMatchIndex-dictIndexDelta);
mLength = ZSTD_count_2segments(ip+4, dictMatch+4, iend, dictEnd, prefixStart) + 4; mLength = ZSTD_count_2segments(ip+4, dictMatch+4, iend, dictEnd, prefixStart) + 4;
while (((ip>anchor) & (dictMatch>dictStart)) while (((ip>anchor) & (dictMatch>dictStart))
&& (ip[-1] == dictMatch[-1])) { && (ip[-1] == dictMatch[-1])) {
ip--; dictMatch--; mLength++; ip--; dictMatch--; mLength++;
} /* catch up */ } /* catch up */
offset_2 = offset_1; offset_2 = offset_1;
offset_1 = offset; offset_1 = offset;
ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, offset + ZSTD_REP_MOVE, mLength-MINMATCH); ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
} }
} else if (MEM_read32(match) != MEM_read32(ip)) { } else if (MEM_read32(match) != MEM_read32(ip)) {
/* it's not a match, and we're not going to check the dictionary */ /* it's not a match, and we're not going to check the dictionary */
assert(stepSize >= 1); assert(stepSize >= 1);
ip += ((ip-anchor) >> kSearchStrength) + stepSize; ip += ((ip-anchor) >> kSearchStrength) + stepSize;
continue; continue;
} else { } else {
/* found a regular match */ /* found a regular match */
U32 const offset = (U32)(ip-match); U32 const offset = (U32)(ip-match);
mLength = ZSTD_count(ip+4, match+4, iend) + 4; mLength = ZSTD_count(ip+4, match+4, iend) + 4;
while (((ip>anchor) & (match>prefixStart)) while (((ip>anchor) & (match>prefixStart))
&& (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */ && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
offset_2 = offset_1; offset_2 = offset_1;
offset_1 = offset; offset_1 = offset;
ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, offset + ZSTD_REP_MOVE, mLength-MINMATCH); ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
} }
/* match found */ /* match found */
ip += mLength; ip += mLength;
anchor = ip; anchor = ip;
if (ip <= ilimit) { if (ip <= ilimit) {
/* Fill Table */ /* Fill Table */
assert(base+current+2 > istart); /* check base overflow */ assert(base+current+2 > istart); /* check base overflow */
hashTable[ZSTD_hashPtr(base+current+2, hlog, mls)] = current+2; /* here because current+2 could be > iend-8 */ hashTable[ZSTD_hashPtr(base+current+2, hlog, mls)] = current+2; /* here because current+2 could be > iend-8 */
hashTable[ZSTD_hashPtr(ip-2, hlog, mls)] = (U32)(ip-2-base); hashTable[ZSTD_hashPtr(ip-2, hlog, mls)] = (U32)(ip-2-base);
/* check immediate repcode */ /* check immediate repcode */
while (ip <= ilimit) { while (ip <= ilimit) {
U32 const current2 = (U32)(ip-base); U32 const current2 = (U32)(ip-base);
U32 const repIndex2 = current2 - offset_2; U32 const repIndex2 = current2 - offset_2;
const BYTE* repMatch2 = repIndex2 < prefixStartIndex ? const BYTE* repMatch2 = repIndex2 < prefixStartIndex ?
dictBase - dictIndexDelta + repIndex2 : dictBase - dictIndexDelta + repIndex2 :
base + repIndex2; base + repIndex2;
if ( ((U32)((prefixStartIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */) if ( ((U32)((prefixStartIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */)
&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) { && (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend; const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend;
size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4; size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4;
U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */ U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
ZSTD_storeSeq(seqStore, 0, anchor, 0, repLength2-MINMATCH); ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, repLength2-MINMATCH);
hashTable[ZSTD_hashPtr(ip, hlog, mls)] = current2; hashTable[ZSTD_hashPtr(ip, hlog, mls)] = current2;
ip += repLength2; ip += repLength2;
anchor = ip; anchor = ip;
continue; continue;
} }
break; break;
} }
} }
} }
/* save reps for next block */ /* save reps for next block */
rep[0] = offset_1 ? offset_1 : offsetSaved; rep[0] = offset_1 ? offset_1 : offsetSaved;
rep[1] = offset_2 ? offset_2 : offsetSaved; rep[1] = offset_2 ? offset_2 : offsetSaved;
/* Return the last literals size */ /* Return the last literals size */
return (size_t)(iend - anchor); return (size_t)(iend - anchor);
} }
size_t ZSTD_compressBlock_fast_dictMatchState( size_t ZSTD_compressBlock_fast_dictMatchState(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize) void const* src, size_t srcSize)
{ {
ZSTD_compressionParameters const* cParams = &ms->cParams; U32 const mls = ms->cParams.minMatch;
U32 const mls = cParams->minMatch;
assert(ms->dictMatchState != NULL); assert(ms->dictMatchState != NULL);
switch(mls) switch(mls)
{ {
default: /* includes case 3 */ default: /* includes case 3 */
case 4 : case 4 :
return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 4); return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 4);
case 5 : case 5 :
return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 5); return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 5);
const size_t h = ZSTD_hashPtr(ip, hlog, mls); const size_t h = ZSTD_hashPtr(ip, hlog, mls);
const U32 matchIndex = hashTable[h]; const U32 matchIndex = hashTable[h];
const BYTE* const matchBase = matchIndex < prefixStartIndex ? dictBase : base; const BYTE* const matchBase = matchIndex < prefixStartIndex ? dictBase : base;
const BYTE* match = matchBase + matchIndex; const BYTE* match = matchBase + matchIndex;
const U32 current = (U32)(ip-base); const U32 current = (U32)(ip-base);
const U32 repIndex = current + 1 - offset_1; const U32 repIndex = current + 1 - offset_1;
const BYTE* const repBase = repIndex < prefixStartIndex ? dictBase : base; const BYTE* const repBase = repIndex < prefixStartIndex ? dictBase : base;
const BYTE* const repMatch = repBase + repIndex; const BYTE* const repMatch = repBase + repIndex;
size_t mLength;
hashTable[h] = current; /* update hash table */ hashTable[h] = current; /* update hash table */
assert(offset_1 <= current +1); /* check repIndex */ assert(offset_1 <= current +1); /* check repIndex */
if ( (((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow */ & (repIndex > dictStartIndex)) if ( (((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow */ & (repIndex > dictStartIndex))
&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) { && (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
const BYTE* const repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend; const BYTE* const repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend;
mLength = ZSTD_count_2segments(ip+1 +4, repMatch +4, iend, repMatchEnd, prefixStart) + 4; size_t const rLength = ZSTD_count_2segments(ip+1 +4, repMatch +4, iend, repMatchEnd, prefixStart) + 4;
ip++; ip++;
ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, 0, mLength-MINMATCH); ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, rLength-MINMATCH);
ip += rLength;
anchor = ip;
} else { } else {
if ( (matchIndex < dictStartIndex) || if ( (matchIndex < dictStartIndex) ||
(MEM_read32(match) != MEM_read32(ip)) ) { (MEM_read32(match) != MEM_read32(ip)) ) {
assert(stepSize >= 1); assert(stepSize >= 1);
ip += ((ip-anchor) >> kSearchStrength) + stepSize; ip += ((ip-anchor) >> kSearchStrength) + stepSize;
continue; continue;
} }
{ const BYTE* const matchEnd = matchIndex < prefixStartIndex ? dictEnd : iend; { const BYTE* const matchEnd = matchIndex < prefixStartIndex ? dictEnd : iend;
const BYTE* const lowMatchPtr = matchIndex < prefixStartIndex ? dictStart : prefixStart; const BYTE* const lowMatchPtr = matchIndex < prefixStartIndex ? dictStart : prefixStart;
U32 offset; U32 const offset = current - matchIndex;
mLength = ZSTD_count_2segments(ip+4, match+4, iend, matchEnd, prefixStart) + 4; size_t mLength = ZSTD_count_2segments(ip+4, match+4, iend, matchEnd, prefixStart) + 4;
while (((ip>anchor) & (match>lowMatchPtr)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */ while (((ip>anchor) & (match>lowMatchPtr)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
offset = current - matchIndex; offset_2 = offset_1; offset_1 = offset; /* update offset history */
offset_2 = offset_1; ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
offset_1 = offset;
ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
} }
/* found a match : store it */
ip += mLength; ip += mLength;
anchor = ip; anchor = ip;
} }
if (ip <= ilimit) { if (ip <= ilimit) {
/* Fill Table */ /* Fill Table */
hashTable[ZSTD_hashPtr(base+current+2, hlog, mls)] = current+2; hashTable[ZSTD_hashPtr(base+current+2, hlog, mls)] = current+2;
hashTable[ZSTD_hashPtr(ip-2, hlog, mls)] = (U32)(ip-2-base); hashTable[ZSTD_hashPtr(ip-2, hlog, mls)] = (U32)(ip-2-base);
/* check immediate repcode */ /* check immediate repcode */
while (ip <= ilimit) { while (ip <= ilimit) {
U32 const current2 = (U32)(ip-base); U32 const current2 = (U32)(ip-base);
U32 const repIndex2 = current2 - offset_2; U32 const repIndex2 = current2 - offset_2;
const BYTE* repMatch2 = repIndex2 < prefixStartIndex ? dictBase + repIndex2 : base + repIndex2; const BYTE* const repMatch2 = repIndex2 < prefixStartIndex ? dictBase + repIndex2 : base + repIndex2;
if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) & (repIndex2 > dictStartIndex)) /* intentional overflow */ if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) & (repIndex2 > dictStartIndex)) /* intentional overflow */
&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) { && (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend; const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend;
size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4; size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4;
U32 const tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */ { U32 const tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; } /* swap offset_2 <=> offset_1 */
ZSTD_storeSeq(seqStore, 0, anchor, 0, repLength2-MINMATCH); ZSTD_storeSeq(seqStore, 0 /*litlen*/, anchor, iend, 0 /*offcode*/, repLength2-MINMATCH);
hashTable[ZSTD_hashPtr(ip, hlog, mls)] = current2; hashTable[ZSTD_hashPtr(ip, hlog, mls)] = current2;
ip += repLength2; ip += repLength2;
anchor = ip; anchor = ip;
continue; continue;
} }
break; break;
} } } } } }
/* save reps for next block */ /* save reps for next block */
rep[0] = offset_1; rep[0] = offset_1;
rep[1] = offset_2; rep[1] = offset_2;
/* Return the last literals size */ /* Return the last literals size */
return (size_t)(iend - anchor); return (size_t)(iend - anchor);
} }
size_t ZSTD_compressBlock_fast_extDict( size_t ZSTD_compressBlock_fast_extDict(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize) void const* src, size_t srcSize)
{ {
ZSTD_compressionParameters const* cParams = &ms->cParams; U32 const mls = ms->cParams.minMatch;
U32 const mls = cParams->minMatch;
switch(mls) switch(mls)
{ {
default: /* includes case 3 */ default: /* includes case 3 */
case 4 : case 4 :
return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 4); return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 4);
case 5 : case 5 :
return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 5); return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 5);
case 6 : case 6 :
return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 6); return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 6);
case 7 : case 7 :
return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 7); return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 7);
} }
} }

contrib/python-zstandard/zstd/compress/zstd_lazy.c

const BYTE* const mStart = (matchIndex < prefixLowestIndex) ? dictLowest : prefixLowest; const BYTE* const mStart = (matchIndex < prefixLowestIndex) ? dictLowest : prefixLowest;
while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */ while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */
} }
offset_2 = offset_1; offset_1 = (U32)(offset - ZSTD_REP_MOVE); offset_2 = offset_1; offset_1 = (U32)(offset - ZSTD_REP_MOVE);
} }
/* store sequence */ /* store sequence */
_storeSequence: _storeSequence:
{ size_t const litLength = start - anchor; { size_t const litLength = start - anchor;
ZSTD_storeSeq(seqStore, litLength, anchor, (U32)offset, matchLength-MINMATCH); ZSTD_storeSeq(seqStore, litLength, anchor, iend, (U32)offset, matchLength-MINMATCH);
anchor = ip = start + matchLength; anchor = ip = start + matchLength;
} }
/* check immediate repcode */ /* check immediate repcode */
if (dictMode == ZSTD_dictMatchState) { if (dictMode == ZSTD_dictMatchState) {
while (ip <= ilimit) { while (ip <= ilimit) {
U32 const current2 = (U32)(ip-base); U32 const current2 = (U32)(ip-base);
U32 const repIndex = current2 - offset_2; U32 const repIndex = current2 - offset_2;
const BYTE* repMatch = dictMode == ZSTD_dictMatchState const BYTE* repMatch = dictMode == ZSTD_dictMatchState
&& repIndex < prefixLowestIndex ? && repIndex < prefixLowestIndex ?
dictBase - dictIndexDelta + repIndex : dictBase - dictIndexDelta + repIndex :
base + repIndex; base + repIndex;
if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex) >= 3 /* intentional overflow */) if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex) >= 3 /* intentional overflow */)
&& (MEM_read32(repMatch) == MEM_read32(ip)) ) { && (MEM_read32(repMatch) == MEM_read32(ip)) ) {
const BYTE* const repEnd2 = repIndex < prefixLowestIndex ? dictEnd : iend; const BYTE* const repEnd2 = repIndex < prefixLowestIndex ? dictEnd : iend;
matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd2, prefixLowest) + 4; matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd2, prefixLowest) + 4;
offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap offset_2 <=> offset_1 */ offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap offset_2 <=> offset_1 */
ZSTD_storeSeq(seqStore, 0, anchor, 0, matchLength-MINMATCH); ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, matchLength-MINMATCH);
ip += matchLength; ip += matchLength;
anchor = ip; anchor = ip;
continue; continue;
} }
break; break;
} }
} }
if (dictMode == ZSTD_noDict) { if (dictMode == ZSTD_noDict) {
while ( ((ip <= ilimit) & (offset_2>0)) while ( ((ip <= ilimit) & (offset_2>0))
&& (MEM_read32(ip) == MEM_read32(ip - offset_2)) ) { && (MEM_read32(ip) == MEM_read32(ip - offset_2)) ) {
/* store sequence */ /* store sequence */
matchLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4; matchLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4;
offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap repcodes */ offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap repcodes */
ZSTD_storeSeq(seqStore, 0, anchor, 0, matchLength-MINMATCH); ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, matchLength-MINMATCH);
ip += matchLength; ip += matchLength;
anchor = ip; anchor = ip;
continue; /* faster when present ... (?) */ continue; /* faster when present ... (?) */
} } } } } }
/* Save reps for next block */ /* Save reps for next block */
rep[0] = offset_1 ? offset_1 : savedOffset; rep[0] = offset_1 ? offset_1 : savedOffset;
rep[1] = offset_2 ? offset_2 : savedOffset; rep[1] = offset_2 ? offset_2 : savedOffset;
const BYTE* const mStart = (matchIndex < dictLimit) ? dictStart : prefixStart; const BYTE* const mStart = (matchIndex < dictLimit) ? dictStart : prefixStart;
while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */ while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */
offset_2 = offset_1; offset_1 = (U32)(offset - ZSTD_REP_MOVE); offset_2 = offset_1; offset_1 = (U32)(offset - ZSTD_REP_MOVE);
} }
/* store sequence */ /* store sequence */
_storeSequence: _storeSequence:
{ size_t const litLength = start - anchor; { size_t const litLength = start - anchor;
ZSTD_storeSeq(seqStore, litLength, anchor, (U32)offset, matchLength-MINMATCH); ZSTD_storeSeq(seqStore, litLength, anchor, iend, (U32)offset, matchLength-MINMATCH);
anchor = ip = start + matchLength; anchor = ip = start + matchLength;
} }
/* check immediate repcode */ /* check immediate repcode */
while (ip <= ilimit) { while (ip <= ilimit) {
const U32 repIndex = (U32)((ip-base) - offset_2); const U32 repIndex = (U32)((ip-base) - offset_2);
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base; const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
const BYTE* const repMatch = repBase + repIndex; const BYTE* const repMatch = repBase + repIndex;
if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > lowestIndex)) /* intentional overflow */ if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > lowestIndex)) /* intentional overflow */
if (MEM_read32(ip) == MEM_read32(repMatch)) { if (MEM_read32(ip) == MEM_read32(repMatch)) {
/* repcode detected we should take it */ /* repcode detected we should take it */
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend; const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4; matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap offset history */ offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap offset history */
ZSTD_storeSeq(seqStore, 0, anchor, 0, matchLength-MINMATCH); ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, matchLength-MINMATCH);
ip += matchLength; ip += matchLength;
anchor = ip; anchor = ip;
continue; /* faster when present ... (?) */ continue; /* faster when present ... (?) */
} }
break; break;
} } } }
/* Save reps for next block */ /* Save reps for next block */

contrib/python-zstandard/zstd/compress/zstd_ldm.c

} }
params->bucketSizeLog = MIN(params->bucketSizeLog, params->hashLog); params->bucketSizeLog = MIN(params->bucketSizeLog, params->hashLog);
} }
size_t ZSTD_ldm_getTableSize(ldmParams_t params) size_t ZSTD_ldm_getTableSize(ldmParams_t params)
{ {
size_t const ldmHSize = ((size_t)1) << params.hashLog; size_t const ldmHSize = ((size_t)1) << params.hashLog;
size_t const ldmBucketSizeLog = MIN(params.bucketSizeLog, params.hashLog); size_t const ldmBucketSizeLog = MIN(params.bucketSizeLog, params.hashLog);
size_t const ldmBucketSize = size_t const ldmBucketSize = ((size_t)1) << (params.hashLog - ldmBucketSizeLog);
((size_t)1) << (params.hashLog - ldmBucketSizeLog); size_t const totalSize = ZSTD_cwksp_alloc_size(ldmBucketSize)
size_t const totalSize = ldmBucketSize + ldmHSize * sizeof(ldmEntry_t); + ZSTD_cwksp_alloc_size(ldmHSize * sizeof(ldmEntry_t));
return params.enableLdm ? totalSize : 0; return params.enableLdm ? totalSize : 0;
} }
size_t ZSTD_ldm_getMaxNbSeq(ldmParams_t params, size_t maxChunkSize) size_t ZSTD_ldm_getMaxNbSeq(ldmParams_t params, size_t maxChunkSize)
{ {
return params.enableLdm ? (maxChunkSize / params.minMatchLength) : 0; return params.enableLdm ? (maxChunkSize / params.minMatchLength) : 0;
} }
size_t const newLitLength = size_t const newLitLength =
blockCompressor(ms, seqStore, rep, ip, sequence.litLength); blockCompressor(ms, seqStore, rep, ip, sequence.litLength);
ip += sequence.litLength; ip += sequence.litLength;
/* Update the repcodes */ /* Update the repcodes */
for (i = ZSTD_REP_NUM - 1; i > 0; i--) for (i = ZSTD_REP_NUM - 1; i > 0; i--)
rep[i] = rep[i-1]; rep[i] = rep[i-1];
rep[0] = sequence.offset; rep[0] = sequence.offset;
/* Store the sequence */ /* Store the sequence */
ZSTD_storeSeq(seqStore, newLitLength, ip - newLitLength, ZSTD_storeSeq(seqStore, newLitLength, ip - newLitLength, iend,
sequence.offset + ZSTD_REP_MOVE, sequence.offset + ZSTD_REP_MOVE,
sequence.matchLength - MINMATCH); sequence.matchLength - MINMATCH);
ip += sequence.matchLength; ip += sequence.matchLength;
} }
} }
/* Fill the tables for the block compressor */ /* Fill the tables for the block compressor */
ZSTD_ldm_limitTableUpdate(ms, ip); ZSTD_ldm_limitTableUpdate(ms, ip);
ZSTD_ldm_fillFastTables(ms, ip); ZSTD_ldm_fillFastTables(ms, ip);
/* Compress the last literals */ /* Compress the last literals */
return blockCompressor(ms, seqStore, rep, ip, iend - ip); return blockCompressor(ms, seqStore, rep, ip, iend - ip);
} }

contrib/python-zstandard/zstd/compress/zstd_opt.c

U32 const currentOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode]; U32 const currentOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode];
if (repCode >= 2) rep[2] = rep[1]; if (repCode >= 2) rep[2] = rep[1];
rep[1] = rep[0]; rep[1] = rep[0];
rep[0] = currentOffset; rep[0] = currentOffset;
} } } }
assert(anchor + llen <= iend); assert(anchor + llen <= iend);
ZSTD_updateStats(optStatePtr, llen, anchor, offCode, mlen); ZSTD_updateStats(optStatePtr, llen, anchor, offCode, mlen);
ZSTD_storeSeq(seqStore, llen, anchor, offCode, mlen-MINMATCH); ZSTD_storeSeq(seqStore, llen, anchor, iend, offCode, mlen-MINMATCH);
anchor += advance; anchor += advance;
ip = anchor; ip = anchor;
} } } }
ZSTD_setBasePrices(optStatePtr, optLevel); ZSTD_setBasePrices(optStatePtr, optLevel);
} }
} /* while (ip < ilimit) */ } /* while (ip < ilimit) */

contrib/python-zstandard/zstd/compress/zstdmt_compress.c

*/ */
if (job->jobID != 0) jobParams.fParams.checksumFlag = 0; if (job->jobID != 0) jobParams.fParams.checksumFlag = 0;
/* Don't run LDM for the chunks, since we handle it externally */ /* Don't run LDM for the chunks, since we handle it externally */
jobParams.ldmParams.enableLdm = 0; jobParams.ldmParams.enableLdm = 0;
/* init */ /* init */
if (job->cdict) { if (job->cdict) {
size_t const initError = ZSTD_compressBegin_advanced_internal(cctx, NULL, 0, ZSTD_dct_auto, ZSTD_dtlm_fast, job->cdict, jobParams, job->fullFrameSize); size_t const initError = ZSTD_compressBegin_advanced_internal(cctx, NULL, 0, ZSTD_dct_auto, ZSTD_dtlm_fast, job->cdict, &jobParams, job->fullFrameSize);
assert(job->firstJob); /* only allowed for first job */ assert(job->firstJob); /* only allowed for first job */
if (ZSTD_isError(initError)) JOB_ERROR(initError); if (ZSTD_isError(initError)) JOB_ERROR(initError);
} else { /* srcStart points at reloaded section */ } else { /* srcStart points at reloaded section */
U64 const pledgedSrcSize = job->firstJob ? job->fullFrameSize : job->src.size; U64 const pledgedSrcSize = job->firstJob ? job->fullFrameSize : job->src.size;
{ size_t const forceWindowError = ZSTD_CCtxParams_setParameter(&jobParams, ZSTD_c_forceMaxWindow, !job->firstJob); { size_t const forceWindowError = ZSTD_CCtxParams_setParameter(&jobParams, ZSTD_c_forceMaxWindow, !job->firstJob);
if (ZSTD_isError(forceWindowError)) JOB_ERROR(forceWindowError); if (ZSTD_isError(forceWindowError)) JOB_ERROR(forceWindowError);
} }
{ size_t const initError = ZSTD_compressBegin_advanced_internal(cctx, { size_t const initError = ZSTD_compressBegin_advanced_internal(cctx,
job->prefix.start, job->prefix.size, ZSTD_dct_rawContent, /* load dictionary in "content-only" mode (no header analysis) */ job->prefix.start, job->prefix.size, ZSTD_dct_rawContent, /* load dictionary in "content-only" mode (no header analysis) */
ZSTD_dtlm_fast, ZSTD_dtlm_fast,
NULL, /*cdict*/ NULL, /*cdict*/
jobParams, pledgedSrcSize); &jobParams, pledgedSrcSize);
if (ZSTD_isError(initError)) JOB_ERROR(initError); if (ZSTD_isError(initError)) JOB_ERROR(initError);
} } } }
/* Perform serial step as early as possible, but after CCtx initialization */ /* Perform serial step as early as possible, but after CCtx initialization */
ZSTDMT_serialState_update(job->serial, cctx, rawSeqStore, job->src, job->jobID); ZSTDMT_serialState_update(job->serial, cctx, rawSeqStore, job->src, job->jobID);
if (!job->firstJob) { /* flush and overwrite frame header when it's not first job */ if (!job->firstJob) { /* flush and overwrite frame header when it's not first job */
size_t const hSize = ZSTD_compressContinue(cctx, dstBuff.start, dstBuff.capacity, job->src.start, 0); size_t const hSize = ZSTD_compressContinue(cctx, dstBuff.start, dstBuff.capacity, job->src.start, 0);
/* ZSTDMT_releaseAllJobResources() : /* ZSTDMT_releaseAllJobResources() :
* note : ensure all workers are killed first ! */ * note : ensure all workers are killed first ! */
static void ZSTDMT_releaseAllJobResources(ZSTDMT_CCtx* mtctx) static void ZSTDMT_releaseAllJobResources(ZSTDMT_CCtx* mtctx)
{ {
unsigned jobID; unsigned jobID;
DEBUGLOG(3, "ZSTDMT_releaseAllJobResources"); DEBUGLOG(3, "ZSTDMT_releaseAllJobResources");
for (jobID=0; jobID <= mtctx->jobIDMask; jobID++) { for (jobID=0; jobID <= mtctx->jobIDMask; jobID++) {
/* Copy the mutex/cond out */
ZSTD_pthread_mutex_t const mutex = mtctx->jobs[jobID].job_mutex;
ZSTD_pthread_cond_t const cond = mtctx->jobs[jobID].job_cond;
DEBUGLOG(4, "job%02u: release dst address %08X", jobID, (U32)(size_t)mtctx->jobs[jobID].dstBuff.start); DEBUGLOG(4, "job%02u: release dst address %08X", jobID, (U32)(size_t)mtctx->jobs[jobID].dstBuff.start);
ZSTDMT_releaseBuffer(mtctx->bufPool, mtctx->jobs[jobID].dstBuff); ZSTDMT_releaseBuffer(mtctx->bufPool, mtctx->jobs[jobID].dstBuff);
mtctx->jobs[jobID].dstBuff = g_nullBuffer;
mtctx->jobs[jobID].cSize = 0; /* Clear the job description, but keep the mutex/cond */
memset(&mtctx->jobs[jobID], 0, sizeof(mtctx->jobs[jobID]));
mtctx->jobs[jobID].job_mutex = mutex;
mtctx->jobs[jobID].job_cond = cond;
} }
memset(mtctx->jobs, 0, (mtctx->jobIDMask+1)*sizeof(ZSTDMT_jobDescription));
mtctx->inBuff.buffer = g_nullBuffer; mtctx->inBuff.buffer = g_nullBuffer;
mtctx->inBuff.filled = 0; mtctx->inBuff.filled = 0;
mtctx->allJobsCompleted = 1; mtctx->allJobsCompleted = 1;
} }
static void ZSTDMT_waitForAllJobsCompleted(ZSTDMT_CCtx* mtctx) static void ZSTDMT_waitForAllJobsCompleted(ZSTDMT_CCtx* mtctx)
{ {
DEBUGLOG(4, "ZSTDMT_waitForAllJobsCompleted"); DEBUGLOG(4, "ZSTDMT_waitForAllJobsCompleted");
return ZSTD_CCtxParams_getParameter(&mtctx->params, ZSTD_c_rsyncable, value); return ZSTD_CCtxParams_getParameter(&mtctx->params, ZSTD_c_rsyncable, value);
default: default:
return ERROR(parameter_unsupported); return ERROR(parameter_unsupported);
} }
} }
/* Sets parameters relevant to the compression job, /* Sets parameters relevant to the compression job,
* initializing others to default values. */ * initializing others to default values. */
static ZSTD_CCtx_params ZSTDMT_initJobCCtxParams(ZSTD_CCtx_params const params) static ZSTD_CCtx_params ZSTDMT_initJobCCtxParams(const ZSTD_CCtx_params* params)
{ {
ZSTD_CCtx_params jobParams = params; ZSTD_CCtx_params jobParams = *params;
/* Clear parameters related to multithreading */ /* Clear parameters related to multithreading */
jobParams.forceWindow = 0; jobParams.forceWindow = 0;
jobParams.nbWorkers = 0; jobParams.nbWorkers = 0;
jobParams.jobSize = 0; jobParams.jobSize = 0;
jobParams.overlapLog = 0; jobParams.overlapLog = 0;
jobParams.rsyncable = 0; jobParams.rsyncable = 0;
memset(&jobParams.ldmParams, 0, sizeof(ldmParams_t)); memset(&jobParams.ldmParams, 0, sizeof(ldmParams_t));
memset(&jobParams.customMem, 0, sizeof(ZSTD_customMem)); memset(&jobParams.customMem, 0, sizeof(ZSTD_customMem));
return toFlush; return toFlush;
} }
/* ------------------------------------------ */ /* ------------------------------------------ */
/* ===== Multi-threaded compression ===== */ /* ===== Multi-threaded compression ===== */
/* ------------------------------------------ */ /* ------------------------------------------ */
static unsigned ZSTDMT_computeTargetJobLog(ZSTD_CCtx_params const params) static unsigned ZSTDMT_computeTargetJobLog(const ZSTD_CCtx_params* params)
{ {
unsigned jobLog; unsigned jobLog;
if (params.ldmParams.enableLdm) { if (params->ldmParams.enableLdm) {
/* In Long Range Mode, the windowLog is typically oversized. /* In Long Range Mode, the windowLog is typically oversized.
* In which case, it's preferable to determine the jobSize * In which case, it's preferable to determine the jobSize
* based on chainLog instead. */ * based on chainLog instead. */
jobLog = MAX(21, params.cParams.chainLog + 4); jobLog = MAX(21, params->cParams.chainLog + 4);
} else { } else {
jobLog = MAX(20, params.cParams.windowLog + 2); jobLog = MAX(20, params->cParams.windowLog + 2);
} }
return MIN(jobLog, (unsigned)ZSTDMT_JOBLOG_MAX); return MIN(jobLog, (unsigned)ZSTDMT_JOBLOG_MAX);
} }
static int ZSTDMT_overlapLog_default(ZSTD_strategy strat) static int ZSTDMT_overlapLog_default(ZSTD_strategy strat)
{ {
switch(strat) switch(strat)
{ {
static int ZSTDMT_overlapLog(int ovlog, ZSTD_strategy strat) static int ZSTDMT_overlapLog(int ovlog, ZSTD_strategy strat)
{ {
assert(0 <= ovlog && ovlog <= 9); assert(0 <= ovlog && ovlog <= 9);
if (ovlog == 0) return ZSTDMT_overlapLog_default(strat); if (ovlog == 0) return ZSTDMT_overlapLog_default(strat);
return ovlog; return ovlog;
} }
static size_t ZSTDMT_computeOverlapSize(ZSTD_CCtx_params const params) static size_t ZSTDMT_computeOverlapSize(const ZSTD_CCtx_params* params)
{ {
int const overlapRLog = 9 - ZSTDMT_overlapLog(params.overlapLog, params.cParams.strategy); int const overlapRLog = 9 - ZSTDMT_overlapLog(params->overlapLog, params->cParams.strategy);
int ovLog = (overlapRLog >= 8) ? 0 : (params.cParams.windowLog - overlapRLog); int ovLog = (overlapRLog >= 8) ? 0 : (params->cParams.windowLog - overlapRLog);
assert(0 <= overlapRLog && overlapRLog <= 8); assert(0 <= overlapRLog && overlapRLog <= 8);
if (params.ldmParams.enableLdm) { if (params->ldmParams.enableLdm) {
/* In Long Range Mode, the windowLog is typically oversized. /* In Long Range Mode, the windowLog is typically oversized.
* In which case, it's preferable to determine the jobSize * In which case, it's preferable to determine the jobSize
* based on chainLog instead. * based on chainLog instead.
* Then, ovLog becomes a fraction of the jobSize, rather than windowSize */ * Then, ovLog becomes a fraction of the jobSize, rather than windowSize */
ovLog = MIN(params.cParams.windowLog, ZSTDMT_computeTargetJobLog(params) - 2) ovLog = MIN(params->cParams.windowLog, ZSTDMT_computeTargetJobLog(params) - 2)
- overlapRLog; - overlapRLog;
} }
assert(0 <= ovLog && ovLog <= ZSTD_WINDOWLOG_MAX); assert(0 <= ovLog && ovLog <= ZSTD_WINDOWLOG_MAX);
DEBUGLOG(4, "overlapLog : %i", params.overlapLog); DEBUGLOG(4, "overlapLog : %i", params->overlapLog);
DEBUGLOG(4, "overlap size : %i", 1 << ovLog); DEBUGLOG(4, "overlap size : %i", 1 << ovLog);
return (ovLog==0) ? 0 : (size_t)1 << ovLog; return (ovLog==0) ? 0 : (size_t)1 << ovLog;
} }
static unsigned static unsigned
ZSTDMT_computeNbJobs(ZSTD_CCtx_params params, size_t srcSize, unsigned nbWorkers) ZSTDMT_computeNbJobs(const ZSTD_CCtx_params* params, size_t srcSize, unsigned nbWorkers)
{ {
assert(nbWorkers>0); assert(nbWorkers>0);
{ size_t const jobSizeTarget = (size_t)1 << ZSTDMT_computeTargetJobLog(params); { size_t const jobSizeTarget = (size_t)1 << ZSTDMT_computeTargetJobLog(params);
size_t const jobMaxSize = jobSizeTarget << 2; size_t const jobMaxSize = jobSizeTarget << 2;
size_t const passSizeMax = jobMaxSize * nbWorkers; size_t const passSizeMax = jobMaxSize * nbWorkers;
unsigned const multiplier = (unsigned)(srcSize / passSizeMax) + 1; unsigned const multiplier = (unsigned)(srcSize / passSizeMax) + 1;
unsigned const nbJobsLarge = multiplier * nbWorkers; unsigned const nbJobsLarge = multiplier * nbWorkers;
unsigned const nbJobsMax = (unsigned)(srcSize / jobSizeTarget) + 1; unsigned const nbJobsMax = (unsigned)(srcSize / jobSizeTarget) + 1;
unsigned const nbJobsSmall = MIN(nbJobsMax, nbWorkers); unsigned const nbJobsSmall = MIN(nbJobsMax, nbWorkers);
return (multiplier>1) ? nbJobsLarge : nbJobsSmall; return (multiplier>1) ? nbJobsLarge : nbJobsSmall;
} } } }
/* ZSTDMT_compress_advanced_internal() : /* ZSTDMT_compress_advanced_internal() :
* This is a blocking function : it will only give back control to caller after finishing its compression job. * This is a blocking function : it will only give back control to caller after finishing its compression job.
*/ */
static size_t ZSTDMT_compress_advanced_internal( static size_t ZSTDMT_compress_advanced_internal(
ZSTDMT_CCtx* mtctx, ZSTDMT_CCtx* mtctx,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize, const void* src, size_t srcSize,
const ZSTD_CDict* cdict, const ZSTD_CDict* cdict,
ZSTD_CCtx_params params) ZSTD_CCtx_params params)
{ {
ZSTD_CCtx_params const jobParams = ZSTDMT_initJobCCtxParams(params); ZSTD_CCtx_params const jobParams = ZSTDMT_initJobCCtxParams(&params);
size_t const overlapSize = ZSTDMT_computeOverlapSize(params); size_t const overlapSize = ZSTDMT_computeOverlapSize(&params);
unsigned const nbJobs = ZSTDMT_computeNbJobs(params, srcSize, params.nbWorkers); unsigned const nbJobs = ZSTDMT_computeNbJobs(&params, srcSize, params.nbWorkers);
size_t const proposedJobSize = (srcSize + (nbJobs-1)) / nbJobs; size_t const proposedJobSize = (srcSize + (nbJobs-1)) / nbJobs;
size_t const avgJobSize = (((proposedJobSize-1) & 0x1FFFF) < 0x7FFF) ? proposedJobSize + 0xFFFF : proposedJobSize; /* avoid too small last block */ size_t const avgJobSize = (((proposedJobSize-1) & 0x1FFFF) < 0x7FFF) ? proposedJobSize + 0xFFFF : proposedJobSize; /* avoid too small last block */
const char* const srcStart = (const char*)src; const char* const srcStart = (const char*)src;
size_t remainingSrcSize = srcSize; size_t remainingSrcSize = srcSize;
unsigned const compressWithinDst = (dstCapacity >= ZSTD_compressBound(srcSize)) ? nbJobs : (unsigned)(dstCapacity / ZSTD_compressBound(avgJobSize)); /* presumes avgJobSize >= 256 KB, which should be the case */ unsigned const compressWithinDst = (dstCapacity >= ZSTD_compressBound(srcSize)) ? nbJobs : (unsigned)(dstCapacity / ZSTD_compressBound(avgJobSize)); /* presumes avgJobSize >= 256 KB, which should be the case */
size_t frameStartPos = 0, dstBufferPos = 0; size_t frameStartPos = 0, dstBufferPos = 0;
assert(jobParams.nbWorkers == 0); assert(jobParams.nbWorkers == 0);
assert(mtctx->cctxPool->totalCCtx == params.nbWorkers); assert(mtctx->cctxPool->totalCCtx == params.nbWorkers);
params.jobSize = (U32)avgJobSize; params.jobSize = (U32)avgJobSize;
DEBUGLOG(4, "ZSTDMT_compress_advanced_internal: nbJobs=%2u (rawSize=%u bytes; fixedSize=%u) ", DEBUGLOG(4, "ZSTDMT_compress_advanced_internal: nbJobs=%2u (rawSize=%u bytes; fixedSize=%u) ",
nbJobs, (U32)proposedJobSize, (U32)avgJobSize); nbJobs, (U32)proposedJobSize, (U32)avgJobSize);
if ((nbJobs==1) | (params.nbWorkers<=1)) { /* fallback to single-thread mode : this is a blocking invocation anyway */ if ((nbJobs==1) | (params.nbWorkers<=1)) { /* fallback to single-thread mode : this is a blocking invocation anyway */
ZSTD_CCtx* const cctx = mtctx->cctxPool->cctx[0]; ZSTD_CCtx* const cctx = mtctx->cctxPool->cctx[0];
DEBUGLOG(4, "ZSTDMT_compress_advanced_internal: fallback to single-thread mode"); DEBUGLOG(4, "ZSTDMT_compress_advanced_internal: fallback to single-thread mode");
if (cdict) return ZSTD_compress_usingCDict_advanced(cctx, dst, dstCapacity, src, srcSize, cdict, jobParams.fParams); if (cdict) return ZSTD_compress_usingCDict_advanced(cctx, dst, dstCapacity, src, srcSize, cdict, jobParams.fParams);
return ZSTD_compress_advanced_internal(cctx, dst, dstCapacity, src, srcSize, NULL, 0, jobParams); return ZSTD_compress_advanced_internal(cctx, dst, dstCapacity, src, srcSize, NULL, 0, &jobParams);
} }
assert(avgJobSize >= 256 KB); /* condition for ZSTD_compressBound(A) + ZSTD_compressBound(B) <= ZSTD_compressBound(A+B), required to compress directly into Dst (no additional buffer) */ assert(avgJobSize >= 256 KB); /* condition for ZSTD_compressBound(A) + ZSTD_compressBound(B) <= ZSTD_compressBound(A+B), required to compress directly into Dst (no additional buffer) */
ZSTDMT_setBufferSize(mtctx->bufPool, ZSTD_compressBound(avgJobSize) ); ZSTDMT_setBufferSize(mtctx->bufPool, ZSTD_compressBound(avgJobSize) );
if (ZSTDMT_serialState_reset(&mtctx->serial, mtctx->seqPool, params, avgJobSize)) if (ZSTDMT_serialState_reset(&mtctx->serial, mtctx->seqPool, params, avgJobSize))
return ERROR(memory_allocation); return ERROR(memory_allocation);
FORWARD_IF_ERROR( ZSTDMT_expandJobsTable(mtctx, nbJobs) ); /* only expands if necessary */ FORWARD_IF_ERROR( ZSTDMT_expandJobsTable(mtctx, nbJobs) ); /* only expands if necessary */
if (params.nbWorkers != mtctx->params.nbWorkers) if (params.nbWorkers != mtctx->params.nbWorkers)
FORWARD_IF_ERROR( ZSTDMT_resize(mtctx, params.nbWorkers) ); FORWARD_IF_ERROR( ZSTDMT_resize(mtctx, params.nbWorkers) );
if (params.jobSize != 0 && params.jobSize < ZSTDMT_JOBSIZE_MIN) params.jobSize = ZSTDMT_JOBSIZE_MIN; if (params.jobSize != 0 && params.jobSize < ZSTDMT_JOBSIZE_MIN) params.jobSize = ZSTDMT_JOBSIZE_MIN;
if (params.jobSize > (size_t)ZSTDMT_JOBSIZE_MAX) params.jobSize = (size_t)ZSTDMT_JOBSIZE_MAX; if (params.jobSize > (size_t)ZSTDMT_JOBSIZE_MAX) params.jobSize = (size_t)ZSTDMT_JOBSIZE_MAX;
mtctx->singleBlockingThread = (pledgedSrcSize <= ZSTDMT_JOBSIZE_MIN); /* do not trigger multi-threading when srcSize is too small */ mtctx->singleBlockingThread = (pledgedSrcSize <= ZSTDMT_JOBSIZE_MIN); /* do not trigger multi-threading when srcSize is too small */
if (mtctx->singleBlockingThread) { if (mtctx->singleBlockingThread) {
ZSTD_CCtx_params const singleThreadParams = ZSTDMT_initJobCCtxParams(params); ZSTD_CCtx_params const singleThreadParams = ZSTDMT_initJobCCtxParams(&params);
DEBUGLOG(5, "ZSTDMT_initCStream_internal: switch to single blocking thread mode"); DEBUGLOG(5, "ZSTDMT_initCStream_internal: switch to single blocking thread mode");
assert(singleThreadParams.nbWorkers == 0); assert(singleThreadParams.nbWorkers == 0);
return ZSTD_initCStream_internal(mtctx->cctxPool->cctx[0], return ZSTD_initCStream_internal(mtctx->cctxPool->cctx[0],
dict, dictSize, cdict, dict, dictSize, cdict,
singleThreadParams, pledgedSrcSize); &singleThreadParams, pledgedSrcSize);
} }
DEBUGLOG(4, "ZSTDMT_initCStream_internal: %u workers", params.nbWorkers); DEBUGLOG(4, "ZSTDMT_initCStream_internal: %u workers", params.nbWorkers);
if (mtctx->allJobsCompleted == 0) { /* previous compression not correctly finished */ if (mtctx->allJobsCompleted == 0) { /* previous compression not correctly finished */
ZSTDMT_waitForAllJobsCompleted(mtctx); ZSTDMT_waitForAllJobsCompleted(mtctx);
ZSTDMT_releaseAllJobResources(mtctx); ZSTDMT_releaseAllJobResources(mtctx);
mtctx->allJobsCompleted = 1; mtctx->allJobsCompleted = 1;
mtctx->cdict = mtctx->cdictLocal; mtctx->cdict = mtctx->cdictLocal;
if (mtctx->cdictLocal == NULL) return ERROR(memory_allocation); if (mtctx->cdictLocal == NULL) return ERROR(memory_allocation);
} else { } else {
ZSTD_freeCDict(mtctx->cdictLocal); ZSTD_freeCDict(mtctx->cdictLocal);
mtctx->cdictLocal = NULL; mtctx->cdictLocal = NULL;
mtctx->cdict = cdict; mtctx->cdict = cdict;
} }
mtctx->targetPrefixSize = ZSTDMT_computeOverlapSize(params); mtctx->targetPrefixSize = ZSTDMT_computeOverlapSize(&params);
DEBUGLOG(4, "overlapLog=%i => %u KB", params.overlapLog, (U32)(mtctx->targetPrefixSize>>10)); DEBUGLOG(4, "overlapLog=%i => %u KB", params.overlapLog, (U32)(mtctx->targetPrefixSize>>10));
mtctx->targetSectionSize = params.jobSize; mtctx->targetSectionSize = params.jobSize;
if (mtctx->targetSectionSize == 0) { if (mtctx->targetSectionSize == 0) {
mtctx->targetSectionSize = 1ULL << ZSTDMT_computeTargetJobLog(params); mtctx->targetSectionSize = 1ULL << ZSTDMT_computeTargetJobLog(&params);
} }
assert(mtctx->targetSectionSize <= (size_t)ZSTDMT_JOBSIZE_MAX); assert(mtctx->targetSectionSize <= (size_t)ZSTDMT_JOBSIZE_MAX);
if (params.rsyncable) { if (params.rsyncable) {
/* Aim for the targetsectionSize as the average job size. */ /* Aim for the targetsectionSize as the average job size. */
U32 const jobSizeMB = (U32)(mtctx->targetSectionSize >> 20); U32 const jobSizeMB = (U32)(mtctx->targetSectionSize >> 20);
U32 const rsyncBits = ZSTD_highbit32(jobSizeMB) + 20; U32 const rsyncBits = ZSTD_highbit32(jobSizeMB) + 20;
assert(jobSizeMB >= 1); assert(jobSizeMB >= 1);

contrib/python-zstandard/zstd/decompress/huf_decompress.c

#error "Cannot force the use of the X1 and X2 decoders at the same time!" #error "Cannot force the use of the X1 and X2 decoders at the same time!"
#endif #endif
/* ************************************************************** /* **************************************************************
* Error Management * Error Management
****************************************************************/ ****************************************************************/
#define HUF_isError ERR_isError #define HUF_isError ERR_isError
#ifndef CHECK_F
#define CHECK_F(f) { size_t const err_ = (f); if (HUF_isError(err_)) return err_; } #define CHECK_F(f) { size_t const err_ = (f); if (HUF_isError(err_)) return err_; }
#endif
/* ************************************************************** /* **************************************************************
* Byte alignment for workSpace management * Byte alignment for workSpace management
****************************************************************/ ****************************************************************/
#define HUF_ALIGN(x, a) HUF_ALIGN_MASK((x), (a) - 1) #define HUF_ALIGN(x, a) HUF_ALIGN_MASK((x), (a) - 1)
#define HUF_ALIGN_MASK(x, mask) (((x) + (mask)) & ~(mask)) #define HUF_ALIGN_MASK(x, mask) (((x) + (mask)) & ~(mask))

contrib/python-zstandard/zstd/decompress/zstd_decompress.c

+ dctx->inBuffSize + dctx->outBuffSize; + dctx->inBuffSize + dctx->outBuffSize;
} }
size_t ZSTD_estimateDCtxSize(void) { return sizeof(ZSTD_DCtx); } size_t ZSTD_estimateDCtxSize(void) { return sizeof(ZSTD_DCtx); }
static size_t ZSTD_startingInputLength(ZSTD_format_e format) static size_t ZSTD_startingInputLength(ZSTD_format_e format)
{ {
size_t const startingInputLength = (format==ZSTD_f_zstd1_magicless) ? size_t const startingInputLength = ZSTD_FRAMEHEADERSIZE_PREFIX(format);
ZSTD_FRAMEHEADERSIZE_PREFIX - ZSTD_FRAMEIDSIZE :
ZSTD_FRAMEHEADERSIZE_PREFIX;
ZSTD_STATIC_ASSERT(ZSTD_FRAMEHEADERSIZE_PREFIX >= ZSTD_FRAMEIDSIZE);
/* only supports formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless */ /* only supports formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless */
assert( (format == ZSTD_f_zstd1) || (format == ZSTD_f_zstd1_magicless) ); assert( (format == ZSTD_f_zstd1) || (format == ZSTD_f_zstd1_magicless) );
return startingInputLength; return startingInputLength;
} }
static void ZSTD_initDCtx_internal(ZSTD_DCtx* dctx) static void ZSTD_initDCtx_internal(ZSTD_DCtx* dctx)
{ {
dctx->format = ZSTD_f_zstd1; /* ZSTD_decompressBegin() invokes ZSTD_startingInputLength() with argument dctx->format */ dctx->format = ZSTD_f_zstd1; /* ZSTD_decompressBegin() invokes ZSTD_startingInputLength() with argument dctx->format */
* compatible with legacy mode * compatible with legacy mode
* `srcSize` must be the exact length of some number of ZSTD compressed and/or * `srcSize` must be the exact length of some number of ZSTD compressed and/or
* skippable frames * skippable frames
* @return : decompressed size of the frames contained */ * @return : decompressed size of the frames contained */
unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize) unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize)
{ {
unsigned long long totalDstSize = 0; unsigned long long totalDstSize = 0;
while (srcSize >= ZSTD_FRAMEHEADERSIZE_PREFIX) { while (srcSize >= ZSTD_startingInputLength(ZSTD_f_zstd1)) {
U32 const magicNumber = MEM_readLE32(src); U32 const magicNumber = MEM_readLE32(src);
if ((magicNumber & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) { if ((magicNumber & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) {
size_t const skippableSize = readSkippableFrameSize(src, srcSize); size_t const skippableSize = readSkippableFrameSize(src, srcSize);
if (ZSTD_isError(skippableSize)) { if (ZSTD_isError(skippableSize)) {
return ZSTD_CONTENTSIZE_ERROR; return ZSTD_CONTENTSIZE_ERROR;
} }
assert(skippableSize <= srcSize); assert(skippableSize <= srcSize);
BYTE* const oend = ostart + dstCapacity; BYTE* const oend = ostart + dstCapacity;
BYTE* op = ostart; BYTE* op = ostart;
size_t remainingSrcSize = *srcSizePtr; size_t remainingSrcSize = *srcSizePtr;
DEBUGLOG(4, "ZSTD_decompressFrame (srcSize:%i)", (int)*srcSizePtr); DEBUGLOG(4, "ZSTD_decompressFrame (srcSize:%i)", (int)*srcSizePtr);
/* check */ /* check */
RETURN_ERROR_IF( RETURN_ERROR_IF(
remainingSrcSize < ZSTD_FRAMEHEADERSIZE_MIN+ZSTD_blockHeaderSize, remainingSrcSize < ZSTD_FRAMEHEADERSIZE_MIN(dctx->format)+ZSTD_blockHeaderSize,
srcSize_wrong); srcSize_wrong);
/* Frame Header */ /* Frame Header */
{ size_t const frameHeaderSize = ZSTD_frameHeaderSize(ip, ZSTD_FRAMEHEADERSIZE_PREFIX); { size_t const frameHeaderSize = ZSTD_frameHeaderSize_internal(
ip, ZSTD_FRAMEHEADERSIZE_PREFIX(dctx->format), dctx->format);
if (ZSTD_isError(frameHeaderSize)) return frameHeaderSize; if (ZSTD_isError(frameHeaderSize)) return frameHeaderSize;
RETURN_ERROR_IF(remainingSrcSize < frameHeaderSize+ZSTD_blockHeaderSize, RETURN_ERROR_IF(remainingSrcSize < frameHeaderSize+ZSTD_blockHeaderSize,
srcSize_wrong); srcSize_wrong);
FORWARD_IF_ERROR( ZSTD_decodeFrameHeader(dctx, ip, frameHeaderSize) ); FORWARD_IF_ERROR( ZSTD_decodeFrameHeader(dctx, ip, frameHeaderSize) );
ip += frameHeaderSize; remainingSrcSize -= frameHeaderSize; ip += frameHeaderSize; remainingSrcSize -= frameHeaderSize;
} }
/* Loop on each block */ /* Loop on each block */
DEBUGLOG(5, "ZSTD_decompressMultiFrame"); DEBUGLOG(5, "ZSTD_decompressMultiFrame");
assert(dict==NULL || ddict==NULL); /* either dict or ddict set, not both */ assert(dict==NULL || ddict==NULL); /* either dict or ddict set, not both */
if (ddict) { if (ddict) {
dict = ZSTD_DDict_dictContent(ddict); dict = ZSTD_DDict_dictContent(ddict);
dictSize = ZSTD_DDict_dictSize(ddict); dictSize = ZSTD_DDict_dictSize(ddict);
} }
while (srcSize >= ZSTD_FRAMEHEADERSIZE_PREFIX) { while (srcSize >= ZSTD_startingInputLength(dctx->format)) {
#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1) #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1)
if (ZSTD_isLegacy(src, srcSize)) { if (ZSTD_isLegacy(src, srcSize)) {
size_t decodedSize; size_t decodedSize;
size_t const frameSize = ZSTD_findFrameCompressedSizeLegacy(src, srcSize); size_t const frameSize = ZSTD_findFrameCompressedSizeLegacy(src, srcSize);
if (ZSTD_isError(frameSize)) return frameSize; if (ZSTD_isError(frameSize)) return frameSize;
RETURN_ERROR_IF(dctx->staticSize, memory_allocation, RETURN_ERROR_IF(dctx->staticSize, memory_allocation,
"legacy support is not compatible with static dctx"); "legacy support is not compatible with static dctx");
dictPtr += litlengthHeaderSize; dictPtr += litlengthHeaderSize;
} }
RETURN_ERROR_IF(dictPtr+12 > dictEnd, dictionary_corrupted); RETURN_ERROR_IF(dictPtr+12 > dictEnd, dictionary_corrupted);
{ int i; { int i;
size_t const dictContentSize = (size_t)(dictEnd - (dictPtr+12)); size_t const dictContentSize = (size_t)(dictEnd - (dictPtr+12));
for (i=0; i<3; i++) { for (i=0; i<3; i++) {
U32 const rep = MEM_readLE32(dictPtr); dictPtr += 4; U32 const rep = MEM_readLE32(dictPtr); dictPtr += 4;
RETURN_ERROR_IF(rep==0 || rep >= dictContentSize, RETURN_ERROR_IF(rep==0 || rep > dictContentSize,
dictionary_corrupted); dictionary_corrupted);
entropy->rep[i] = rep; entropy->rep[i] = rep;
} } } }
return dictPtr - (const BYTE*)dict; return dictPtr - (const BYTE*)dict;
} }
static size_t ZSTD_decompress_insertDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize) static size_t ZSTD_decompress_insertDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx, size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx,
const void* dict, size_t dictSize, const void* dict, size_t dictSize,
ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictLoadMethod_e dictLoadMethod,
ZSTD_dictContentType_e dictContentType) ZSTD_dictContentType_e dictContentType)
{ {
RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong); RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong);
ZSTD_clearDict(dctx); ZSTD_clearDict(dctx);
if (dict && dictSize >= 8) { if (dict && dictSize != 0) {
dctx->ddictLocal = ZSTD_createDDict_advanced(dict, dictSize, dictLoadMethod, dictContentType, dctx->customMem); dctx->ddictLocal = ZSTD_createDDict_advanced(dict, dictSize, dictLoadMethod, dictContentType, dctx->customMem);
RETURN_ERROR_IF(dctx->ddictLocal == NULL, memory_allocation); RETURN_ERROR_IF(dctx->ddictLocal == NULL, memory_allocation);
dctx->ddict = dctx->ddictLocal; dctx->ddict = dctx->ddictLocal;
dctx->dictUses = ZSTD_use_indefinitely; dctx->dictUses = ZSTD_use_indefinitely;
} }
return 0; return 0;
} }
size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize) size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize)
{ {
return ZSTD_DCtx_refPrefix_advanced(dctx, prefix, prefixSize, ZSTD_dct_rawContent); return ZSTD_DCtx_refPrefix_advanced(dctx, prefix, prefixSize, ZSTD_dct_rawContent);
} }
/* ZSTD_initDStream_usingDict() : /* ZSTD_initDStream_usingDict() :
* return : expected size, aka ZSTD_FRAMEHEADERSIZE_PREFIX. * return : expected size, aka ZSTD_startingInputLength().
* this function cannot fail */ * this function cannot fail */
size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize) size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize)
{ {
DEBUGLOG(4, "ZSTD_initDStream_usingDict"); DEBUGLOG(4, "ZSTD_initDStream_usingDict");
FORWARD_IF_ERROR( ZSTD_DCtx_reset(zds, ZSTD_reset_session_only) ); FORWARD_IF_ERROR( ZSTD_DCtx_reset(zds, ZSTD_reset_session_only) );
FORWARD_IF_ERROR( ZSTD_DCtx_loadDictionary(zds, dict, dictSize) ); FORWARD_IF_ERROR( ZSTD_DCtx_loadDictionary(zds, dict, dictSize) );
return ZSTD_FRAMEHEADERSIZE_PREFIX; return ZSTD_startingInputLength(zds->format);
} }
/* note : this variant can't fail */ /* note : this variant can't fail */
size_t ZSTD_initDStream(ZSTD_DStream* zds) size_t ZSTD_initDStream(ZSTD_DStream* zds)
{ {
DEBUGLOG(4, "ZSTD_initDStream"); DEBUGLOG(4, "ZSTD_initDStream");
return ZSTD_initDStream_usingDDict(zds, NULL); return ZSTD_initDStream_usingDDict(zds, NULL);
} }
/* ZSTD_initDStream_usingDDict() : /* ZSTD_initDStream_usingDDict() :
* ddict will just be referenced, and must outlive decompression session * ddict will just be referenced, and must outlive decompression session
* this function cannot fail */ * this function cannot fail */
size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* dctx, const ZSTD_DDict* ddict) size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* dctx, const ZSTD_DDict* ddict)
{ {
FORWARD_IF_ERROR( ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only) ); FORWARD_IF_ERROR( ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only) );
FORWARD_IF_ERROR( ZSTD_DCtx_refDDict(dctx, ddict) ); FORWARD_IF_ERROR( ZSTD_DCtx_refDDict(dctx, ddict) );
return ZSTD_FRAMEHEADERSIZE_PREFIX; return ZSTD_startingInputLength(dctx->format);
} }
/* ZSTD_resetDStream() : /* ZSTD_resetDStream() :
* return : expected size, aka ZSTD_FRAMEHEADERSIZE_PREFIX. * return : expected size, aka ZSTD_startingInputLength().
* this function cannot fail */ * this function cannot fail */
size_t ZSTD_resetDStream(ZSTD_DStream* dctx) size_t ZSTD_resetDStream(ZSTD_DStream* dctx)
{ {
FORWARD_IF_ERROR(ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only)); FORWARD_IF_ERROR(ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only));
return ZSTD_FRAMEHEADERSIZE_PREFIX; return ZSTD_startingInputLength(dctx->format);
} }
size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict) size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict)
{ {
RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong); RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong);
ZSTD_clearDict(dctx); ZSTD_clearDict(dctx);
if (ddict) { if (ddict) {
size_t const remainingInput = (size_t)(iend-ip); size_t const remainingInput = (size_t)(iend-ip);
assert(iend >= ip); assert(iend >= ip);
if (toLoad > remainingInput) { /* not enough input to load full header */ if (toLoad > remainingInput) { /* not enough input to load full header */
if (remainingInput > 0) { if (remainingInput > 0) {
memcpy(zds->headerBuffer + zds->lhSize, ip, remainingInput); memcpy(zds->headerBuffer + zds->lhSize, ip, remainingInput);
zds->lhSize += remainingInput; zds->lhSize += remainingInput;
} }
input->pos = input->size; input->pos = input->size;
return (MAX(ZSTD_FRAMEHEADERSIZE_MIN, hSize) - zds->lhSize) + ZSTD_blockHeaderSize; /* remaining header bytes + next block header */ return (MAX((size_t)ZSTD_FRAMEHEADERSIZE_MIN(zds->format), hSize) - zds->lhSize) + ZSTD_blockHeaderSize; /* remaining header bytes + next block header */
} }
assert(ip != NULL); assert(ip != NULL);
memcpy(zds->headerBuffer + zds->lhSize, ip, toLoad); zds->lhSize = hSize; ip += toLoad; memcpy(zds->headerBuffer + zds->lhSize, ip, toLoad); zds->lhSize = hSize; ip += toLoad;
break; break;
} } } }
/* check for single-pass mode opportunity */ /* check for single-pass mode opportunity */
if (zds->fParams.frameContentSize && zds->fParams.windowSize /* skippable frame if == 0 */ if (zds->fParams.frameContentSize && zds->fParams.windowSize /* skippable frame if == 0 */

contrib/python-zstandard/zstd/decompress/zstd_decompress_block.c

ZSTD_fseState stateOffb; ZSTD_fseState stateOffb;
ZSTD_fseState stateML; ZSTD_fseState stateML;
size_t prevOffset[ZSTD_REP_NUM]; size_t prevOffset[ZSTD_REP_NUM];
const BYTE* prefixStart; const BYTE* prefixStart;
const BYTE* dictEnd; const BYTE* dictEnd;
size_t pos; size_t pos;
} seqState_t; } seqState_t;
/*! ZSTD_overlapCopy8() :
* Copies 8 bytes from ip to op and updates op and ip where ip <= op.
* If the offset is < 8 then the offset is spread to at least 8 bytes.
*
* Precondition: *ip <= *op
* Postcondition: *op - *op >= 8
*/
static void ZSTD_overlapCopy8(BYTE** op, BYTE const** ip, size_t offset) {
assert(*ip <= *op);
if (offset < 8) {
/* close range match, overlap */
static const U32 dec32table[] = { 0, 1, 2, 1, 4, 4, 4, 4 }; /* added */
static const int dec64table[] = { 8, 8, 8, 7, 8, 9,10,11 }; /* subtracted */
int const sub2 = dec64table[offset];
(*op)[0] = (*ip)[0];
(*op)[1] = (*ip)[1];
(*op)[2] = (*ip)[2];
(*op)[3] = (*ip)[3];
*ip += dec32table[offset];
ZSTD_copy4(*op+4, *ip);
*ip -= sub2;
} else {
ZSTD_copy8(*op, *ip);
}
*ip += 8;
*op += 8;
assert(*op - *ip >= 8);
}
/* ZSTD_execSequenceLast7(): /*! ZSTD_safecopy() :
* exceptional case : decompress a match starting within last 7 bytes of output buffer. * Specialized version of memcpy() that is allowed to READ up to WILDCOPY_OVERLENGTH past the input buffer
* requires more careful checks, to ensure there is no overflow. * and write up to 16 bytes past oend_w (op >= oend_w is allowed).
* performance does not matter though. * This function is only called in the uncommon case where the sequence is near the end of the block. It
* note : this case is supposed to be never generated "naturally" by reference encoder, * should be fast for a single long sequence, but can be slow for several short sequences.
* since in most cases it needs at least 8 bytes to look for a match. *
* but it's allowed by the specification. */ * @param ovtype controls the overlap detection
* - ZSTD_no_overlap: The source and destination are guaranteed to be at least WILDCOPY_VECLEN bytes apart.
* - ZSTD_overlap_src_before_dst: The src and dst may overlap and may be any distance apart.
* The src buffer must be before the dst buffer.
*/
static void ZSTD_safecopy(BYTE* op, BYTE* const oend_w, BYTE const* ip, ptrdiff_t length, ZSTD_overlap_e ovtype) {
ptrdiff_t const diff = op - ip;
BYTE* const oend = op + length;
assert((ovtype == ZSTD_no_overlap && (diff <= -8 || diff >= 8 || op >= oend_w)) ||
(ovtype == ZSTD_overlap_src_before_dst && diff >= 0));
if (length < 8) {
/* Handle short lengths. */
while (op < oend) *op++ = *ip++;
return;
}
if (ovtype == ZSTD_overlap_src_before_dst) {
/* Copy 8 bytes and ensure the offset >= 8 when there can be overlap. */
assert(length >= 8);
ZSTD_overlapCopy8(&op, &ip, diff);
assert(op - ip >= 8);
assert(op <= oend);
}
if (oend <= oend_w) {
/* No risk of overwrite. */
ZSTD_wildcopy(op, ip, length, ovtype);
return;
}
if (op <= oend_w) {
/* Wildcopy until we get close to the end. */
assert(oend > oend_w);
ZSTD_wildcopy(op, ip, oend_w - op, ovtype);
ip += oend_w - op;
op = oend_w;
}
/* Handle the leftovers. */
while (op < oend) *op++ = *ip++;
}
/* ZSTD_execSequenceEnd():
* This version handles cases that are near the end of the output buffer. It requires
* more careful checks to make sure there is no overflow. By separating out these hard
* and unlikely cases, we can speed up the common cases.
*
* NOTE: This function needs to be fast for a single long sequence, but doesn't need
* to be optimized for many small sequences, since those fall into ZSTD_execSequence().
*/
FORCE_NOINLINE FORCE_NOINLINE
size_t ZSTD_execSequenceLast7(BYTE* op, size_t ZSTD_execSequenceEnd(BYTE* op,
BYTE* const oend, seq_t sequence, BYTE* const oend, seq_t sequence,
const BYTE** litPtr, const BYTE* const litLimit, const BYTE** litPtr, const BYTE* const litLimit,
const BYTE* const base, const BYTE* const vBase, const BYTE* const dictEnd) const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd)
{ {
BYTE* const oLitEnd = op + sequence.litLength; BYTE* const oLitEnd = op + sequence.litLength;
size_t const sequenceLength = sequence.litLength + sequence.matchLength; size_t const sequenceLength = sequence.litLength + sequence.matchLength;
BYTE* const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */ BYTE* const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */
const BYTE* const iLitEnd = *litPtr + sequence.litLength; const BYTE* const iLitEnd = *litPtr + sequence.litLength;
const BYTE* match = oLitEnd - sequence.offset; const BYTE* match = oLitEnd - sequence.offset;
BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH;
/* check */ /* bounds checks */
assert(oLitEnd < oMatchEnd);
RETURN_ERROR_IF(oMatchEnd>oend, dstSize_tooSmall, "last match must fit within dstBuffer"); RETURN_ERROR_IF(oMatchEnd > oend, dstSize_tooSmall, "last match must fit within dstBuffer");
RETURN_ERROR_IF(iLitEnd > litLimit, corruption_detected, "try to read beyond literal buffer"); RETURN_ERROR_IF(iLitEnd > litLimit, corruption_detected, "try to read beyond literal buffer");
/* copy literals */ /* copy literals */
while (op < oLitEnd) *op++ = *(*litPtr)++; ZSTD_safecopy(op, oend_w, *litPtr, sequence.litLength, ZSTD_no_overlap);
op = oLitEnd;
*litPtr = iLitEnd;
/* copy Match */ /* copy Match */
if (sequence.offset > (size_t)(oLitEnd - base)) { if (sequence.offset > (size_t)(oLitEnd - prefixStart)) {
/* offset beyond prefix */ /* offset beyond prefix */
RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - vBase),corruption_detected); RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart), corruption_detected);
match = dictEnd - (base-match); match = dictEnd - (prefixStart-match);
if (match + sequence.matchLength <= dictEnd) { if (match + sequence.matchLength <= dictEnd) {
memmove(oLitEnd, match, sequence.matchLength); memmove(oLitEnd, match, sequence.matchLength);
return sequenceLength; return sequenceLength;
} }
/* span extDict & currentPrefixSegment */ /* span extDict & currentPrefixSegment */
{ size_t const length1 = dictEnd - match; { size_t const length1 = dictEnd - match;
memmove(oLitEnd, match, length1); memmove(oLitEnd, match, length1);
op = oLitEnd + length1; op = oLitEnd + length1;
sequence.matchLength -= length1; sequence.matchLength -= length1;
match = base; match = prefixStart;
} } } }
while (op < oMatchEnd) *op++ = *match++; ZSTD_safecopy(op, oend_w, match, sequence.matchLength, ZSTD_overlap_src_before_dst);
return sequenceLength; return sequenceLength;
} }
HINT_INLINE HINT_INLINE
size_t ZSTD_execSequence(BYTE* op, size_t ZSTD_execSequence(BYTE* op,
BYTE* const oend, seq_t sequence, BYTE* const oend, seq_t sequence,
const BYTE** litPtr, const BYTE* const litLimit, const BYTE** litPtr, const BYTE* const litLimit,
const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd) const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd)
{ {
BYTE* const oLitEnd = op + sequence.litLength; BYTE* const oLitEnd = op + sequence.litLength;
size_t const sequenceLength = sequence.litLength + sequence.matchLength; size_t const sequenceLength = sequence.litLength + sequence.matchLength;
BYTE* const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */ BYTE* const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */
BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH; BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH;
const BYTE* const iLitEnd = *litPtr + sequence.litLength; const BYTE* const iLitEnd = *litPtr + sequence.litLength;
const BYTE* match = oLitEnd - sequence.offset; const BYTE* match = oLitEnd - sequence.offset;
/* check */ /* Errors and uncommon cases handled here. */
RETURN_ERROR_IF(oMatchEnd>oend, dstSize_tooSmall, "last match must start at a minimum distance of WILDCOPY_OVERLENGTH from oend"); assert(oLitEnd < oMatchEnd);
RETURN_ERROR_IF(iLitEnd > litLimit, corruption_detected, "over-read beyond lit buffer"); if (iLitEnd > litLimit || oMatchEnd > oend_w)
if (oLitEnd>oend_w) return ZSTD_execSequenceLast7(op, oend, sequence, litPtr, litLimit, prefixStart, virtualStart, dictEnd); return ZSTD_execSequenceEnd(op, oend, sequence, litPtr, litLimit, prefixStart, virtualStart, dictEnd);
/* copy Literals */ /* Assumptions (everything else goes into ZSTD_execSequenceEnd()) */
if (sequence.litLength > 8) assert(iLitEnd <= litLimit /* Literal length is in bounds */);
ZSTD_wildcopy_16min(op, (*litPtr), sequence.litLength, ZSTD_no_overlap); /* note : since oLitEnd <= oend-WILDCOPY_OVERLENGTH, no risk of overwrite beyond oend */ assert(oLitEnd <= oend_w /* Can wildcopy literals */);
else assert(oMatchEnd <= oend_w /* Can wildcopy matches */);
ZSTD_copy8(op, *litPtr);
/* Copy Literals:
* Split out litLength <= 16 since it is nearly always true. +1.6% on gcc-9.
* We likely don't need the full 32-byte wildcopy.
*/
assert(WILDCOPY_OVERLENGTH >= 16);
ZSTD_copy16(op, (*litPtr));
if (sequence.litLength > 16) {
ZSTD_wildcopy(op+16, (*litPtr)+16, sequence.litLength-16, ZSTD_no_overlap);
}
op = oLitEnd; op = oLitEnd;
*litPtr = iLitEnd; /* update for next sequence */ *litPtr = iLitEnd; /* update for next sequence */
/* copy Match */ /* Copy Match */
if (sequence.offset > (size_t)(oLitEnd - prefixStart)) { if (sequence.offset > (size_t)(oLitEnd - prefixStart)) {
/* offset beyond prefix -> go into extDict */ /* offset beyond prefix -> go into extDict */
RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart), corruption_detected); RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart), corruption_detected);
match = dictEnd + (match - prefixStart); match = dictEnd + (match - prefixStart);
if (match + sequence.matchLength <= dictEnd) { if (match + sequence.matchLength <= dictEnd) {
memmove(oLitEnd, match, sequence.matchLength); memmove(oLitEnd, match, sequence.matchLength);
return sequenceLength; return sequenceLength;
} }
/* span extDict & currentPrefixSegment */ /* span extDict & currentPrefixSegment */
{ size_t const length1 = dictEnd - match; { size_t const length1 = dictEnd - match;
memmove(oLitEnd, match, length1); memmove(oLitEnd, match, length1);
op = oLitEnd + length1; op = oLitEnd + length1;
sequence.matchLength -= length1; sequence.matchLength -= length1;
match = prefixStart; match = prefixStart;
if (op > oend_w || sequence.matchLength < MINMATCH) {
U32 i;
for (i = 0; i < sequence.matchLength; ++i) op[i] = match[i];
return sequenceLength;
}
} } } }
/* Requirement: op <= oend_w && sequence.matchLength >= MINMATCH */ /* Match within prefix of 1 or more bytes */
assert(op <= oMatchEnd);
assert(oMatchEnd <= oend_w);
assert(match >= prefixStart);
assert(sequence.matchLength >= 1);
/* match within prefix */ /* Nearly all offsets are >= WILDCOPY_VECLEN bytes, which means we can use wildcopy
if (sequence.offset < 8) { * without overlap checking.
/* close range match, overlap */ */
static const U32 dec32table[] = { 0, 1, 2, 1, 4, 4, 4, 4 }; /* added */ if (sequence.offset >= WILDCOPY_VECLEN) {
static const int dec64table[] = { 8, 8, 8, 7, 8, 9,10,11 }; /* subtracted */ /* We bet on a full wildcopy for matches, since we expect matches to be
int const sub2 = dec64table[sequence.offset]; * longer than literals (in general). In silesia, ~10% of matches are longer
op[0] = match[0]; * than 16 bytes.
op[1] = match[1]; */
op[2] = match[2]; ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength, ZSTD_no_overlap);
op[3] = match[3];
match += dec32table[sequence.offset];
ZSTD_copy4(op+4, match);
match -= sub2;
} else {
ZSTD_copy8(op, match);
}
op += 8; match += 8;
if (oMatchEnd > oend-(16-MINMATCH)) {
if (op < oend_w) {
ZSTD_wildcopy(op, match, oend_w - op, ZSTD_overlap_src_before_dst);
match += oend_w - op;
op = oend_w;
}
while (op < oMatchEnd) *op++ = *match++;
} else {
ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength-8, ZSTD_overlap_src_before_dst); /* works even if matchLength < 8 */
}
return sequenceLength;
}
HINT_INLINE
size_t ZSTD_execSequenceLong(BYTE* op,
BYTE* const oend, seq_t sequence,
const BYTE** litPtr, const BYTE* const litLimit,
const BYTE* const prefixStart, const BYTE* const dictStart, const BYTE* const dictEnd)
{
BYTE* const oLitEnd = op + sequence.litLength;
size_t const sequenceLength = sequence.litLength + sequence.matchLength;
BYTE* const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */
BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH;
const BYTE* const iLitEnd = *litPtr + sequence.litLength;
const BYTE* match = sequence.match;
/* check */
RETURN_ERROR_IF(oMatchEnd > oend, dstSize_tooSmall, "last match must start at a minimum distance of WILDCOPY_OVERLENGTH from oend");
RETURN_ERROR_IF(iLitEnd > litLimit, corruption_detected, "over-read beyond lit buffer");
if (oLitEnd > oend_w) return ZSTD_execSequenceLast7(op, oend, sequence, litPtr, litLimit, prefixStart, dictStart, dictEnd);
/* copy Literals */
if (sequence.litLength > 8)
ZSTD_wildcopy_16min(op, *litPtr, sequence.litLength, ZSTD_no_overlap); /* note : since oLitEnd <= oend-WILDCOPY_OVERLENGTH, no risk of overwrite beyond oend */
else
ZSTD_copy8(op, *litPtr); /* note : op <= oLitEnd <= oend_w == oend - 8 */
op = oLitEnd;
*litPtr = iLitEnd; /* update for next sequence */
/* copy Match */
if (sequence.offset > (size_t)(oLitEnd - prefixStart)) {
/* offset beyond prefix */
RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - dictStart), corruption_detected);
if (match + sequence.matchLength <= dictEnd) {
memmove(oLitEnd, match, sequence.matchLength);
return sequenceLength;
}
/* span extDict & currentPrefixSegment */
{ size_t const length1 = dictEnd - match;
memmove(oLitEnd, match, length1);
op = oLitEnd + length1;
sequence.matchLength -= length1;
match = prefixStart;
if (op > oend_w || sequence.matchLength < MINMATCH) {
U32 i;
for (i = 0; i < sequence.matchLength; ++i) op[i] = match[i];
return sequenceLength; return sequenceLength;
} }
} } assert(sequence.offset < WILDCOPY_VECLEN);
assert(op <= oend_w);
assert(sequence.matchLength >= MINMATCH);
/* match within prefix */ /* Copy 8 bytes and spread the offset to be >= 8. */
if (sequence.offset < 8) { ZSTD_overlapCopy8(&op, &match, sequence.offset);
/* close range match, overlap */
static const U32 dec32table[] = { 0, 1, 2, 1, 4, 4, 4, 4 }; /* added */
static const int dec64table[] = { 8, 8, 8, 7, 8, 9,10,11 }; /* subtracted */
int const sub2 = dec64table[sequence.offset];
op[0] = match[0];
op[1] = match[1];
op[2] = match[2];
op[3] = match[3];
match += dec32table[sequence.offset];
ZSTD_copy4(op+4, match);
match -= sub2;
} else {
ZSTD_copy8(op, match);
}
op += 8; match += 8;
if (oMatchEnd > oend-(16-MINMATCH)) { /* If the match length is > 8 bytes, then continue with the wildcopy. */
if (op < oend_w) { if (sequence.matchLength > 8) {
ZSTD_wildcopy(op, match, oend_w - op, ZSTD_overlap_src_before_dst); assert(op < oMatchEnd);
match += oend_w - op; ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength-8, ZSTD_overlap_src_before_dst);
op = oend_w;
}
while (op < oMatchEnd) *op++ = *match++;
} else {
ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength-8, ZSTD_overlap_src_before_dst); /* works even if matchLength < 8 */
} }
return sequenceLength; return sequenceLength;
} }
static void static void
ZSTD_initFseState(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, const ZSTD_seqSymbol* dt) ZSTD_initFseState(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, const ZSTD_seqSymbol* dt)
{ {
const void* ptr = dt; const void* ptr = dt;
sequences[seqNb] = ZSTD_decodeSequenceLong(&seqState, isLongOffset); sequences[seqNb] = ZSTD_decodeSequenceLong(&seqState, isLongOffset);
PREFETCH_L1(sequences[seqNb].match); PREFETCH_L1(sequences[seqNb].match + sequences[seqNb].matchLength - 1); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */ PREFETCH_L1(sequences[seqNb].match); PREFETCH_L1(sequences[seqNb].match + sequences[seqNb].matchLength - 1); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */
} }
RETURN_ERROR_IF(seqNb<seqAdvance, corruption_detected); RETURN_ERROR_IF(seqNb<seqAdvance, corruption_detected);
/* decode and decompress */ /* decode and decompress */
for ( ; (BIT_reloadDStream(&(seqState.DStream)) <= BIT_DStream_completed) && (seqNb<nbSeq) ; seqNb++) { for ( ; (BIT_reloadDStream(&(seqState.DStream)) <= BIT_DStream_completed) && (seqNb<nbSeq) ; seqNb++) {
seq_t const sequence = ZSTD_decodeSequenceLong(&seqState, isLongOffset); seq_t const sequence = ZSTD_decodeSequenceLong(&seqState, isLongOffset);
size_t const oneSeqSize = ZSTD_execSequenceLong(op, oend, sequences[(seqNb-ADVANCED_SEQS) & STORED_SEQS_MASK], &litPtr, litEnd, prefixStart, dictStart, dictEnd); size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequences[(seqNb-ADVANCED_SEQS) & STORED_SEQS_MASK], &litPtr, litEnd, prefixStart, dictStart, dictEnd);
if (ZSTD_isError(oneSeqSize)) return oneSeqSize; if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
PREFETCH_L1(sequence.match); PREFETCH_L1(sequence.match + sequence.matchLength - 1); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */ PREFETCH_L1(sequence.match); PREFETCH_L1(sequence.match + sequence.matchLength - 1); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */
sequences[seqNb & STORED_SEQS_MASK] = sequence; sequences[seqNb & STORED_SEQS_MASK] = sequence;
op += oneSeqSize; op += oneSeqSize;
} }
RETURN_ERROR_IF(seqNb<nbSeq, corruption_detected); RETURN_ERROR_IF(seqNb<nbSeq, corruption_detected);
/* finish queue */ /* finish queue */
seqNb -= seqAdvance; seqNb -= seqAdvance;
for ( ; seqNb<nbSeq ; seqNb++) { for ( ; seqNb<nbSeq ; seqNb++) {
size_t const oneSeqSize = ZSTD_execSequenceLong(op, oend, sequences[seqNb&STORED_SEQS_MASK], &litPtr, litEnd, prefixStart, dictStart, dictEnd); size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequences[seqNb&STORED_SEQS_MASK], &litPtr, litEnd, prefixStart, dictStart, dictEnd);
if (ZSTD_isError(oneSeqSize)) return oneSeqSize; if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
op += oneSeqSize; op += oneSeqSize;
} }
/* save reps for next block */ /* save reps for next block */
{ U32 i; for (i=0; i<ZSTD_REP_NUM; i++) dctx->entropy.rep[i] = (U32)(seqState.prevOffset[i]); } { U32 i; for (i=0; i<ZSTD_REP_NUM; i++) dctx->entropy.rep[i] = (U32)(seqState.prevOffset[i]); }
} }

contrib/python-zstandard/zstd/deprecated/zbuff.h

#include "zstd.h" /* ZSTD_CStream, ZSTD_DStream, ZSTDLIB_API */ #include "zstd.h" /* ZSTD_CStream, ZSTD_DStream, ZSTDLIB_API */
/* *************************************************************** /* ***************************************************************
* Compiler specifics * Compiler specifics
*****************************************************************/ *****************************************************************/
/* Deprecation warnings */ /* Deprecation warnings */
/* Should these warnings be a problem, /* Should these warnings be a problem,
it is generally possible to disable them, * it is generally possible to disable them,
typically with -Wno-deprecated-declarations for gcc * typically with -Wno-deprecated-declarations for gcc
or _CRT_SECURE_NO_WARNINGS in Visual. * or _CRT_SECURE_NO_WARNINGS in Visual.
Otherwise, it's also possible to define ZBUFF_DISABLE_DEPRECATE_WARNINGS */ * Otherwise, it's also possible to define ZBUFF_DISABLE_DEPRECATE_WARNINGS
*/
#ifdef ZBUFF_DISABLE_DEPRECATE_WARNINGS #ifdef ZBUFF_DISABLE_DEPRECATE_WARNINGS
# define ZBUFF_DEPRECATED(message) ZSTDLIB_API /* disable deprecation warnings */ # define ZBUFF_DEPRECATED(message) ZSTDLIB_API /* disable deprecation warnings */
#else #else
# if defined (__cplusplus) && (__cplusplus >= 201402) /* C++14 or greater */ # if defined (__cplusplus) && (__cplusplus >= 201402) /* C++14 or greater */
# define ZBUFF_DEPRECATED(message) [[deprecated(message)]] ZSTDLIB_API # define ZBUFF_DEPRECATED(message) [[deprecated(message)]] ZSTDLIB_API
# elif (defined(__GNUC__) && (__GNUC__ >= 5)) || defined(__clang__) # elif (defined(GNUC) && (GNUC > 4 || (GNUC == 4 && GNUC_MINOR >= 5))) || defined(__clang__)
# define ZBUFF_DEPRECATED(message) ZSTDLIB_API __attribute__((deprecated(message))) # define ZBUFF_DEPRECATED(message) ZSTDLIB_API __attribute__((deprecated(message)))
# elif defined(__GNUC__) && (__GNUC__ >= 3) # elif defined(__GNUC__) && (__GNUC__ >= 3)
# define ZBUFF_DEPRECATED(message) ZSTDLIB_API __attribute__((deprecated)) # define ZBUFF_DEPRECATED(message) ZSTDLIB_API __attribute__((deprecated))
# elif defined(_MSC_VER) # elif defined(_MSC_VER)
# define ZBUFF_DEPRECATED(message) ZSTDLIB_API __declspec(deprecated(message)) # define ZBUFF_DEPRECATED(message) ZSTDLIB_API __declspec(deprecated(message))
# else # else
# pragma message("WARNING: You need to implement ZBUFF_DEPRECATED for this compiler") # pragma message("WARNING: You need to implement ZBUFF_DEPRECATED for this compiler")
# define ZBUFF_DEPRECATED(message) ZSTDLIB_API # define ZBUFF_DEPRECATED(message) ZSTDLIB_API

contrib/python-zstandard/zstd/dictBuilder/cover.c

if (ratio >= 10) { if (ratio >= 10) {
return; return;
} }
LOCALDISPLAYLEVEL(displayLevel, 1, LOCALDISPLAYLEVEL(displayLevel, 1,
"WARNING: The maximum dictionary size %u is too large " "WARNING: The maximum dictionary size %u is too large "
"compared to the source size %u! " "compared to the source size %u! "
"size(source)/size(dictionary) = %f, but it should be >= " "size(source)/size(dictionary) = %f, but it should be >= "
"10! This may lead to a subpar dictionary! We recommend " "10! This may lead to a subpar dictionary! We recommend "
"training on sources at least 10x, and up to 100x the " "training on sources at least 10x, and preferably 100x "
"size of the dictionary!\n", (U32)maxDictSize, "the size of the dictionary! \n", (U32)maxDictSize,
(U32)nbDmers, ratio); (U32)nbDmers, ratio);
} }
COVER_epoch_info_t COVER_computeEpochs(U32 maxDictSize, COVER_epoch_info_t COVER_computeEpochs(U32 maxDictSize,
U32 nbDmers, U32 k, U32 passes) U32 nbDmers, U32 k, U32 passes)
{ {
const U32 minEpochSize = k * 10; const U32 minEpochSize = k * 10;
COVER_epoch_info_t epochs; COVER_epoch_info_t epochs;

contrib/python-zstandard/zstd/dictBuilder/zdict.c

} }
static void ZDICT_fillNoise(void* buffer, size_t length) static void ZDICT_fillNoise(void* buffer, size_t length)
{ {
unsigned const prime1 = 2654435761U; unsigned const prime1 = 2654435761U;
unsigned const prime2 = 2246822519U; unsigned const prime2 = 2246822519U;
unsigned acc = prime1; unsigned acc = prime1;
size_t p=0;; size_t p=0;
for (p=0; p<length; p++) { for (p=0; p<length; p++) {
acc *= prime2; acc *= prime2;
((unsigned char*)buffer)[p] = (unsigned char)(acc >> 21); ((unsigned char*)buffer)[p] = (unsigned char)(acc >> 21);
} }
} }
typedef struct typedef struct

contrib/python-zstandard/zstd/zstd.h

#if defined (__cplusplus) #if defined (__cplusplus)
extern "C" { extern "C" {
#endif #endif
#ifndef ZSTD_H_235446 #ifndef ZSTD_H_235446
#define ZSTD_H_235446 #define ZSTD_H_235446
/* ====== Dependency ======*/ /* ====== Dependency ======*/
#include <limits.h> /* INT_MAX */
#include <stddef.h> /* size_t */ #include <stddef.h> /* size_t */
/* ===== ZSTDLIB_API : control library symbols visibility ===== */ /* ===== ZSTDLIB_API : control library symbols visibility ===== */
#ifndef ZSTDLIB_VISIBILITY #ifndef ZSTDLIB_VISIBILITY
# if defined(__GNUC__) && (__GNUC__ >= 4) # if defined(__GNUC__) && (__GNUC__ >= 4)
# define ZSTDLIB_VISIBILITY __attribute__ ((visibility ("default"))) # define ZSTDLIB_VISIBILITY __attribute__ ((visibility ("default")))
# else # else
Advanced experimental APIs should never be used with a dynamically-linked Advanced experimental APIs should never be used with a dynamically-linked
library. They are not "stable"; their definitions or signatures may change in library. They are not "stable"; their definitions or signatures may change in
the future. Only static linking is allowed. the future. Only static linking is allowed.
*******************************************************************************/ *******************************************************************************/
/*------ Version ------*/ /*------ Version ------*/
#define ZSTD_VERSION_MAJOR 1 #define ZSTD_VERSION_MAJOR 1
#define ZSTD_VERSION_MINOR 4 #define ZSTD_VERSION_MINOR 4
#define ZSTD_VERSION_RELEASE 3 #define ZSTD_VERSION_RELEASE 4
#define ZSTD_VERSION_NUMBER (ZSTD_VERSION_MAJOR *100*100 + ZSTD_VERSION_MINOR *100 + ZSTD_VERSION_RELEASE) #define ZSTD_VERSION_NUMBER (ZSTD_VERSION_MAJOR *100*100 + ZSTD_VERSION_MINOR *100 + ZSTD_VERSION_RELEASE)
ZSTDLIB_API unsigned ZSTD_versionNumber(void); /**< to check runtime library version */ ZSTDLIB_API unsigned ZSTD_versionNumber(void); /**< to check runtime library version */
#define ZSTD_LIB_VERSION ZSTD_VERSION_MAJOR.ZSTD_VERSION_MINOR.ZSTD_VERSION_RELEASE #define ZSTD_LIB_VERSION ZSTD_VERSION_MAJOR.ZSTD_VERSION_MINOR.ZSTD_VERSION_RELEASE
#define ZSTD_QUOTE(str) #str #define ZSTD_QUOTE(str) #str
#define ZSTD_EXPAND_AND_QUOTE(str) ZSTD_QUOTE(str) #define ZSTD_EXPAND_AND_QUOTE(str) ZSTD_QUOTE(str)
#define ZSTD_VERSION_STRING ZSTD_EXPAND_AND_QUOTE(ZSTD_LIB_VERSION) #define ZSTD_VERSION_STRING ZSTD_EXPAND_AND_QUOTE(ZSTD_LIB_VERSION)
* Note 2 : In multi-threaded environments, * Note 2 : In multi-threaded environments,
* use one different context per thread for parallel execution. * use one different context per thread for parallel execution.
*/ */
typedef struct ZSTD_CCtx_s ZSTD_CCtx; typedef struct ZSTD_CCtx_s ZSTD_CCtx;
ZSTDLIB_API ZSTD_CCtx* ZSTD_createCCtx(void); ZSTDLIB_API ZSTD_CCtx* ZSTD_createCCtx(void);
ZSTDLIB_API size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx); ZSTDLIB_API size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx);
/*! ZSTD_compressCCtx() : /*! ZSTD_compressCCtx() :
* Same as ZSTD_compress(), using an explicit ZSTD_CCtx * Same as ZSTD_compress(), using an explicit ZSTD_CCtx.
* The function will compress at requested compression level, * Important : in order to behave similarly to `ZSTD_compress()`,
* ignoring any other parameter */ * this function compresses at requested compression level,
* __ignoring any other parameter__ .
* If any advanced parameter was set using the advanced API,
* they will all be reset. Only `compressionLevel` remains.
*/
ZSTDLIB_API size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx, ZSTDLIB_API size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize, const void* src, size_t srcSize,
int compressionLevel); int compressionLevel);
/*= Decompression context /*= Decompression context
* When decompressing many times, * When decompressing many times,
* it is recommended to allocate a context only once, * it is recommended to allocate a context only once,
* Advanced compression API * Advanced compression API
***************************************/ ***************************************/
/* API design : /* API design :
* Parameters are pushed one by one into an existing context, * Parameters are pushed one by one into an existing context,
* using ZSTD_CCtx_set*() functions. * using ZSTD_CCtx_set*() functions.
* Pushed parameters are sticky : they are valid for next compressed frame, and any subsequent frame. * Pushed parameters are sticky : they are valid for next compressed frame, and any subsequent frame.
* "sticky" parameters are applicable to `ZSTD_compress2()` and `ZSTD_compressStream*()` ! * "sticky" parameters are applicable to `ZSTD_compress2()` and `ZSTD_compressStream*()` !
* They do not apply to "simple" one-shot variants such as ZSTD_compressCCtx() * __They do not apply to "simple" one-shot variants such as ZSTD_compressCCtx()__ .
* *
* It's possible to reset all parameters to "default" using ZSTD_CCtx_reset(). * It's possible to reset all parameters to "default" using ZSTD_CCtx_reset().
* *
* This API supercedes all other "advanced" API entry points in the experimental section. * This API supercedes all other "advanced" API entry points in the experimental section.
* In the future, we expect to remove from experimental API entry points which are redundant with this API. * In the future, we expect to remove from experimental API entry points which are redundant with this API.
*/ */
Only the order (from fast to strong) is guaranteed */ Only the order (from fast to strong) is guaranteed */
} ZSTD_strategy; } ZSTD_strategy;
typedef enum { typedef enum {
/* compression parameters /* compression parameters
* Note: When compressing with a ZSTD_CDict these parameters are superseded * Note: When compressing with a ZSTD_CDict these parameters are superseded
* by the parameters used to construct the ZSTD_CDict. See ZSTD_CCtx_refCDict() * by the parameters used to construct the ZSTD_CDict.
* for more info (superseded-by-cdict). */ * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */
ZSTD_c_compressionLevel=100, /* Update all compression parameters according to pre-defined cLevel table ZSTD_c_compressionLevel=100, /* Set compression parameters according to pre-defined cLevel table.
* Note that exact compression parameters are dynamically determined,
* depending on both compression level and srcSize (when known).
* Default level is ZSTD_CLEVEL_DEFAULT==3. * Default level is ZSTD_CLEVEL_DEFAULT==3.
* Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT. * Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT.
* Note 1 : it's possible to pass a negative compression level. * Note 1 : it's possible to pass a negative compression level.
* Note 2 : setting a level sets all default values of other compression parameters */ * Note 2 : setting a level resets all other compression parameters to default */
/* Advanced compression parameters :
* It's possible to pin down compression parameters to some specific values.
* In which case, these values are no longer dynamically selected by the compressor */
ZSTD_c_windowLog=101, /* Maximum allowed back-reference distance, expressed as power of 2. ZSTD_c_windowLog=101, /* Maximum allowed back-reference distance, expressed as power of 2.
* This will set a memory budget for streaming decompression,
* with larger values requiring more memory
* and typically compressing more.
* Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX. * Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX.
* Special: value 0 means "use default windowLog". * Special: value 0 means "use default windowLog".
* Note: Using a windowLog greater than ZSTD_WINDOWLOG_LIMIT_DEFAULT * Note: Using a windowLog greater than ZSTD_WINDOWLOG_LIMIT_DEFAULT
* requires explicitly allowing such window size at decompression stage if using streaming. */ * requires explicitly allowing such size at streaming decompression stage. */
ZSTD_c_hashLog=102, /* Size of the initial probe table, as a power of 2. ZSTD_c_hashLog=102, /* Size of the initial probe table, as a power of 2.
* Resulting memory usage is (1 << (hashLog+2)). * Resulting memory usage is (1 << (hashLog+2)).
* Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX. * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX.
* Larger tables improve compression ratio of strategies <= dFast, * Larger tables improve compression ratio of strategies <= dFast,
* and improve speed of strategies > dFast. * and improve speed of strategies > dFast.
* Special: value 0 means "use default hashLog". */ * Special: value 0 means "use default hashLog". */
ZSTD_c_chainLog=103, /* Size of the multi-probe search table, as a power of 2. ZSTD_c_chainLog=103, /* Size of the multi-probe search table, as a power of 2.
* Resulting memory usage is (1 << (chainLog+2)). * Resulting memory usage is (1 << (chainLog+2)).
* Must be clamped between ZSTD_CHAINLOG_MIN and ZSTD_CHAINLOG_MAX. * Must be clamped between ZSTD_CHAINLOG_MIN and ZSTD_CHAINLOG_MAX.
* Larger tables result in better and slower compression. * Larger tables result in better and slower compression.
* This parameter is useless when using "fast" strategy. * This parameter is useless for "fast" strategy.
* It's still useful when using "dfast" strategy, * It's still useful when using "dfast" strategy,
* in which case it defines a secondary probe table. * in which case it defines a secondary probe table.
* Special: value 0 means "use default chainLog". */ * Special: value 0 means "use default chainLog". */
ZSTD_c_searchLog=104, /* Number of search attempts, as a power of 2. ZSTD_c_searchLog=104, /* Number of search attempts, as a power of 2.
* More attempts result in better and slower compression. * More attempts result in better and slower compression.
* This parameter is useless when using "fast" and "dFast" strategies. * This parameter is useless for "fast" and "dFast" strategies.
* Special: value 0 means "use default searchLog". */ * Special: value 0 means "use default searchLog". */
ZSTD_c_minMatch=105, /* Minimum size of searched matches. ZSTD_c_minMatch=105, /* Minimum size of searched matches.
* Note that Zstandard can still find matches of smaller size, * Note that Zstandard can still find matches of smaller size,
* it just tweaks its search algorithm to look for this size and larger. * it just tweaks its search algorithm to look for this size and larger.
* Larger values increase compression and decompression speed, but decrease ratio. * Larger values increase compression and decompression speed, but decrease ratio.
* Must be clamped between ZSTD_MINMATCH_MIN and ZSTD_MINMATCH_MAX. * Must be clamped between ZSTD_MINMATCH_MIN and ZSTD_MINMATCH_MAX.
* Note that currently, for all strategies < btopt, effective minimum is 4. * Note that currently, for all strategies < btopt, effective minimum is 4.
* , for all strategies > fast, effective maximum is 6. * , for all strategies > fast, effective maximum is 6.
* Larger values improve compression speed. * Larger values improve compression speed.
* Deviating far from default value will likely result in a compression ratio decrease. * Deviating far from default value will likely result in a compression ratio decrease.
* Special: value 0 means "automatically determine hashRateLog". */ * Special: value 0 means "automatically determine hashRateLog". */
/* frame parameters */ /* frame parameters */
ZSTD_c_contentSizeFlag=200, /* Content size will be written into frame header _whenever known_ (default:1) ZSTD_c_contentSizeFlag=200, /* Content size will be written into frame header _whenever known_ (default:1)
* Content size must be known at the beginning of compression. * Content size must be known at the beginning of compression.
* This is automatically the case when using ZSTD_compress2(), * This is automatically the case when using ZSTD_compress2(),
* For streaming variants, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */ * For streaming scenarios, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */
ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */ ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */
ZSTD_c_dictIDFlag=202, /* When applicable, dictionary's ID is written into frame header (default:1) */ ZSTD_c_dictIDFlag=202, /* When applicable, dictionary's ID is written into frame header (default:1) */
/* multi-threading parameters */ /* multi-threading parameters */
/* These parameters are only useful if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD). /* These parameters are only useful if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).
* They return an error otherwise. */ * They return an error otherwise. */
ZSTD_c_nbWorkers=400, /* Select how many threads will be spawned to compress in parallel. ZSTD_c_nbWorkers=400, /* Select how many threads will be spawned to compress in parallel.
* When nbWorkers >= 1, triggers asynchronous mode when used with ZSTD_compressStream*() : * When nbWorkers >= 1, triggers asynchronous mode when used with ZSTD_compressStream*() :
* ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller, * ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller,
* while compression work is performed in parallel, within worker threads. * while compression work is performed in parallel, within worker threads.
* (note : a strong exception to this rule is when first invocation of ZSTD_compressStream2() sets ZSTD_e_end : * (note : a strong exception to this rule is when first invocation of ZSTD_compressStream2() sets ZSTD_e_end :
* in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call). * in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call).
* More workers improve speed, but also increase memory usage. * More workers improve speed, but also increase memory usage.
* Default value is `0`, aka "single-threaded mode" : no worker is spawned, compression is performed inside Caller's thread, all invocations are blocking */ * Default value is `0`, aka "single-threaded mode" : no worker is spawned, compression is performed inside Caller's thread, all invocations are blocking */
ZSTD_c_jobSize=401, /* Size of a compression job. This value is enforced only when nbWorkers >= 1. ZSTD_c_jobSize=401, /* Size of a compression job. This value is enforced only when nbWorkers >= 1.
* Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads. * Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads.
* 0 means default, which is dynamically determined based on compression parameters. * 0 means default, which is dynamically determined based on compression parameters.
* Job size must be a minimum of overlap size, or 1 MB, whichever is largest. * Job size must be a minimum of overlap size, or 1 MB, whichever is largest.
* The minimum size is automatically and transparently enforced */ * The minimum size is automatically and transparently enforced. */
ZSTD_c_overlapLog=402, /* Control the overlap size, as a fraction of window size. ZSTD_c_overlapLog=402, /* Control the overlap size, as a fraction of window size.
* The overlap size is an amount of data reloaded from previous job at the beginning of a new job. * The overlap size is an amount of data reloaded from previous job at the beginning of a new job.
* It helps preserve compression ratio, while each job is compressed in parallel. * It helps preserve compression ratio, while each job is compressed in parallel.
* This value is enforced only when nbWorkers >= 1. * This value is enforced only when nbWorkers >= 1.
* Larger values increase compression ratio, but decrease speed. * Larger values increase compression ratio, but decrease speed.
* Possible values range from 0 to 9 : * Possible values range from 0 to 9 :
* - 0 means "default" : value will be determined by the library, depending on strategy * - 0 means "default" : value will be determined by the library, depending on strategy
* - 1 means "no overlap" * - 1 means "no overlap"
* - 9 means "full overlap", using a full window size. * - 9 means "full overlap", using a full window size.
* Each intermediate rank increases/decreases load size by a factor 2 : * Each intermediate rank increases/decreases load size by a factor 2 :
* 9: full window; 8: w/2; 7: w/4; 6: w/8; 5:w/16; 4: w/32; 3:w/64; 2:w/128; 1:no overlap; 0:default * 9: full window; 8: w/2; 7: w/4; 6: w/8; 5:w/16; 4: w/32; 3:w/64; 2:w/128; 1:no overlap; 0:default
* default value varies between 6 and 9, depending on strategy */ * default value varies between 6 and 9, depending on strategy */
/* note : additional experimental parameters are also available /* note : additional experimental parameters are also available
* within the experimental section of the API. * within the experimental section of the API.
* At the time of this writing, they include : * At the time of this writing, they include :
* ZSTD_c_rsyncable * ZSTD_c_rsyncable
* ZSTD_c_format * ZSTD_c_format
* ZSTD_c_forceMaxWindow * ZSTD_c_forceMaxWindow
* ZSTD_c_forceAttachDict * ZSTD_c_forceAttachDict
* ZSTD_c_literalCompressionMode * ZSTD_c_literalCompressionMode
* ZSTD_c_targetCBlockSize * ZSTD_c_targetCBlockSize
* ZSTD_c_srcSizeHint
* Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them. * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
* note : never ever use experimentalParam? names directly; * note : never ever use experimentalParam? names directly;
* also, the enums values themselves are unstable and can still change. * also, the enums values themselves are unstable and can still change.
*/ */
ZSTD_c_experimentalParam1=500, ZSTD_c_experimentalParam1=500,
ZSTD_c_experimentalParam2=10, ZSTD_c_experimentalParam2=10,
ZSTD_c_experimentalParam3=1000, ZSTD_c_experimentalParam3=1000,
ZSTD_c_experimentalParam4=1001, ZSTD_c_experimentalParam4=1001,
ZSTD_c_experimentalParam5=1002, ZSTD_c_experimentalParam5=1002,
ZSTD_c_experimentalParam6=1003, ZSTD_c_experimentalParam6=1003,
ZSTD_c_experimentalParam7=1004
} ZSTD_cParameter; } ZSTD_cParameter;
typedef struct { typedef struct {
size_t error; size_t error;
int lowerBound; int lowerBound;
int upperBound; int upperBound;
} ZSTD_bounds; } ZSTD_bounds;
/*********************************** /***********************************
* Bulk processing dictionary API * Bulk processing dictionary API
**********************************/ **********************************/
typedef struct ZSTD_CDict_s ZSTD_CDict; typedef struct ZSTD_CDict_s ZSTD_CDict;
/*! ZSTD_createCDict() : /*! ZSTD_createCDict() :
* When compressing multiple messages / blocks using the same dictionary, it's recommended to load it only once. * When compressing multiple messages or blocks using the same dictionary,
* ZSTD_createCDict() will create a digested dictionary, ready to start future compression operations without startup cost. * it's recommended to digest the dictionary only once, since it's a costly operation.
* ZSTD_createCDict() will create a state from digesting a dictionary.
* The resulting state can be used for future compression operations with very limited startup cost.
* ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only. * ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only.
* `dictBuffer` can be released after ZSTD_CDict creation, because its content is copied within CDict. * @dictBuffer can be released after ZSTD_CDict creation, because its content is copied within CDict.
* Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate `dictBuffer` content. * Note 1 : Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate @dictBuffer content.
* Note : A ZSTD_CDict can be created from an empty dictBuffer, but it is inefficient when used to compress small data. */ * Note 2 : A ZSTD_CDict can be created from an empty @dictBuffer,
* in which case the only thing that it transports is the @compressionLevel.
* This can be useful in a pipeline featuring ZSTD_compress_usingCDict() exclusively,
* expecting a ZSTD_CDict parameter with any data, including those without a known dictionary. */
ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict(const void* dictBuffer, size_t dictSize, ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict(const void* dictBuffer, size_t dictSize,
int compressionLevel); int compressionLevel);
/*! ZSTD_freeCDict() : /*! ZSTD_freeCDict() :
* Function frees memory allocated by ZSTD_createCDict(). */ * Function frees memory allocated by ZSTD_createCDict(). */
ZSTDLIB_API size_t ZSTD_freeCDict(ZSTD_CDict* CDict); ZSTDLIB_API size_t ZSTD_freeCDict(ZSTD_CDict* CDict);
/*! ZSTD_compress_usingCDict() : /*! ZSTD_compress_usingCDict() :
* Note 1 : Prefix buffer is referenced. It **must** outlive compression. * Note 1 : Prefix buffer is referenced. It **must** outlive compression.
* Its content must remain unmodified during compression. * Its content must remain unmodified during compression.
* Note 2 : If the intention is to diff some large src data blob with some prior version of itself, * Note 2 : If the intention is to diff some large src data blob with some prior version of itself,
* ensure that the window size is large enough to contain the entire source. * ensure that the window size is large enough to contain the entire source.
* See ZSTD_c_windowLog. * See ZSTD_c_windowLog.
* Note 3 : Referencing a prefix involves building tables, which are dependent on compression parameters. * Note 3 : Referencing a prefix involves building tables, which are dependent on compression parameters.
* It's a CPU consuming operation, with non-negligible impact on latency. * It's a CPU consuming operation, with non-negligible impact on latency.
* If there is a need to use the same prefix multiple times, consider loadDictionary instead. * If there is a need to use the same prefix multiple times, consider loadDictionary instead.
* Note 4 : By default, the prefix is interpreted as raw content (ZSTD_dm_rawContent). * Note 4 : By default, the prefix is interpreted as raw content (ZSTD_dct_rawContent).
* Use experimental ZSTD_CCtx_refPrefix_advanced() to alter dictionary interpretation. */ * Use experimental ZSTD_CCtx_refPrefix_advanced() to alter dictionary interpretation. */
ZSTDLIB_API size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx, ZSTDLIB_API size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx,
const void* prefix, size_t prefixSize); const void* prefix, size_t prefixSize);
/*! ZSTD_DCtx_loadDictionary() : /*! ZSTD_DCtx_loadDictionary() :
* Create an internal DDict from dict buffer, * Create an internal DDict from dict buffer,
* to be used to decompress next frames. * to be used to decompress next frames.
* The dictionary remains valid for all future frames, until explicitly invalidated. * The dictionary remains valid for all future frames, until explicitly invalidated.
* and must use the same prefix as the one used during compression. * and must use the same prefix as the one used during compression.
* Prefix is **only used once**. Reference is discarded at end of frame. * Prefix is **only used once**. Reference is discarded at end of frame.
* End of frame is reached when ZSTD_decompressStream() returns 0. * End of frame is reached when ZSTD_decompressStream() returns 0.
* @result : 0, or an error code (which can be tested with ZSTD_isError()). * @result : 0, or an error code (which can be tested with ZSTD_isError()).
* Note 1 : Adding any prefix (including NULL) invalidates any previously set prefix or dictionary * Note 1 : Adding any prefix (including NULL) invalidates any previously set prefix or dictionary
* Note 2 : Prefix buffer is referenced. It **must** outlive decompression. * Note 2 : Prefix buffer is referenced. It **must** outlive decompression.
* Prefix buffer must remain unmodified up to the end of frame, * Prefix buffer must remain unmodified up to the end of frame,
* reached when ZSTD_decompressStream() returns 0. * reached when ZSTD_decompressStream() returns 0.
* Note 3 : By default, the prefix is treated as raw content (ZSTD_dm_rawContent). * Note 3 : By default, the prefix is treated as raw content (ZSTD_dct_rawContent).
* Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode (Experimental section) * Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode (Experimental section)
* Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost. * Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost.
* A full dictionary is more costly, as it requires building tables. * A full dictionary is more costly, as it requires building tables.
*/ */
ZSTDLIB_API size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx, ZSTDLIB_API size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx,
const void* prefix, size_t prefixSize); const void* prefix, size_t prefixSize);
/* === Memory management === */ /* === Memory management === */
**************************************************************************************** ****************************************************************************************
* The following symbols and constants * The following symbols and constants
* are not planned to join "stable API" status in the near future. * are not planned to join "stable API" status in the near future.
* They can still change in future versions. * They can still change in future versions.
* Some of them are planned to remain in the static_only section indefinitely. * Some of them are planned to remain in the static_only section indefinitely.
* Some of them might be removed in the future (especially when redundant with existing stable functions) * Some of them might be removed in the future (especially when redundant with existing stable functions)
* ***************************************************************************************/ * ***************************************************************************************/
#define ZSTD_FRAMEHEADERSIZE_PREFIX 5 /* minimum input size required to query frame header size */ #define ZSTD_FRAMEHEADERSIZE_PREFIX(format) ((format) == ZSTD_f_zstd1 ? 5 : 1) /* minimum input size required to query frame header size */
#define ZSTD_FRAMEHEADERSIZE_MIN 6 #define ZSTD_FRAMEHEADERSIZE_MIN(format) ((format) == ZSTD_f_zstd1 ? 6 : 2)
#define ZSTD_FRAMEHEADERSIZE_MAX 18 /* can be useful for static allocation */ #define ZSTD_FRAMEHEADERSIZE_MAX 18 /* can be useful for static allocation */
#define ZSTD_SKIPPABLEHEADERSIZE 8 #define ZSTD_SKIPPABLEHEADERSIZE 8
/* compression parameter bounds */ /* compression parameter bounds */
#define ZSTD_WINDOWLOG_MAX_32 30 #define ZSTD_WINDOWLOG_MAX_32 30
#define ZSTD_WINDOWLOG_MAX_64 31 #define ZSTD_WINDOWLOG_MAX_64 31
#define ZSTD_WINDOWLOG_MAX ((int)(sizeof(size_t) == 4 ? ZSTD_WINDOWLOG_MAX_32 : ZSTD_WINDOWLOG_MAX_64)) #define ZSTD_WINDOWLOG_MAX ((int)(sizeof(size_t) == 4 ? ZSTD_WINDOWLOG_MAX_32 : ZSTD_WINDOWLOG_MAX_64))
#define ZSTD_WINDOWLOG_MIN 10 #define ZSTD_WINDOWLOG_MIN 10
#define ZSTD_LDM_BUCKETSIZELOG_MIN 1 #define ZSTD_LDM_BUCKETSIZELOG_MIN 1
#define ZSTD_LDM_BUCKETSIZELOG_MAX 8 #define ZSTD_LDM_BUCKETSIZELOG_MAX 8
#define ZSTD_LDM_HASHRATELOG_MIN 0 #define ZSTD_LDM_HASHRATELOG_MIN 0
#define ZSTD_LDM_HASHRATELOG_MAX (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN) #define ZSTD_LDM_HASHRATELOG_MAX (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN)
/* Advanced parameter bounds */ /* Advanced parameter bounds */
#define ZSTD_TARGETCBLOCKSIZE_MIN 64 #define ZSTD_TARGETCBLOCKSIZE_MIN 64
#define ZSTD_TARGETCBLOCKSIZE_MAX ZSTD_BLOCKSIZE_MAX #define ZSTD_TARGETCBLOCKSIZE_MAX ZSTD_BLOCKSIZE_MAX
#define ZSTD_SRCSIZEHINT_MIN 0
#define ZSTD_SRCSIZEHINT_MAX INT_MAX
/* internal */ /* internal */
#define ZSTD_HASHLOG3_MAX 17 #define ZSTD_HASHLOG3_MAX 17
/* --- Advanced types --- */ /* --- Advanced types --- */
typedef struct ZSTD_CCtx_params_s ZSTD_CCtx_params; typedef struct ZSTD_CCtx_params_s ZSTD_CCtx_params;
typedef struct { typedef struct {
unsigned int matchPos; /* Match pos in dst */
/* If seqDef.offset > 3, then this is seqDef.offset - 3
* If seqDef.offset < 3, then this is the corresponding repeat offset
* But if seqDef.offset < 3 and litLength == 0, this is the
* repeat offset before the corresponding repeat offset
* And if seqDef.offset == 3 and litLength == 0, this is the
* most recent repeat offset - 1
*/
unsigned int offset;
unsigned int litLength; /* Literal length */
unsigned int matchLength; /* Match length */
/* 0 when seq not rep and seqDef.offset otherwise
* when litLength == 0 this will be <= 4, otherwise <= 3 like normal
*/
unsigned int rep;
} ZSTD_Sequence;
typedef struct {
unsigned windowLog; /**< largest match distance : larger == more compression, more memory needed during decompression */ unsigned windowLog; /**< largest match distance : larger == more compression, more memory needed during decompression */
unsigned chainLog; /**< fully searched segment : larger == more compression, slower, more memory (useless for fast) */ unsigned chainLog; /**< fully searched segment : larger == more compression, slower, more memory (useless for fast) */
unsigned hashLog; /**< dispatch table : larger == faster, more memory */ unsigned hashLog; /**< dispatch table : larger == faster, more memory */
unsigned searchLog; /**< nb of searches : larger == more compression, slower */ unsigned searchLog; /**< nb of searches : larger == more compression, slower */
unsigned minMatch; /**< match length searched : larger == faster decompression, sometimes less compression */ unsigned minMatch; /**< match length searched : larger == faster decompression, sometimes less compression */
unsigned targetLength; /**< acceptable match size for optimal parser (only) : larger == more compression, slower */ unsigned targetLength; /**< acceptable match size for optimal parser (only) : larger == more compression, slower */
ZSTD_strategy strategy; /**< see ZSTD_strategy definition above */ ZSTD_strategy strategy; /**< see ZSTD_strategy definition above */
} ZSTD_compressionParameters; } ZSTD_compressionParameters;
typedef enum { typedef enum {
ZSTD_dct_auto = 0, /* dictionary is "full" when starting with ZSTD_MAGIC_DICTIONARY, otherwise it is "rawContent" */ ZSTD_dct_auto = 0, /* dictionary is "full" when starting with ZSTD_MAGIC_DICTIONARY, otherwise it is "rawContent" */
ZSTD_dct_rawContent = 1, /* ensures dictionary is always loaded as rawContent, even if it starts with ZSTD_MAGIC_DICTIONARY */ ZSTD_dct_rawContent = 1, /* ensures dictionary is always loaded as rawContent, even if it starts with ZSTD_MAGIC_DICTIONARY */
ZSTD_dct_fullDict = 2 /* refuses to load a dictionary if it does not respect Zstandard's specification, starting with ZSTD_MAGIC_DICTIONARY */ ZSTD_dct_fullDict = 2 /* refuses to load a dictionary if it does not respect Zstandard's specification, starting with ZSTD_MAGIC_DICTIONARY */
} ZSTD_dictContentType_e; } ZSTD_dictContentType_e;
typedef enum { typedef enum {
ZSTD_dlm_byCopy = 0, /**< Copy dictionary content internally */ ZSTD_dlm_byCopy = 0, /**< Copy dictionary content internally */
ZSTD_dlm_byRef = 1, /**< Reference dictionary content -- the dictionary buffer must outlive its users. */ ZSTD_dlm_byRef = 1 /**< Reference dictionary content -- the dictionary buffer must outlive its users. */
} ZSTD_dictLoadMethod_e; } ZSTD_dictLoadMethod_e;
typedef enum { typedef enum {
/* Opened question : should we have a format ZSTD_f_auto ?
* Today, it would mean exactly the same as ZSTD_f_zstd1.
* But, in the future, should several formats become supported,
* on the compression side, it would mean "default format".
* On the decompression side, it would mean "automatic format detection",
* so that ZSTD_f_zstd1 would mean "accept *only* zstd frames".
* Since meaning is a little different, another option could be to define different enums for compression and decompression.
* This question could be kept for later, when there are actually multiple formats to support,
* but there is also the question of pinning enum values, and pinning value `0` is especially important */
ZSTD_f_zstd1 = 0, /* zstd frame format, specified in zstd_compression_format.md (default) */ ZSTD_f_zstd1 = 0, /* zstd frame format, specified in zstd_compression_format.md (default) */
ZSTD_f_zstd1_magicless = 1, /* Variant of zstd frame format, without initial 4-bytes magic number. ZSTD_f_zstd1_magicless = 1 /* Variant of zstd frame format, without initial 4-bytes magic number.
* Useful to save 4 bytes per generated frame. * Useful to save 4 bytes per generated frame.
* Decoder cannot recognise automatically this format, requiring this instruction. */ * Decoder cannot recognise automatically this format, requiring this instruction. */
} ZSTD_format_e; } ZSTD_format_e;
typedef enum { typedef enum {
/* Note: this enum and the behavior it controls are effectively internal /* Note: this enum and the behavior it controls are effectively internal
* implementation details of the compressor. They are expected to continue * implementation details of the compressor. They are expected to continue
* to evolve and should be considered only in the context of extremely * to evolve and should be considered only in the context of extremely
* advanced performance tuning. * advanced performance tuning.
* *
* Zstd currently supports the use of a CDict in two ways: * Zstd currently supports the use of a CDict in three ways:
* *
* - The contents of the CDict can be copied into the working context. This * - The contents of the CDict can be copied into the working context. This
* means that the compression can search both the dictionary and input * means that the compression can search both the dictionary and input
* while operating on a single set of internal tables. This makes * while operating on a single set of internal tables. This makes
* the compression faster per-byte of input. However, the initial copy of * the compression faster per-byte of input. However, the initial copy of
* the CDict's tables incurs a fixed cost at the beginning of the * the CDict's tables incurs a fixed cost at the beginning of the
* compression. For small compressions (< 8 KB), that copy can dominate * compression. For small compressions (< 8 KB), that copy can dominate
* the cost of the compression. * the cost of the compression.
* *
* - The CDict's tables can be used in-place. In this model, compression is * - The CDict's tables can be used in-place. In this model, compression is
* slower per input byte, because the compressor has to search two sets of * slower per input byte, because the compressor has to search two sets of
* tables. However, this model incurs no start-up cost (as long as the * tables. However, this model incurs no start-up cost (as long as the
* working context's tables can be reused). For small inputs, this can be * working context's tables can be reused). For small inputs, this can be
* faster than copying the CDict's tables. * faster than copying the CDict's tables.
* *
* - The CDict's tables are not used at all, and instead we use the working
* context alone to reload the dictionary and use params based on the source
* size. See ZSTD_compress_insertDictionary() and ZSTD_compress_usingDict().
* This method is effective when the dictionary sizes are very small relative
* to the input size, and the input size is fairly large to begin with.
*
* Zstd has a simple internal heuristic that selects which strategy to use * Zstd has a simple internal heuristic that selects which strategy to use
* at the beginning of a compression. However, if experimentation shows that * at the beginning of a compression. However, if experimentation shows that
* Zstd is making poor choices, it is possible to override that choice with * Zstd is making poor choices, it is possible to override that choice with
* this enum. * this enum.
*/ */
ZSTD_dictDefaultAttach = 0, /* Use the default heuristic. */ ZSTD_dictDefaultAttach = 0, /* Use the default heuristic. */
ZSTD_dictForceAttach = 1, /* Never copy the dictionary. */ ZSTD_dictForceAttach = 1, /* Never copy the dictionary. */
ZSTD_dictForceCopy = 2, /* Always copy the dictionary. */ ZSTD_dictForceCopy = 2, /* Always copy the dictionary. */
ZSTD_dictForceLoad = 3 /* Always reload the dictionary */
} ZSTD_dictAttachPref_e; } ZSTD_dictAttachPref_e;
typedef enum { typedef enum {
ZSTD_lcm_auto = 0, /**< Automatically determine the compression mode based on the compression level. ZSTD_lcm_auto = 0, /**< Automatically determine the compression mode based on the compression level.
* Negative compression levels will be uncompressed, and positive compression * Negative compression levels will be uncompressed, and positive compression
* levels will be compressed. */ * levels will be compressed. */
ZSTD_lcm_huffman = 1, /**< Always attempt Huffman compression. Uncompressed literals will still be ZSTD_lcm_huffman = 1, /**< Always attempt Huffman compression. Uncompressed literals will still be
* emitted if Huffman compression is not profitable. */ * emitted if Huffman compression is not profitable. */
ZSTD_lcm_uncompressed = 2, /**< Always emit uncompressed literals. */ ZSTD_lcm_uncompressed = 2 /**< Always emit uncompressed literals. */
} ZSTD_literalCompressionMode_e; } ZSTD_literalCompressionMode_e;
/*************************************** /***************************************
* Frame size functions * Frame size functions
***************************************/ ***************************************/
/*! ZSTD_findDecompressedSize() : /*! ZSTD_findDecompressedSize() :
ZSTDLIB_API unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize); ZSTDLIB_API unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize);
/*! ZSTD_frameHeaderSize() : /*! ZSTD_frameHeaderSize() :
* srcSize must be >= ZSTD_FRAMEHEADERSIZE_PREFIX. * srcSize must be >= ZSTD_FRAMEHEADERSIZE_PREFIX.
* @return : size of the Frame Header, * @return : size of the Frame Header,
* or an error code (if srcSize is too small) */ * or an error code (if srcSize is too small) */
ZSTDLIB_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize); ZSTDLIB_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize);
/*! ZSTD_getSequences() :
* Extract sequences from the sequence store
* zc can be used to insert custom compression params.
* This function invokes ZSTD_compress2
* @return : number of sequences extracted
*/
ZSTDLIB_API size_t ZSTD_getSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
size_t outSeqsSize, const void* src, size_t srcSize);
/*************************************** /***************************************
* Memory management * Memory management
***************************************/ ***************************************/
/*! ZSTD_estimate*() : /*! ZSTD_estimate*() :
* These functions make it possible to estimate memory usage * These functions make it possible to estimate memory usage of a future
* of a future {D,C}Ctx, before its creation. * {D,C}Ctx, before its creation.
* ZSTD_estimateCCtxSize() will provide a budget large enough for any compression level up to selected one. *
* It will also consider src size to be arbitrarily "large", which is worst case. * ZSTD_estimateCCtxSize() will provide a budget large enough for any
* If srcSize is known to always be small, ZSTD_estimateCCtxSize_usingCParams() can provide a tighter estimation. * compression level up to selected one. Unlike ZSTD_estimateCStreamSize*(),
* ZSTD_estimateCCtxSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel. * this estimate does not include space for a window buffer, so this estimate
* ZSTD_estimateCCtxSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_c_nbWorkers is >= 1. * is guaranteed to be enough for single-shot compressions, but not streaming
* Note : CCtx size estimation is only correct for single-threaded compression. */ * compressions. It will however assume the input may be arbitrarily large,
* which is the worst case. If srcSize is known to always be small,
* ZSTD_estimateCCtxSize_usingCParams() can provide a tighter estimation.
* ZSTD_estimateCCtxSize_usingCParams() can be used in tandem with
* ZSTD_getCParams() to create cParams from compressionLevel.
* ZSTD_estimateCCtxSize_usingCCtxParams() can be used in tandem with
* ZSTD_CCtxParams_setParameter().
*
* Note: only single-threaded compression is supported. This function will
* return an error code if ZSTD_c_nbWorkers is >= 1. */
ZSTDLIB_API size_t ZSTD_estimateCCtxSize(int compressionLevel); ZSTDLIB_API size_t ZSTD_estimateCCtxSize(int compressionLevel);
ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams); ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams);
ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params); ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params);
ZSTDLIB_API size_t ZSTD_estimateDCtxSize(void); ZSTDLIB_API size_t ZSTD_estimateDCtxSize(void);
/*! ZSTD_estimateCStreamSize() : /*! ZSTD_estimateCStreamSize() :
* ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one. * ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one.
* It will also consider src size to be arbitrarily "large", which is worst case. * It will also consider src size to be arbitrarily "large", which is worst case.
/*************************************** /***************************************
* Advanced compression functions * Advanced compression functions
***************************************/ ***************************************/
/*! ZSTD_createCDict_byReference() : /*! ZSTD_createCDict_byReference() :
* Create a digested dictionary for compression * Create a digested dictionary for compression
* Dictionary content is just referenced, not duplicated. * Dictionary content is just referenced, not duplicated.
* As a consequence, `dictBuffer` **must** outlive CDict, * As a consequence, `dictBuffer` **must** outlive CDict,
* and its content must remain unmodified throughout the lifetime of CDict. */ * and its content must remain unmodified throughout the lifetime of CDict.
* note: equivalent to ZSTD_createCDict_advanced(), with dictLoadMethod==ZSTD_dlm_byRef */
ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_byReference(const void* dictBuffer, size_t dictSize, int compressionLevel); ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_byReference(const void* dictBuffer, size_t dictSize, int compressionLevel);
/*! ZSTD_getCParams() : /*! ZSTD_getCParams() :
* @return ZSTD_compressionParameters structure for a selected compression level and estimated srcSize. * @return ZSTD_compressionParameters structure for a selected compression level and estimated srcSize.
* `estimatedSrcSize` value is optional, select 0 if not known */ * `estimatedSrcSize` value is optional, select 0 if not known */
ZSTDLIB_API ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize); ZSTDLIB_API ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize);
/*! ZSTD_getParams() : /*! ZSTD_getParams() :
* optimize params for a given `srcSize` and `dictSize`. * optimize params for a given `srcSize` and `dictSize`.
* `srcSize` can be unknown, in which case use ZSTD_CONTENTSIZE_UNKNOWN. * `srcSize` can be unknown, in which case use ZSTD_CONTENTSIZE_UNKNOWN.
* `dictSize` must be `0` when there is no dictionary. * `dictSize` must be `0` when there is no dictionary.
* cPar can be invalid : all parameters will be clamped within valid range in the @return struct. * cPar can be invalid : all parameters will be clamped within valid range in the @return struct.
* This function never fails (wide contract) */ * This function never fails (wide contract) */
ZSTDLIB_API ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize); ZSTDLIB_API ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize);
/*! ZSTD_compress_advanced() : /*! ZSTD_compress_advanced() :
* Same as ZSTD_compress_usingDict(), with fine-tune control over compression parameters (by structure) */ * Note : this function is now DEPRECATED.
* It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_setParameter() and other parameter setters.
* This prototype will be marked as deprecated and generate compilation warning on reaching v1.5.x */
ZSTDLIB_API size_t ZSTD_compress_advanced(ZSTD_CCtx* cctx, ZSTDLIB_API size_t ZSTD_compress_advanced(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize, const void* src, size_t srcSize,
const void* dict,size_t dictSize, const void* dict,size_t dictSize,
ZSTD_parameters params); ZSTD_parameters params);
/*! ZSTD_compress_usingCDict_advanced() : /*! ZSTD_compress_usingCDict_advanced() :
* Same as ZSTD_compress_usingCDict(), with fine-tune control over frame parameters */ * Note : this function is now REDUNDANT.
* It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_loadDictionary() and other parameter setters.
* This prototype will be marked as deprecated and generate compilation warning in some future version */
ZSTDLIB_API size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx, ZSTDLIB_API size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
const void* src, size_t srcSize, const void* src, size_t srcSize,
const ZSTD_CDict* cdict, const ZSTD_CDict* cdict,
ZSTD_frameParameters fParams); ZSTD_frameParameters fParams);
/*! ZSTD_CCtx_loadDictionary_byReference() : /*! ZSTD_CCtx_loadDictionary_byReference() :
*/ */
#define ZSTD_c_literalCompressionMode ZSTD_c_experimentalParam5 #define ZSTD_c_literalCompressionMode ZSTD_c_experimentalParam5
/* Tries to fit compressed block size to be around targetCBlockSize. /* Tries to fit compressed block size to be around targetCBlockSize.
* No target when targetCBlockSize == 0. * No target when targetCBlockSize == 0.
* There is no guarantee on compressed block size (default:0) */ * There is no guarantee on compressed block size (default:0) */
#define ZSTD_c_targetCBlockSize ZSTD_c_experimentalParam6 #define ZSTD_c_targetCBlockSize ZSTD_c_experimentalParam6
/* User's best guess of source size.
* Hint is not valid when srcSizeHint == 0.
* There is no guarantee that hint is close to actual source size,
* but compression ratio may regress significantly if guess considerably underestimates */
#define ZSTD_c_srcSizeHint ZSTD_c_experimentalParam7
/*! ZSTD_CCtx_getParameter() : /*! ZSTD_CCtx_getParameter() :
* Get the requested compression parameter value, selected by enum ZSTD_cParameter, * Get the requested compression parameter value, selected by enum ZSTD_cParameter,
* and store it into int* value. * and store it into int* value.
* @return : 0, or an error code (which can be tested with ZSTD_isError()). * @return : 0, or an error code (which can be tested with ZSTD_isError()).
*/ */
ZSTDLIB_API size_t ZSTD_CCtx_getParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int* value); ZSTDLIB_API size_t ZSTD_CCtx_getParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int* value);
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any) * ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any)
* ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); * ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
* ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
* *
* pledgedSrcSize must be correct. If it is not known at init time, use * pledgedSrcSize must be correct. If it is not known at init time, use
* ZSTD_CONTENTSIZE_UNKNOWN. Note that, for compatibility with older programs, * ZSTD_CONTENTSIZE_UNKNOWN. Note that, for compatibility with older programs,
* "0" also disables frame content size field. It may be enabled in the future. * "0" also disables frame content size field. It may be enabled in the future.
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/ */
ZSTDLIB_API size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, int compressionLevel, unsigned long long pledgedSrcSize); ZSTDLIB_API size_t
ZSTD_initCStream_srcSize(ZSTD_CStream* zcs,
int compressionLevel,
unsigned long long pledgedSrcSize);
/**! ZSTD_initCStream_usingDict() : /**! ZSTD_initCStream_usingDict() :
* This function is deprecated, and is equivalent to: * This function is deprecated, and is equivalent to:
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel); * ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
* ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); * ZSTD_CCtx_loadDictionary(zcs, dict, dictSize);
* *
* Creates of an internal CDict (incompatible with static CCtx), except if * Creates of an internal CDict (incompatible with static CCtx), except if
* dict == NULL or dictSize < 8, in which case no dict is used. * dict == NULL or dictSize < 8, in which case no dict is used.
* Note: dict is loaded with ZSTD_dm_auto (treated as a full zstd dictionary if * Note: dict is loaded with ZSTD_dct_auto (treated as a full zstd dictionary if
* it begins with ZSTD_MAGIC_DICTIONARY, else as raw content) and ZSTD_dlm_byCopy. * it begins with ZSTD_MAGIC_DICTIONARY, else as raw content) and ZSTD_dlm_byCopy.
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/ */
ZSTDLIB_API size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel); ZSTDLIB_API size_t
ZSTD_initCStream_usingDict(ZSTD_CStream* zcs,
const void* dict, size_t dictSize,
int compressionLevel);
/**! ZSTD_initCStream_advanced() : /**! ZSTD_initCStream_advanced() :
* This function is deprecated, and is approximately equivalent to: * This function is deprecated, and is approximately equivalent to:
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* ZSTD_CCtx_setZstdParams(zcs, params); // Set the zstd params and leave the rest as-is * // Pseudocode: Set each zstd parameter and leave the rest as-is.
* for ((param, value) : params) {
* ZSTD_CCtx_setParameter(zcs, param, value);
* }
* ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
* ZSTD_CCtx_loadDictionary(zcs, dict, dictSize); * ZSTD_CCtx_loadDictionary(zcs, dict, dictSize);
* *
* pledgedSrcSize must be correct. If srcSize is not known at init time, use * dict is loaded with ZSTD_dct_auto and ZSTD_dlm_byCopy.
* value ZSTD_CONTENTSIZE_UNKNOWN. dict is loaded with ZSTD_dm_auto and ZSTD_dlm_byCopy. * pledgedSrcSize must be correct.
* If srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN.
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/ */
ZSTDLIB_API size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs, const void* dict, size_t dictSize, ZSTDLIB_API size_t
ZSTD_parameters params, unsigned long long pledgedSrcSize); ZSTD_initCStream_advanced(ZSTD_CStream* zcs,
const void* dict, size_t dictSize,
ZSTD_parameters params,
unsigned long long pledgedSrcSize);
/**! ZSTD_initCStream_usingCDict() : /**! ZSTD_initCStream_usingCDict() :
* This function is deprecated, and equivalent to: * This function is deprecated, and equivalent to:
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* ZSTD_CCtx_refCDict(zcs, cdict); * ZSTD_CCtx_refCDict(zcs, cdict);
* *
* note : cdict will just be referenced, and must outlive compression session * note : cdict will just be referenced, and must outlive compression session
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/ */
ZSTDLIB_API size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict); ZSTDLIB_API size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict);
/**! ZSTD_initCStream_usingCDict_advanced() : /**! ZSTD_initCStream_usingCDict_advanced() :
* This function is deprecated, and is approximately equivalent to: * This function is DEPRECATED, and is approximately equivalent to:
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* ZSTD_CCtx_setZstdFrameParams(zcs, fParams); // Set the zstd frame params and leave the rest as-is * // Pseudocode: Set each zstd frame parameter and leave the rest as-is.
* for ((fParam, value) : fParams) {
* ZSTD_CCtx_setParameter(zcs, fParam, value);
* }
* ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
* ZSTD_CCtx_refCDict(zcs, cdict); * ZSTD_CCtx_refCDict(zcs, cdict);
* *
* same as ZSTD_initCStream_usingCDict(), with control over frame parameters. * same as ZSTD_initCStream_usingCDict(), with control over frame parameters.
* pledgedSrcSize must be correct. If srcSize is not known at init time, use * pledgedSrcSize must be correct. If srcSize is not known at init time, use
* value ZSTD_CONTENTSIZE_UNKNOWN. * value ZSTD_CONTENTSIZE_UNKNOWN.
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/ */
ZSTDLIB_API size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, const ZSTD_CDict* cdict, ZSTD_frameParameters fParams, unsigned long long pledgedSrcSize); ZSTDLIB_API size_t
ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs,
const ZSTD_CDict* cdict,
ZSTD_frameParameters fParams,
unsigned long long pledgedSrcSize);
/*! ZSTD_resetCStream() : /*! ZSTD_resetCStream() :
* This function is deprecated, and is equivalent to: * This function is deprecated, and is equivalent to:
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only); * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize); * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
* *
* start a new frame, using same parameters from previous frame. * start a new frame, using same parameters from previous frame.
* This is typically useful to skip dictionary loading stage, since it will re-use it in-place. * This is typically useful to skip dictionary loading stage, since it will re-use it in-place.
* Note that zcs must be init at least once before using ZSTD_resetCStream(). * Note that zcs must be init at least once before using ZSTD_resetCStream().
* If pledgedSrcSize is not known at reset time, use macro ZSTD_CONTENTSIZE_UNKNOWN. * If pledgedSrcSize is not known at reset time, use macro ZSTD_CONTENTSIZE_UNKNOWN.
* If pledgedSrcSize > 0, its value must be correct, as it will be written in header, and controlled at the end. * If pledgedSrcSize > 0, its value must be correct, as it will be written in header, and controlled at the end.
* For the time being, pledgedSrcSize==0 is interpreted as "srcSize unknown" for compatibility with older programs, * For the time being, pledgedSrcSize==0 is interpreted as "srcSize unknown" for compatibility with older programs,
* but it will change to mean "empty" in future version, so use macro ZSTD_CONTENTSIZE_UNKNOWN instead. * but it will change to mean "empty" in future version, so use macro ZSTD_CONTENTSIZE_UNKNOWN instead.
* @return : 0, or an error code (which can be tested using ZSTD_isError()) * @return : 0, or an error code (which can be tested using ZSTD_isError())
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/ */
ZSTDLIB_API size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize); ZSTDLIB_API size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize);
typedef struct { typedef struct {
unsigned long long ingested; /* nb input bytes read and buffered */ unsigned long long ingested; /* nb input bytes read and buffered */
unsigned long long consumed; /* nb input bytes actually compressed */ unsigned long long consumed; /* nb input bytes actually compressed */
unsigned long long produced; /* nb of compressed bytes generated and buffered */ unsigned long long produced; /* nb of compressed bytes generated and buffered */
/*===== Advanced Streaming decompression functions =====*/ /*===== Advanced Streaming decompression functions =====*/
/** /**
* This function is deprecated, and is equivalent to: * This function is deprecated, and is equivalent to:
* *
* ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
* ZSTD_DCtx_loadDictionary(zds, dict, dictSize); * ZSTD_DCtx_loadDictionary(zds, dict, dictSize);
* *
* note: no dictionary will be used if dict == NULL or dictSize < 8 * note: no dictionary will be used if dict == NULL or dictSize < 8
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/ */
ZSTDLIB_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize); ZSTDLIB_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize);
/** /**
* This function is deprecated, and is equivalent to: * This function is deprecated, and is equivalent to:
* *
* ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
* ZSTD_DCtx_refDDict(zds, ddict); * ZSTD_DCtx_refDDict(zds, ddict);
* *
* note : ddict is referenced, it must outlive decompression session * note : ddict is referenced, it must outlive decompression session
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/ */
ZSTDLIB_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict); ZSTDLIB_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict);
/** /**
* This function is deprecated, and is equivalent to: * This function is deprecated, and is equivalent to:
* *
* ZSTD_DCtx_reset(zds, ZSTD_reset_session_only); * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
* *
* re-use decompression parameters from previous init; saves dictionary loading * re-use decompression parameters from previous init; saves dictionary loading
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/ */
ZSTDLIB_API size_t ZSTD_resetDStream(ZSTD_DStream* zds); ZSTDLIB_API size_t ZSTD_resetDStream(ZSTD_DStream* zds);
/********************************************************************* /*********************************************************************
* Buffer-less and synchronous inner streaming functions * Buffer-less and synchronous inner streaming functions
* *
* This is an advanced API, giving full control over buffer management, for users which need direct control over memory. * This is an advanced API, giving full control over buffer management, for users which need direct control over memory.
/* ============================ */ /* ============================ */
/** Block level API */ /** Block level API */
/* ============================ */ /* ============================ */
/*! /*!
Block functions produce and decode raw zstd blocks, without frame metadata. Block functions produce and decode raw zstd blocks, without frame metadata.
Frame metadata cost is typically ~18 bytes, which can be non-negligible for very small blocks (< 100 bytes). Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 100 bytes).
But users will have to take in charge needed metadata to regenerate data, such as compressed and content sizes. But users will have to take in charge needed metadata to regenerate data, such as compressed and content sizes.
A few rules to respect : A few rules to respect :
- Compressing and decompressing require a context structure - Compressing and decompressing require a context structure
+ Use ZSTD_createCCtx() and ZSTD_createDCtx() + Use ZSTD_createCCtx() and ZSTD_createDCtx()
- It is necessary to init context before starting - It is necessary to init context before starting
+ compression : any ZSTD_compressBegin*() variant, including with dictionary + compression : any ZSTD_compressBegin*() variant, including with dictionary
+ decompression : any ZSTD_decompressBegin*() variant, including with dictionary + decompression : any ZSTD_decompressBegin*() variant, including with dictionary