XZ Embedded

XZ Embedded is a relatively small, limited implementation of the .xz
file format. Currently only decoding is implemented.

XZ Embedded was written for use in the Linux kernel, but the code can
be easily used in other environments too, including regular userspace
applications. See userspace/xzminidec.c for an example program.

    NOTE: The version of XZ Embedded in the Linux kernel lacks a few
    build-time-selectable optional features that are present in the
    upstream XZ Embedded project: support for concatated .xz files,
    CRC64, and ignoring unsupported check. These aren't in Linux
    because they don't seem useful there but they would add to the
    code size.

This README contains information that is useful only when the copy
of XZ Embedded isn't part of the Linux kernel tree. You should also
read linux/Documentation/staging/xz.rst even if you aren't using
XZ Embedded as part of Linux; information in that file is not
repeated in this README.

Conformance to the .xz file format specification

As of the .xz file format specification version 1.2.0, this
decompressor implementation has the following limitations:

  - Delta filter is not included.

  - BCJ filters don't support non-default start offset.

  - LZMA2 supports at most 3 GiB dictionary.

There are a couple of corner cases where things have been simplified
at expense of detecting errors as early as possible. These should not
matter in practice at all since they don't cause security issues. But
it is good to know this if testing the code with the test files from
XZ Utils.

Compiler requirements

XZ Embedded should compile with any C99 or C11 compiler. The code
should also be GNU-C89 compatible still. GNU-C89 was used in the
Linux kernel until 2022. GNU-C89 support likely will be dropped
at some point.

Embedding into userspace applications

To embed the XZ decoder, copy the following files into a single
directory in your source code tree:

    linux/include/linux/xz.h
    linux/lib/xz/xz_crc32.c
    linux/lib/xz/xz_dec_lzma2.c
    linux/lib/xz/xz_dec_stream.c
    linux/lib/xz/xz_lzma2.h
    linux/lib/xz/xz_private.h
    linux/lib/xz/xz_stream.h
    userspace/xz_config.h

Alternatively, xz.h may be placed into a different directory but then
that directory must be in the compiler include path when compiling
the .c files.

Your code should use only the functions declared in xz.h. The rest of
the .h files are meant only for internal use in XZ Embedded.

You may want to modify xz_config.h to be more suitable for your build
environment. Probably you should at least skim through it even if the
default file works as is.

Supporting concatenated .xz files

Regular .xz files can be concatenated as is and the xz command line
tool will decompress all streams from a concatenated file (a few
other popular formats and tools support this too). This kind of .xz
files are more common than one might think because pxz, an early
threaded XZ compressor, created this kind of .xz files.

The xz_dec_run() function will stop after decompressing one stream.
This is good when XZ data is stored inside some other file format.
However, if one is decompressing regular standalone .xz files, one
will want to decompress all streams in the file. This is easy with
xz_dec_catrun(). To include support for xz_dec_catrun(), you need
to #define XZ_DEC_CONCATENATED in xz_config.h or in compiler flags.

Integrity check support

XZ Embedded always supports the integrity check types None and
CRC32. Support for CRC64 and SHA-256 is optional. The xz tool
from XZ Utils uses CRC64 by default, but CRC32 is usually
enough in embedded systems to keep the code size smaller.

If you want support for CRC64, you need to copy linux/lib/xz/xz_crc64.c
into your application, and #define XZ_USE_CRC64 in xz_config.h or in
compiler flags.

When using the internal CRC32 or CRC64, their lookup tables need to be
initialized with xz_crc32_init() and xz_crc64_init(), respectively.
See xz.h for details.

To use external CRC32 or CRC64 code instead of the code from
xz_crc32.c or xz_crc64.c, the following #defines may be used
in xz_config.h or in compiler flags:

    #define XZ_INTERNAL_CRC32 0
    #define XZ_INTERNAL_CRC64 0

Then it is up to you to provide compatible xz_crc32() or xz_crc64()
functions.

If the .xz file being decompressed uses an integrity check type that
isn't supported by XZ Embedded, it is treated as an error and the
file cannot be decompressed. For multi-call mode, this can be modified
by #defining XZ_DEC_ANY_CHECK. Then xz_dec_run() will return
XZ_UNSUPPORTED_CHECK when unsupported check type is detected. After
that decompression can be continued normally except that the
integrity check won't be verified. In single-call mode there's
no way to continue decoding, so XZ_DEC_ANY_CHECK is almost useless
in single-call mode.

BCJ filter support

If you want support for one or more BCJ filters, you need to copy
linux/lib/xz/xz_dec_bcj.c into your application, and use appropriate
#defines in xz_config.h or in compiler flags. You don't need these
#defines in the code that just uses XZ Embedded via xz.h, but having
them always #defined doesn't hurt either.

    #define             Instruction set     BCJ filter endianness
    XZ_DEC_X86          x86-32 or x86-64    Little endian only
    XZ_DEC_POWERPC      PowerPC             Big endian only
    XZ_DEC_IA64         Itanium (IA-64)     Big or little endian
    XZ_DEC_ARM          ARM                 Little endian instructions
    XZ_DEC_ARMTHUMB     ARM-Thumb           Big or little endian
    XZ_DEC_ARM64        ARM64               Big or little endian
    XZ_DEC_SPARC        SPARC               Big or little endian
    XZ_DEC_RISCV        RISC-V              Big or little endian

While some architectures are (partially) bi-endian, the endianness
setting doesn't change the endianness of the instructions on all
architectures. That's why many filters work for both big and little
endian executables (Itanium and ARM based architectures have little
endian instructions and SPARC has big endian instructions).

Notes about shared libraries

If you are including XZ Embedded into a shared library, you should
rename the xz_* functions to prevent symbol conflicts in case your
library is linked against some other library or application that
also has XZ Embedded in it (which may even be a different version
of XZ Embedded).

Please don't create a shared library of XZ Embedded itself unless
it is fine to rebuild everything depending on that shared library
every time you upgrade to a newer version of XZ Embedded. There are
no API or ABI stability guarantees between different versions of
XZ Embedded.

Contact information

Email: Lasse Collin <lasse.collin@tukaani.org>
IRC: Larhzu on #tukaani on Libera Chat
GitHub: https://github.com/tukaani-project/xz-embedded