lz4.stream sub-package¶
This sub-package provides the capability to compress and decompress data using the stream specification, especially the stream specification based on a double buffer.
Because the LZ4 stream format does not define a container format, the Python bindings will by default insert the compressed data size as an integer at the start of the compressed payload. However, it is possible to set the bit depth of this compressed data size.
So far, only the double-buffer based approach is implemented.
Example usage¶
To use the lz4 stream format bindings is straightforward:
>>> from lz4.stream import LZ4StreamCompressor, LZ4StreamDecompressor
>>> import os
>>> block_size_length = 2 # LZ4 compressed block size stored on 2 bytes
>>> page_size = 8192 # LZ4 context double buffer page size
>>> origin_stream = 10 * 1024 * os.urandom(1024) # 10MiB
>>> # LZ4 stream compression of origin_stream into compressed_stream:
>>> compressed_stream = bytearray()
>>> with LZ4StreamCompressor("double_buffer", page_size, store_comp_size=block_size_length) as proc:
... offset = 0
... while offset < len(origin_stream):
... chunk = origin_stream[offset:offset + page_size]
... block = proc.compress(chunk)
... compressed_stream.extend(block)
... offset += page_size
>>> # LZ4 stream decompression of compressed_stream into decompressed_stream:
>>> decompressed_stream = bytearray()
>>> with LZ4StreamDecompressor("double_buffer", page_size, store_comp_size=block_size_length) as proc:
... offset = 0
... while offset < len(compressed_stream):
... block = proc.get_block(compressed_stream[offset:])
... chunk = proc.decompress(block)
... decompressed_stream.extend(chunk)
... offset += block_size_length + len(block)
>>> decompressed_stream == origin_stream
True
Contents¶
A Python wrapper for the LZ4 stream protocol.
- class lz4.stream.LZ4StreamCompressor(strategy, buffer_size, mode='default', acceleration=True, compression_level=9, return_bytearray=False, store_comp_size=4, dictionary='')¶
LZ4 stream compressing context.
- __enter__()¶
Enter the LZ4 stream context.
- __exit__(exc_type, exc, exc_tb)¶
Exit the LZ4 stream context.
- compress(chunk)¶
Stream compress given
chunk
of data.Compress the given
chunk
, using the given LZ4 stream context, returning the compressed data as abytearray
or as abytes
object.- Parameters
chunk (str, bytes or buffer-compatible object) – Data to compress
- Returns
Compressed data.
- Return type
bytes or bytearray
- Raises
Exceptions occuring during compression. –
OverflowError – raised if the source is too large for being compressed in the given context.
LZ4StreamError – raised if the call to the LZ4 library fails.
- class lz4.stream.LZ4StreamDecompressor(strategy, buffer_size, return_bytearray=False, store_comp_size=4, dictionary='')¶
LZ4 stream decompression context.
- __enter__()¶
Enter the LZ4 stream context.
- __exit__(exc_type, exc, exc_tb)¶
Exit the LZ4 stream context.
- decompress(chunk)¶
Decompress streamed compressed data.
Decompress the given
chunk
, using the given LZ4 stream context, Raises an exception if any error occurs.- Parameters
chunk (str, bytes or buffer-compatible object) – Data to decompress
- Returns
Decompressed data.
- Return type
bytes or bytearray
- Raises
Exceptions occuring during decompression. –
ValueError – raised if the source is inconsistent with a finite LZ4 stream block chain.
MemoryError – raised if the work output buffer cannot be allocated.
OverflowError – raised if the source is too large for being decompressed in the given context.
LZ4StreamError – raised if the call to the LZ4 library fails. This can be caused by
decompressed_size
being too small, or invalid data.
- get_block(stream)¶
Return the first LZ4 compressed block from
stream
.- Parameters
stream (str, bytes or buffer-compatible object) – LZ4 compressed stream.
- Returns
LZ4 compressed data block.
- Return type
bytes or bytearray
- Raises
Exceptions occuring while getting the first block from stream. –
BufferError – raised if the function cannot return a complete LZ4 compressed block from the stream (i.e. the stream does not hold a complete block).
MemoryError – raised if the output buffer cannot be allocated.
OverflowError – raised if the source is too large for being handled by the given context.