dart_lz4 #

Pure Dart implementation of LZ4 (block + frame) and LZ4HC, including streaming frame encode/decode.

📚 Read the Official Documentation

Repository: https://github.com/jxoesneon/dart_lz4

Status #

This package provides a feature-complete pure Dart implementation of the LZ4 frame format.

Implemented:

• LZ4 block encode/decode
• LZ4 frame encode/decode (including skippable and legacy frames)
• Streaming frame encode/decode (StreamTransformer)
• LZ4HC block compression
• Dictionary support (encode + decode)
• 64-bit content size (encode + decode)
• xxHash32 with VM + Web parity

Goals #

• Pure Dart (no FFI)
• Web-safe core (no dart:io in library code)
• Strict, bounds-safe decoding with deterministic errors
• Streaming-friendly APIs with output limits
• Compatibility with LZ4 frame format (current)

Limitations #

• No FFI acceleration. This is a pure Dart implementation. Native LZ4 bindings (e.g. via dart:ffi) will be faster for large payloads.
• Single-threaded. Compression and decompression run on a single isolate. Multi-threaded compression is a roadmap item.
• Web precision. Content sizes above 2^53 bytes may lose precision on Web (JS) targets due to 64-bit integer limitations.
• Checksums are not cryptographic. blockChecksum and contentChecksum detect corruption, not tampering. Use a MAC or signature for authentication.
• Dictionary allocation. Dictionary encode/decode allocates a concatenation buffer proportional to dictionary + input size.

Security / untrusted input #

• Always set a reasonable maxOutputBytes when decoding frames (lz4FrameDecode / lz4FrameDecoder) to mitigate decompression bombs.
• Use blockChecksum and/or contentChecksum when encoding if you want corruption detection. These checksums are not cryptographic authentication.
• For block decompression (lz4Decompress), decompressedSize must be known and trusted/validated.

Interop / compatibility #

Tested against the reference lz4 CLI (lz4 v1.10.0) via embedded decode vectors and a CLI decode test.

Usage #

Block #

Block decompression requires the decompressed size.

import 'dart:typed_data';
import 'package:dart_lz4/dart_lz4.dart';

final src = Uint8List.fromList('hello'.codeUnits);
final compressed = lz4Compress(src);
final decoded = lz4Decompress(compressed, decompressedSize: src.length);

Zero-Copy Block Decompression #

Decompress directly into a pre-allocated buffer without intermediary allocations:

final dst = Uint8List(src.length);
final bytesWritten = lz4DecompressInto(compressed, dst);

// Or into an offset within a shared buffer:
final offsetBytes = lz4DecompressInto(compressed, dst, dstOffset: 64);

LZ4HC #

final compressed = lz4Compress(
  src,
  level: Lz4CompressionLevel.hc,
  hcOptions: Lz4HcOptions(maxSearchDepth: 64), // Optional tuning
);

Frame #

final frame = lz4FrameEncode(src);
final decoded = lz4FrameDecode(frame);

Frame with options #

final frame = lz4FrameEncodeWithOptions(
  src,
  options: Lz4FrameOptions(
    blockSize: Lz4FrameBlockSize.k64KB,
    blockChecksum: true,
    contentChecksum: true,
    contentSize: src.length,
    compression: Lz4FrameCompression.fast,
    acceleration: 1,
  ),
);
final decoded = lz4FrameDecode(frame);

Dependent blocks are supported by setting blockIndependence: false. When enabled, blocks may reference up to 64KiB of history from prior blocks.

Frame Inspection #

Inspect a frame header without decoding the payload:

final info = lz4FrameInfo(frameBytes);
print('Content Size: ${info.contentSize}');
print('Dictionary ID: ${info.dictId}');

Dictionary Support #

To decode frames that use a preset dictionary (identified by dictId):

final decoded = lz4FrameDecode(
  frameBytes,
  dictionaryResolver: (dictId) {
    if (dictId == 0x123456) return myDictionaryBytes;
    return null; // Dictionary not found
  },
);

Skippable Frames #

Embed custom metadata in an LZ4 stream using skippable frames:

import 'dart:convert';

final metadata = Uint8List.fromList(utf8.encode('{"version": 1}'));
final skippable = lz4SkippableEncode(metadata, index: 0);

// Concatenate with a regular frame
final combined = Uint8List.fromList([...skippable, ...lz4FrameEncode(data)]);

// Decoders will skip the metadata and decode only the payload
final decoded = lz4FrameDecode(combined);

Legacy Frames #

Encode data using the legacy LZ4 format (compatible with lz4 -l):

final frame = lz4LegacyEncode(src);
final decoded = lz4FrameDecode(frame);

Sized Blocks #

Simple helper for block compression with prepended 4-byte length header:

final compressed = lz4CompressWithSize(src);
final decoded = lz4DecompressWithSize(compressed);

Streaming frame decode #

final decodedChunks = byteChunksStream.transform(
  lz4FrameDecoder(maxOutputBytes: 128 * 1024 * 1024),
);

Streaming frame encode #

final encodedChunks = byteChunksStream.transform(
  lz4FrameEncoder(),
);

Streaming encoding also supports Lz4FrameOptions:

final encodedChunks = byteChunksStream.transform(
  lz4FrameEncoderWithOptions(
    options: Lz4FrameOptions(
      blockSize: Lz4FrameBlockSize.k64KB,
      blockIndependence: false,
    ),
  ),
);

Buffer Pooling #

Reuse allocations across streaming decode and encode workloads using power-of-two slab pools:

// Standard slab pool (64B to 8MB)
final pool = SimpleLz4BufferPool(
  maxTotalBuffers: 64,
  maxBuffersPerBucket: 8,
);

// Secure zeroized pool (CWE-226 residual memory mitigation)
final securePool = SecureLz4BufferPool();

// Supply to streaming decoder or encoder
final decodedStream = byteChunksStream.transform(
  lz4FrameDecoder(bufferPool: pool),
);

dart:convert Codec #

Compose LZ4 with the standard Dart conversion ecosystem:

import 'dart:convert';

final codec = Lz4Codec();
final compressed = codec.encode(data);
final decoded = codec.decode(compressed);

// Fuse with other codecs:
final jsonLz4 = json.fuse(codec);
final payload = utf8.encode('{"hello":"world"}');
final compressedJson = jsonLz4.encode(payload);
final restored = jsonLz4.decode(compressedJson);

The decoder enforces a 256 MiB default maxOutputBytes to prevent decompression bombs. Pass an explicit limit via Lz4Codec(maxOutputBytes: ...) to override.

Benchmarks #

Run:

dart run benchmark/lz4_benchmark.dart

It reports throughput (MiB/s) and ratio for:

• Block compress/decompress (fast + hc)
• Frame (sync) encode/decode (fast + hc)
• Frame (streaming) encode/decode (fast + hc)

License #

Apache-2.0. See LICENSE.