memory_mapped_list 0.1.0
memory_mapped_list: ^0.1.0 copied to clipboard
A high-performance memory-mapped List for Dart that handles very large datasets with minimal, constant RAM usage.
Changelog #
All notable changes to memory_mapped_list will be documented here.
The format follows Keep a Changelog. This project adheres to Semantic Versioning.
Unreleased #
(nothing yet)
0.1.0 — 2026-05-30 #
Added #
Core
MemoryMappedList<T>abstract base class mixing inListMixin<T>for fullListAPI compatibility (sort,where,map,sublist,indexOf, etc.)- LRU page-buffer (
PageBuffer) with configurablepageSizeandmaxPages MmlFileManager—RandomAccessFileI/O with sync and async variantsMmlFileHeader— 128-byte self-describing binary header (magic, version, element count, type, timestamps, page size, flags, metadata)- Three flush modes:
immediate,onClose,batched - Three access modes:
create,readWrite,readOnly
Typed Lists
MmlDoubleList— 64-bit IEEE 754 doubles (8 bytes/element)MmlFloat32List— 32-bit IEEE 754 floats (4 bytes/element)MmlInt32List— 32-bit signed integers (4 bytes/element)MmlInt64List— 64-bit signed integers (8 bytes/element)MmlGenericList<T>— variable-length elements with separate.mml.idxindex file; element count and size tracked per entry
Serialisers
MmlSerializer<T>abstract baseJsonSerializer<T>— JSON UTF-8, variable sizeStringSerializer— UTF-8 strings, variable sizeInt32Serializer,Int64Serializer,Float64Serializer,Float32Serializer— fixed-size binaryBytesSerializer— rawUint8Listpass-through
Streaming & Batch API
stream— async generator yielding all elements one by onechunkedStream({int chunkSize})— yieldsList<T>chunksreadRange(int start, int end)— batch async readwriteRange(int start, Iterable<T>)— batch async write
Aggregates (on typed lists)
sum(),mean(),minMax(),standardDeviation()— single-pass streamingmedian()(doubles only) — loads sorted copy into RAM, explicit O(n) warningfill(T value)— bulk fill viawriteRangeisAllZeros()(int32 only)timestampAt(int)/setTimestamp(int, DateTime)(int64 only)
Exceptions
MmlExceptionsealed base classMmlClosedException,MmlReadOnlyException,MmlRangeException,MmlFileNotFoundException,MmlCorruptFileException,MmlDiskFullException,MmlTypeMismatchException,MmlDirtyShutdownException,MmlDeserializationException,MmlIoExceptionMmlErrorCodeenum with stable numeric codes (1000–1599)
Utilities
MmlEndian— little-endian read/write helpersMmlMathUtils— offset arithmetic, alignment, human-readable sizes, savings ratioPageBufferConfigpresets:defaultConfig,small,large,lowMemoryMmlStats— snapshot of hit rate, file size, cached pages, pending writes
FFI (optional)
MmlFfiLoader— platform-aware dynamic library loaderMmapBindings— Dart FFI bindings tomml_nativeC librarymml_native.c— cross-platform wrapper: POSIXmmap+ Win32CreateFileMappingCMakeLists.txt+build.shfor compiling the native layer
Tests
- Unit tests:
mml_header_test.dart,mml_page_buffer_test.dart,mml_file_manager_test.dart - Type tests:
mml_double_list_test.dart,mml_int32_list_test.dart - Integration tests (tagged
integration): large file, streaming, concurrent, resize
Examples & Benchmarks
example/basic_usage.dart— minimal API walkthroughexample/scientific_analysis.dart— Z-score normalisation + anomaly detectionexample/log_processor.dart—MmlGenericList<LogEntry>withJsonSerializerbenchmark/vs_dart_list.dart— sequential read/write vsList<double>benchmark/vs_file_read.dart— full-file read vs paged MML vs chunkedStream