memory_mapped_list

Process billions of elements. Use constant RAM. πŸš€

A pure-Dart package that provides a List-compatible interface backed by binary files using an LRU page-buffer strategy. Store and process datasets far larger than the available RAM, while writing code that looks exactly like ordinary Dart list manipulation.

Pub Version Dart SDK License: MIT Platform

The Problem

// ❌ Crashes with OutOfMemoryError on most machines
final list = List<double>.filled(1_000_000_000, 0.0); // 8 GB of RAM needed

The Solution

// βœ… Works with ~4 MB of RAM regardless of list size
final list = await MemoryMappedList.doubles(
  path: 'my_data.mml',
  length: 1_000_000_000, // one billion elements!
);
// ↑ exactly 4 MB RAM Β· ~7.6 GB file on disk

Features

Feature Description
🎯 Drop-in List API Full List<T> interface via ListMixin β€” sort, where, map, sublist, indexOf, etc. all work out of the box
πŸ’Ύ Disk-backed storage Data lives on disk; RAM stays constant at pageSize Γ— maxPages
πŸ“„ LRU page cache Configurable cache keeps hot pages in memory for high hit rates
πŸ”’ Typed lists int32, int64, float32, float64 β€” each with optimal byte layout
🧩 Generic list Any Dart object via a custom MmlSerializer
🌊 Streaming API stream and chunkedStream for zero-copy sequential processing
⚑ Batch I/O readRange / writeRange for high-throughput bulk operations
πŸ“ˆ Aggregate stats sum, mean, standardDeviation, minMax β€” all streaming
πŸ” Flush modes immediate, onClose, batched β€” balance durability vs speed
πŸ›‘οΈ Rich exceptions Typed exceptions with stable numeric error codes
🌍 Cross-platform Windows, Linux, macOS, iOS, Android β€” Pure Dart, no native required
πŸ”Œ Optional FFI Native mmap / CreateFileMapping layer for maximum throughput

Installation

dependencies:
  memory_mapped_list: ^0.1.0

dart pub get

Quick Start

import 'package:memory_mapped_list/memory_mapped_list.dart';

void main() async {
  // ── Create ──────────────────────────────────────────────────────────
  final list = await MemoryMappedList.doubles(
    path: 'data.mml',
    length: 100_000_000, // 100 million doubles = 762 MB on disk, 4 MB RAM
  );

  // ── Write β€” identical to List<double> ─────────────────────────────
  list[0] = 3.14159;
  list[99_999_999] = 2.71828;
  for (var i = 0; i < 1000; i++) list[i] = i * 0.01;

  // ── Read ──────────────────────────────────────────────────────────
  print(list[0]);            // 3.14159
  print(list.length);        // 100000000

  // ── Standard List operations ──────────────────────────────────────
  list.sort();               // sorts in-place (with page cache)
  list.where((v) => v > 0);  // lazy iterable
  list.sublist(0, 10);       // returns List<double>

  // ── Streaming aggregates (O(1) RAM) ───────────────────────────────
  print(await list.sum());
  print(await list.mean());
  print(await list.standardDeviation());
  final bounds = await list.minMax();
  print('${bounds.min} … ${bounds.max}');

  // ── Stream API ────────────────────────────────────────────────────
  await for (final value in list.stream) {
    // process one element at a time
  }
  await for (final chunk in list.chunkedStream(chunkSize: 50_000)) {
    // process 50 000 elements at a time
  }

  // ── Diagnostics ───────────────────────────────────────────────────
  print(list.stats); // hit rate, file size, pending writes …

  // ── Always close! ─────────────────────────────────────────────────
  await list.close(); // flushes dirty pages, releases file descriptor
}

Supported Types

Factory Dart type Bytes / element Notes
MemoryMappedList.doubles() double 8 64-bit IEEE 754
MemoryMappedList.float32s() double 4 32-bit IEEE 754; half the disk space
MemoryMappedList.int32s() int 4 Range: βˆ’2 147 483 648 … 2 147 483 647
MemoryMappedList.int64s() int 8 Full 64-bit integer range
MemoryMappedList.generic<T>() T variable Requires MmlSerializer<T>

Access Modes

// Create new file (or overwrite existing)
final list = await MemoryMappedList.doubles(
  path: 'data.mml', length: 1000, mode: AccessMode.create);

// Open existing for read + write
final list = await MemoryMappedList.doubles(
  path: 'data.mml', length: 1000, mode: AccessMode.readWrite);

// Open existing, read-only
final list = await MemoryMappedList.doubles(
  path: 'data.mml', length: 1000, mode: AccessMode.readOnly);

Flush Modes

// Safest: every write is immediately persisted
final list = await MemoryMappedList.doubles(
  path: 'x.mml', length: 100, flushMode: FlushMode.immediate);

// Fastest: only flush on explicit flush() or close()
final list = await MemoryMappedList.doubles(
  path: 'x.mml', length: 100, flushMode: FlushMode.onClose);

// Balanced: auto-flush every 1 000 writes
final list = await MemoryMappedList.doubles(
  path: 'x.mml', length: 100,
  flushMode: FlushMode.batched, batchSize: 1000);

Page Buffer Configuration

// Balanced default (4 MB cache)
PageBufferConfig.defaultConfig()     // 16 KB pages Γ— 256 = 4 MB

// Larger cache for random-access patterns (64 MB)
PageBufferConfig.large()             // 64 KB pages Γ— 1024 = 64 MB

// Tiny footprint for constrained devices (64 KB)
PageBufferConfig.lowMemory()         // 4 KB pages Γ— 16 = 64 KB

// Custom
PageBufferConfig(pageSize: 32 * 1024, maxPages: 512) // 16 MB

Generic List

class Point {
  final double x, y;
  const Point(this.x, this.y);
  Map<String, dynamic> toJson() => {'x': x, 'y': y};
  factory Point.fromJson(Map<String, dynamic> j) =>
      Point(j['x'] as double, j['y'] as double);
}

final serializer = JsonSerializer<Point>(
  fromJson: Point.fromJson,
  toJson: (p) => p.toJson(),
);

final points = await MemoryMappedList.generic<Point>(
  path: 'points.mml',
  length: 10_000_000,
  serializer: serializer,
);

points[0] = const Point(1.0, 2.0);
print(points[0].x); // 1.0

await points.close();

Memory Model

RAM consumed  =  pageSize Γ— maxPages  (constant)

Example β€” 10 million doubles, default config:
  File on disk :  128B header + 10M Γ— 8B  =  76.3 MB
  RAM (MML)    :  16 KB Γ— 256            =   4.0 MB  ← always!
  RAM (List)   :  10M Γ— 8B              =  76.3 MB

  Savings: 94.7 %

Formula:

byteOffset  = headerSize + index Γ— elementSize
pageIndex   = byteOffset Γ· pageSize
offsetInPage = byteOffset mod pageSize

File Format

Every .mml file starts with a 128-byte binary header:

 Offset β”‚ Size β”‚ Field
────────┼──────┼──────────────────────────────────────────────────
  0     β”‚  4   β”‚ Magic: 0x4D4D4C01 ("MML\x01")
  4     β”‚  4   β”‚ Format version (currently 1)
  8     β”‚  8   β”‚ Element count (int64, little-endian)
 16     β”‚  4   β”‚ Element size in bytes
 20     β”‚  4   β”‚ Element type ID (1=int32 2=int64 3=float32 4=float64 99=generic)
 24     β”‚  8   β”‚ Created-at  (Β΅s since Unix epoch)
 32     β”‚  8   β”‚ Modified-at (Β΅s since Unix epoch)
 40     β”‚  4   β”‚ Page size used
 44     β”‚  4   β”‚ Flags (0x01=readOnly, 0x02=compressed, 0x04=dirtyShutdown)
 48     β”‚ 64   β”‚ User metadata (UTF-8, NUL-terminated)
112     β”‚ 16   β”‚ Reserved
128     β”‚  …   β”‚ DATA

This makes .mml files self-describing β€” they can be reopened without knowing the element count or type in advance.

When to Use

Scenario Dart List memory_mapped_list
< 1 M elements βœ… fastest βœ…
1 M – 100 M elements ⚠️ heavy RAM βœ… recommended
> 100 M elements ❌ OOM βœ…
Files > available RAM ❌ βœ…
Scientific data analysis ⚠️ βœ… streaming aggregates
Log processing pipelines ⚠️ βœ… chunkedStream
ML feature stores ❌ βœ… float32s list
Time-series databases ❌ βœ… int64s + doubles

Performance Tips

  1. Use PageBufferConfig.large() for random-access workloads.
  2. Use chunkedStream instead of element-by-element iteration β€” it reads whole pages at a time.
  3. Use writeRange for bulk writes β€” it batches 50 000 writes per event-loop yield to reduce overhead.
  4. Use FlushMode.batched when writing large amounts of data to reduce syscall pressure.
  5. Sequential access achieves >95 % cache hit rate with the default config.

Error Handling

try {
  final list = await MemoryMappedList.doubles(
    path: 'missing.mml', length: 0, mode: AccessMode.readOnly);
} on MmlFileNotFoundException catch (e) {
  print('File not found: ${e.filePath}');  // typed exception
  print('Error code: ${e.errorCode.code}'); // numeric code
}

All exceptions extend MmlException and carry a MmlErrorCode for programmatic handling without string parsing.

Running the Examples

dart run example/basic_usage.dart
dart run example/scientific_analysis.dart
dart run example/log_processor.dart

Running the Benchmarks

dart run benchmark/vs_dart_list.dart
dart run benchmark/vs_file_read.dart

Running the Tests

# Unit tests
dart test

# Integration tests (creates files up to ~10 MB, takes a few seconds)
dart test --tags integration

Contributing

Pull requests welcome! Please:

  1. Run dart analyze β€” zero warnings required.
  2. Add tests for any new functionality.
  3. Update CHANGELOG.md following Keep a Changelog.

License

MIT Β© 2026 yourname

Libraries

memory_mapped_list
memory_mapped_list