Fixed #

• Conversation memory ordering no longer depends on the system clock. ConversationMemory allocated message ids from DateTime.now(), so on a platform whose clock ticks coarsely (Windows: a millisecond or more) two handles appending to one conversation inside a single tick received equal stamps, and their relative order came down to the random part of the id — recent() and context() could then return two concurrent messages in the wrong order. Each append now catches up to the newest stored message before allocating, so the sequence increases from stored state rather than from the clock and holds at any resolution, at the cost of one read per append. No message was ever lost, and ordering within a single handle was always correct.

Reactive queries, on-device agents, collection maintenance and salvage tooling — plus fixes for everything a full audit of the 4.0 code turned up.

Added #

• Reactive queries. AsyncPhoenixDB.changes() and AsyncPhoenixCollection.changes() stream committed changes (keys, or document ids) as they happen; PhoenixDatabase.watch() and PhoenixCollection.watch() are the synchronous form. Only committed writes are published, in commit order, after the log is durable; each subscription has a bounded queue and reports how many changes a slow consumer missed; a restore delivers one reset. New C entry points phoenix_watch_* and phoenix_collection_watch_open.
• Tool calling and agents in package:phoenixdb/ai.dart: Tool, ToolCallingModel (implemented by both the Claude and OpenAI-compatible clients) and Agent, a bounded, inspectable loop that runs the tools a model asks for. knowledgeBaseTool hands the model a search tool over the local collection.
• Collection maintenance: verify (documents cross-checked against the vector index, plus a structural report), compact (reclaims tombstoned and orphaned vectors and free pages) and backup (a consistent, compacted copy), on both clients and over the ABI.
• Salvage tooling: PhoenixDatabase.salvage, phoenix_salvage and the phoenixdb_salvage binary recover every readable pair from a file too damaged to open, newer copies of a key winning. Closes the roadmap's checksum/repair item.
• Newest-first paging: list(newestFirst: true) and documents(newestFirst: true) on collections, with streaming pagination that costs the page rather than the collection.
• CollectionStats.textIndex reports the layout actually in force.

Fixed #

• A failed collection upsert no longer destroys the previous embedding. Replacing a document wrote its new vector before the commit; if the commit then failed, the old embedding was already gone. The previous vectors are now restored (and new ones withdrawn) when a commit fails.
• Vectors the documents disown are never reported. A vector whose document is absent, or whose document says it has none, is dropped by recovery and excluded from search results, so search cannot return an id that get and list both deny. A document whose embedding was lost is recorded as having none, so the three agree.
• Corrupt document metadata is an error, not silent null — reading it as null left every index entry for that document behind on delete.
• text_documents no longer grows forever on a collection created with text_index: false, and a text query against such a collection is an error rather than an empty result. Reopening with a different text_index is refused instead of silently reinterpreted.
• Single-character CJK queries match again: runs are indexed as unigrams as well as bigrams, so 猫 finds 子猫.
• Index keys can no longer exceed the engine's key limit: metadata field paths are bounded at 256 bytes, and a string's encoded length (NUL escapes included) decides whether it is indexable. Metadata walks are depth-bounded, so deeply nested metadata cannot overflow the stack, and a truncated index key reports corruption instead of panicking.
• Filter.and([]) matches everything from the index, as the in-memory semantics always did; weighted fusion with one empty retriever falls back to the side that found something instead of zeroing every score; MMR without embeddings is refused.
• A second close of a handle is safe again. Closing freed the handle struct, so a second close read freed memory — and could release a different handle that reused the address. Handles now leave a poisoned tombstone.
• AI toolkit: a Claude stream ends at message_stop instead of waiting for the connection to close (a gateway that holds it open used to turn a good answer into a timeout); a stream that ends before its terminal event is reported rather than passed off as a short answer; the semantic cache embeds stored prompts and lookups the same way (an asymmetric embedder never hit before), treats a NaN similarity as a miss, and never caches an empty or interrupted answer; conversation memory keys messages by timestamp, so two handles cannot overwrite each other's messages; RAG prunes per document (a repeated id in one batch used to leave a mixture), tolerates any citation a model writes, and reports only the sources that fitted the context budget.
• HTTP transport: one deadline covers a whole call including retries, the response body is read inside the retry loop, a non-JSON 200 and an odd error shape surface as LlmException rather than raw Dart errors, and SSE parsing tolerates malformed UTF-8 and split frames.
• Collection clients validate arguments identically, documents() validates at the call site, a failed open no longer leaks the worker isolate or the directory lock, and unexpected JSON is reported as a Phoenix error rather than a TypeError.

Notes #

• Native ABI is 5 (collections and change notifications). A v4 library is rejected by the loader, so upgrade the native binaries with the package.
• CollectionStats gained a required field and DocumentStore.list a named parameter; both are breaking only for code that constructed them itself.

A hardening release for every engine, plus document collections with hybrid search and an AI toolkit for on-device retrieval-augmented generation.

Breaking #

• Native ABI 4. The Dart package requires a v4 native library; the loader checks the version before binding symbols and keeps searching past stale libraries.
• Exclusive file locks. A database, vector index or collection opened by another process now fails with the new status -11 (busy). Opens within one process share the running engine, so multiple isolates and Flutter hot restart keep working.
• SQL is stricter: reserved words and duplicate column names are rejected, PRIMARY KEY is enforced, and = NULL means IS NULL.
• Rust: Database::open takes an Options struct; tree read APIs take &Pager. MSRV is 1.89.

Added #

• Document collections (PhoenixCollection, AsyncPhoenixCollection): text, JSON metadata and embeddings in one store, searched together with HNSW vector similarity, BM25 full text (Unicode tokenizer with CJK bigrams), MongoDB-style metadata filters served from an index (Filter DSL with &, |, ~), Reciprocal Rank Fusion or weighted fusion, and MMR diversification. Upserts are atomic and crash-safe. New C entry points phoenix_collection_*.
• AI toolkit (package:phoenixdb/ai.dart): AnthropicChatModel (Claude via the Messages API — streaming, adaptive thinking, prompt caching, server-side refusal fallbacks), OpenAICompatibleChatModel (OpenAI, Ollama, Gemini, vLLM, LM Studio, …), OpenAICompatibleEmbedder (OpenAI, Voyage AI, Ollama), HashingEmbedder, a persistent CachedEmbedder, TextChunker, RagPipeline with cited answers, SemanticCache and ConversationMemory. No new dependencies.
• Key/value: PhoenixOptions, ordered range and prefix scans (scan, scanPrefix, lazy entries, scanWhile), atomic WriteBatch, backup/restore/compact, stats, structural check, Prometheus metrics, tracing spans, and full async parity. PhoenixPrefs gained getKeys, getAll and clear.
• SQL: bound parameters (?, ?N), statements inside a caller's transaction (txnId:), NOT/parentheses, IN, BETWEEN, LIKE/ILIKE, IS [NOT] NULL, COUNT/SUM/AVG/MIN/MAX with DISTINCT and GROUP BY, aliases, multi-key ORDER BY and OFFSET.
• Vectors: filtered search, exact search within an id subset, batch search and batch insert (insertAll, searchIds, searchBatch).

Fixed #

• Data loss in the B+Tree: growing a value in a full leaf deleted its neighbour; large cells could wedge splits and every later checkpoint.
• WAL and checkpoints: a checkpoint could strand half of an in-flight transaction or truncate versions a live snapshot still needed; a torn log tail hid every later commit from recovery.
• Torn pages: page flushes are now journaled, so a crash cannot tear a page or the meta page.
• Scans returned unordered and duplicate keys.
• SQL lost concurrent writes, leaked conflicted transactions, sorted ORDER BY inconsistently, and returned floats as ints.
• Vector index: compact() could lose every vector on a crash, a torn tail made the index unopenable, bulk loads were quadratic and each search was O(N).
• Dart: the scan callback ABI was wrong (scans could stop at random), metricsReport double-freed, trace events grew without bound, and async workers hung forever when their isolate died.
• LSM (standalone): torn or corrupt manifests could delete every SSTable; compaction swaps are now atomic.
• Security (standalone): RBAC fell open when emptied, the KDF input was ambiguous, and audit records could carry terminal escapes or interleave.

Phase 4 observability: engine metrics are now reachable from Dart.

Added #

• PhoenixDatabase.metricsReport() — returns the engine's metrics snapshot as a human-readable string. The report covers WAL fsync latency percentiles, page-cache hit ratio, LSM compaction throughput, and transaction counters. The engine records begin/insert latencies into the same EngineMetrics histograms that power the Rust-side report.
• New C entry point phoenix_metrics_report(handle, out_buf, out_len) exports the report into a caller-provided buffer.
• TraceListener lifecycle hooks now fire for open, beginTransaction, insert, get, delete, and commit — enough to reconstruct a request's critical path from the Dart side.

Fixed #

• Rust 2024 let-chains in lsm/manifest.rs and lsm/compaction.rs that failed to parse on the current toolchain (converted to nested if let).
• c"..." c-string literals in ffi/vector_ffi.rs that broke cargo check under edition 2024 (replaced with CStr::from_bytes_with_nul_unchecked).

Notes #

• Native ABI remains 3 (additive change — no new signatures on existing entry points). The new phoenix_metrics_report is covered by the ABI version guard, so a v3 native library loaded by this package is guaranteed to expose it.
• AUDIT_AND_ROADMAP.md remains an untracked scratchpad.

Embedded vector search: approximate k-NN over f32 embeddings, so a local-first app can do semantic retrieval with no server and no second database.

Breaking #

• Native ABI raised from 2 to 3. The change is additive (every v2 entry point keeps its signature) but the Dart loader enforces an exact match, so a v2 native library and this package will not load together. Both are rebuilt and shipped in this release; anyone building the library themselves must rebuild it.

Added #

• HNSW vector index. PhoenixVectorDB (sync) and AsyncPhoenixVectorDB (worker isolate) expose insert, search, get, remove, save, flush, compact and stats.
  • Metrics: cosine, Euclidean (L2) and dot product, selected at index creation and fixed thereafter.
  • VectorQuery carries the query vector, k, and an optional efSearch beam width; VectorMatch carries the id, the metric distance, and a "higher is better" score.
  • Vectors up to 65 536 dimensions; ids up to 128 bytes; k up to 4096.
• SIMD distance kernels. Runtime-dispatched AVX2+FMA on x86_64, NEON on AArch64, and a portable auto-vectorised fallback everywhere else. PhoenixVectorDB.kernel reports which one this CPU selected.
• Durable, memory-mapped vector storage. A fixed-stride append-only file with a per-record CRC32, read zero-copy through the existing mmap layer. The HNSW graph is snapshotted separately with bincode, written atomically via a temporary file and a rename.
• Crash and corruption recovery. A torn tail record is ignored, a corrupt or stale graph snapshot is rebuilt from the vectors (which are the source of truth), and reopening an index with the wrong dimensionality or metric is refused rather than silently reinterpreted.
• Nine new C entry points (phoenix_vector_*) plus phoenix_free_string_array and phoenix_has_vector, all with the same null-check, length-check and catch_unwind guarantees as the existing surface.
• .github/workflows/build_native.yml: a SIMD-aware build matrix covering Linux, macOS, Windows, Android, iOS, and cross-compilation-only targets.

Notes #

• +avx2 is deliberately not passed in RUSTFLAGS. This package ships prebuilt binaries, and a crate-wide AVX2 build would SIGILL on pre-2013 x86_64 CPUs — on a user's machine, not in CI. AVX2 is instead enabled per-function and selected by is_x86_feature_detected!. +neon is passed on AArch64, where NEON is part of the base architecture and therefore always safe.
• No new dependencies. hnsw_rs would have pulled in roughly 100 transitive crates (anndists, mmap-rs, rayon, env_logger, jiff), several of which do not cross-compile cleanly to every mobile target, so the graph is implemented directly in rust/src/vector/hnsw.rs.
• Indexes with fewer than 512 live vectors are searched exhaustively, so small collections return exact rather than approximate results.

Multi-modal storage: a hybrid LSM layer, a SQL front end, encryption at rest, RBAC, audit logging, and metrics — all behind feature flags so the embedded core stays small.

Breaking #

• Native ABI raised from 1 to 2. The change is additive (every v1 entry point keeps its signature) but the Dart loader enforces an exact match, so a v1 native library and this package will not load together. Both are rebuilt and shipped in this release; anyone building the library themselves must rebuild it.

Added #

LSM storage layer (lsm) — library only, see Known limitations

• MemTable, SSTable (block-based, with a Bloom filter per table), leveled compaction, and a compaction scheduler prioritised by write amplification.
• A crash-safe, CRC-framed durable manifest recording which SSTables are live at which level. A torn tail from a power loss is truncated rather than treated as corruption; the layout, level placement, tombstones, and checkpoint sequence number all survive a restart. Unreferenced .sst files are reclaimed on open; a referenced table that fails its checksum is a hard error rather than silent data loss.
• The manifest log is snapshotted once it grows past a threshold, so startup replay does not slow down without bound.

SQL front end (sql, opt-in)

• Hand-written lexer, recursive-descent parser, and executor supporting CREATE TABLE, DROP TABLE, INSERT, SELECT (projection, WHERE, AND/OR, ORDER BY, LIMIT), UPDATE, and DELETE, plus IF NOT EXISTS / IF EXISTS.
• SQL semantics where they matter: NULL is never equal to anything under an ordinary comparison, ORDER BY is applied before LIMIT, integers and floats compare numerically, and mismatched types yield no rows rather than an arbitrary ordering.
• Mutations are transactional — a multi-row INSERT either lands completely or not at all.
• Reachable from Dart as db.query(...) (synchronous) and AsyncPhoenixDB.query(...) (on the worker isolate, so a slow query cannot block a Flutter frame). Results arrive as a typed SqlResult with scalar, firstOrNull, cell(), and asMaps helpers.

Security (encryption, opt-in) — library only, see Known limitations

• Transparent AES-256-GCM encryption at rest. Pages are encrypted before checksumming and decrypted after verification, so a tampered or swapped page is rejected rather than decrypted into garbage.
• Role-based access control with constant-time credential comparison.
• An append-only audit log kept separate from the WAL, resistant to log injection.

Observability (metrics, opt-in) — library only, see Known limitations

• Latency histograms with p50/p99/p999, covering WAL fsync, compaction throughput, and cache hit/miss ratios.
• Structured tracing spans with trace ids.

Dart API

• PhoenixPrefs, a shared_preferences-style typed facade (getString, setInt, getBool, …) over the key/value store.
• phoenix_has_sql() reports whether the loaded library includes the SQL layer, so an app can degrade gracefully on a lean embedded build.

Fixed #

• The package could not be used as an ordinary dependency. Every native library search path was relative to the consumer's working directory, but the binaries ship inside the installed package (in the pub cache, or at a path: dependency's location). A plain dart pub get followed by dart run failed with "Could not load phoenixdb.dll". The loader now resolves its own package root first — via Isolate.resolvePackageUriSync, falling back to reading .dart_tool/package_config.json, which is what the flutter test runner needs.

• Windows lookups missed MSVC builds. Only x86_64-pc-windows-gnu was searched, so the MSVC library that CI ships would not be found. Both triples are now searched, and windows_arm64 was added.

• Flutter and script builds produced an unloadable library. The Linux and Windows CMake files, the Apple build script, build.sh, and build.ps1 all ran cargo build without --features sql, yielding an ABI v1 library that the v2 loader rejects. All build paths now enable the feature.

• Platform manifests still declared 0.1.0. The iOS and macOS podspecs and android/build.gradle were never bumped, so CocoaPods and Gradle advertised a version that no longer matched the package.

• dart pub get failed for plain Dart consumers. pubspec.yaml declared a flutter: constraint under environment:, which makes the whole package require the Flutter SDK:

  Because phoenixdb requires the Flutter SDK, version solving failed.
  

  Nothing under lib/ imports package:flutter — the only package imports are ffi and phoenixdb — so the constraint was never warranted. Flutter support comes from the flutter: plugin: section, which plain Dart ignores. The constraint is removed and CI now guards against its return.

• A missing Rust toolchain failed opaquely. A desktop Flutter build without cargo on PATH stopped at Error 1 from the custom build command, with no indication that Rust was the cause. The Linux and Windows CMake files now fail configuration with an explicit message pointing at rustup.

Changed #

• Feature flags (encryption, json, sql, metrics, async-runtime, full) keep the default build lean for Flutter. The default build adds no heavy dependencies.
• serde_json is now optional, behind the json feature.

Notes on dependencies #

Three crates named in the original design could not be used, because they require a C toolchain that is unavailable on the Flutter/Android cross-compilation path. Substitutions with the same guarantees were used instead:

Planned	Shipped	Reason
ring	aes-gcm	Pure Rust, same AES-256-GCM construction
sqlparser-rs	hand-written parser	Its stacker dependency needs a C compiler
OpenTelemetry OTLP	internal tracing	tonic pulls in cc-based dependencies

Known limitations #

• The lsm, security::encryption, security::rbac, security::audit and observability modules are libraries, not yet engine behaviour. They are implemented and tested, but Database still writes through the B+Tree only, the pager does not encrypt, no permission check runs at the FFI boundary, and the engine emits no metrics. Key and value length validation from security is enforced on every FFI call.
• SELECT scans the visible key space per query rather than using a prefix-bounded iterator: appropriate for embedded workloads, O(database) on large tables.
• The SQL layer has no planner, joins, subqueries, or aggregate functions.
• Raft replication, gossip anti-entropy, full-text search, and the REPL are not implemented.
• Web is not supported: the engine is native code reached over dart:ffi.

Initial release.

Added #

Storage engine (Rust)

• B+Tree index with configurable fill factors (minimum 50%, maximum 100%).
• Fixed 4096-byte slotted pages with a 32-byte header carrying page_id, is_leaf, num_keys, parent_id and a CRC32 checksum.
• CRC32 written before every page write and verified on every read; corruption is reported as an error and never returned as data.
• Structural validation of the slot directory in addition to the checksum, so a page cannot direct an accessor outside its own buffer.
• Overflow-page chains for values larger than 1 KiB, with cycle guards.
• Free-list recycling of released pages.
• Zero-copy reads through mmap (Unix) and CreateFileMappingW (Windows), paired with positional writes and fsync/sync_all for durability.
• LRU page cache in front of the mapping.

Transactions

• MVCC snapshot isolation: many concurrent readers, one writer serialised by parking_lot::RwLock.
• Write-ahead log with Begin, Insert, Delete, Commit and Rollback records, each framed with its own CRC32.
• fsync on commit, so a transaction is durable when commit returns.
• Crash recovery that replays only committed transactions and tolerates a torn log tail.
• Write-write conflict detection with a dedicated status code for retry.
• Checkpointing that merges versions into the tree, flushes, then truncates the log.

FFI layer

• Pure C ABI (extern "C") covering open, close, insert, get, delete, begin/commit/rollback, checkpoint, flush, verify, count and free.
• phoenixdb.h generated automatically by cbindgen during the build.
• Pointer non-nullness and length limits (key 1 MiB, value 10 MiB) validated before any dereference, returning -2.
• Constant-time handle-tag verification, poisoned on close, so use-after-free and double close are rejected rather than followed.
• Panics contained with catch_unwind and mapped to -7; nothing unwinds into Dart.

Dart package

• Type-safe API using Uint8List for keys and values.
• NativeFinalizer attached to the native handle for automatic cleanup.
• AsyncPhoenixDB, an isolate-backed client that keeps blocking disk I/O off the UI thread.
• Automatic native-library discovery with an ABI-version check on load.

Packaging

• Installable with dart pub add phoenixdb and flutter pub add phoenixdb.
• Declared as a Flutter FFI plugin (ffiPlugin: true) for Android, iOS, macOS, Linux and Windows, so no method-channel registrant code is generated.
• Android: prebuilt .so for arm64-v8a, armeabi-v7a, x86_64 and x86 shipped in android/src/main/jniLibs/, packaged into the host APK/AAB with no NDK required by the consumer (minSdk 21).
• iOS and macOS: CocoaPods podspecs that compile a static archive during the Xcode build via rust/build-apple.sh; the archive is -force_loaded so the phoenix_* symbols survive dead-stripping.
• Linux and Windows: CMake integration that runs cargo build and bundles the resulting shared library with the app.
• Platform-aware library loading: bare-name dlopen on Android, DynamicLibrary.process() on iOS (statically linked), and a per-target-triple filesystem search on desktop.

Tooling

• build.sh and build.ps1 with cross-compilation support via CC/AR.
• libfuzzer harnesses for the B+Tree, page parsing and the FFI surface.
• 86 Rust tests and 33 Dart tests; dart analyze --fatal-infos clean.
• Scores 160/160 on pub.flutter-io.cn's package analysis (pana).

License #

Released under the BSD 3-Clause License.