The official 2.0 release of flutter_ai_scrapper — an architectural rebuild delivering an HTML5 DOM parser, zero-cost deterministic metadata harvesting, on-device Gemma & cloud AI extraction, pure-CSS site recipes, and zero-dependency Material 3 widgets.

⚠️ Breaking Changes & Migration #

• HTML5 DOM Engine: Replaced legacy regular expression matching with an HTML5 spec-compliant DOM parser. Element extraction correctly pairs nested tags and respects true DOM hierarchies.
• Class Filtering: .select('.class') evaluates exact class name tokens instead of substring containment.
• Entity Decoding: All text nodes automatically decode HTML5 entities (e.g. & becomes &).
• Platform Minimums:
  • Android: minSdkVersion 24 (Android 7.0), compileSdkVersion 36.
  • iOS: iOS 15.0+.
• API Modernization:
  • MobileScraper is @Deprecated in favor of AiScrapper.open() and ScrapedPage.
  • See doc/MIGRATION.md for full migration instructions.

🌟 Key Features Added in 2.0 #

• Tier 1 (Deterministic Foundation): Zero AI tokens, <2ms execution. Full DOM querying, Readability article prose extraction, and automatic JSON-LD / Microdata / OpenGraph structured harvesting.
• Tier 2 (Typed Schema Extraction): Schema-as-tool extraction with on-device Gemma models (flutter_gemma) or cloud fallback (OpenAiProvider, AnthropicProvider, CustomProvider).
• Tier 3 (Natural Language Asking): page.ask('query') infers typed schemas and executes automated extractions.
• Tier 4 (Pure-CSS Selector Recipes): RecipeRunner runs synthesized CSS selectors with 0 AI tokens, sub-millisecond latency, and automatic layout drift detection.
• Output & Codecs: Deterministic normalizers (dates, money, phone, URL, numbers) and RFC 4180 CSV, Markdown table, and typed object codecs.
• Privacy & Security: allowCloudEgress: false privacy gate by default, plus KeySanitizer API key redaction.
• Material 3 UI Suite: ModelManagerSheet, ProviderSettingsSheet, ResultViewer, StreamingTextView, and ExtractionConsole.

Phase 6: Output codecs, deterministic normalizers, and zero-dependency Material 3 UI widgets.

Added #

• Deterministic Value Normalizers:
  • DateNormalizer: Parses ISO-8601, RFC-2822, human-readable dates, and relative times ("yesterday", "N hours ago") to standard UTC ISO-8601 strings.
  • MoneyNormalizer: Multi-currency symbol detection ($, €, £, ¥, ₹) and ISO codes with European/US separator handling; completely closes the fabricated-$ issue.
  • PhoneNormalizer: E.164 standardization with domestic region hints.
  • UrlNormalizer: Resolves relative URLs and strips marketing/tracking query parameters (utm_*, fbclid, gclid).
  • NumberNormalizer: Locale-aware thousands/decimal separators and shorthand suffixes (k, M, B, %).
• Output Codecs (codecs.dart):
  • toJsonString: Serializes extraction results with optional provenance audit blocks.
  • toCsv: Exports to RFC 4180 CSV with nested map flattening and cell escaping.
  • toMarkdownTable: Clean GitHub Flavored Markdown table generation.
  • toTyped<T>: Deserializes results into strongly-typed domain model instances.
  • toPrettyString: Colorized/indented console debugging string.
• Material 3 UI Widgets (Zero External State Management):
  • ModelManagerSheet: Manages on-device Hugging Face models, download progress, Wi-Fi toggle, storage statistics, and deletion.
  • ProviderSettingsSheet: Fallback chain configuration, per-provider settings, test connection, token spend, and allowCloudEgress privacy switch.
  • ResultViewer: Interactive tabbed view (Table, JSON, Markdown, Raw) with field-level Provenance Badges.
  • StreamingTextView: Real-time token streaming with collapsible thinking/reasoning blocks.
  • ExtractionConsole: End-to-end interactive scraping console with pipeline stage indicators.
• Demo App (example/lib/main.dart):
  • Rebuilt with full Material 3 dynamic themes in light and dark mode, demonstrating all four API tiers (Quick Scrape, Typed Schema, Ask Page, Recipes).

Phase 5: Selector recipes, DOM structural skeleton, and natural-language planner.

Added #

• DOM Structural Skeleton (StructuralSkeleton):
  • Condenses arbitrary HTML trees down into a token-efficient DOM skeleton.
  • Elides textual content and collapses consecutive repeated siblings with count="N" and comment annotations (<!-- repeated N times -->).
  • Automatically bounds depth and budget under 1,500 tokens for massive 400 KB+ catalogue pages.
• Pure-CSS Recipe Runner (RecipeRunner):
  • Executes site extraction recipes using pure CSS selectors and regex parsing with zero AI inference tokens.
  • Sub-millisecond execution for both single-entity objects and collections.
  • Automatically flags structural drift if container selectors or key fields match 0 elements.
• Recipe Model & Store (Recipe, FieldSelector, RecipeStore):
  • Schema-stable recipe hashes (Recipe.hashSchema(schema)).
  • Time-to-live expiration and eviction.
  • RepairPolicy support (resynthesize, fallbackToAi, fail).
• Self-Verifying Recipe Synthesizer (RecipeSynthesizer):
  • Synthesizes candidate selector recipes from a DOM skeleton using language models.
  • Runs candidate recipes against the source page and self-verifies data yield before storage, rejecting faulty selectors immediately.
• Natural-Language Planner (Planner):
  • Translates free-form natural language queries into typed schemas (Schema.object and Schema.list).
  • Infers field types (money, number, date, url, string) and query cardinality (list vs object).
  • Emits PlannedExtraction exposing the inferred schema to callers for inspection.
• Public page.ask() API:
  • One-line natural-language scraping with automatic selector recipe caching: expensive AI judgment on page 1, deterministic zero-token CSS execution on pages 2–N.

Phase 4: Cloud providers and fallback engine.

Added #

• OpenAI-compatible adapter (OpenAiProvider):
  • Targets POST {baseUrl}/chat/completions with first-class baseUrl parameter covering OpenAI, Azure, Groq, Together, Fireworks, OpenRouter, DeepSeek, Mistral, xAI, Ollama, LM Studio, and vLLM.
  • Three structured output modes: json_schema strict mode, tools function calling, and prompted JSON repair fallback.
  • Endpoint capability caching per baseUrl + model.
  • SSE token streaming.
• Anthropic adapter (AnthropicProvider):
  • Targets POST {baseUrl}/v1/messages with tools + tool_choice forced function calling.
  • Direct decoding of tool_use content blocks and content_block_delta SSE streaming.
• Custom adapter (CustomProvider):
  • Allows wrapping arbitrary functions or enterprise API gateways into an AiProvider.
• Fallback engine (ProviderChain):
  • Ordered fallback with exact failure escalation: handles unconfigured providers, offline/network errors, 429 rate limits (with backoff), 5xx server errors (with retry), and 401/403 auth errors (loud warnings).
  • Terminal fallback to on-device Gemma with configuration warnings when an offline floor is absent.
  • Per-provider circuit breaker to stop hammering down endpoints.
  • preferLocal mode prioritizing on-device inference before cloud escalation.
• Privacy & security controls:
  • allowCloudEgress: false by default, ensuring zero scraped content is sent off-device without explicit opt-in.
  • KeySanitizer redacts sensitive API keys and bearer tokens from all logs and error strings.
  • Published comprehensive security guide in doc/PROVIDERS.md emphasizing the backend proxy pattern.
• Cost & usage accounting (cost_tracker.dart):
  • ModelPricing per-model price catalog.
  • UsageSession aggregating prompt/completion tokens, dollar spend, and savings from deterministic short-circuits.

Phase 3: AI Provider layer and on-device Gemma integration.

Added #

• flutter_gemma integration: added flutter_gemma: ^1.7.0 to library dependencies, with opt-in inference engines (flutter_gemma_litertlm and flutter_gemma_mediapipe) in the demo app.
• AiProvider contract (lib/src/ai/ai_provider.dart):
  • Defines extract(Schema, content) as the primary seam rather than simple string completions.
  • Full capability profiling with AiCapabilities, execution metrics with TokenUsage, and typed AiResult.
• FakeAiProvider (lib/src/ai/fake_ai_provider.dart):
  • Scripted, deterministic mock provider for CI and offline unit testing without requiring a 550 MB model download.
• GemmaProvider (lib/src/ai/providers/gemma_provider.dart):
  • Real on-device provider utilizing flutter_gemma with native function calling.
• ModelManager & GemmaModels catalogue (lib/src/ai/model_manager.dart):
  • Model lifecycle management (install, uninstallModel, isModelInstalled, listInstalledModels, getStorageInfo).
  • Curated catalogue: Gemma 3 1B, FunctionGemma 270M, Qwen3 0.6B, Gemma 4 E2B (excluding Gemma 3 270M).
  • Gated repository error mapping: maps 401 and 403 HTTP errors to clear Hugging Face token and access agreement instructions.
• Schema-as-tool bridge (lib/src/ai/tool_bridge.dart):
  • Translates Schema DSL to Tool specifications with Draft-07 JSON Schema parameters.
  • Sets ToolChoice.required and parses structured arguments from FunctionCallResponse.args.
  • Self-healing retry feedback loop when validation fails.
• Extraction strategies & provenance (lib/src/ai/extractor.dart):
  • Object extraction over BM25-ranked top-K chunks.
  • Map-reduce list extraction with chunking and configurable deduplication.
  • Strict provenance guarantee: deterministic structured data always takes priority over AI guesses.
  • Short-circuiting: zero inference when structured data already satisfies the schema.
  • Graceful degradation on timeout.
• Provider-derived token budget (lib/src/reduce/budget.dart):
  • TokenBudget.fromCapabilities() and TokenBudget.fromProvider().
• Public API: ScrapedPage.extractWithAi() and ScrapedPage.extractAsync().

Phase 2: deterministic reduction and structured data harvesting. Shrinks pages deterministically and satisfies many schemas with zero inference.

Added #

• Schema DSL (Field, Schema) with validation, automatic type coercion, and Draft-07 JSON Schema generation (Schema.toJsonSchema()).
• Structured data harvesters:
  • JsonLdHarvester: parses <script type="application/ld+json">, handles @graph and arrays.
  • MicrodataHarvester: W3C HTML Microdata (itemscope, itemtype, itemprop).
  • RdfaHarvester: W3C RDFa Lite (vocab, typeof, property).
  • OpenGraphHarvester: OpenGraph, Twitter Cards, canonical links, standard meta tags.
  • StructuredMapper: maps Schema.org types onto target schemas with field synonym resolution and provenance reporting (ExtractionCoverage).
• Readability scoring engine (ReadabilityScorer) with DOM node scoring, link density penalties, chrome filtering, sibling inclusion, and image preservation.
• GFM Markdown serializer (MarkdownWriter) with MarkdownOptions and real GitHub-Flavoured table rendering with th detection and pipe escaping.
• Chunking & ranking pipeline:
  • TokenEstimator: 4 chars/token heuristic.
  • Chunker: structure-preserving chunking that never splits tables, code blocks, or lists mid-structure.
  • Bm25Ranker: BM25 relevance scoring with query synonym expansion.
  • TokenBudget: context window management with reserved generation headroom.
• Tier-1 Public API:
  • AiScrapper.open() and AiScrapper.fromHtml() returning ScrapedPage.
  • ScrapedPage.article(), .markdown, .plainText, .metadata, .links, .images, .tables, .extract(schema).
• Golden fixture corpus grown to 15 fixtures (added commerce_rdfa, recipe_jsonld, event_microdata, article_blog_sidebar, jobs_jsonld).

Fixed #

• Fabricated currency bug: money parsing detects real symbols and codes (£ -> GBP, € -> EUR, $ -> USD, ¥ -> JPY). €99 never becomes $99.
• Phone extraction: strict format validation requiring real international/national phone shapes; rejects dates, SKUs, and prices.

Phase 1: the parsing core is replaced. Every extraction path now runs through package:html — a real HTML5 parser — instead of regular expressions.

Fixed #

• Nested containers no longer truncate. <div>(.*?)</div> stopped at the first closing tag, so querying any container returned a fragment and silently dropped the rest.
• Class filters no longer leak across the document. The regex lookahead ran with dotAll, so it scanned the whole remaining page — an element with no class at all matched a class filter because an unrelated later element carried it. Wrong data, no error.
• Timeouts are caught. The package declared its own TimeoutException, which shadowed dart:async's inside the same library, so on TimeoutException never matched what .timeout() throws. Every timeout was mislabelled as an unexpected network error.
• Backoff is exponential. It computed initialDelay × (multiplier × attempt) — linear — while documenting exponential. Jitter added, so clients no longer retry a struggling host in lockstep.
• Group-less regex patterns work. queryWithRegex read group 1 unconditionally, so a pattern of only non-capturing (?:…) groups threw RangeError, relabelled as "Failed to parse HTML" — pointing the caller at their markup instead of the group index.
• Regex is case-sensitive by default, matching Dart's own RegExp. 1.x forced case-insensitivity with no opt-out, so [A-Z][a-z]+ also matched lowercase.
• Non-breaking spaces are folded.   decodes to U+00A0 and survived whitespace normalisation, so text.contains('12 %') failed against 12\u{00A0}% invisibly.
• The size limit works. It was checked after the whole body had been buffered into memory; it now aborts mid-download.
• The cache is durable. It lived in Directory.systemTemp, which the OS may clear at any time, and rewrote every entry into one JSON file on each write.

Added #

• HtmlDocument / HtmlNode with CSS selectors, blockText, and URL resolution that honours <base href>.
• SelectorGuard, which refuses selectors package:html answers wrongly. Structural pseudo-classes like li:nth-child(2) return zero matches on markup that plainly contains them; a confidently empty result is worse than a refusal, so these raise InvalidSelectorException naming the workaround.
• RegexTarget, so a pattern can run against visible text instead of markup — the fix for URLs captured with a trailing </p>.
• robots.txt support, per-host rate limiting and a truthful User-Agent, all on by default.
• Conditional requests with ETag/Last-Modified, so an unchanged page costs a 304.
• Charset detection: BOM, then Content-Type, then <meta charset>. 1.x read only the header, so pages declaring encoding in markup came back as mojibake.
• CancellationToken. 1.x's cancel() completed an error on a Completer nobody awaited, producing an unhandled async error rather than stopping the work.
• A 10-page golden fixture corpus and tool/capture_fixture.dart.

Changed — breaking #

• TimeoutException → ScraperTimeoutException (the old name shadowed dart:async's).
• ScraperException is now sealed, with InvalidUrlException, HttpStatusException, RobotsDisallowedException, CancelledException and InvalidSelectorException split out. Every exception carries a userMessage safe to show a person.
• ContentFormatter and SmartExtractor take an HtmlDocument, not a String.
• ContentFormatter.toCleanHtml, removeClutter and extractReadableText removed.
• CacheManager replaced by CacheStore.
• estimateReadingTime reports seconds rather than rounding up to whole minutes.
• The platform gate no longer inspects Platform.environment for FLUTTER_TEST. Tests inject a PlatformInfo instead, so no test awareness ships in production code.

Removed #

• Price extraction. It stamped $ on every match regardless of currency, so €99 came back as $99. Fabricating currency is worse than returning nothing; real money parsing arrives with the structured-data work.
• Phone extraction. Its pattern matched dates, SKUs and IDs as freely as phone numbers. Returning when it can be done with region-aware parsing.

Phase 0 of the 2.0 rebuild: the package is renamed and correctly shaped. No scraping behaviour changed in this release — the 7 inherited test failures fail identically before and after, which is the evidence for that claim.

Breaking #

• Renamed flutter_scrapper → flutter_ai_scrapper.
• Single entrypoint. lib/mobile_scraper.dart and lib/flutter_mobile_scraper.dart are replaced by package:flutter_ai_scrapper/flutter_ai_scrapper.dart. Everything else moved under lib/src/ and is no longer importable directly.
• ScraperViewModel moved to the example app. A package should not ship a ChangeNotifier bound to a particular state-management library. Purpose-built widgets arrive in a later phase.
• provider is no longer a dependency, so the package imposes no state-management choice.
• Platform floors raised to Android minSdk 24 (was API 21) and iOS 15.0 (was iOS 12), as required by flutter_gemma 1.7.0. The old floors were never achievable alongside on-device AI.
• SDK floors raised to Dart >=3.12.0 and Flutter >=3.44.0.
• Android and iOS only, now declared in pubspec.yaml so pub.flutter-io.cn states it rather than leaving consumers to hit it at runtime.

Changed #

• Default User-Agent is now truthful and carries a contact URL.
• Analyzer tightened (strict-casts, strict-raw-types, strict-inference, plus correctness lints). Issue count went from 194 to 0.
• The demo app moved to example/, and the stale example_app/ — whose own widget test never compiled — was removed.

Known issues #

7 inherited tests fail, all traced to the regex parsing engine that Phase 1 replaces. Three are cases where the test is right and the library is wrong. See test/KNOWN_FAILURES.md.

Changed #

• Upgraded package version to 1.1.0
• Updated dependencies to latest versions

Added #

• Initial release of flutter_scrapper
• Basic HTML scraping functionality for mobile platforms
• Support for tag-based content extraction
• Support for regex-based content extraction
• Smart content extraction with auto-detection
• High-performance caching system
• Content formatting (plain text, markdown, clean HTML, readable)
• Comprehensive error handling
• Platform validation (Android/iOS only)
• Retry mechanism with exponential backoff
• Configurable timeout and headers
• Complete test coverage

Features #

• Smart Content Extraction: Auto-detect titles, descriptions, images, prices, and more
• High-Performance Caching: 50x faster repeated requests with intelligent caching
• Professional Content Formatting: Clean text, Markdown, readability mode
• Production Ready: Error handling, retry logic, resource management

Platform Support #

• ✅ Android
• ✅ iOS
• ❌ Web (by design)
• ❌ Desktop (by design)

Documentation #

• Comprehensive README with examples
• API documentation with dartdoc comments
• MVVM architecture explanation
• Usage examples and best practices