extractDataStream method

Stream<String> extractDataStream({
  1. required String url,
  2. required String selector,
  3. String? attribute,
  4. bool asText = true,
  5. Map<String, String>? headers,
  6. int? timeout,
  7. int? retries,
  8. int priority = 0,
  9. int chunkSize = 1024 * 1024,
})

Extracts data from a URL using streaming for memory efficiency

url is the URL to fetch selector is the CSS selector to use attribute is the attribute to extract (optional) asText whether to extract the text content (default: true) headers are additional headers to send with the request timeout is the timeout for the request in milliseconds retries is the number of retry attempts priority is the priority of the request (higher values = higher priority) chunkSize is the size of each chunk to process (default: 1024 * 1024 bytes)

Implementation

Stream<String> extractDataStream({
  required String url,
  required String selector,
  String? attribute,
  bool asText = true,
  Map<String, String>? headers,
  int? timeout,
  int? retries,
  int priority = 0,
  int chunkSize = 1024 * 1024, // 1MB chunks
}) async* {
  final htmlStream = await fetchHtmlStream(
    url: url,
    headers: headers,
    timeout: timeout,
    retries: retries,
    priority: priority,
  );

  final dataStream = _streamingParser.extractDataStream(
    htmlStream: htmlStream,
    selector: selector,
    attribute: attribute,
    asText: asText,
    chunkSize: chunkSize,
  );

  await for (final item in dataStream) {
    yield item;
  }
}