scrapeWithPagination<T> method

Future<PaginationResult<T>> scrapeWithPagination<T>({
  1. required String url,
  2. required PaginationConfig config,
  3. required Future<T> extractor(
    1. String html,
    2. String pageUrl
    ),
  4. Map<String, String>? headers,
  5. int? timeout,
  6. int? retries,
})

Scrapes multiple pages with pagination

url is the starting URL config is the pagination configuration extractor is a function that extracts data from each page headers are additional headers to send with the request timeout is the timeout for the request in milliseconds retries is the number of retry attempts

Implementation

Future<PaginationResult<T>> scrapeWithPagination<T>({
  required String url,
  required PaginationConfig config,
  required Future<T> Function(String html, String pageUrl) extractor,
  Map<String, String>? headers,
  int? timeout,
  int? retries,
}) async {
  return paginationHandler.scrapeWithPagination(
    url: url,
    config: config,
    extractor: extractor,
    headers: headers,
    timeout: timeout,
    retries: retries,
  );
}