sloth_link_crawler 0.0.2 
sloth_link_crawler: ^0.0.2 copied to clipboard
Published 3 years ago • 
ynnob.com
ynnob.com(WIP!) Web crawler to crawl all links of a website. Looking foward to respect robots.txt and use custom user agents.
2 versions
Stable versions of sloth_link_crawler
| Version | Min Dart SDK | Uploaded | Documentation | Archive | |
|---|---|---|---|---|---|
| 0.0.2 | 2.18 | 3 years ago |  |  | |
| 0.0.1 | 2.18 | 3 years ago | | |
Metadata
3
likes
140
points
9
downloads
Publisher
Weekly Downloads
Metadata
(WIP!) Web crawler to crawl all links of a website. Looking foward to respect robots.txt and use custom user agents.
Homepage
Repository (GitHub)
View/report issues
Documentation
License
BSD-3-Clause (license)
Dependencies
beautiful_soup_dart, http, http_interceptor, queue, robots_txt
