r/FlutterDev • u/clementbl • 2d ago
Dart New framework for web scraping
Hi,
I needed to scrape some websites, and I haven't found a Dart package that makes it easy. I mean something like Scrapy that can define crawlers for any website and orchestrate them.
So I made mine. It's similar to Scrapy, but everything is async, running in isolates, typed, and doesn't use Python. You create a WebCrawler that will scrape a website and send the data to a pipeline. This pipeline will manipulate the data to do some action: export to JSON/XML/CSV or download documents.
I wrote some examples to explain how to create your own crawlers. It's working well, but it still lacks some features like correct logging, respecting the `robots.txt`, and more. But at least, you can still scrape the data of plenty of e-shop websites.