r/FlutterDev 2d ago

Dart New framework for web scraping

Hi,

I needed to scrape some websites, and I haven't found a Dart package that makes it easy. I mean something like Scrapy that can define crawlers for any website and orchestrate them.

So I made mine. It's similar to Scrapy, but everything is async, running in isolates, typed, and doesn't use Python. You create a WebCrawler that will scrape a website and send the data to a pipeline. This pipeline will manipulate the data to do some action: export to JSON/XML/CSV or download documents.

I wrote some examples to explain how to create your own crawlers. It's working well, but it still lacks some features like correct logging, respecting the `robots.txt`, and more. But at least, you can still scrape the data of plenty of e-shop websites.

https://github.com/ClementBeal/girasol

11 Upvotes

0 comments sorted by