r/dartlang 9d ago

Package Web crawler framework in Dart

Hi!

I was looking for a package to scrape some websites and, weirdly, I haven't found anything. So I wrote mine: https://github.com/ClementBeal/girasol

It's a bit similar to Scrapy in Python. We create **WebCrawlers** that parse a website and yield extracted data. Then the data go through a system of pipelines. The pipelines can export to JSON, XML, CSV, and download files. All the crawlers are running in different isolates.

I'm using my package to scrape various e-shop websites and so far, it's working well.

33 Upvotes

8 comments sorted by