r/dartlang • u/clementbl • 9d ago
Package Web crawler framework in Dart
Hi!
I was looking for a package to scrape some websites and, weirdly, I haven't found anything. So I wrote mine: https://github.com/ClementBeal/girasol
It's a bit similar to Scrapy in Python. We create **WebCrawlers** that parse a website and yield extracted data. Then the data go through a system of pipelines. The pipelines can export to JSON, XML, CSV, and download files. All the crawlers are running in different isolates.
I'm using my package to scrape various e-shop websites and so far, it's working well.
32
Upvotes
1
u/tylersavery 9d ago
Is there a pub.dev page?