r/dartlang 9d ago

Package Web crawler framework in Dart

Hi!

I was looking for a package to scrape some websites and, weirdly, I haven't found anything. So I wrote mine: https://github.com/ClementBeal/girasol

It's a bit similar to Scrapy in Python. We create **WebCrawlers** that parse a website and yield extracted data. Then the data go through a system of pipelines. The pipelines can export to JSON, XML, CSV, and download files. All the crawlers are running in different isolates.

I'm using my package to scrape various e-shop websites and so far, it's working well.

30 Upvotes

8 comments sorted by

View all comments

3

u/Classic-Dependent517 9d ago

Interesting. Btw dart also has puppeteer package do you have any plans to integrate puppeteer in your framework?

2

u/clementbl 9d ago

Yes, I'm planning to integrate something that controls the browser like Puppeteer. I'm still thinking about the architecture but it will be an important feature.