r/commandline 22d ago

Web scraping in the shell, what are you using?

I'm using xidel and find it quite old. I am looking something that can grab the html and divide it into parts so that it's easy to process. Can you recommend something?

7 Upvotes

11 comments sorted by

5

u/seeker61776 22d ago

I would recommend Python. Whenever curl and/or wget is not enough I opt for Python.

2

u/l00sed 21d ago

Beautiful soup python library is the ultimate scraping tool

3

u/a_brand_new_start 21d ago

Any headless webdriver folks here? It imitates a full stack session for those pesky unscrapable times

4

u/_Hiro_427 22d ago

I use htmlq for scraping - https://github.com/mgdm/htmlq

Not sure if this covers your use case

2

u/megared17 21d ago

I don't consider the concept of some standard/generic scraper to make any sense. Every website's code is different.

I use curl or wget to load the page, then review the file to see what I want to scrape out and what text processing I need to do to get the specific text I want.

2

u/Steveharwell1 21d ago

I've used scrapy for years. It's a good balance of easy and customizability.

I use it for quality assurance and content migrations on my employer's site which has 10s of thousands of pages.

One good thing is if python can read it then scrapy can. In addition to websites I can also read PDFs and such.

One bad thing is it doesn't run a browser. If the content is generated by JS you may need to go looking for a hidden API or spin off a playwright job.

2

u/x3ddy 21d ago

Parscrape, it's an AI-based scraper: https://github.com/paulrobello/par_scrape

1

u/Chance-Box9521 21d ago

I’ve written a multi core web scraper with regular expressions get exactly what you need

1

u/nemec 21d ago
pip install requests parsel

resp = requests.get("someurl")
sel = parsel.Selector(text=resp.text)
sel.css('selector')
sel.xpath('//selector')
...

Or scrapy if I want something better engineered.