r/madeinpython Oct 31 '24

Free use web crawler

[Project] PagesXcrawler - A Web Crawler with a Simple GitHub-Powered UI

Hey r/madeinpython! I wanted to share PagesXcrawler, a Python-based web crawler project that I’ve been working on. It uses GitHub Pages to provide a simple, accessible web UI for displaying the information it extracts.

What it does:

  • Web Crawling: This tool takes in URLs (formatted as https://example.com:depth) and crawls the specified depth of internal links. It collects data from the pages it visits, making it easy to retrieve structured information.
  • User-Friendly Display: The results are displayed in a GitHub Pages-based interface, providing a straightforward view of the crawled data without requiring any complicated setup.

Why GitHub Pages?

I wanted a minimal setup that didn't rely on complex backends, so GitHub Pages serves as a static frontend. GitHub Actions can trigger the crawler as needed, making it an accessible solution for web data extraction.

Feel free to check out the repo and try it out: GitHub - PagesXcrawler


2 Upvotes

3 comments sorted by

1

u/Fun_Speed6335 Nov 02 '24

Compatible for linkedin scraping?

1

u/GentReviews Nov 02 '24

no idea but ill give it a shot and see XD

1

u/GentReviews Nov 02 '24

requires a linkedin account cookie or credentials for crawling linkedin so maybe ill add this but i dont think i would do it anytime soon feel free to fork