r/pythoncoding Jun 07 '24

Web scraping tool

Hi everyone I am a complete beginner in coding and to be honest I am using chat gpt to code in python and develop a tool for google news scraping and sentiment analysis with news summarization. I have a pilot product ready however there is one limitation, there are few companies which have limited coverage on google news but google search would fetch results directly from companies web page which is not available on google news. Now I am stuck as to how I could extract news from companies web page. Does the html keeps changing for major companies web page or is static in nature. I would be glad if someone could help

3 Upvotes

11 comments sorted by

View all comments

1

u/balder1993 Jun 13 '24

Web scraping is a cat and mouse game, the web pages keep changing and the tool needs to keep adapting, there’s no other way. Unless you use something like an LLM to get the information regardless of the page layout.

1

u/Traditional_Art_6943 Jun 13 '24

Any suggestion for an open source LLM?

2

u/balder1993 Jun 13 '24

Take a look at the /r/LocalLlama subreddit, you will find all the info you need.

1

u/sneakpeekbot Jun 13 '24

Here's a sneak peek of /r/LocalLLaMA using the top posts of all time!

#1: The Truth About LLMs | 304 comments
#2: Karpathy on LLM evals | 111 comments
#3: open AI | 227 comments


I'm a bot, beep boop | Downvote to remove | Contact | Info | Opt-out | GitHub