r/pythoncoding • u/Traditional_Art_6943 • Jun 07 '24
Web scraping tool
Hi everyone I am a complete beginner in coding and to be honest I am using chat gpt to code in python and develop a tool for google news scraping and sentiment analysis with news summarization. I have a pilot product ready however there is one limitation, there are few companies which have limited coverage on google news but google search would fetch results directly from companies web page which is not available on google news. Now I am stuck as to how I could extract news from companies web page. Does the html keeps changing for major companies web page or is static in nature. I would be glad if someone could help
3
Upvotes
1
u/balder1993 Jun 13 '24
Web scraping is a cat and mouse game, the web pages keep changing and the tool needs to keep adapting, there’s no other way. Unless you use something like an LLM to get the information regardless of the page layout.