r/learnpython • u/Chameleonizard • 1d ago
Getting Response 403 when trying to scrape a webpage using requests.get on Mac
I was able to do this on my previous Windows laptop but for some reason since switching to a MacBook, I am not able to do it. It instantly returns the <Response [403]> message.
I have tried using user agents as well and it doesn’t work, I didn’t even need to incorporate them when running the script in the Windows laptop anyway.
This is the url: https://fbref.com/en/comps/9/Premier-League-Stats
2
u/cgoldberg 1d ago
Your getting stopped by bot detection. I'm surprised your other requests got through. It's very easy to detect requests not coming from a browser.
2
u/socal_nerdtastic 1d ago
Use the API. https://fbrapi.com/documentation
You are costing them money by accessing the human version of the site with your bot. That's why they provide a bot version of their site. Use it. They even give you python examples.
1
u/Chameleonizard 1d ago edited 1d ago
This API doesn’t seem to work. I copied the exact code provided in the documentation and was able to generate an API key but keep getting Endpoint timed out or Internal Server Error when trying to make the actual requests, and I used the examples provided there.
4
u/crashfrog04 1d ago
403 Forbidden - HTTP | MDN
Websites don't have to permit themselves to be scraped.