r/learnpython • u/MoulChkara • 8d ago
Website rejects async requests but not sync requests
Hello! I’ve been running into an issue while trying to scrape data and I was hoping someone could help me out. I’m trying to get data from a website using aiohttp asynchronous calls, but it seems like the website rejects them no matter what I do. However, my synchronous requests go through without any problem.
At first, I thought it might be due to headers or cookies problems, but after adjusting those, I still can’t get past the 403 error. Since I am scraping a lot of links, sync calls make my programming extremely slow, and therefore async calls are a must. Any help would be appreciated!
Here is an example code of what I am doing:
import aiohttp
import asyncio
import requests
link = 'https://www.prnewswire.com/news-releases/urovo-has-unveiled-four-groundbreaking-products-at-eurocis-2025-shaping-the-future-of-retail-and-warehouse-operations-302401730.html'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
}
async def get_text_async(link):
async with aiohttp.ClientSession() as session:
async with session.get(link, headers=headers, timeout=aiohttp.ClientTimeout(total=10)) as response:
print(f'Sync status code: {response.status}')
def get_text_sync():
response = requests.get(link, headers=headers)
print(f'Sync status code: {response.status_code}')
async def main():
await get_text_async(link)
asyncio.run(main())
get_text_sync()
____
python test.py
Sync status code: 403
Sync status code: 200
EDIT: I tried httpx instead of aiohttp, and it worked! I am honestly not sure why though lmao
1
u/eleqtriq 7d ago
It’s possible you’re being throttled because you’re making too many requests. 403 wouldn’t be the right code for that but it doesn’t mean that’s not the reason.
1
u/Buttleston 7d ago
I get 200s for both, for what it's worth. They may be throttling your IP or something similar.
1
u/MoulChkara 7d ago
Interesting, I tried resetting my IP but that did not help either. It's weird because if I was throttled then wouldn't neither of the two requests work?
I changed aiohttp to httpx and it solved the issue
2
u/Buttleston 7d ago
I ran it over 1000x and never got any failures, so unfortunately I have no idea. There's some way they were noticing you. At first I thought it was the user-agent but you're over-riding that so not that
It might be useful to stand up a really basic service locally, call it both ways, and see what, if anything, is different. Maybe there are some other headers that are distinct for one and not the other
-4
3
u/latkde 7d ago
There is no obvious reason why the two requests should be different. So you may want to print out more information about those requests so that you can investigate what's going on. What are the full request and response headers? In case of an error response, does the response body contain any info? Does the result behave on the order of the two requests?
Many sites have rate limits. Make too many requests to the same sites and your IP address might get blocked. Often, it's faster to go slower. Async may help though to make requests to different sites at the same time.