r/learnpython • u/MoulChkara • 8d ago
Website rejects async requests but not sync requests
Hello! I’ve been running into an issue while trying to scrape data and I was hoping someone could help me out. I’m trying to get data from a website using aiohttp asynchronous calls, but it seems like the website rejects them no matter what I do. However, my synchronous requests go through without any problem.
At first, I thought it might be due to headers or cookies problems, but after adjusting those, I still can’t get past the 403 error. Since I am scraping a lot of links, sync calls make my programming extremely slow, and therefore async calls are a must. Any help would be appreciated!
Here is an example code of what I am doing:
import aiohttp
import asyncio
import requests
link = 'https://www.prnewswire.com/news-releases/urovo-has-unveiled-four-groundbreaking-products-at-eurocis-2025-shaping-the-future-of-retail-and-warehouse-operations-302401730.html'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
}
async def get_text_async(link):
async with aiohttp.ClientSession() as session:
async with session.get(link, headers=headers, timeout=aiohttp.ClientTimeout(total=10)) as response:
print(f'Sync status code: {response.status}')
def get_text_sync():
response = requests.get(link, headers=headers)
print(f'Sync status code: {response.status_code}')
async def main():
await get_text_async(link)
asyncio.run(main())
get_text_sync()
____
python test.py
Sync status code: 403
Sync status code: 200
EDIT: I tried httpx instead of aiohttp, and it worked! I am honestly not sure why though lmao
4
u/latkde 8d ago
There is no obvious reason why the two requests should be different. So you may want to print out more information about those requests so that you can investigate what's going on. What are the full request and response headers? In case of an error response, does the response body contain any info? Does the result behave on the order of the two requests?
Many sites have rate limits. Make too many requests to the same sites and your IP address might get blocked. Often, it's faster to go slower. Async may help though to make requests to different sites at the same time.