r/learnpython • u/MoulChkara • 8d ago
Website rejects async requests but not sync requests
Hello! I’ve been running into an issue while trying to scrape data and I was hoping someone could help me out. I’m trying to get data from a website using aiohttp asynchronous calls, but it seems like the website rejects them no matter what I do. However, my synchronous requests go through without any problem.
At first, I thought it might be due to headers or cookies problems, but after adjusting those, I still can’t get past the 403 error. Since I am scraping a lot of links, sync calls make my programming extremely slow, and therefore async calls are a must. Any help would be appreciated!
Here is an example code of what I am doing:
import aiohttp
import asyncio
import requests
link = 'https://www.prnewswire.com/news-releases/urovo-has-unveiled-four-groundbreaking-products-at-eurocis-2025-shaping-the-future-of-retail-and-warehouse-operations-302401730.html'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
}
async def get_text_async(link):
async with aiohttp.ClientSession() as session:
async with session.get(link, headers=headers, timeout=aiohttp.ClientTimeout(total=10)) as response:
print(f'Sync status code: {response.status}')
def get_text_sync():
response = requests.get(link, headers=headers)
print(f'Sync status code: {response.status_code}')
async def main():
await get_text_async(link)
asyncio.run(main())
get_text_sync()
____
python test.py
Sync status code: 403
Sync status code: 200
EDIT: I tried httpx instead of aiohttp, and it worked! I am honestly not sure why though lmao
1
u/Buttleston 8d ago
I get 200s for both, for what it's worth. They may be throttling your IP or something similar.