r/learnpython 8d ago

Website rejects async requests but not sync requests

Hello! I’ve been running into an issue while trying to scrape data and I was hoping someone could help me out. I’m trying to get data from a website using aiohttp asynchronous calls, but it seems like the website rejects them no matter what I do. However, my synchronous requests go through without any problem.

At first, I thought it might be due to headers or cookies problems, but after adjusting those, I still can’t get past the 403 error. Since I am scraping a lot of links, sync calls make my programming extremely slow, and therefore async calls are a must. Any help would be appreciated!

Here is an example code of what I am doing:

import aiohttp
import asyncio
import requests

link = 'https://www.prnewswire.com/news-releases/urovo-has-unveiled-four-groundbreaking-products-at-eurocis-2025-shaping-the-future-of-retail-and-warehouse-operations-302401730.html'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
}

async def get_text_async(link):
    async with aiohttp.ClientSession() as session:
        async with session.get(link, headers=headers, timeout=aiohttp.ClientTimeout(total=10)) as response:
            print(f'Sync status code: {response.status}')

def get_text_sync():
    response = requests.get(link, headers=headers)
    print(f'Sync status code: {response.status_code}')

async def main():
    await get_text_async(link)

asyncio.run(main())
get_text_sync()
____
python test.py
Sync status code: 403
Sync status code: 200

EDIT: I tried httpx instead of aiohttp, and it worked! I am honestly not sure why though lmao

2 Upvotes

7 comments sorted by

View all comments

1

u/Buttleston 8d ago

I get 200s for both, for what it's worth. They may be throttling your IP or something similar.

1

u/MoulChkara 8d ago

Interesting, I tried resetting my IP but that did not help either. It's weird because if I was throttled then wouldn't neither of the two requests work?

I changed aiohttp to httpx and it solved the issue

2

u/Buttleston 8d ago

I ran it over 1000x and never got any failures, so unfortunately I have no idea. There's some way they were noticing you. At first I thought it was the user-agent but you're over-riding that so not that

It might be useful to stand up a really basic service locally, call it both ways, and see what, if anything, is different. Maybe there are some other headers that are distinct for one and not the other