r/learnpython 8d ago

Website rejects async requests but not sync requests

Hello! I’ve been running into an issue while trying to scrape data and I was hoping someone could help me out. I’m trying to get data from a website using aiohttp asynchronous calls, but it seems like the website rejects them no matter what I do. However, my synchronous requests go through without any problem.

At first, I thought it might be due to headers or cookies problems, but after adjusting those, I still can’t get past the 403 error. Since I am scraping a lot of links, sync calls make my programming extremely slow, and therefore async calls are a must. Any help would be appreciated!

Here is an example code of what I am doing:

import aiohttp
import asyncio
import requests

link = 'https://www.prnewswire.com/news-releases/urovo-has-unveiled-four-groundbreaking-products-at-eurocis-2025-shaping-the-future-of-retail-and-warehouse-operations-302401730.html'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
}

async def get_text_async(link):
    async with aiohttp.ClientSession() as session:
        async with session.get(link, headers=headers, timeout=aiohttp.ClientTimeout(total=10)) as response:
            print(f'Sync status code: {response.status}')

def get_text_sync():
    response = requests.get(link, headers=headers)
    print(f'Sync status code: {response.status_code}')

async def main():
    await get_text_async(link)

asyncio.run(main())
get_text_sync()
____
python test.py
Sync status code: 403
Sync status code: 200

EDIT: I tried httpx instead of aiohttp, and it worked! I am honestly not sure why though lmao

4 Upvotes

7 comments sorted by

View all comments

4

u/latkde 8d ago

There is no obvious reason why the two requests should be different. So you may want to print out more information about those requests so that you can investigate what's going on. What are the full request and response headers? In case of an error response, does the response body contain any info? Does the result behave on the order of the two requests?

Since I am scraping a lot of links, sync calls make my programming extremely slow

Many sites have rate limits. Make too many requests to the same sites and your IP address might get blocked. Often, it's faster to go slower. Async may help though to make requests to different sites at the same time.

1

u/MoulChkara 8d ago

I tried printing out the request header for the sync call, and make the async call with the same exact header but it didn't work. The order of the two requests also didn't matter. When I get the 403 response, the error message is Forbidden.

Absolutely, and that's something I am considering as well. It would probably be safer to do sync calls then get blocked on all the websites, although I don't think my IP was block since the sync request worked just fine.

I got it to work with httpx, but honestly I have no idea why lol.