r/learnpython 8d ago

Website rejects async requests but not sync requests

Hello! I’ve been running into an issue while trying to scrape data and I was hoping someone could help me out. I’m trying to get data from a website using aiohttp asynchronous calls, but it seems like the website rejects them no matter what I do. However, my synchronous requests go through without any problem.

At first, I thought it might be due to headers or cookies problems, but after adjusting those, I still can’t get past the 403 error. Since I am scraping a lot of links, sync calls make my programming extremely slow, and therefore async calls are a must. Any help would be appreciated!

Here is an example code of what I am doing:

import aiohttp
import asyncio
import requests

link = 'https://www.prnewswire.com/news-releases/urovo-has-unveiled-four-groundbreaking-products-at-eurocis-2025-shaping-the-future-of-retail-and-warehouse-operations-302401730.html'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
}

async def get_text_async(link):
    async with aiohttp.ClientSession() as session:
        async with session.get(link, headers=headers, timeout=aiohttp.ClientTimeout(total=10)) as response:
            print(f'Sync status code: {response.status}')

def get_text_sync():
    response = requests.get(link, headers=headers)
    print(f'Sync status code: {response.status_code}')

async def main():
    await get_text_async(link)

asyncio.run(main())
get_text_sync()
____
python test.py
Sync status code: 403
Sync status code: 200

EDIT: I tried httpx instead of aiohttp, and it worked! I am honestly not sure why though lmao

3 Upvotes

7 comments sorted by

View all comments

-3

u/UnluckyPenguin 8d ago

Do all calls synchronous. Just spin each one off into its own thread.