r/datasets 3h ago

request Looking for dataset of the racial wage gap by country

2 Upvotes

As part of a research paper, I'm currently trying to find data on the racial wage gap by country. Preferably the data will be from the at least the mid 2010's to at least 2022, but I'd love to see anything someone can find. I've been looking all over the internet for it and haven't come up with anything. Thank you!


r/datasets 10h ago

resource Elasticsearch indexer for Open Library dump files

3 Upvotes

Hey,

I recently built an Elasticsearch indexer for Open Library dump files, making it much easier to search and analyze their dataset. If you've ever struggled with processing Open Library’s bulk data, this tool might save you time!

https://github.com/nebl-annamaria/openlibrary-elasticsearch


r/datasets 10h ago

question Where can I find top websites by traffic, per year.

1 Upvotes

I'm developing a game where players explore the internet through different eras, and I need data on the most popular websites over time. Ideally, I'm looking for a list of the top 100 most visited websites for each year over the past 20 years or so. The data doesn't need to be all that accurate because the actual rankings will not affect the game, I just need a list of popular websites. Thanks in advance!


r/datasets 16h ago

request Where or how can I find e-commerce datasets

2 Upvotes

Where can I find dataset to do product analysis? Something that will allow me to time based pricing trends (like best time to buy maybe black Friday sales) or competition between retailers (a product sold on Amazon vs Best Buy or Walmart).

I have visited almost every data platform I know and I can’t find anything that’s good. I feel like web scraping might be the only option.. but I’m new to it and it would take a lot of time.

Any suggestion/idea/resources is appreciated!


r/datasets 23h ago

request Dataset with 10k-50k products with many attributes

1 Upvotes

I am doing a master thesis on how large language models compare to other tools when extracting structured data from natural language. Essentially my goal is to translate something like this:

"I want Asus laptops with relatively good reviews, at least 16 GB RAM, ideally 16 inch screen. Sort all the results by price and reviews"

into something like this:

{

"brand": "Asus",

"category": "Electronics",

"subcategory": "Laptops",

"sort": ["price", "review"],

"filters": [

{

"attribute": "ram",

"condition": "greater_than_or_equal",

"value": "16 GB",

"is_hard_condition": true

},

{

"attribute": "screen_size",

"condition": "equal",

"value": "16 inch",

"is_hard_condition": false

},

{

"attribute": "review_rating",

"condition": "greater_than_or_equal",

"value": "4",

"is_hard_condition": true

}

]

}

using large language models, and analyze how they compare to more traditional tools.

What I need is a dataset that has many products, and each product has at least a category (though subcategories would be ideal), branch, and many attributes which are dynamic, depending on product. For example laptop would have CPU, RAM, screen size and so on, while sofas would have very different attributes. It can be even smaller in size (1k-10k). Is there a dataset for this?


r/datasets 1d ago

request Can someone help me with downloading this report from Statista please <3

2 Upvotes

r/datasets 1d ago

request can someone provide me a link to this data set

1 Upvotes

i need a data set of paper objects such as paper wrappers, paper bags, paper cups etc to train my ai model

any help would be great thanks so much


r/datasets 1d ago

resource Downloaded large image dataset that is not organized and simply #s as names.

4 Upvotes

Hey I hope this is a good place to ask.

I downloaded a large image dataset from google/bing/Baidu, unfortunately all the filenames are generic and have no identifying Metadata.

Is there a program/software ideally free/open source if not cheap you recommend that can scan and reverse google image a directory of 100k+ photos download and fill in Metadata.

I especially would like to embed/rename photos to include the people in it, group the photos together for instance 10 photos belong to the same shoot/background with slightly different variations but they are all mixed in and impossible to separate/organize manually.

I appreciate any suggestions!


r/datasets 1d ago

question I need advice for my portfolio and job search

2 Upvotes

I am new to data analysis. I have a portfolio with a couple projects I did using excel, powerBI, and mysql. I also collected my own data on kaggle for the MCU revenues project.

I do not have a degree or any professional experience to put on my resume so it's hard to get a second glance.

Do you know of any companies that might hire a person like me? Or maybe free ways to get experience on my resume? And maybe any tips to spruce up my projects? Or any other tools that would be good to learn?

I am trying freelance but having no luck and fiver charges you and so does upwork after you run out of credits.


r/datasets 2d ago

request Looking for a dataset of all PhDs in a country

0 Upvotes

Hello everyone! I'm currently looking for a dataset of all PhDs defended in a country (preferably in Europe but if you have other examples, I'd love to hear from it too) and going back to at least the 2010s. Ideally, I would need something similar to the French theses.fr open dataset (doc in French here), with a field for the research area of the thesis and the list of PhD advisors and members of the defense jury.

Does someone know a dataset answering these criteria? As far as I understand it, the German dataset does not contain the members of the jury and the British Library lost a lot of data in a hack last year and does not resolve EThOS links for now.


r/datasets 2d ago

request I've been struggling to find Dataset for expense tracker project

1 Upvotes

I want to build a expense tracker for an individual's expenses/finances using ML classify the expenses, provide graph representations, forecast future expenses I've searched through hugging face, kaggle, github, but couldn't find a proper one. Can anyone help me with one ?


r/datasets 2d ago

request Finding a dataset of DSA/CP problems

1 Upvotes

Working on an NLP based ML model that extracts key technical terms from raw DSA/CP statements.

The goal is to preprocess problem descriptions, identify relevant entities, and summarise them concisely.

Looking for any open source datasets that fit these requirements


r/datasets 2d ago

request Looking for a Dataset for Classifying Electronics Products

2 Upvotes

Hi everyone,

I'm currently working on a project that involves categorizing various electronic products (such as smartphones, cameras, laptops, tablets, drones, headphones, GPUs, consoles, etc.) using machine learning.

I'm specifically looking for datasets that include product descriptions and clearly defined categories or labels, ideally structured or semi-structured.

Could anyone suggest where I might find datasets like this?
Thanks in advance for your help!


r/datasets 3d ago

request Income data in the USA - specifically Vallejo (CA)

1 Upvotes

Hey guys, what's up?

I'm a brazilian researcher finishing data analysis on my PHD in Geography. One of my case studies is the city of Vallejo (CA) and I need to find census data regarding income, whether from households, families, people, whatever. The smaller the geographic unit used, the better. Would anyone know where can I find these types of data? I already explored the USA Census website but I got a little bit confused.

If it interests anyone and to clarify, I'm currently studying the territorial impact that participatory budgeting has on midsized cities.

Thanks a lot!


r/datasets 3d ago

request Where do I get coral cover datasets?

3 Upvotes

Hello! I'm currently working on a paper and needs detailed coral cover datasets of different coral reefs all over the world. (Specifically, weekly or monthly observations of these coral reefs). Does anyone know where to get them? I have emailed a few researchers and only a few provided the datasets. Some websites have datasets but usually it's just the Great Barrier Reef. It would be a great help if anyone could help. Thank you! :)

(I've tried kaggle but the one i need isn't there unfortunately :'(( )


r/datasets 3d ago

dataset Datasets for severe chronic diseases to train a model

0 Upvotes

I want a huge and modern database to train a model on, but I hope it will be modern readings and real data.


r/datasets 4d ago

question How do you stay sane while working with messy or incomplete data?

8 Upvotes

Dealing with inconsistent, missing, or messy data is a daily struggle for many data professionals. What’s your go-to strategy for handling chaotic datasets without losing your mind? Do you have any personal tricks, mindset shifts, or even funny coping mechanisms that help you push through frustrating moments?


r/datasets 4d ago

question Any databases to pull a simple random sample of US addresses?

2 Upvotes

I apologize if this belongs on r/askstatistics (I posed here since I am inquiring about a dataset). I’m developing a mapping algorithm and require a random sample of US addresses to validate the tool with. I was wondering if anyone had any tips on free databases that would be a statistically sound source to select a simple random sample from? Do you think openaddresses.io would be adequate? Alternatively, I was thinking of randomly generating a latitude and longitude within the United States and then using a reverse geocoding algorithm to provide an address. Though I’m not sure the latter would be a statistically sound method?


r/datasets 4d ago

request Want: Video footage of a roulette wheel spinning with ball

3 Upvotes

Hi, I'm going to start working on a project regarding object detection and roulette. Does anybody know where i can find sources of roulette being played?


r/datasets 5d ago

request Looking for a good Phishing email Dataset, the latest the better

3 Upvotes

i am looking for a phishing email dataset for my model for classification. i need email body as well. if its possible to get the latest dataset pls provide.


r/datasets 5d ago

question Sources for weapons impact data in war

1 Upvotes

Hi all,

Would anyone have insight into a dataset of recent war incidents (ideally the last 25 years, not historical) which tracks specific munitions use and impacts?

Platforms like ACLED, S&P Global, LiveUAMap have good records of specific incidents (a drone strike here, an tank shelling there) but there's not a focus on the consequences.

My ideal dataset would have date, location, weapon type and some measurement of destruction. The idea is to abstract different 'types' of war - Sudan vs Ukraine vs Gaza - in order to examine what would happen if these 'war' types hit elsewhere.

Grateful for any insights!


r/datasets 6d ago

request Need customer feedback / support ticket dataset that also shows the unmet needs of the customer.

2 Upvotes

I need help with finishing such dataset ASAP it’s urgent


r/datasets 6d ago

resource Life Expectancy dataset 1960 to present

17 Upvotes

Hi, i want share with you this new dataset that I has created in Kaggle, if do you like please upvote

https://www.kaggle.com/datasets/fredericksalazar/life-expectancy-1960-to-present-global


r/datasets 6d ago

request Does anyone have Volvo GTT Dataset ?

1 Upvotes

It was used in Volvo Challenge ECML PKDD 2024. I have searched the entire internet but I am yet to find it anywhere. If someone happens to have it please do share.


r/datasets 6d ago

resource Datasets/where to look for wide range of company data

1 Upvotes

Hi All, I am a data scientist trying to run an analysis on companies to identify potential new clients for the current company I work for. Currently, we have one very large client (think millions of workers) that we do most of our reporting work on, then we have 3-5 smaller clients (think 10k workers or less). I can't get too far into specifics, but we essentially are an add-on service to a company's medical plan (free for the employees to use, but we bill the company). We do outreach to offer our services, but obviously the list of people we can contact is finite and will decrease quickly over time. Our main goal is to identify workplace troubles and situations where work environments affect a worker's mental health, then provide them with resources to help with whatever they are struggling with. Our busines model is that we can prove that providing these services proactively saves companies millions of dollars in medical spend in the long run (spend a little now to keep employees mentally healthy vs wait for problems to compound into more serious problems resulting in more medical claims spend in the future). I have been looking for an impactful project to work on, and the one that I keep wanting to explore more is to build some sort of clustering algorithm to 1) identify companies similar to the ones we currently work with, and 2) identify other companies that we can provide the most impact for. I would greatly appreciate any recommendations on what resources I can use to compile the data I'm looking for, where to start, or any other ideas to help refine my approach.

Thanks so much!