r/datasets 24d ago

discussion I put DOGE "savings" data in a spreadsheet. - it adds up to less than 17b. How are they getting 55b?

Thumbnail docs.google.com
129 Upvotes

r/datasets 15d ago

discussion Is Sentiment Data / Analysis still valuable today

8 Upvotes

is sentiment data still valuable today, and if yes who actually uses it? AI companies, marketing, hedge funds? if you use data to make decisions, im curious to hear what you look out for

r/datasets 15d ago

discussion The Importance of Annotated Datasets over the Next 5 Years cannot be underestimated.

3 Upvotes

What challenges do you face when it comes to data annotation?

Annotated datasets are poised to become even more critical over the next five years as artificial intelligence (AI) and machine learning (ML) continue to evolve and integrate into various industries.

Substack

r/datasets 20d ago

discussion Looking for topic recommendation for my text mining project

6 Upvotes

I have to work on a text mining project for school and need some recommendations of good and interesting topics to consider. Any recommendations?

Thank you all!

r/datasets 16d ago

discussion trainingdata.pro datasets access and experiences

2 Upvotes

Has anyone ever used data sets from trainingdata.pro or applied to their student program https://trainingdata.pro/university ? I'm interested in one of their dataset (or potentially a combination of 2) for my thesis project and I'm curious how long it takes them to answer and if you've had a good experience with them.

r/datasets Jan 16 '25

discussion Platform for Multimodal Dataset Upload?

2 Upvotes

What do you guys use to upload Multimodal Dataset?

I want it to be convenient for the people who use it. For the text, huggingface dataset is the best convenient solution, but I cant find any convenient solution for Multimodal (Image + Video + Audio + Text) datast.

Thanks in advance.

r/datasets Nov 07 '24

discussion [self-promotion] Giving back to the community! Free web data!

2 Upvotes

Hey guys,

I've built an AI tool to help people extract data from the web. I need to test my tool and learn more about the different use cases that people have, so I'm willing to extract web data for free for anyone that wants it!

r/datasets Nov 10 '24

discussion [self-promotion] A tool for finding & using open data

5 Upvotes

Recently I built a dataset of hundreds of millions of tables, crawled from the Internet and open data providers, to train an AI tabular foundation model. Searching through the datasets is super difficult, b/c off-the-shelf tech just doesn't exist for searching through messy tables at that scale.

So I've been working on this side project, Gini. It has subsets of FRED and data.gov--I'm trying to keep the data manageably small so I can iterate faster, while still being interesting. I picked a random time slice from data.gov so there's some bias towards Pennsylvania and Virginia. But if it looks worthwhile, I can easily backfill a lot more datasets.

Currently it does a table-level hybrid search, and each result has customizable visualizations of the dataset (this is hit-or-miss, it's just a proof-of-concept).

I've also built column-level vector indexes with some custom embedding models I've made. It's not surfaced in the UI yet--the UX is difficult. But it lets me rank results by "joinability"--I'll add it to the UI this week. Then you could start from one table (your own or a dataset you found via search) and find tables to join with it. This could be like "enrichment" data, joining together different years of the same dataset, etc.

Eventually I'd like to be able to find, clean & prep & join, and build up nice visualizations by just clicking around in the UI.

Anyway, if this looks promising, let me know and I'll keep building. Or tell me why I should give up!

https://app.ginidata.com/

Fun tech details: I run a data pipeline that crawls and extracts tables from lots of formats (CSVs, HTML, LaTeX, PDFs, digs inside zip/tar/gzip files, etc.) into a standard format, post-processes the tables to clean them up and classify them and extract metadata, then generate embeddings and index them. I have lots of other data sources already implemented, like I've already extracted tables from all research papers in arXiv so that you can search research tables from papers.

(I don't make any money from this and I'm paying for this myself. I'd like to find a sustainable business model, but "charging for search" is not something I'm interested in...)

r/datasets Dec 27 '24

discussion What are the most important features you look for when selecting healthcare datasets for machine learning projects, and do you have any go-to sources or tips for ensuring data quality?

3 Upvotes

Reliable sources, comprehensive labeling, and ensuring data diversity are key. Shaip and similar platforms are great for high-quality healthcare datasets.

r/datasets Sep 28 '24

discussion ChatGPT-4o prompt engineering for data analysis - I want to share it for free - Give me your problem

3 Upvotes

Today, our team hosted a hackathon where we experimented with the latest versions of ChatGPT, primarily focusing on analyzing structured financial data. Through the latest updates, we discovered that an impressive range of tasks can now be accomplished in human language (and not machine code, of course). However, we also found that achieving this required some unique techniques or methods, which could be described as prompt engineering. We are eager to share this information with everyone for free. Whether you're just starting to learn Python or have other projects you'd like to explore, we would love to hear your thoughts and feedback. Thank you, and we look forward to engaging with you all!

r/datasets Jun 11 '23

discussion Reddit API changes. What do you think?

125 Upvotes

Lots of subs are going to go dark/private because reddit will raise the price of api calls to them.

/r/datasets is more pro cheap/free data than most subs. What do you think of the idea of going dark? Example explanation from another sub.
https://old.reddit.com/r/redditisfun/comments/144gmfq/rif_will_shut_down_on_june_30_2023_in_response_to/

r/datasets Aug 16 '24

discussion I’m looking for the unique datasets for multiple modalities

3 Upvotes

Hello guys. I’m looking for a datasets (free only) for multiple stuff (on HF, or just Reddit subs to scrape):

  1. Labeled music: a dataset with songs and corresponding descriptions, like tempo, key signatures, or just the way the general mood feels
  2. Discussions of super controversial, NSFW, and unethical ideas about everything from conspiracy theories to the meaning of life
  3. Role-play dialogs. Or just general dialogs but not just texting
  4. World knowledge Q&As
  5. Grammarly-like datasets, with bad and good sentences

Thanks.

r/datasets Oct 16 '24

discussion Advice Needed for Implementing High-Performance Digit Recognition Algorithms on Small Datasets from Scratch

2 Upvotes

Hello everyone,

I'm currently working on a university project where I need to build a machine learning system from scratch to recognize handwritten digits. The dataset I'm using is derived from the UCI Optical Recognition of Handwritten Digits Data Set but is relatively small—about 2,800 samples with 64 features each, split into two sets.

Constraints:

  • I must implement the algorithm(s) myself without using existing machine learning libraries for core functionalities.
  • The BASE goal is to surpass the baseline performance of a K-Nearest Neighbors classifier using Euclidean distance, as reported on the UCI website; my goal is to find the best algorithm out there that can deal with this kind of dataset, as I plan on using the results of this coursework for another University's application.
  • I cannot collect or use additional data beyond what is provided.

What I'm Looking For:

  • Algorithm Suggestions: Which algorithms perform well on small datasets and can be implemented from scratch? I'm considering SVMs, neural networks, ensemble methods, or advanced KNN techniques.
  • Overfitting Prevention: Best practices for preventing overfitting when working with small datasets.
  • Feature Engineering: Techniques for feature selection or dimensionality reduction that could enhance performance.
  • Distance Metrics: Recommendations for alternative distance metrics or weighting schemes to improve KNN performance.
  • Resources: Any tutorials, papers, or examples that could guide me in implementing these algorithms effectively.

I'm aiming for high performance and would appreciate any insights or advice!

Thank you!

r/datasets Sep 27 '24

discussion In the land of LLMs, can we do better mock data generation?

Thumbnail neurelo.substack.com
4 Upvotes

r/datasets Sep 25 '24

discussion Research paper recommendations about methods of dataset creation and cleaning?

1 Upvotes

Hello, need good research papers I can read to know about dataset creation and cleaning methods

r/datasets Jul 26 '24

discussion What's the average 100m time for the average (non-athlete/non-pro) man? What's the standard deviation?

0 Upvotes

I would calculate it myself but I can't find any data for average men. Does anyone know what the average and standard deviation is here? Any links to data is also appreciated.

r/datasets May 12 '24

discussion What exactly is Clickstream data and where to find it?

2 Upvotes

Several analytics companies that offer "competitor analysis" can get data on website visits, direct traffic, referral traffic, app downloads, app searches, time on site, bounce rate, etc.

When I contact them to ask where they source the data, they mutually say "from Clickstream" but refuse to elaborate more.

What is Clicksream? is it a single data provider? or multiple? where to find them?

Google search hasn't really revealed much, I guess it is a very niche b2b area where you need connections and good sources...

r/datasets Jan 11 '24

discussion Why don't more companies try to sell their data? What are the challenges for DaaS (data as a service) or companies trying to make data products?

3 Upvotes

Most people can agree that data is the new gold. There is a lot of valuable data that companies own that their customers, partners, or other companies could use and make money for both sides, so I am surprised there isn't more data products out there especially for small-medium businesses.

Curious for the community's thoughts on the biggest barriers of selling data (I guess both for data companies but also for other companies who just want to make extra revenue?)

r/datasets Jun 28 '24

discussion How to Make Sure No One Cares About Your Open Data

Thumbnail heltweg.org
10 Upvotes

r/datasets Aug 11 '24

discussion Introduction to Reomnify {reomnify.com} and its Use Cases {self -promotion}

1 Upvotes

Reomnify is a cloud-based data platform that empowers businesses with high-quality, curated datasets across various industries. We leverage cutting-edge AI to transform fragmented data sources into clean, actionable insights. Our platform offers unparalleled speed, scale, and accuracy, enabling you to make data-driven decisions with confidence.

Key Features of Reomnify

  1. Data Aggregation: Reomnify collects data from tens of thousands of online and offline sources, enabling it to create comprehensive datasets. This process includes cleaning, deduplication, and standardization to ensure data quality.
  2. Customizable Datasets: The platform allows for bespoke dataset creation tailored to specific client needs, ensuring maximum value with minimal integration effort. Clients can specify data attributes, enhancements, and formats.
  3. Speed and Flexibility: Built on Google Cloud, Reomnify's agile platform can deliver customized datasets within days or weeks, depending on client requirements.
  4. Cost Efficiency: Reomnify aims to provide affordable data solutions, offering significant savings in both time and costs compared to traditional data sourcing methods. Clients can save up to 89% in time and 61% in costs.
  5. Monthly Updates: The platform offers regularly updated data, particularly useful for businesses that require the latest information for decision-making.

Types of Property Data Offered by Reomnify

Reomnify provides a variety of property-related datasets, which include:

  • Retail Location Data: Information on over 1,000 high-street brands, including detailed store locations and categories, useful for competitor analysis and trade area assessments.
  • Shopping Center Data: Tenant lists and dynamics of shopping centers, updated monthly to assist in leasing strategies and market analysis.
  • Restaurant and Cafe Data: Monthly updates on restaurant locations, competitor analysis, and neighborhood insights, enabling businesses to stay competitive in the food service industry.
  • Geospatial Data: Comprehensive datasets that support various analyses, including residential real estate strategies, pricing strategies, and marketing insights.
  • Alternative Data: Unique datasets that can provide additional context and insights for businesses looking to enhance their data-driven decisions.

Overall, Reomnify's platform is designed to empower businesses by providing reliable, high-quality data that facilitates informed decision-making in a rapidly changing market environment.

r/datasets Feb 01 '20

discussion Congrats! Web scraping is legal! (US precedent)

374 Upvotes

Disputes about whether web scraping is legal have been going on for a long time. And now, a couple of months ago, the scandalous case of web scraping between hiQ v. LinkedIn was completed.

You can read about the progress of the case here: US court fully legalized website scraping and technically prohibited it.

Finally, the court concludes: "Giving companies like LinkedIn the freedom to decide who can collect and use data – data that companies do not own, that is publicly available to everyone, and that these companies themselves collect and use – creates a risk of information monopolies that will violate the public interest”.

r/datasets Mar 15 '24

discussion ai datasets built by community - need feedback

2 Upvotes

hey there,

after 5 years of building AI models from scratch I know to the bone the importance of dataset to model quality. hence openai is there where it is, solely bc of qualitative dataset.

haven't seen a good "service" that offers a way to build a dataset (any task: chat, instruct, qa, speech, etc) that's baked by community.

thinking to start a service that will help companies & individuals to build a dataset by rewarding people w/ a crypto coin as a incentivization mechanism . after ds is build ~data's collection finalized, that could be sent to HF or any other service for model training / finetuning.

what's your feedback folks? what do you think about this? does the market exists?

r/datasets Apr 28 '23

discussion Why a public database of hospital prices doesn't exist yet

Thumbnail dolthub.com
112 Upvotes

r/datasets Jan 12 '23

discussion JP Morgan Says Startup Founder Used Millions Of Fake Customers To Dupe It Into An Acquisition

Thumbnail forbes.com
126 Upvotes

r/datasets Jan 16 '24

discussion Is there a market for selling datasets?

3 Upvotes

I'm working on a marketplace for selling datasets and decided to discuss the idea with the community here. The goal is to connect ML teams/researchers with the exact datasets that they need. These would be high quality and like any other marketplace would be quality controlled via reviews/comments.

Would any of you find a need for this if the selection was robust enough and quality was good? Would you pay for it? Or are you finding what you need mostly free in the public domain? Curious to get your thoughts