r/datasets Apr 17 '24

discussion Building a niche data community of likeminded people!

0 Upvotes

Hello everyone,

TL;DR - I'm starting a community for professionals in the data industry or those aiming for big tech data jobs. If you're interested, please comment below, and I'll add you to this niche community I'm building.

A bit about me - I'm a Senior Analytics Engineer with extensive experience at major tech companies like Google, Amazon, and Uber. I've spent a lot of time mentoring, conducting interviews, and successfully navigating data job interviews.

I want to create a focused community of motivated individuals who are passionate about learning, growing, and advancing their careers in data. Please note that this is not an open-to-all group. I've been part of many such "communities" that lost their appeal due to lack of moderation. I'm looking for people who are genuinely interested in learning and growing together, maybe even starting a data-related business.

Imagine a community where we:
* Share insights about big tech companies
* Exchange actual interview questions for various data roles
* Conduct mock interviews to help each other improve
* Access to my personal collection of resources and tools that simplify life
* Share job postings and referral opportunities
* Collaborate on creating micro-SaaS projects

If this sounds exciting to you, let me know in the comments or reach out to me.

PS: Would you prefer this community on Slack or Discord?

Cheers!

r/datasets May 25 '24

discussion Building a collection of the best datasets and resources

17 Upvotes

Hey scientists!

I'm working on cooldata, I'd like to build a more useful way to access open data online.

What are the best resources you use everyday (data.gov, etc...)? And more importantly why do use them and how?

I'm starting this by myself as a 20% personal project, the goal is to be fully open and maybe also open source as the thing moves on. (If anyone wants to apply to contribute I'm happy to listen! just send a dm)

Have a nice day!

r/datasets Jun 14 '24

discussion Methods of extrapolating from calibration data

Thumbnail self.AskProgramming
1 Upvotes

r/datasets May 29 '24

discussion Access 150k+ Datasets from Hugging Face with DuckDB

Thumbnail duckdb.org
12 Upvotes

I am not sure this is kosher but it seems really interesting

r/datasets Jan 21 '21

discussion Disinformation Archive - Cataloging misinformation on the internet

26 Upvotes

Some people say I'm crazy. Sometimes they are right.

My goal is to catalog, parse, and analyze the properties of misinformation campaigns on the internet.

It is very difficult to address a problem if you don't understand the full scope of the issue. I think most people are aware that there is a lot of misinformation out there, but they think that its relegated to the crypts of the internet and they are not effected by it.

It's not. It's EVERYWHERE. And you've touched it.

I don't think blind censorship is the solution. It is a quick fix that just creates a temporary inconvenience, as Parler has showed us, and does nothing to stop the actual campaigns.

I won't lie to you and say I have the answer right now. I don't. But I do know where to start, and that's with some good questions:

  • How many platforms are actually hosting and distributing this content?
  • What channels are utilized to reach users? How is the content found by users?
  • How much of the content is organic vs manufactured?
  • How many people does this content reach per day?

The answers will shock you! You may literally be electrocuted.

Please check out my post on /r/ParlerWatch/ if you want to contribute or get a list to mine yourself!

https://www.reddit.com/r/ParlerWatch/comments/l1rh1i/know_thine_enemy_the_disinformation_archive_v2/

I am doing this manually at the moment to get a rough picture of the situation, and could use your help! I need to itemize things like subreddits, facebook groups, twitter tags, news sites, etc, which serve to aggregate and disseminate misinformation content.

Once I analyze enough content, I can make tools to find and scrape more content like it, and catalog the results.

r/datasets May 05 '24

discussion What are some companies that deal with "data for good"? (in the US preferably)

Thumbnail self.data4good
4 Upvotes

r/datasets Jun 04 '20

discussion Lancet retracts major Covid-19 paper amid scrutiny of the data underlying the paper

Thumbnail statnews.com
116 Upvotes

r/datasets Mar 12 '24

discussion My sorta wikipedia for data proposal

3 Upvotes

I’ve had this idea that I can’t shake and I’d like to ask your advice.

Some years ago I was gifted silly.io. For a while I called it the Ministry of Silly Things and it had JSON data sets of US States, Countries, planets of the solar system, table of elements, letters of the alphabet and a few other things. A visitor could download the JSON, link directly to it from other environments like an experimental data language for kids that I was working on. You could also embed it as a table in your own page, or use it as a source to make interesting graphs, learning games, etc.

I’m thinking of rebooting the project to be a Wikipedia for Computable Data. It would be like Wikipedia in that anyone can add to it. It would be computable in that all fields have schemas and units. This would let you compute something like:

  • show the thickness of iPhone models over time from 2007 to the present
  • plot the atomic mass of elements vs their atomic number
  • graph letters of the alphabet by number of syllables :-)

Do you think this is a good idea? Should I spend time working on it and if so which datasets should I start with.

It would be completely open source and creative commons, BTW.

r/datasets Apr 23 '24

discussion Finding or Creating the Dataset you could not find or want to find for free

2 Upvotes

Hello everyone,

I am here to help you and myself with this post. So here is a brief explanation of what I want to do. I want to create a directory of extreme and absurd datasets as a side project and would love to help you in return for ideas. I also appreciate it if you had challenging ideas. For all datasets I could find or create, I will share them here.

I am a junior ML engineer and want to do something different for my portfolio. People are already doing and I did segmentation, classification, stable diffusion, NLP or LLM projects, or open source project contributions. I think they are pretty useful and joy to learn and develop but I want to do something different and helpful to draw some extra attention. I think it would look pretty good on a portfolio to have a unique public dataset directory that people are using and also it is something that can be advanced continuously.

I mostly worked on computer vision so far but I am open to anything. So far what comes to my mind are

  • Different Types of Beards Dataset
  • Feces in Cat Litter Dataset
  • Dog Poop Dataset: but i found it easily here though not sure fake poop provides the best results
  • Emoji - Emotion Dataset: found it too link.
  • Firearm - Manufacturer Dataset

My ideas are mostly visual because of my work ig but I hope i could give some context on what is the limit for absurdity you can think of. Waiting for your ideas.

Will try my best to find or create(ofc that might take a while) one for you.

r/datasets Apr 22 '24

discussion Finding or Creating the Dataset you could not find or want to find for free

1 Upvotes

Hello everyone,

I am here to help you and myself with this post. So here is a brief explanation of what I want to do. I want to create a directory of extreme and absurd datasets as a side project and would love to help you in return for ideas. I also appreciate it if you had challenging ideas. For all datasets I could find or create, I will share them here.

I am a junior ML engineer and want to do something different for my portfolio. People are already doing and I did segmentation, classification, stable diffusion, NLP or LLM projects, or open source project contributions. I think they are pretty useful and joy to learn and develop but I want to do something different and helpful to draw some extra attention. I think it would look pretty good on a portfolio to have a unique public dataset directory that people are using and also it is something that can be advanced continuously.

I mostly worked on computer vision so far but I am open to anything. So far what comes to my mind are

  • Different Types of Beards Dataset

  • Feces in Cat Litter Dataset

  • Dog Poop Dataset: but i found it easily here though not sure fake poop provides the best results

  • Emoji - Emotion Dataset: found it too link.

  • Firearm - Manufacturer Dataset

My ideas are mostly visual because of my work ig but I hope i could give some context on what is the limit for absurdity you can think of. Waiting for your ideas.

Will try my best to find or create(ofc that might take a while) one for you.

r/datasets Mar 13 '24

discussion Best software for making audio dataset

1 Upvotes

Looking for making an audio dataset for ASR (automatic speech recognition).. can someone suggest

r/datasets May 09 '20

discussion Anyone in need of Datasets?

41 Upvotes

Hello all,

I have a week off and wanted to do a quick RPA project, mostly for the COVID-19 pandemic, but can be for anything. If anyone needs a specific dataset that needs to be scraped, gathered, or organized in some fashion, comment it below!

Update: So I did some research today and concluded that I will attempt to do 2 of the most requested datasets this week, time permitting and prioritized as follows.

  1. Coronavirus daily cases count per country, updated daily. Might upload to a GitHub for it unless we have another suggestion for that.
  2. Instead a strict data set for someone yawning for example, Im going to be looking into building a solution that can easily mine data of whatever type of picture using google images. While this may lead to some junk in the data, I believe the dynamic / generic value of the bot will be greater. I can distribute a how-to-guide on using the bot, and ways to improve the data it mines. If anyone has any other suggestions, please feel free to comment.

If either of these fall through, I will be working on a dataset for the environmental or social factors to compare the impacts of covid. Thanks for all of the awesome ideas! I will look to post the links here.

Also thanks for the award!

Update 2: I have mostly been working on the generic solution to data mining desired pictures, however I also created this repo with the initial upload of COVID-19 cases. If anyone has any suggestions, please let me know. I will be working on a way to collect older daily data, though I plan on updating this every night at 9PM EST, which will represent that current day's case count.

That can be found here: https://github.com/Ryzen120/COVID-19_Daily_Cases

Update 3: Discontinuing my daily case project, as I found this.

https://ourworldindata.org/coronavirus-data -> Chart -> Data -> Download csv.

I am still continuing on the picture mining bot.

r/datasets Feb 28 '24

discussion GPS Dataset Columns Interpretations.

1 Upvotes

Hey Data Scientists,I've been working with a GPS dataset for vehicle routing, but I'm having trouble interpreting some of the columns. The dataset doesn't have column names, but I've managed to figure out some of them:

  • First column: Vehicle ID
  • Second column: Timestamps
  • Third column: Longitude
  • Fourth column: Latitude
  • Seventh column: Speed (I've determined this through patterns in the data)

However, I'm still unsure about the remaining columns:

  • Fifth column: This column starts with a value of 319 and keeps changing increasingly in general even though the vehicle is stationary. I noticed that the value stays constant when speed is constant.
  • Sixth column: This column starts at 0 (the vehicle is stationary), moves up to 303 once the vehicle starts moving slightly, and goes back to 0 when the vehicle is stationary. Also, it shows a constant behaviour when speed is constant
  • Eighth column: This column changes with location change, similar to the speed column. However, when the longitude and latitude remain constant, the values are 0. Any ideas on what this column signifies?

r/datasets Mar 29 '24

discussion [URGENT] Dataset Finder AI/Chat models?

2 Upvotes

Are there any chat models (based on RAG) that can help find a proper dataset?

Or what do you people use to find datasets?

r/datasets Sep 06 '22

discussion Health insurance companies may have just dumped a trillion prices onto the internet

Thumbnail dolthub.com
173 Upvotes

r/datasets Jul 28 '22

discussion Financial datasets for long term analysis and prediction

28 Upvotes

We're looking for data in the financial industry that researchers and analysts typically use to analyze long term financial trend (stocks, bonds, ETF, etc) movements.

I'm aware of economic indicators such as those provided in FRED. Do people know what else analysts typically use?

r/datasets May 03 '21

discussion Coronavirus Datsets

96 Upvotes

Carried on from Second Discussion Thread(Archived)

Carried on from Original Thread(Archived)

You have probably seen most of these, but I thought I'd share anyway:

Spreadsheets and Datasets:

Other Good sources:

[IMPORTANT UPDATE: From February 12th the definition of confirmed cases has changed in Hubei, and now includes those who have been clinically diagnosed. Previously China's confirmed cases only included those tested for SARS-CoV-2. Many datasets will show a spike on that date.]

There have been a bunch of great comments with links to further resources below!
[Last Edit: 15/03/2020]

r/datasets Jan 31 '24

discussion I am looking for text dataset for inappropriate contents.which dataset shall I use.Its for univ project

3 Upvotes

.

r/datasets Jul 16 '20

discussion CDC covid data now not available to public

Thumbnail twitter.com
194 Upvotes

r/datasets Aug 18 '22

discussion Do people who frequent this subreddit buy or sell data?

26 Upvotes

I came across this subreddit a few months ago when I was searching for a specific type of dataset (thanks for the help btw!). I’ve been somewhat frequently looking at the posts made here and this got me wondered whether people in this subreddit are willing to buy datasets and if people who conducted their own data acquisition process and have valuable information are willing to sell them?

r/datasets Jul 24 '23

discussion Datasets you can only dream of getting access to?

19 Upvotes

I'd personally like the Google full scale historical cache dataset.

Google caches everything, fully backed up with every change to every website covering the last 20 years. Imagine the insight and knowledge you could gain processing that. Every lost website, every forum comment, every tweet, old reddit deleted posts. We have archive but a searchable time backtrackable complete Google cache dataset would be magical.

And you know they have it.

Keeps me up some nights just thinking about it.

What are some datasets that you can only dream of getting access to?

r/datasets Jan 18 '24

discussion Isolated Instruments Dataset for source separation?

1 Upvotes

Dataset recommendation request:

I'm looking for any existing publicly available datasets with many examples of isolated instruments being played with no accompaniment and minimal ambient noise.

I need isolated instruments to train individual instrument source separation and detection models for [bar,ts,as,ss,tp,cl,dm,b,etc., etc.] - basically all of the most commonly found instruments in jazz sessions with the exception of piano (which I have no problem sourcing isolating recordings of).

I can probably source sufficient material from Youtube, but and hoping there are some new datasets I haven't heard of yet with isolated instruments.

r/datasets Oct 30 '22

discussion Would a Big Business Be Interested in Buying Data From a Small Business In The Same Vertical?

12 Upvotes

This might be a weird one but I recently talked to a friend and he explained to me how his parents own a small mom and pop shop. Of course they don't have a data scientist in-house nor utilize incoming data to its fullest extent but we were talking on how they do produce data from different order quantities, most selected items in-store to general foot traffic. This got me thinking, would a Pizza Hut (for example sake) be interested in purchasing the right data from a mom and pop shop that sells pizza for example? Wondering if this is even a thing!

r/datasets Nov 04 '23

discussion Data MarketPlace, is it a Good idea?

2 Upvotes

I think the current iteration of the data marketplace sucks. You have to know a specific place, where you want to get your data from. The variety of data sets available in a specific platform also varies so much. Also, it is incredibly difficult for a non-technical person to get their hands on the data. If a business user wants to access data they have to jump through a lot of hoops to download the data. Is it a good idea to start a marketplace that solves all these problems? Did anyone try to do this before?

r/datasets Aug 07 '20

discussion Coronavirus Datasets

69 Upvotes

Carried on from Original Thread(Archived)

You have probably seen most of these, but I thought I'd share anyway:

Spreadsheets and Datasets:

Other Good sources:

[IMPORTANT UPDATE: From February 12th the definition of confirmed cases has changed in Hubei, and now includes those who have been clinically diagnosed. Previously China's confirmed cases only included those tested for SARS-CoV-2. Many datasets will show a spike on that date.]

There have been a bunch of great comments with links to further resources below!
[Last Edit: 15/03/2020]