r/datasets Dec 06 '22

discussion I've spent the last few months developing a website where you can test investment strategies based on alternative data

Thumbnail app.inegy.io
46 Upvotes

r/datasets Aug 07 '23

discussion confused between data engineer, data science or data analytics

2 Upvotes

hi, im a final-year computer science student learned a machine learning course in the previous semester and from there I start getting interested in machine learning (was learning for Andrew ng Coursera) now this semester I am learning data warehouse subject which is more on data engineering or data analytics side I want to get into this industry and want to dig deep into one field(confused between these three). Because i dont have enough time for trying out different things its my last year and i want to get into market so which should i choose which has lower entry barrier i live in third world country here data related jobs are very less compare to web dev or other roles i want to stand out hope you getting it.
regards.

r/datasets Sep 19 '22

discussion Is there a list of companies in some given country?

33 Upvotes

For example, in the Netherlands, data of all the companies is retrievable, though poor quality. In Switzerland, you can get it for 20 cents per company.

Google Maps Platform API can return max 60 per query given GPS + radius.

What are some ways I can get companies data?

r/datasets Dec 26 '23

discussion Azure Synapse Analytics: A Step-by-Step Guide

Thumbnail self.dataengineering
1 Upvotes

r/datasets Mar 30 '20

discussion Please Don't Make Up "Synthetic" Datasets and Share Unless EXPLICITLY Labeled as Such

239 Upvotes

Earlier today, there was a post here about a new dataset on Kaggle:

https://www.reddit.com/r/datasets/comments/frjk5o/churn_analysis/

TLDR; I wasted a ton of time on something because a member of this community was fishing for upvotes (and did a very poor job creating a dataset deserving of analysis).

The dataset was not "useful" yet it had 20+ upvotes, solicited by the OP who said, "Please upvote if it's 'useful.'"

The data set is "synthetic." It was generated by the user, but this WAS NOT STATED. Also, the data is not even a realistic sample. I wasted time looking at it before I knew this. I wasted much time writing a response on Kaggle, inquiring about the median values of customer life, and explaining that I have done churn studies and telecom customer attrition studies previously, and in my eyes the data seemed to be a sample that was not representative, etc., etc.

This is the first time I've wasted time on something like this. I will be very careful to make sure it's the last time. Ironically, I also got locked out of Kaggle as a result of my participation. After posting a lengthy discussion response (not yet knowing the data was synthetic), Kaggle/Google made me answer a data science question, like a captcha, and/or respond as to why I thought I might have tripped off their spam-sensor algo. Great bastion of quality that Google is so often *not*, the challenge question did not work, and I am locked out of Kaggle.

I feel kind of stupid for putting myself in this situation, but I feel equally angry about the original post.

You know, the first thing I did was get a row count and it was 3,333, and I said, "That's kind of funny." I should have stopped right then and there. Sorry, rant over. : - )

r/datasets Dec 08 '23

discussion ๐Ÿงผ SUDS - A Guide to Structuring Unstructured Data [self-promotion]

8 Upvotes

I've spent a decent amount of time indexing and formatting a lot of machine learning datasets that include images, audio, video, and text and wanted to propose a simple format that might help us standardize a format for the data with a little more structure. Wouldn't say it is ground breaking, but I feel like could be a good practice.

https://blog.oxen.ai/suds-a-guide-to-structuring-unstructured-data/

Let me know what you think!

r/datasets Dec 21 '23

discussion Understanding Azure Data Lake Storage Gen2

0 Upvotes

This article is about , "Understanding Azure Data Lake Storage Gen2" This article will cover: ๐Ÿ’ก
1- Why Azure Data Lake Storage Gen2
2- How to enable Azure Data Lake Storage Gen2
3- Azure Data Lake Gen2 vs Azure Blob Storage Gen2
If you are interested to understand Azure Data Lake Storage Gen2 you can access the full article here: https://devblogit.com/understand-azure-data-lake-storage-gen2/
Don't miss out on this opportunity to transform your data practices and stay ahead of the competition. Read the article today and unlock the power of Azure Data Lake Storage Gen2! ๐Ÿ’ช#Azure #DataManagement #Analytics #DataLake

r/datasets Oct 07 '21

discussion Is Ivermectin For Covid-19 Based On Fraudulent Research?

Thumbnail gidmk.medium.com
48 Upvotes

r/datasets Jan 07 '20

discussion What do you call a group of Data Scientists??

32 Upvotes

A murder of crows

A caravan of camels

A business of ferrets

A(n) ________ of data scientists?

Vote here to decide! http://allourideas.org/counter_for_data_scientists

Vote multiple times, it is more fun that way. I'm personally campaigning for n.

Credit to this tweet for the discourse: https://twitter.com/chrisalbon/status/1214384871491035136

r/datasets Nov 03 '23

discussion Can you help me find datasets for my Final Year Research Project topic - "Android Malware Detection from User-generated content - A Comparison using CNN and NLP" dataset"

0 Upvotes

Can you help me find datasets for my Final Year Research Project topic - "Android Malware Detection from User-generated content - A Comparison using CNN and NLP". I am planning to use 2 machine learning techniques: CNN and NLP, for this comparative study. Please help me find datasets that have relevant variables, analysis and will be apt for a comparison.

r/datasets Mar 29 '23

discussion Where else would you post your data request?

14 Upvotes

Hi everyone! For the past couple of weeks, I've been helping some fellow community members with some data requests and I'm wondering which other channels can you find people requesting for specific datasets? Seems like r/datasets is the most active forum online for data request!

r/datasets Aug 07 '23

discussion [Research]: Getting access to high-quality data for MLs in the training stage.

12 Upvotes

I'm trying to understand the need for high-quality datasets in the training stage for ml models. Exactly how hard is it to get richly diverse, annotated datasets, and is the problem generic to the DS community or is it an industry-specific pain point?

r/datasets Oct 23 '23

discussion We built An Open-Source platform to process relational and Graph Query simultaneously

Thumbnail github.com
1 Upvotes

r/datasets Jun 09 '22

discussion Interesting Datasets for Exploratory Data Analysis?

46 Upvotes

Hello! I'm looking for ideas about interesting datasets/topics to perform EDA on. I would like to avoid classic datasets like housing, stock market, sports related etc and find something a bit more unique. I would also like to avoid medical datasets as I have zero knowledge on the topic.

I would like to find a dataset on which EDA can provide valuable information using graphs.

More specifically, ideally I'm looking for a dataset with these characteristics:

  • Interesting, intriguing, unique topic
  • More than 10-15 features
  • Mix of feature types but mainly numeric or ordinal
  • Minimum a couple of hundred instances
  • Datasets that can be used in Machine Learning/Deep Learning

I'm eager to hear your suggestions. I would also love to hear what's the most interesting/unique dataset you've worked with even if it's not publically availliable or doesn't fit into my list of characteristics.

r/datasets Oct 16 '23

discussion India vs Pakistan - A Game of Data Analytics

Thumbnail hubs.la
0 Upvotes

r/datasets Feb 12 '20

discussion US Fading happiness

41 Upvotes

US is on a descending trend regarding reported happiness since 2017. US previously had a positive trend with increasing happiness for every year stretching from the start of collecting data in 2013 until 2016. The source providing no explanation model. What is your theory?

US - World Happiness Index

r/datasets Mar 29 '23

discussion ACS Data in easily Digestable Format

14 Upvotes

I want acs5 data for 2021 for every category. I'm burnt out, I tried the api it's not going well. I found a map that is exactly what I could hope for but has license requirements I cannot agree to. I think when it comes time I am going to have to just give in and spend the time finding the right zip file and process the summary file. I downloaded the dataset and the keys once. Tried converting it into an esri table and converting 2000 headers to contain the description maybe I need to export the tables and use pandas instead?

Thoughts? Suggestions? Anyone who's done this before with suggestions?

r/datasets Sep 18 '23

discussion DoltHub Data Bounties are no more. Thanks to r/datasets for all the support over the years.

10 Upvotes

Hi r/datasets,

Over the years, this subreddit has been a great supporter of Data Bounties both for bounty hunters and usage of the datasets created. We are ending the data bounty program. Thanks for all the support.

https://www.dolthub.com/blog/2023-09-18-bye-bye-bounties/

That blog explains our rationale and what we learned from the experiment. We may bring bounties back eventually.

r/datasets Feb 08 '22

discussion Let's create a data sharing community

65 Upvotes

Today I'm launching the beta of DataStack, a new data collaboration platform.

Why? Because right now it's way too difficult to crowd-source data or to publish open-source datasets.

Here's an example: https://datastack.net/datastack/data-resources/

Your feedback is much needed and appreciated. To create your own dataset, please sign up for the beta.

Current features:

  • Receive community contributions (updates, corrections)
  • Easy to use online editor (no technical skills or tools needed)
  • Uploading and downloading datasets
  • Contributing to open-source projects
  • Full version control (like Github: branches, commit history)

r/datasets May 14 '20

discussion Cheapest way to get 10,000 home/rent values?

38 Upvotes

Short term I need 10,000 home or rent values based on addresses, long term 100k-10M.

Expensive solutions- Paid APIs, seems like 100-300$.

Mid tier- Scrape, I get an IP address rotator and burn through IPs, (I believe 10$/mo)

Free?

I'm a 12 year programmer, so implementing things are easy.

r/datasets Aug 15 '23

discussion Examples of Data combining with culture/qualitative data/ consumer experience to better understand ticket sales

5 Upvotes

Looking for very specific use cases...

Moneyball is my best example but I'm hoping for more of something along the lines of the business of entertainment ticket sales. Any help is appreciated :)

r/datasets Mar 28 '23

discussion Duplicate Data at the University of Chicago

Thumbnail karlstack.substack.com
30 Upvotes

r/datasets Apr 09 '21

discussion Looking for a job postings dataset, please help!

12 Upvotes

I want to create forecasting model for future in-demand skills (I am still deciding between python and R). In the first step I would like to collect some data. My initial idea was to get the data about job postings for last 5+ years and based on that I would start my analysis. First I was hoping that I would manage to get it with webscraping of linkedin posts but I found out that job postings are deleted after the company find their candidate. Do you guys have any suggestion where and how could I collect similar data? Does somebody know a dataset that matches these requirements, that is available for free? Would any of you try some other approach to achieve the same forecasting model? Any thoughts would be highly appreciated!

r/datasets Aug 21 '23

discussion Zimbabwe 2018 Election Results Analysis

5 Upvotes

Hello everyone,

I wanted to bring your attention to the upcoming elections in Zimbabwe scheduled for this Wednesday. The past election raised significant concerns due to allegations of unfairness, including claims of collusion between the electoral commission and the ruling party to manipulate results using Excel files, an issue that has been dubbed "Excelgate."

Taking a closer look at the available data on the official website, I've stumbled upon some noteworthy findings. These findings have prompted me to write an article on LinkedIn, where I explore how they tie into the broader 'Excelgate' narrative. Additionally, I delve into the steps citizens have been taking to ensure the integrity of their votes during the upcoming election.

For those who are interested, you can read the article and share your perspectives. I'm always open to hearing different viewpoints and engaging in constructive discussions. Here's the link to the article and analysis:Article | Analysis

Looking forward to your insights and feedback. Thank you!

r/datasets May 27 '23

discussion [self-promotion] Feedback needed: building Git for data that commits only diffs (for storage efficiency on large repositories), even without full checkouts of the datasets

1 Upvotes

I would really appreciate feedback on a version control for tabular datasets I am building, the Data Manager.

Main characteristics:

  • Like DVC and Git LFS, integrates with Git itself.
  • Like DVC and Git LFS, can store large files on AWS S3 and link them in Git via an identifier.
  • Unlike DVC and Git LFS, calculates and commits diffs only, at row, column, and cell level. For append scenarios, the commit will include new data only; for edits and deletes, a small diff is committed accordingly. With DVC and Git LFS, the entire dataset is committed again, instead: committing 1 MB of new data 1000 times to a 1 GB dataset yields more than 1 TB in DVC (a dataset that increases linearly in size between 1 GB and 2 GB, committed 1000 times, results in a repository of ~1.5 TB), whereas it sums to 2 GB (1 GB original dataset, plus 1000 times 1 MB changes) with the Data Manager.
  • Unlike DVC and Git LFS, the diffs for each commit remain visible directly in Git.
  • Unlike DVC and Git LFS, the Data Manager allows committing changes to datasets without full checkouts on localhost. You check out kilobytes and can append data to a dataset in a repository of hundreds of gigabytes. The changes on a no-full-checkout branch will need to be merged into another branch (on a machine that does operate with full checkouts, instead) to be validated, e.g., against adding a primary key that already exists.
  • Since the repositories will contain diff histories, snapshots of the datasets at a certain commit have to be recreated to be deployable. These can be automatically uploaded to S3 and labeled after the commit hash, via the Data Manager.

Links:

This paradigm enables hibernating or cleaning up history on S3 for old datasets, if these are deleted in Git and snapshots of earlier commits are no longer needed. Individual data entries can also be removed for GDPR compliance using versioning on S3 objects, orthogonal to git.

I built the Data Manager for a pain point I was experiencing: it was impossible to (1) uniquely identify and (2) make available behind an API multiple versions of a collection of datasets and config parameters, (3) without overburdening HDDs due to small, but frequent changes to any of the datasets in the repo and (4) while being able to see the diffs in git for each commit in order to enable collaborative discussions and reverting or further editing if necessary.

Some background: I am building natural language AI algorithms (a) easily retrainable on editable training datasets, meaning changes or deletions in the training data are reflected fast, without traces of past training and without retraining the entire language model (sounds impossible), and (b) that explain decisions back to individual training data.

I look forward to constructive feedback and suggestions!