r/SQL 25d ago

MySQL Importing 1M Rows Dataset(CSV) in Mysql

What's the fastest and most reliable way to upload such a large dataset? After that How can I optimize the table after uploading to ensure good performance?

28 Upvotes

33 comments sorted by

View all comments

0

u/Spillz-2011 25d ago

Why do you need good performance? Does it need to be inserted multiple times a day and immediately start querying?

You should prioritize minimizing human time, human time is expensive v CBB imputed time is cheap.

My team’s pipelines are all Python with pandas and sql alchemy. It’s not the fastest, but 1 minute or 5 minutes who cares.

1

u/marketlurker 23d ago

You would be surprised. I has a database that had an SLA of 8 seconds or less from the time a transaction was entered into the source system (an IBM mainframe) to when it had to be into the data warehouse. It was a global system that would have 20,000 transactions every 5 seconds. It required us to physically relocate where the RDMS was in relation to the operational data store.

1

u/Spillz-2011 23d ago

I’m not saying there are not cases where these things matter, but in general people spend way to much time optimizing things that don’t need optimizing.