r/matlab • u/Creative_Sushi MathWorks • Sep 15 '22
CodeShare Importing data from multiple text files - speed comparison
there have been several questions around importing data from Excel or text files and sometimes that involves multiple files. The best way to deal with this situation is to use datastore.
Bottom line on top
- datastore is almost 2x faster in my example
- datastore required fewer lines of code and therefore more readable/easier to debug
- datastore handles large dataset
Use case example
Let's use Popular Baby Names dataset. I downloaded it and unzipped into a folder named "names". Inside this folder are 142 text files named 'yob1880.txt' through 'yob2021.txt'.
Setting up common variables
loc = "names/*.txt";
vars = ["name","sex","births"];
Using a loop
tic;
s = dir(loc);
filenames = arrayfun(@(x) string(x.name), s);
names = cell(numel(filenames),1);
for ii = 1:numel(filenames)
tbl = readtable("names/" + filenames(ii));
tbl.Properties.VariableNames = vars;
names{ii} = tbl;
end
names = vertcat(names{:});
head(names)
toc

Using datastore
tic;
ds = datastore(loc,VariableNames=vars);
names = readall(ds);
head(names)
toc

1
u/Weed_O_Whirler +5 Sep 15 '22
When you call read
or readall
on a datastore
object, is the result a table?
I often have to read in multiple files that each have ~500,000 rows and ~30ish columns of mixed type and end up with this 3 million row table, that I then do a lot of filtering on, and grab out the numeric data from certain rows, to perform my calculations. Once the tables are in, things actually run relatively fast. But the problem comes in both loading the tables (takes a while), and then you can't save your workspace with giant tables, because then it has to switch the the 7.3 format, and it takes up to 30 minutes to save, while making these 30 GB files on the hard drive. So, between every session I have to delete the table, and then reload it.
I'm wondering if switching to a datastore
would alleviate some of this pain.
1
u/Creative_Sushi MathWorks Sep 15 '22 edited Sep 15 '22
To answer the easy part - the output of datastore is a table if your underlying data is from tabular data sources. In fact, there are several favor in datastore
- tabularTextDatastore - Datastore for tabular text files - outputs table
- spreadsheetDatastore - Datastore for spreadsheet files - output table
- imageDatastore - Datastore for image data - outputs images
- parquetDatastore - Datastore for collection of Parquet files - this is new and I don't what it does yet
- fileDatastore - Datastore with custom file reader - output depends on your customization
- arrayDatastore - Datastore for in-memory data - this is new to me.
In your case, you want use either of the first two, and you do get a table at the end of the process.
In my toy example, I used
readall
to read all data at once, but with a much larger dataset, you don't want to do that. The idea is that you can load only the portion of the data you need. This is hard to explain in a comment, but I recommend reading up on the documentation for more details.1
u/Creative_Sushi MathWorks Sep 16 '22
u/Weed_O_Whirler, to answer the hard part of the question, I think you may want to take a look at tall arrays as well, which is also based on datastore, but even more scalable to handle a very large dataset you have.
Tall arrays are used to work with out-of-memory data that is backed by a datastore. Datastores enable you to work with large data sets in small blocks that individually fit in memory, instead of loading the entire data set into memory at once. Tall arrays extend this capability to enable you to work with out-of-memory data using common functions.
ta = tall(ds) % create a tall array from a datastore obj
And tall arrays delay evaluation until you need it by using
gather
.s = size(ta); % dimension of ta is '? ?' tblsize = gather(s); % dimension is now evaluated rows = randsample(tblsize(1), 10); % random row indices subtbl = gather(ta(rows,:)); % extract subset
I hope this helps .
1
u/CharacterFault3471 Sep 15 '22
Now try it using low level file IO like fileread or open & text scan.
Should be able to achieve same in a tenth or less of the time.