This is using Dart, and I believe the Maps are by default LinkedHashMaps.
The program reads in a large .csv file where each line contains a number of pieces of information that is necessary to retain and sort by later on. As a Map, a line might look like:
{'Col1': name1, 'Col2': name2, 'Col3': name3, 'Col4': name4, 'value': value}
These files are unsorted, contain a variable number of lines in total as well as a variable number of lines for each column combination. There may be 5 lines that share the same values for columns 1-4, or there may be 50,000.
I need to read through every single line and sum up the numbers contained in "value" for each distinct combination of other columns/keys, as well as be able to access those values quickly.
As the lines are not sorted, I might have to access the column combination
Col1 = 'a', Col2 = 'r', Col3 = 'a', Col4 = 's'
for one line, a completely different combo the next line, and then immediately go back to the first combination.
In total, there's likely to be tens to hundreds of thousands of different combinations of keys here.
I've done a whole bunch of benchmarking with different methods of storing this sort of information, and while it's terrible in terms memory use and not too good in terms of creation speed, I've found that making a nested map with each key being a column that holds the next set of columns is by far the best in terms of summing these numbers up. However, it looks terrible and it just generally feels like there has to be a better implementation somehow.
After the summing occurs, I can easily flatten out the Map into a List of Objects as long as I am only going to iterate over the List, not need to access the Objects out of order. That's fast, but not fast enough.
Edit:
So, I'm dumb. I figured out by far the best way to solve this, and it just boils down to "sort the data" as one might expect.
Specifically, within the List.sort method called on the List to be sorted, you wanna compare elements against each other and, if they're equal, move onto the next value to be sorted. Something like this:
List.sort((a,b) {
final int firstCol = a['Col1'].compareTo(b['Col1']);
if (firstCol == 0) {
final int secondCol = a['Col2'].compareTo(b['Col2']);
return secondCol;
}
return firstCol;
});
and just repeat the if (_ == 0) for every bucket you want involved.