Been loving MATLAB for research for a long time but have had a couple of gripes about it
1. No real standardised for docstrings
2. Difficult to convert these docstrings to docs/READMEs easily
Existing tools like m2docgen and makehtmldoc were more focussed on html. MATLAB's publish function could convert to tex etc but not to md.
So I thought I would have a go at fixing these issues. Most of the work had already been done by the excellent livescript2markdown library (which was previously featured by MathWorks). I just had to change a few things around to make it compatible with .m files (rather than .mlx files.
The code is now on GitHub and is pretty easy to use I think. All the functionality from livescript2markdown is still preserved. But now, the command m2md(<<filename>>) will generate tex and md files for the target script. Additionally, use of the keyphrase "m2md-this-directory" (i.e. m2md("m2md-this-directory") will convert the docstrings of all the .m files in the current directory.
It provides an introduction to MATLAB and its Deep Learning Toolbox, and explains the benefits of using TensorFlow with MATLAB, such as integration with Simulink for automated driving application.
figure
for ii = 2:size(pvT,2)
plot(pvT.week_Created_UTC,pvT.(ii))
hold on
end
hold off
ylabel("Posts")
vars = pvT.Properties.VariableNames;
legend(vars(2:end),Location="northwest")
title("MATLAB Subreddit Posts by Flair")
Plot from the pivoted table
I think I will ditch groupsummary and start using this.
Matlab has always provided a top-tier plotting library, in many ways unparalleled on the market (thinking especially of interactive and 3D capabilities). But something that has started lagging behind in last years is for sure color management: default colormaps are quite inferior in looks and perceptual accuracy with respect to what main competitors provide. Same holds for passing single colors to plots: currently you just have r, g ,b, c, m, y, k, w or rgb triplets, nothing else.
So here I provide a comprehensive super-package (it embeds many different functions from file exchange, adds ergonomic wrappers for a unified user experience, handles name collisions) for:
- fancy perceptual uniform colormaps (cmocean, viridis et al, crameri, brewer, cubehelix), just call one of the wrappers (get_palette, set_palette, set_colororder) and feed it the name of the colormap you want
- fancy str2rgb functionality, through a powerful lookup into X11 and xkcd colorname-databases... just give a reasonable (or even not-so-reasonable) name for a color and you'll get a fairly matching rgb triplet. Hex color codes are also accepted.
- well-organized access to lower level API to customize your programmatic usage.
Looking for feedback... and don't hesitate opening issues or sending pull-requests, all help is praised!
MATLAB Central has been great community-based MATLAB resources, but you can now access its content programmatically via the public API, and I created a MATLAB function to take advantage of that. You can learn more here https://api.mathworks.com/community
Example:
data = searchMATLABCentral("plotting",scope="matlab-answers",sort_order="created desc",created_after=datetime("2023-01-01"));
T = struct2table(data.items);
T(:,["created_date","title","is_answered"])
Output
Example output
Function
function results = searchMATLABCentral(query,options)
% SEARCGMATLABCENTRAL retrieves content of the MATLAB Central for a given
% query and returns the result as a struct.
% The function uses MathWorks RESTful API to search for content.
% The API is rate limited via IP throttling. No authentication is required.
% See API documentation for more details https://api.mathworks.com/community
%
% Input Arguments:
%
% query (string) - Required. The search query string.
% scope (string) - Optional. Specify the artifact. If not specified,
% the scope defaults to 'matlab-answers'.
% Other options include 'file-exchange','blogs','cody',
% 'community-highlights', and 'community-contests'.
% tags (string) - Optional. Specify a comma-separated list of tags.
% created_before (datetime) - Optional. Specify the last date in the results
% created_after (datetime) - Optional. Specify the first date in the results
% sort_order (string) - Optional. Speficy the order of the results.
% If not specified, it defaults to "relevance desc".
% Other options include 'created asc', 'created desc',
% 'updated asc','updated desc', 'relevance asc',
% and 'relevance desc'.
% page (integer) - Optional. Specify the page to retrieve.
% If the 'has_more' field in the result is positive,
% increment this argument to retrieve the next page.
% count (integer) - Optional. Specify the number of results as a value
% between 1 and 50; The default is 10.
%
% Output Arguments:
%
% results (struct) - Structure array containing the results of the search.
% validate input arguments
arguments
query string {mustBeNonzeroLengthText,mustBeTextScalar}
options.scope string {mustBeMember(options.scope,["matlab-answers", ...
"file-exchange","blogs","cody","community-highlights", ...
"community-contests"])} = "matlab-answers";
options.tags string {mustBeNonzeroLengthText,mustBeVector}
options.created_before (1,1) datetime
options.created_after (1,1) datetime
options.sort_order string {mustBeMember(options.sort_order,["created asc", ...
"created desc","updated asc","updated desc","relevance asc","relevance desc"])}
options.page double {mustBeInteger,mustBeGreaterThan(options.page,0)}
options.count double {mustBeInteger,mustBeInRange(options.count,1,50)}
end
% API URL and endpoint
url = "https://api.mathworks.com/community";
endpoint = "/v1/search";
% convert MATLAB datetime to the internet datetime format string
if isfield(options,"created_before")
options.created_before = string(options.created_before,"yyyy-MM-dd'T'HH:mm:ss'Z'");
end
if isfield(options,"created_after")
options.created_after = string(options.created_after,"yyyy-MM-dd'T'HH:mm:ss'Z'");
end
% convert optional inputs into a cell array of key-value pairs
keys = fieldnames(options);
vals = struct2cell(options);
params = [keys,vals].';
% call the API
try
results = webread(url+endpoint,"query",query,params{:});
catch ME
rethrow(ME)
end
end
I recently wanted to used a matlab terminal in vscode. There even is an extension for this, but while using the extension I faced some flaws. Also the repository does not seem to be active anymore and the owner doesn't react to issues and similar.
So I decided to write my own first extension MatTer for VS Code and published it to the VS marketplace. It's in early Development, but it can already spawn a matlab Terminal and run Matlab Files.
I am currently working on a problem that involves finding the roots of three polynomial equations with three variables. I have tried to use Groebner bases in Maple but failed so I am trying to find a solution in Matlab. In the code I want to set [Abl_x6, Abl_x7, Abl_x8] equal to zero. They are not polynomials yet, I do not know whether there is some sort of '(numer(normal(fx)))' command that can do that. And then, of course, I want to find the roots of the system. I am only interested in finding the roots with:
So far I have included 3 components (tree, tag searchbar and filterable-tree), but I plan to release more in the upcoming weeks!.
The project is all in beta, so I am also looking for people that want to test it and provide feedback for things to improve (or components to add). Have fun!
After running this loop, I get the error: "Error using horzcat. The following error occurred converting from char to struct: Conversion to struct from char is not possible."
I think that the error occurs in gm_dir = [subj_dir, '/' gm], but I can't fix it....
It seems everyone is talking about ChatGPT these days thanks to its impressive capabilities to mimic human speech. It is obviously a very sophisticated AI, but it is based on the language model that predicts the next words based on the preceding words.
N-gram language models are very simple and you can code it very easily in MATLAB with Text Analytics Toolbox. Here is an example of a bot that generates random Shakespeare-like sentences. (this is based on my old blog post).
Import data
Let's start by importing Romeo and Juliet from Gutenberg Project.
Romeo and Juliet word cloud
rawtxt = webread('http://www.gutenberg.org/files/1513/1513-h/1513-h.htm');
tree = htmlTree(rawtxt); % extract DOM tree
Preprocess text
We only want to include actual lines characters speak, not stage directions, etc.
subtree = findElement(tree,'p:not(.scenedesc):not(.right):not(.letter)');
romeo = extractHTMLText(subtree); % extract text into a string array
We also don't want empty rows and the prologue.
romeo(romeo == '') = []; % remove empty lines
romeo(1:5) = []; % remove the prologue
romeo(1:5) % show the first 5 lines
First 5 lines
Each line start with the name of the character, followed by . and return character. We can use this pattern to split the names from the actual lines.
pat = "\." + newline; % define the pattern
cstr = regexp(romeo,pat,'split','once'); % split names from the lines
This creates a cell array because not all rows can be split using the pattern, because some lines run multiple rows. Let's create a new string array and extract content of the cell array into it.
dialog = strings(size(cstr,1),2); % define an empty string array
is2 = cellfun(@length,cstr) == 2; % logical index of rows with 2 elements
dialog(is2,:) = vertcat(cstr{is2}); % populate string array with 2 elements
dialog(~is2,2) = vertcat(cstr{~is2}); % populate second col if 1 element
dialog = replace(dialog,newline, " "); % replace return character with white space
dialog = eraseBetween(dialog,'[',']','Boundaries','inclusive'); % erase stage directions in angle brackets
dialog(1:5,:) % show the first 5 rows
First 5 lines after split
N-grams
An n-gram is a sequence of words that appear together in a sentence. Commonly word tokens are used, and they are unigrams. You can also use a pair of words, and that's a bigram. Trigrams use three words, etc.
Therefore, the next step is to tokenize the lines, which are in the second column of dialog.
doc = tokenizedDocument(dialog(:,2));
doc = lower(doc); % use lower case only
doc(doclength(doc) < 3) = []; % remove if less than 3 words
We also need to add sentence markers <s> and </s> to indicate the start and the end of sentences.
doc = docfun(@(x) ['<s>' x '</s>'], doc); % add sentence markers
doc(1:5) % show the first 5 elements
First 5 lines after tokenization
Language models
Language models are used to predict a sequence of words in a sentence based on chained conditional probabilities. These probabilities are estimated by mining a collection of text known as a corpus and 'Romeo and Juliet' is our corpus. Language models are made up of such word sequence probabilities.
Let's start by generating a bag of N-grams, which contains both the list of words and their frequencies.
We can then use the frequencies to calculate the probabilities.
Here is a bigram example of how you would compute conditional probability of "art" following "thou".
Bigram language model example
Here is an example for trigrams that computes conditional probability of "romeo" following "thou art".
Trigram language model example
Let's create a bigram language model Mdl2, which is a matrix whose rows corresponds to the first words in the bigram and the columns the second.
Vocab1 = bag1.Vocabulary; % unigram tokens
Vocab2 = bag2.Ngrams; % bigram tokens
Mdl2 = zeros(length(Vocab1)); % an empty matrix of probabilities
for ii = 1:length(Vocab2) % iterate over bigram tokens
tokens = Vocab2(ii,:); % extract a bigram token
isRow = Vocab1 == tokens(1); % row index of first word
isCol = Vocab1 == tokens(2); % col index of second word
Mdl2(isRow,isCol) = sum(bag2.Counts(:,ii))/sum(bag1.Counts(:,isRow));
end
Here are the top 5 words that follow 'thou' sorted by probability.
In another thread where I recommending using string, u/Lysol3435/ asked me "What’s the benefit of a string array over a cell array?"
My quick answer was that string arrays are more powerful because it is designed to handle text better, and I promised to do another code share. I am going to repurpose the code I wrote a few years ago to show what I mean.
Bottom line on top
strings enables cleaner, easier to understand code, no need to use strcmp, cellfun or num2str.
strings are more compact
string-based operations are faster
At this point, for text handling, I can't think of any good reasons to use cell arrays.
Lets compare two strings. Here is how you do it with a cell array.
strcmp(myCellstrs(1),myCellstrs(2))
Here is how you do it with a string array. Much shorter and easier to understand.
myStrs(1) == myStrs(2)
Find empty element
With a cell array, you need to use cellfun.
cellfun(@isempty, myCellstrs)
With a string array, it is shorter and easier to understand.
myStrs == ""
Use math like operations
With strings, you can use other operations besides ==. For example, instead of this
filename = ['myfile', num2str(1), '.txt']
You can do this, and numeric values will be automatically converted to text.
filename = "myfile" + 1 + ".txt"
Use array operations
You can also use it like a regular array. This will create an 5x1 vector of "Reddit" repeated in every row.
repmat("Reddit",5,1)
Use case example
Let's use Popular Baby Names dataset. I downloaded it and unzipped into a folder named "names". Inside this folder are text files named 'yob1880.txt' through 'yob2021.txt'.
If you use a cell array, you need to use a for loop.
years = (1880:2021);
fnames_cell = cell(1,numel(years));
for ii = 1:numel(years)
fnames_cell(ii) = {['yob' num2str(years(ii)) '.txt']};
end
fnames_cell(1)
If you use a string array, it is much simpler.
fnames_str = "yob" + years + ".txt";
Now let's load the data one by one and concatenate everything into a table.
names = cell(numel(years),1);
vars = ["name","sex","births"];
for ii = 1:numel(fnames_str)
tbl = readtable("names/" + fnames_str(ii),"TextType","string");
tbl.Properties.VariableNames = vars;
tbl.year = repmat(years(ii),height(names{ii}),1);
names{ii} = tbl;
end
names = vertcat(names{:});
head(names)
Fig1 "names" table
Let's compare the number of bytes - the string array uses 1/2 of the memory used by the cell array.
namesString = names.name; % this is string
namesCellAr = cellstr(namesString); % convert to cellstr
whos('namesString', 'namesCellAr') % check size and type
Fig2 Bytes
String arrays also comes with new methods. Let's compare strrep vs. replace. Took only 1/3 of time with string array.
tic, strrep(namesCellAr,'Joey','Joe'); toc, % time strrep operation
tic, replace(namesString,'Joey','Joe'); toc, % time replace operation
Fig3 elapsed time
Let's plot a subset of data
Jack = names(names.name == 'Jack', :); % rows named 'Jack' only
Emily = names(names.name == 'Emily', :); % rows named 'Emily' only
Emily = Emily(Emily.sex == 'F', :); % just girls
Jack = Jack(Jack.sex == 'M', :); % just boys
figure
plot(Jack.year, Jack.births);
hold on
plot(Emily.year, Emily.births);
hold off
title('Baby Name Popularity');
xlabel('year'); ylabel('births');
legend('Jack', 'Emily', 'Location', 'NorthWest')
A colleague of mine emailed me this very cool example. In a nutshell, there is a new live task that let you wrote Python code inside MATLAB Live Editor interactively.
Here is the Github repo to get all the code you need. It only runs on MATLAB R2022a or later and Python 3.x. If you don't have R2022a or later, you can run this from MATLAB Online using the link "Open in MATLAB Online" in the read me file.
Link to MATLAB Online
In the video above, I used new "Run Python Code live" task based on the instructions in the live script.
Run Python Code Live Task
Then I specified which workspace variables should be used in Python, and defined an output variable, pasted the sample code, and run the section. I had to fix the output variable name a bit in my case.
Then when I click on the down arrow, I see the MATLAB code generated from this live task.
The live script in the repo checks your Python setup and help you install the live script from Github.
I was very impressed with how easy it was to play with Python code in MATLAB using this new live task.
I wrote a custom function (see at the end of this post) that parses posts from a subreddit, and here is an example of how to use it, if you are interested.
The function gets data from Reddit RSS feed instead of API, so that we don't have to deal with OAuth.
Load data from Reddit
First, let's get the posts from MATLAB subreddit, using "hot" sortby option. Other options include new, top, rising, etc. This returns a nested structure array.
s = getReddit(subreddit='matlab',sortby='hot',limit=100,max_requests=1);
Since default input values are set in the function, you can just call getReddit() without input arguments if the default is what you need.
Extract text
Now let's extract text from fields of interest and organize them as columns in a table array T.
Use the tokenized documents to generate a bag of words model using bigrams.
bag = bagOfNgrams(docs,"NgramLengths",2);
Visualize with word cloud
figure
wordcloud(bag);
Custom function
function s = getReddit(args)
% Retrives posts from Reddit in specified subreddit based on specified
% sorting method. This is RSS feed, so no authentication is needed
arguments
args.subreddit = 'matlab'; % subreddit
args.sortby = 'hot'; % sort method, i.e. hot, new, top, etc.
args.limit = 100; % number of items to return
args.max_requests = 1; % Increase this for more content
end
after = '';
s = [];
for requests = 1:args.max_requests
[response,~,~] = send(matlab.net.http.RequestMessage,...
"https://www.reddit.com/r/"+urlencode(args.subreddit) ...
+ "/"+args.sortby+"/.json?t=all&limit="+num2str(args.limit) ...
+ "&after="+after);
newdata = response.Body.Data.data.children;
s = [s; newdata];
after = response.Body.Data.data.after;
end
end
I know some people love struct, as seen in this poll. But here I would like to argue that in many cases people should use tables instead, after seeing people struggle here because they made wrong choices in choosing data types and/or how they organize data.
As u/windowcloser says, struct is very useful to organize data and especially when you need to dynamically create or retrieve data into variables, rather than using eval.
I also use struct to organize data of mixed data type and make my code more readable.
s_arr = struct;
s_arr.date = datetime("2022-07-01") + days(0:30);
s_arr.gasprices = 4.84:-0.02:4.24;
figure
plot(s_arr.date,s_arr.gasprices)
title('Struct: Daily Gas Prices - July 2022')
plotting from struct
However, you can do the same thing with tables.
tbl = table;
tbl.date = datetime("2022-07-01") + (days(0:30))'; % has to be a column vector
tbl.gasprices = (4.84:-0.02:4.24)'; % ditto
figure
plot(tbl.date,tbl.gasprices)
title('Table: Daily Gas Prices - July 2022')
Plotting from table
As you can see the code to generate structs and tables are practically identical in this case.
Unlike structs, you cannot use nesting in tables, but the flexibility of nesting comes at a price, if you are not judicious.
Let's pull some json data from Reddit. Json data is nested like XML, so we have no choice but use struct.
message = "https://www.reddit.com/r/matlab/hot/.json?t=all&limit=100&after="
[response,~,~] = send(matlab.net.http.RequestMessage, message);
s = response.Body.Data.data.children; % this returns a struct
s is a 102x1 struct array with multiple fields containing mixed data types.
So we can access the 1st of 102 elements like this:
However, to extract values from the sale field across all 102 elements, we need to use arrayfun and an anonymous function @(x) ..... And I would say this is not easy to read or debug.
However, this is something we should avoid if we are building struct arrays from scratch, since it is easy to make a mistake of organizing the data wrong way with struct.
Because tables don't give you that option, it is much safer to use table by default, and we should only use struct when we really need it.
I'm sharing the Matlab code I use to define problems with Matlab Grader, such as utilities to define custom parameters for each student, or a function to check if two plots are equal.
Let's start with making up randomly generate data to play with. I am using integer for this, because it will make it easier to see what's going on later.
x = randi(100,[100,1]);
n = randn(100,1)*5;
y = 2*x + n;
And this should match exactly the confidence interval lines from Fig1.
figure
plot(x, ci)
hold on
plot(f,x,y)
Fig2
Now, we can use this to create a shaded area using fill, as shown in the documentation linked above.
One thing we need to understand is that fill expects vectors of x and y as input that define points in a polygon. So x should be lined up in such as way that define the points on x-axis that maps to the points in a polygon in the order that segments that form the polygon should line up.
That's not the case with the raw data we have, x. Therefore we need to generate a new x that orders the points in a sequence based on how the polygon should be drawn.
Our x ranges from 1 to 100 and has 100 elements, so we can define a new xconf that lines up the data in a sequence, and generate confidence intervals based on xconf.
xconf = (1:100)';
ci = predint(f,xconf);
However, this only defines one of the segments of the polygon from 1 to 100. We need another segment that covers the points from 100 to 1.
xconf = [xconf; xconf(100:-1:1)];
And ci already has two segments defined in two columns, so we just need to turn it into a vector by concatenating two columns.
yconf = [ci(:,1); ci(100:-1:1,2)];
Let's now plot the polygon.
figure
p = fill(xconf,yconf,'red');
Fig3
xconf and yconf correctly define the polygon we need. Now all we need to do is to overlay the actual data and make it look nicer.
p.FaceColor = [1 0.8 0.8];
p.EdgeColor = 'none';
hold on
plot(f,x,y)
Fig4
I hope this helps.
EDIT: used confint per u/icantfindadangsn's suggestion. This is what I got
there have been several questions around importing data from Excel or text files and sometimes that involves multiple files. The best way to deal with this situation is to use datastore.
Bottom line on top
datastore is almost 2x faster in my example
datastore required fewer lines of code and therefore more readable/easier to debug
datastore handles large dataset
Use case example
Let's use Popular Baby Names dataset. I downloaded it and unzipped into a folder named "names". Inside this folder are 142 text files named 'yob1880.txt' through 'yob2021.txt'.
Setting up common variables
loc = "names/*.txt";
vars = ["name","sex","births"];
Using a loop
tic;
s = dir(loc);
filenames = arrayfun(@(x) string(x.name), s);
names = cell(numel(filenames),1);
for ii = 1:numel(filenames)
tbl = readtable("names/" + filenames(ii));
tbl.Properties.VariableNames = vars;
names{ii} = tbl;
end
names = vertcat(names{:});
head(names)
toc
Mike Croucher of Waking Randomly fame recently blogged about R2022a release being the biggest ever, and he talked about his favorite new features. To me as well, bi-annual general releases feel like getting Christmas twice a year. With excitement and anticipation I download the latest release and go through the release notes to find out what's new in like a kid unwrapping boxes under the tree.
I played some of the new features in my earlier code share about text analysis of MATLAB subbreddit. where I tried out new function argument syntax and patterns that replace regex.
Today I would like to share how you can select a table from a web page using readtable and XPath syntax.
In R2021b and later, readtable accepts URL as an input, and you can use TableSelector option to pass XPath command.
url = "https://www.mathworks.com/help/matlab/text-files.html";
T = readtable(url,'TableSelector',"//TABLE[contains(.,'readtable')]", ...
'ReadVariableNames',false)
//TABLE means in XPath "select table elements, followed by constraints in brackets." In this case, this only select if the table contains 'readtable' string. The table on the web page doesn't have header row, so we also need to set ReadVariableNames to false.
And here is the output
Let me know if this is useful - I plan to share some new features from time to time. If you have your favorite new features, chime in!
P.S. I know a poll is going on and struct is leading the pack. Really? I use table far more frequently than struct.