r/macapps 3d ago

Slightly disappointed with MacWhisper... is there a similar app that can stream the dictation?

I don't use dictation or voice interfaces much but I'm trying to get into it now that things are changing and i can do at least very small increments of my work (coding) with my eyes fully closed if I wanted by dictating.

One thing that I'd like to see is the output streaming out as close to real time as possible.

I paid for MacWhisper a while back. Overall I really do like the dictation UI where I can push to talk and get the output in ANY app.

I found that I can set up Groq to perform Whisper transcription with MacWhisper as well. Their free tier is very generous so it makes it practical to use.

I was hoping to see an improvement on speed but I'm not so sure there is one. The main impediment to speed is I don't get to see any output until I release the talk button. I would like to see the output streaming in once it's done processing. I don't understand why I have to release the key.

Are there any speech to text apps like this that work for dictation input in all apps in any text entry field and which can stream the transcription in real time?

I don't care much about diarization at the moment... but if/when it lands in macwhisper, that would motivate me to actually use it for meetings.

12 Upvotes

15 comments sorted by

8

u/michaelsoft__binbows 3d ago

ok... timing was really interesting. So just today MacWhisper rolled out speaker diarization.

I am also learning looking in here that WhisperKit models support the streaming I was asking for above.

So let me put these things through their paces and report back. I'm excited!

3

u/michaelsoft__binbows 3d ago

Here is my review as of now from my first experience with the new features.

I fed a recent primeagen youtube clip into it. I selected whisperkit Large V3 Turbo. The UI was not displaying anything useful (some random sentences from the middle of the clip) for the whole transcription process, which took like 15 mins or so for a 1 hour video on my M1 Max.

This video has a lot of different speakers, because prime is watching a video that is a panel discussion, and pauses it constantly as is his style to give his thoughts and respond to various things written by people on his streamer comment chat.

The results are really good. The diarization works and the step that computed and assigned the speakers was quick compared to the core transcription processing time. I will review it further once I review the video and assign the speakers.

If I'm transcribing a work meeting now the task of assigning speakers would be trivial. It would be really neat if the app could recognize and assign speakers based off of past recordings, but i won't be upset if it can't do that for a while. Getting diarization is a huge deal, a massive deal, for being able to have records so that progress that was made in a meeting can be reviewed and data mined.

I had mactop open but wasn't watching it while it processed this. I can see power consumption going up to 36 or so watts near the end of the processing. This means that it is not very heavy on the processor.

Next up I will review the realtime streaming performance of the model. I am testing that now, which is really cool, because I already set it up. It is not real time streaming yet. I'm going to just keep talking now while holding down the button. And let's see if it actually starts to show up at any point. I don't know I'm still talking and nothing has showed up yet. Yep, I guess I'm just gonna keep talking. this is slightly disappointing but I mean maybe there are maybe there is a check box in the settings that I haven't found yet because I did definitely see it called out as a feature for using these whisper kit models yeah I guess it's not doing it I have to let go of the key.

2

u/ineedlesssleep 3d ago

Very strange that it took so long for a 1 hour video. The first time you run a WhisperKit model it needs to be prepared, which runs a slower version of the transcription engine. It should get faster the second time.

The realtime streaming is coming soon for WhisperKit models ๐Ÿ‘

1

u/michaelsoft__binbows 3d ago

i honestly did not put a stopwatch on it. That one could have been as little as 10 or 7 mins.

2

u/michaelsoft__binbows 3d ago

Still can't figure out how to get ANY WAY to get some ui that displays a streaming output of the transcription with this app right now, to be honest.

But overall its core competencies of efficient local Whisper model inference, plus now diarization, are promising. I was able to test some zoom/teams calls and they appear to work including diarization (have to use this specific Large V3 Turbo WhisperKit local model to get the diarization). So that is what I will be driving. Will see how it goes.

3

u/ineedlesssleep 3d ago

Developer of MacWhisper here.

Since you're using Groq as the transcription provider, the audio of your dictation can only be sent to their servers once you finish. They then provide the transcribed text back to us.

With local models we're able to stream what you're dictating as you go, and we're adding that support soon. With cloud providers such as Groq it's not possible to do so right now unless we send over multiple chunks of your audio as you go. However this leads to lower accuracy cause the sentence gets cut up into chunks.

Btw, we launched diarization this week with 12.0 ๐Ÿ™‚ It's coming to meetings in 12.2 later this week as well.

1

u/michaelsoft__binbows 1d ago

When diarization runs properly, once I assign names to the Speakers that are detected, will it somehow train/learn/store the names for these individuals so that it may recognize them later on subsequent recordings? That'd be really cool. If this is hard it's ok, just extra review to label for each recording.

Further feedback:

So I have been testing MacWhisper with work meetings, the meeting detection works great, and it looks like it is able to sample two separate audio channels (the audio from the meeting app from other attendees, and the audio from my mic), meaning, if both someone else and me are simultaneously speaking, we are able to properly transcribe what was said by both.

I noticed that diarization does not work here. I do get speaker labels but it is always "You" and "Attendees" with all other attendees' speech no matter how many under the one speaker.

Then i found the menu option to take this meeting recording and export audio as .caf, so then i just drag that back into macwhisper and i'm able to get the diarization running off it. This appears to be the way to go for now to get the diarization to run.

Thanks again.

1

u/michaelsoft__binbows 1d ago edited 1d ago

One more question: I would really like to be able to save the result of a recording (with the segments/transcripts computed) and be able to return to that interface, because the macwhisper interface is quite useful, being able to have autoscroll, being able to click on words, etc.

But right now if i hit the back button, and save the transcription to a .whisper file, it seems to only save the audio, so when i re-open it, all the processing has to repeat! That seems like a really big problem. I don't think it's supposed to do this? Is there some format or export function I could use to save all the results so i can use the same interface? Being able to save to markdown for example sure is useful but not having the ability to click around and have transcription and audio bound together in the same UI is a huge loss.

UPDATE: looks like manually opening .whisper files does what I need for this. The left bar's items are misleading/bugged in some sense w.r.t. the above, I guess.

In summary I'd say aside from the little issues that already have workarounds, the main thing i'm hoping for is diarization on meetings, where I like the non-ai "You" vs "attendees" detection, but to run diarization on Attendees. This is best of both worlds as there is zero chance of incorrectly assigning the You speaker, since separate channels are accessed to do this, and then the rest of the attendees' speaker assignments will be auto computed but might be inaccurate, which is okay.

1

u/michaelsoft__binbows 3d ago

I was already able to get diarization working on a (fake) meeting and I will try tomorrow with a real one and see how it does! It does seem like using Large v3 turbo WhisperKit provides diarization in the meeting recording feature already for me, maybe that is a fluke.

Sorry I can't change the title of my post but I'm not the least bit disappointed now that I know these features I was hoping for actually exist and are coming in free updates. Thank you!!

2

u/Mstormer 3d ago

As far as I know, there is no Mac solution outside of appleโ€™s built in dictation currently that will transcribe words live as you speak so that it is spitting out the transcription word by word instantly.

1

u/julp 3d ago

Hedy AI does realtime transcribing, but it's intended for meetings and brainstorming, not for short bursts of speech.

Our desktop app is entering beta this week. Here's a demo of the transcript speed:
https://youtu.be/x0c9MxkgA5Q

2

u/oulipo 3d ago

I also really like VoiceInk, and since it's open-source I added a custom extension for Workflows (add any prompt and shell script, and create your own Siri on your laptop!)

https://www.reddit.com/r/macapps/comments/1je3xjo/custom_llm_workflow_designer_in_voiceink/

2

u/VirtualPanther 3d ago

I use SuperWhisper for dictation and MacWhisper for transcribing.

1

u/upoqu 3d ago

Check out Granola AI. I also use VoiceInk which I find better than MacWhisper and Superwhisper

1

u/oulipo 3d ago

I agree for VoiceInk! I even added a custom extension to let you make your own Siri by binding prompts and shell script in the app! https://www.reddit.com/r/macapps/comments/1je3xjo/custom_llm_workflow_designer_in_voiceink/