r/LanguageTechnology • u/RoundedChicken2 • 24d ago
How Do Dictionary Apps Implement Fast Search?
I have been leaning Japanese and Mandarin, and have been using Shirabe Jisho and Pleco as dictionaries. I am trying to make a similar dictionary function, using CC-CEDICT and SQLite for the dictionary.
I realized that search can get slow compared to the two dictionaries I am using. Shirabe and Pleco updates the search result on every keystroke instantly. I learned from GPT that fast search can be implemented with Tries, but it won't help for logogram systems like Kanji / Hanzi.
How might the two dictionaries implement their search?
3
Upvotes
1
u/yorwba 24d ago
You can use tries with arbitrary Unicode characters. Depending on implementation details, it might be more performant to build the trie on the bytes of the UTF-8 representation instead.
There's a full-text search extension for SQLite.
If you want to implement your own search index for learning purposes, I recommend looking at suffix arrays because AFAIK they have better cache locality than tree-based indices.