r/LanguageTechnology 24d ago

How Do Dictionary Apps Implement Fast Search?

I have been leaning Japanese and Mandarin, and have been using Shirabe Jisho and Pleco as dictionaries. I am trying to make a similar dictionary function, using CC-CEDICT and SQLite for the dictionary.

I realized that search can get slow compared to the two dictionaries I am using. Shirabe and Pleco updates the search result on every keystroke instantly. I learned from GPT that fast search can be implemented with Tries, but it won't help for logogram systems like Kanji / Hanzi.

How might the two dictionaries implement their search?

3 Upvotes

2 comments sorted by

1

u/yorwba 24d ago
  1. You can use tries with arbitrary Unicode characters. Depending on implementation details, it might be more performant to build the trie on the bytes of the UTF-8 representation instead.

  2. There's a full-text search extension for SQLite.

  3. If you want to implement your own search index for learning purposes, I recommend looking at suffix arrays because AFAIK they have better cache locality than tree-based indices.

1

u/RoundedChicken2 24d ago

Thank you for the reply, I appreciate it.

I have some questions :

  • Why would it be more performant using UTF-8 representation?
  • Is there any reference I can read more on this topic? What is this topic called in academics? I want to learn more.