← Back to the site
Engineering
Small models on small devices
4×
smaller than the base model
120 ms
median latency on a phone
Offline
no network needed
Notes
The goal
People wanted voice notes that work on a train with no signal. The base model was too big and too slow for the phones most of them own.
The approach
We distilled the model into a smaller student, then fine-tuned it on the vocabulary people actually used in the app.
The result
The new model is four times smaller and answers in about 120 ms on a mid-range phone. Accuracy on everyday words stayed within a point of the original.