From Frames to a Speech Detector
Introduction
Project introduction
From Frames to a Speech Detector
You have written the pieces of a speech detector, which are frame energy in decibels, a threshold with a hangover, frame-level precision, recall and F1, a mel filterbank, log-mel features, and the step from frame decisions to segments with start and end times.
In the notebook they run on real recordings. Short spoken words from Google's Speech Commands data set are mixed into 4 second clips with real background noise at known times and noise levels, so every frame has an exact true answer. The functions from the coding problems come already written. Seven cells ask you for new work that only makes sense on real recordings, such as tuning the hangover by measuring it, finding out why the threshold fails in noise, testing whether the network needs each frame's neighbours, and scoring the final segments against the true words.
What you will find out
In quiet recordings the loudness detector with a hangover reaches a frame F1 of about 0.9. Add a running tap 20 dB below the speech, quiet enough that the words are perfectly clear, and its F1 falls to about 0.5 because every frame is now above the threshold. Following the noise level instead of a fixed line helps at 20 dB but finds almost nothing once the noise is as loud as the voice.
A small network that reads log-mel features keeps an F1 of about 0.7 at that point, on speakers it never heard. It is far from perfect, and you will see where it struggles, including a cat whose meows look like a voice. Then you will clean its decisions into about one segment per spoken word.
Where is someone speaking?
4 s, two spoken wordsHow the notebook is organized
01
Loudness
Frames, decibels and a threshold detector with a hangover, scored on quiet recordings.
02
Add noise
Watch the threshold fail, try tracking the noise level, and look at the frequencies instead.
03
Learn the shape
Log-mel features, a small network, segments, and noises the network never heard.
Build a speech detector that survives noise
Run the whole build in the browser, about 45 minutes.