Hear what a voice is feeling.
Record half a minute of speech. A light model listens first and calls the heavy model only when your clip needs it. You get one answer, the moments that drove it, and a spoken explanation of why.
How a run works
Four steps. Your audio is sent to the analysis server, processed in memory and discarded. Nothing is kept.
Give it audio
Record from your microphone or upload an audio file. Up to thirty seconds of natural speech; longer gives a steadier answer.
The light model listens first
An MFCC model hears the whole clip in milliseconds. A router reads the opening seconds and the light model's confidence, and decides whether that answer can stand.
The heavy model steps in when needed
Emotion2Vec+, a 94-million-parameter speech encoder, analyses the clip only when the router escalates. Or pick Heavy model only to always use it.
You hear why
The timeline shows which five-second stretches pushed toward the answer and which pushed against it, and the page can read its reasoning out loud.
Read this before you trust a result
It is right about one time in three
Measured at 35% on 516 real recordings from four people, across six emotions. Better than the 17% you'd get by guessing, nowhere near good enough to decide anything that matters about a person.
Confidence is not correctness
A 90% score means the model found the clip unambiguous, not that it is right. Read the confidence badge and the timeline together.
Two models, one answer
Each recording is answered by either the light model or the heavy model, never both. The page always tells you which one answered.
It is not a diagnosis
No clinical, hiring, or safety use. This is a demonstration of a speech-emotion pipeline and its explainability, nothing more.
Run a recording
Record or upload, choose a model, then analyze. You get one answer, the moments behind it, and an explanation you can listen to.
Source
Longer recordings give a steadier result.Tips for a cleaner recording
- Pick a quiet room and switch off fans, air-con and the TV.
- Keep the mic about 15–20 cm from your mouth.
- A wired headset or earphone mic beats a laptop mic. Bluetooth mics lower the audio quality.
- Speak naturally for as long as the recording allows; short pauses are fine.
Server settings
All six emotions
Where the answer came from
Spend the heavy model only where it pays
A light model listens to every recording. A router, trained on real voices, predicts whether the light answer would match the heavy model's. If not, the clip is worth ~24× more compute, and the heavy model answers instead. Watch it decide.
Pick a recording
Three acted clips from public datasets, with known labels. Acted speech is exaggerated and studio-clean, which is unlike the real voices this router learned from, so it sends all three to the heavy model. For a clip that stays light, try your own voice on Analyze.
What this buys
Record a labelled clip
Real voices are what this model is short of. Record a clip, tell us what you were going for, and save it — the file downloads to your machine, correctly named and ready to send back.
1 · Record
No clip yet.
2 · Label it
How the answer is produced
The whole pipeline, the two models behind it, the numbers, and an honest account of what it cannot do.
The pipeline
Prepare the audio
Downmixed to mono and resampled to 16 kHz. Nothing else: no denoising, trimming or normalisation, so the models hear the recording as it was made.
Light model
13 MFCCs plus their deltas and delta-deltas (coefficient 0 dropped), averaged over the whole clip into 72 numbers, then a small neural network (72 → 256 → 128 → 6).
Router (ASCA)
Twelve acoustic complexity features from the opening three seconds (SNR, energy, spectral shape, flux, entropy, MFCC statistics) plus the light model's own confidence: fifteen numbers into a logistic regression. It escalates when P(needs heavy) ≥ 0.63.
Heavy model
Emotion2Vec+ turns the clip into a 768-number embedding; a standard-scaled logistic regression turns that into six probabilities. It runs only on escalation, or always in Heavy-only mode.
Explain the answer
The clip is cut into five-second windows. For the heavy model each window's push toward the answer is linear arithmetic on its own embedding; for the light model each window is scored by the light model on its own. Pitch and loudness come from signal processing, not the model.
The two models and the router
Sizes from the saved files. Accuracy is on real voices from people the models never trained on.
| Part | What it is | Size | Real-voice accuracy |
|---|---|---|---|
| Light | MFCC (72) → MLP 256-128. Trained on 5,805 acted clips plus real voices with fan, chatter, echo and phone-line copies, to copy the heavy model's answers. | 52,358 params | 28.3% alone |
| Router | Logistic regression on 15 features. Learns "will the light answer match the heavy one?" Threshold set with Learn-then-Test on 516 real clips. | 16 params | — |
| Heavy | Emotion2Vec+ base (frozen, 94M params) + logistic regression trained on acted speech, MELD and IEMOCAP. | 94M + 4,614 | 34.9% |
| Routed | Green Inference Engine end to end. | — | 34.1% |
Leave-one-person-out on 516 clean recordings from four people (notebook 31). Chance is 16.7%.
Known limits
Routing saves compute, not accuracy
On new people the light model answered 31% of clips, with accuracy 34.1% against 34.9% for the heavy model alone. The target was for the final answer to match the heavy model's on at least 90% of clips; on new people it matched on 75%. Four people is a small sample.
Trained mostly on acted speech
Much of the training material is performed emotion. Real conversation is flatter and more ambiguous, which is most of the gap between benchmark numbers and real-voice results.
Long clips get approximate evidence
A window is re-analysed on its own, without the context around it, so on clips longer than five seconds the per-window bars are supporting evidence rather than exact arithmetic.
Six boxes for something that is not six boxes
Real speech carries blends, sarcasm, and no emotion at all. Forcing every clip into one of six labels throws that away by construction.
The people behind it
A final-year project built by three students with their project guide.
Adaptive, Efficient and Explainable Audio-Only Speech Emotion Recognition using Green Inference and Audible Saliency
Built by
Rishikesh Patil
@Rishi-Is-Cool- Website design and build
- Heavy model (Emotion2Vec+ branch)
- Research: adaptive routing (Green Inference Engine)
- Hardware
- Deployment, debugging and fixing errors across the project
Maitry Mahesh Mohite
@maitry-mohite- Data collection
- Preprocessing
- Research: HuBERT layer-wise representations and CKA
- Hardware
Tanvi Yerram
@tanviyerram08- Data collection
- Preprocessing
- Research: acoustic explainability
- Hardware
- Testing
Guided by
Dr. Sheetal Patil
Project guide. Research guidance throughout the project, and helped build the real-voice dataset.