Development of an AI Depression Detector Using Text and Speech

AI depression detection: where accuracy is lost

AI depression detection breaks not at the model but at the input: assessment of state from text and speech begins with which signals actually reach processing.

In practice, text and voice requests often pass by specialists' attention — the person answers briefly, speech sounds even, there are no explicit complaints, and the material goes into the archive without a label.

The main blind spot is missing state features. In speech, these are flattened intonation, lengthened pauses, reduced tempo, and impoverished prosody; in text, generalized wording, negations, short answers, simplified syntax.

Individually, each feature is weak and indistinguishable from fatigue, so modalities are processed separately, and decisions are combined at the last step without mixing raw signals.

The second source of errors is fragmented specialist conclusions: different scales, free-form notes, absence of subthreshold cases in labeling.

The model fine-tunes on what is easily agreed upon and begins to catch writing style instead of state — hence false positives on formally "heavy" text and misses where the state is expressed with restraint.

Noise is also added by shift between recordings: a different headset, a different site, a different language of description change the feature distribution, and quality drops with the same model.

We discuss how this is addressed — with a unified labeling scheme, separate processing of modalities, threshold calibration on target recordings, and stability monitoring across samples.

Measurable effect: speed, coverage, stability

Before implementation, initial assessment was limited by specialist resources: one review took tens of minutes, the queue grew, and disputed cases were interpreted differently depending on who read the request. Only a small part made it into the sample — the rest simply did not reach assessment.

The model takes over initial assessment, and a specialist checks and confirms it. Time to initial assessment drops from hours of waiting in queue to minutes: requests with a high risk score go first, routine ones are reviewed in the background.

Coverage grows from a selective 10–15% to all incoming — text messages and speech recordings, including transcription and voice features: tempo, pause duration, evenness of intonation.

Unified assessment criteria produce a second effect. Labeling is done according to the validated PHQ-9 scale, rules are consolidated into one document, and disputed cases are described with examples.

Re-labeling of the same subset by two specialists agrees in 85–90% of cases, and discrepancies are reviewed and incorporated into the rules.

Reproducibility also rests on the engineering side: the model version is fixed, every decision is saved to a log together with input data and assessment, and the archive can be rerun to get the same picture.

When a version is updated, the old and new are compared on the hold-out set — it is visible where accuracy improved and where the false positive rate increased.

The outcome is measured by three numbers over time: time to initial assessment, share of covered requests, and agreement of decisions on recheck.

Detection operating modes: batch and streaming

Detection relies on the same set of features — vocabulary, speech tempo and pauses, changes in sentiment over the course of a conversation.

The difference between the modes is not in the model but in when the result is needed: to analyze accumulated material or to prompt directly during a dialogue. Therefore the modes do not exclude each other and are often enabled together.

The key limitation of the streaming mode is response latency. A prompt to the operator is useful only if it arrives while the conversation is still ongoing, so the assessment is updated as utterances come in rather than after the conversation ends.

The price for this is a narrow observation window: the streaming mode may fail to capture rare state changes and slow shifts over a long period.

Batch mode, by contrast, sees the material as a whole — over a long span, dynamics and recurring episodes appear that are lost in a short window.

The limitation here is the opposite: the result appears with a delay, and accuracy depends on speech recognition in the recording and on how completely the dialogue is preserved.

How implementation proceeds: stages of engineering work

Implementation of the detector starts not with the model but with data: text and speech provide different sensitivity, and modalities cannot be mixed without checking recording quality.

The work proceeds in stages; after each one, the client receives an artifact and decides whether to continue.

1. Data and scenario audit. We look at what recordings exist — interviews, diary texts, calls — where sampling bias lies, and which labels can actually be obtained. The output is a labeling protocol and requirements for recording quality.
2. Labeling of the record corpus. We translate clinical scales into level labels; each fragment is labeled by at least two people, and a third resolves discrepancies. We calculate agreement — if it is low, we revise the instructions rather than averaging disputed cases.
3. Hold-out set. We split by people, not by recordings: otherwise a patient's voice ends up in both training and validation, and the acceptance metric is inflated. The hold-out portion is not opened until final validation.
4. Training and feature selection. Text: topic, syntax, negations. Speech: tempo, pauses, intonation. We keep what genuinely adds discriminative power.
5. Calibration of decision thresholds. The cost of a miss and a false positive differs, so we select the threshold for the scenario — screening or monitoring — recording sensitivity and specificity and behavior on borderline cases.
6. Acceptance. Validation on the hold-out set, error analysis by subgroups, testing on noise and truncated fragments. Metrics are recorded in writing before refinements.
7. Handover to operation. Channel connection scheme, decision logging, escalation rule to a specialist, and procedure for rechecking on new data.

Control points are after labeling, the hold-out set, and acceptance: if label quality or dataset separation is not confirmed, work stops until the causes are eliminated rather than being masked by fine-tuning.

Project artifacts: models, labeling rules, procedures

A project in the field of AI depression detection ends not with a demonstration but with a package that lives inside the client's environment.

Below is the minimum set of project artifacts that we hand over: without any one of them, the model remains a black box that cannot be checked or updated by your own team.

• Trained models of two modalities — text and speech branches — together with inference settings, audio preprocessing parameters, and a fixed version of dictionaries and features.
• Labeling rules: an annotator guide with a definition of each class, examples of disputed cases, a cross-check scheme, and a requirement for inter-annotator agreement above a specified threshold.
• Quality control procedure: a set of metrics (sensitivity, specificity, missed severe case rate), the procedure for calibrating the decision threshold, and the escalation rule to a human specialist.
• Model documentation: feature map, validation report on the hold-out set, analysis of common errors, and an explicit list of applicability limitations.
• Retraining procedure on new data: frequency, signs of drift in incoming texts and speech recordings, launch procedure, and version log with rollback capability.
• Instructions for specialists: how to read the model report, what each risk level means, and when a human gets involved.
• A set of de-identified test data and environment reproduction tools so that updates on the client side yield the same result as ours.

Case study: how we reduced misses in initial assessment

Domain: initial assessment of requests in a psychological support service: several hundred forms and text messages per week, manual selection by complaint and by the PHQ-9 scale.

Specialists physically reviewed only part of the flow, the rest waited in queue, and some severe cases surfaced only on a repeat request.

To keep metrics fair, measurement was done on a labeled sample of past requests: two specialists independently labeled cases, and disputed ones were resolved by a third. Before and after were calculated on the same data, without cherry-picking convenient examples.

Metrics before and after

What produced the result

A two-stage cascade works. The model evaluates text and speech — lexical markers, tempo, pauses, prosody changes — and outputs a risk score; the decision remains with the specialist, the system does not diagnose but prioritizes the queue.

The cutoff threshold was tuned toward sensitivity so as not to miss severe cases, and excess false positives were suppressed by a second filter — a short structured interview.

Drift was monitored separately: once a quarter we run a frozen set of examples and compare metrics. If the false positive rate grows, thresholds are revised before the load on specialists returns to its previous level.

We transfer the same pipeline to adjacent domains — there it is enough to replace the labeling and recalibrate thresholds.

How text analysis differs from speech analysis

Short answer: not one source but their consistency is more reliable. Text answers the question of what a person says about themselves; speech answers in what state they say it.

The discrepancy between them is informative in itself: cheerful wording with a slowed, even voice is a typical picture of masking.

analysis of request text works with vocabulary and structure: the share of negations and generalizations, words with semantics of hopelessness, vocabulary impoverishment, shifting of "I" toward impersonal constructions, breakdown of coherence between paragraphs.

Features are robust to noise but depend on language, dialect, and how willing a person is to write at length.

analysis of speech and intonation takes tempo, pause duration, monotonicity, volume dynamics, response latency.

These are state features that a person does not consciously control, so they are harder to fake — but they are the first to suffer from poor recording: echo, narrowband channel, background conversation.

audio recording quality is a limiter, not a reason to abandon acoustics. If the recording is noisy or truncated, the weight of speech features decreases, the result shifts toward text, and assessment confidence drops explicitly rather than silently.

joint assessment of signals is a single assessment based on two feature sets, calibrated on the validated PHQ-9 scale: a weak signal does not drag the result along, a strong one compensates for it.

choice of data source is dictated by the channel — chat, call transcript, voice message, or free-form narrative; feature sets and thresholds differ for them. The output is not a diagnosis but a priority: with a high score, the system directs the person to a specialist.

Concerns about data, privacy, and implementation

Before a pilot project, five technical questions usually come up: where recordings are stored, whether the corpus is sufficient, whether the conclusion can be explained, whether data remains within the environment, and what lawyers sign off on. We address each.

We have few recordings — will the model train?

Yes. We fine-tune pretrained language and speech representations: several hundred labeled cases provide stable separation. Class balance and labeling quality matter more than volume.

Can a doctor understand why the conclusion is what it is?

For each assessment, we attach feature contributions — which text fragments and speech segments influenced it. The doctor sees the supports of the decision, not an abstract score.

Can it be deployed on our infrastructure?

Yes, training and inference run in your environment: your servers or an isolated cloud, data does not leave. We hand over the model, configurations, and deployment scripts.

What will need to be coordinated with lawyers?

We create a data map, describe the grounds for processing, retention periods, and deletion procedure. This is the first thing lawyers check — we prepare documents according to your policies.

Let's discuss your task and come back with a plan

To come back with an assessment of the task and scope of work, we need a snapshot of your process: what data already exists, what form it is in, and who makes decisions based on the model results.

We go through this in the first conversation — no generic questions, but specific points that determine the architecture and pilot timeline.

• Modalities and sources: we work only with text (requests, correspondence, transcripts) or add speech — then recordings and labeling of delivery, tempo, and pauses are needed; we separately discuss what to do with noise, dialects, and mixed speech.
• Corpus and labeling: how many labeled examples exist now, who labeled them — a clinician or an assessor, whether labels are tied to validated scales. This determines whether we start with fine-tuning or from scratch.
• Acceptance criteria: target recall at a given precision, separately the rate of missed severe states, the decision threshold, and the model response format: score, category, or flag for the specialist.
• Privacy and environment: where data is stored, whether external processing is allowed, whether a closed environment on your servers is needed, how access to sensitive records is restricted and how long they are retained.
• Integration points: where the stream comes from, in what form the result is needed, which specialists see it, and how feedback for fine-tuning is returned.
• Pilot launch: which area and data volume we take, over what period we validate, who reviews disputed cases, and how we calculate the discrepancy between the model and human assessment.
• Implementation plan: stages from prototype to working environment, drift monitoring on new data, retraining and threshold revision procedure.

The model here is a decision support tool, not a replacement for a specialist: we leave final interpretation to the human. Describe your task in free form — we will come back with an estimate of timelines, scope of work, and a pilot plan.