Expert report review
How accurate is forensic voice comparison, and how does a reviewer check the error rate?
No single accuracy figure covers forensic voice comparison. An FBI examiner wrote in 1993 that the actual error rate of spectrographic comparison under most investigative conditions was unknown, and a federal court said the same in 2003. In a 2018 study a semi-automatic system had an equal error rate of 10.33% on matched studio recordings and about 26% to 32% when one recording came over a telephone or mobile channel.

Is there a known error rate for forensic voice comparison?
No single figure covers the field. The rate depends on the method, the recordings and the examiner's protocol, and a federal court in 2003 found the error rate of aural spectrographic comparison unknown.
Bruce E. Koenig of the FBI wrote in a 1993 article in Crime Laboratory Digest (vol. 20, no. 4) that "the scientific support of this examination is limited" and that "the actual error rate under most investigative conditions is unknown" (p. 79). He described the FBI's policy "not to provide testimony on spectrographic comparisons" because of the "inconclusive nature of the examination" and the unknown error rate (p. 80).
Ten years later, in United States v. Angleton, decided June 26, 2003, Judge Rosenthal of the Southern District of Texas found that "the potential rate of error of the aural spectrographic method is unknown." The court wrote that error rates "vary considerably among applications of the method and depend heavily on the protocol used by the analyst," and that "the one percent error rate Koenig observed at the Federal Bureau of Investigation has not been reliably repeated."
The error rate matters in court because it is one of the factors in Daubert v. Merrell Dow Pharmaceuticals, decided June 28, 1993: the Court listed the technique's "known or potential error rate" along with testing, peer review, standards and general acceptance. A report that gives no accuracy figure for its method under the conditions of the case leaves that factor without evidence.
Which recording conditions lower accuracy?
Short samples, noise, narrow bandwidth and whispering. Koenig's 1993 review of listener studies reports lower identification rates as each condition worsens.
Koenig summarized studies of human listeners. For familiar voices, median accuracy was 98% for sentences under three seconds and 88% for a single two-syllable word such as "mama," with a range of 73% to 98% for that word. Whispered samples needed more than three times the duration of normal speech for the same level of identification (p. 78).
For unfamiliar voices, correct identifications were about 90% at a signal-to-noise ratio of 30 dB and about 80% at 0 dB. With the frequency response cut to 1,000 Hz, correct identifications were about 78%, compared with about 88% at 4,500 Hz (p. 79). From a 1966 study by Clarke and colleagues, Koenig projected that correct identifications of unfamiliar voices would range from 78% to 90%, with most in the 78% to 83% range.
He also reported that spectrographic comparison has "a certain error rate that is independent of examiner-induced errors," and that errors of false elimination, where two samples of one person are judged different, are appreciably higher than false identifications (pp. 79-80).
A reviewer reads the opposing report for each of these conditions: the net duration of speech in the questioned and known recordings, the noise level, the channel and whether any speech was whispered or disguised. A report that does not state them gives no basis to judge the accuracy of its conclusion.
How much does a channel mismatch change the result?
In a 2018 study, a semi-automatic system had an equal error rate of 10.33% with studio recordings on both sides and 25.95% to 31.99% when the questioned recording came over a telephone or mobile channel.
Vincent Hughes, Philip Harrison, Paul Foulkes, Peter French, Colleen Kavanagh and Eugenia San Segundo tested a semi-automatic system built on long-term formant distributions (F1, F2 and F3) with recordings of 97 male speakers from the DyViS corpus, presented at Interspeech 2018. The suspect sample was always a studio recording. The offender sample was studio, landline, or mobile at 12.2 or 4.75 kbit/s.
The equal error rate, the point at which the share of wrong same-speaker decisions equals the share of wrong different-speaker decisions, was 10.33% for studio against studio, 25.95% for studio against landline, 31.71% for studio against mobile at the higher bit rate and 31.99% at the lower. The log-likelihood-ratio cost (Cllr), a second measure in which lower is better, was 0.37, 0.73, 0.81 and 0.83.
The gap between the matched and the worst mismatched condition was 21.66 percentage points of equal error rate. For a reviewer the comparison is direct: a known sample recorded in a quiet office and a questioned call recorded from a phone line are a mismatched pair, and the report has to say how the method performs on such pairs.
Does accuracy vary by speaker?
Yes. In the 2018 study, speakers differed both in the strength of the evidence and in how stable their results were across recording conditions. The authors concluded that testing should analyze individual speakers.
Hughes and colleagues found that speakers who produced stronger evidence were also more variable across conditions, and that mean F3 predicted variability in same-speaker log-likelihood ratios, while mean F1 and voice quality features did not.
The authors concluded that analysis of individual speakers should be an essential part of testing such systems. A validation figure averaged over a test population says how the system performs on average, and the reviewer asks whether the report says anything about the speaker in the case.
Morrison and Thompson, in a 2017 article in the Columbia Science and Technology Law Review (vol. 18, no. 2), discuss procedures for empirically validating forensic voice comparison systems and the risk of contextual bias, review Angleton and the Daubert hearing in United States v. Ahmed (2015), and set out how a proponent of newer methods could meet a rigorous Daubert standard.
What minimum material did examiners' standards require?
In the 1993 table, both the International Association for Identification and the FBI asked for 20 words for the highest confidence level, the original recording and an independent second decision.
Koenig's Table 2 compares the IAI and FBI requirements for spectrographic comparison. Words needed for the highest confidence level: 20 for both. Words needed for the lowest confidence level: 10 for the IAI, 20 for the FBI. Original recording required: yes for both. Completely verbatim known samples: usually. Speech frequency range: above 2 kHz for the IAI, above 2.5 kHz for the FBI. An affirming independent second decision: yes for both.
Angleton shows what happens when a report departs from such standards. The IAI and the American Board of Recorded Evidence standards called for a list of usable words on both recordings, similar and dissimilar, and the examiner listed only dissimilar ones. The ABRE standards called for an impartial person familiar with the speaker's voice to be present, whenever possible, when the exemplar was recorded, and the examiner did not ask the officers to arrange this. The exemplar was not recorded with equipment or conditions that matched the original; the speaker's distance from the microphone differed, and the original had background reverberation that the exemplar lacked.
The court added that "an examiner's report should also contain a statement of accuracy," and that the defense examiner's report had none.
How should the report state the strength of a voice comparison?
Within the limits of the method. The 2023 committee note to Rule 702 says each opinion must stay within what a reliable application of the basis and method can support and warns against claims of absolute certainty.
The committee note to the 2023 amendment of Rule 702 says that "each expert opinion must stay within the bounds of what can be concluded from a reliable application of the expert's basis and methodology," and that forensic experts should avoid assertions of absolute or one hundred percent certainty, or of a reasonable degree of scientific certainty, if the methodology is subjective and thus potentially subject to error.
In United States v. Baller, 519 F.2d 463 (4th Cir. 1975), the court of appeals said that absolute certainty is not required for admissibility, and it recommended the word "spectrogram" in place of "voiceprint." The jury in that case was told that the spectrograms were only a basis for the opinion and that it could disregard the opinion if the method was not sufficiently reliable.
Angleton gives the reviewer a test of stability. The defense examiner's conclusion moved over time: "possible elimination" in 1998, "probable elimination" in 2003. A second analyst reached a different conclusion and found that most phrases lacked sufficient quality for comparison. A conclusion that changes without new data is a point for cross-examination.
How do jurors weigh voice evidence and error rates?
In a 2020 study of 897 mock jurors, voice comparison evidence drew fewer guilty votes than fingerprint evidence, and the drop in guilty votes that error rate information produced for fingerprints was not found for voice evidence.
Brandon L. Garrett, William E. Crozier and Rebecca Grady published the study in the Journal of Forensic Sciences (vol. 65, no. 4, pp. 1199-1209). The 897 participants, recruited on Amazon Mechanical Turk, evaluated written testimony on fingerprint or voice comparison evidence, qualified by error rates, by likelihood ratios or by no qualifying information.
Mock jurors were more skeptical of voice analysis than of fingerprint analysis. Error rate information reduced guilty votes for fingerprint evidence, and the study did not find that effect for voice evidence. The authors concluded that jurors adjust the weight they give forensic evidence according to their prior views of its reliability.
The study tested general error-rate information. A review that names the channel mismatch or the short sample in the opposing report gives the factfinder reasons tied to the case.
What does a reviewer ask about accuracy in the opposing voice report?
Which method was used, how it was validated, what its error rate is under conditions like those in the case, how much speech was compared, and whether the conclusion is worded within those limits.
The facts or data considered must be listed in the report under Rule 26(a)(2)(B)(ii), so the reviewer starts with the list: the questioned recording, the known recordings, the segments used, the features measured, the software and settings, and any validation data.
Rule 702(b) asks whether the testimony is based on sufficient facts or data, 702(c) whether it is the product of reliable principles and methods, and 702(d) whether the method was reliably applied to the facts. The 2000 committee note adds two questions that fit voice work: whether the expert has adequately accounted for obvious alternative explanations, and whether the field is known to reach reliable results for the type of opinion the expert would give.
From the sources above the reviewer builds the questions. What is the net duration of speech in each recording? Were the known and questioned recordings made over the same kind of channel? Has the method been tested on mismatched pairs, and with what equal error rate? Did a second examiner make an independent decision? Did the examiner know who the suspect was before comparing?
A lay witness may still identify a voice under Rule 901(b)(5), and the advisory committee note says that aural voice identification is not a subject of expert testimony. The accuracy questions above apply to the expert comparison, which is governed by Rule 702.
What does IAE do when the accuracy of a voice comparison is disputed?
An IAE examiner prepares a review of the opposing audio examiner's report for a federal or state case, in up to 5 business days after the format and cost are confirmed.
On the IAE audio and voice page, the examiner compares the questioned voice with known recordings by pitch, formants, speech rate and pronunciation, and states the limits of the comparison in the report. The review checks which file the examiner worked from, whether its hash and metadata are listed among the facts or data considered, and whether the known samples were enough for the opinion, the test of Rule 702(b).
IAE describes a review of an expert report as an assessment of the method, the completeness of the examination, the soundness of the conclusions and the expert's competence. The questions are confirmed with the client before work starts.
Sources
- Koenig B. E., Selected Topics in Forensic Voice Identification, Crime Laboratory Digest 20(4), 1993 (NCJRS)
- United States v. Angleton, 269 F. Supp. 2d 892 (S.D. Tex. 2003) (Justia)
- United States v. Baller, 519 F.2d 463 (4th Cir. 1975) (Justia)
- Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993) (Cornell LII)
- Federal Rules of Evidence, Rule 702 and committee notes (Cornell LII)
- Federal Rules of Evidence, Rule 901 (Cornell LII)
- Federal Rules of Civil Procedure, Rule 26 (Cornell LII)
- Hughes V. et al., The individual and the system: assessing the stability of the output of a semi-automatic forensic voice comparison system, Interspeech 2018
- Morrison G. S., Thompson W. C., Assessing the Admissibility of a New Generation of Forensic Voice Comparison Testimony, Columbia Science and Technology Law Review 18(2), 2017 (NIJ abstract)
- Garrett B. L., Crozier W. E., Grady R., Error Rates, Likelihood Ratios, and Jury Evaluation of Forensic Evidence, Journal of Forensic Sciences 65(4), 2020 (Duke Scholars)
Read also
All articles
ShotSpotter evidence: is it admissible, and how does a reviewer test the other side's acoustic gunshot report?
An acoustic gunshot report tells the jury how many shots were fired, when and where, on the basis of sound picked up by sensors. Courts have admitted
The opposing audio report rests on a compressed file from a messaging app or a phone: what does a reviewer check?
Most recordings in disputes today reach the examiner as compressed files: voice messages, call recordings, memos made on a phone. Compression
Was the opposing audio examiner told what to find, and how does a reviewer check for contextual bias?
An audio examiner who knows that the suspect confessed, or that other evidence points to him, is working with information that the examination doesExpert report review
Ready to discuss your case?
Send the materials: the secretariat will confirm the format, cost, deadline and the specialist within one business day.

