Expert report review
How often do footwear and toolmark examiners get it wrong?
Error rates for footwear and toolmark comparisons come from black-box studies in which examiners compare samples whose source is known to the researchers. A 2022 footwear study found erroneous identifications in 0.2 percent of conclusions on non-mated pairs, and firearm studies report false positives near 1 percent. How inconclusive answers and skipped items are counted changes those figures, and a reviewer checks how the opposing report uses them.

Why does the error rate matter when an impression report is reviewed?
Courts weigh the known or potential error rate of a method under Rule 702 and Daubert. A reviewer uses the published rates to test how strongly the opposing examiner may state a source conclusion.
Federal courts list error rates among the Daubert factors. In Fields v. Ashford (E.D. Mich., November 5, 2019) the court excluded an expert after finding no evidence that he "used techniques that had been tested or whose error-rates had been determined." The court added that this failure, "while not dispositive, further weighs against admission." The 2000 committee note to Rule 702 lists "the known or potential rate of error of the technique or theory when applied" among the Daubert factors.
The 2023 committee note to Rule 702 ties the error question to the wording of the opinion: forensic experts should avoid assertions of absolute or 100 percent certainty, or of a reasonable degree of scientific certainty, if the methodology is subjective and potentially subject to error.
The Department of Justice draws the same line for its footwear examiners. Its footwear ULTR, effective August 15, 2020, bars an examiner from stating that footwear examinations are infallible or have a "zero error rate," and from giving a statistic or numerical probability that is not based on relevant and appropriate data.
A review therefore asks two questions about the opposing report. What do the studies say about error rates for this kind of comparison? Does the strength of the stated conclusion stay within what those rates support?
What error rates did the footwear black-box study find?
In a 2022 study of 84 examiners and 6,610 comparisons, 0.2 percent of conclusions on non-mated pairs were erroneous identifications and 6.0 percent of conclusions on mated pairs were erroneous exclusions.
Hicklin and colleagues published Accuracy, reproducibility, and repeatability of forensic footwear examiner decisions in Forensic Science International, volume 339 (2022), article 111418. Eighty-four practicing footwear examiners completed up to 100 comparisons each, 6,610 comparisons in all. Forty percent of conclusions were definitive (identification or exclusion), 14 percent probable, 40 percent class association and 6 percent neutral.
On non-mated pairs, comparisons where the shoe did not make the print, 0.2 percent of conclusions were erroneous identifications and another 1.4 percent were incorrect "high degree of association" responses. Most of the erroneous identifications came from a single participant. On mated pairs, 6.0 percent of conclusions were erroneous exclusions and another 1.8 percent were incorrect "indications of non-association" responses.
Other examiners reproduced 1.1 percent of the erroneous identifications and 19.9 percent of the erroneous exclusions. No erroneous identification was repeated by the same examiner, and one erroneous exclusion was. The authors found the examiners' judgments on whether a questioned impression was suitable for comparison notably inconsistent, and linked correct definitive conclusions with impression quality and the degree of class similarity or difference.
The FBI Laboratory described its black-box programme for footwear, handwriting and firearms/toolmarks at the 2020 meeting of the American Academy of Forensic Sciences. Each footwear participant reviewed 100 packets of known and questioned samples, and the studies were double-blinded.
What error rates do firearm and toolmark studies report?
The Supreme Court of Maryland reviewed them in 2023. In the Ames I study the false-positive rate was about 1 percent; in Ames II it was 0.7 percent for bullets and 0.92 percent for cartridge cases.
In Abruquah v. State (Md., June 20, 2023) the court went through the studies in the record. In the closed-set Hamby study, 10,447 of 10,455 unknown bullets were correctly identified, 8 were inconclusive and none were misidentified. In the open-set Ames I study of cartridge cases, 22 of 2,180 different-source comparisons were wrongly matched, a false-positive rate of 1.01 percent, and 4 of 1,090 same-source comparisons were wrongly excluded. Of the non-matching sets, 33.7 percent were called inconclusive.
Ames II, completed in 2020, reported false-positive rates of 0.7 percent for bullets and 0.92 percent for cartridge cases. Inconclusive results made up 20.5 percent of matching bullet sets and 65.3 percent of non-matching bullet sets. A Miami-Dade study, which the court described in a footnote from the PCAST report, reported a 2.1 percent false-positive rate and 41.8 percent inconclusive results for non-matching samples.
The court quoted the 2016 report of the President's Council of Advisors on Science and Technology, which called Ames I "the first appropriately designed black-box study" of firearms identification and found that firearms analysis "currently falls short of the criteria for foundational validity."
These studies concern marks left by firearms on bullets and cartridge cases. For tools such as pry bars, screwdrivers and pliers the data are thinner. VanderPlas, Khan, Hofmann and Carriquiry wrote in Firearms and Toolmark Error Rates (2022), a paper submitted by the defense in Illinois v. Winfield: "Fundamentally, we do not know what the error rate is for these types of comparisons."
How do inconclusive results change the numbers?
Most studies count inconclusives as correct or leave them out. The Maryland court noted that counting one category of inconclusives as errors would raise the Ames II false-positive rate for bullets to 10.13 percent.
In Abruquah the court observed that closed-set studies produced few inconclusives while open-set studies produced many, which suggests that examiners pick "inconclusive" even when it is not a correct response. Counting only the "Inconclusive-A" results for non-matching Ames II bullets as incorrect matches raises the false-positive rate to 10.13 percent, against the reported 0.7 percent.
A witness at the hearing, Dean Faigman, testified that counting inconclusives as errors would push the Ames I error rate over 30 percent and the Ames II rates to 53 percent for bullets and 44 percent for cartridge cases. The court did not adopt any single figure, but concluded that the record did not establish that the reported rate is reliable, "especially when it comes to actual casework."
Kori Khan and Alicia Carriquiry, in Shining a Light on Forensic Black-Box Studies (arXiv, 2023), describe the convention. The practice of counting inconclusive decisions as correct, they write, apparently originated in the late 1990s; the Collaborative Testing Service treated them as errors until about 1998. The authors write that "Treating inconclusives as 'correct' is problematic" and that there is no agreed method for handling them.
For the reviewer the consequence is practical. When the opposing report cites an error rate, the review states which study it comes from, whether inconclusives were counted as correct, excluded or treated as errors, and what the rate becomes under each treatment.
Why can study error rates understate casework errors?
Because participants skip items and drop out, and those who skip may be the ones who err. A 2023 statistical paper found nonresponse above 30 percent in several studies and simulated how it can hide false positives.
Khan and Carriquiry report unit nonresponse often above 30 percent. One footwear black-box study had 33.0 percent unit nonresponse and 9.6 percent item nonresponse. Item nonresponse was 35.6 percent in the FBI/Ames bullet study and 27.5 percent in a palmar print study by Eldridge and colleagues.
In the palmar print data, examiners were less likely to answer items rated very difficult, and examiners with high nonresponse were more likely to work for non-U.S. employers or unaccredited laboratories. The authors treat this as evidence that the missing answers are not random.
Their simulation, built on the FBI/Ames design with about 17.9 percent nonresponse, produced an observed false-positive rate of 0.7 percent against a full-data rate of 3.6 percent. The authors write that "It is possible to grossly underestimate the false positive error rate" by ignoring nonresponse, and stress that the simulation illustrates the mechanism and does not estimate the true rate of any existing study.
The OSAC Footwear and Tire Subcommittee has listed the gap as a research need. Its examiner reliability need (version 2 of January 13, 2020) asks researchers to quantify intra-examiner, inter-examiner and examiner-layperson variability, and a white box need approved on October 24, 2022 asks how examiners evaluate the quality and quantity of information in an impression.
What errors surface in casework reviews?
Misidentifications linked to non-blind verification, and conclusions stated with a certainty the method cannot support. A 2024 review of the Rhode Island laboratory and a 2023 Colorado ruling are recent examples.
The Rhode Island Current reported that an independent review by consultant Ronald Nichols, presented to the State Crime Laboratory Commission on October 23, 2024, found that the laboratory had attributed 13 fired cartridge casings in a 2021 Pawtucket murder case to a Glock that could not have fired them. Nichols found a lack of diligence and confirmation bias among the three examiners, and one examiner told him the verification was not truly blind. In 15 other cases he found no further misidentifications and disagreed with two inconclusive findings.
The ABA Journal reported that on July 7, 2023 Colorado Judge Gurley in Grand Junction found that a 1993 trial opinion matching a defendant's pliers and wire cutters to pipe-bomb fragments "to the exclusion of every tool in the world" would be rejected by the relevant scientific community today. The ruling concerned a non-firearm toolmark comparison and led to an order for a new trial.
Neither example is a study, and neither gives a rate. Both point the reviewer to the same records: who verified the conclusion and what the verifier knew, and how the certainty in the report was worded and on what basis.
Which statements in a footwear report does a reviewer flag?
Claims of uniqueness or of exclusion of all other footwear, a zero error rate, absolute certainty, unsupported statistics, and inclusion conclusions that do not say other shoes could have made the print.
The Department of Justice footwear ULTR, approved June 8, 2020 and effective August 15, 2020, permits six conclusions: source identification; inclusion based on class and randomly acquired characteristics; inclusion based on class characteristics; inconclusive; support for exclusion; and source exclusion.
The examiner may not base a source identification or exclusion on the "uniqueness" of the evidence, use the terms "individualize" or "individualization," or assert that the item made the impression "to the exclusion of all other footwear items." Either inclusion conclusion must be given with an explanation that other items indistinguishable in the observed features could have made the impression. The examiner may not claim infallibility or a zero error rate, give a statistic not based on relevant and appropriate data, cite the number of his examinations as a measure of accuracy, or assert absolute certainty or a "reasonable degree of scientific certainty" unless a judge or applicable law requires it.
The 2022 footwear study adds a further check. Because examiners disagreed often on whether a questioned impression was suitable for comparison, the reviewer looks for the stated reasons behind the suitability decision and for the quality of the impression on which the conclusion rests.
Does the opposing report contain what the rules require?
In a federal civil case, such as a cargo seal dispute, a retained examiner's report must state all opinions with their basis, the facts or data considered, exhibits, qualifications, prior testimony and compensation.
Under Rule 26(a)(2)(B) the report of a retained expert contains "a complete statement of all opinions the witness will express and the basis and reasons for them," the facts or data considered, the exhibits, the witness's qualifications with publications from the previous 10 years, a list of cases with testimony in the previous 4 years and the compensation. A footwear or toolmark report that gives a conclusion without the comparison record does not state the basis of the opinion.
Under Rule 26(b)(4)(C) the facts, data and assumptions that counsel supplied to the opposing examiner can be discovered. In an impression case this covers, for example, what the examiner was told about the suspect before the comparison, which bears on the confirmation bias that the Rhode Island review described.
The reviewer lists each missing element separately from the scientific criticism. The court can order a missing element produced, while the scientific points go to admissibility or weight.
How do courts use error-rate evidence?
Maryland limited firearm toolmark testimony to "consistent" or "inconsistent"; New Jersey admitted full identifications and left bias and documentation to cross-examination. Error-rate figures from a review support either a motion or the cross-examination.
In Abruquah the Supreme Court of Maryland held, 4-3, that the methodology can support reliable conclusions that patterns and markings are "consistent or inconsistent" with those from a particular firearm, but not an unqualified opinion that the bullets were fired from the defendant's revolver. The court wrote that such a conclusion "is likely to lead much more directly to a conviction," and reversed for a new trial.
In People v. Ross (Sup Ct, Bronx County, 2020) the court limited the examiner to class characteristics. In State v. Caneiro (N.J. Super. Ct. Law Div., August 7, 2025) the court denied a motion to limit the expert to class characteristics or "cannot be excluded" language, and held that concerns about cognitive bias and documentation went to weight.
The reviewer gives counsel the figures for both settings: the study, the rate, the treatment of inconclusives and nonresponse, and the wording of the opposing conclusion set against the DOJ language. Which use the court permits is for the court.
What does IAE do when a party needs a review of a footwear or toolmark report?
IAE prepares a review of the opposing examiner's report or an expert report on the client's questions for criminal defense and cargo loss cases in US courts. Delivery takes up to 5 business days, or 48 hours on rush.
The questions cover whether a shoe, tire or tool could have left the mark, how strong the association is, and whether a lock or seal was defeated. Each question is tied to the ANSI/ASB or ISO standard for that kind of mark. For a footwear report the review checks whether the examiner analyzed the questioned print before the known shoe, documented class characteristics and characteristics of use, and recorded the conclusion. For a toolmark report it checks whether the case file records the verification and who performed it.
IAE confirms the questions with the client before work starts. The secretariat confirms receipt the same day, and within one business day IAE confirms the format, the cost, the deadline and the specialist from its registry; the client approves the candidate and the CV.
Every specialist is a member of the public IAE registry, verified by Member ID or QR code, and confirms in the document that he or she has no interest in the outcome. Materials go through the contact form or to info@iaexperts.com and are not passed to third parties without the client's consent. The court weighs a review with the other evidence, and neither IAE nor the examiner guarantees the outcome. IAE is not a law firm and does not advise on case strategy.
Sources
- Hicklin A. et al., Accuracy, reproducibility, and repeatability of forensic footwear examiner decisions, Forensic Science International 339 (2022) 111418 (Noblis)
- Abruquah v. State, No. 10, Sept. Term 2022 (Md. June 20, 2023)
- Khan K., Carriquiry A. L., Shining a Light on Forensic Black-Box Studies, arXiv:2209.14216 (2023)
- VanderPlas S., Khan K., Hofmann H., Carriquiry A. L., Firearms and Toolmark Error Rates (2022), University of Nebraska-Lincoln Digital Commons
- U.S. Department of Justice, Uniform Language for Testimony and Reports for the Forensic Footwear Discipline (effective August 15, 2020)
- Federal Rules of Evidence, Rule 702, Testimony by Expert Witnesses, with Committee Notes (LII)
- Fields v. Ashford, No. 17-cv-11812 (E.D. Mich. Nov. 5, 2019), ECF No. 88 (Justia)
- Federal Rules of Civil Procedure, Rule 26, Duty to Disclose; General Provisions Governing Discovery (LII)
- Ryman C. et al., On-Going Decision Analysis (a.k.a. Black Box) Studies at the FBI Laboratory, AAFS 2020, E92
- OSAC Footwear and Tire Subcommittee, Examiner Reliability Study: Black/White Box Study on Footwear and Tire Examiners, research need (NIST)
- OSAC Footwear and Tire Subcommittee, R1: Examiner Reliability Study: White Box Study on Footwear and Tire Examiners, research need (NIST)
- Rhode Island Current, report on the independent review of toolmark testing at the Rhode Island State Crime Laboratory (2024)
- ABA Journal, Expert's assertions about toolmark evidence lead judge to order new trial in pipe bombings, July 24, 2023
- People v. Ross, 68 Misc 3d 899, 2020 NY Slip Op 20153 (Sup Ct, Bronx County June 30, 2020)
- State v. Caneiro, Indictment No. 19-02-283 (N.J. Super. Ct. Law Div., Monmouth County Aug. 7, 2025), order and opinion
Read also
All articles
Can the prosecution use a toolmark or footwear report without calling the examiner who did the comparison?
A toolmark or footwear report rests on a comparison that a particular examiner made, and the case file can show who else made test marks, verified
Footwear and toolmark conclusion scales: what do identification, probably made and inconclusive mean in the examiner's report?
A footwear report can end with one of seven conclusions under the SWGTREAD standard of 2013 or one of six under the Department of Justice language
Contextual bias in footwear and toolmark examinations: what was the examiner told, and what does a reviewer check?
A footwear or toolmark examiner who knows that the suspect confessed, has a record or was linked by other evidence has received informationExpert report review
Ready to discuss your case?
Send the materials: the secretariat will confirm the format, cost, deadline and the specialist within one business day.

