Expert report review

Contextual bias in footwear and toolmark examinations: what was the examiner told, and what does a reviewer check?

A footwear or toolmark examiner who knows that the suspect confessed, has a record or was linked by other evidence has received information the comparison does not need. The National Commission on Forensic Science called such information task-irrelevant in 2015 and recommended that laboratories keep it from analysts. A review of the opposing report checks what the examiner was told, in what order the items were compared, and whether the verification was blind.

A gloved hand slides a case folder away from a plaster shoe cast and a microscope

What is contextual bias in a footwear or toolmark examination?

The risk that information about the case, rather than the marks, shapes the conclusion. The National Commission on Forensic Science separated task-relevant information, which the examiner should consider, from task-irrelevant information, which should not be considered.

The National Commission on Forensic Science, a federal advisory committee established by the Department of Justice, adopted "Ensuring That Forensic Analysis Is Based Upon Task-Relevant Information" on December 8, 2015, with 93 percent voting yes. It uses "task-relevant to describe information that FSSPs should consider when performing a particular task" and "task-irrelevant to describe information that should not be considered." FSSPs are forensic science service providers.

The document lists examples of task-irrelevant information in a fingerprint comparison: that "the suspect's criminal history (or lack thereof) is not necessary for drawing conclusions," "information that the suspect confessed to the crime," information that the suspect was implicated by other physical evidence at the scene, and information that another examiner had identified the suspect as the source of a print.

The document also names the risk directly: it describes the problem that arises "if analytic conclusions were influenced by whether the suspect made incriminating statements or had a criminal record." A footwear or toolmark comparison needs the questioned mark, the known shoe or tool, and information about how each was collected and made. The suspect's statements are not part of that list.

What did the National Commission on Forensic Science recommend?

That forensic science service providers rely solely on task-relevant information, specify which information is task-relevant for common tasks, and take steps to avoid exposing analysts to task-irrelevant information, for example through case managers and sequential unmasking.

The three views of the Commission are short. "FSSPs should rely solely on task-relevant information when performing forensic analyses." Providers "should specify what types of information are task-relevant and task-irrelevant for common forensic tasks." And "forensic laboratories should take appropriate steps to avoid exposing analysts to task-irrelevant information."

The document describes two methods. One uses "a case manager, who is fully informed about the facts of the underlying case" and passes to the analyst only what the task needs. The other is sequential unmasking, in which evidentiary samples from the scene are typed and interpreted before analysts see the reference samples.

The Commission also wrote that laboratory information management systems "can and should be programmed to facilitate context management" and that laboratories should monitor how new procedures work. A reviewer can ask whether the laboratory that produced the opposing report had any such procedure and whether it was followed in the case.

Does keeping case information from the examiner change the result?

Not in one 2024 study. It found no effect of blinding examiners from case information or of redacting task-irrelevant information, and expert fingerprint examiners did no better than students at comparing footwear impressions.

Michelle M. Pena, Stephanie Stoiloff, Maria Sparacino and Nadja Schreiber Compo published "The effects of cognitive bias, examiner expertise, and stimulus material on forensic evidence analysis" in the Journal of Forensic Sciences, volume 69, issue 5 (2024), pages 1740-1757. Expert fingerprint examiners and student participants received varying amounts of pre-comparison case information and compared matching and non-matching fingerprint and footwear impressions.

The abstract reports: "Results suggest no effects of blinding examiners from case information or redacting task-irrelevant information." Experts outperformed students on fingerprint comparisons, while "expert fingerprint examiners were no better than student participants at comparing footwear impression evidence."

A review uses both sources as they stand. The NCFS document gives the recommended practice; the 2024 study did not find a measurable effect in its setting. A review does not claim that the opposing examiner was biased; it records what information reached the examiner and whether the laboratory's own procedures were followed.

Why does the order of the comparison matter?

Because an examiner who documents the questioned mark before seeing the known shoe or tool commits to its features before any match can suggest them. Sequential unmasking, which NCFS described, follows the same order.

In sequential unmasking, as the NCFS document describes it, the evidentiary samples from the crime scene are typed and interpreted before analysts see the reference samples. Applied to a shoe print, the examiner records the class characteristics and wear of the questioned print first and only then examines the suspect's shoe.

The case file shows whether the order was kept. Bench notes with dates and times, photographs of the questioned impression marked before the comparison, and a separate record for the known item are what a reviewer looks for.

ASTM E3392-24, the guide for physical fit examination published in January 2024, describes a similar order for broken items: pieces may be assessed or reassembled before a questioned sample is compared with a possible known source (section 4.2).

What does verification add, and when is it blind?

A second examiner's independent review of the conclusion. ASTM E3392-24 rates blind verification, in which the verifier works without the first examiner's conclusion, above open verification, and a reviewer checks which kind was used and who performed it.

ASTM E3392-24 defines verification as an independent review by a second examiner (section 3.2.8) and rates blind verification above open verification in section 3.2.8.2. In an open verification the second examiner knows the first conclusion, and the same contextual information that reached the first examiner can reach the second.

The case file should name the verifier and the date and record what the verifier was given. A verification sheet with only a signature does not say whether the verifier knew the answer, knew about the suspect or saw the first examiner's annotated photographs.

For a toolmark report, the IAE review checks whether the case file records the verification and who performed it. Where the verifier worked in the same unit and under the same supervisor, the reviewer notes that as a fact for the court to weigh.

What does a reviewer ask for to see what the examiner knew?

The request for examination or submission form, the case notes and communications, the bench notes, and in a civil case the facts and assumptions counsel gave the expert, which Rule 26(b)(4)(C) leaves open to discovery.

The submission form or request letter tells the examiner what the case is about. If it names the suspect, says that he confessed or lists other evidence, the examiner received task-irrelevant information in the sense of the NCFS document. The reviewer asks for the form, any emails or call notes with investigators, and the laboratory's case record.

In a federal criminal case, Rule 16(a)(1)(G)(iii) of the Federal Rules of Criminal Procedure requires the government's expert disclosure to contain all opinions with "the bases and reasons for them" and the examiner's qualifications. A basis that includes other evidence in the case is a basis that the comparison did not need.

In a civil case over a seal or a lock, Rule 26(b)(4)(C) of the Federal Rules of Civil Procedure protects communications between counsel and a reporting expert, except to the extent they identify "facts or data that the party's attorney provided and that the expert considered" or "assumptions that the party's attorney provided and that the expert relied on." Those communications tell the reviewer what theory of the case the expert was given.

What in a report suggests that context entered the conclusion?

References to the suspect's statements, record or other evidence in the reasoning, conclusions stated with certainty the DOJ rules bar, and a description of the questioned mark written only after the known item was examined.

A comparison conclusion that cites other evidence, for example a footwear conclusion supported by the fact that the suspect was arrested nearby, rests in part on information the NCFS document calls task-irrelevant. The reviewer quotes the sentence and separates what the marks show from what the case file adds.

Wording is the second sign. The DOJ footwear ULTR bars claims that the known shoe made the impression to the exclusion of all other footwear and claims of absolute or 100 percent certainty. The 2023 committee note to Rule 702 of the Federal Rules of Evidence asks forensic experts to avoid assertions of absolute certainty when the method is subjective and potentially subject to error.

The third sign is sequence. Dates and times in the bench notes show whether the questioned mark was documented before the known item reached the bench. A review that finds the known item photographed first records that fact; it leaves to the court what weight to give it.

What does a UK police-funded research project propose for contextual information?

A University College London project funded by a College of Policing bursary, due for completion in December 2026, develops footwear evaluation methods in which a model, not the practitioner, handles contextual information.

The project "Investigation and development of interpretative methods and data for the evaluation of forensic footwear examination" aims to "mitigate the risk of cognitive bias" by giving the evaluation of contextual information to a model instead of the practitioner. Adding other variables to the model is expected to reduce confirmation bias and avoid double-counting.

The project cites misinterpretation of forensic evidence and failure to communicate it as key issues, drawing on miscarriages of justice. Its approach follows the same idea as the NCFS document: the person comparing the marks should not be the person weighing the rest of the case.

For a reviewer in a US or Russian case, these documents give published reference points. The review compares the opposing laboratory's practice with them and states the differences as facts.

What does IAE do when a party needs an impression or toolmark report reviewed for bias?

IAE reviews the opposing examiner's report and case file for US criminal defense and cargo loss cases, including the record of verification and who performed it. The review takes up to 5 business days, or 48 hours on rush.

The review checks the case file against the standard for that kind of mark. For a footwear report it checks whether the examiner analyzed the questioned print before the known shoe, documented class characteristics and characteristics of use, and recorded the conclusion, as ANSI/ASB Standard 137 (2023) requires. For a toolmark report it checks whether the case file records the verification and who performed it. Each point is linked to a page of the report or the bench notes and to a standard, a DOJ rule or a ruling.

IAE confirms the questions with the client before work starts. The secretariat confirms receipt the same day, and IAE then confirms the format, the cost, the deadline and the specialist from its registry. The client approves the candidate and the CV before work starts.

Every specialist is a member of the public IAE registry, verified by Member ID or QR code, and confirms in the document that he or she has no interest in the outcome. Materials are not passed to third parties without the client's consent. The court weighs a review with the other evidence, and neither IAE nor the reviewer guarantees the outcome. IAE is not a law firm and does not advise on case strategy.

Sources

Read also

All articles

Expert report review

Ready to discuss your case?

Send the materials: the secretariat will confirm the format, cost, deadline and the specialist within one business day.

Expertise request

Received

Cost and timeline estimate

WorkReview of an opinion
TimelineUp to 5 business days
ExpertIAE registry member