Expert report review
Cross-examining a toolmark or footwear examiner: which questions come from the review?
A cross-examination of a toolmark or footwear examiner tests what the report leaves out: what the examiner recorded, how the test marks were made, who checked the conclusion and which words the examiner may use. The questions come from a review of the case file against the draft OSAC toolmark standard, the Department of Justice rules and the Federal Rules of Evidence. Courts in Virginia and Maryland left these points to cross-examination, and Maryland's highest court barred unqualified source opinions.

Why do courts send criticism of toolmark examiners to cross-examination?
Because courts usually admit the comparison and treat the remaining criticism as a question of weight. A federal court in Virginia in 2024 and a Maryland trial court in 2023 called that criticism material for cross-examination, not exclusion.
In United States v. Graham, decided on February 20, 2024, Judge Thomas T. Cullen of the Western District of Virginia ruled on a defense motion to exclude Laura Hollenbeck, a forensic scientist with the Virginia Department of Forensic Science who had compared a cartridge case found at the store with cases test-fired from a recovered pistol. The defense argued that error rates in the discipline are unknown and vary widely, that validation studies use closed sets and count only false positives, that the AFTE Journal is not double-blind, and that her report did not say which marks drove her conclusion.
The court declined to exclude the opinion. It called the remaining skepticism "proper fodder not for the outright exclusion of evidence on Daubert grounds" and left it for cross-examination at trial. It also quoted the Crisp decision for the point that "an able defense lawyer will bring that fact to the jury's attention, both through skillful cross-examination" and through expert testimony of the defense's own.
Eight months earlier, on June 14, 2023, Judge Kathleen L. Beckstead of the Circuit Court for Wicomico County, Maryland, denied a defense motion in limine in State v. Delonte Bryant. The examiner, Susan Kim, was allowed to testify within limits the State had agreed to and under the Department of Justice uniform language. The court weighed the Rochkind factors and found that every factor it treated as relevant favored admission except one: standards and controls. It held that the defense criticisms "go[] to the weight, not the admissibility of the testimony" and called the testimony "proper fodder for crossexamination rather than exclusion." A footnote noted that Abruquah v. State was pending before the Supreme Court of Maryland. That court decided it on June 20, 2023: by 4 votes to 3 it held that a firearms examiner should not have been allowed to give an unqualified opinion that the bullets were fired from the defendant's revolver, said the method can support testimony that markings are "consistent or inconsistent" with those on bullets fired from a particular firearm, and reversed for a new trial.
Federal Rule of Evidence 702 draws the same line. The proponent must show the court that it is more likely than not that the expert's knowledge will help the jury, that the testimony is based on sufficient facts or data and that the method is reliable and reliably applied. The committee note to the 2023 amendment says the amendment "clarifies that the preponderance standard applies to the three reliability-based requirements added in 2000." Testimony that passes that test goes to the jury, and the jury learns its limits on cross-examination.
What should the cross-examiner have from the examiner's file?
The notes, worksheets and photographs behind every observation. Under the draft OSAC toolmark standard another examiner must be able to follow the analysis from the record alone, and a conclusion without supporting documentation is insufficient.
OSAC Proposed Standard 2024-S-0002, Standard Test Method for the Examination and Comparison of Toolmarks for Source Attribution, is a draft of the Firearms and Toolmarks Subcommittee of the Organization of Scientific Area Committees, marked version 1.0 and dated November 2023. Its disclaimer says it will be submitted to a standards developing organization and is subject to change. Its scope covers the microscopic evaluation, classification and comparison of toolmarks and "the minimum requirements for supporting documentation."
Section 4.7 of the draft requires depictions or descriptions of the observations to the extent that another examiner, "without the benefit of the specimens themselves, can review the case record and understand what analysis was conducted" and the basis for any conclusions. Section 4.7.4.2 states: "A statement of conclusion alone, without supporting documentation, is insufficient." Under section 4.7.5.1, an identification is documented with photographs of the toolmark agreement in all areas of the specimens used for the conclusion.
Laboratory procedures say where those records are kept. The FBI Laboratory's procedure FTD-241-00, Toolmark and Fracture Examinations, issued on March 3, 2023, provides that observations of a questioned toolmark "will be recorded on the FTD Worksheet" and that a change in the classification of a mark is documented on the same worksheet. With the worksheet in hand, counsel can compare each sentence of the testimony with what was written at the bench.
Federal Rule of Evidence 705 lets an expert state an opinion and the reasons for it "without first testifying to the underlying facts or data," unless the court orders otherwise. The rule continues: "But the expert may be required to disclose those facts or data on cross-examination." For a toolmark opinion the underlying facts are the observations in the notes and the photographs, so the review lists, page by page, which statement in the report has no record behind it.
A thin file does not by itself remove the opinion. In Bryant the defense argued that the examiner had not documented subclass and individual characteristics. The court answered that the defense "falsely equates a failure to document certain observations with a failure to make those observations" and accepted her account of her process. After such a ruling the question moves to the jury: was an observation that left no trace in the file made at all? The examiner has to answer it on the stand.
In a civil case about a cut seal or a forced lock, Rule 26(a)(2)(B) of the Federal Rules of Civil Procedure requires a retained expert's report to contain all opinions with their basis and reasons, "the facts or data considered by the witness in forming them" and the exhibits that will support them. Communications between the attorney and the expert are protected, with three exceptions in Rule 26(b)(4)(C): compensation, facts or data the attorney provided and the expert considered, and assumptions the attorney provided and the expert relied on.
How does a review become a cross-examination outline?
The reviewer follows the examiner's method step by step and records, for each step, what the file contains and what it lacks. Each gap becomes a short question that the examiner must answer from his or her own file.
The Association of Firearm and Tool Mark Examiners describes the examination in four phases: evaluation, comparison, conclusion and verification. In evaluation the examiner checks whether the specimens are suitable for examination and whether the observed class characteristics are the same. In comparison a comparison microscope is used to decide whether marks are subclass or individual characteristics and how well the individual characteristics correspond. A conclusion follows, and verification follows the laboratory's quality assurance policy, under which a mechanism should be in place to determine which cases require it.
A review of the report uses the same order. For each phase it sets out three things: what the report states, what the notes and photographs record, and what the standard requires. Where all three agree, there is nothing to ask. Where the report states more than the notes record, counsel has a question with a document behind it.
Questions built this way are narrow. The examiner is asked to point to the worksheet page where a characteristic was recorded, to the photograph that shows it, or to the policy that required or did not require verification. Each finding in the review cites the page of the file and the clause of the standard, so that counsel can put both in front of the witness at the same time.
The review also separates questions for a Rule 702 hearing from questions for trial. Gaps that go to whether the method was applied at all belong in the motion. Gaps that go to how much the conclusion is worth stay for the jury, and Graham and Bryant put most criticism of established comparison methods in that second group.
Which questions test class and subclass characteristics?
Questions about which marks the examiner treated as individual and why they could not be subclass marks. The draft OSAC standard bars a same-source opinion based solely on agreement of potential subclass characteristics.
The draft OSAC standard defines three kinds of features. Class characteristics are "observable features of a specimen which indicate a restricted group source" and result from design and manufacturing decisions. Subclass characteristics are "toolmarks produced by a single tool that repeat with little, if any, change on a limited series of sequentially manufactured items." Individual characteristics are "marks produced by the random imperfections or irregularities of tool surfaces."
Two clauses of the draft turn the definitions into limits. Section 4.2.3.3: "An opinion of the same source shall not be based solely on agreement of potential subclass characteristics." Section 4.4.2: "Toolmarks that are believed to be subclass characteristics shall not be used for identification conclusions." The archived National Institute of Justice training page on intercomparison lists among possible reasons for an inconclusive result that "numerous subclass characteristics may be present."
The AFTE paper by R. Grzybowski and five co-authors, Firearm/Toolmark Identification: Passing the Reliability Test Under Federal and State Evidentiary Standards, describes the consecutive matching striae criteria used by some examiners. For three-dimensional toolmarks, an identification under these criteria needs at least two groups of at least three consecutive matching striae in the same relative position, or one group of six. For two-dimensional toolmarks the thresholds are two groups of at least five, or one group of eight. The authors add that "the possibility of subclass characteristics must be ruled out" before the criteria apply.
From these texts the review draws the questions. Which marks did the examiner treat as individual? How was the tool made, and could that process leave marks that repeat on a run of tools? Did the examiner look at other tools from the same production or rely on general knowledge? Where in the file is the answer recorded? If the examiner used a striae count, how many consecutive striae agreed, and in which photograph can the jury see them?
What should the examiner be asked about the test marks?
What material the test marks were made in, how many were made, whether they were compared with each other before the comparison with the questioned mark, and where they are now. The draft OSAC standard addresses each point.
A toolmark examiner compares the questioned mark with marks the laboratory makes with the suspect tool. The Missouri State Highway Patrol crime laboratory describes it this way: "Minute scratches (striations) produced by tests made in the crime laboratory by the questioned tool, will be compared." The same page lists bolt cutters, screwdrivers, scissors, knives and pliers among submitted tools, and wire, sheet metal, chains and safety-deposit boxes among marked objects.
The draft OSAC standard sets rules for these exemplars. Section 4.3.1 tells the examiner to "select the appropriate test material, considering the relative hardness of the tool working surface and the test material," to label exemplars under laboratory policy and, if possible, to mark the physical specimens with a unique identifier. The same section requires that exemplars "shall be preserved," whether kept in the laboratory, returned to the submitting agency or held some other way. The FBI procedure adds that exemplars may include toolmarks produced by a known tool and casts taken from a marked item or fractured surface.
Section 4.4.2 of the draft requires that exemplars made by the suspected tool be intercompared under the microscope "to determine which markings reliably reproduce with sufficient detail for comparison to questioned toolmarks." A mark that does not repeat across the test marks cannot carry an identification. The National Institute of Justice training page on intercomparison, which covers comparing evidence marks with each other, says at step 8.1: "If the class characteristics are different, the examination should end."
The questions follow. What was the test material, and how did its hardness compare with the door frame, wire or seal? How many test marks were made, at what angles and with what force? Were they compared with each other first, and which marks repeated? Where are the test marks now, and has the defense been able to examine them? Each answer either matches the file or exposes a gap.
How is the examiner asked about the differences he or she saw?
By asking which differences the examiner saw and how each was explained. The draft OSAC standard requires examiners to consider all similarities and differences before a conclusion, and the FBI procedure names conditions that can prevent a source conclusion.
Section 4.4.3 of the draft OSAC standard reads: "Examiners shall consider all similarities and differences observed prior to reaching a conclusion." A report that lists only points of agreement does not show that the second half of that sentence was done. The review asks for the notes on differences and for the reason each difference was set aside.
The FBI procedure FTD-241-00 names the reasons a mark may not support a conclusion. Section 8.1 says that because of variation in the marked material, tool wear and similar factors, "it may not be possible for an examiner to reach a source conclusion." Section 8.2 says fracture marks are not always identifiable because of variation in the material, corrosion and abuse.
AFTE's summary of the method gives three outcomes. An identification is rendered when individual agreement is sufficient. An inconclusive determination is rendered when class characteristics agree but individual agreement is insufficient. An elimination may be rendered in exceptional situations, based on differences in individual characteristics. The Department of Justice footwear rules use a longer scale of six conclusions, including "support for exclusion," defined as a conclusion that the known item "probably did not make the questioned impression."
In Graham the court said that an inconclusive result "is not an error in the sense that it falsely attributes a cartridge or casing to the wrong firearm." For cross-examination the point runs the other way: when the examiner saw differences and still reported an identification, the jury is entitled to hear what they were and why they did not matter.
What should the examiner be asked about verification?
Who verified the conclusion, whether the verifier knew the first result, and where the verification is recorded. AFTE leaves the choice of cases that require verification to each laboratory's quality assurance policy.
Verification is the fourth phase in AFTE's summary of the method, and AFTE says it follows the laboratory's quality assurance policy, under which a mechanism should be in place to determine which cases require it. The first questions are therefore about the policy: does the laboratory verify every identification, or only some, and did this case fall within the policy?
The AFTE paper by Grzybowski and co-authors describes "internal quality control procedures of peer and administrative review" as a further check on error, and states that once a laboratory, "following technical peer review, issues a report, the results are assumed to be correct." The same paper says that most such cases "are never critically evaluated by anyone else." A cross-examiner can ask whether the verifier examined the specimens or only read the notes, and whether the verifier knew the first examiner's answer.
The court in Graham put a number on the stakes. It estimated the potential error rate for an examiner like Ms. Hollenbeck at about 1 percent, and at roughly 0.01 percent with independent verification. The second figure assumes that the verification was independent. If the verifier saw the first conclusion before looking, the case falls outside that assumption, and counsel can ask the examiner to say so.
The review checks the record for the verifier's name, the date, the items examined and the result. If any of these is absent, the question is put to the examiner in that form: where in the file is the verification, and what did the verifier look at?
Which words is the examiner not allowed to use?
Department of Justice examiners may not claim uniqueness, exclusion of all other sources, a zero error rate or 100 percent certainty. Courts in Virginia and Maryland imposed similar limits on state laboratory examiners.
The Department of Justice Uniform Language for Testimony and Reports for the forensic footwear discipline took effect on August 15, 2020, and applies to Department of Justice examiners who prepare reports and testify about footwear examination. It lists six conclusions: source identification, inclusion based on class and randomly acquired characteristics, inclusion based on class characteristics, inconclusive, support for exclusion and source exclusion. A source identification is the examiner's conclusion "that the known footwear item made the questioned impression."
The same document states that a conclusion "is not based on a statistically-derived or verified measurement or comparison to all other footwear items." An examiner therefore shall not assert that a source conclusion rests on the uniqueness of an item, shall not use the terms individualize or individualization, and shall not assert that the known shoe made the impression to the exclusion of all other footwear. The examiner may not assert that footwear examinations are infallible or have a zero error rate, may not cite the number of examinations performed in a career as a direct measure of accuracy, may not give a statistic or numerical degree of probability unless it rests on relevant and appropriate data, and may not claim absolute or 100 percent certainty or use expressions such as reasonable degree of scientific certainty unless required to do so by a judge or applicable law.
Both inclusion conclusions come with a required statement. The examiner must explain "that there may be other footwear items with characteristics that are indistinguishable from the known footwear item that could have also made the questioned impression." If the report gives an inclusion without that sentence, the question writes itself.
For firearms and toolmarks the Graham court applied the department's uniform language as the government stipulated. The examiner could not assert that two toolmarks originated from the same source "to the exclusion of all other sources," that examinations in the discipline "are infallible or have a zero error rate," or that the marks share a source "with absolute or 100% certainty," and a footnote barred "reasonable degree of scientific certainty." The court refused the defense's alternative request to limit her to saying the patterns appear similar. Under the limits accepted in Bryant, the examiner "will not use terms such as match" and will not state her opinion with any level of statistical certainty.
The footwear rules say they do not create rights enforceable by any party and should not be read to imply that different terminology used by non-department laboratories is erroneous. A state examiner may therefore follow a different policy. The first question is which language policy the laboratory follows, and the second is whether the report or the testimony stayed inside it.
How are error rates raised on cross-examination?
Through studies and publications put to the examiner. Rule 803(18) lets counsel read a statement from a reliable treatise or periodical to an expert on cross-examination, and the examiner's own professional literature contains the figures.
Rule 803(18) of the Federal Rules of Evidence covers a statement in a treatise, periodical or pamphlet if it is "called to the attention of an expert witness on cross-examination" and the publication is established as a reliable authority by the expert's admission or testimony, by another expert's testimony or by judicial notice. If admitted, the statement "may be read into evidence but not received as an exhibit." The cross-examiner first asks whether the examiner accepts the publication as reliable, then reads the passage.
The AFTE paper by Grzybowski and co-authors is a publication an examiner is likely to accept. It states that "error rates associated with normal day-to-day forensic firearm/toolmark casework are impossible to establish" and that proficiency tests "were never intended to be used to calculate a national error rate." It reports that earlier figures by Peterson and Markham, 12 percent for firearms and 26 percent for toolmarks, counted inconclusive responses as incorrect.
The Graham court reviewed study figures for firearms comparisons. It said there is no error rate for toolmark identification as a whole, only for particular validation studies, and that "the focal point of the inquiry should be on the rate of false positives." It cited a false-positive rate of 1.01 percent in the Ames I study, which PCAST listed as 1.5 percent, 0.56 percent in the Smith study and a rate well below 1 percent in Ames II. It also agreed that closed-set designs may "artificially deflate the false-positive error rate." The court noted that Ms. Hollenbeck takes annual proficiency tests.
The questions follow from those passages. Which studies cover the kind of tool and material in this case? Were they open-set or closed-set? How were inconclusive answers counted? What proficiency tests has the examiner taken, who made them, and were they designed to measure error? The footwear uniform language adds one more: the examiner may not offer the number of examinations in a career as a direct measure of accuracy, so an answer of that kind can be met with the rule itself.
What happens if overstated testimony draws no objection?
The appellate court reviews it only for plain error. In Williams v. United States in 2016 the District of Columbia Court of Appeals affirmed convictions after an examiner testified without objection that markings were unique to one gun.
In Williams v. United States, No. 13-CF-1312, decided on January 21, 2016, a firearms examiner was asked whether he had any doubt and answered "[n]o sir." He testified: "Those markings are unique to that gun and that gun only," and "Item Number 58 fired these three bullets." He described the markings as "similar to a fingerprint, basically." Defense counsel did not object.
Without an objection the court asked only whether the error was plain. It found none, because "there is no precedent in this jurisdiction that limits a toolmark and firearms examiner's testimony," and it held that "the government's internal policy does not constitute binding law." The court affirmed the judgment, except that it remanded with instructions to vacate the attempted robbery conviction and the related weapons count, which merge into the felony murder conviction.
Judge Easterly, who also wrote the opinion for the court, concurred. She wrote that the law in the District "does not clearly preclude" such statements, that a certainty statement about toolmark pattern matching "has the same probative value as the vision of a psychic," and that the courts "must bar the admission of these certainty statements." The concurrence did not change the result for Mr. Williams.
The record is made before and during trial. A review delivered before trial gives counsel the specific terms in the report that exceed the uniform language, so that a motion to limit the testimony or an objection can be made when the words are spoken.
Can the defense call its own examiner after cross-examination?
Yes. Graham quoted the principle that forensic evidence is tested through cross-examination and through expert testimony of the defense's own. In federal civil cases a rebuttal report is due within 30 days after the other side's disclosure, absent another order.
The passage the Graham court quoted names two tools: skillful cross-examination and "presenting expert testimony of his own." Cross-examination puts the gaps in the government's file to its examiner. A defense examiner can tell the jury what the draft OSAC standard and the uniform language require and compare that with the file.
In civil cases Rule 26(a)(2)(D)(ii) of the Federal Rules of Civil Procedure sets the default time for a rebuttal report: within 30 days after the other party's disclosure, if the evidence is intended solely to contradict or rebut evidence on the same subject matter. The rebuttal expert's report has the same six parts as any retained expert's report: opinions with basis and reasons, facts or data considered, exhibits, qualifications with publications from the previous 10 years, prior testimony from the previous 4 years and compensation.
A specialist who reviews the file only to prepare counsel's cross-examination may stay a consulting expert. Under Rule 26(b)(4)(D) a party ordinarily may not discover the facts known or opinions held by an expert retained in anticipation of litigation who is not expected to testify, except under Rule 35(b) or on a showing of exceptional circumstances. Whether to name the reviewer as a witness is a decision for counsel.
What does IAE do when counsel prepares to cross-examine a toolmark or footwear examiner?
IAE reviews the opposing examiner's report for criminal defense and cargo loss cases in US courts, including whether the verification is recorded and who performed it. Delivery takes up to 5 business days, or 48 hours on rush.
The review checks the case file against the standard for that kind of mark. For a footwear report it checks whether the examiner analyzed the questioned print before the known shoe, documented class characteristics and characteristics of use, and recorded the conclusion. Toolmarks on lock cylinders and seals are compared with test marks made by the suspect tool, and for a toolmark report the review checks whether the case file records the verification and who performed it. The questions the review answers are whether a shoe, tire or tool could have left the mark, how strong the association is, and whether a lock or seal was defeated.
IAE confirms the questions with the client before work starts. The secretariat confirms receipt of the materials the same day, and within one business day IAE confirms the format, the cost, the deadline and the specialist from its registry. The client approves the candidate and the CV before work begins. The signed PDF is issued within up to 5 business days.
Every specialist is a member of the public IAE registry and can be verified by Member ID or QR code, and confirms in the document that he or she has no interest in the outcome. Materials are sent through the contact form or by email and are not passed to third parties without the client's consent. The court weighs a review with the other evidence, and neither IAE nor the reviewer guarantees the outcome. IAE is not a law firm and does not advise on case strategy; legal questions are for the client's attorney.
Sources
- OSAC Proposed Standard 2024-S-0002, Standard Test Method for the Examination and Comparison of Toolmarks for Source Attribution, draft version 1.0, November 2023 (NIST)
- U.S. Department of Justice, Uniform Language for Testimony and Reports for the Forensic Footwear Discipline, effective August 15, 2020
- Federal Rules of Evidence, Rule 702, with committee note to the 2023 amendment (LII)
- Federal Rules of Evidence, Rule 705 (LII)
- Federal Rules of Evidence, Rule 803(18), statements in learned treatises, periodicals or pamphlets (LII)
- Federal Rules of Civil Procedure, Rule 26 (LII)
- FBI Laboratory, Toolmark and Fracture Examinations, FTD-241-00, issued March 3, 2023
- Association of Firearm and Tool Mark Examiners, Summary of the Examination Method
- Grzybowski R. et al., Firearm/Toolmark Identification: Passing the Reliability Test Under Federal and State Evidentiary Standards (AFTE)
- National Institute of Justice, Firearms Examiner Training, Module 13, Intercomparison (archived)
- Missouri State Highway Patrol Crime Laboratory, Toolmarks
- United States v. Graham, No. 4:23-cr-00006 (W.D. Va. Feb. 20, 2024), memorandum opinion (copy posted by AFTE)
- State of Maryland v. Delonte Bryant, No. 22-K-15-000572 (Cir. Ct. Wicomico County June 14, 2023), opinion (copy posted by AFTE)
- Williams v. United States, No. 13-CF-1312 (D.C. Jan. 21, 2016), District of Columbia Court of Appeals
- Abruquah v. State, No. 10, September Term 2022 (Md. June 20, 2023), Supreme Court of Maryland
Read also
All articles
Can the prosecution use a toolmark or footwear report without calling the examiner who did the comparison?
A toolmark or footwear report rests on a comparison that a particular examiner made, and the case file can show who else made test marks, verified
Footwear and toolmark conclusion scales: what do identification, probably made and inconclusive mean in the examiner's report?
A footwear report can end with one of seven conclusions under the SWGTREAD standard of 2013 or one of six under the Department of Justice language
Contextual bias in footwear and toolmark examinations: what was the examiner told, and what does a reviewer check?
A footwear or toolmark examiner who knows that the suspect confessed, has a record or was linked by other evidence has received informationExpert report review
Ready to discuss your case?
Send the materials: the secretariat will confirm the format, cost, deadline and the specialist within one business day.

