Expert report review

Was the opposing audio examiner told what to find, and how does a reviewer check for contextual bias?

An audio examiner who knows that the suspect confessed, or that other evidence points to him, is working with information that the examination does not need. The National Commission on Forensic Science called such information task-irrelevant in 2015 and recommended that examiners rely only on task-relevant information. A reviewer checks what the opposing examiner was told, when, and whether it shows in the report.

A reviewer from behind slides a sealed case folder away from headphones and a small recorder

What is contextual bias in a forensic audio examination?

The risk that information unrelated to the recording, such as a confession, a criminal record or other evidence against a suspect, shapes the examiner's reading of the audio. The 2015 NCFS views call such information task-irrelevant.

The National Commission on Forensic Science, a federal advisory committee established by the Department of Justice, adopted its views "Ensuring That Forensic Analysis Is Based Upon Task-Relevant Information" on December 8, 2015, by a vote of 93 percent yes, 7 percent no and no abstentions. The document was prepared by the Human Factors subcommittee and states that it does not necessarily represent the views of the Department of Justice.

The document draws one line. It "uses the term task-relevant to describe information that FSSPs should consider when performing a particular task" and "task-irrelevant to describe information that should not be considered." FSSP stands for forensic science service provider. The test is formal: "Information is task-irrelevant if it is not necessary for drawing conclusions about the propositions in question".

The examples in the document come from fingerprint work, but they translate directly. "Information that the suspect confessed to the crime" is task-irrelevant. So is a suspect's criminal history, which "(or lack thereof) is not necessary for drawing conclusions from the physical evidence", and information that the suspect was implicated by other evidence such as DNA.

What did the National Commission on Forensic Science recommend to laboratories about case information?

That forensic science service providers rely solely on task-relevant information, adopt written context management policies, and use measures such as case managers and sequential unmasking to keep analysts away from task-irrelevant information.

The central recommendation reads: "FSSPs should rely solely on task-relevant information when performing forensic analyses." The Commission asked that standards "should specify what types of information are task-relevant and task-irrelevant for common forensic tasks", a task it assigned to the Organization of Scientific Area Committees.

Laboratories should "take appropriate steps to avoid exposing analysts to task-irrelevant information" through written context management policies. One measure is a case manager: "The case manager decides what information is task-relevant and task-irrelevant for each task to be performed", and passes on only the first kind. Another is sequential unmasking, which the document describes for DNA testing: evidentiary samples from the crime scene are typed and interpreted before analysts learn the reference profiles. Laboratory information systems "can and should be programmed to facilitate context management", for example by restricting access to witness statements and police reports.

The recommendation is addressed to forensic science service providers in general. Its worked examples come from latent print examination, and the studies it discusses include bloodstain pattern analysis, forensic anthropology, bite marks and DNA.

Which information does an audio examiner need, and which is irrelevant?

The examiner needs the questions, the file and its history, and how and on what device the recording is said to have been made. Confessions, criminal records, other evidence and counsel's theory of the case are not needed.

Applying the Commission's test to audio means asking, for each piece of information, whether it is necessary for a conclusion about the recording. The SWGDE Best Practices for Digital Audio Authentication, version 1.3 of September 20, 2018, list some of what is necessary: section 3.1 asks for the date and time the recording was purportedly made, and section 4.7 provides for test recordings on the claimed recorder or on one of the same manufacturer and model. Without that information the examiner cannot test the claim.

For a voice comparison the examiner needs the questioned recording, the known recordings and the conditions under which each was made. The examiner does not need to know that the detective is certain of the speaker, that the speaker has prior convictions or that a witness identified the voice.

For an authenticity question the examiner needs to know what the proponent says happened: who recorded, on what, whether the device was paused. The examiner does not need to know which result would help the client.

How can you find out what the opposing examiner was told?

From the report's list of facts or data considered, and through discovery of attorney-expert communications that identify facts, data or assumptions provided by counsel, which Rule 26(b)(4)(C) leaves unprotected.

Rule 26(a)(2)(B)(ii) of the Federal Rules of Civil Procedure requires a retained expert's report to list "the facts or data considered by the witness in forming them". If the opposing report lists a police report, a deposition transcript or a witness statement among the materials considered, the examiner read them.

Rule 26(b)(4)(C) protects communications between the attorney and the expert, "except to the extent that the communications" relate to compensation, "identify facts or data that the party's attorney provided and that the expert considered in forming the opinions to be expressed", or "identify assumptions that the party's attorney provided and that the expert relied on in forming the opinions to be expressed." An engagement letter that says "the voice is believed to be the defendant's" falls under the third exception.

Under Rule 26(b)(4)(A) the examiner may be deposed after the report is provided. The deposition is the place to ask what the examiner knew about the case before listening to the recording and in what order the work was done.

What does a reviewer look for in the opposing report?

Case background that the analysis did not need, references to confessions or other evidence, the order in which questioned and known recordings were analyzed, and conclusions that track the case theory more closely than the data.

Many reports open with a summary of the case. A reviewer reads that summary with the Commission's test in mind: does the report mention that the defendant confessed, that a witness recognized the voice, that the police believe the recording is genuine? Each of those facts shows what the examiner knew while working.

The order of the work matters as well. The Commission described sequential unmasking for DNA testing, where crime scene samples are interpreted before analysts learn the reference profiles. For a voice comparison the analogous question is whether the examiner described and measured the questioned recording before listening to the known recordings of the suspect, and whether the report records that order.

Finally the reviewer compares the conclusion with the measurements. SWGDE section 5.2 warns that conclusions "should not be over or understated." A conclusion that is firmer than the documented data allow, in a report that also recites the case against the defendant, gives the reviewer a concrete point to raise.

Does exposure to case information make the opinion inadmissible?

Not by itself. The NCFS views are recommendations, not rules of evidence. Exposure goes to whether the method was reliably applied under Rule 702 and to the weight the jury gives the opinion.

Rule 702 of the Federal Rules of Evidence asks whether the testimony is based on sufficient facts or data, is the product of reliable principles and methods, and whether the opinion is a reliable application of those methods to the facts of the case. An examiner who read a confession before comparing voices has not necessarily applied the method badly. The question is what the documentation shows.

If the report documents measurements that another examiner can repeat, exposure to case information is a matter for cross-examination. The 2000 committee note to Rule 702 quotes Daubert on the point: "Vigorous cross-examination, presentation of contrary evidence, and careful instruction on the burden of proof are the traditional and appropriate means of attacking shaky but admissible evidence."

If the report offers little more than the examiner's impression, and the examiner knew the case against the defendant, the reviewer can argue that the opinion rests on the context rather than on the data, which is a challenge to reliable application under Rule 702(d).

Which questions about context should the opposing examiner be asked?

What the examiner was told about the case and by whom, what materials beyond the recordings were received, in what order the work was done, and whether the conclusion would change if that information were false.

The questions follow the Commission's distinction. What did counsel or the investigator tell you about the case before you started? Did you receive police reports, statements or transcripts prepared by one side? Were you told that the speaker had been identified, or that the recording had been accepted as genuine by anyone?

Then the order of the work. Did you analyze the questioned recording before you listened to the known recordings? Did you record your observations on the questioned recording before the comparison? Were you given known recordings of more than one person, or only of the defendant?

Then the effect. Which of your measurements would be the same if the defendant had not confessed? Which part of your conclusion rests on information other than the recordings? The answers go to the rebuttal report and, if the record supports it, to a Rule 702 motion.

What should your own examiner receive?

The questions, the files and their history, and information about how the recordings were made. Your own examiner is held to the same standard, and the same discovery rules apply to what counsel provides.

The Commission's recommendation applies to every forensic science service provider, including the one a party retains. Counsel who send their examiner a summary of the case theory hand the other side a cross-examination, because under Rule 26(b)(4)(C) the facts, data and assumptions counsel provided are discoverable.

A clean engagement letter states the questions, lists the files with their sources and hash values, describes how the recordings are said to have been made and asks the examiner to record what was received and when. Background beyond that can be supplied later, after the measurements are documented, if the examiner needs it.

What does IAE do when the opposing audio examiner may have been influenced by case information?

An IAE examiner reviews the opposing audio report: which file was received, which tests were run and documented, whether the exemplars were comparable and whether the conclusion stays within the method. Up to 5 business days.

The IAE review of the opposing forensic audio report is for a party facing the other side's examiner on whether a recording was edited, whose voice is on it or what was said. The reviewer checks which file the examiner received, which tests were run and documented, whether the voice exemplars were comparable and whether the conclusion stays within the method. Each point in the review is tied to a page of the opposing report and to a rule, a standard or a court decision.

The document can take the form of a review of the opposing report, a rebuttal report or a statement of grounds for a Rule 702 motion. The client sends the opposing report and the case materials and describes what to check. The secretariat confirms receipt the same day, and within one business day IAE confirms the format, the cost, the deadline and the specialist from its registry. The client approves the candidate and the CV before work starts. The signed PDF is issued in up to 5 business days after the format and cost are confirmed, and a 48-hour rush is available for tight deadlines.

Specialists are members of the public IAE registry, and the Member ID or QR code in the document opens the specialist's record. The specialist confirms in the document that he or she has no interest in the outcome. A review is evidence the court weighs with the other materials; neither IAE nor the reviewer guarantees the outcome, and IAE is not a law firm.

Sources

Read also

All articles

Expert report review

Ready to discuss your case?

Send the materials: the secretariat will confirm the format, cost, deadline and the specialist within one business day.

Expertise request

Received

Cost and timeline estimate

WorkReview of an opinion
TimelineUp to 5 business days
ExpertIAE registry member