Expert report review

Contextual bias in handwriting examination: what was the examiner told, and what does a reviewer check?

Contextual bias is the effect of case information that a handwriting comparison does not need, such as a confession, other evidence or the client's theory, on how the examiner reads a signature. The NIST expert working group on human factors in handwriting examination says laboratories must use a documented protocol for managing such information. A reviewer checks what the examiner was told, the order of examination and whether a second examiner verified the result blind.

A hand slides a face-down case file away from a signature sheet and a desk loupe

What is contextual bias in handwriting examination?

The influence of task-irrelevant information on the examiner's observations or conclusion. Task-irrelevant information is anything not pertinent to the material or the question being examined, such as other evidence against a suspect.

The definitions come from NIST Interagency Report 8282r1, "Forensic Handwriting Examination and Human Factors: Improving the Practice Through a Systems Approach", prepared by the Expert Working Group for Human Factors in Handwriting Examination and updated in May 2021. Its glossary defines relevant information as information "pertinent and applicable to the subject, material, or question being considered."

Contextual information management is defined as actions that optimize the flow of information to and from a forensic expert so that the potential for contextual bias is minimized. Blinding is "Systematically shielding an examiner from task-irrelevant contextual information."

Handwriting is exposed to the problem in a particular way. The questioned document itself carries text: the amount of a check, the beneficiary of a will, the words of a note. Kunii, whose proposal is described below, points out that documents carry this kind of information themselves, in their printed content and in the context of the writing.

Is there evidence that context changes handwriting conclusions?

Yes, though the research base is small. Experiments with lay people and with document examination trainees found that biasing context changed decisions, and the NIST report calls research on handwriting examination itself limited.

NIST IR 8282r1 states that there is "currently limited research about this issue's impact on handwriting examination specifically." Its notes say that early work on the issue, a 1984 study by Miller, used trainee examiners, and that in a 2014 study by Kukucka and Kassin lay people judged handwriting samples in the presence or absence of a confession.

Marijn Kouwenhoven's 2019 doctoral thesis at the University of Otago, "Focus on the task at hand: Contextual bias in the forensic examination of handwriting", tested lay participants judging signatures as genuine or forged. In Study 1A the context had no significant effect. In Study 1B, which corrected its limitations, high-biasing context increased participants' response bias and reduced their ability to discriminate. Study 2 found that asking participants to focus on similarities instead of differences did not remove the effect, and Study 3 interviewed 19 professional document examiners from government laboratories and private practice about how they meet and manage contextual information.

A 2024 study in Science and Justice (volume 64, No. 4, pages 360 to 366), "Contextual bias by Forensic Document Examination trainees: An empirical study from China", tested 24 students. The context groups differed significantly in their decision scores (p < 0.05, Cohen's d > 0.8), and the authors wrote that the participants' reports disclosed a biased evaluation of handwriting features.

What information should the examiner not receive?

Anything the comparison does not need: confessions, other evidence, the results of other disciplines, motives and the client's theory. The examiner needs the questioned item, the known writing and facts about how each was written.

Section 2.1.7 of NIST IR 8282r1 says that a confession is relevant to the case as a whole but should never be considered by forensic scientists who give opinions on scientific evidence. In its worked example of a disputed suicide note, the report lists what the examiner should not know: the suspicion against a relative, the financial troubles and the inheritance, the violent behavior, and the DNA and fingerprint results. An opinion from another discipline, such as a fingerprint result, should not reach the handwriting examiner either.

Relevant information remains. The examiner needs to know which items are questioned, which are known and where they came from, and facts that bear on the writing, such as the writer's age, illness or the conditions of writing. The report notes that a list of task-irrelevant information is difficult to identify exhaustively, while a list of task-relevant information is much easier to define; in work by Mattijssen and colleagues that it cites, the second approach gave more consistent results and was faster.

In a civil case the retaining party's theory is task-irrelevant information too. A cover letter that says the client never signed the guaranty tells the examiner which answer is wanted.

Why does the order of examination matter?

Because known writing seen first can steer which features of the questioned writing the examiner looks for. The NIST report says to analyze the questioned material first, before any writing of the suspect.

Linear sequential unmasking, in the glossary of NIST IR 8282r1, requires the examiner to examine the unknown material first and the suspect's writing after it. Section 1.2.2 says the conclusion on the quality of the questioned writing should ideally be drawn before the known writing has been seen and with no knowledge of the context of the case. Section 2.1.3 describes the reverse order, in which examiners start with the suspect and then turn to the evidence, as "a potentially dangerous method that should be avoided."

The report adds a limit: a strict approach to sequential unmasking in handwriting "could result in a loss of evidential strength." A review therefore looks for a record of what was documented before the known writing was opened and what was added after.

Paulo Akira Kunii of the Brazilian Federal Police, in a presentation published by NIST, lists as one option for digitized documents a workflow in which known writings are submitted only after the questioned writings have been examined, and any change of opinion after that point must be included and explained in the final report.

What is a context manager?

A person who filters the case file before it reaches the examiner, removing task-irrelevant information and passing on what the comparison needs. In handwriting work the manager may also mask irrelevant text on the document images.

NIST IR 8282r1 describes the context-manager model: a context manager removes task-irrelevant information from the case file before it reaches the examiner. The report calls this the main approach suggested to reduce bias from that kind of contextual information.

Kunii's proposal goes further for documents. The context manager prepares images by cropping and obscuring irrelevant content and must be a qualified handwriting examiner. The examiner can ask for more or better images, and the manager can ask the investigator for more information. Kunii cites the forensic services of Victoria Police and the Netherlands Forensic Institute as laboratories that use context managers. He also lists the limits: media coverage of a case can defeat context management, and some documents, such as a suicide note, carry biasing content in the questioned writing itself.

A private examiner working alone has no second person to act as manager. The record of what counsel sent and when still exists, and the review asks for it.

Do laboratories have to manage contextual information?

The NIST working group says they must. Recommendation 2.2 requires laboratories performing handwriting examinations to use a contextual information management protocol documented within their quality management system.

Recommendation 2.2 of NIST IR 8282r1 reads: "Forensic document examiner laboratories performing handwriting examinations must use a contextual information management protocol", which must be documented within the quality management system. Recommendation 2.1 asks the research community, with examiners, to study the impact of various sources of contextual information on handwriting examinations.

Recommendation 2.3 limits the wording of conclusions: examiners must not report or testify, directly or by implication, that questioned handwriting has been written by an individual to the exclusion of all others.

The report is a set of recommendations of an expert working group, not a rule of court. A review uses it as the published benchmark for what the examiner's quality system should contain and asks whether the examiner's laboratory has such a protocol and followed it in the case.

How do you find out what the examiner was told?

Through the report and discovery. The report lists the facts or data considered, and communications with counsel are open to discovery where they identify facts, data or assumptions that the attorney supplied and the examiner used.

Rule 26(a)(2)(B)(ii) of the Federal Rules of Civil Procedure requires the expert's report to contain "the facts or data considered by the witness in forming" the opinions. Rule 26(b)(4)(C) protects communications between the attorney and a reporting expert, except to the extent that they relate to compensation, identify facts or data that the attorney provided and that the expert considered, or identify assumptions that the attorney provided and that the expert relied on.

For a review of contextual information, the useful items are the engagement letter, the cover letters that came with each delivery of documents, any summary of the case the examiner received and the examiner's notes of telephone calls. Each of them shows what the examiner knew before and during the comparison.

Was the result verified by a second examiner who did not know the answer?

A review asks for the verification record. The NIST report calls blind technical reviews and independent re-examinations particularly important, and says the re-examiner should not know the original conclusion.

Section 2.1.7 of NIST IR 8282r1 says blind technical reviews and independent re-examinations matter because the reviewer does not know the case information or the original opinion. When a re-examination is done, the second examiner should be blind to the first conclusion.

Section 2.1.5 suggests telling examiners that their caseload will include simulated cases with "innocent" writers and recommends studying whether such blind cases can be inserted. A verification by a colleague who read the first report before looking at the writing is not a blind verification. The review compares the verification record with what the verifier had seen.

Which questions does a reviewer put on contextual information?

Questions about the file, the order of work and the verification. Each one can be answered from the examiner's records, and missing records are themselves a finding of the review.

The review lists the points where the records should answer, and the same points go into the deposition outline.

  • What did the examiner receive with the documents: a case summary, pleadings, the client's account, other experts' reports?
  • Did the cover letter or the engagement letter state the result the client expected?
  • Was the questioned writing documented before the known writing was examined, and is there a dated record of that step?
  • Does the examiner's laboratory have a written contextual information management protocol, and was it applied in this case?
  • Was the conclusion verified, by whom, and did the verifier know the first conclusion or the case information?
  • Did the examiner change any observation after seeing the known writing or the case information, and is the change explained?

What does IAE do when the opposing handwriting examiner may have been told too much?

IAE prepares a review of the opposing handwriting examiner's report, checking the file against Rule 26 and the published standards, in up to 5 business days after the format and cost are confirmed.

The IAE review page for handwriting reports names this check among the reviewer's points: NIST IR 8282r1 (Recommendation 3.2) puts the background information, propositions and assumptions given to the examiner into the case record, and under Rule 26(b)(4)(C) facts and assumptions from counsel are discoverable if the examiner used them. The reviewer also asks for the bench notes, images and the list of known writing, which ANSI/ASB Standard 070 (6.1) requires to be documented in enough detail for review, and checks limitations and verification.

Send the opposing report and the produced file through the form on the contact page. The secretariat confirms receipt the same day and, within one business day, the format, the cost, the deadline and the specialist from the registry; you approve the reviewer and the CV before work starts. Under Rule 26(b)(4)(D) a consulting expert's opinions are ordinarily not discoverable, so counsel can read the review before deciding to disclose the reviewer. Cases with tight deadlines have a 48-hour rush.

Registry members passed the collegial attestation of the council under the published Criteria Matrix, and the Member ID in the document can be checked by QR code. The reviewer confirms in the document that he or she has no interest in the outcome, and materials are not passed to third parties without your consent. Neither IAE nor the reviewer guarantees the outcome, and IAE is not a law firm and does not advise on case strategy.

Sources

Read also

All articles

Expert report review

Ready to discuss your case?

Send the materials: the secretariat will confirm the format, cost, deadline and the specialist within one business day.

Expertise request

Received

Cost and timeline estimate

WorkReview of an opinion
TimelineUp to 5 business days
ExpertIAE registry member