Expert report review
Reviewing an authorship attribution report: are the known writings enough?
An authorship opinion compares a questioned text with known writings of the possible authors, so a review starts with those known writings: where they came from, how many there are, how long they are and whether they match the questioned text in genre and topic. In Tim Grant's 2007 worked example, as the comparison set shrank toward three texts per author, the conclusion that should be reached was that no attribution can be made.

What are known writings, and why does an authorship opinion depend on them?
Known writings are texts whose author is not in dispute. The linguist measures features in them, compares the result with the questioned text and infers who wrote it, so every conclusion rests on that comparison set.
Carole E. Chaski described the usual casework in Linguistic Authentication and Reliability, a paper on research done during a National Institute of Justice fellowship: the analyst receives a questioned document together with known writing samples from one or more potential suspects. Chaski and John R. Middleton put the task in one line in a 2009 abstract for the American Academy of Forensic Sciences: author identification problems are classification problems.
A classifier learns from examples. Efstathios Stamatatos, in A Survey of Modern Authorship Attribution Methods (Journal of the American Society for Information Science and Technology, 2009, vol. 60, no. 3, pp. 538-556), wrote that such classification algorithms require multiple training instances per class for extracting a reliable model. In an authorship case the classes are the candidate authors and the training instances are their known writings.
The Federal Rules of Evidence point the same way. Rule 901(b)(3) lists, as one way to authenticate an item, a comparison with an authenticated specimen by an expert witness or the trier of fact. Under Rule 702(b) the proponent must show the court that it is more likely than not that the expert's testimony is based on sufficient facts or data. For an authorship opinion the known writings are the core of those facts or data, and a review tests them before it tests the method.
Do the known writings themselves have to be authenticated?
Yes. A comparison under Rule 901(b)(3) is made with an authenticated specimen, and a text of uncertain authorship placed among the known writings contaminates the comparison it is meant to anchor.
Rule 901(a) requires the proponent to produce evidence sufficient to support a finding that an item is what the proponent claims it is. Blake Stephen Howald opened his 2009 article in the International Journal of Speech, Language and the Law (vol. 15, no. 2, pp. 219-247) with the same rule: all documentary evidence admitted at trial must be authenticated, and for authorship attribution this means showing that the document in question has or has not been authored by a specific person.
The same requirement applies one step earlier, to the known writings. Chaski warned against using as test data any document whose authorship is in question. Texts with more than one possible writer, such as messages from a shared account or letters prepared by an assistant, need that showing before they enter the comparison set.
The review therefore lists, for each known text, its source, the person who identified it as written by the candidate and the basis for that identification: a signature, an admission, a witness, a device seized from the candidate. A known text with no stated provenance is flagged, and the reviewer notes which conclusions in the report would change without it.
How many known texts does an authorship comparison need?
No validated minimum exists. Published tests show that results degrade as known texts get fewer and shorter, and that with too little data the defensible conclusion is that no attribution can be made.
Tim Grant tested this directly in Quantifying Evidence in Forensic Authorship Analysis (International Journal of Speech, Language and the Law, 2007, vol. 14, no. 1, pp. 1-25). His worked example started with twenty comparison texts for each of three potential authors and then used a progressively smaller comparison corpus: fifteen, ten, five and finally three texts per author. In his words, "With greater numbers of reference texts quantified and safe attributions are shown to be possible." As the number of reference texts fell, "the conclusion which should be reached is that no attribution can be made." Grant reported that "The testing process at no point results in instances of a misattribution."
Hans van Halteren benchmarked an author recognition system on British National Corpus data in Linguistic Evidence in Security, Law and Intelligence (2019, vol. 3). He varied the sample size among 1,000, 500, 250 and 125 words and the number of training samples between 2 and 20, using 28 authors with three or more texts. With 7 training and 3 test samples of 1,000 words, text attribution had an error rate of 0.06 percent. With 2 training samples and 1 test sample of 125 words of sequential text, the equal error rates for text attribution and text verification were 26.6 and 42.2 percent, and author verification did not perform better than chance.
Stamatatos found no fixed threshold. He wrote that various studies reported promising results with short texts under 1,000 words, but that it is not yet possible to define a text-length threshold, and that some methods work well on long texts but poorly on short ones. A report that does not state the number and length of the known texts, or that claims a firm attribution from a handful of short messages, gives the reviewer a point under Rule 702(b).
Does it matter if the known writings are a different genre or topic?
Yes. Stamatatos wrote that test corpora should be controlled for genre and topic, and van Halteren concluded that a benchmark should imitate the case situation as closely as possible.
If the known writings are formal letters and the questioned text is a string of short messages, the comparison crosses genres. The features that separate the candidates in their letters have to be shown to exist, and to behave the same way, in short messages.
Stamatatos named this problem in his survey: evaluation corpora should be controlled for genre and topic, and he listed cross-genre attribution, where a method is trained on texts of one genre and applied to texts of another genre by the same authors, among the questions that research had not yet examined well. Van Halteren reached a related conclusion from his benchmark: the benchmark situation should imitate the case situation as closely as possible, and a corpus-based benchmark cannot estimate the quality of the result in a specific case; it can only serve as a gatekeeper.
The reviewer compares the known writings with the questioned text on genre, topic, medium, length and, where the dates are known, time of writing, and asks whether the report tested its method on material of the same kind. If it did not, the error rate the report cites belongs to a different task.
Is the list of possible authors closed or open?
A closed set assumes the author is one of the named candidates. An open set leaves room for an author who is none of them. Stamatatos wrote that the open-set setting fits forensic work better.
Stamatatos observed that most attribution studies assumed a closed set, in which the true author is one of the candidates, and that the open-set setting fits forensic applications better, because the true author may not be among the suspects. In the open-set case, he wrote, special attention must be paid to the confidence of the decisions.
Patrick Juola illustrated the difference in his presentation Error Treatment in Forensic Authorship Attribution, posted on the website of the National Institute of Standards and Technology. He labeled the Rowling case an open-class problem. For an anonymized email case he noted that the physical evidence confined the possible authors to four people.
A report that compares the questioned text with two or three candidates and names the closest one has answered a closed-set question. The reviewer asks what grounds the report gives for excluding everyone else, and whether that limit comes from facts in the case, such as access to the account or the device, or only from the list counsel supplied. The 1998 study Daubert-Inspired Assessment of Language-Based Author Identification, catalogued by the Department of Justice as NCJ 172234, concluded that determining authorship from a fixed set of suspect documents is not the same as determining individuality in language.
Which features did the linguist count, and were they tested?
The reviewer lists every feature the report relies on and asks whether it was validated on known data. Spelling errors and favored words are the features most often criticized in the published research.
The 1998 study catalogued as NCJ 172234 tested language-based author identification techniques on four writers drawn from a writing sample database. Its abstract states that the authorship of a document should not be decided based on common conceptions of language such as spelling errors and vocabulary words, and that language experts who offer such evidence should not testify about the authorship of a document.
Chaski's paper on linguistic authentication reported pilot tests in which techniques built on common misconceptions about language were unreliable. Her example: an analyst who relied on spelling errors would have concluded that there were three known writers when there were four. Her summary table recorded incorrect clusters produced by type-token ratio, readability scores and spelling analysis.
Grant's approach adds a case-specific step. In his sampling strategy, "markers identified as being generally valid and reliable are retested for use in specific authorship analysis cases." The reviewer checks whether the report did anything comparable: whether each feature was shown to separate the candidates in their known writings before it was applied to the questioned text.
How strongly may the report word its conclusion?
Only as strongly as the tested method supports. The 2023 committee note to Rule 702 tells forensic experts to avoid claims of absolute certainty when the methodology is subjective and potentially subject to error.
Under the amended Rule 702(d) the proponent must show that the expert's opinion is a reliable application of the principles and methods to the facts of the case. The 2023 committee note says that each expert opinion must stay within the bounds of what can be concluded from a reliable application of the expert's basis and methodology, and that forensic experts should avoid assertions of absolute or one hundred percent certainty, or to a reasonable degree of scientific certainty, if the methodology is subjective and thus potentially subject to error.
The published research gives the reviewer the measure. Juola argued for testing methods, not practitioners, and for third-party method validation in a non-adversarial setting, and he described combining tests to produce Daubert-style error measures. Chaski and Middleton wrote in 2009 that independent research had found the forensic stylistics method to have extremely low accuracy and referred to court rulings that kept such testimony out of trial.
Georgina Heydon, describing an expert report on authorship in a murder trial in a chapter titled Authorship Attribution Case File, wrote that in some rare cases a simple corpus analysis can be applied effectively, and that the forensic linguistic community is in urgent need of a reliable and valid tool for establishing authorship. A report that states an attribution without an error rate, a validation study or a reason why this case is one of the rare ones gives the reviewer the core of a Rule 702 objection.
What should the review ask the other side to produce?
Every known writing and reference text the linguist considered, the source of each, the counts behind each feature, any validation runs, and the facts or assumptions that counsel supplied about who could have written the text.
Rule 26(a)(2)(B)(ii) of the Federal Rules of Civil Procedure requires a retained expert's report to list the facts or data considered in forming the opinions. For an authorship report that list should contain each known writing, each reference corpus and the questioned text in the form the linguist analyzed it.
Communications between counsel and the expert are protected under Rule 26(b)(4)(C), with three exceptions. They are discoverable to the extent they relate to compensation, identify facts or data that counsel provided and that the expert considered, or identify assumptions that counsel provided and that the expert relied on. When counsel supplied the list of candidates or selected the known writings, those communications fall within the second and third exceptions.
A party may depose the opposing linguist once the report is provided, under Rule 26(b)(4)(A). Absent a stipulation or court order, a rebuttal report intended solely to contradict or rebut evidence on the same subject matter is due within 30 days after the other party's disclosure under Rule 26(a)(2)(D)(ii). The review is timed so that its findings can go into that report or into the deposition outline.
What does IAE do when the other side offers an authorship report?
A linguist from the public IAE registry reviews the opposing report: which texts were considered, whether the method can be tested and whether the conclusions go beyond the data. The document is issued in up to 5 business days.
The review covers the known writings, the reference texts, the features and the wording of the conclusion, and each point is tied to a page of the opposing report and to a rule or a court decision. The outputs are a review of the opposing report, a rebuttal report and the grounds for a Rule 702 motion.
The reviewer needs the opposing report with every exhibit, each text the linguist lists as considered under Rule 26(a)(2)(B)(ii), the known writings used for comparison and any reference texts. Materials go through the contact form or to info@iaexperts.com and are not passed to third parties without the client's consent.
IAE confirms the format, cost, deadline and the specialist from the registry within one business day, and the client approves the candidate and the CV before work starts. The document is issued in up to 5 business days after the format and cost are confirmed, with a 48-hour rush for tight deadlines. IAE is not a law firm, and the ruling on any motion is the court's; neither IAE nor the reviewer guarantees the outcome.
Sources
- Federal Rules of Evidence, Rule 702 and 2023 committee note (LII)
- Federal Rules of Evidence, Rule 901 (LII)
- Federal Rules of Civil Procedure, Rule 26 (LII)
- Tim Grant, Quantifying evidence in forensic authorship analysis, International Journal of Speech, Language and the Law 14(1), 2007, pp. 1-25 (Aston University)
- Hans van Halteren, Benchmarking Author Recognition Systems for Forensic Application, Linguistic Evidence in Security, Law and Intelligence 3, 2019
- Efstathios Stamatatos, A Survey of Modern Authorship Attribution Methods, JASIST 60(3), 2009, pp. 538-556
- Carole E. Chaski, Linguistic Authentication and Reliability (NCJRS)
- Daubert-Inspired Assessment of Language-Based Author Identification, NCJ 172234 (1998) (Office of Justice Programs)
- Carole E. Chaski and John R. Middleton, Forensic Linguistics, AAFS 2009, abstract C1 (PDF)
- Patrick Juola, Error Treatment in Forensic Authorship Attribution (NIST)
- Blake Stephen Howald, Authorship Attribution under the Rules of Evidence, International Journal of Speech, Language and the Law 15(2), 2009, pp. 219-247
- Georgina Heydon, Authorship attribution case file (Routledge chapter abstract)
Read also
All articles
The opposing linguist reads an emoji as a promise, a threat or an insult: how is the meaning of an emoji tested?
A federal court has held that "Emojis may be actionable if they communicate an idea that would otherwise be actionable," and that liability "will
The opposing linguist worked from a translation: how is the meaning of foreign-language words reviewed?
When the words at issue were spoken or written in another language, every opinion about their meaning passes through a translation. If the opposing
Is the word a slur, and how does a reviewer test the opposing linguist's reading of offensive language?
Whether a word was a slur depends on how it was used. In Ash v. Tyson Foods the Supreme Court held in 2006 that the word "boy", standing alone, isExpert report review
Ready to discuss your case?
Send the materials: the secretariat will confirm the format, cost, deadline and the specialist within one business day.

