Expert report review

ShotSpotter evidence: is it admissible, and how does a reviewer test the other side's acoustic gunshot report?

An acoustic gunshot report tells the jury how many shots were fired, when and where, on the basis of sound picked up by sensors. Courts have admitted such reports, and in 2021 a federal appeals court held that one was admitted without the examination Rule 702 requires. A review of the other side's report asks which document was produced, who changed it and what supports its figures.

A person seen from behind studies a small rooftop sensor box above a quiet golden street

What does a ShotSpotter report claim?

It claims that sensors recorded gunfire and gives a shot count, a time and a location. The location comes out of a calculation on the times at which the sound reached different sensors.

The Seventh Circuit described the system in United States v. Godinez, decided August 4, 2021, as "an acoustic gunshot detection and location system." Sensors "listen specifically for the sound of gunfire," and software classifies each sound as gunfire or not. Locations are calculated through what the opinion calls multilateration: differences in arrival time across pairs of sensors are used to plot intersecting curves.

A summary document quoted by the Supreme Court of Pennsylvania in Commonwealth v. Weeden puts it this way: the system "uses strategically placed acoustic sensors to detect and locate gunshots within a coverage area," and "machine learning algorithms analyze and classify the sounds before they are reviewed by acoustic experts."

So a single report carries three separate claims. The sound was gunfire. There were this many shots at this second. They were fired at this address. A reviewer tests each claim against the data behind it, and the data are different for each.

Which document did the other side produce: an alert summary or a reviewed analysis?

The two differ. In Weeden the prosecution used an Investigative Lead Summary that said on its face it had not been independently reviewed by the company's forensic engineers and should only be used for initial investigative purposes.

The incident in Weeden took place on December 15, 2018 in Pittsburgh: the system detected two possible gunshots near 3400 Shadeland Avenue at about 7:43 p.m. The Investigative Lead Summary used at trial was created on July 3, 2019, about six months later, at the request of a company employee. It listed the date, time and location of the suspected gunshots, a map, a general description of the program, and the shot count and timing.

According to the opinion, the shot data in the summary was "automatically calculated by the ShotSpotter system at the time of detection." The summary stated that it "has not been independently reviewed by our Forensic Engineers" and that it "should only be used for initial investigative purposes."

The first line of a review therefore records what kind of document the opposing party relies on, when it was generated, at whose request and whether any person examined the audio before the document was issued. A detective who described the program at the Weeden trial had not prepared the summary and was unsure whether a human operator had reviewed the detection.

Did the shot count, the time or the location change after the first alert?

In Godinez they did. The first result was two shots near one street corner; after police asked for more audio, the company identified five shots at a different address and a time four seconds earlier.

According to the Seventh Circuit, the system first identified two shots near the southeast corner of 44th and Hermitage in Chicago, between 3:18:14 and 3:18:15 a.m. Later the police asked for more audio, and the company identified five shots at 4338 South Hermitage, between 3:18:10 and 3:18:12 a.m.

Justice Wecht, concurring in Weeden, pointed to similar marks in the Pittsburgh summary: a reviewer's email address and an adjusted shot count. He wrote that "only the initial stages of a ShotSpotter 'hit' involve automated systems" and that, according to the detective's testimony, every detected sound traveled through a human operator.

A reviewer asks for every version of the result with its timestamp: the automated detection, the alert sent to police, any later re-analysis and the document offered in court. For each change the file should name the person who made it, the audio that person listened to and the reason recorded at the time.

What stands behind an accuracy figure such as "10 to 12 feet"?

In Godinez a company employee testified that the system typically has an accuracy of 10 to 12 feet. The defense answered that the programming and algorithms had never been peer reviewed. A reviewer asks for the tests behind the figure.

The witness in Godinez, Paul Greene, was a ShotSpotter employee. The defendant challenged the methodology and the underlying data as unreliable, argued that "ShotSpotter's programming and algorithms have never been peer reviewed" and asked for a Daubert hearing.

The committee note to the 2023 amendment of Rule 702 says the amendment is especially pertinent to the testimony of forensic experts and that the judge should, where possible, receive an estimate of the known or potential rate of error of the method. A single typical figure is no such estimate until the reviewer knows how it was measured.

The questions are concrete. Was the figure measured by live-fire tests, and in which city? How many sensors heard each test shot? Were the tests run in the coverage area where the incident happened, with its buildings and its sensor layout? How often did the system miss a shot or report one that was not fired? A Pittsburgh detective conceded in Weeden that the system is "not completely foolproof" and that officers have been sent to nonexistent gunshots, while calling it "very accurate."

Can a court admit ShotSpotter testimony without examining the method?

The Seventh Circuit said no in Godinez: the district court abused its discretion by admitting the testimony without a more searching examination of the method under Daubert. The error was held harmless, and the convictions were affirmed.

The case arose out of a shooting in Chicago's Back of the Yards neighborhood in which an ATF special agent was seriously injured. The district court denied a Daubert hearing as unnecessary, reasoning that this type of evidence had been tested in 80 cases involving the same witness. It relied on a Nebraska decision, State v. Hill, 851 N.W.2d 670 (Neb. 2014).

Judge Brennan, writing for the court, distinguished that decision: the defendant in Hill had not challenged the GPS triangulation methodology on which the location is based, and the Nebraska court had held a Daubert hearing. The Nebraska opinion, filed August 8, 2014, records the same. Hill argued that blind tests were never performed, that the witness did not know the system's capacity at the time of the shooting and that incident review staff used subjective judgments to classify sounds as gunfire. The trial court heard the witness at a hearing on the motion, and the Supreme Court of Nebraska found no abuse of discretion. The evidence admitted at the Godinez trial, the court said, "implicated that system's methodology," and "the district court never sufficiently explored this issue." The fact that the witness had been qualified before did not establish that the method was reliable in this case.

The court added that the system "may adhere to the requirements of the best available science" and that without a more searching examination of its methods under Daubert it could not conclude that the evidence was properly admitted against Godinez. It found the error harmless because other evidence of the shooting location, including casings, witness testimony and a plat map, made the acoustic evidence cumulative: it "did not have a substantial influence on the verdicts." Judge Wood dissented on a different ground: in her view the evidence against Godinez was insufficient for a conviction.

Must the person behind the report testify?

Not under the Confrontation Clause, according to the Supreme Court of Pennsylvania in Weeden in 2023: it held the summary nontestimonial. The court did not decide the reliability challenge, which had not been raised at trial.

Chief Justice Todd wrote that the primary purpose of the summary was not "to establish or prove past events potentially relevant to later criminal prosecution." The data, the court said, "was created to aid law enforcement in combating a particular instance of gun violence." The order of the Superior Court was affirmed.

Two limits of the ruling matter for a review. Footnote 13 says the appellant "failed to preserve his challenge to the Summary on reliability grounds." In footnote 27 the court declined to determine whether the summary is generated solely by machine and left for another day whether a purely machine-generated statement implicates the Confrontation Clause. The reliability objection in Weeden was lost because it had not been made in the trial court.

Justice Wecht's concurrence states the problem he saw: "The problem is that the law currently does not require the Commonwealth to put that person on the witness stand." He wrote that the vendor itself warns that the data may change on review and should be corroborated with physical evidence.

How often does an alert lead to evidence of a shooting?

Chicago's Office of Inspector General counted 50,176 alerts between January 1, 2020 and May 31, 2021. Police recorded a gun-related criminal disposition for 4,556 of them, 9.1 percent of all responses.

The report, The Chicago Police Department's Use of ShotSpotter Technology, was issued on August 24, 2021. Of the 50,176 alerts, 41,830 had a recorded disposition and 8,346 had none. The 4,556 gun-related dispositions are 10.9 percent of the alerts with a recorded disposition. At least one investigatory stop report matched 1,056 alerts, or 2.1 percent.

The Inspector General concluded that "CPD responses to ShotSpotter alerts rarely produce documented evidence of a gun-related crime, investigatory stop, or recovery of a firearm." The report says its technical information about the system is sourced primarily from publicly available records.

These figures are not an error rate. An alert with no evidence found may be a real shot with no casing left behind, and the analysis started with alerts, so shots that produced no alert are outside it. They do bear on one question a reviewer puts to the opposing witness: on what data does the witness base a statement that an alert means a shot was fired at that spot?

What does a reviewer ask the other side to produce?

The audio from each sensor that detected the event, the sensor positions and detection times used in the calculation, the classification history with every human review, each version of the report and the testing behind any accuracy figure.

The Seventh Circuit's criticism in Godinez gives the outline. The district court had not explored how the system collects, analyzes and reports its data. Collection means the sensors: where they were, which of them detected the event and what each recorded. Analysis means the classification of the sound and the location calculation. Reporting means the path between the first alert and the document in court.

In a federal criminal case Rule 16(a)(1)(G) of the Federal Rules of Criminal Procedure requires the government, on the defendant's request, to disclose a complete statement of all opinions it will elicit and the bases and reasons for them, the witness's qualifications with publications of the previous 10 years, and a list of cases in the previous 4 years in which the witness testified as an expert. Rule 16(a)(1)(F) covers the results or reports of scientific tests in the government's possession that are material to preparing the defense.

The location calculation the opinion describes takes sensor positions and arrival times as its inputs. Without the recordings, positions and times the review is limited to the documents, and it says so.

How do the review points fit Rule 702?

Rule 702 requires the proponent to show that the testimony more likely than not rests on sufficient facts or data, is the product of reliable principles and methods, and applies them reliably to the facts of the case.

Each point of the review goes to one condition. A report produced without the sensor audio goes to sufficient facts or data. An accuracy figure without test records goes to principles and methods. A shot count that changed after a request by police, with no record of why, goes to the application of the method to this incident.

The 2023 committee note adds that each expert opinion must stay within the bounds of what can be concluded on a reliable application of the expert's basis and methodology, and that forensic experts should avoid assertions of absolute or one hundred percent certainty. A statement that shots were fired at a street number, rather than within a stated distance of a calculated point, is tested against that note.

What other evidence is compared with the acoustic data?

Casings, bullet damage, witness accounts, video and maps. In Godinez such evidence made the acoustic report cumulative; in Weeden officers found no physical evidence of gunfire at the reported location.

In Weeden, searches of the scene and of the appellant's home and car found no physical evidence of gunfire. A detective saw damage to the victim's car that he believed was caused by bullets, and photographs of it were admitted without objection. Two witnesses testified consistently about the shooting, and the appellant called alibi witnesses.

A reviewer sets the calculated point and time beside every other fixed fact in the file: where casings lay, where the damaged car stood, what a camera recorded and at what clock time. A gap of a few seconds between two clocks or a few houses between two points is either explained by the stated accuracy of the system or it is a finding of the review.

What does the IAE review of an opposing forensic audio report cover?

The IAE service page describes a review of the other side's forensic audio report on three subjects: whether a recording was edited, whose voice is on it and what was said. Acoustic gunshot location is not listed there.

The reviewer checks which file the opposing examiner received, which tests were run and documented, whether the voice exemplars were comparable and whether the conclusion stays within the method. Each point in the review is tied to a page of the opposing report and to a rule, a standard or a court decision. A request starts the same way for any report: the client sends the report or the case materials and describes what to check, and the secretariat confirms receipt the same day.

Within one business day IAE confirms the format, the cost, the deadline and the specialist from its registry. The client approves the candidate and the CV before work starts, so a party with an acoustic gunshot report sees the proposed specialist's field before the review begins. The document can take the form of a review of the opposing report, a rebuttal report or a statement of grounds for a Rule 702 motion. The signed PDF is issued in up to 5 business days after the format and cost are confirmed, and a 48-hour rush is available for tight deadlines.

Specialists are members of the public IAE registry, and the Member ID or QR code in the document opens the specialist's record. The specialist confirms in the document that he or she has no interest in the outcome. A review is evidence the court weighs with the other materials; neither IAE nor the reviewer guarantees the outcome, and IAE is not a law firm.

Sources

Read also

All articles

Expert report review

Ready to discuss your case?

Send the materials: the secretariat will confirm the format, cost, deadline and the specialist within one business day.

Expertise request

Received

Cost and timeline estimate

WorkReview of an opinion
TimelineUp to 5 business days
ExpertIAE registry member