Cylab.info

Accred Qual Assur (2003) 8:179–183DOI 10.1007/s00769-003-0609-9 D. Brynn Hibbert
Abstract An Australian case study
Keywords Forensic science ·
Received: 23 October 2002Accepted: 17 February 2003 of abuse were acquitted on appealbecause of shortcomings in the prosecution’s case that establishedthe identity of the material seized.
The need to have proper standard operating procedures that can be followed routinely and correctly ishighlighted.
2052 Sydney, NSW, Australiae-mail: [email protected] au A recent judgment in the Court of Criminal Appeal of In this paper the analytical chemistry of the case is New South Wales highlights a problem faced by courts explained, and broader issues are discussed that relate to and scientists [1]. Australia has an adversarial system of matters relevant to accreditation and quality assurance.
justice that is derived from the English system. In each The nature of expert opinion in the Australian system is case, a plaintiff has to persuade the court of his or her case, while the defendant argues the opposite. The end ofthe case is when the Court, a judge acting as a jury, or ajury of ordinary men and women, determine for oragainst the plaintiff’s case. The case in hand is the trialof Regina (the Queen) v three gentlemen who werefound guilty of ‘knowingly taking part in the manufac-ture of not less than the commercial quantity of a prohib-ited drug, namely methcathinone.’ On appeal, the con-victions were overturned on the grounds that the jury didnot take sufficient account of the shortcomings of theCrown’s evidence of identity of the alleged drug.
In the local press, much was made of the suggestion that they were really trying to make the male anti-impo-tence drug ‘Viagra’ (sildenafil)! If this were indeed a se- rious defence it is difficult to see how a synthetic chem- Structures of 1-[[3-(6,7-dihydro-1-methyl-7-oxo-3-propyl- 1H-pyrazolo [4,3-d]pyrimidin-5-yl)-4-ethoxyphenyl] sulfonyl]-4- ist might have decided to make structure (1) via structure
methylpiperazine monocitrate (Sildenafil marketed as Viagra) and (2) (Fig. 1).
(2) methcathinone
It is easy for scientists to believe that lawyers cannot ap- preciate the proper scientific method, but in this case thetrial judge stated the problem with some clarity [4].
The case discussed here highlights the importance of fo-rensic evidence being of the highest quality, and being – . gave in evidence that, in relation to these tests, he able to be shown to be so in court. The issue was not had not done them in accordance with the manual – complex, and was essentially: was a quantity of white “technically”, again he said – in that he had not run a powder found in the premises of the defendants the standard at the end of each blank, nor had he included chemical substance methcathinone (Fig. 1 (structure 2))
duplicate runs as indicated by the manual. He claims, however, that the reference to “duplicates” relates to The initial analysis was by gas chromatography mass quantitation, not identity testing, even though the spectrometry (GCMS), identifying methcathinone against manual apparently indicates otherwise. He agreed that a literature spectrum (in the absence of a reference sam- his runs in respect of these samples, however, was ple). The GC retention time was later compared with that “technically” deficient, because they did not comply of a reference standard of methcathinone. The transcript of the trial refers to a ‘manual’, which appears to have You have heard the evidence relating to the impor- been the standard operating procedure (SOP), which tance of complying with these set standards and coun- specified the degree of matching of peak intensities be- sel’s submissions to you in relation to that . You tween the mass spectrum of the unknown and a literature may ask yourselves whether substantial non-compli- reference spectrum (±20%). Under cross examination the ance with these set standards, designed to produce analyst admitted that the practice of the laboratory was validity in the results of an analysis, can be properly to perform visual matching only, and that in fact the dif- ferences of peak heights between the sample spectrumand a literature spectrum were indeed greater than 20%.
Those interested in quality assurance in the chemical There is no doubt that the laboratory should have fol- laboratory may thank the learned judge for pointing out lowed the manual. However, it may be seen why the ar- what the laboratory may have temporarily forgotten.
bitrary ±20% was ignored. First the barrister read the cri- When a standard was finally sourced, the attempts of the terion “±20% of reference spectra” as an instruction to laboratory to match the retention time with that of the take the peak height of the reference and determine lim- seized material were also riddled with error. Given (from its at ±20% of that height. In this case the most abundant where it is not clear) a window of ±0.2 min to match the ion intensity was of m/e 58, and measured against this GC retention time of a sample with a standard, the one intensity the next peaks in the literature were m/e 77 chromatogram presented to the court showing a match (11%) and m/e 51 (11%). These literature values are giv- had a retention time of 9.55 min compared with the stan- en in whole number percentages which implies at least a dard 9.8 min. Under cross examination it was admitted range of ±0.5% before any other considerations of preci- that the window might have been increased without doc- sion are made. In cross examination the barrister deter- umentation or reason, in what must be said was unpro- mined that the allowable ±20% range of m/e 51 and 77 was ±2.2%. As 1% of this is already taken up in the It is clear to me that the scientists were convinced that significant figures, it may not be surprising that it is dif- they had identified methcathinone, but fell into the trap ficult to find a match between the ion intensities mea- of believing that their own belief was all that was needed sured in the forensic laboratory and those in the literature to satisfy the court. The fact that the jury found the de- taken under possibly very different conditions. The mea- fendants guilty shows that, despite the shortcomings of sured ion intensities of m/e 58 and m/e 77 were 29% the evidence, they believed the laboratory’s story. It wasand 24% respectively, and much was made of the dis- left to the judges of appeal to understand the need for ab- solute and demonstrable reliability of the chemical evi- It was argued by the witness from the forensic labora- tory that the manual was ‘not supposed to be read in a The importance of quality assurance in forensic evi- black and white fashion’ [2] which merely compounded dence has been highlighted by a judge of the New South the problem, faced with the interpretation by the defence – “5.28 It is essential, in those circumstances, that there – It’s a part of the quality accreditation re laboratory.
be an added emphasis on the adoption of high stan- You must have adequate documentation of the proce- dards of competence, adherence by laboratories to dures that you use for analytical chemistry, and these protocols, and acceptance of a duty to the court on the procedures and protocols are to be followed by all part of criminalists and forensic scientists, as an ad- junct to the adoption by the Courts themselves of pro- cedures, which are designed to improve the quality and the manner of presentation of expert evidence.
5.29 In this regard the need for quality assurance To clarify the role of an expert witness, and to provide and quality control is imperative, carrying with it guidance for experts and lawyers, the Federal and State matters such as the regular checking of forensic labo- courts of Australia have adopted similar codes of con- ratories by random blind sampling, the recording of error rates, and the giving of attention to the training The Federal guidelines start with a clear definition of of technicians not only in a way which ensures their competence, but which also brings home to them the significance and potential consequences of their work, so far as suspects and victims of crime are con- – An expert witness has an overriding duty to assist the Court on matters relevant to the expert’s area ofexpertise An expert is, at the court’s discretion, absolved from thehearsay and opinion rules and so must take care that he The saying ‘to be wise after the event’ may well be ap- or she does not abuse this freedom. The guidelines then plied to this paper. However, the essence of good quality explain how an expert should establish his or her exper- assurance systems is that they promote being wise before tise, how to set out evidence, and how to document prop- the event, thus avoiding the unfortunate consequences of erly the exact nature of the expertise on which the evi- dence is based. Finally the expert must add the following The problem in this case was the difference between the analyst’s absolute certainty about the identity of thesample, and his inability to demonstrate this to the court.
– “[The expert] has made all the inquiries which [the While professional judgement should count for a great expert] believes are desirable and appropriate and that deal, the courts demand solid and very formal proof.
no matters of significance which [the expert] regards Therefore members of a forensic laboratory should: as relevant have, to [the expert’s] knowledge, beenwithheld from the Court.” 1. Be well aware of the standard operating procedures and other systems underpinning the accreditation of The need for this declaration has helped experts stand up to clients who, while not wishing to suborn a witness in- 2. Expect to follow those procedures until a satisfactory to actively lying, have suggested there is no need to mention everything that has come to light (especially the 3. Be able to explain in court the scientific basis of their bits that are prejudicial to the client’s case). I would argue that this approach has an advantage over practice 4. Not to allow themselves to be pressured by manage- in the USA courts, where the expert is very much part of ment, police or judicial officers to produce a substan- In the case at hand, one of the problems identified by the Appeal Judges was the very poor quality of the evi- This case also highlights the need to involve everyone in dence given by the defendants’ expert analytical chemist.
continual rounds of quality improvements. The matching On the witness stand he railed against the work done by requirements using library spectra clearly do not work. A the forensic laboratory in quite intemperate language.
simplistic ±20%, although sounding plausible, was not Despite correctly pointing out the obvious deficiencies in suitable for this situation. It may be that a technical per- the identification of the alleged drug, he allowed himself son in the laboratory had noticed this problem before, to overstate his case to the point where the jury may but without the culture of preventative action he or she have stopped listening to the science. Perhaps the fol- would not have been encouraged to raise the matter.
lowing taken from the judgment, as an example of the Changing SOPs is not an easy matter once a laboratory language used by this expert, could well be used in train- has been accredited, but it can be done if properly docu- ing expert witnesses how not to conduct themselves in mented and justified. The effort would have been well Accreditation to ISO/IEC 17025 [6, 7] should ensure – Some of the results he described as “absolutely amaz- that the proper infrastructure, methods, staff and proce- ing”. He described the GCMS results of August 2000 dures are in place, and it will be of interest to see what as “.this rubbish proffered in evidence.”. He de- steps are taken by the Australian accreditation body scribed test results as “dreadful” and obliquely ac- cused the Crown experts of being biased.
The jury were in the best position to assess his de- will never be able to convict a drunk driver again. Law- meanour and these illustrations show why they might yers have also advanced that opinion, and barristers have not have been inclined to place much weight on his occasionally taken the opportunity to pretend ignorance of even the most basic statistics. In a patent case inwhich I had measured blood glucose levels by two meth-ods on each of eight replicate samples, and had shown by a t-test that the means of each method differed signifi-cantly, I was asked the following. “Professor, you have In the case presented here, neither expert came out with made 16 measurements?” “Yes”. “And they are all dif- any credit. The forensic laboratory failed to maintain its ferent?” “Yes”. “So surely, 15 of them must be wrong!” standards, did not produce reliable results and so the I forbore from correcting him – “at least 15 of them are prosecution eventually failed. In particular the evidence given in court by the prosecution’s analyst appeared to Against this pessimistic view of the courts’ ability to suggest that he was not fully prepared nor did he under- embrace uncertainty comes discussions (unpublished) in stand the importance of the admissions he was forced to the Australian Academy of Forensic Science, in which make under cross examination. The defence expert had senior judges have pointed out that although the ultimate the easier task of explaining to the court the nature of judgment (Guilty/Not Guilty) has no uncertainty, to ar- standard operating procedures and their place in a quali- rive at that point the court has to weigh many matters ty system, and why the disregard shown by the forensic that are uncertain. In assessing the credibility of witness- analyst rendered the identification of the substance unre- es, the weight a certain piece of evidence is to be given, liable. However he also did not manage to do his job and so on, a judge or jury must grapple with impondera- properly, because the jury placed no weight on his evi- bles. Scientific evidence is of a better quality because dence, possibly because of the manner in which he gave quantifying uncertainty places bounds on results that are usually absent from other types of evidence. There may It is important that expert witnesses understand that be a conception that scientific evidence has to be abso- their job is to provide the court with appropriate opinion, lute, otherwise the very credibility of ‘Science’ is at and not to try the case in lieu of the judge and jury. It stake, but this is to make too much of the scientist’s evi- may sometimes be difficult. In a recent case, I discov- dence, which must take its place with all other aspects of ered that an expert called by the opposition had used the case. A better dialog with the courts should allow un- atomic numbers instead of atomic weights to determine derstanding of an uncertainty expressed with a given the amount of substance from a measured weight. In confidence interval. It may be that a court will want conference with the barrister, I explained that this measurements made to a greater precision than the con- schoolboy howler would allow him to repudiate totally ventional 95% confidence interval, which, of course, can the witness’ evidence. However the barrister remained unimpressed, reminding me that I had explained that anatomic number is about half the atomic weight of an ele-ment and so the relative values calculated would not change that much. Despite my entreaties that this wassuch an egregious fault that the so-called expert could Grappling with statistics is not, as we have seen, a strong not have survived the ignominy of its revelation in court, point of the legal profession. The concept of probability the barrister did not make the point. He had realised that is crucial to assessing the worth of DNA matching [12], there was enough against the opinion of this witness, and or indeed the matching of any chemical data. More com- that the correction would not have made a deal of differ- plex issues have the power to confuse, if handled in such ence to the discussion of the results, while confusing the a way. A famous case of illogical reasoning was during court still further over the nature of atomic chemistry.
the trial of the American personality O J Simpson, in The barrister’s job is to win the case, not to strike a blow which a statistician advised that the probability of a known wife beater actually killing his wife was verysmall (1/10,000). Writing it as a conditional probabilityPr(M|B)=0.0001 where M is a murdered wife, and we are given the prior information that the husband is a wifebeater (B). As pointed out in a letter to Nature after the There is an opinion that courts cannot accept the concept trial [13], the real probability that was of significance is of uncertainty. It is said by some forensic scientists that Pr(G|M,B) – the probability that given a murdered wife if evidence is presented on, for example, blood alcohol (M), and the fact that the husband was a wife beater (B), levels, with a certain (low) probability that the concen- that the husband was the killer (G). This can be estimat- tration was actually below the legal limit [10], courts ed by a Bayesian approach to be at least one third.
Courts in England [14] and Australia [15] have set surance, SOPs and the like, is to bring demonstrable their faces against the overuse of statistics to give what rigour to a complex procedure. Courts bring out the appears to be a definitive probability of the guilt or inno- worst in the system. Clever lawyers can tease out in- cence of the defendant. The case of R v Adams estab- consistencies of approach or transgressions of SOPs.
lished that to use Bayes’ Theorem or similar method in a When they are desperate, the system allows for consid- criminal trial to bring together probabilities “plunges the erable obfuscation in cross examination. However this jury into inappropriate and unnecessary realms of theory, is not the fault of the legal system. When liberty is at and complexity deflecting them from their proper task” stake mere opinion, no matter how well founded, is no substitute for evidence correctly acquired, interpretedand defended in court. If, as may be the case here, thelaboratory was accredited to follow SOPs that were not appropriate, the way forward is not to disregard theSOPs, thus invalidating the whole process, but to seek An analyst has a primary duty to determine properly what he or she is asked to do, whether for court, or any There is clearly plenty of scope for education, of legal other client. The existence of accreditation, quality as- 9. Jasanoff S (1995) Science at the bar.
quirements for accreditation in the field tion of Testing Authorities, Sydney,Australia expert witnesses in proceedings in the Federal Court of Australia. Federal Court Practice direction, Federal Court of Australiahttp://www.fedcourt.gov.au/pracproc/practice_direct.html

Source: http://cylab.info/u/tek/Accred.Qual.Assur.20038p.179.pdf