In my job as a trainer of Quality Auditors, one of the things that I run into regularly is a misunderstanding of the requirement for auditors to have an evidence-based approach. Evidence is one of those terms that everyone automatically assumes that they understand, but when it comes to determining what is evidence they often run into problems. The problem is that few people naturally recognise evidence or fully understand the mechanics of evidence.
Evidence is defined as: the available body of facts or information indicating whether a belief or proposition is true or valid. Evidence is also described as: Information drawn from personal testimony, documents, or material objects used to establish facts.
But there are rules for evidence and there are different kinds of evidence. These details must be understood before you can fully appreciate a concept like “Evidence-Based Approach”.
The first rule of evidence is that it must be relevant to the investigation. For example – stating “I’ve seen this approach used before in another organization and it didn’t work very well so it won’t work here” is not evidence nor is it relevant! If it is not directly related to the case at hand it isn’t relevant evidence. That said, there are many types of evidence that can be valuable and others that can be detrimental to an auditor trying to reach a conclusion in a workplace investigation.
TYPES OF EVIDENCE
Anecdotal Evidence – Anecdotal evidence is based on hearsay rather than hard facts. People like to share stories about things that happened to them, or that they heard about. This is anecdotal evidence based on personal accounts. The biggest problem with Anecdotal Evidence is that it is often “cherry picked” to support a particular conclusion. Auditors should consider anecdotal evidence with serious skepticism, and use it ONLY in combination with other, more reliable kinds of evidence.
Character Evidence – is testimony or documentation that is used to help prove that someone acted in a particular way based on their character. For example saying they are lazy or inattentive or intentionally not doing the job as expected. This can’t be used by an auditor as evidence because auditors will never have sufficient information to draw such a conclusion. However, it is often used in workplace investigations to prove intent, motive, or opportunity when determining root cause. Auditors should discourage this process because there is almost always a further root cause, which is systemic in nature and causes the behaviour. Auditors may be given this type of evidence as anecdotal input which should be considered unreliable. Example: “The inspection report is incomplete and that is because the inspector is not conscientious.” Is this evidence of an operator who isn’t conscientious or maybe the operator was called away before the job was finished?
Circumstantial Evidence – is used to infer something based on a series of facts separate from the fact the argument is trying to prove. It requires a deduction of inferred facts from other provable facts. For instance: “Mary’s training records couldn’t be found, therefore Mary is not trained.” Circumstantial evidence is available related to Mary’s training, but what is known is that Mary’s records could not be found. No evidence is provided that Mary isn’t trained. Auditors should never consider circumstantial evidence as strong evidence.
Demonstrative Evidence – An object or document is considered to be demonstrative evidence when it directly demonstrates a fact. It is a common and reliable kind of evidence. Examples of this kind of evidence are photographs, records, charts, reports, minutes etc. Demonstrative evidence is common and reliable in systems auditing.
Digital Evidence – is Demonstrative Evidence from any sort of digital file from an electronic source.
Direct Evidence – is a powerful type of evidence that requires no inference. The evidence alone is the proof. This could be the testimony of a witness who saw an event first-hand. “I watched John inspect the product and he did not use all of the required gauges and did not complete the required documentation.”
Documentary Evidence – is commonly written forms of proof, such as documents, records, reports or minutes. Documentary evidence can also include other types of media, such as images, video or audio recordings, etc.
Hearsay Evidence – consists of statements made by witnesses who were not present to witness an event. Hearsay evidence should not be considered by auditors unless it is directly supported by other more reliable forms of evidence.
Physical Evidence – is also known as “real” or “material” evidence. It can be presented as a physical object, captured in still or moving images, described in text or referred to in documents. “During the audit review gage number 16T was found to be damaged and unable to effectively inspect the product.”
Prima Facia Evidence – Also called presumptive evidence – is evidence which at first and without full investigation and analysis would appear to be evidence. Prima Facia evidence must be verified before it can be considered.
Opinion – in reference to evidence this refers to someone’s personal beliefs, views, or judgments on a matter. Due to the subjective nature of opinions, they can at the very least be quite unreliable. Evidence is specific, factual observation, not opinions. In other words, auditors should keep their statements as fact-based as possible, with little or no interjection of their own beliefs.
Statistics – Often evidence provided to auditors is in the form of statistics. Most often charts showing performance over time. All statistics are not created equal and chart data can be manipulated intentionally or unintentionally.
Examine first “who created the chart.” We live in a world of statistics – you can find numbers in support of or against just about any idea. You can find statistics that prove that diet soft drinks will give you cancer and other statistics that show conclusively that soft drinks have no effect on anything but your thirst. Both sets of statistics are probably absolutely true. It is the choice of the data used and the data discarded that determines the conclusion.
The phrase “numbers don’t lie” is true. However what you need to examine is who is publishing the numbers, and what are they trying to prove with them. Statistics can be used to help an organization improve or they can be used to convince outsiders that the systems are performing better than they really are. Both use valid data to drive very different objectives. Any point of view can use statistics to support its ideas. It’s the auditor’s job to examine all statistics supporting all points of view, to arrive at your own conclusions.
Once you have determined whether or not there is prejudice involved in the statistics then it is time to move on to the next question: What are the statistics measuring?
When asking yourself, “what are the statistics measuring,” bear in mind the old saw about measuring apples and oranges. Most people will say that you can’t compare apples and oranges. This is both true and false. It depends on what you are measuring! If you are measuring color you cannot compare apples and oranges. Neither can you compare them if your are measuring texture or overall appearance. But if you are measuring acidity, sugar content, vitamin, mineral, carbohydrate or fat content then yes, a comparative analysis is quite possible.
As you can see, it is possible to compare apples and oranges, if you know what you are measuring. Your job, in reviewing statistics as evidence, is to determine what exactly is being measured, who created the statistic and are you looking at factual data and not simply numbers that seem to apply to your topic.
There are many tricks used in presenting data. Manipulating scales, cherry picking data points, creating measurement charts that measure extraneous or misleading data that is not directed toward the desired objective. For example: “Here is a chart for the performance of our Internal Audit process. We measure audits completed on time.” In other words the measure is that the auditors arrived on time. The objective of internal audit is usually to identify weaknesses in the system so that they can be improved. Shouldn’t the measure address that end? A chart showing that all audits were done per a schedule isn’t evidence of an effective process.
The only evidence that can be used by an auditor is evidence that can be described, documented, verified, and relevant to the identified purpose of the investigation, and is not modified for effect and is specific to the intent.