ISO Audit Group / Planning

An audit can find a system fully conformant and completely ineffective

Auditing tests whether the described system is the operating system. Whether that system achieves anything is a separate question, and the standards increasingly ask it.

Management system auditing tests a specific relationship: whether what the organisation says it does is what it does. The auditor reads the documented arrangements, then looks for evidence that they operated, and reports where they did not. Performed properly this is a genuine discipline requiring sampling judgement, evidence evaluation and the ability to interview people who would rather be doing something else.

It is also a check on the system's relationship to itself. An organisation can describe a process, follow it precisely, produce complete records, and be achieving nothing whatever, because the process was designed to be auditable rather than to be useful. That system is fully conformant and the audit that reports it as such has answered accurately a question nobody outside the audit actually cares about.

This is why the more recent revisions of the major standards pushed toward outcomes: risk-based thinking, context of the organisation, effectiveness of controls, and the requirement that the system deliver intended results. Those changes were an attempt to make the second question auditable, and they are the parts of an audit most often performed thinly, because they require judgement rather than verification.

The reason is structural rather than a failing of auditors. Conformity findings are defensible: here is the requirement, here is the record, they do not match. Effectiveness findings are arguable, they implicate management decisions rather than clerical ones, and an auditor raising them is on ground where the auditee can and will push back. The incentive runs toward the checkable.

The second structural pressure is commercial. In certification auditing the auditor is engaged through a body that the client pays, over multiple cycles, and the relationship continues. Nobody involved has to behave improperly for that arrangement to produce audits that are thorough about documents and gentle about management systems.

The practical consequence for anybody commissioning an audit is that they should say which question they want answered. An audit for certification has a defined scope and is what it is. An internal audit commissioned to find out whether something is working should be scoped and staffed for that, and it will produce findings that are less tidy and more useful.

For the auditor the discipline is to keep asking what the process is for. A record exists, it was completed, and the question that follows is what decision it informed and what would have happened differently if it had said something else. Where the honest answer is nothing, the process is documentation rather than control, and saying so is the most valuable thing an audit can do.

One further observation about what makes this work sustainable for the auditor. Auditing well requires being willing to be unpopular in a room, repeatedly, on matters where the auditee is senior and confident and the evidence is arguable. That is a professional demand rather than a technical one, and it is the reason the difference between two auditors with identical qualifications can be enormous. It is also why an audit function that has never generated any friction should be examined as carefully as one that generates too much.