Structured observation and assessment

SOMAT is the software Harlow Stewart uses for safety audits and surveys with airlines, and we're now adapting it for other organisations whose practice has to stand up to scrutiny, from housing providers to financial services.

Observers and assessors record what happens during ordinary work, an independent evaluator reviews each record, and the results are then compared with outcomes you already track, such as complaints or readmissions.

Supplier audit S-0158 · site visitIllustrative example
  1. Threat

    A new filling line was commissioned two months before the audit.

  2. Error · documentation

    The supplier's self-assessment still describes cleaning validation for the old line.

  3. Undesired state

    The auditor can't confirm cleaning validation for the line now making the product.

  4. Outcome

    A corrective action is raised, and a related deviation is recorded at goods-in the following month.

Evaluator: "Answers matched what we found everywhere else on site. Raised so that self-assessments are refreshed after any change to a line."

Pharmaceuticals

What SOMAT does

SOMAT brings together work that is often spread across several spreadsheets and systems, from designing a questionnaire or observation scheme to analysing the results. Keeping it in one place means that when an analyst finds a pattern, they can open the sessions behind it and see exactly what was recorded and by whom.

Assessment

Questionnaires and self-assessments, scored, with documents or photographs attached to the answers as evidence. A survey that changes is replaced by a new version, so earlier results still refer to the questions that were asked at the time.

Observation

An observer records what happened as a sequence of events, for example a cue from a customer, the agent's response and how the call ended. Each scheme sets which events may follow which, which keeps observers' records consistent with one another.

Evaluation

An evaluator reviews each session and can record their own answer beside the original, so both are kept and you can see where they differ. Anything that needs attention straight away can be flagged.

Analysis

Dashboards and Excel exports compare what was recorded with the outcomes you measure. You can click through from a chart to the sessions behind it, and an analyst checks each pattern before it's reported as a finding.

Findings

Each finding is reported with the number of cases behind it, whether it held in each area or team when they were looked at separately, and a note from the analyst who checked it.

We're careful with the wording. Most findings show that two things tend to go together, which is a good reason to look more closely or to try a change, although on its own it can't show that one causes the other, and we say so on the finding itself. If later sessions stop supporting a finding, we withdraw it.

How we report a finding

Damp and mould inspections · social housing · illustrative

Homes where the inspector recorded a likely cause had a repeat damp report less often

Cause recorded27/251 · 11%
Not recorded38/136 · 28%

Repeat reports within six months of the inspection. The difference appeared in each of the three areas when they were analysed separately. The two were found together, which on its own can't show that one causes the other.

Analyst's note: supports making "likely cause" a required, coded field from next quarter.

Other sectors

Outside aviation, SOMAT is at the pilot stage. We've built example versions for seven sectors so that you can see how it might look in yours, using fictional organisations and made-up data, and the regulations named on each are there to show the kind of question that sector is being asked. A working version would be designed with a pilot partner, around their own questions and the outcome measures they already report.

Background

SOMAT grew out of aviation. On a line observation audit, a trained observer sits on the flight deck during a normal flight and notes the threats and errors that came up and how the crew dealt with them, using a framework called threat and error management (TEM). Harlow Stewart runs these audits for airlines, and SOMAT is the software it records them in.

Much of that structure transfers to other work where people have to deal with the unexpected. The table shows how the steps of an observation on the flight deck correspond to those on a customer call.

More about the background

StepOn the flight deckOn a customer call
OpensThreatCustomer cue
HandledManaged or notHeard or missed
SlipCrew errorAgent error
EndsUndesired aircraft stateUnsuitable outcome

The people behind it

Harlow Stewart

Harlow Stewart brings the aviation safety expertise. It runs observation audits and safety surveys for airlines, and SOMAT records an observation in the same way that Harlow Stewart's observers carry one out.

harlowstewart.com

Elan Smart Solutions

Elan designs, builds and supports the software. We began rebuilding SOMAT in September 2024, and if you get in touch about a pilot, you'll be talking to the people who built it.

elansmart.com

Pilot partnerships

We're looking for a small number of organisations in each sector to run a six-month pilot with us. You'd help shape the version you use, and if you carry on afterwards, you keep a founding-partner rate.