The Madness Of Measuring Nothing

cg_measuring-cup

By Phil La Duke

These days’ organizations live and die by measurements. It seems that no matter where we work we are confronted with the dreaded balanced score card and so we are tasked with measuring “safety”. I’ve said for a long time, “the absence of injuries does not denote the presence of safety” and zero injuries doesn’t tell us a heck of a lot about the risk of injuries within a given population.

The traditional measures of safety, i.e. injuries or days away or restricted time, don’t help us predict the likelihood of future performance, and yet that is the best we seem able to come up with. We play at leading indicators like near miss reporting (as somehow indicating the level of participation of workers in safety) but even these measurements are fraught with statistical noise that can lead us to conclusions not in evidence; so many of our indicators mislead us that one has to wonder if there is any value in them at all.

Recently I was asked to address a meeting of the senior leaders of a multi-national manufacturer and I was asked what some good predictive measures would be for safety. I was pressed for time and I’m afraid I didn’t have the luxury of a prolonged discussion on metrics (my topic was Creating A Culture Of Safety Excellence). So given the fervor set in motion from my last three posts I thought I would add a bit of metaphorical fuel to the fire and lay out for professional debate, what I see as some good ways for correlating business measures to future performance in safety.

Risk Factor #1: Worker Stress and Distraction

Worker stress has a profound impact not only on human error, but on risk taking, and worker’s health as well.  Highly stressed workers are distracted and distraction leads to mistakes which lead to injuries.  Some measures that I think directly correlate to worker stress are:

  • Worker absenteeism.  Absenteeism rates are indicators of both worker stress and worker competence. Research has shown that stressed workers tend to miss more work, and when a worker misses work, his or her job is done by someone less skilled, less practiced at the job, and therefore more likely to deviate from the standard.  In other words, the worker stuck doing the job is at greater risk of injury than the worker whose muscle memory is completing many of the tasks by rote.  Of course this isn’t universally the case, but it is true often enough to correlate, and when it comes to prediction, correlation is the best we’ve got.
  • Number of calls to employee assistance programs.  When we talk about worker distraction, we tend to think in terms of distractions borne in the workplace.  Workers who are worried about financial problems, divorce, or other “off-hours” problems while working face the same dangers as those distracted by work issues.  The number of calls to EAP lines can provide a good idea of how much distraction is in the workplace which correlates to human error, behavioral drift, lapses in judgement and ultimately  workplace injuries.
  • Worker turnover.  Employee turn over creates risk in much the same way absenteeism does: it introduces greater variation into our work processes which in increases the risk of injuries.  The greater the worker turnover rates the higher the risk of injuries as newer, less competent and skilled workers replace higher performing, more experienced workers.
  • Engagement survey scores. Engaged workers tend to do things because these things are the right thing to do.  The lower the level of employee engagement the higher the risk of worker injuries.

In all these cases we have to remember that we seldom have a perfect correlation (a case where everytime factor A is true factor B is also true) and even in those rare cases where there is a perfect correlation such a condition does not mean that there is a cause and effect relationship between two factors.  But since we are looking at the measurement’s predictive value there is always a margin for error, statistical anomolies and statistical outliers.  If we had a perfect way of predicting exactly where and when an injury would occur we would be using it. 

Risk Factor #2: Worker Incompetence

When we talk about worker incompetence, we’re not talking about the nincompoop  who doesn’t seem able to do even the most rudimentary task without screwing things up, rather, we are talking about the skill level at which a worker is able to perform his or her job.

There is a strong correlation between level of mastery at which a worker performs the tasks associated with his or her job and the risk of injuries.  To that end these measurements are appropriate and predictive:

  • Required training % complete. Assuming that we require training because it is necessary to do one’s job, the lack of this training would indicate process variability.  Tracking the percentage of training provides us with a glimpse of how much risk a worker faces of being injured because he or she performed a task improperly.  The greater the percentage of people who have completed training the lower the risk of injury because of a gap in essential skills.
  • % of licenses and certificate expired. Just as the percentage of required training that is complete provides us with an understanding of approximately how many people are likely working out of process (it’s tough to do the job right simply by guessing) so too does the percentage of workers who are working despite having expired licenses and certificates.
  • Time to complete required training.  The longer it takes to complete required training the longer a worker is exposed to workplace risk associated with a skills gape.
  • Worker performance appraisal scores. This particular measure is tricky—it assumes that the worker appraisals are fair assessments of the worker’s ability to accurately complete tasks and do the job. Assuming that there is a robust worker performance appraisal assessment the lower scored individuals should be at greater risk that those who are peforming at higher levels. 

Risk Factor #3: Leader Incompetence.

Workers generally perform in ways for which they are rewarded and eschew behaviors for which they are punished. Low-performing leaders often exacerbate safety issues by behaving inappropriately in their interactions with workers. Some measures that I think directly correlate to leadership competency are:

  • 360 Reviews. 360 Reviews, that is, reviews where a leader’s team members, boss, and peers all contribute to the review, are often excellent indicators of how well a leader interacts with his or her team. The weaker the leader the higher the risk of process variation and hence a rise in the risk of injuries.
  • Leader performance review.  Leaders who perform poorly are generally allowing more variation into the work area the higher the performance of the leader the less likely workers will be harmed on his or her watch. It’s important to note that the leader’s performance review will most likely include things like the productivity of his or her team, general performance in things like cost, quality, and efficiency, in other words, things that will either directly or indirectly impact the risk of injuries.
  • Worker morale.  Of course worker morale can be effected by a host of things unrelated to the leader, but worker morale is heavily influenced by the performance of the leader.  Workers suffering from poor morale generally perform at lower levels which fall outside the processes control limits.  The worse the morale the higher the risk of variation and ultimately injuries.
  • % of safety reviews completed on time. I am not a fan of “behavioral observations”; I’ve always felt the time watching someone work could be better spent taking a more holistic view of worker safety by reviewing the risk conditions (procedural, physical, or behavioral).  That having been said, it is important that leaders conduct routine and repeated inspections of the workplace to identify hazards.  The percent of safety reviews/tours/inspections/observations completed on time is a, at least ostensibly, an indicator of the time to which workers are exposed to hazards.
  • % or performance reviews completed on time.  Completing performance reviews on time isn’t just about making employees feel good, it is also about assessing competency.  The more reviews that are completed on time, the more skills and performance gaps are identified in a timely manner.
  • % Attendance at safety meetings.  The percentage of safety meetings that a leader attends provides a good insight into the level of priority on which the leader places on safety. 

Risk Factor #5: Process Capability

Process variability creates risk; to the product, to the equipment, and to the workers.  The frequency and duration of non-standard or out of process work is a good predictive indicator of risk of injury.  Good measures of process capability (relative to safety) are:

  • % of nonstandard work.  Statistically speaking nonstandard work tends to be more dangerous and the injuries associated with nonstandard work tend to be more lethal than its standard counterpart.  The percentage of work that is nonstandard can indicate a substantial bump in risk associated with any operation.
  • % of jobs with completed JSAs.  A complete and current Job Safety Analysis (JSA) is crucial for the safe execution of work, yet I don’t know any company that has 100% of it’s jobs with JSAs, and many companies don’t have a good track record of keeping the JSA’s current with the standard operating procedure. Understanding the percentage of your tasks have good and current JSAs is a good predictor of future risk (the higher the percentage the lower the risk).
  • % of jobs with Standard Work Instructions. Personally, I prefer Standard Work Instructions (SWI) to JSAs (a good SWI should address all the safety concerns of a job), but SWIs suffer from the same problems that I discussed regarding JSAs above.
  • % behind in production.  I still have nightmares about my days working an assembly line and falling “in the hole” screams of “man in the hole” booming above the cacophony of hand tools, presses, and industrial vehicles still give me chills.  Whenever ever workers are struggling to catch up because they are behind in production the risk of injuries rises.
  • % parts shortages.  When there are part shortages (or tools shortages, or materials shortages, or labor shortages for those of you who work outside manufacturing) workers are forced to work outside the standard process.  This is incredibly dangerous because the standard process is designed with protections against injuries embedded in the tasks.  When a worker is working outside the process the organization is relying on luck to protect them.

Risk Factor #6: Worker Engagement In Safety

We’ve discussed worker engagement in a broad sense, but I think it is important enough to look at worker engagement specific to safety.  Engaged workers will work safely for no more reward than because working safely is the right thing to do.  Worker engagement in safety can be measured by:

  • Number of reported near misses.  Some will argue, correctly, that near misses are lagging indicators, but whether or not a worker choses to report a near miss correlates to the level of worker engagement in the safety process. This meaurement, admittedly, is difficult to get accurately.  Since we don’t know the total actual number of near misses we can’t say with certainty whether the current level of reporting is a high or low percentage.  Even so, the number of workers who report, even more so than the raw numbers of near misses, can provide a good glimpse into the level of importance workers place on safety.
  • Number improvement suggestions.  Workers who take an interest in improving the organization are generally interested in finding and eliminating failure modes, which will include those failure modes that will ultimately place workers at risk of injury.  The greater the number of suggestions the lower the risk.
  • Participation in continuous improvement workshops.  Elimanating variation, risk, and hazards are part and parcel of the continous improvement process so it should surprise no one that the level of participation in these activities correlate to the level of risk.
  • Number of worker grievances.  Worker grievances shed valuable light in to many of the other risk factors identified here and generally the greater the number of grievances the higher the level of risk of injuries.
  • Number of disciplinary actions for safety violations. The number of disciplinary actions for safet violations are indicative of two things: the number of unsafe acts being committed and the extent to which these incidents are taken serioiusly.

Of course one has to be careful in designing and managing these measurements to avoid unintended consequences (for example, one could easily reduce the number of disciplinary actions by not applying appropriate discipline, or one could raise worker performance evaluation simply through “score inflation” but the risk of these unintended consequences can be reduced by solid management practices and random sampling audits.

The Imperfection Of Predictive Measures

To some extent we can never have a perfect set of measures.  In many ways it’s like predicting the weather, since we are talking about probability there is always a chance that the organization will beat the odds.  In fact, there isn’t one of these measures that I couldn’t construct a convincing argument against.  What’s important is to use those of these measures that make sense and use them in conjunction with each other.  One correlation does not a pattern make, but when we look at multiple areas of risk and analyze them in a holistic context we can find a more useful way to measure safety than counting bodies and broken bones.

 

 

Advertisements

#economics, #education, #engineering, #ergonomics, #injury-prevention, #institutionalization, #lagging-indicators, #leading-indicators, #measuring-safety, #misleading-indicators-for-safety, #occupational-safety, #occupational-safety-and-health, #occupational-safety-and-health-law, #organizational-culture, #participation-in-safety-meetings, #personality-characteristics, #predictive-indicators, #predictive-measures-for-safety, #risk-assessment-act, #safety-at-work, #safety-metrics, #traffic, #transition, #worker-safety

Forget Injuries—They Have Nothing to Do With Safety

 

Workplace-Injuries

 

By Phil La Duke

Last week I posted an article defending (to some extent) Zero-Injury goals that touched off a powder-keg of on-line debate.  I have gone back and forth on the idea (on one hand zero-injury (or zero-harm, or zero-anything) goals don’t work very well (for a variety of reasons that I don’t feel like getting into right now) and on the other hand if our goal isn’t zero than how many people can we kill in the workplace and still call it a job well done?) until I finally landed on a position with which I can live: who cares?

Now before you start rounding up pitchforks and lighting torches to drive me from the village, hear me out. We as a society have been using injuries as proof of an unsafe workplace and the absence of injuries as proof of safety and nothing could be further from the truth.  Is it safe to leave a toddler home alone? Is it safe to walk around an unfamiliar and bad neighborhood at night? Why? After all most toddlers wouldn’t be harmed and most people don’t get mugged, and yet most people I’ve talked to agree that many practices like this (or using tools with the guards removed) aren’t safe.

So if we can agree that there are many, many activities that aren’t safe irrespective of the outcome, why do we persist in using injuries as the chief criteria for determining what is safe and what is unsafe? In some organizations safety professionals claim credit for saving lives simply because they reminded people not to die. In other organizations safety professionals are hammered by leaders for injuries that they didn’t cause, but failed to prevent.  Nobody much likes the system, and nobody wants injuries and fatalities. And yet we persist in chasing numbers that don’t matter and juking stats that tell us nothing about the safety of the workplace.

The Measurement Craze

Industries’ fierce desire to measure every element of the business is a by-product of the quality revolution, and in many people’s eyes, if you can’t measure it, it doesn’t exist. Unfortunately, the Safety function directly copied the Quality function’s approach to measurement.  In some respects this makes sense, because an injury is not unlike a defect; both are the outgrowth of a process defect (and no I won’t be baited into an argument over whether or not the cause is procedural, behavioral, or systemic. I’m sorry to disappoint but it really doesn’t matter to a meaningful degree—unless, of course you are selling some new whiz bang approach and you have to differentiate it from the pack). In other respects it makes no sense whatsoever, since, as we’ve already established while there is a quantifiable relationship between the absence of a defect (the part either is within the tolerance limits of its standard or it is not) there is no such quantifiable relationship between a worker’s safety and injuries.  Let me put that another way: we have a good understanding of what constitutes a defect (since we also have a clear understanding of the specifics criteria for an acceptable product) but we don’t have a clear understanding of the specifics of safety, that is, we don’t really have a clue how much risk a worker faces at any given moment so it’s tough to measure safety in any meaningful way. Many organizations have become so obsessed with measurement that they are losing site of the real purpose of the safety function: to help both the organization and the individual to make better choices when it comes to safety.

It’s About Risk

When we talk about safety we’re really talking about risk, that is, how probable is it that our workers will be injured in the normal course of their work days?  The gross misunderstanding of basic statistics in general, and probability in specific, lies at the heart of the trouble so many organizations have in tackling worker safety.  I know it sounds like heresy but in a real sense injuries have little to do with safety and in fact often distract the organization from the real task of lowering operational risk.  Individuals who would never gamble with company funds blithely roll the dice when it comes to the safe execution of work. If we continue to concentrate on injuries at the exclusion of risk we lull ourselves into the false sense of security and when we achieve a year with no injuries we throw a big party and pat ourselves on the back for a job well done, despite the very real risks that lurk unseen in our midst.

More and more companies are celebrating the hard work that seems to have paid off, when in reality, they don’t have a clue whether or not the results they are getting are a product of hard work, voodoo, divine intervention, or blind luck. As a friend of mine recently said to me, we don’t understand the real problem (process variation) so we invent a problem we can solve and call ourselves heroes.  Since we’ve solved the problem that we do understand (worker injuries) we no longer work to solve the real problem (process variation, i.e. risk of injury). The processes (and for the purposes of this post I am including both behaviors and systems in the word “processes”) continue to drift away from the standard and soon exceed the control limits.  Before too long we are operating completely on luck while congratulating ourselves for slaying the dragon of injuries.  By the way, it’s a myth that sooner or later our luck will run out; we’d like to believe it, and statistics support the belief that in most cases risk will catch up to us, but probability being what it is there is always a chance that the organization will keep humming along on a wing and a prayer and never have risk come up and bite it on the ass. It’s theoretically possible that an organization will survive on luck alone, but more often organizations who fight the injury battle continue to win (there is a lot of overlap between the efforts to end injuries and the efforts to reduce process variation) because of good luck will ultimately have a catastrophe that corresponds to some change in the workplace.  Those who understand risk know that given enough risk the probability of injury becomes so likely that for all intents and purposes a serious injury is certain, but the self-congratulatory organizations who trumpet their zero-injury achievements tend to ascribe causation to some external force that has nothing but timing to do with the spike in injuries.

Whose Job Is It Anyway?

Some argue it that the problem is that safety professionals don’t spend enough time “on the floor” or “on the site”. While certainly out of touch safety professionals that don’t understand the nature of the business can’t be effective, I doubt that’s the real issue.  Safety professionals need to be agents of change and continuous improvement, not safety know-it-alls who scurry through the worksite trying to “catch someone doing something safe”.  The front-line supervisor owns safety, which is not to say that that worker don’t have a role, in fact a central role, in safety. After all, it is their fitness to work, decisions, competence, commitment, and judgment that collectively create what we call “safety”.

But when speaking operationally, when everyone is responsible for safety (and that responsibility is not clearly delineated) effectively no one is responsible for it. Certainly the worker must be responsible for his or her safety, not just at work but everywhere. But the individual’s responsibility for safety does not obviate the supervisor’s responsibilities. As a former automotive production worker, trainer in healthcare, construction laborer, consultant, security guard, food service worker and more, I can honestly say that I had a tendency to focus on the rigors of my job and tried to do it as safely as possible. I took it for granted that other workers were doing their job safely, that my boss was ensuring that the equipment and facilities were safe to operate and work in and yes, that my boss was ensuring that my coworkers weren’t doing something that would get us all killed.

My bosses had the decision rights to intervene in unsafe situations that I flat out didn’t have (short of losing my job). I depended on my bosses to keep me safe from the things over which I had no direct control. Too many people believe that safety is the responsibility of the individual alone. Leaders play a key role in all of this and owning the safety of the area is far different from individual ownership of safety.

Consider this: Every day we as individuals go through life responsible for our own safety, and yet we take for granted that someone is acting on our behalf. Don’t believe me? I’m willing to bet that within the last month you (while firmly responsible for your own safety) ate a meal where: a stranger harvested the ingredients, another stranger delivered them to a restaurant, where they were accepted by a person we’ve never met who also decided that the ingredients were safe to use, another stranger prepped the ingredients for cooking, still another stranger cooked you a meal using utensils washed by another stranger who then placed the food on a plate (also washed by a stranger), it was then delivered to you by a stranger, and you ate it using silverware washed, yet again, by a stranger. If at some point you were to die (or merely get really sick) because there was some breakdown in the supply chain, would society have the right to say, “well you never should have trusted so many strangers so you deserve what you get”? of course not, and yet many people bemoan the worker’s lack of ownership of safety.

My point is that we often assume, as workers, that someone else has inspected the tools, made sure the machines are in good shape, checked to ensure my coworkers are fit to work, and in general has looked out for my safety, at least those things that I cannot practically do for myself.

I believe this is the role (primarily) of the first line supervisor. While everyone should have the right to stop work if they feel it is unsafe the front line supervisor often makes the choices that directly affect the safety of dozens of people.

#forget-injuries, #injuries, #measuring-safety, #worker-safety

Misleading Indicators

trash graphs

“If you don’t know where you’re going, how do you know you aren’t already there?”

By Phil La Duke

Nearly every safety professional worth his or her salt has been told that he or she needs to look at both leading and lagging indicators; it’s good advice, in fact, it’s advice I’ve given many times in articles and speeches over the years.  But in my last post (two weeks ago—I spent the last week at a customer site and with the travel travails I just couldn’t bring myself to hammer out a post, deepest apologies to my fans and detractors alike) I questioned the value of tracking (not reporting or investigating, mind you, just tracking) near misses.  Well, as you can imagine the weirdoes, fanatics, and dullards came out in droves to sound off and huff and puff about things I never said (reading comprehension skills are at a disgraceful low these days).  Not everyone one who reads my stuff is a whack-job however, and some of the cooler heads insisted that tracking near misses was important because near miss reporting is a key leading indicator; it’s not…and it is, but like so much of life, it’s complicated.

Near misses in themselves aren’t leading indicators; they are things that almost killed or injured someone, and most importantly, they are events that happened in the past.  Not that anything that happens in the past has to be automatically counted out as a lagging indicator, but unless you still cling to the idea proffered by Heinrich that there is a strict statistical correlation between the number of near misses and fatalities, near misses are no more a leading indicator than your injury rate, lost work days, or first aid cases.  They simply tell you that something almost happened, and nothing more.  Now some of you might try to argue that if you have ENOUGH near misses you are bound to eventually have a fatality, but that does hold up to careful scrutiny.  Leading indicators are often expressions of probability, and like the proverbial coin that is tossed an infinite number of times, the probability of the outcome does not change because of the frequency of the toss.  If you were to toss the coin 400 times and it came up tails, the probability that the 401st toss would come up heads is still 50:50. So knowing that tracking near misses doesn’t really shed any light on what is likely to happen mean we should stop investigating near misses? Certainly not, but we really do need to stop thinking that the data is telling us things that it isn’t.  On the other hand, near miss reporting is indeed a leading indicator; if we accept (as I do) that when people report near misses they: a) are more actively engaged in safety day-to-day (and I suppose someone could argue that this doesn’t necessarily correlate) and b) the more the individual reports near misses the better he or she is at identifying hazards (again, this is a leap of faith, but  I believe in most cases this to be true.) So if you want to gage the robustness of your safety process I suppose the level of participation in near miss reporting is a good indicator.

The whole exercise got me thinking about indicators, and how often safety professionals (and everyone else on God’s green Earth for that matter) tend to be mislead by data because of the erroneous belief that the data is saying things that it isn’t.

Causefusion

Regular readers of my blog will recognize the concept of “causefusion”.  The term was coined by Zachery Shore in his book, Blunder: Why Smart People Make Bad Decisions which he uses to explain how people mistake correlation and cause-and-effect.  According to Shore, causefusion works something like this[1]: People who floss their teeth live longer than people who don’t floss or who floss irregularly therefore flossing your teeth makes you live longer.  It makes sense, right? Yes, except that it is wrong.  There are other possibilities for this correlation, for instance, isn’t it possible that people who are more interested in their health overall might be more likely to floss regularly? In a world where eager safety professionals provide data to Operations people who are hungry for quick fixes, Causefusion happens a lot; and it’s a real danger because it leads us away from the true causes of injuries and may blind us to real shortcomings in our processes.

Another way that we can be lead by indicators is the paradigm effect. When we think of the word “paradigm” we think of the definition, “a typical example” or “viewpoint”, but in the world of science paradigm there is another, lesser known definition, “a worldview underlying the theories and methodology of a particular scientific subject” Joel Barker pointed out how damaging paradigms (in the scientific sense) can be.  Barker believed that there were many instances where the worldview is so powerfully believed that any new evidence that does not support the worldview is ignored. Consider the dangers of ignoring critical new information relative to worker safety because you believe in a particular tool or methodology so strongly that you can’t even consider another viewpoint.

A third way that we mislead ourselves is when we see patterns that aren’t there.  This phenomena is wonderfully described in another book that I really believe is important to the world of safety, Why We Make Mistakes: How We Look Without Seeing, Forget Things in Seconds, and Are All Pretty Sure We Are Way Above Average by Joseph T. Hallinan. According to Hallinan—and the latest in brain research supports his contention—the human brain tends to see patterns even where there are none.  So in cases where safety professionals desperately seek answers and are under pressure to initiate action, the pressure to see patterns where there are none can be extreme.

Perhaps the most misleading indicator is one of the most common: zero recordables.  Too often safety professionals (and operations, as well, for that matter) see a trend of recordables as evidence that they are at far less risk of injuries and fatalities than they are.  This isn’t to say that they AREN’T at less risk, but there isn’t anything more than a correlation between the two elements; they might be good but they are just as likely to be lucky.


[1] The example is mine and mine alone, don’t get all huffy and bother Shore.

#lagging-indicators, #leading-indicators, #measuring-safety, #phil-la-duke, #phil-laduke, #philip-la-duke, #philip-laduke, #predicting-injuries, #safety-measures, #worker-safety