How I Learned to Stop Worrying and Love the Data

For more than a year, the service scores at the Moreno Valley Optometry department zigged and zagged in no discernable pattern. Asked whether receptionists were helpful and courteous, 100 percent of patients answering the Ambulatory Service Questionnaire gave the highest score one month.

But two months later, only 78 percent of respondents were that enthusiastic. Two months after that, scores were back up into the 90s. The huge swings were discovered in May 2011 by Stephanie Valencia, the department’s new manager, who excavated two years’ worth of data.

“We had never looked at it before,” she says. “There was no trend. The scores were inconsistent.” Worse, says Valencia, the feedback from the most recent months was headed “on a downhill streak.”

Working with labor co-lead Gina Hitt, an optician and a member of Teamsters Local 166, Valencia and the unit-based team gathered information and set a baseline. For two days in September, the medical assistants asked all patients whether they found the receptionists to be helpful and then tallied the results.

The team used these to measure the effectiveness of a rapid string of small tests of change. These included adding a smile, positive tone of voice and eye contact on successive days. Each of those days, Hitt and her colleagues asked patients whether their receptionist was courteous and helpful. With each successive effort, the chorus of “yes” got louder and more effusive.

The act of simply examining the service scores seemed to set the team on an upward trajectory: The April 2011 score of 79.55 percent jumped to 89.09 percent in September and then 92.73 percent in October.

 “It is so neat to see how involved people are,” Valencia says. “Everyone is in sync.”

So, that’s a happily-ever-after story, right? Once upon a time, there was an optometry team in Southern California that never looked at its service scores. Suddenly, team members learned their scores were inconsistent and heading in the wrong direction. They focused on key data and tried out small tests of change. Their new practices boosted the score. Everyone lived happily ever after.

This happens every day with every UBT throughout all of Kaiser Permanente.

Right?

Maybe not.

Some UBTs are adept at using data to guide their attempts to improve performance, whether it be raising service scores, reducing infections, creating a safer workplace or boosting attendance. But for others, fear and anxiety about data and numbers are a significant obstacle on teams’ path to high performance.

In order to qualify as a Level 4 team on the Path to Performance, the team has to collect its own data and review it to see whether changes are helping improve performance. To ascend to Level 5, teams must be measuring their progress using annotated run charts.

But what if you break out in a cold sweat and experience shortness of breath at the sight of anything vaguely resembling math or numbers? Do you simply resign yourself (and your team) to being roadkill on the Path to Performance?

No. Read on.

1) Realize you are plenty smart enough.  

Kaiser Permanente, like all large health care organizations, collects and stores vast amounts of data in a variety of complex databases and websites. It employs people with a huge variation in their knowledge of and comfort with data. Just because you’re not at ease with numbers now doesn’t mean you never can be.

Even Bob Lloyd, the executive director of performance improvement at the Institute for Healthcare Improvement, an independent nonprofit in Massachusetts, jokingly refers to statistics as “sadistics.”

Luckily, the data you will need to turbocharge your team’s efforts to improve performance is probably a lot less complex than you fear.

 “It’s not really ‘math’ with formulas, statistics and calculations,” says Michael Mertens, a Kaiser Permanente performance improvement mentor in Southern California. “It’s mostly about before and after, addition and subtraction.”

2) Whether you acknowledge it or not, you collect data every day.  

 “My role in the tests of change has been soliciting feedback from the patients,” says Hitt, the Moreno Valley optician. She didn’t need a computer program or spreadsheet. A piece of paper and pencil did the trick. 

 “We are all data collectors,” proclaims Stacy Dietz, the UBT consultant for regional operations in Southern California. “And every day, we alter our behavior based on data.” For instance, we ask, “What is the temperature outside?” Then we decide whether to wear a wool turtleneck or tank top. We ask, “What is the length of my commute?” Then we decide whether it makes more sense to drive or take the train.

If you can collect and analyze data to determine your wardrobe, you can also do it to improve the performance of your team.

3) Before diving into the numbers, focus on the “why.”

As the new Kaiser Permanente ads challenge viewers, “Find your motivation.” For unit-based teams, the Value Compass offers a handy cheat sheet on motivation: The patient is at the center. Every data point on every chart represents the impact—positive or negative—that a Kaiser Permanente team had on a patient.

IHI’s Bob Lloyd explains there are three distinct reasons in health care for collecting and examining data:

• For research, such as KP’s recent study that found women in their late 60s who break a bone are five times more likely to die within a year than women that age who do not break a bone.
• For judgment, a category that would include the federal government’s recent rankings of Medicare insurance plans on quality and service (several KP plans got five out of five stars). This category also includes scores that determine whether or not a medical center or department earns its Performance Sharing Program (PSP) bonus.
• For improvement.

This last is the reason UBTs should be collecting and examining data.

 “The purpose of measurement in quality improvement work is for learning, not judgment,” Lloyd says. 

Data answers questions like, “How are we doing right now?” “Over time, are we getting better? Or getting worse?” “Is our small test of change making a difference? Or not?” In the absence of data, we have a tendency to fall back on relying on guesses, gut instinct, anecdotes—and to blame or give credit to specific individuals, justifiably or not.

 “You need data. Otherwise, you don’t have any solid information,” Hitt says. “You just have word of mouth.”

4) Only gather the data you actually need.

The holy grail of data for UBTs is the run chart. Don’t let the name throw you. It’s simply a chart that tracks some number (say, a service score, or number of last-minute sick calls) over time (day, week, month, quarter).

 “The most crucial question to ask is, ‘What are the few, vital pieces of information that are important?’ ” says Dennis Benton, executive director of the Panorama City Medical Center in Southern California. Any graph or data set that requires its intended audience to get special training to read is probably too complex for the task at hand, he says.

 “You can do a quick, just-in-time training at a UBT meeting,” says Benton. “We do it in leadership rounds. I point to the graphs and talk about them.” 

Run charts make it clear at a glance how your team's tests of change are working. Use this tool to walk through how to make one.

4 1/2) But, get the data often enough to support your improvement efforts.

For most teams’ small tests of change, data that can be collected daily, weekly or—at most—monthly will be most useful. Waiting for quarterly reports is generally not going to cut it. The Moreno Valley Optometry department did not wait for the Ambulatory Service Questionnaire results—which are posted monthly—to come in. It’s called the Rapid Improvement Model, folks. Not the Slow-as-Molasses Improvement Model.

Bottom line: The data should be useful for the team and be determined by the team.

Keys to Liking Data

When data starts helping your team do its work better and improve performance—you’ll begin to find satisfaction in using it.

You might even become a fan.

At Panorama City Medical Center, executive director Dennis Benton and his staff prepare and email graphs on a regular schedule.

“If we’re a little tardy getting them out, people start calling me and saying, ‘Where are my graphs?’” he says. “We see them plastered on bulletin boards everywhere.” 

5) Think art class, not math class.

 “I hate numbers,” admits Jenny Yang, a receptionist at the Moreno Valley Optometry department and a member of the UBT’s representative group. When the notion of using service scores to guide improvement first came up, Yang says she told her teammates, “I’m not going to do it. Make someone else do it.”

To help others like Yang, Benton says, when it comes to data, “Make a picture out of it. I am a big believer in graphs. With a graph, you can say, ‘We dipped here. What is the reason? What can we do about it?’ You can look at a trend relative to the goal.”

 “Graphs are visual,” Valencia adds. UBT members have a variety of learning styles and preferences: “Everyone learns differently.”  

And think in terms of moving video, not still photographs that capture single moments in time. IHI’s Lloyd asks, would nurses measure an ICU patient’s vital signs only when the patient arrived and when she left the unit? Or would they monitor vitals constantly via a telemetry machine? The second option is better, so caregivers can intervene in real time to help the patient’s recovery.  

6) You didn’t like art class? How about creative writing?

Numbers can tell a story. “There is narrative in data,” says Nancy Duarte, the author of “Slide:ology” and “Resonate,” two popular books about how to give compelling and memorable presentations. “What makes the numbers go up and down? How big are the numbers? How do the numbers contrast with other information?”

Yang agrees. Graphs with data “give you key points, high points and low points and trends,” she says. As a member of the representative UBT, Yang—a member of Steelworkers Local 7600—sees herself as a storyteller: “My audience is the UBT. The graphs help UBT members make sense of everything.”

Hey, if you liked math class, more power to you. “I love math,” says Hitt. “I am a number cruncher. But for me, charts and graphs? Not so much.”

7) It’s OK to ask for help.

So that graph you pored over in your UBT meeting is still making you break out into a cold sweat?

 “It’s OK to find a safe place to say, ‘I don’t get this,’ ” says UBT consultant Stacy Dietz. That might not be in a big group, but it could be one on one with a trusted peer.

Mertens, the Southern California performance improvement mentor, says the best way to learn to use data is to try it out. At the request of Susie Bulf, a UBT consultant, Mertens led a training for UBT co-leads in Fontana on how to create a run chart. He led an in-class exercise using sample data—and then another exercise where each team used its own data.

 “You get over the anxiety by doing it the first time,” Mertens says.

Each KP region boasts a roster of experienced performance improvement mentors. In addition, most UBT consultants have had some training in performance improvement strategies.