Quality roll-up indicator: An “acceptable” measure of comprehensive, patient-centered primary care

The quality roll-up indicator, introduced in D2D 2.0, shows promise as a measure of comprehensive, patient-centred primary care. Statistically speaking, it is classed as an “acceptable” measure of quality; this means that while there is still room for improvement, this composite indicator really does reflect the quality of care being delivered. Read on to learn more about how the indicator works, what we’ve learned from it so far, and how we’re working to develop and refine it.

Questions asked and answered

The indicator was created as a “leap of faith,” with a goal of creating a single measure of quality that would reflect the comprehensive nature of primary care while honouring what is important to patients.  The idea was to compare quality to cost to better understand the value of team-based primary care. It sounded good in principle but would it actually work? Thanks to the over 100 teams who contributed data to D2D 2.0, with 60 contributing even more than the core D2D indicators, AFHTO is now able to answer that question and several more besides! Here’s what we learned when we analyzed all that data with the able support of Chris Meaney, statistician with UTOPIAN and the University of Toronto: Does the quality roll-up indicator “work”? Yes! First of all, it was possible to access enough data to calculate scores.  Second, there was enough data to assess the “reliability” of the measure.  Reliability is a statistical term that reflects how well the items included in a scale (or in this case a composite measure) represent the concept being measured. The analysis of D2D 2.0 data suggests that the quality roll-up indicator has what is called “acceptable” reliability –  see below how reliability is measured. However, our work on this is not done yet – i.e. it is not “excellent”. And at “acceptable”, it is good enough as a starting place to measure the overall quality of comprehensive patient centered care. How much data is needed to make the quality roll-up indicator work? About 14 indicators (see table below). The “acceptable” reliability described above is based on these. It could be that these are the only 14 indicators that matter. And it could be that data from a few additional indicators might help move the needle from acceptable closer to excellent. So while the immediate focus for D2D 3.0 is on the 14 listed below, teams that can provide more data are invited to do so, in hopes of making the quality roll-up indicator work even better.

Patient experience survey data	Administrative data (ie from ICES)
Reasonable wait for an appointment	% of physicians visits to same team
Patients involved in decision-making	Cervical cancer screening
Patient opportunity to ask question	Colorectal cancer screening
Providers spend enough time with patients	Mammograms (ie for breast cancer screening)
Availability of same/next day appointment	Physician billing claim for diabetes assessment
	30-day readmission
EMR data	Ambulatory-care sensitive hospitalizations
Childhood immunization	ED visits

Does higher quality comprehensive patient-centered care cost less? The data are looking very encouraging! Looking at the overall results teams with higher scores on the quality roll-up indicator had lower overall system costs. This is very promising – and it does not mean we draw a direct line between quality and cost yet.  There are a LOT of other things that are related to cost besides quality, not the least of which is the complexity of patients involved.  And the roll-up indicator is still “acceptable”, not excellent yet.  However, even when considering all of that, it is very encouraging to see hints of a relationship between high quality and lower costs.

Next steps

These are exciting but early days for the quality roll-up indicator.  There is much work yet to be done.  Next steps include the following:

• Getting better at telling the story of what and how the quality roll-up indicator works. This is a complicated way of measuring quality. It is distinctly different from the kinds of reports AFHTO members are used to making and seeing. It takes a bit of work to get familiar with the concept and members understandably have little spare time to do that work.  AFHTO will be working with QIDS Specialists and others to find ways to make this easier.
• Moving forward from “acceptable” towards “excellent”. This means getting more data from more teams for D2D 3.0.  Please watch ebulletins and connect with your QIDS Specialists to learn about how your team can help with that.
• Setting “threshold” levels of performance. To get to a place where the quality roll-up score means something concrete to individual teams, thresholds for good performance need to be set for each of the 14 (or more) indicators included in the score.  Please help set these thresholds by completing this survey by Nov 5.
• Learning more about the “domains” of quality. The theory (as described here) is that there are 6 domains in the relationship between patients and their providers that is at the heart of comprehensive primary care.  The analysis so far are not conclusive about these 6 domains.  Data from more teams is needed to sort this out definitively.  As noted above, please consider how your team can help with this.

What do the quality roll-up scores tell teams at this point?

The data teams submitted for D2D 2.0 made it possible to start to answer the three important questions above. However, in themselves, the scores are still too raw to support action to improve at the team-level.  This is not about being lazy or sloppy – the leap of faith members made in contributing these data made it possible to do the analyses that are necessary to identify and properly address these issues.  In subsequent iterations, as more of the questions get answered through the data contributed by teams, the quality roll-up indicator will work better and therefore be as useful at the team level as it currently is for the membership as a whole. For more information, including technical details of the statistical analyses please contact Carol Mulder or check out the quality roll-up indicator FAQs.

How is reliability measured?

Reliability is measured with a statistic called Cronbach alpha, the most important characteristic of which is that higher values are better.

Figure 1 shows that the roll-up indicator has some reliability even with just core D2D 2.0 indicators (0.587). This increases to 0.697 with the addition of the ICES data from the expanded set of measures listed below. Since these additional indicators are already available to any team that has requested ICES data for D2D 2.0, contributing them may generate increased value in the roll-up indicator without much additional effort. The reliability of the roll-up indicator increases to 0.846 with the addition of the “high priority” indicators listed below in bold.  This increase in reliability may need to be balanced by the extra effort associated with getting access to these data.  Contributing data from the lower-priority set of indicators does not increase reliability much at all (i.e. increase to 0.855), something teams might weigh against what might be considerable extra effort to access and contribute these data.