Updated as of January 22, 2016

Estimate impact of data quality:

• Click here to access the Imperfect Data Impact Calculator
• Please see below for more information about this tool.

Increase quality of the data

…if the “imperfect data impact calculator” shows that the issues in your data may point you to a different action than suggested in the report. You might consider:

• Administer your patient survey in a way that includes patients not recently in the office.
• Revise your team’s patient survey to align with standard questions and/or add additional (standard) questions, even if it is one question at a time.
  • See the HQO Primary Care Patient Experience Survey
  • Tips and Tools to Capture the Patient’s Experience in Primary Care using the NHS’s Experience-Based Design (EBD) Methodology
  • What are they thinking? Approaches, Ideas and Tools to Measure Patient Experience in Your Practice
• Consider changing your patient experience survey process from once a year to an ongoing cycle using some of the new tools integrated with your EMR to enhance and track patient experience. Check out how other teams administer patient surveys (see below).  You may also want to consider international and provincial patient survey initiatives.
  • Using new tools to capture and track patient experience survey data to improve the efficiency of administrating surveys, and especially improve the timing and quality of patient responses (e.g. administer surveys by email, phone, tablets – all integrated with your EMR)
  • Extending the EMR with Patient Tablets: Using Interactive, Point-of-Care Patient Surveys in the Waiting Room to Generate Clinical Content and Save Time
  • Using an automated patient reminder service and survey to collect information on patient experiences
  • Add some SaaS to your patient experience survey
• Other ideas: please share!

Additional information for estimating the impact of data quality for this measure:

The data are almost certainly not a definitive estimate of your team’s actual performance in the area of access.  However, they might be “good enough” to help you decide if your team needs to improve access or not.  To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes are a problem with your team.  Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement.  To do this, work with your clinical leaders and staff to establish an approximate impact of data quality (i.e. is it the data quality issue that is causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is?). Plug that number into the “imperfect data impact calculator”.  It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve. You may find it hard to generate consensus about the possible impact of data quality issues on the level of performance shown in the D2D report.  In that case, try the following options:

• Track the next 10 (or 20 or other small number) encounters to get a better estimate of the extent of the data quality issue.  Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report.  Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Compare to other sources of data to see if the rate with other/better data is TWICE or HALF or 10% (or some other number) of the rate in the report.  Plug that number into the “imperfect data impact calculator” and proceed accordingly.  Other sources of data may include your scheduling system (e.g. actual appointment data), your EMR (queries or chart audits on a small number of patients), other pre-existing reports (CPCSSN, EMRALD, SAR, Physician Profile, MOHLTC, QIP, commercially provided or others) or personal interviews with a few patients, to name a few.
• If none of the above is helpful, consider instead experimenting with possible “error” rates to see how much error (i.e. TWICE or HALF or 10% of some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D.  If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality.  Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D.
• If the “imperfect data impact calculator” points to the same decision (e.g. a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve.  The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team.  If your data are not “good enough”, you may then consider taking action to improve your data quality, before or at the same time as you try to improve processes of care.

Updated as of January 22, 2016

Sampling Bias:

• The performance shown on these indicators in D2D is based only on patients either currently in the office or having just had a recent visit. It is possible that this may not be representative of patients who did not have appointments. For example, patients who feel they were not involved as much as they wanted or felt that the office staff were not courteous may be choosing to get care elsewhere (e.g. walk-in clinic, Emergency Department, other provider) or nowhere at all.
• Teams that do not feel able to do anything to improve patient experiences in these areas may decline to ask these questions to avoid setting false expectations among patients that their input will prompt changes. As a result, the performance level may represent only teams that would consider interventions to improve patient experience in these areas and thus may not be representative of all teams.
  • It could be an over-estimate of actual patient experience if one assumes that teams with good patient experience are more likely to consider interventions.
  • Or, it could be an under-estimate if one assumes that teams for whom patient experience is not good are more likely to be considering interventions to improve their performance in this regard.

Social Bias:

• There appears to be little difference in performance between patient experience survey (PES) questions over time for the same questions, with most showing, a high percent of patients with positive experiences. This may be real and it may also be what is known as “ceiling effect” (i.e. bunching of responses at the top end of the scale), possibly because patients want to be positive about their experience with their provider.

Watch a detailed tour on the components of the D2D 2.0 membership-wide report: This includes sections on the following (start and end times included): 1) Getting Started on D2D (Length 6:33) 2) Quality roll-up indicator (Length 4:39) 3) Cost indicator (Length 2:52) 4) Capacity indicator (Length 0:44) 5) Core D2D measures:

• • Introduction to Core D2D measures (Length 1:03)

• • Patient centeredness (Length 1:48)

• Effectiveness (Length 2:24)

• • Access (Length 3:57)

• Integration (Length 1:55)

6) Comparing D2D 1.0 vs 2.0 (Length 1:45) 7) Exploratory Indicator: 7 day follow-up (Length 2:53) 8) Comparative data (29:55-33:53) [youtube e0YFp6rcTVw] Click here to go back to D2D 2.0 Orientation and Supporting Materials

Updated as of January 22, 2016

Estimate impact of data quality:

• Access the Imperfect Data Impact Calculator to find out whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve
• Please see below for more information about this tool.

Increase quality of the data

…if the “imperfect data impact calculator” shows that the issues in your data may point you to a different action than suggested in the report. You might consider:

• Most of the work to improve data quality for this indicator lies in refining the definitions as the data are captured via administrative information systems across all health care sectors and thus beyond the influence of primary care providers.
• Consider generating a more local estimate of continuity based on who patients see in your team, as indicated by your EMR data.  Consider collaborating with the QIDSS group to improve definitions of encounter types and provider types so a more local, team-based measure can be extracted from the EMR.

Additional information for estimating the impact of data quality for this measure:

The data are almost certainly not a definitive estimate of your team’s actual performance in the area of access.  However, they might be “good enough” to help you decide if your team needs to improve access or not.  To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes are a problem with your team.  Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement.  To do this, work with your clinical leaders and staff to establish an approximate impact of data quality – i.e. is the data quality issue causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is. Plug that number into the “imperfect data impact calculator”.  It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve. You may find it hard to generate consensus about the possible impact of data quality issues on the level of performance shown in the D2D report.  In that case, try the following options:

• If you have part-time physicians in your team, try the following:
  • extract data from the EMR to determine the number of visits that patients of part-time physicians made to other physicians in the team (i.e. NOT seen by their “own” doctor). Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report.  Plug that number into the “imperfect data impact calculator” and proceed accordingly.
  • If not possible to extract all these data from the EMR, extract as much as possible and project mathematically based on the number of patients, the average number of visits for all patients and how many hours the part-time physician normally works.  Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report.  Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Determine how many of the patients visiting your team are not formally rostered but might be virtually rostered and estimate how many of them usually see the same doctor. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Attempt to estimate how many patients would have seen their own physician, in the absence of efforts to increase same/next day access.  This will almost certainly be a judgement call, rather than an “estimate” in the truest sense of the word. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report.  Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• If none of the above is helpful, consider instead experimenting with possible “error” rates to see how much error (i.e. TWICE or HALF or 10% of some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D.  If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality.  Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D.

If the “imperfect data impact calculator” points to the same decision (e.g. a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve.  The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team.  If your data are not “good enough”, you may then consider taking action to improve your data quality, before or at the same time as you try to improve processes of care.

Updated as of January 22, 2016 Assuming you have established that the data are good enough to direct action AND that improving performance in this area is a priority for your team, you may wish to discuss the following options with your clinical leaders, Quality Improvement committees, team staff and/or patients:

• Consider sharing your stories about how you are using these data to join other teams working to improve performance related to this measure and/or request other teams who have enjoyed success in improving in this area to identify themselves and provide suggestions.
• Consider exploring the Choosing Wisely campaign for change ideas and share your ideas about inclusion of some of the Choosing Wisely metrics in subsequent iterations of D2D.

Updated as of January 22, 2016

Estimate impact of data quality:

• Access the Imperfect Data Impact Calculator to find out whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve

Increase quality of the data

…if the “imperfect data impact calculator” shows that the issues in your data may point you to a different action than suggested in the report. You might consider:

• Most of the work to improve data quality for this indicator lies in refining the definitions as the data are captured via administrative information systems across all health care sectors and thus beyond the influence of primary care providers.  Primary care contributions to improving data quality would therefore be thoughtful reflections on refinements to the definition, to be considered for presentation in subsequent iterations of D2D.
• Other ideas: please share!
  
   

Additional information for estimating the impact of data quality for this measure:

The data are almost certainly not a definitive estimate of your team’s actual performance.  However, they might be “good enough” to help you decide if your team needs to improve or not. To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes above are a problem with your team.  Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement.  To do this, work with your clinical leaders and staff to establish an approximate impact of data quality – i.e. is the data quality issue is causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is.  Plug that number into the “imperfect data impact calculator”.  It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve.

You may find it hard to generate consensus about the possible impact of data quality issues on the level of performance shown in the D2D report. In that case, try the following options:

• Explore the proportion of your patients who are long term care residents to estimate how much impact their costs are having on your overall team cost.  Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report.  Plug that number into the “imperfect data impact calculator” and proceed accordingly.

• Alternatively, consider instead experimenting with possible “error” rates to see how much error (i.e. TWICE or HALF or 10% of some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D. If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality.  Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D.

If the “imperfect data impact calculator” points to the same decision (e.g. a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve. The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team.  If your data are not “good enough”, you may then consider taking action to better understand the issues affecting data quality, before or at the same time as you try to improve processes of care.

Updated as of January 22, 2016

• The EMR Data Quality (DQ) score is a snapshot of how well data are being recorded in your EMR based on measures of completeness (smoking status complete) and concordance (EMR/SAR screening rate comparison). Because the indicator currently only includes 3 data elements, it may not reflect your team’s general approach to data quality in your EMR. Work continues to further refine the measures of EMR data quality even as efforts are underway to improve it. The EMR DQ measure is an attempt to operationalize a data quality framework described by Bowen and Lau.  
• Your EMR DQ score may be low if the cancer screening data in your EMR does NOT match the CCO cancer screening data. This might be due to lab results not coming in a consistent or standard way, or due to variations in documenting cancers, hysterectomies etc. so these patients can be excluded.
• Your EMR DQ score might not represent what is happening across your whole team, if you were only able to get the CCO SAR and EMR cancer screening rates for a few doctors.
• Your EMR DQ score might actually not represent the quality of data in your EMR, if you do better or worse on other aspects of data quality/standardization in your EMR (e.g. using SNOMED coding for your patients with diabetes or COPD).
• Teams might think their EMR data quality is WAY better than their score shows. If this is the case, have a look at the data quality actions page for other ways of improving data quality. Also, try using the imperfect data quality impact calculator to determine if your data are good enough and whether to proceed with improving the quality of your data, or not.

Updated as of January 22, 2016 Cost has been identified as one of the priority measures for system-level performance by HQO and will therefore be eventually included in system-level performance reports.  In the meantime, D2D remains the only primary care reporting process to include per capita cost data. D2D 1.0 was the first time cost data were shared with primary care providers at a team-level, although these data have been used in research and policy decision-making for several years.  The inclusion of cost data fully embodies the intent of D2D to be a “START-egy”, a tool to get started at meaningful measurement in primary care.  As such, the main value of these data is to initiate conversations to refine the measure based on the wisdom of frontline primary care providers to make this measure meaningful and actionable over time.  It is possible that cost will function more as a system-level indicator than a metric for particular attention at the team-level.

• Unadjusted total costs do not take into account how sick patients are.  Consider focusing on ADJUSTED total costs to allow comparisons between teams to be more meaningful.

• Because costs for long term care are considerably higher than costs in most other categories, costs are broken down into 4 categories: primary care, services, settings and institutions (see technical notes). Further exploration with AFHTO members may help clarify the extent to which any of these categories are sensitive to primary care interventions.

Readers are referred to emerging research (Wodchis and Laberge and others, personal communication) on health care system costs which seems to indicate that differences in costs for patient care by different models, pre-dated the implementation of the models and thus may be related to factors beyond the model of care itself.

• For more information, refer to the Health System Performance Research Network’s Guidelines on Person-Level Costing Using Administrative Databases in Ontario

Updated as of January 22, 2016 Assuming you have established that the data are good enough to direct action AND that improving performance in this area is a priority for your team, you may wish to discuss the following options with your clinical leaders, Quality Improvement committees, team staff and/or patients:

• Get more current data to move beyond the idea that “the data are too old to make improvements” OR test out a new question that wasn’t previously included in your survey
  • Step 1: Work with a physician who is interested in improving patient experience or work through your ED to get the conversation started with your team’s Med Lead
  • Step 2: Suggest tracking patient experience for a small subset of patients (i.e. the next 10 patients that come in) or for a narrowly defined timeframe (i.e. whatever you get in the next 2 weeks).
  • Step 3: Have a look at the data to see how this current “snapshot” compares to the “old” data
  • Step 4: Take this opportunity to discuss sample size – see resources on web (i.e. you do NOT need hundreds of surveys to know how your patients are feeling and to start making improvements)
  • Step 5: Consider setting up a frequent, ongoing automated patient experience survey process (See Data Quality Actions – how to increase quality of the data using new tools to capture and track patient experience data)
• Consult these tools to help improve access and efficiency in your team
  • Join machealth – an HQO supported group that works on Advanced Access and Efficiency for Primary Care : review the online modules to better understand the concepts of Access and Efficiency and how to guide improvement efforts and implement sustainable changes in your team.
  • Access similar resources available through HQO`s Learning Community for Advanced Access and Efficiency for Primary Care
• Check out how your peers are doing by looking at the D2D report to determine their performance with access and connect with them to either spread any processes they find helpful or collaboratively test some new changes that might work for you AND your peers.
• Check out what your peers are doing to improve their patients’ experience of care
  • Presentations from past AFHTO conference :
    • Engaging Patients in Their Care (AFHTO 2014)
    • Clinical Innovations in comprehensive primary care (AFHTO 2014)
    • Improving Patient’s Experience of Care (AFHTO 2012)