Updated as of January 22, 2016

Estimate impact of data quality:

• Click here to access the Imperfect Data Impact Calculator
• Please see below for more information about this tool.

Increase quality of the data

…if the “imperfect data impact calculator” shows that the issues in your data may point you to a different action than suggested in the report. You might consider:

• Administering your patient survey more often to capture input from patients more than once a year and possibly from patients not recently in the office.

• Revising your team’s patient survey to align with standard questions and/or add additional (standard) questions, even if it is one question at a time.
  • See the HQO Primary Care Patient Experience Survey
  • Tips and Tools to Capture the Patient’s Experience in Primary Care using the NHS’s Experience-Based Design (EBD) Methodology
  • What are they thinking? Approaches, Ideas and Tools to Measure Patient Experience in Your Practice 
• Using new tools to capture and track patient experience survey data to improve the efficiency of administrating surveys, and especially improve the timing and quality of patient responses (e.g. administer surveys by email, phone, tablets – all integrated with your EMR)
  • Extending the EMR with Patient Tablets: Using Interactive, Point-of-Care Patient Surveys in the Waiting Room to Generate Clinical Content and Save Time
  • Using an automated patient reminder service and survey to collect information on patient experiences
  • Add some SaaS to your patient experience survey
• Participate in TRANSFORM
• Other ideas: please share!

Additional information for estimating the impact of data quality for this measure:

The data are almost certainly not a definitive estimate of your team’s actual performance in the area of access.  However, they might be “good enough” to help you decide if your team needs to improve access or not.  To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes are a problem with your team.  Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement.  To do this, work with your clinical leaders and staff to establish an approximate impact of data quality (i.e. is it the data quality issue that is causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is?) Plug that number into the “imperfect data impact calculator”.  It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve. You may find it hard to generate consensus about the possible impact of data quality issues on the level of performance shown in the D2D report.  In that case, try the following options:

• Track the next 10 (or 20 or other small number) encounters to get a better estimate of the extent of the data quality issue.  Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report.  Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Compare to other sources of data to see if the rate with other/better data is TWICE or HALF or 10% (or some other number) of the rate in the report.  Plug that number into the “imperfect data impact calculator” and proceed accordingly.  Other sources of data may include your scheduling system (e.g. actual appointment data), your EMR (queries or chart audits on a small number of patients), other pre-existing reports (CPCSSN, EMRALD, SAR, Physician Profile, MOHLTC, QIP, commercially provided or others) or personal interviews with a few patients, to name a few.
• If none of the above is helpful, consider instead experimenting with possible “error” rates to see how much error (i.e. TWICE or HALF or 10% of some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D.  If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality.  Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D.

If the “imperfect data impact calculator” points to the same decision (e.g. a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve.  The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team.  If your data are not “good enough”, you may then consider taking action to improve your data quality, before or at the same time as you try to improve processes of care.

Updated as of January 22, 2016

Estimate impact of data quality:

• Access the Imperfect Data Impact Calculator to find out whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve
• Please see below for more information about this tool.

Increase quality of the data

…if the “imperfect data impact calculator” shows that the issues in your data may point you to a different action than suggested in the report. You might consider:

• Most of the work to improve data quality for this indicator lies in refining the definitions as the data are captured via administrative information systems across all health care sectors and thus beyond the influence of primary care providers.
• Consider generating a more local estimate of continuity based on who patients see in your team, as indicated by your EMR data.  Consider collaborating with the QIDSS group to improve definitions of encounter types and provider types so a more local, team-based measure can be extracted from the EMR.

Review list of “rostered” patients with each physician and identify patients who are likely virtually rostered (ie see the doctor frequently but are not rostered) for consideration for formal inclusion in the roster.

Additional information for estimating the impact of data quality for this measure:

The data are almost certainly not a definitive estimate of your team’s actual performance in the area of access.  However, they might be “good enough” to help you decide if your team needs to improve access or not.  To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes are a problem with your team.  Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement.  To do this, work with your clinical leaders and staff to establish an approximate impact of data quality – i.e. is the data quality issue causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is. Plug that number into the “imperfect data impact calculator”.  It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve. You may find it hard to generate consensus about the possible impact of data quality issues on the level of performance shown in the D2D report.  In that case, try the following options:

• Determine how many of the patients visiting your team are not formally rostered but might be virtually rostered and estimate how many of them usually see the same doctor. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Attempt to estimate how many patients would have seen their own physician, in the absence of efforts to increase same/next day access.  This will almost certainly be a judgement call, rather than an “estimate” in the truest sense of the word. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report.  Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• If none of the above is helpful, consider instead experimenting with possible “error” rates to see how much error (i.e. TWICE or HALF or 10% of some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D.  If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality.  Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D.

If the “imperfect data impact calculator” points to the same decision (e.g. a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve.  The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team.  If your data are not “good enough”, you may then consider taking action to improve your data quality, before or at the same time as you try to improve processes of care.

Updated as of January 22, 2016

• Team, in this indicator, is “physician group” ie FHO or FHN, not FHT.
• Virtual rostering assigns patients to the primary care physician that provided the highest dollar amount of services within a defined set of primary care services. Your team may not be aware of which patients have been virtually rostered to them so may erroneously think that these patients are not “their” patients. Hence, your team’s sense of how many of “their” patient’s visits were made to a provider in the team may be different than the rate shown in D2D.
• Visits to health care providers other than physicians are not included in this measure, however this does not necessarily skew the measure.
  • g. if a patient visits a primary care team 10 times and sees a physician 8 times, regardless of whether or not it was their “own” physician , they will score 100% (8 out of 8) for “regular care provider – team”. If, however, they visit 10 times, receiving care from multiple providers but only saw a physician once, they would still score 100% on this measure, regardless of whether they say their “own” physician
• Teams with part-time physicians and teaching teams may have developed strong relationships between physicians to jointly care for patients, such that patients may feel equally comfortable and familiar with more than one physician. This principle is at the core of team-based care and is reflected in higher performance on the “regular care provider – team” vs. “regular care provider – individual” measure across most teams in D2D 2.0.

Updated as of January 22, 2016

Recall Bias:

• There is a chance that patients may not recall how long they waited for an appointment the last time they visited their provider.
  • There could be bias in either direction, with the patient wanting to be positive (i.e. under-estimating the time) or the patient being generally unsatisfied and therefore over-estimating the time.
• There is likely inconsistency in the recall of patients regarding whether the person they saw was their “own” provider or someone else on the team.

Sampling Bias:

• The level of access to care in your team might be lower than indicated by D2D because it only considers the access of patients who eventually had an appointment.
• Patients who didn’t get an appointment soon enough (or when they wanted it) might never have shown up and instead may have either gone nowhere or to a walk-in clinic, an Emergency Department or another team. Therefore, these patients would never have been included in the survey.

Patient Choice:

• Performance on this measure does not necessarily reflect patient choice because some patients would PREFER to have an appointment set several days in advance, instead of on the same or next day, as it makes it easier for them to plan.
• The measure might also be counter-intuitive to the known desire of patients to see their “own” regular provider.  Efforts to increase the proportion of patients getting appointments within the same/next day might be counter-productive to efforts aimed at helping patients see their “own” regular provider as much as possible.

Measurement Error:

• In spite of best efforts, there is likely inconsistency in the wording of the question between teams.

Other Issues with Current Definition:

• The measure as it is currently defined does not distinguish between appointments for an urgent issue vs. appointments for more elective concerns such as well-baby check-ups, follow-ups and preventive visits.
• The measure is counter-intuitive to the purpose of interprofessional team-based care when the wording specifies access to an appointment with the patient’s“own” regular provider. Team-based care suggests that anyone in the team could and should be able to help the patient, whenever they need it.

Updated as of January 22, 2016 Please see PAGE 10 of the D2D 3.0 Data Dictionary VERSION 4

Please refer to page 23 in the D2D 3.0 Data Dictionary

Updated as of January 22, 2016 Please see PAGE 7 of the D2D 3.0 Data Dictionary VERSION 4 Breakdown of Cost Sub-Categories

1. Primary Care

• GP – FFS visits
• FHO/FHN capitation costs
• Non-FFS GP/FP visits

  2. Services

• Non-FFS radiation oncologists
• Non-FFS medical oncologists
• EDAFA non-FFS visits
• Other non-FFS visits
• OHIP non-physician cost
• OHIP lab cost
• NACRS ED
• Home care services costs
• ODB drug cost
• OHIP specialty physician FFS costs

3. Settings

• Inpatient (CIHI/DAD)
• Same day surgery (SDS)
• NACRS cancer
• NACRS dialysis

4. Institutions

• LTC cost
• Inpatient MH
• CCC cost
• Rehab (NRS)

Updated as of January 22, 2016 Indicators based on administrative data tend to be the oldest of all indicators in D2D. Improving the timeliness of administrative data is a priority for AFHTO and HQO and others. And in the meantime, there are things teams can do to use these “old” data to fuel current, local efforts to improve. Assuming you have established that the data are good enough to direct action AND that improving performance in this area is a priority for your team, you may wish to discuss the following options with your clinical leaders, Quality Improvement committees, team staff and/or patients:

• Collaboration with HealthLinks
• Use of Patient Navigators
• Since some of the reasons for readmission include lack of coordination or delivery of home care supports and poor patient compliance with discharge instructions, follow-up after hospitalization might help reduce readmissions. Ideas to increase follow-up include
  • Train administrative staff (e.g. reception) to call patients who were recently in hospital or ER to set up follow-up appointments, with or without a triage process informed by clinical staff to exclude those for whom follow-up is not needed.
  • Implement electronic reminders in EMR to prompt clinicians to decide if follow-up is necessary and if so, who should do it (i.e. physician, other clinician) how (i.e. phone or in-person) and when, so that the health team staff can initiate the follow-up process.
  • Share follow-up rates with providers (anonymously or shared within the team) for their review to identify potential areas to intervene to prevent readmissions, if possible.
• Contact your peers to determine their performance and work with them to either spread any processes they find have helped them or collaboratively test some new changes that might work for you AND your peers. To that end, consider reviewing the following presentations from past AFHTO conferences to find out more about how teams are keeping people at home and out of hospital
  • Reducing the revolving-door syndrome: hospital and primary care working together to reduce 30-day readmission rate for COPD and CHF patients
  • The evolution of telehomecare: targeting more chronic conditions and offering customized approaches
  • Aging at home: Interprofessional care to keep seniors home and out of hospital
  • Post-hospital transition of care: from hospital to family practice

Updated as of January 22, 2016

Estimate impact of data quality:

• Access the Imperfect Data Impact Calculator to find out whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve

Increase quality of the data

…if the “imperfect data impact calculator” shows that the issues in your data may point you to a different action than suggested in the report. You might consider the following:

• Sign up with OntarioMD to implement direct EMR access to hospital discharge data via Hospital Report Manager (HRM). Some teams already have this access via HRM or similar regional systems such as SPIRE, POI or TDIS. This process takes some time to implement so it is worthwhile to act as soon as possible to get the process started. Check out the Ontario MD site for progress reports on HRM implementation and stories about how teams are using it to improve work flow and patient outcomes.
• Make arrangements with the hospitals most commonly visited by your team’s patients to receive extracts of data from hospital discharges and ER visits. You can then and manually update the EMR with this information. > Many teams are already doing this with the help of their QIDS Specialists. Contact your QIDS Specialist or AFHTO QIDS program staff for help.
• Establish a communication process for direct notification of your team about hospitalizations on an individual-patient basis via fax, phone calls or other method. This also requires manual updating of the EMR. It also may involve some work on the part of the hospital and therefore involve some negotiation.
• Review hospital information systems to find hospitalizations of your patients and then manually update the EMR. This assumes appropriate permission to access these systems is in place as is often the case when your team’s physicians also attend at the hospital’s Emergency Department or inpatient units.
• Other ideas: please share!

Additional information for estimating the impact of data quality for this measure:

The data are almost certainly not a definitive estimate of your team’s actual performance in the area of access.  However, they might be “good enough” to help you decide if your team needs to improve access or not.  To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes are a problem with your team.  Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement.  To do this, work with your clinical leaders and staff to establish an approximate impact of data quality – i.e. is the data quality issue causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is. Plug that number into the “imperfect data impact calculator”.  It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve. You may find it hard to generate consensus about the possible impact of data quality issues on the level of performance shown in the D2D report.  In that case, try the following options:

• Track the next 10 (or 20 or other small number) encounters to get a better estimate of the extent of the data quality issue. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Estimate how many of your patients have chronic disease conditions that could be expected to result in hospitalizations as part of the normal course of the disease. Also, estimate how many of your patients are likely to be hospitalized for issues not usually associated with readmissions (e.g. birth, minor, elective surgeries). Compare to the proportion in other teams to get a sense of whether the difference is likely to lead to readmissions at TWICE or HALF or 10% (or some other number) of the rate in report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Estimate how many hospitalizations your team is notified of and, by extension, how many you are NOT made aware of in a timely way. Consider how many readmissions might be related to your team’s lack of awareness of the initial hospitalization and estimate whether that would lead to readmissions at TWICE or HALF or 10% (or some other number) of the rate in report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• If none of the above is helpful, consider instead experimenting with possible “error” rates to see how much error (i.e. TWICE or HALF or 10% of some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D. If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality. Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D.

If the “imperfect data impact calculator” points to the same decision (e.g. a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve. The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team. If your data are not “good enough”, you may then consider taking action to improve your data quality, before or at the same time as you try to improve processes of care.