Updated as of January 22, 2016

Estimate impact of data quality:

• Access the Imperfect Data Impact Calculator to find out whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve

Increase quality of the data

…if the “imperfect data impact calculator” shows that the issues in your data may point you to a different action than suggested in the report. You might consider:

• Get current cancer screening data from your EMR: The QIDSS have developed standardized EMR queries for cancer screening. Try them. Now that they are developed, they should take very little time to run on an ongoing basis (rather than just once a year for reporting purposes). The data might not be directly comparable to what is in D2D (because it is from a different time-period and may have more information about patient eligibility for screening). However, it will give you a sense of how your team is doing over time. More importantly, having a list of specific patients that might be overdue for screening gives your team something concrete to do now about something they care about (i.e. patients).
• Record exclusion criteria more completely and more consistently on both cancer screening test documentation and in the EMR.
  • e.g., Exclude patients who have had cervical, endometrial or ovarian cancer and patients who have had a hysterectomy Ask your patients about their participation in screening in hospital labs and update the EMR manually to provide a more accurate estimate of screening performance locally, even if this gap remains in the central database.
• Devise a mechanism to formally, consistently record invitations for screening for these and other cancers, to make it easier to extract screening information that incorporates patient choice.
• Perform a manageable sized chart audit (e.g. 10% of eligible patients) and double check that appropriate screening was conducted and recorded in the chart or that patient choice to forego screening is noted.
  • Consider hiring a student to help generate a list of eligible patients and check against information in their chart.
• Other ideas: please share!

Additional information for estimating the impact of data quality for this measure:

The data are almost certainly not a definitive estimate of your team’s actual performance in the area of access.  However, they might be “good enough” to help you decide if your team needs to improve access or not.  To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes are a problem with your team.  Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement.  To do this, work with your clinical leaders and staff to establish an approximate impact of data quality – i.e. is the data quality issue causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is. Plug that number into the “imperfect data impact calculator”.  It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve. You may find it hard to generate consensus about the possible impact of data quality issues on the level of performance shown in the D2D report.  In that case, try the following options:

• Track the next 10 (or 20 or other small number) encounters with patients who meet the inclusion criteria for cancer screening to get a better estimate of the extent of the data quality issues (i.e. check their chart for any record of screening, if absent ask patient whether they have been screened and educate patient on the purpose of screening, record patient choice, especially where the choice to not screen is made).Estimate how many of your patients get their tests done in hospital laboratories and estimate what the screening rate is among those patients.
• Estimate how many of your patients decline screening when offered.
• Extract data from the EMR to determine how many eligible patients were screened. Consult with your QIDSS to ensure the definitions used for the data extraction are consistent with those being developed and deployed across the QIDSS network of approximately 150 AFHTO member organizations. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.

• If none of the above is helpful, consider instead experimenting with possible “error” rates to see how much error (i.e. TWICE or HALF or 10% of some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D. If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality. Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D.

If the “imperfect data impact calculator” points to the same decision (e.g. a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve. The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team.  If your data are not “good enough”, you may then consider taking action to improve your data quality, before or at the same time as you try to improve processes of care.

Updated as of January 22, 2016 Assuming you have established that the data are good enough to direct action AND that improving performance in this area is a priority for your team, you may wish to discuss the following options with your clinical leaders, Quality Improvement committees, team staff and/or patients:

• Interventions to improve cancer screening rates using the CCO Cancer Screening Quality Improvement Toolkit designed specifically for Family Health Teams (.doc)
• Identify and set an improvement target and work towards it using the change ideas presented in HQO’s Primary Care Practice Group Report (to be released by HQO early in 2016)
• Contact your peers to determine their performance and work with them to either spread any processes they find have helped them or collaboratively test some new changes that might work for you AND your peers.
• Learn about what other family physician leaders are doing working as part of the Provincial Primary Care and Cancer Network
• Sign your physicians up for monthly screening reports via CCO SAR. Once they get through the sign-up process, most physicians agree that these reports are very helpful, especially if you or they have trouble getting or trusting your EMR data for cancer screening.

Updated as of January 22, 2016 Since SAMI score is essentially a description of patient primary care needs, it is not a reflection of quality of care and there is no “target” measure teams could or should strive for.  Therefore, any improvement efforts are probably most usefully targeted at data quality, not at changing the SAMI score for a team.

Updated as of January 22, 2016

Increase quality of the data

…if addressing SAMI score accuracy is an immediate priority for your team.

• Work with engaged clinicians to increase the specificity of diagnosis data in your billing processes.
• Other ideas: please share!

Updated as of January 22, 2016

• The SAMI score displayed is the average SAMI score for teams contributing data for this indicator that are among the “peer group” you selected.  If you didn’t select a peer group, then it is the average SAMI score for all teams contributing data for the indicator.
• For teams who signed up for the HQO Primary Care Practice Group Report, your SAMI score can be found by logging in through the HQO portal and accessing the additional excel worksheet (addendum to the core report). If your team did not sign up for this report, you will likely not know what your SAMI score is – prepare for the next iterations by signing up for the Group-Level report.
• The ‘average’ patient or population has a SAMI score of 1.0.
  • A SAMI score of 1.40 can be interpreted as an expected need for primary health care that is 40% higher than in the average patient.
  • A SAMI score of 0.88 can be interpreted as a 12% lower expected need.
• Patients who have very complex needs for specialized care (ie oncologist for cancer, endocrinologists for diabetes) might not have higher than average needs for PRIMARY care and therefore may not contribute to a higher SAMI score.
• Among CHCs, where SAMI scores have been reported for several years, some patient populations have scores of nearly 3.0 (very high) with scores of 1.4 considered to be low.
• The range of SAMI scores among FHTs contributing data to D2D 2.0 was 0.81 to 1.23 with a provincial range of 0.95 to 1.84 (depending on the type of primary care model).
• Primary care documentation can theoretically affect SAMI score. If providers routinely use the same, non-specific code for visits by patients for different issues (e.g. “visit for medication renewal” instead of a more specific diagnosis-related code) the SAMI score could theoretically under-estimate the needs for primary care. However, the scoring system has been validated in both Ontario and Manitoba and shown to be very stable, even with the current state of primary care documentation.

Updated as of January 22, 2016 Assuming you have established that the data are good enough to direct action AND that improving performance in this area is a priority for your team, you may wish to discuss the following options with your clinical leaders, Quality Improvement committees, team staff and/or patients:

• Contact your peers to determine their performance and work with them to either spread any processes they find have helped them or collaboratively test some new changes that might work for you AND your peers.

Updated as of January 22, 2016

Estimate impact of data quality:

• Click here to access the Imperfect Data Impact Calculator
• Please see below for more information about this tool.

Increase quality of the data

…if the “imperfect data impact calculator” shows that the issues in your data may point you to a different action than suggested in the report. You might consider:

• Administering your patient survey more often to capture input from patients more than once a year and possibly from patients not recently in the office.

• Revising your team’s patient survey to align with standard questions and/or add additional (standard) questions, even if it is one question at a time.
  • See the HQO Primary Care Patient Experience Survey
  • Tips and Tools to Capture the Patient’s Experience in Primary Care using the NHS’s Experience-Based Design (EBD) Methodology
  • What are they thinking? Approaches, Ideas and Tools to Measure Patient Experience in Your Practice 
• Using new tools to capture and track patient experience survey data to improve the efficiency of administrating surveys, and especially improve the timing and quality of patient responses (e.g. administer surveys by email, phone, tablets – all integrated with your EMR)
  • Extending the EMR with Patient Tablets: Using Interactive, Point-of-Care Patient Surveys in the Waiting Room to Generate Clinical Content and Save Time
  • Using an automated patient reminder service and survey to collect information on patient experiences
  • Add some SaaS to your patient experience survey
• Participate in TRANSFORM
• Other ideas: please share!

Additional information for estimating the impact of data quality for this measure:

The data are almost certainly not a definitive estimate of your team’s actual performance in the area of access.  However, they might be “good enough” to help you decide if your team needs to improve access or not.  To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes are a problem with your team.  Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement.  To do this, work with your clinical leaders and staff to establish an approximate impact of data quality (i.e. is it the data quality issue that is causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is?) Plug that number into the “imperfect data impact calculator”.  It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve. You may find it hard to generate consensus about the possible impact of data quality issues on the level of performance shown in the D2D report.  In that case, try the following options:

• Track the next 10 (or 20 or other small number) encounters to get a better estimate of the extent of the data quality issue.  Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report.  Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Compare to other sources of data to see if the rate with other/better data is TWICE or HALF or 10% (or some other number) of the rate in the report.  Plug that number into the “imperfect data impact calculator” and proceed accordingly.  Other sources of data may include your scheduling system (e.g. actual appointment data), your EMR (queries or chart audits on a small number of patients), other pre-existing reports (CPCSSN, EMRALD, SAR, Physician Profile, MOHLTC, QIP, commercially provided or others) or personal interviews with a few patients, to name a few.
• If none of the above is helpful, consider instead experimenting with possible “error” rates to see how much error (i.e. TWICE or HALF or 10% of some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D.  If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality.  Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D.

If the “imperfect data impact calculator” points to the same decision (e.g. a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve.  The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team.  If your data are not “good enough”, you may then consider taking action to improve your data quality, before or at the same time as you try to improve processes of care.

Updated as of January 22, 2016

Estimate impact of data quality:

• Access the Imperfect Data Impact Calculator to find out whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve
• Please see below for more information about this tool.

Increase quality of the data

…if the “imperfect data impact calculator” shows that the issues in your data may point you to a different action than suggested in the report. You might consider:

• Most of the work to improve data quality for this indicator lies in refining the definitions as the data are captured via administrative information systems across all health care sectors and thus beyond the influence of primary care providers.
• Consider generating a more local estimate of continuity based on who patients see in your team, as indicated by your EMR data.  Consider collaborating with the QIDSS group to improve definitions of encounter types and provider types so a more local, team-based measure can be extracted from the EMR.

Review list of “rostered” patients with each physician and identify patients who are likely virtually rostered (ie see the doctor frequently but are not rostered) for consideration for formal inclusion in the roster.

Additional information for estimating the impact of data quality for this measure:

The data are almost certainly not a definitive estimate of your team’s actual performance in the area of access.  However, they might be “good enough” to help you decide if your team needs to improve access or not.  To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes are a problem with your team.  Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement.  To do this, work with your clinical leaders and staff to establish an approximate impact of data quality – i.e. is the data quality issue causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is. Plug that number into the “imperfect data impact calculator”.  It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve. You may find it hard to generate consensus about the possible impact of data quality issues on the level of performance shown in the D2D report.  In that case, try the following options:

• Determine how many of the patients visiting your team are not formally rostered but might be virtually rostered and estimate how many of them usually see the same doctor. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Attempt to estimate how many patients would have seen their own physician, in the absence of efforts to increase same/next day access.  This will almost certainly be a judgement call, rather than an “estimate” in the truest sense of the word. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report.  Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• If none of the above is helpful, consider instead experimenting with possible “error” rates to see how much error (i.e. TWICE or HALF or 10% of some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D.  If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality.  Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D.

If the “imperfect data impact calculator” points to the same decision (e.g. a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve.  The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team.  If your data are not “good enough”, you may then consider taking action to improve your data quality, before or at the same time as you try to improve processes of care.

Updated as of January 22, 2016

• Team, in this indicator, is “physician group” ie FHO or FHN, not FHT.
• Virtual rostering assigns patients to the primary care physician that provided the highest dollar amount of services within a defined set of primary care services. Your team may not be aware of which patients have been virtually rostered to them so may erroneously think that these patients are not “their” patients. Hence, your team’s sense of how many of “their” patient’s visits were made to a provider in the team may be different than the rate shown in D2D.
• Visits to health care providers other than physicians are not included in this measure, however this does not necessarily skew the measure.
  • g. if a patient visits a primary care team 10 times and sees a physician 8 times, regardless of whether or not it was their “own” physician , they will score 100% (8 out of 8) for “regular care provider – team”. If, however, they visit 10 times, receiving care from multiple providers but only saw a physician once, they would still score 100% on this measure, regardless of whether they say their “own” physician
• Teams with part-time physicians and teaching teams may have developed strong relationships between physicians to jointly care for patients, such that patients may feel equally comfortable and familiar with more than one physician. This principle is at the core of team-based care and is reflected in higher performance on the “regular care provider – team” vs. “regular care provider – individual” measure across most teams in D2D 2.0.

Updated as of January 22, 2016

Recall Bias:

• There is a chance that patients may not recall how long they waited for an appointment the last time they visited their provider.
  • There could be bias in either direction, with the patient wanting to be positive (i.e. under-estimating the time) or the patient being generally unsatisfied and therefore over-estimating the time.
• There is likely inconsistency in the recall of patients regarding whether the person they saw was their “own” provider or someone else on the team.

Sampling Bias:

• The level of access to care in your team might be lower than indicated by D2D because it only considers the access of patients who eventually had an appointment.
• Patients who didn’t get an appointment soon enough (or when they wanted it) might never have shown up and instead may have either gone nowhere or to a walk-in clinic, an Emergency Department or another team. Therefore, these patients would never have been included in the survey.

Patient Choice:

• Performance on this measure does not necessarily reflect patient choice because some patients would PREFER to have an appointment set several days in advance, instead of on the same or next day, as it makes it easier for them to plan.
• The measure might also be counter-intuitive to the known desire of patients to see their “own” regular provider.  Efforts to increase the proportion of patients getting appointments within the same/next day might be counter-productive to efforts aimed at helping patients see their “own” regular provider as much as possible.

Measurement Error:

• In spite of best efforts, there is likely inconsistency in the wording of the question between teams.

Other Issues with Current Definition:

• The measure as it is currently defined does not distinguish between appointments for an urgent issue vs. appointments for more elective concerns such as well-baby check-ups, follow-ups and preventive visits.
• The measure is counter-intuitive to the purpose of interprofessional team-based care when the wording specifies access to an appointment with the patient’s“own” regular provider. Team-based care suggests that anyone in the team could and should be able to help the patient, whenever they need it.