Updated as of January 22, 2016

• The EMR Data Quality (DQ) score is a snapshot of how well data are being recorded in your EMR based on measures of completeness (smoking status complete) and concordance (EMR/SAR screening rate comparison). Because the indicator currently only includes 3 data elements, it may not reflect your team’s general approach to data quality in your EMR. Work continues to further refine the measures of EMR data quality even as efforts are underway to improve it. The EMR DQ measure is an attempt to operationalize a data quality framework described by Bowen and Lau.  
• Your EMR DQ score may be low if the cancer screening data in your EMR does NOT match the CCO cancer screening data. This might be due to lab results not coming in a consistent or standard way, or due to variations in documenting cancers, hysterectomies etc. so these patients can be excluded.
• Your EMR DQ score might not represent what is happening across your whole team, if you were only able to get the CCO SAR and EMR cancer screening rates for a few doctors.
• Your EMR DQ score might actually not represent the quality of data in your EMR, if you do better or worse on other aspects of data quality/standardization in your EMR (e.g. using SNOMED coding for your patients with diabetes or COPD).
• Teams might think their EMR data quality is WAY better than their score shows. If this is the case, have a look at the data quality actions page for other ways of improving data quality. Also, try using the imperfect data quality impact calculator to determine if your data are good enough and whether to proceed with improving the quality of your data, or not.

Updated as of January 22, 2016 Cost has been identified as one of the priority measures for system-level performance by HQO and will therefore be eventually included in system-level performance reports.  In the meantime, D2D remains the only primary care reporting process to include per capita cost data. D2D 1.0 was the first time cost data were shared with primary care providers at a team-level, although these data have been used in research and policy decision-making for several years.  The inclusion of cost data fully embodies the intent of D2D to be a “START-egy”, a tool to get started at meaningful measurement in primary care.  As such, the main value of these data is to initiate conversations to refine the measure based on the wisdom of frontline primary care providers to make this measure meaningful and actionable over time.  It is possible that cost will function more as a system-level indicator than a metric for particular attention at the team-level.

• Unadjusted total costs do not take into account how sick patients are.  Consider focusing on ADJUSTED total costs to allow comparisons between teams to be more meaningful.

• Because costs for long term care are considerably higher than costs in most other categories, costs are broken down into 4 categories: primary care, services, settings and institutions (see technical notes). Further exploration with AFHTO members may help clarify the extent to which any of these categories are sensitive to primary care interventions.

Readers are referred to emerging research (Wodchis and Laberge and others, personal communication) on health care system costs which seems to indicate that differences in costs for patient care by different models, pre-dated the implementation of the models and thus may be related to factors beyond the model of care itself.

• For more information, refer to the Health System Performance Research Network’s Guidelines on Person-Level Costing Using Administrative Databases in Ontario

Updated as of January 22, 2016 Assuming you have established that the data are good enough to direct action AND that improving performance in this area is a priority for your team, you may wish to discuss the following options with your clinical leaders, Quality Improvement committees, team staff and/or patients:

• Get more current data to move beyond the idea that “the data are too old to make improvements” OR test out a new question that wasn’t previously included in your survey
  • Step 1: Work with a physician who is interested in improving patient experience or work through your ED to get the conversation started with your team’s Med Lead
  • Step 2: Suggest tracking patient experience for a small subset of patients (i.e. the next 10 patients that come in) or for a narrowly defined timeframe (i.e. whatever you get in the next 2 weeks).
  • Step 3: Have a look at the data to see how this current “snapshot” compares to the “old” data
  • Step 4: Take this opportunity to discuss sample size – see resources on web (i.e. you do NOT need hundreds of surveys to know how your patients are feeling and to start making improvements)
  • Step 5: Consider setting up a frequent, ongoing automated patient experience survey process (See Data Quality Actions – how to increase quality of the data using new tools to capture and track patient experience data)
• Consult these tools to help improve access and efficiency in your team
  • Join machealth – an HQO supported group that works on Advanced Access and Efficiency for Primary Care : review the online modules to better understand the concepts of Access and Efficiency and how to guide improvement efforts and implement sustainable changes in your team.
  • Access similar resources available through HQO`s Learning Community for Advanced Access and Efficiency for Primary Care
• Check out how your peers are doing by looking at the D2D report to determine their performance with access and connect with them to either spread any processes they find helpful or collaboratively test some new changes that might work for you AND your peers.
• Check out what your peers are doing to improve their patients’ experience of care
  • Presentations from past AFHTO conference :
    • Engaging Patients in Their Care (AFHTO 2014)
    • Clinical Innovations in comprehensive primary care (AFHTO 2014)
    • Improving Patient’s Experience of Care (AFHTO 2012)

Updated as of January 22, 2016

Estimate impact of data quality:

• Access the Imperfect Data Impact Calculator to find out whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve

Increase quality of the data

…if the “imperfect data impact calculator” shows that the issues in your data may point you to a different action than suggested in the report. You might consider:

• Get current cancer screening data from your EMR: The QIDSS have developed standardized EMR queries for cancer screening. Try them. Now that they are developed, they should take very little time to run on an ongoing basis (rather than just once a year for reporting purposes). The data might not be directly comparable to what is in D2D (because it is from a different time-period and may have more information about patient eligibility for screening). However, it will give you a sense of how your team is doing over time. More importantly, having a list of specific patients that might be overdue for screening gives your team something concrete to do now about something they care about (i.e. patients).
• Record exclusion criteria more completely and more consistently on both cancer screening test documentation and in the EMR.
  • e.g., Exclude patients who have had cervical, endometrial or ovarian cancer and patients who have had a hysterectomy Ask your patients about their participation in screening in hospital labs and update the EMR manually to provide a more accurate estimate of screening performance locally, even if this gap remains in the central database.
• Devise a mechanism to formally, consistently record invitations for screening for these and other cancers, to make it easier to extract screening information that incorporates patient choice.
• Perform a manageable sized chart audit (e.g. 10% of eligible patients) and double check that appropriate screening was conducted and recorded in the chart or that patient choice to forego screening is noted.
  • Consider hiring a student to help generate a list of eligible patients and check against information in their chart.
• Other ideas: please share!

Additional information for estimating the impact of data quality for this measure:

The data are almost certainly not a definitive estimate of your team’s actual performance in the area of access.  However, they might be “good enough” to help you decide if your team needs to improve access or not.  To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes are a problem with your team.  Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement.  To do this, work with your clinical leaders and staff to establish an approximate impact of data quality – i.e. is the data quality issue causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is. Plug that number into the “imperfect data impact calculator”.  It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve. You may find it hard to generate consensus about the possible impact of data quality issues on the level of performance shown in the D2D report.  In that case, try the following options:

• Track the next 10 (or 20 or other small number) encounters with patients who meet the inclusion criteria for cancer screening to get a better estimate of the extent of the data quality issues (i.e. check their chart for any record of screening, if absent ask patient whether they have been screened and educate patient on the purpose of screening, record patient choice, especially where the choice to not screen is made).Estimate how many of your patients get their tests done in hospital laboratories and estimate what the screening rate is among those patients.
• Estimate how many of your patients decline screening when offered.
• Extract data from the EMR to determine how many eligible patients were screened. Consult with your QIDSS to ensure the definitions used for the data extraction are consistent with those being developed and deployed across the QIDSS network of approximately 150 AFHTO member organizations. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.

• If none of the above is helpful, consider instead experimenting with possible “error” rates to see how much error (i.e. TWICE or HALF or 10% of some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D. If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality. Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D.

If the “imperfect data impact calculator” points to the same decision (e.g. a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve. The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team.  If your data are not “good enough”, you may then consider taking action to improve your data quality, before or at the same time as you try to improve processes of care.

Updated as of January 22, 2016 Assuming you have established that the data are good enough to direct action AND that improving performance in this area is a priority for your team, you may wish to discuss the following options with your clinical leaders, Quality Improvement committees, team staff and/or patients:

• Interventions to improve cancer screening rates using the CCO Cancer Screening Quality Improvement Toolkit designed specifically for Family Health Teams (.doc)
• Identify and set an improvement target and work towards it using the change ideas presented in HQO’s Primary Care Practice Group Report (to be released by HQO early in 2016)
• Contact your peers to determine their performance and work with them to either spread any processes they find have helped them or collaboratively test some new changes that might work for you AND your peers.
• Learn about what other family physician leaders are doing working as part of the Provincial Primary Care and Cancer Network
• Sign your physicians up for monthly screening reports via CCO SAR. Once they get through the sign-up process, most physicians agree that these reports are very helpful, especially if you or they have trouble getting or trusting your EMR data for cancer screening.

Updated as of January 22, 2016 Since SAMI score is essentially a description of patient primary care needs, it is not a reflection of quality of care and there is no “target” measure teams could or should strive for.  Therefore, any improvement efforts are probably most usefully targeted at data quality, not at changing the SAMI score for a team.

Updated as of January 22, 2016

Increase quality of the data

…if addressing SAMI score accuracy is an immediate priority for your team.

• Work with engaged clinicians to increase the specificity of diagnosis data in your billing processes.
• Other ideas: please share!

Updated as of January 22, 2016

• The SAMI score displayed is the average SAMI score for teams contributing data for this indicator that are among the “peer group” you selected.  If you didn’t select a peer group, then it is the average SAMI score for all teams contributing data for the indicator.
• For teams who signed up for the HQO Primary Care Practice Group Report, your SAMI score can be found by logging in through the HQO portal and accessing the additional excel worksheet (addendum to the core report). If your team did not sign up for this report, you will likely not know what your SAMI score is – prepare for the next iterations by signing up for the Group-Level report.
• The ‘average’ patient or population has a SAMI score of 1.0.
  • A SAMI score of 1.40 can be interpreted as an expected need for primary health care that is 40% higher than in the average patient.
  • A SAMI score of 0.88 can be interpreted as a 12% lower expected need.
• Patients who have very complex needs for specialized care (ie oncologist for cancer, endocrinologists for diabetes) might not have higher than average needs for PRIMARY care and therefore may not contribute to a higher SAMI score.
• Among CHCs, where SAMI scores have been reported for several years, some patient populations have scores of nearly 3.0 (very high) with scores of 1.4 considered to be low.
• The range of SAMI scores among FHTs contributing data to D2D 2.0 was 0.81 to 1.23 with a provincial range of 0.95 to 1.84 (depending on the type of primary care model).
• Primary care documentation can theoretically affect SAMI score. If providers routinely use the same, non-specific code for visits by patients for different issues (e.g. “visit for medication renewal” instead of a more specific diagnosis-related code) the SAMI score could theoretically under-estimate the needs for primary care. However, the scoring system has been validated in both Ontario and Manitoba and shown to be very stable, even with the current state of primary care documentation.

Updated as of January 22, 2016 Assuming you have established that the data are good enough to direct action AND that improving performance in this area is a priority for your team, you may wish to discuss the following options with your clinical leaders, Quality Improvement committees, team staff and/or patients:

• Contact your peers to determine their performance and work with them to either spread any processes they find have helped them or collaboratively test some new changes that might work for you AND your peers.