For UPDATED technical notes, please see page 28 in the Data Dictionary.

Interpretive Notes

Tips to help you understand the data and put it in context.

• The EMR Data Quality (EMR-DQ) score is a snapshot of how well data are being recorded in your EMR based on measures of completeness (smoking status complete) and concordance (comparison of EMR-based colorectal and cervical cancer screening to CCO rates reported in the Screening Activity Report (SAR), coded diagnosis of diabetes). Because the indicator currently only includes 4 data elements, it may not reflect your team’s general approach to data quality in your EMR. Work continues to further refine the measures of EMR data quality even as efforts are underway to improve it. The EMR DQ measure is an attempt to operationalize a data quality framework described by Bowen and Lau.
• Your EMR DQ score may be low if the cancer screening data in your EMR does NOT match the CCO cancer screening data. This might be due to lab results not coming in a consistent or standard way, or due to variations in documenting cancers, hysterectomies etc., so these patients can be excluded.
• Your EMR DQ score might not represent what is happening across your whole team, if you were only able to get the CCO SAR and EMR cancer screening rates for a few doctors.
• Your EMR DQ score might actually not represent the quality of data in your EMR, if you do better or worse on other aspects of data quality/standardization in your EMR (e.g. using SNOMED coding for your patients with diabetes or COPD).
• Teams might think their EMR data quality is WAY better than their score shows. If this is the case, have a look at the data quality actions for other ways of improving data quality.

Want to compare your data across iterations? The EMR-DQ Score has changed across multiple iterations of D2D, with the addition of new components. Moreover, teams may choose to enter data on more or different components from one iteration to another. For this reason, a simple comparison of EMR-DQ scores across two or more iterations of D2D is not necessarily a reliable indicator of whether your team’s data quality is getting better, getting worse, or staying the same. You could be comparing “apples to oranges” – i.e., a score based on one particular set of measures to a score based on a larger (or simply different) set of measures. The good news is you still have the necessary data to make an accurate “apples to apples” comparison. Here’s how:

1.  Access your D2D data (from the Data-Input Toolkit) for each iteration you want to compare.
2. Identify the measures for which you submitted data in all of these iterations.
3. For each iteration you wish to compare, compile a straight average of the scores for each measure to calculate your modified EMR-DQ score.

Example:  In D2D 3.0, you submitted EMR-DQ data for cervical and colorectal cancer screening and smoking status.  In D2D 4.0, you submitted data on cervical cancer screening, colorectal cancer screening, smoking status, and coded diagnosis of diabetes. In D2D 5.0, you submitted data on cervical cancer screening, smoking status, coded diagnosis of diabetes, and coded diagnosis of COPD (but not colorectal cancer screening, coded diagnosis of CHF, or coded diagnosis of depression)

• To compare D2D 3.0 and D2D 4.0, you will add your scores for cervical cancer screening, colorectal cancer screening, and smoking status, then divide the total for each iteration by 3.
• To compare D2D 3.0 and D2D 4.0, you will add your scores for cervical cancer screening, smoking status, and coded diagnosis of diabetes, then divide the total for each iteration by 3.
• To compare D2D 3.0 through D2D 6.0, you will add your scores for cervical cancer screening and smoking status, then divide the total for each iteration by 2.

Of course, every team is unique, and there may be other factors making it challenging to track your EMR-DQ performance across multiple years. For assistance, please contact improve@afhto.ca.

Data Quality Actions

Tips to help you understand the quality of your data and, if necessary, take steps to improve it. Estimate impact of data quality

• Access the Imperfect Data Impact Calculator to find out whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve.

The data are almost certainly not a definitive estimate of your team’s actual performance. However, they might be “good enough” to help you decide if your team needs to improve or not. To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes are a problem with your team. Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement. To do this, work with your clinical leaders and staff to establish an approximate impact of data quality – i.e., is the data quality issue causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is? Plug that number into the “imperfect data impact calculator.” It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve. You may find it hard to generate consensus about the impact of data quality issues on the level of performance shown in the D2D report. In that case, consider the following options:

• Track the next 10 (or 20 or other small number) encounters to get a better estimate of the extent of the data quality issue. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Estimate how many of your patients eligible for cervical or colorectal cancer screening have tests or labs recorded properly in the EMR. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Estimate how many of your patients have a smoking status coded in a consistent manner in your EMR. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Estimate how many of your patients with diabetes have a diagnosis code in the appropriate place in the EMR. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Estimate how many of your patients with COPD have a diagnosis code in the appropriate place in the EMR. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Estimate how many of your patients with depression have a diagnosis code in the appropriate place in the EMR. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Estimate how many of your patients with CHF have a diagnosis code in the appropriate place in the EMR. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• If none of the above is helpful, consider instead experimenting with possible “error” rates to see how much error (i.e., TWICE or HALF or 10% or some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D 3.0. If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality. Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D 3.0.

If the “imperfect data impact calculator” points to the same decision (e.g., a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve. The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team. If your data are not “good enough”, you may then consider taking action to improve your data quality, before or at the same time as you try to improve processes of care. Increase quality of the data The goal of this indicator is to inform and motivate action to improve data quality in the EMR and also serve as a measure to monitor progress for actions to improve data quality such as those described for all the other D2D indicators. If the “imperfect data impact calculator” shows that the issues in your data may point you to a different action than suggested in the report, you might consider:

•  Exploring other aspects of EMR data quality: The actions to improve data quality do not need to be limited to cancer screening smoking status or diabetes diagnosis data, although these might be of immediate interest. Accurately identifying deceased patients could be a focus, or accurately recording lab results in appropriate fields and in appropriate language, or coding diagnoses consistently are other areas.
• Confirming patient volume data: Confirm which patients are alive and active (by whatever definition you use in your EMR/team) as rates of indicators based on incorrect denominators (i.e., all patients ever seen, vs. just the patients who are alive and who are active) will be incorrect.
• Measuring your EMR Maturity: Have your QIDS Specialist and/or physicians complete  OntarioMD’s EMR Progress Assessment tool (EPA) to measure the current and desired level of EMR maturity. By completing the EPA, teams will be connected with the resources of OntarioMD’s EMR Practice Enhancement Program to help enhance their EMR skills, efficiency and data quality.
• Providing feedback to clinicians:
  • Create lists of patients with particular characteristics identified via standardized EMR queries for their review/confirmation. These have been developed by QIDS Specialists and the EMR CoPs (e.g., childhood immunization, COPD, diabetes, and CHF; a query for depression is coming soon).
  • Participate with external partners like CPCSSN, EMRALD, and eCHN to use your EMR data in more meaningful ways
• Hiring a student to help you clean up your data (see suggestions in this handbook for cleaning up your roster and smoking/alcohol status).
• Participating in the EMR communities of practice and join your peers in developing new tools and processes for standardizing access to EMR data. Contact us to get connected.
• Tap into external resources to support clinical process changes using PDSAs from HQO (also check with your QIDS Specialist).