Assuming you have established that the data are good enough to direct action AND that improving performance in this area is a priority for your team, you may wish to discuss the following options with your clinical leaders, Quality Improvement committees, team staff and/or patients:

• Interventions to improve immunization rates through conversation with Public Health units and others
• Contact your peers to determine their performance and work with them to either spread any processes they find have helped them or collaboratively test some new changes that might work for you AND your peers.
  • Learn how the Queen’s FHT used quality improvement planning to exceed provincial targets for pediatric immunizations
  • Check out how the Markham team deals with measles outbreaks due to patients with out-of-date vaccinations

Estimate the impact of data quality

• Access the Imperfect Data Impact Calculator to find out whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve.
• Please see below for more information about this tool.

Increase the quality of the data

…if the “imperfect data impact calculator” shows that the issues in your data may point you to a different decision than suggested in D2D 3.0. You might consider the following:

• Work with engaged clinicians to increase the consistency of data entry related to immunization. Refer to the Standardized EMR queries for guidance on ways to increase consistency in data entry such that extraction of the data can be more efficient and accurate over time and between teams.
• Track your immunization rates on an ongoing basis vs just at year-end. Work with QIDS Specialists in the application of consistent queries across teams and EMRs to increase access to these data in an efficient way.
• Ask patients about their immunizations, partly to update the EMR and partly to engage them in an important discussion about preventive strategies. Tilbury District FHT uses a simple paper form for patients to share information regarding flu shots received elsewhere. A similar idea could be applied to childhood immunizations. Better yet, contact them directly to find out exactly how this works!
• Support AFHTO’s efforts to work for inclusion of a standard process for recording and reporting immunization data aligned with PHAC guidelines as part of the technical specification for EMRs. This is already in progress through the EMR Data Management committee and individual EMR Communities of Practice.
• Develop data sharing processes with public health units and other partners engaged in immunization to ensure accurate records regarding immunization status, not only for reporting purposes but also for better risk management for patients.
• Other ideas: please share!

Additional  information for estimating the impact of data quality for this measure:

The data are almost certainly not a definitive estimate of your team’s actual performance. However, they might be “good enough” to help you decide if your team needs to improve or not. To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes above are a problem with your team.  Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement. To do this, work with your clinical leaders and staff to establish an approximate impact of data quality – i.e. is the data quality issue causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is. Plug that number into the “imperfect data impact calculator”.  It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve. Click here to access the Imperfect Data Impact Calculator. You may find it hard to generate consensus about the impact of data quality issues on the level of performance shown in the D2D 3.0 report. In that case, consider the following options:

• Track the next 10 (or 20 or other small number) encounters to get a better estimate of the extent of the data quality issue. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.

• Estimate how many of your patients get their immunizations elsewhere. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly. [If you are having trouble estimating how many patients are getting immunizations/flu vaccines elsewhere you might want to use 43% as a ball park figure. This was the rate observed by Tilbury District FHT who use a simple paper form for patients to share information regarding flu shots received elsewhere.]

• Estimate how many of your patient’s decline immunization when offered. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.

• If none of the above is helpful, consider instead experimenting with possible “error” rates to see how much error (i.e. TWICE or HALF or 10% of some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D 3.0. If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality. Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D 3.0. If the “imperfect data impact calculator” points to the same decision (e.g. a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve. The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team. If your data are not “good enough”, you may then consider taking action to improve your data quality, before or at the same time as you try to improve processes of care.

Please refer to the D2D 3.0 Data Dictionary

• The D2D 3.0 definition fully aligns with Public Health criteria. This means Rotavirus is now included in the definition and EMR queries. It is not part of the preventive care bonus; therefore, performance may appear lower than in your preventive care bonus reports which exclude Rotavirus.
• The D2D definition does not reflect patient choice. Patients who decline immunization and therefore are not immunized may be the reason why your rates are lower than you expect. See “data quality actions” for ideas to examine the extent to which this is affecting your rates.
• Data for patients immunized outside of the primary care team (e.g. at a health unit) might not be recorded consistently in all EMRs and teams. The performance seen in D2D 3.0 might therefore under-estimate actual immunization rates. See “data quality actions” for ideas to examine the extent to which this is affecting your rates.
• Rates for teams with very few children in their panel may be more variable than rates based on larger eligible patient populations. For example, if 2 less children are immunized this year out of a population of 10 children, your immunization rate will drop by 20%. However, if 2 less children out of 100 are not immunized, your rate will only drop by 2%. So consider how many children are eligible for immunization when interpreting differences for your team year to year or relative to another team.
• D2D 3.0 includes data for all children as opposed to only rostered children. This might generate different rates than those the team might be used to seeing in reports based on rostered children only (e.g. the MOHLTC Preventive Care Target Population/Service Report (TPSR)). The MOHLTC provides eligible physicians in Patient Enrolment Models (PEMs) with a Projected Preventive Care Target Population/Service Report semi-annually, in April and September, to assist them in determining their Target Population and the delivery of preventive care services.
• The timing of D2D reporting may not coincide with the reporting time period for the MOHLTC Preventive Care Target Population/Service Report. There may be differences in rates related to these differences in time periods.
• Although a consistent definition was developed to create queries that were shared among members for purposes of extracting these data for D2D 3.0, it is possible that the extent to which these data were consistently recorded and therefore extracted in your team might vary. To explore this issue, look at the processes used in your team to record and extract immunization data and work to align it as much as possible with the standard process being developed by QIDS Specialists.

Please refer to PAGE 21 in the D2D 3.0 Data Dictionary VERSION 4  

Estimate the impact of data quality

• Access the Imperfect Data Impact Calculator to find out whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve.
• Please see below for more information about this tool.

Increase the quality of the data

…if the “imperfect data impact calculator” shows that the issues in your data may point you to a different decision than suggested in D2D 3.0. You might consider the following:

• Increase awareness of your team about the importance of having “clean” data in your EMR: Project ALIVE shows that having clean data in your EMR allows you to create quick and flexible reports to better inform your team about the needs of patients and which patients require follow-up care.
• Create a diabetes registry: identify patients with diabetes more accurately using the standard queries and processes developed by QIDS Specialists to get started on a diabetes registry
• Track and demonstrate your progress cleaning up your data to improve data quality. Before you start the cleanup process run a “coded” query to capture a baseline, then every few months re-run the query and plot your results over time. You may want to use a tracking form to help you document your progress.
• Consider hiring a student to help you clean up your diabetes data. Check page 26 in this handbook for details about cleaning up diabetes data.
• Once your diabetes registry is clean, run the D2D diabetes queries on an ongoing basis – don’t just wait till the end of the year. Do you notice any changes?
• Consider signing up for CPCSSN or EMRALD to get ongoing, patient-specific reports to help you help your patients manage their diabetes.
• Join an EMR CoP to share new tools and solutions to help you make better use of your EMR.

Additional  information for estimating the impact of data quality for this measure:

The data are almost certainly not a definitive estimate of your team’s actual performance. However, they might be “good enough” to help you decide if your team needs to improve or not. To determine if the data are “good enough” for that, estimate how likely it is that one or more of the issues outlined in the interpretive notes are a problem with your team. Then, run the “imperfect data impact calculator” to see if the issue(s) could lead to a different decision related to the need for improvement. To do this, work with your clinical leaders and staff to establish an approximate impact of data quality – i.e. is the data quality issue causing your performance to look like TWICE or HALF or 10% (or other number) less or more than it actually is? Plug that number into the “imperfect data impact calculator”. It will show you whether the data quality issue(s) you think you have would change your initial decision regarding the need to improve. Click here to access the Imperfect Data Impact Calculator. You may find it hard to generate consensus about the impact of data quality issues on the level of performance shown in the D2D 3.0 report. In that case, consider the following options:

• Track the next 10 (or 20 or other small number) encounters to get a better estimate of the extent of the data quality issue. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Estimate how many of your patients with diabetes have blood pressure (or HBA1c) recorded properly in the EMR. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• Estimate how many of your patients with diabetes are not coded in a consistent manner in your EMR. Perhaps the rate among these patients will shift your team’s overall rate to be TWICE or HALF or 10% (or some other number) of the rate in the report. Plug that number into the “imperfect data impact calculator” and proceed accordingly.
• If none of the above is helpful, consider instead experimenting with possible “error” rates to see how much error (i.e. TWICE or HALF or 10% of some other number) would be needed to change the decision made on the basis of the performance of the indicator in D2D 3.0. If, in the opinion of the team, such an amount of error is reasonable, then it may be worth considering efforts to improve data quality. Alternatively, if that amount of error is considered to be unlikely, then the data are likely good enough to support the initial decision regarding the need to improve, based on the performance shown in D2D 3.0.

If the “imperfect data impact calculator” points to the same decision (e.g. a need to improve or NOT) even after data quality issues are considered, the data are likely “good enough” to base your decision on regarding the need to improve. The next step is to consider strategies to improve, assuming the area of care measured by the indicator is a priority for your team.  If your data are not “good enough”, you may then consider taking action to improve your data quality, before or at the same time as you try to improve processes of care.  

• The Diabetes Care Score represents the % of diabetes measures (aspects of care) that a team’s patient population has achieved. For example, if your team’s score is .68, this means that your population or registry of patients with diabetes has achieved 68% of the 3 measures included in the calculation (HbA1C testing, HbA1C level and blood pressure level). In future iterations of D2D, the composition of the indicator will be modified to include other measures of diabetes care like foot and eye exams, based on increasing EMR maturity/data quality and capacity to access data on personalized targets.
• Your score may be low if you have a lot of patients with diabetes that have only one process/outcome measure within the appropriate target.
• Your score may also be low if you have patients with no measures in range, even though others have most of the measures in range.
• How you document and are able to access blood pressure and HBA1c data in your EMR will affect the numerator i.e. your score will be low if documentation is an issue for your team.
• The way your team documents diabetes diagnoses in the EMR affects your denominator (i.e. number of patients with diabetes). Your diabetes score may be over or understated depending on how “clean” your diabetes registry is.

REGISTER NOW: A joint webinar between AFHTO and the Ministry will be offered to review the Annual Operating Plan (AOP) timelines and expectations, with a specific focus on the Program Planning & Evaluation Framework, Indicator Catalogue and Schedule A reporting requirements for FHTs.

• Click here to register and pick the session you wish to attend:
  • February 10^th 12:00 – 1:30 pm OR
  • February 17^th 2:30 – 4:00 pm

(Note: the webinar is being offered twice to provide flexibility & maximize participation. Please register for one session only. In the event that you cannot attend, the webinars will be recorded and available on AFHTO website)

√  Annual Operating Plan (AOP): AFHTO is anticipating Ministry release of the AOP to FHTs/NPLCs by the first week of February. Expected submission deadline is mid-April.

√  Program Planning & Evaluation Framework: developed jointly between AFHTO member ED Work Group and the Ministry, the framework is intended to be a guide for FHTs and NPLCs to use when developing new or evaluating current programs, and to help promote the delivery of effective programs. The Framework will be a valuable reference to support teams in completing their program reporting requirements. (will be shared end of January)

√  Indicator Catalogue: the indicator catalogue is another supportive guide developed for FHTs/NPLCs to use when selecting meaningful measures for their programs that are based on clinical guidelines. The catalogue will enable teams to find sample indicators that can be used to measure progress on specific objectives and select indicators that align most appropriately with the goals of their programs. (will be shared end of January)

Privacy Webinar: Reminder – FREE Privacy Training & Tools
To assist our members in meeting and understanding the new privacy criteria set out by Office of the Information and Privacy Commissioner of Ontario , the following FREE training and tools will be made available:

•  A 1hr Privacy Training Webinar for Executive Directors – 12-1pm WEDNESDAY JANUARY 27^th 2016 – click here to register
• A 1hr Privacy Training Webinar for Board Chairs – 12-1pm WEDNESDAY FEBRUARY 3^rd 2016   –  click here to register
• Privacy Tools: to answer the top 5 privacy questions asked by FHTs/NPLCs and FHT/NPLC staff  – tools/templates to be released end of JANUARY 2016