News and Updates
Important Information: Changes to methodology. Final Learner Responsive Success Rate Reports
The Final Learner Responsive Success Rate Reports for the Academic Year 2008/09 have been released today (25 January 2010). There are two important changes to methodology which have been implemented for the first time in these reports.
Please read the information below for details of the changes and how they affect the latest reports, and the impact on future reports.
Use of learning start date in matching ILR records
The 2008/09 success rate methodology for Learner Responsive provision introduced a technical change to the way that multiple learning aim records from across multiple academic years are appropriately matched together. The aim of this change was to increase the accuracy of the success rate calculation.
The essence of the change was that the specific “Learning Start Date” field from the ILR replaced the derived variable “Start Year”, to enable more accurate matching, consistent with guidance within the ILR Specification.
This change in methodology was first implemented in the initial 2008/09 success rates reports in December 2009. Subsequently, some learning providers have reported that their databases were generating random changes to some start dates for learning aims and that they were now unable to submit corrections to historic ILR data. As a result of this feedback the change in methodology and the technical change that supports it have been reviewed with the following outcomes:
· for learning aims records where the matching process only needs to look back to 2007/08 learning start date will be used
· where the matching process needs to look at ILR data earlier than 2007/08 start year will be used.
The above are effective immediately in the Final Learner Responsive Success Rate Reports for the Academic Year 2008/09, issued on 26 January 2010, and in future years’ reports until the earliest historic data used in the success rate calculation is 2007/08.
This means that use of the start year match will eventually be phased out and replaced with the learning start date matching process in subsequent years’ reports.
This approach was endorsed by the Harmonisation Group at its meeting on Monday 18th January 2010 and is deemed to be the fairest possible because it takes both of the following into account:
· it accepts the limitations on which ILR data providers can legitimately change to ensure consistency and remove errors
· it maintains the introduction of a greater accuracy in the matching process, which in turn leads to greater accuracy in the success rate results.
Matching records for generic learning aims
The 2008/09 success rate methodology for Learner Responsive provision introduced a technical change (in the form of a sub-routine) to remove duplicate learning aim records from the ILR data. This change was based on the assumption that these learning aim records had all been erroneously submitted.
This change was first implemented in the initial 2008/09 success rates reports in December 2009. Subsequently, some learning providers have reported that there were legitimate instances where learners had enrolled concurrently onto multiple learning aims with generic references. As a result of this feedback the sub routine which removed duplicates has been discontinued and a new sub routine has been introduced to ensure the maximum matching of ILR records. This means that all duplicate ILR records will remain in providers’ data sets for calculating the success rates in future.
This approach was endorsed by the Harmonisation Group at its meeting on Monday 18th January 2010.
This new matching process ensures maximum matching of ILR records whilst giving the benefit of doubt to multiple records matching rather than towards generating fails/withdrawals/non-achievers and potentially decrease the provider success rate result.
This maintains an important concept of the success rate process which is to report on the learning aim records that are submitted via the ILR without making any unnecessary assumptions that could potentially mask data errors and/or matters of poor data quality.
It will be prudent for the Harmonisation Group to review the identification and consideration of generic learning aims within a success rate context in future years, to further enable a maximisation of the accuracy of this important measure of providers’ performance.
