CSPS Strategic Directions Initiative Integrated Summative Evaluation: Summary Report

May 2018

Integrated Summative Evaluation – Final – Summary Report

 Report summary

Background and context

In November 2014, the Treasury Board (TB) approved the Strategic Directions Initiative (SDI, or the Initiative) as a phased project over 3 years (2014–15 through 2016–17) to examine the delivery and management of learning at the Canada School of Public Service (CSPS, or the School), as well as programming, the underlying business model, and the supporting strategies needed to implement a new model. The Initiative aimed to put in place a more accessible, relevant and responsive learning platform. The Initiative included plans to modernize the School's business operations and change the funding model to establish a new approach to learning. In this new approach, departments would focus on mandate-specific learning while the School would deliver a relevant, common curriculum for the public service, and advances a public service-wide culture of learning.

The past 3 years saw considerable activity in terms of consultations with federal government departments and agencies, the phasing in of the new funding model, the revision of the curriculum, and the development of supportive learning technologies and business intelligence systems. As of the end of 2016–17, public servants can now access the School's online platform, GCcampus, with its suite of learning products, from anywhere internet access is provided.

Evaluation purpose and scope

An evaluation of the Initiative was undertaken to provide neutral, credible, and timely feedback to School decision-makers based on the commitments made to the Prime Minister and the Treasury Board in 2014. Two evaluation exercises were conducted. A formative evaluation providing a mid-term assessment of the Initiative was delivered in December 2016. The formative evaluation focused on implementation and addressed questions primarily related to SDI activities, outputs and immediate outcomes. The CSPS Strategic Directions Initiative: Integrated Summative Evaluation focuses on questions related primarily to intermediate and long-term outcomes; specifically, the summative evaluation examined, post-SDI, the adequacy of School funding and the appropriateness of School authorities, the extent to which learner satisfaction, knowledge and application of knowledge increased, the relevance of the School's curriculum, the impact of SDI on the range of CSPS learning products, the extent to which a common, core curriculum was established, and the School's performance in terms of reporting on training outcomes. The summative evaluation also examined the question of how the School can measure, in the future, the outcomes of its learning ecosystem.

Lines of evidence

Findings from 5 lines of evidence were synthesized to address each outcomes review and funding model review question.

Data collection methods included a document review (over 150 documents were reviewed, including internal School documents ranging from project plans to reports, as well as various tracking tools) and the collection of related administrative data pertaining to product registrations and evaluations recorded by the School from 2013–14 through 2016–17. Interviews were conducted with 45 key informants (17 external senior interviewees, 18 additional external interviewees and 10 School interviewees). Finally, a literature review was conducted on methods to measure and report on the learning outcomes of the School's learning ecosystem. A review of the School's funding model was also conducted as a 5th line of evidence.

Key evaluation findings

Conclusions and recommendations

1. The new CSPS funding model has been successfully adopted, and planned reference levels and spending authorities appear to be appropriate. Because activities undertaken as part of SDI are still stabilizing and will continue to do so for at least 2 more years, and future demand is difficult to predict so early in the life span of the new model, it is premature to assess with certainty the future resource requirements of the School. That said, it would appear that current reference levels for 2017–18 and beyond will be sufficient in the near term, with the caveat that further augmentations in School responsibilities might need to be accompanied by additional funding.

   Retaining the authority to collect fees and to carry over revenues from these fees to the next fiscal year, provides an appropriate level of flexibility for the School to efficiently and effectively implement the new business model.

   Recommendation 1. It is recommended that, in future, estimated costs associated with the development and delivery of learning become a distinct element of consideration in Treasury Board submissions and Cabinet requests.

2. The School has established a relevant core common curriculum, as defined by senior officials. The School made substantial progress toward the establishment of a relevant core common curriculum. The School's focus is seen to have been sharpened, and department and agency representatives have greater confidence that the School can meet basic learning needs, leaving to departments and agencies mandate-specific training.

   The core common curriculum is defined, in essence, as the topics and priorities deemed to be relevant to the performance responsibilities of a critical mass of public servants enterprise-wide, along with priorities identified at senior levels. The definition has not been systematically validated at the front-line level, however. Also, the idea of learner needs having to be "enterprise-wide" appears to have precluded training for significant subpopulations; officials in some departments and agencies believe their employees' needs fall largely outside the purview of the School's new definition, and are left having to provide required training themselves.

   Recommendation 2. It is recommended that the School, in consultation with senior officials as well as middle managers and front-line staff across all regions, continues to clarify, validate and refine the definition of its core common curriculum.

3. The School's range of learning products was extensively overhauled, improving accessibility and choice among learning modes. Under SDI the range of CSPS learning products changed substantially. The number of classroom courses fell from 262 to 87 while the number of events increased nearly fivefold to 372. The number of self-paced online learning products remained relatively stable (at 219 as of the end of 2016–17), but almost all were replaced with newer versions. In addition, new types of learning products were made available.

   Department and agency representatives are generally positive respecting the renewed range of learning products, noting significant improvements in terms of learning mode choice and in the accessibility of learning opportunities. The School appears better able to respond rapidly to emergent priorities. Concerns remain, however, related to the shift from a model emphasizing classroom-based training to one relying more on online learning activities. Fewer classroom options can lead to perceptions among users about possible seat shortages. Also, an overreliance on online modes can impede training for employees without ready access to the internet.

   Recommendation 3. It is recommended that the School continues to develop and optimize the range of learning modes associated with its learning ecosystem.

4. The School has yet to establish a robust outcomes measurement regime. CSPS has collected Level 1 evaluation data since long before SDI, and found through its Level 1 data tracking efforts that satisfaction with self-paced online learning products increased over the SDI period. Level 1 observations are, however, of the least interest to department and agency representatives who are keener to know the extent to which their employees are acquiring new knowledge and skills from their School experiences and the extent to which they are applying these learnings on the job.

   The School collects elementary knowledge acquisition and learning application data on most activities for which registration is required, plus detailed data on a small sample of learning activities each year, but it has not developed an analysis protocol sufficient to satisfy client organizations. Moreover, in light of the evolution of the School's learning ecosystem, a system that appropriately and effectively measures learning outcomes will need to go beyond traditional evaluation concepts (the Kirkpatrick model) and traditional evaluation data collection approaches. Departments and agencies must be involved in data collection. In short, a more comprehensive and strategic approach is required to meet the demands of departments and agencies, as well as the needs of the School, by which methodologies are adopted in line with evaluation needs.

   Recommendation 4. It is recommended that the School builds upon existing learning evaluation practices, and invests appropriately to create a comprehensive evaluation regime capable of collecting, analyzing, and reporting on data in relation to satisfaction, learning, application and other concepts according to the needs of users.

5. SDI has given the School an upgraded business intelligence system capable of generating a wide range of reports; internal reports have been well received, however, client organizations remain unsatisfied. As a result of the activities undertaken as part of the Initiative, the School has invested in an upgraded business intelligence system that, combined with existing data collection practices, is capable of providing a wide range of outcome metrics. School officials are generally satisfied with reports generated for internal use. Various iterations of external report models, however, have yet to fully satisfy the needs of client departments and agencies. Department and agency officials want monthly usage figures, information on completions and cancellations, and learning outcome metrics for their employees including statistical information such as response rates.

   Recommendation 5. It is recommended that the School improve its external reporting by establishing a system that generates reports that meet client requirements in terms of frequency, content, and format.

Management response and action plan

Contact us Subscribe to the GCLearning newsletter

---

Features

Embedding Risk Management into Decision-Making (TRN5-E44)

(November 18, 2024)

2024 Government of Canada Workplace Charitable Campaign: Quiz for a Cause – DM Trivia (WMT2-E16)

(November 28, 2024)

Practical Artificial Intelligence Series: Reimagining Government Communications and Access to Services (DDN1-E11)

(December 5, 2024)

Follow:

• Twitter
• YouTube
• LinkedIn
• GCLearning newsletter
• RSS