Virtual Internships Step 7 - Programme Evaluation & Intern Appraisal

Virtual Internships Step 7 - Programme Evaluation & Intern Appraisal

The final step in any virtual internship programme should be programme evaluation.  As outlined in step one, the objectives of the programme should form the basis for any such evaluation.  When organisations invest time, money, and resources into a programme they expect to see tangible outcomes matched to the programme objectives.[1]

 

Programme Evaluation

Programme evaluation offers a key means to improve the design and delivery of any HR programme, to engage stakeholders in the programme and raise its profile and to measure the return on the investment in the programme14. Programme evaluation can be conducted at multiple levels and the level(s) of evaluation undertaken should be determined by the objectives. For example, through an L&D lens evaluation ranges from

  1. Reaction or satisfaction with the programme

     
  2. Learning of knowledge, skills, and attitudes

     
  3. Transfer and application of learning to the job

     
  4. Return on investment

 

Level Of Satisfaction With the Programme

While almost all programmes were evaluated in some manner, the approaches to programme evaluation were often limited and focused on Level 1 evaluation i.e. satisfaction with the programme. This is an important level of evaluation for organisations where a key objective of their internship was to give students a “a good experience” and therefore build the employer brand. Some frame this evaluation in terms of net promoter scores (NPS) or the positivity with which participants spoke about the organisation. This feedback was generally collected through surveys or informal feedback from interns and/or their managers. A range of approaches for managing feedback were evident ranging from monthly, end of the programme to year-on-year comparative feedback. This example was typical of how this feedback was collected and utilised.

 

“We send…all of our interns and our intern managers a survey at the end of the internship where we ask them questions around how satisfied they were etc…We ask a number of questions, but the number one is really around that experience. So how satisfied are the intern population and the manager population with the internship experience? And that's how my team evaluates whether or not we're doing a good job in terms of what we're offering from a programmatic point of view. One of our key measurements is experience. We anticipated a hit to our intern experience score [owing to the virtual programme]…and not at all. We’re still at, like, 98% for 2020 and that’s been amazing … Obviously of course I’m sure most interns would prefer to do an internship in an office where they can actually physically be collocated”

 

 

 

IDEA IN PRACTICE

Tracking Development

One telecoms company has developed a very structured approach to measuring intern progress and satisfaction with the programme. The intern programme manager meets each intern on a monthly basis to understand how they're finding the programme, where are they seeing the benefits of the internship in terms of their skills development etc. There is also a monthly pulse survey for interns. This was designed to understand how the programme was delivering on objectives, how interns were feeling and where returns were most evident. The questions on the survey were linked to the programme design priorities and monthly surveys test against these design principles. There was additional check-in at the mid-point and end of the programme. The mid-point check in focused on development and social leaning with an emphasis on self-reflection. At the end of the programme the check in is focused on interns playing back their intern experience in what they've done on the job, their development, and any self-directed learning they would have taken.


 

Levels 2 & 3: Learning and Learning Transfer & Application

Levels two and three in this model of evaluation relate primarily to learning transfer and application and given the prioritisation on learning varied from organisation to organisation, it was not surprising that the focus on these metrics was mixed.  Indeed, learning and transfer or application of learning to the job were also not formally assessed in the majority of organisations. (See the ideas in practice box above as an example of an effective approach.) This is an important level of evaluation for organisations where a key programme objective was to develop interns’ competence and to build their work-readiness. However, the reality was that many organisations measured outcomes at this level in a more ad-hoc way.  For example, feedback provided by immediate managers and project leaders represents a form of evaluation of intern’s performance in the in-company projects. Equally, feedback from panels observing end of programme internship project presentations represents a form of evaluation with regard to their internship project.

Formalising such measures does seem like a useful initiative, as a number of managers spoke about the importance of the rate of development evident in interns as key in hiring decisions at the end of the programme. A number of organisations also tracked engagement throughout the programme. For example, to what extent did interns engage with online resources, attend social and other events. These represented an important pulse check on engagement with the programme.

 



Level 4: ROI

Evaluation of ROI was generally limited. For most organisations, the objective of the programme was to use the internship programme to create a talent pipeline. Many organisations used conversion rates of interns to graduate hires as a key metric in terms of measuring the ROI of the programme.  For many organisations target conversion rates were as high as 50-70%. A number of organisations also tracked diversity and inclusion metrics in terms of the conversion rates for different demographic groups.

Some tensions between perceived programme objectives and the more suitable evaluation approaches did emerge. In one organisation the evaluation survey distributed to intern managers focused wholistically on the programme. However, the line manager we interviewed perceived that the feedback form did not reflect the priority of the programme for managers on the ground, which was to attract and retain suitable talent with specific competencies. This highlighted the importance of developing a shared understating of the objectives amongst key stakeholders in the organisation.

Although not raised as a KPI in any organisation, we suggest that organisations should track longer-term career outcomes of interns versus other routes of entry to the graduate programme. How do their careers play out relative to other hires for example? These longer-term outcomes would provide a very useful metric for organisations to track.

 

Intern Performance

Similar to programme evaluation, the approaches taken to appraising the interns were generally focused around gathering feedback from their line manager with regard to their daily work and work-related projects. The added importance of the line manager in virtual programmes was called out again here in terms of their role of engaging more actively and regularly in tracking performance with the intern due to the remote nature of working. This varied from informal check-in’s, which was the more common approach,  to a few organisations that had a more formal process, often driven more by the manager than HR.

“[the intern] doesn’t have formal goals updated on to our performance management system. However, as a manager, we created a document to list out his goals and we manage that as very much an organic document and the way I see it is this will be a really useful tool when he comes to pull together his CV so he can articulate really clearly what his goal was, how he met it and key outcomes, etcetera. We’re just trying to set him up as best we can for life after his college course.”

One manager who did not have a formal approach for managing the intern’s performance, did call out to the intern the expectations, deliverables and targets that would be expected of a graduate on the organisations graduate programme and used this as a means to motivate them to strive towards those goals during their internship.

The intern specific projects and related presentations also provided opportunity for HR, L&D, and panels of judges to evaluate the projects, the presentations, and the interns. Depending on the nature of the projects, performance of interns could be evaluated and compared here if the project had very precise measurable outcomes e.g. selecting the best stocks.

In contrast to how organisations approach the performance management of full-time employees, interns were generally not set and managed against precise performance or indeed learning and development objectives. One organisation did however set performance expectations for the intern which were the “same as other employees but toned down a little”. Interns were evaluated against key performance indicators around collaboration, development, growth in skillset, impact of their work and making connections. Another organisation set and tracked interns against formal performance objectives related to their role. A food organisation tracked performance, learning and knowledge transfer and it was in the remit of the line manager to set these objectives from the outset with the intern and review progress every 3 months. The more formal performance management of interns was considered useful to decisions about recruiting the intern for future positions and indeed fast-tracking this recruitment process. The decision on whether or not to set more formal objectives and goals depends on the objectives of the internship programme. If the programme objectives include intern performance and satisfaction with their jobs, then academic evidence consistently finds that setting specific, challenging goals improves performance substantially[2]. Combining this with feedback improves performance further still.

 

“Yeah, so we would actually … We would measure them the same way that we would measure across the organisation obviously at a more junior level, but all our measurement would be of our behaviours. So the “for the customer”, create the future, experiment and learn fast, and get it done together which is great for us because we're really measuring them on how they've performed in their role against our culture in terms of the collaboration piece. And it's not about just what they've done. It's about how they've done it as well, would be the key performance indicators for us. So we would gather that feedback near the end of the programme because that would then inform the decision to direct to offer for the likes of our grad programme”. 

 

 

 

ACTION CHECKLIST

Programme Evaluation

  1. Match the key objectives of the programme and key metrics to the evaluation method[3].
    • Intern experience: Measured by surveys of interns’ perceptions of programme. Supplemented by input from other stakeholders.
       
    • Graduate pipeline: Measured by translation of interns to graduate hires.
      • Track diversity metrics.
      • Track quality and quantity of applicants.
         
    • Learning and skills development:
      • Pre and post scores on training.
      • Manager feedback and ratings on intern projects or work tasks.
      • Intern self-evaluation.
      • Measure pre and post programme self-efficacy which is a reliable indicator of development in the absence of more specific programme measures of learning [4].
         
  2. Evaluate which elements of programme design, content and delivery were most effective in terms of programme satisfaction, learning and results.
    • Track longer term outcomes in terms of the career path of interns who are hired on internship programmes relatively to other hires.


Intern Performance

  1. Set specific goals for the intern matched to the objectives of the internship programme and the values of the organisation
    • Work, projects, learning, collaboration, networks, growth.
    • Use SMART goals.
       
  2. Regularly check-ins focused on progress, challenges, success.
     

  3. Provide feedback to the intern on work, progress, processes, and objectives.
     

  4. Complete an end-of-programme appraisal of the intern against goals.

 


[1] Garavan, T.N., McCarthy, A., Sheehan, M. et al., (2020), Training and Organisational Performance: A Meta-Analysis of temporal, institutional and organisational context moderators. Human Resource Management Journal, 1-26.

Phillips, J.J. & Phillips, P.P. (2011). Measuring ROI in Learning and Development: Case Studies from Global Organizations. Alexandria, VA: ASTD

[2] Centre for Evidence Based Management (2016) Rapid evidence assessment of the research literature on the effect of goal setting on workplace performance, CIPD.

[3] Garavan, T.N., Hogan, C., Cahir-O’Donnell, A., Gubbins, C. (2020), Learning & Development in Organisations: Strategy, Evidence and Practice, Oaktree Press. Chapter 13.

[4] Blume, B.D., Ford, J.K., Baldwin, T.T & Huang, J.L. (2010), Transfer of Training: A META-Analytic Review, Journal of Management, 36 (4), 1065-1105.