by Simon Junker and Lea Zanola – Results measurement is an issue that has already been on the agenda at the last f2f two years ago. By then, a working group with members from SDC and partners in Switzerland and in different partner countries decided to start the attempt to define a set of common outcome indicators (COI). For this year’s f2f we prepared a draft version of the instrument which I presented. The working group came up with four indicators:
- Gainful Employment
- Systemic Change
- Relevance of Training
- Outreach / Scale
Each of these indicators is defined and illustrated with practical indicators for measurement in the presentation:
[slideshare id=21223980&doc=ppt-coisinvsd-simonjunker-130515182853-phpapp02]
Trouble viewing? Try fullscreen mode or download the PDF here: COIs in VSD [PDF]
For a first field test we asked Siroco Messerli, Team Leader of HELVETAS Swiss Intercooperation in Nepal, to have a close look at the instrument and to evaluate its usefulness and practicability by applying it on the Employment Fund project (see ↓ infobox below). He presented the results of this exercise and made useful proposals for some adaptations in the instrument. Siroco’s recommendations will be included into the final version of the instrument, which will then be distributed to the network members, with the recommendation to use and test the instrument.
[slideshare id=21223937&doc=ppt-coisfieldtest-siroccomesserli-130515182551-phpapp02]
Trouble viewing? Try fullscreen mode or download the PDF here: COIs Field Test [PDF]
Infobox: The Employment Fund Project in Nepal
by Siroco Messerli – The Employment Fund (EF) was established jointly by the Government of Nepal and the Swiss Agency for Development and Cooperation (SDC) in 2008 and is managed by the Employment Fund Secretariat (EFS) as a project of HELVETAS Swiss Intercooperation. EF is currently funded by UKAid from the Department for International Development (DFID), SDC and the World Bank. The purpose of EF is to support the gainful employment of unemployed youths by imparting skills through private sector Training and Employment Service Providers (T&Es). The primary stakeholders of EF are unemployed, out-of-school youths from disadvantaged groups, aged between 16 to 40 years. In the past five years EF trained over 70,000 youth in about 80 occupations, out of whom 75–80% were gainfully employed thereafter.
EFS applies an annual competitive procurement process in order to identify the best performing T&Es for providing services which consist not only of skills training but – most importantly – the subsequent placement of the graduates in gainful employment. A key step in the process is the Rapid Market Appraisal (RMA), which is being conducted by the T&E among potential employers and clients in the locality where the T&E proposes to implement the training.
EFS neither sets geographical nor occupational targets as these are solely defined by the findings of the RMA of each T&E. With regards to the numbers of trainees, the T&Es are given a target range which is derived from an assessment of their implementation capacity. The final target numbers and the direct training costs for each occupation as well as the implementation arrangements are agreed between EFS and each T&E during contract negotiation.
Upon completion of the contracting, the T&E has received a clear mandate and the role of EFS is mainly limited to monitoring. Guidelines developed by EFS as well as coaching by the EFS field staff support the T&Es in applying the agreed standards with regards to training announcements, trainee selection, training implementation as well as placement and counselling support. The skills trainings are either provided as mobile training in smaller market towns or in training centres and typically consist of 1-3 months practical instruction which usually includes on-the-job training. Training is provided in about 80 occupations and follows the standardised occupational profiles and curricula of the Council for Technical Education and Vocational Training (CTEVT) under the Ministry of Education. A key proxy indicator for the quality of the skills training is the pass rate of the graduates during the skills test conducted by the National Skills Testing Board (NSTB). Apart from the technical skills, all trainees are provided with a training package on labour rights, HIV/ AIDS and reproductive health. Women trainees are offered a life skills training package, those interested in self-employment may opt to attend a business skills training and trainees intending to migrate abroad are provided with a pre-departure orientation.
Beyond procurement and monitoring, EFS supports the capacity building of the T&Es (mainly through methodological training and skills testing of trainers, monitoring and management capacity building), conducts labour market studies, explores and pilots new occupations jointly with the T&Es and supports CTEVT/NSTB in developing and revising occupational profiles and curricula.
In the discussion, the instrument was welcomed both by COOF and HQ staff. As a general comment it was underlined that the application of the indicators could not only be considered for the project but also for the programme level, meaning using the indicators for the monitoring of country strategies. Indeed, the COIs are backed by the SDC directorate and will be used in the Bill to the parliament 2013 – 2016. This issue was not further discussed but was taken up by Franz Kehl from KEK-CDC Consultants in his presentation on the results frameworks and by Adrian Maitre from the Quality Assurance Section of SDC (see below).
Hereafter some specific statements concerning the four proposed indicators:
Gainful employment…
… is a highly context-specific concept and is not about competing with minimal wage standards.
… should not only look at the income but also consider quality aspects of employment (decent work).
Systemic change…
… is difficult to define, thus the indicator remains vague.
… is assessed by the involved authorities themselves, thus the dialogue and the process of “sitting together” is the core issue and not the exact measurement and figures.
… should be assessed by actors of both, the public and the private sector (if the private sector is organized).
Relevance of training / employability…
… is a very important indicator as it shows the link to the labour market and recognizes the importance of the private sector.
… does not reflect the social importance and the empowerment aspect of education, which is particularly important for target groups with a very difficult background; not only the employment, but the educational process itself has a value.
Outreach / scale…
… is crucial as VET interventions (especially the long-term programmes) are often relatively expensive and thus run the risk to exclude a majority of the society.
… should also look at the indirect effects of the training, meaning the number of successfully established businesses and persons employed in those companies.
In the second half of the morning, Franz Kehl presented the results of a general assessment done on result frameworks, developped over the last two years. The evaluation showed that the definition of the result statements in the frameworks is not an easy task and that it might be worth to dig a bit deeper into the subject. One of the suggestions of Adrian Maitre, Head of the Quality Assurance section of SDC, was therefore to establish a joint learning project, which will address proposals on how to improve the quality of result frameworks.
Franz’ presentation:
[slideshare id=21224073&doc=ppt-analysisofvetresultsframeworks-franzkehlmarkusengler-130515183538-phpapp02]
Trouble viewing? Try fullscreen mode or download the PDF here: Results Frameworks [PDF]
Feedback from Adrian Maître, Head of SDC Quality Assurance
First of all I have a supportive message regarding the indicator work. This is an important step forward. I noted several convincing elements: (1) population and system levels are included, (2) qualitative and quantitative aspects are integrated, (3) due to the bottom up approach, thematic expertise is mobilized and the guidance is easy to understand, (4) the Nepali experience showed the overall utility of the exercise and also good practice while bringing national partners on board of results measurement.
A further step that should be done is screening the proposed indicators according to their more strategic (→ results frameworks of Domains of intervention) and operational (→ project monitoring) nature. This could be appropriately done as a joint effort of VSD/E+I network and QA.
We discussed whether employment and income is enough as outcome indicators on population level. What about decent work? What about income availability among household members? What about sustainable jobs? I would suggest we engage in a discussion on which of these important aspects could be covered by (project) monitoring and which ones require additional measures on the level of the cooperation strategy monitoring like case studies or beneficiary assessments or then also: evaluations?
Regarding results frameworks: It is encouraging that we already achieved a good initial quality while working on these demanding and new tools. It is also important to improve them over time. I suggest here that we analyze carefully the recommendations made by KEK consultants in a small and mixed VSD and QA team and then bring our conclusions back to the larger VSD and QA community.
Finally I would point out the importance of elaborating good results statements in future annual reports. I am very confident that in the next rounds of Annual reports we will improve substantially the quality of our VSD results statements based on (1) general guidance for results statements provided by QA and (2) common VSD outcome indicators and improved results frameworks/monitoring systems. The VSD community is on track when it comes to results management!
– Adrian Maître, Head of the SDC Quality Assurance Section
Lucia Bordone’s presented the Education Network’s work on Results Measurement in basic education projects. The network is at an advanced stage of developing a tool for results measurement that was tested in Benin. For the second version of the tool, a glossary to promote the use of a common language, as well as an indicator dashboard to link the tool’s core assessment criteria (Acceptability, Adaptability, Accessibility and Availability) with classical criteria and indicators.
Download the presentation: Education Results Measurement [PDF]
