PAs - What to consider

 

 

​Down​load as PDF​​

Realising potential: What to consider when planning and implementing a participatory assessment

​Participatory assessment implies more than organising peer​ interviews – it is an assessment methodology that involves target groups and peers actively in producing and collecting data. 
​

​In the comprehensive and coherent design of the assessment process lies the power to produce useful and focused results.
Thus, a successful participatory assessment depends on a variety of issues. 

​
10 questions to consider: ​


1) Is the participatory assessment methodology adequate for the intended purpose, in the current context?

The participatory assessment makes sense when the SDC or its implementation partners want to acquire food for thought and are ready to invest their own time and resources to validate the conclusions and deeply reflect on the results and its consequences. The participatory assessment is particularly useful for planning a new phase of an already established intervention. If the organisers have little flexibility to take up new ideas and adapt their approaches, the participatory assessment methodology may be too resource intensive.

The participatory assessment is adequate, if and when:

  • There is a perceived need for getting out of the box of current thinking around the intervention.
  • The assessment has a clear forward-looking learning purpose, to find out how best to design and implement a future intervention or a next phase of a project.
  • The context is appropriate for target groups and peers openly expressing and discussing their views.
  • Target groups and peers would see an interest and intrinsic motivation in contributing to the assessment pro
  • cess.
  • The organisers have the necessary commitment and the required financial resources, human resources and logistics are available.
  • The assessment process can be expected not to do harm to the target groups or their peers ('do no harm' principles).​

​​2) Does the organiser’s framing of the assessment allow for purposeful reflection and useful results?​

The organisers (SDC office or implementers who are mandating the PA) are responsible for setting out a coherent and adequate framework for the assessment process, designing, steering and accompanying the process and implementing its result
s. 
The preliminary considerations are the purpose, objectives and scope of the assessment. What do we want to find out through this assessment? Do we want to check whether our theories of change are adequate and relevant to the realities of targeted groups? Or do we rather want to find out whether and to what extent a certain methodology or training topic responds to the needs of targeted individuals and groups? ​
The framing and design of the assessment process should be carefully considered, with external support for the organisers when needed. If the purpose of the assessment is not clear (enough) the PA carries the risk of being very resource intensive and producing few useful results.

Main tasks of the organisers:

  • Set the scope and focus. Define the purpose of the exercise, based on a clear view of the intervention, the partners and the institutions involved: For what and how will we use the results of the assessment? What are the questions that need to be answered?
  • Recruit a facilitator for the process, ensuring that he/she has the required human and financial resources.
  • Accompany the process, steer it towards its purpose at all stages; make sure that the conflict dynamics are dealt with and the 'do no harm' principles are followed.
  • Provide space for reflection and feedback loops to adapt the process if the conditions prove to be different than expected.
  • Provide space for individual and institutional learning, clarify who should be informed about and learn what, when, how, and for what purpose.
  • Make sure that stakeholders, target groups and peers are informed adequately about the process and its results.
  • Plan for integrating the assessment's results into its own decision-making on future interventions (in terms of ToCs, project approaches, project design, partnerships, risk assessments). ​​

​

3) ​Does the facilitator (and his/her team) fulfil his/her responsibility and role in designing, monitoring and steering the process?

Main tasks of the facilitator (in cooperation with the project management)

  • Designs the PA process steps and defines the adequate methodology in detail, respecting context-specific challenges e.g. conflict sensitivity.
  • Recruits team of co-facilitators (if needed), with the required profile.
  • Sets out the work programme.
  • Selects targeted individuals and peers to be involved in data collection.
  • Designs and organises the appropriate methodology for data collection ('peer interview process').
  • Steers, manages, organises and monitors the process of data collection.​

Key competences of a national facilitator

To fulfil complex tasks, the facilitator must have a strong professional and personal profile.

The facilitators should understand the participatory assessment approach and qualitative assessment methodologies, and have professional experience in this respect. He/she should know the intervention's context, have competences in designing and facilitating participatory learning processes, strong communication skills, be able to interact with people from different backgrounds and status, identify and deal with different types of biases, and be able to identify and manage conflict and power relations.

The attitude of a curious researcher rather than an evaluator should guide all interventions of the national facilitator. He/she needs to work with a genuine appreciation of and trust in the competences of the peers.

Finally, analytical and synthesizing capacities are needed to produce, from a wealth of data, a concise report for the organisers, as a basis for their management decisions.​

​​​​

4​) ​Is the assessment process designed carefully and implemented in a flexible way, responding to the risks involved?

Specific attention must be paid to the scoping, design and planning. Experience shows that a scoping workshop is useful for developing a common understanding of the purpose, identifying the expectations from the assessment, focusing on relevant questions, establishing the cornerstones of a process that is fit for purpose, taking to account the risks involved. It also helps identify the required resources and the financial costs to budget for the assessment – and to make an estimation of the cost-benefit relationship. A careful scoping and clear thematic focus considerably help reduce the investment.

Careful conflict and power analysis regarding the intervention and assessment process is needed. The PA itself may involve specific risks, for the participating individuals as well as for the assessed intervention. If the process is not designed and managed in a conflict-sensitive way and discrimination issues are not treated properly, the PA process may even do harm. For example, it may expose targeted individuals and groups or peers to reprisals. Or the results of the participatory assessment may only reflect the views of individuals that have vested personal interests and are far from representing the target groups. Since many actors need to be coordinated in a PA, a written documentation of key information and findings reflects good organisational practice.

Feedback loops between the organisers and the facilitator and his/her team must be established throughout the process, and the planning should be adapted flexibly to emerging risks and potential benefits. 

​

5) ​Are the criteria for the selection of individuals targeted by the intervention to take part in the assessment clear and transparent, and do they take diversity into account?​​

Target groups are not homogenous. Some groups and individuals will have benefitted more than others from an assessed intervention due to power relations and other factors – and they all might express very different views on the results of a particular intervention. A key challenge is selecting individuals who can represent the target groups that the intervention was planned to be responsive to, taking into account the diversity while keeping sight of practicability and costs. 

Participatory assessment is a qualitative approach. A certain quantity and diversity of views will be needed to ensure a certain representativeness of the collected information. For example, to get gender disaggregated data, both men and women from target groups will need to be involved. However, larger samples do not add quality per se. The rigour of the results stems from the careful and well thought-out selection and design of interviews and not from numbers in samples. E.g. involving more school teachers with the same contextual background might not add new qualitative insights, and just create more costs. 

In general, interlocutors should be selected based on their proximity to the project goals and activities, their knowledge and experience on the topic, and their motivation to contribute. ​

Whom to select? 

The criteria will depend on the purpose of the assessment, the questions to answer, the intervention at stake and the context. For example: ​

  • Do we want to compare the impact of different learning methods? Then we might focus on teachers and students who were involved in the different approaches, and we might add a group of students that were not involved in the project at all. 
  • Do we want to assess the usefulness of the learning content? Then we might focus on former students who are in a position to express their opinion on its practical usefulness. 
  • Do we want to know more about the performance of teachers? Then we might select current and former students and their parents. ​ ​​

M​otivation of target groups by personal interest in the results, for example: 

  • If target groups have been constantly and continuously engaging with the programme (e.g. farmers in a local agriculture programme), they may have a strong motivation and interest in contributing to the learning and improving the intervention they will continue to benefit from. 
  • In other cases, there might not be a strong genuine interest among target groups to contribute to the PA exercise. For example, if students are not aware of the fact that they are benefitting from a project intervention, or if they benefitted from one activity or phase of an educational programme but have since moved on. In this case it might be difficult to motivate them to contribute if no additional mutual benefits are identified.


6) ​Are identified peers ‘similar’ to the targeted individuals and groups – and capable of producing valuable insights together with them?​

The selected peers share important characteristics with the target groups but should be outside the intervention's direct reach and impact. It is assumed that familiarity with the context and similarities between peers establishes trust, so that target groups will buy into exchange with them and openly share their views and opinions – more openly and genuinely than with external stakeholders or experts. 

However, people may be 'similar' with regard to one aspect – and very different in other respects. Organisers and facilitators may consider in their subjective external view two individuals or groups as peers while the individuals concerned may not consider themselves peers at all (see "Similarity" below). 

Peers should be recruited according to clear criteria of 'similarity', taking to account their capabilities, their own motivations and interests in the intervention and/or the assessment, and their relationship with the beneficiaries. In recruiting peers, it is important to consider what could motivate them to invest their time in such an exchange and contribute to the assessment process. It may be difficult to establish any intrinsic motivation. 

Peers are assumed to play a neutral and impartial role of data collector for the organisers. This might not be well understood either by the peers themselves or by the involved target groups. Some peers might not stick to the set of interview questions and may take on a more investigative role, so that peers can be perceived as intruding or even spying into the lives of interviewees, which could entail personal risk for all involved, particularly in conflict-affected situations. Target groups and peers both live in a reality of complex interactions and relationships (including with the intervention's implementing partners). 

Peers and target groups might have specific motivations to be involved in the assessment – be it only the concern of satisfying the donor to continue the support. While this can't be avoided per se, it is the facilitators' role to build a shared understanding, instruct the peers on their role, provide adequate information to the target groups, monitor the data collection process and detect the factors that risk hampering the peers' neutrality and the authenticity of the expressed views. It will also be his /her task to take such factors to account when analysing and synthesizing the data collected from peers.

Facilitators will need to find ways to bring forward and strengthen the knowledge and capacity of the peers to make their interlocutors express their views, document and collect data according to the design of the assessment. They will need specific information, training (including interview techniques) and coaching to help them understand und fulfil their particular role and the important tasks in the process. A specific challenge is to engage with institutions in a participatory way (see key questions in 'step 2'). 


‘Similarity’: What decisive features to potential peers have in common? 

It is always challenging to define 'similarity'. For example, is it the belonging to a similar territorial community? Similar living conditions? A neighbouring village? A similar social or educational status? The same beliefs or shared values? Similar development challenges? The same gender? The criteria for 'similarity' must be considered carefully in the light of the intervention and the context at stake, and the aim of establishing a basis for trust between the interview partners. ​​​​​​


7) Does the exchange between target groups and their peers build on mutual trust, and is it organised so as to produce the expected results?

Relationships among target groups and peers must be examined carefully, not only when selecting the peers but throughout the process of exchange. In the light of complex power relations within societies, trust and confidence is never to be taken for granted and must often be built or strengthened, particularly in fragile and conflict-affected situations. Possible security risks for the facilitators, target groups and peers and their individual fears must be dealt with throughout the assessment process and are particularly important for framing questions, selecting the interlocutors and organising the logistics.

The exchange between selected target groups and peers is framed by the context of the programme and the objectives of the PA. The challenge for the organisers and the facilitator is to keep this framing but not distort the data and unduly influence the exchange. They must be careful in their communication and explanations of the exercise. It might seem that communicating as little as possible around the purpose of the assessment might reduce expert bias, however that is not the case. If peers and target groups are not clear about the objectives of the exercise, they will find their own explanations and (mis-)interpretations – and might fulfil a wrongly perceived role, most of the time with the best intentions. In such cases, the collected data will present strong biases and be difficult to interpret (link to level 3.1 Bias).

Participatory assessment is not only about bilateral interviewing. Depending on the purpose and the context, the exchange between target groups and their peers can take place in different formats according to the purpose, context and available resources. For example, facilitated group discussions may simplify the approach and make it lighter while providing differentiated views of stakeholder groups. Remote interviews, phone calls or video calls can also be considered.

The questions for the interviews and/or discussions as well as the interview methodology should be identified and formulated in close cooperation between the facilitator and the peers, with a view to incorporating the realities of the target groups and collecting relevant data to answer the general questions of the assessment, but also to allow for new and unexpected insights. (link to level 3.2 examples for phrasing the purpose)

In many cases, the task at hand will need to be expressed using concepts, terminology and a common language shared by peers and target groups. A story-telling methodology, the use of pictures, photos, metaphors, anecdotes (and other creative methods) may help produce useful data for analysis, even in contexts where written reports will be challenging. (link to 3.3 Facilitation)

Also decisive for building trust are a precise, reliable location for the meetings and confidentiality rules. (link to level 3.4 The environment of meetings)

Peers must receive basic training to develop trust in the process and acquire a common understanding of the main features of solid data collection. These include a 'neutral', impartial, empathetic perspective; a common format, with clear interview questions for accurate data collection and reporting; basic interviewing and reporting skills. For example, it is important to make sure that peers know the difference between closed (yes/no) questions and open questions (that allow for a more extensive/flexible response).

In general, organisers, facilitators, target groups and peers must be aware of their different roles in the exchange, and the expectations of all stakeholders must be managed. Good messaging and communication are key, also towards other stakeholders in the intervention (e.g. local authorities, implementing partners).


Bias management​


​​
Example for phrasing the purpose:

The assessment questions are part of the ToR of the facilitator. The interview questions are a separate set of questions which are framed by the purpose of the assessment and the assessment questions. It is the task of the facilitator to explain the purpose and the assessment questions to the interlocutors of the peer interviews, who then develop their own interview questions. 
As an example, in the case of vocational training, different assessment questions are possible. Their phrasing will determine the direction and the space for discussion and interviews. The seemingly subtle differences have a large impact on the peers’ understanding of the task. This phrasing is thus of key importance for the PA. 
Examples for the different framing of the purpose of the PA in the introductory communication of the facilitator with the peers are:  
  • “We want to find out what your interlocutors think might be a good education.” This leads to a narrow approach focusing in the interviews on views of “good” and “bad” education.
  • “We want to find out what the strengths and weaknesses are of the training methodology used.” This implies that peers and target groups are able to compare different methodologies and are aware or even specialists on adult learning.
  • “We want to find out what your interlocutors think this education programme has achieved? What did they learn from it?” This leads to linear causal thinking on the results of an intervention – not taking to account the systemic dimension. 
  • “We want to find out how and why the life of your interlocutors has changed since the training events.” This opens the discussion to a broader awareness of systemic interactions of different factors and gives an opportunity to challenge the underlying theories of change of the intervention.
From this introduction as to the purpose, the peers will develop their interview questions. The PA process must allow space for additional questions from peers and interviewees. 
While the format and content of interview questions will differ depending on the context and the interlocutors, a certain rigour in following an interview guide and uniform formatting of the questions for discussion will help make a sound analysis and to synthesise the answers. Special attention will be needed to translate the questions from the ‘development jargon’ into everyday language and from the project’s working language into local languages. It may impact strongly on the results and on the process itself if the interviews are not conducted in the mother tongue of either peer or interviewee, or facilitator.

​​
Facilitation

An adequate interview methodology is key for getting good results worthy of the considerable investment. The facilitator's team must possess strong communication skills.
It is primordial to construct a safe space for the exchange. Security requires an atmosphere of trust, not just physical safety. The facilitators must pay attention to group dynamics, the inclusion of extra- and introverted individuals, culturally acceptable ways of expressing criticism, the importance of saving face, the presence of other people (e.g. authorities of any kind) in the room etc. 
It may be necessary to work with alternative methods of data collection such as story-telling, or to link different exchange formats, such as exchanges in small groups and exchanges in peer/interviewee settings, in order to adapt the methodology to the context.  
Conducting interviews by phone has the advantage that you can reach out to stakeholders who are relatively inaccessible, e.g. due to the security situation. This can be part of a remote management set-up.​

​
The environment of the meetings frames the results. Example from experience

Depending on the location and the set-up, meetings can be perceived very differently. If meetings are organised in the municipality premises and in the presence of the authorities, it might be difficult to make people understand that the assessment has nothing to do with authoritative decisions or the mayor's power. 
If the implementing partner's representative or the donor is present, it will be obvious for interlocutors that their views will influence the personal relationship with them – and they will tend to be polite if they have an interest in continuing relations. Interviews and discussions might not reveal honest critical viewpoints – why should those present risk provoking those in power for the sake of 'telling the truth' or pleasing a donor?​​

​​​​​

8) Are the views expressed inte​​rpreted and analysed in the light of motivations, interests, power relations and conflicts?

The collected views and their documentation by peers do not provide an ‘unbiased’ view of the intervention at stake. The statements of interlocutors depend on their personal motivations, interests and expectations of the project as well as – more broadly – their ideas and values, existing power relations or conflict lines in society, within their own social group or in their municipalities. Power relations impact on the quality of the exchange, on data collection and data quality – and carry with them a set of specific biases. 
The facilitator steering the process must carefully assess the situation of target groups and peers, with a view to being able to interpret the results accurately. Power dynamics, alliances and conflict lines that can be observed among targeted individuals and groups and their peers must be analysed and documented. This then helps assess the collected data and draw adequate findings and conclusions. ​

​​​​​

9) ​Does the data analysis bring the «stories» into so​und conclusions?

Participatory and empowering approaches allow for a great variety of stories told by target groups – this is the great advantage of this approach. However, the raw data and 'stories' are complex; they are just a basis for the analysis. Data must be interpreted and developed into sound conclusions for programme management, according to the purpose that was set for the assessment. Illustrative cases or stories might help make the report more accessible.

The challenging task of the facilitator is to frame their data analysis from the viewpoint of the original purpose of the organisers – without detracting from the complexity of the data. In this moment, he/she will switch from the role of facilitator to the role and tasks of an evaluator, which may be difficult. A validation phase involving peers and interviewees can help sharpen their conclusions and allow for a clear focus on the target groups' views. In any case, the process of interpretation and synthesis needs to be as transparent as possible.

​​​​

10) Is open communication at all levels ensured to allow for transparent outcomes?

Last but not least, open and targeted communication is important throughout the process. A communication strategy establishes different forms of participation and feedback loops. Open communication can establish trust and help improve the assessment process and results as well as the relations between project management and target groups. Lastly, open co​mmunication has an important empowering dimension – meaning that target groups are able to contribute more effectively.

Organisers, programme management and facilitators must ensure that relevant information is shared at all times and a common understanding is built on the main issues at stake, the challenges encountered and the risks taken. Depending on who should learn and understand which content, clear messaging and communication with target groups and their peers, but also with programme stakeholders (authorities, partner institutions) is key to motivate participation, get effective results and manage risks. A communication strategy will help reflect on and develop a comprehensive approach to messaging and communicating throughout the process, with the various stakeholders.


​​​​​​​​​​​

​



​​
To get started with your participatory assessment, contact us:
Stephanie Guha, stephanie.guha(at)eda.admin.ch   
Ursula König, uk(at)ximpulse.ch

​​

​




​​


​