What do we know about Joint Evaluations?

In June of 2011, the ECB Project published the latest version of What we know about joint evaluations of humanitarian action: Learning from NGO Experiences. This paper aims to share the experiences and learnings of NGO staff who have conducted joint evaluations and serve as a resource for agencies considering conducting  joint evaluations in the future.

The Guide section of the booklet can be considered a ‘how‐to’ for those closely involved in joint evaluations. It discusses the benefits and disadvantages of the process, and what to do before, during and after a joint evaluation.

The Stories section shares three case studies from the ECB Project’s experiences.

1. Joint Independent Evaluation of the Humanitarian Response of CARE, Catholic Relief Services, Save the Children and World Vision to the 2005 Food Crisis in the Republic of Niger
2. Multi‐Agency Evaluation of the Response to the Emergency Created By Tropical Storm Stan in Guatemala – CARE, Catholic Relief Services, Oxfam
3. CARE, Catholic Relief Services, Save the Children and World Vision Indonesia Joint Evaluation of their Responses to the Yogyakarta Earthquake in Indonesia

The Tools section includes templates and tools that can be adapted for evaluations, including sample terms of references, agreement documents, a joint evaluations readiness checklist, and suggested topics for discussion with prospective partner agencies.

Advantages of a Joint Evaluation

• Like a single‐agency evaluation, a joint evaluation provides an opportunity to learn from past action so as to improve future decision‐making.
• It allows agencies to see a bigger picture of the collective response and what gaps still exist.
• By looking at a non-joint response of different agencies side by side, you can see where a coordinated effort would have been beneficial and can plan accordingly for the next response.

“Evaluation reports repeatedly show that better coordination would have led to a more effective response.”

• When agencies open up to one another by sharing weaknesses and strengths, they increase transparency and make it easier for them to hold one another accountable for acting upon the recommendations.
• Conducting the evaluation with other agencies allows sharing of perspectives and technical knowledge and builds trust for future cooperation.

Disadvantages

• It takes greater time, funds and skills for agencies to agree to do and conduct a joint evaluation.
• Less depth on the work of each agency is covered.

So check out What we know about Joint Evaluations and tell us what you think!

Advertisement

Read Full Post »

Beyond the written report: Serious content can be presented in interesting ways!

Silva Ferretti, consultant to our project,  and facilitator of the two workshops for the AIM Standing Team, gave an online presentation in February 2012 to share her innovative approach to share knowledge and learning. This approach is using mediums other than written reports. Silva prefers that her evaluations (of trainings or programs) and other types of “reports” are portrayed in a more appealing way.

The following are the different types of mediums Silva discussed. You will recognize a few from the workshops with Silva.

1. Blog through WordPress to summarize daily activities for a later evaluation of the activity (a workshop, for example)
2. Photos and Videos

• Participatory activity is better documented by photography, such as through a powerpoint presentation, or video. For example, Silva showed her Powerpoint using photography that documents the process of a vulnerability analysis.
• For a Knowledge, Attitudes and Practices analysis, Silva, as the consultant, prefers to use photo and video to document the analysis with the community. Using this medium promotes ownership by the community.
• Video is Silva’s preferred medium to capture people’s reflections, opinions, and explanations, such as beneficiaries’ feedback or staff’s explanation of an activity or approach.
• Use 1-2 minute clips to capture learning from workshop participants when the learning happens
• A video can portray emotion and a picture is worth a thousand words!
• use Youtube to post the video

3. Prezi is a free downloadable presentation software that presents information in one screen and allows the reader to zoom in to different parts. A prezi presentation allows the reader to see the creator’s structure of thinking behind the presentation of information which is not possible in a linear report. Silver showed an evaluation she did, which includes all the regular parts of a report. This prezi presentation also includes an embedded video of a staff person explaining a program component!
4. Google Doc to conduct an online survey and get results quickly.
5. Animation using xtranormal software allows you to create characters to act out a skit (which always turns out funny). Animation can be used to depict a sensitive or controversial topic when there is no real voice to do so, such as problems in a program found through an evaluation. Have you seen the cowboy video created by ___ during the first Standing Team workshop in Bangkok? Not only was this video super fun to watch, it sparked a great discussion of the importance of high quality evaluations.
6. Diagrams, flow charts, and maps can be used to break down and present a big idea, problem or concept in a visual way. There are tools online to do mind mapping, problem solving, flow chart diagramming.

Advantages to these alternative mediums of sharing knowledge and learning:

• They are interesting, fun and cool!
• They can spark an interesting discussion among participants.
• These mediums are more likely to be given attention than written reports.
• The images are memorable.
• Using these mediums is at no cost.
• They only requires a computer and internet.
• Presenting knowledge through these mediums allows participants to work together in a different and fun way.

Serious can be Cool too!

Silva believes that just because the content (like an impact evaluation) is serious doesn’t mean it can’t be presented in a more interesting way. She explained that she is trying to overcome the “report wall” whereby many in the field, including big donors and report commissioners, prefer a “proper” written report over the alternative mediums of presentation of information, such as video, diagrams and infographics. Silva and the participants in the online presentation discussed the challenges of getting big donors and report commissioners to accept these new formats.

Which medium do you prefer?

Read Full Post »

Timeline of the evaluation.. and our learning around it!

The timeline is the skeleton on which we noted many of our learning and tips.

We looked in detail at two areas of concern along the timeline, in two group:

• the role of management
• communication and documentation of results

Our tips, ideas, questions were noted on the timeline with post it and will be captured on our presentation on the timeline (above)

Read Full Post »

What is the added value of a joint evaluation?

What is the added value of a joint evaluation? There are of course some very practical implications (e.g. maximizing the use of resources and funding), but the value of a joint evaluation is not limited to this!

We had already discussed some of these issues on the first day, but we now wanted to capture new thoughts and leanings, at the end of the workshop. In a “bus-stop” session, 4 participants were assigned the role of reporters on 4 main themes:

• accountability
• learning
• policy
• coordination

The other 4 participants rotated amongst them hand had a 2 minute conversation on each of these topics. Ideas were noted down, and results  presented. These are the ideas gathered on the flipcharts!

Read Full Post »

The role of the team leader

The session started by reflecting on how Pauline acted in the scenario, as a team leader. What was her role? What did she do directly? What did she facilitate?

• Lots of preparation!
• Facilitated
• Provided technical advice
• Organized the group
• Delegated tasks
• Worked closely on logistics herself

We did then had a close look – and revised –  the RACI(s) matrix and the timeline. For each action in the timeline, what is the role of the team leader? Is she Accountable? Responsible? Consulted?

Read Full Post »

Evaluation of Day 3

Each person described what s/he learned from the joint evaluation simulation, and what areas s/he still wants to learn more about.

What I learned	What I want to learn more about
Adhong: How to draw conclusions from evidence	How to effectively draw conclusions
Angela: Visual way of documenting evidence, so can continue to refer to the flip charts to draw conclusions and generate recommendations. This is helpful to visual people like me.	Tips on how to pull it all together into a report. How to organize (e.g. headings) complex information
Amadou: One needs to have a base of evidence for everything. A high quality preparation of the evaluation results in a good evaluation.	How to draw a recommendation from the analysis.
Sekai: Importance of planning, esp. for team leader. Who, what, when…	Consolidation of final report: what are the important parts?
Saji: Tabulation. Different teams divided up the work of data collection, but then tabulation brought it all together.	Planning/preparing for data collection.
Kassoum: How to motivate a team with chocolate! (team building). How to simplify each step, accomplish them in a relatively short time, then synthesize information so one can digest/use it easily (this is applicable in things other than evals!). Team leader can facilitate this greatly.	Self assessment to explore further areas of capacity building needs. How to become as proficient as Pauline?
Matt: OECD criteria – using them as a framework.	Certain preparatory steps, like negotiating TORs, etc. A check list may be helpful to make sure not miss anything.
Oliver: Evaluation is a process, not an event. Need to attend to the process to get the desired results. Gained experience in qualitative analysis. Qualitative methods can be reliable!!	In context of ECB and JE, how to organize a final report that meets needs of ECB as well as other standards that exist.
Pauline: Figure out ways to reduce questions, especially for testing.

 

 

 

Read Full Post »

Scenario. Developing recommendations

Recommendations mainly  come from the “no”s in the conclusions – areas or examples where one or more agencies did not meet the criterion.

Tips for good recommendations:

• Keep list short. Want to aim for 1-2 recommendations to be acted upon, otherwise entire exercise was useless!
• Could do a priority setting with management, or identify immediately actionable items.
• Be clear who are the primary users of the evaluation as that will help determine what types of recommendations are generated; ensure they match the users.
• If helpful, categorize the recommendations (e.g. along the lines of the different users such as management, programming, etc.)

The challenge of joint evaluations is you are trying to please a lot of people and but can’t make everyone happy.  But evaluators do need to make sure management accepts most conclusions/recommendations, otherwise the evaluation won’t be used.There are constructive ways to phrase these recommendations, which is important so one does not alienate the audience.

• Emphasize the positive. Be practical and specific (if a particular agency was strong, can use that as the basis for a recommendation – “other agencies can learn from agency A”). Focus on things under agencies’ control.
• “Improve in this area…”
• Don’t change the findings, but way it is pitched can help temper how the message is received, e.g. lessons learned

Read Full Post »

Scenario. Getting to conclusions

Documenting the findings and data analysis:

Lay out the findings (summarized from the data as tabulated in the matrix) by criterion, and write it as a close ended question.

e.g. Was coordination across ECB agencies effective? Yes? No? Why?

• Evidence 1
• Evidence 2
• Evidence 3

Disaggregate data sources especially when perceptions were different (women said A, then said B)

If evidence is scant, the team may go and collect more data to fill in gaps

First level conclusion drawing – it’s all about evidence!

An evaluator’s job is to make judgments based on evidence. Looking at the evidence available for each evaluation criterion, the evaluation team can discuss the findings and determine if the agencies collectively met or did not meet the criterion.

Yes or no? Were the agencies effective in meeting needs? Did the agencies coordinate well?

Question: What happens if there are three sets of evidence that point to “yes,” one set that points to “no”?

Answer: Areas where the findings are mixed or unclear are areas to explore deeper with stakeholders. In the formal report, the answer won’t be yes/no, but rather explaining the things that going well (“yes”), and things that are going less well (“no”). Making it a close ended question pushes the evaluators to make a judgment. Such a judgment helps create concrete recommendations.

Again, it’s about the evidence. For every statement and conclusion, evaluators need to cite evidence. The more specific you can be, the more specific the recommendations can be.

 

Tip: Type up the flip charts of the tabulation as well as the conclusion drawing notes and keep as body of evidence, just in case someone asks for justification.

Next level analysis/conclusion drawing: Reflection workshop

To increase use of the evaluation for improving practice, organize a reflection workshop where agencies and possibly other stakeholders analyze the data and draw conclusions.

The evaluation team does not present their conclusions, but lays out the questions and evidence. They facilitate the agency managers in drawing out conclusions. The team’s preliminary conclusions are either confirmed, deepened and improved, or different. The team can use the conclusions that are different or controversial as the basis of discussion.

Question: What do you do when the answer an evaluation question is “yes” for 2 agencies, and “no” for another.

Answer: Naming agencies is a delicate issue. Many times, it is done without much controversy. But some issues that are revealed in an evaluation are delicate and you want to remember to be ethical and ensure the evaluation itself doesn’t do any harm (e.g. harm relationships). Discuss with the steering committee on what to do.

Read Full Post »

Scenario. Flipcharts of the conclusions

Read Full Post »

Scenario. Findings. Other findings, other informants.

 

Read Full Post »