The reason-of-being of evaluations is to support evidence-informed decision-making. However, each study should also provide added value from the methodological perspective. The quality of the evaluation design and its implementation is ultimately a critical contributing factor to the usefulness of the results. In this blog, I highlight some methodologically interesting parts of the Evaluation of the Finnish Development Policy Influencing in the European Union. The evaluation used multiple methods framed around a theory-based approach, which integrated policy-level outcome harvesting and hypotheses from the Small States Theory. Also, other methods were applied; however, they are not discussed in this blog.
Evaluating policy influencing
The evaluation was designed around a theory-based approach. The logic model was developed based on a methodology tested in the evaluation of Finnish Development Policy Influencing Activities in Multilateral Organisations (Palenberg et al., 2020) and applied to the context of EU influencing. In turn, the conceptual framework by Palenberg was built on an overview of literature discussing advocacy and influencing approaches (see Annex 5 in Volume 2).
Influencing EU’s development policy includes both similarities and differences to (other) multilateral influencing (see p. 112 of Volume 1). Combining the tested evaluation framework of the evaluation of multilateral influencing and the specificities of how the EU works, a theory of change was developed, see Figure 1 below.
Figure 1 Framework of the Evaluation of the Finnish Development Policy Influencing in the European Union
Source: Mackie et al. (2022)
The theory of change articulates with the evaluation questions. The first level; how Finland has organised itself internally for EU influencing (the lowest level, EQ1), how and how well the influencing activities unfolded in connection with EU actors (middle level, EQ2), and what influencing outcomes were achieved (higher levels, EQ3). Like in most theories of change, the level of control of the actor (here, the influencer, i.e. the MFA) reduces towards the higher levels of the diagram.
A detailed evaluation matrix guided the evaluation implementation (see p. 117 in Volume 1). The evaluation criteria embedded in the evaluation questions were Relevance, Efficiency, Effectiveness, and Coherence.
As in a typical theory-based evaluation, the logic model was tested by unpacking the underlying assumptions. The overall methodological conclusion was that the theory holds ground well.
Policy-level outcome harvesting
Effectiveness, i.e. the influencing outcomes obtained, was assessed using a dual-track approach. In the first track of analysis, the evaluation looked at what Finland has achieved vis-à-vis the objectives defined in various influencing plans. The challenges here were that (1) the evaluation period covered the years 2014-2021 and the first MFA’s EU development policy influencing plans was from 2019, (2) a rich hierarchy of government policy documents contributed to lower-level influencing objectives (e.g. government programmes, foreign and security policies, government’s general EU influencing plans, to mention a few), (3) a large number of priority topics, themes, and processes were included in the policies and plans making it challenging to pack these into manageable benchmarks.
As a result, a group of key topics were selected for in-depth analysis. These included, for example, the new multi-annual financial framework negotiations and Finland’s EU Presidency period. In addition, the evaluation looked at some of Finland’s thematic priorities, such as gender equality, governance & human rights, education, and climate change.
The second track was an outcome-harvesting process adapted to this specific case. First, outcome leads were harvested from available documents and initial interviews with MFA staff. Here, leads from any sector or process were accepted. However, given the broad scope of the evaluation (Figure 1 above), the outcome leads could be naturally grouped under the pre-determined clusters. Subsequently, the outcomes were verified through additional desk review and interviews with external individuals (mainly EU Commission staff). As a result, leads were either rejected or maintained. In total, 18 outcomes were documented.
Finally, the outcomes were rated based on the significance and Finland’s contribution.
Figure 2 The system of outcome rating used in the evaluation
In other words, the judgement criteria for the Effectiveness analysis were derived the above-mentioned rating as follows:
- Significance of the influencing outcomes and Finland’s contribution to achieving them
- Achievements against Finland’s influencing objectives
- Contributing and hindering factors (internal/external).
For the detailed rating results of the harvested outcomes, see Annex 6 of Volume 1. The narrative description of the results can be found in the main report on page 44 (Section 4.3).
Small States Theory
The evaluation also tested the use of Small States Theory hypothesis as part of the analysis. There are ways to characterise small states beyond mere population size, surface area, and Gross Domestic Product; those states typically use different influencing strategies compared to large countries. An important reference was the article ‘Finland – An EU member with a Small State Identity’ by Tiilikainen (2006; see reference list in the report).
Figure 3 The Small States Theory hypotheses tested in the evaluation
The general conclusion was that Finland’s behaviour on influencing conforms well to the hypotheses developed by the Small States Theory. Furthermore, the exercise was helpful because it confirmed and provided reassurance that Finland’s approach to EU influencing is recognised more widely in research.
To read more on the Small States Theory (SST), see Annex 4 (p. 152 onwards) and Annex 5 in Volume 1.
A tiny step towards integrating futures-thinking in evaluation
The evaluation included a basic timeline of relevant future events until 2030 that bear significance to EU influencing activities (see Figure 2, p. 15 in Volume 1). The purpose was to strengthen the relevance and usefulness of evaluation conclusions and recommendations; they should take into account the evolving operational and political environment of EU influencing.
Futures-thinking is in its infancy in the field of evaluation. However, this evaluation shows that even a rudimentary overview of future events provides a useful platform for anchoring the recommendations to concrete milestones.
For the debate on the “marriage” between evaluation and futures thinking, follow Petra Mikkolainen’s LinkedIn profile. A discussion paper dealing with this matter will be published in December 2022 or January 2023.
Final thoughts
I believe that the evaluation provides added value because it shows how data-driven outcome harvesting can be combined with normative analysis (i.e. comparison against what was supposed to be achieved). The methodology worked well despite the complexity of the evaluand. Furthermore, the Small States Theory has applicability to other sectors and contexts beyond this study. Finally, the timeline of future events allows for sparking discussion on the role of futures thinking in evaluation practice.
Quick facts about the evaluation:
Commissioner: Development Evaluation Unit of the Ministry for Foreign Affairs of Finland
Evaluation Team: James Mackie (Team Leader), Petra Mikkolainen, Charlotte Vaillant, Tamara Ulla, Maria Kontro supported by Jörn Dosch, Sari Laaksonen, Daniel Di Marzo, Michael Lieckefett, Julia Schwarz, and Luciana Capuano Mascarenhas
Implemented by: Particip-NIRAS Consortium
Year: 2021-2022
Petra Mikkolainen, Senior Consultant in evaluation at Niras International Consulting
