One of the primary activities of our sister organisation in Sweden, The Swedish Evaluation society SVUF (Svenska utätningsföreningen), founded in 2003, is the holding of a bi-annual conference. In 2022, after a long pause due to the Covid crisis, a large group of evaluation professionals from the Nordic countries and elsewhere in Europe gathered together in Stockholm to discuss the theme, ‘Better Societies through evaluation”.
The meeting was billed as the largest evaluation event in Norden, though an international flavour was clearly present, with numerous presentations made from across Europe. The Nordic perspective was equally strong and clearly visible: two separate Nordic Panels were organised, one to discuss the prospect of deeper collaboration between the five Nordic countries, while the other focused on social development and sustainability across the Nordics. In the panel held by the Nordic evaluation associations, Finland’s evaluation association was represented by the authors of this blog, chair of Finnish evaluation society Mari Räkköläinen and Kaisa Lähteenmäki-Smith, currently a deputy member of the SAYFES board. In this article, we examine some of the seminar contributions, particularly in terms of the current challenges faced by evaluation associations and the future opportunities for collaboration. We also try to crystallise which societal and professional themes, challenges and opportunities were discussed.
Maria Bergström (SVUF), Mari Räkköläinen & Kaisa Lähteenmäki-Smith (SAYFES). Picture SVUF
Waste is a common problem – also in evaluation
In the panel discussions, as well as in several of the introductions to the conference, attention was drawn to the many fundamental challenges we currently face in relation to the undertaking of evaluation activities. Both the status of ‘information’ as an essential part of decision-making, and the Nordic welfare state with its structures and operating methods based on accountability, transparency and critical social debate can be considered to be under pressure.
Problems with the use of evaluation knowledge and information are emblematic of this problem. In recent years, evaluation has also been marked by a ‘skeptical turn’, where many Nordic evaluation experts have self-critically reflected on the limitations of evaluation, such as the lack of time and attention in procurement organisations and the barriers to the use and usefulness of evaluation results. In addition, unrealistic expectations or costs in respect of evaluation assignments can sometimes be identified as constraints, particularly as cost consciousness increases in economic downturns. This point has been highlighted by several of the pioneers of evaluation thinking, some of whom also participated in the conference (e.g., see, Jan-Eric Furubo & Nicoletta Stame (ed.) The Evaluation Enterprise).
That waste can occur – also within evaluation – is no surprise in itself, but it is clear that even improved access to evaluation information does not guarantee that it is in fact used in parliamentary debates and decision-making. The waste and under-utilisation that often occurs in societal decision-making is also a problem in evaluation. The openness and easy availability of information also promotes its use, but even in the Nordic countries, evaluation knowledge is still wasted. The public sector has already learned how to better commission evaluation missions, but the information produced is not always used, nor when it is to a sufficient extent. Constraints such as silo-based thinking are not easily abandoned while evaluation usage can also be limited by the fact that it continues to be viewed as a rather technical and formal matter: it is ordered because it is required, but its meaning and role at the core of systemic decision-making and guidance is not always sufficiently understood.
Evaluation is not the answer to everything, but it can of course be the answer to many things! Evaluation is specifically required to ensure accountability, an information base, responsibility and legitimacy, as well as to test and analyse alternatives and to analyse the outcomes of policy actions. The conference presented a number of good examples of practices and working methods that strengthen the usability and utilisation of evaluation information. The well-known method of ‘peer review’ can be considered as one alternative that promotes a common understanding and development of evaluability and evaluation criteria. The evaluator, as a critical friend, remains a valuable partner to the decision maker, as long as their independent critical voice is retained.
Sustainability as an epistemic community and its infrastructure
In several panels, sustainability themes were discussed as a common frame of reference and as the basic goals of any evaluation process. The panel of Nordic representatives critically considered whether the production of information on sustainability is sufficient and whether current evaluation capacity is sufficient to comprehensively produce information, strengthen knowledge and contribute to closer follow-up in relation to social sustainability and its trends.
Epistemic communities are formed around shared methodological starting points and values, something which is expressed, among other things, in the sharing of special knowledge and skills, in the way of thinking and in the rhetoric of the debate. The sustainable development agenda closely associated with the SDGs already has its own living and breathing epistemic infrastructure. Its 17 goals provide the principles and vocabulary for the work being done in both the public, private and third sectors in addressing the sustainability equation, in more ways than one (see e.g. Bandola-Gill, J., Grek, S., Tichenor , M. (2022). ‘Epistemic Infrastructures: SDGs and the Making of Global Public Policy’. In: Governing the Sustainable Development Goals). The building blocks of the evaluation infrastructure are information content and the ways of describing and conveying it as an input for a common interpretation (e.g. checklists, workbooks, road maps, and other similar instruments for analysis).
On the other hand, the epistemic infrastructure also includes its community-building dimension. The social distribution of knowledge is a common theme of many different interdependencies connecting experts, decision makers, civil society and activists in their different roles and networks. Here, the evaluator can act as a bridge- and network-builder.
The third dimension could herald a new governance and management paradigm, emerging as a systemic response to the challenges of fragmented infrastructure and culture in governance. During the green transition and the sustainability crisis, global public decision-making processes are clearly coming under increasing pressure, but sustainability can also provide new tools for building new infrastructure solutions. Beyond the green transition or double transition (SDGs and digitalisation) presented at the conference, the SDGs can form an epistemic infrastructure that can help to structure our understanding and knowledge base for evaluation. At the same time, we can perhaps foresee a triple transition, where increasing participation and thus democratic responsibility, is included together with ecological and digital transformation and their evaluation.
Evaluation as a profession and community
The Nordic panel discussion began with a round up of the various nation evaluation associations’ histories and their varying roles in the five Nordic countries. The importance of the professional peer community was felt to be significant in all five. At best, the effect is liberating and empowering and it is considered important to have a professional peer community where one can talk about evaluation in depth and comprehensively,with many of the participants claiming the identity of ‘evaluation geek’ as their own.
Across the Nordic countries, the associations generally have strong foundations, the Finnish Evaluation Association (founded in 1999) was, somewhat surprisingly, the first of its kind. Sweden’s corresponding association was founded in 2003, Norway’s in 2009 and Denmark’s in 2000. Iceland currently has no association of its own, only a small group of evaluation professionals, several of whom work in the field of development policy. Small and fragmented communities particularly benefit from professional peer learning platforms for evaluation communities, the use of which could be further expanded.
All of the Nordic associations are currently striving to strengthen the evaluation culture, they take care of the development of evaluation competence and aim to promote evaluation as a profession. On the other hand, the associations also recognise the dangers associated with strict qualification requirements for evaluators which can lead to one-sidedness in the evaluation activity and a narrowing of the task and, ultimately, to closed or isolated communities. At worst, the result can induce a cycle of curtailment in respect of evaluation skills and a weakening of their effectiveness. The associations want to offer a forum that brings together and involves evaluation actors from different corners of society, from the public, private and third sectors. As such, strict professional requirements are not suitable here. Multifunctional associations strive to strengthen the visibility of evaluation in important social processes by advocating for data-based decision-making encompassing a broad spectrum of political opinion. On the other hand, evaluators want to keep a sufficient distance from political control in order to guarantee the independence of the evaluation. What evaluators have in common is a belief in the importance of evaluation communities and the work they do in the pursuit of better societies.
Evaluations of education and development policy are often seen as pioneers for evaluation in the Nordic countries. The importance of global actors, international standards and funders in the professionalisation of evaluation has been higher than the average in these sectors which has created a need for, among other things, accountability evaluation and its resources.
On the other hand, the Nordic countries differ in terms of the status of assessment as a discipline and in the structures of learning and education. Although all Nordic countries are united by an emphasis of certain policy areas in the general area of evaluation expertise and markets, there are significant differences in the type of academic training available to evaluators. In Finland, there is no special chair for evaluation, with education in this area focusing on professional development or continuing education rather than on introducing students to areas of evaluation and even scientific evaluation research. The role of higher education and evaluation science in scientific communities and in the tertiary education sphere thus varies. Denmark and Sweden in particular are countries where evaluation is also taught at university level and where eminent academic chairs in the field have been established.
All Nordic evaluation groups however face a common challenge to remain relevant and important. The need to profile oneself in the social debate and to maintain evaluation as one of the research areas where research and knowledge work related to society and its upheavals is conducted, is common. With the Finnish Evaluation Association’s new award, Evaluation Examination of the Year, together with the universities, we want to promote both the production of research information on evaluation and to strengthen competence in evaluation research. Over time, this collaboration can also promote evaluation’s status as a profession in its own right. The Evaluation Association of Finland also makes evaluation expertise visible through the annual awards day award for significant evaluation work or lifelong career in evaluation. Other Nordic associations also have similar awards which make the association’s activities visible and emphasise the special role of evaluation in societal processes.
A previous European comparative study has shown that Finland’s ranking amongst its peers in respect of the professionalisation of evaluation is weakened precisely by the lack of professional academic training, such as the existence of master’s programmes in evaluation which in turn affects the development of evaluation expertise. Although Finland ranks first in terms of the institutionalisation comparison on other criteria, we do not have national evaluation legislation or comprehensive national guidance for evaluation activities that would be binding and would cross the boundaries of government sectors. In some cases, Finland also lags behind the leading European countries in this field in terms of the demand for and utilisation of evaluation data. The other Nordic countries often also grapple with the same challenges. (See, in more detail, Stockmann R., Meyer W., & Taube L. (toim.): The institutionalisation of Evaluation in Europe)
Conclusions for closer future collaboration
The Nordic evaluation communities are brought together by these basic challenges, for example in terms of knowledge and information use, with regard to professional identity and peer community and in relation to globally relevant information needs such as sustainability and the green transition which were the central themes of the conference.
In the Nordic countries there is a common expectation that evaluation communities will play a key role both in strengthening evaluation awareness in political decision-making and in highlighting the importance of evaluation and bringing it to the attention of the general public. The joint task here was to promote the Nordic culture of democratic evaluation. Together, the Nordic communities could develop into a modern learning community which also offers avenues for the development of evaluation skills, mentoring and opportunities for peer learning and joint development. The various Nordic national associations could also draft and design a common mission and vision to guide their activities. In the panel discussion, the Nordic associations already informally agreed to intensify cooperation and join forces. Strengthening the ‘Nordic perspective’ of the evaluation could also contribute to increasing the effectiveness of the evaluation when solving global challenges and our common sustainability crisis.
Link to the seminar: https://svuf.nu/ and to the presentations and workshop sessions: https://svuf.nu/konferens-2022/
