Holger Bähr on the strengths and pitfalls of evidence-based policy advice in policy-making precesses
Evidence-based policy advice and evidence-based policy-making constitute two related concepts that are widely Evidence-based policy advice and evidence-based policy-making constitute two related concepts that are widely supported. Both political and scientific actors argue that political decisions should be based on scientific evidence in order to manage societal problems; two examples may be mentioned: the European Commission and the German National Academy of Sciences Leopoldina. By evidence-based policy advice, as it is commonly understood, individual experts, organisations or committees provide decision-makers in parliaments, governments, and public administrations with evidence. The evidence results from scientific research, and is based on empirical data and on inferences from these data that are theoretically informed and methodically controlled. The evidence is supposed to assess the impact of a political programme in advance and to evaluate a political measure after it has been decided and implemented.
Many authors have written about science, politics, and the relationship between both spheres. Some research focuses on scientific policy advice (e.g. Kraul/Stoll 2011, Lentsch/Weingart 2011, and Weingart/Wagner 2015). Authors with backgrounds in various academic disciplines and theoretical positions agree that science and politics pursue different rationales and that policy advice faces difficulties – greater or lesser depending on the theoretical perspective – when transposing scientific knowledge into policy-making. On the one hand, science strives for truth, although scientific knowledge is always preliminary and uncertain. On the other hand, politics is about power relations between political actors and societal groups and socially binding decisions are taken on the basis of interests and ideologies. Taking both rationales into account, evidence-based policy advice faces – at least and in general – two challenges. First, how can policy advice obtain evidence when scientific knowledge is uncertain? Second, how does policy advice contribute to evidence-based policy-making when politics struggles for power? Starting with the uncertainty of scientific knowledge, the following thoughts will deal with the second question. Based on studies and frameworks of the policy process, I will argue that evidence constitutes but one aspect of policy-making besides interests, ideas, values, and beliefs.
Evidence-based policy advice faces scientific uncertainty and values-led policy-making
In an enlightening article, Gebhard Kirchgässner (2013) shows how econometric analyses of empirical data result in diverging conclusions on cause and effect. As examples he uses two socially relevant issues: the relation between government size and economic growth and the deterrent effect of the death penalty. The studies analysing these relations and considered by Kirchgässner have all met sound scientific standards. Nevertheless, they present contradicting results. Thus, there is no clear evidence, whether an increase or a decrease in government size boosts economic growth. It is also unclear, whether the death penalty deters people from homicide. Minor methodical decisions and a priori assumptions during the research process have made the difference between the econometric analyses. The argument explicitly applies to the analysis of concrete empirical questions – while other questions are even contested within the scientific community between paradigms like Keynesianism and Monetarism. In his seminal book “The Honest Broker” Roger A. Pielke (2007) considers scientific uncertainty on the one hand, and interest and value driven policy-making on the other. The honest broker represents the idealised role scientific experts may hold in the policy process. The honest broker reveals theoretical perspectives and methods of scientific inference, applies scientific knowledge to policy problems and political debates, and engages with decision-makers by providing policy alternatives. Pielke argues that policy advice in the role of the honest broker is well suited to decision-making when scientific uncertainty is high and values consensus low – a situation that prevails when policy decisions are taken.
To avoid misunderstandings, evidence-based policy advice acknowledges scientific uncertainty, and evidence-based policy-making leaves decision-making to political actors. However, societal problems may be discussed differently; either expertise or values may dominate policy-making, as Alexander Bogner and Wolfgang Menz (2021) argue. They show how at the beginning of the Covid-19 pandemic political actors agreed on saving lives as the main policy goal. Scientists provided expertise on the means to achieve that goal. Questions like ways of infection spread or the effectiveness of wearing masks characterised the political debate. Later, the consensus on the policy goal weakened. Further values and policy goals entered the political debate, such as the prevention of constant unemployment, the education of children or the individual freedom of citizens. In this situation, political actors have to weigh policy goals; scientific policy advice may provide general knowledge, reveal diverging positions, and systematise argumentation. Thus, as far as political debates on values are concerned, policy advice backs political decisions by revealing the limitations of scientific evidence.
Policy goals and policy instruments are subject to values and beliefs
Policy advice provides political actors with knowledge both when policy goals are contested and when policy instruments, i.e. means to achieve these goals, are discussed in the policy process. However, evidence-based policy advice should not be equated with advice in respect of policy instruments and their effectiveness and efficiency. As with policy goals, policy instruments are also subject to normative values and ideological beliefs. In the advocacy coalition framework, Paul A. Sabatier and collaborators (Sabatier 1998, Jenkins-Smith/Nohrstedt/Weible/Ingold 2018) conceptualise both the motivation of political actors and the structure of public policies as belief systems. Encompassing three levels, a belief system consists of fundamental ontological and normative beliefs at the highest level, policy positions, strategies and instruments at the medium level, and secondary, more technical aspects at the lowest level. In this view, policy instruments realise normative values in a policy area. Minimum wage, for example, represents a means to allow employees a decent standard of living. Social democratic politicians support both the policy goal and the policy instruments. As far as the policy instrument is concerned, conservative politicians favour collective agreements, and liberal politicians trust in demand and supply at the labour market.
From the perspective of the advocacy coalition framework, scientific expertise is mainly relevant to the lowest level of the belief system. At this level, new information provided by evidence-based policy advice may affect policy decisions by clarifying dimensions of policy problems, revealing causal relations, and evaluating policy alternatives. The amount of a minimum wage is situated at that level. However, whether a minimum wage is introduced or not, mainly results from perturbations in the external environment of policy-making, as the advocacy coalition framework argues. A decrease in collective agreements covering fewer enterprises and employees, as well as a replacement of liberals by social democrats in the government coalition represent such external changes. The Fukushima nuclear disaster, which induced the nuclear power phase-out in Germany is another example taken from energy and environmental policy. While perturbations may change political actors’ attitude towards policy instruments, normative beliefs and general policy goals at the highest level of the belief system are regarded as internalised and, thus, unchangeable by evidence-based policy advice (Sabatier 1998, Jenkins-Smith/Nohrstedt/Weible/Ingold 2018).
Most policy problems are characterised by redistribution and ambiguity
Evidence-based policy advice is supposed to further the solving of societal problems. However, the process of solving societal problems is political by its nature, as Renate Mayntz and Fritz W. Scharpf (1995, Scharpf 1997) explain, using their framework of actor-centred institutionalism. Societal problems are policy problems because a single actor is unable to solve the respective problem by themselves. Either they require the resources of other actors or the problem is caused by other actors. There are some problems that may be solved just by coordination, for example if every car drives on the same side of the road – no matter if it is left or right – everybody benefits and nobody is discriminated against. However, in most cases, solving policy problems results in a redistribution of material values. For example, environmental protection is not only a matter of pollution limits and technological improvements but also involves questions of redistribution, like taxation and subsidies as well as a change in the behaviour of citizens and companies. Thus, in the policy process political actors even contest whether a situation is considered a policy problem. Climate change has long been neglected as a policy problem, and now, facing overwhelming scientific evidence for man-made climate change, aspects of redistribution still occur throughout debates of climate policy.
Evidence-based policy advice may reduce uncertainty in policy decisions. However, it may not reduce ambiguity. In ambiguous situations, political actors perceive and interpret policy problems differently and have various problem solutions of equal standard at their disposal. Further information may not resolve that situation. Taking the ambiguity of policy-making into account, John W. Kingdon (2014, Herweg/Zahariadis/Zolnhöfer 2018) argues in the multiple streams framework, that societal problems are set on the political agenda if there is an opportunity for political actors to focus attention on a problem and push a solution to that problem. Such an opportunity emerges when three processes meet: the perception of a problem specifically as a policy problem, the debate on ideas and policy alternatives, and the competition and negotiation of political actors – in Kingdon’s terms: the problem stream, the policy stream, and the politics stream join up with another and open a policy window. Policy windows may open due to changes either in the problem stream or in the politics stream. The politics stream may change as a result of a change in government, for example. In the problem stream, evidence may highlight a problem, e.g. by unemployment figures, measurements of environmental pollution or evaluations of policy measures. As a prerequisite for a policy window, at least one policy alternative has to be accepted in the policy stream. The policy stream is regarded similarly to the process of evolution: policy proposals compete and mutate and are refined and recombined. Evidence supports or opposes policy positions but political debates are mainly governed by ideas, beliefs, and values.
Evidence may strengthen policy positions
Interests, ideas, beliefs, and values drive policy-making. Furthermore, politicians employ beliefs in order to cope with myriads of information. Beliefs systematise information and provide criteria for assessing information and ending a search for information. Thus, Paul Cairney (2022) argues that the conception of policy-making as being evidence-based misses the main characteristics of the policy process; societal problems are not solved in subsequent stages of the policy process that are led by evidence. Rather, evidence-based policy making represents an ideal that actors may use in order to depoliticise societal problems. Nevertheless, evidence provided by policy advice impacts on the policy process in specific situations, as shown above. In general, evidence can support the argumentation in favour of a policy position. If, in democracies, political actors compete with one another for decision power by applying arguments in political debates, as Joseph A. Schumpeter (1993) argues, evidence-based arguments constitute an advantage over other forms of arguments, for example arguments based on fake news or religious faith, as long as citizens and political competitors appreciate argumentation and decision-making based on evidence.