Based on Ellen Mastenbroek’s inaugural lecture ‘More than the sum of its parts’, held at Radboud University, 15 September 2017 and two related articles (Van Voorst and Mastenbroek, 2017; Mastenbroek et al, 2016).
In this contribution, we critically assess the EU’s system for ex-post evaluation, using fresh empirical data. We find that the Commission has not met its repeated ambition to produce systematic and high-quality ex-post evaluation of all important EU legislation. Both the coverage and quality of ex-post evaluations have been variant. This lack of systematic high-quality evaluation is problematic, as such evaluations can fulfill important functions in the EU policy cycle: learning, accountability and improved enforcement. It also bring strategic risks for the Commission, primarily reduced credibility.
The importance of ex-post evaluation in the EU
The European Union (EU) primarily relies on legislation to reach policy aims. Much of this legislation is established for functional reasons: the desire to solve interdependent societal problems such as air pollution or terrorism. At the same time, the EU is heavily criticized. Some critics target the EU’s limited democratic character. Others criticize the system’s output: EU legislation. While some find EU laws too modest and ineffective, others argue them to be invasive, cumbersome and restrictive. The EU’s regulatory burden was one of the key themes in the British BREXIT campaign, the British telegraph arguing that BREXIT offers a ‘golden chance to throw EU regulations on a bonfire’.
The picture of a UK suffocated by EU red tape may be somewhat of an exaggeration, given the fact that it is still comparatively lightly regulated. Yet, there is no smoke without fire: EU legislation incurs both benefits and costs on member states’ authorities and target groups. They must devote time and capacity to speedy and correct implementation of EU rules, and they see their policy discretion reduced. A case in point are national parliaments. According to Vivienne Schmidt these have become arenas for ‘politics without policy’.
As many EU laws are a mixed blessing, a crucial question is whether they succeed in solving the societal problems they seek to solve. This implies a strong case for careful scrutiny of the practical implementation and effects of EU legislation. Evaluation reports may enable such scrutiny, and thus enable policy learning and accountability, two key tasks in any representative democracy. Additionally, by uncovering deficiencies in national implementation, they may form a basis for the enforcement of EU law, so as to improve implementation and effectiveness.
The Commission’s ambitions
The European Commission recognizes the importance of strengthening the ‘evidence-base’ under EU legislation. Since 2001 this is evident from its ‘better regulation’ program. Central to this program is the Commission’s desire for effectiveness: laws that really solve the problems that citizens and politicians want to be solved. In the words of Commission president Juncker: ‘Only by focusing our resources on the issues where Europe can (…) deliver results (…) will we be able to make Europe a better place’.
This ambition to produce effective legislation has primarily materialized in the first part of the EU’s regulatory cycle: policy development. As the flagship of Better Regulation, the Commission has developed an extensive system of ex ante evaluation (Radaelli and Meuwese, 2010): evaluation before legislation is enacted. Accordingly, most legislative proposals are assessed on their expected environmental, social, and economic impacts.
From ambition to practice
Since 2000, the Commission has repeatedly stated its ambition to produce systematic, high-quality ex-post evaluations of EU laws. Despite these clear statements, actual practice has lagged behind. Firstly, only 42% of all important legislation enacted in the period 2000-2004 has been followed by an evaluation report. Thus, more than half of the EU’s major directives and regulations enacted in this period still have not been evaluated. Secondly, the quality of evaluations varied greatly. Crucially, 43 percent of the ex-post evaluations scored insufficiently on a set of rather basic methodological criteria, such as replicability and robust methodology. In addition, a mere 9% of the reports indicated serious involvement of stakeholders in the evaluation process. Finally, about half of the evaluations identified concerned effects. The other reports concerned only procedural matters like transposition, implementation or enforcement of the rules by national authorities.
In sum, there is a gap between the ambitions and practice of ex-post evaluation in the European Union. This is problematic with an eye on the important democratic functions evaluations may fulfill: learning, accountability, and enforcement. Also, patchy evaluation brings a strategic risk. According to the OECD (2015), an evaluation system is only credible if its procedures for initiating evaluations are systematic and transparent. Absent or low-quality evaluations could thus reduce the credibility of the Commission. According to Poptcheva, the European Parliament already tends to disregard the Commission’s evaluations because of their variant quality.
Explaining the gap
In a recent article, we set out to uncover the reasons for the variant initiation of ex-post legislative evaluations by the European Commission. In doing so, we distinguished between two political motives. The first political motive was the enforcement of EU legislation in the member states. We expected that legislation running a higher risk of non-compliance is relatively likely to be evaluated. We thus expected directives to be evaluated more often than regulations, and evaluation to depend on the extent of legislative complexity and politicization of decision making. The second political motive was the strategic protection of competences, which is in line with the image of the Commission as a political actor seeking to expand its powers. We expected evaluations to be more likely if the relevant legislation is harder to change, which is the case under EP involvement and unanimity rather than the consultation procedure and qualified majority voting. To this, we added two control variables: the presence of an evaluation clause in the legislation, and the evaluation capacity of the DG responsible for the evaluation.
We tested these hypotheses using a dataset developed for this purpose, containing all major legislation from 2000 to 2004 and all ex-post legislative evaluations conducted by the Commission from 2000 through 2014. The findings of our study were mostly in line with the enforcement perspective: directives indeed are more likely to be evaluated ex-post than regulations. Legislative complexity also increases the chances of evaluation, while politicization does not have a significant effect. We did not find evidence for the second political perspective: the strategic protection of competences. The presence of evaluation clauses and a DG’s evaluation capacity, finally, produce further explanations of the variance in the initiation of ex-post legislative evaluations. In sum, enforcement concerns combined with evaluation clauses and evaluation capacity explain the Commission’s variant ex-post evaluation practice.
Conclusion and reflection
To conclude, the Commission has not fully lived up to its own ambitions to step up ex-post evaluation of major EU legislation, despite repeated pledges to do so. These deficiencies are problematic with an eye on EU representative democracy, given the fact that evaluations can provide an important basis for policy learning, accountability, and enforcement. They also may compromise the Commission’s credibility.
This having been said, one may wonder whether the glass is half-empty or half-full. Compared to most member states, the Commission is a frontrunner when it comes to ex-post legislative evaluation. Second, this analysis offers a snapshot only. Given the fact that the Commission has revamped its evaluation ambitions under Commission president Juncker, it may have stepped up its evaluation practice in the meantime, making it ‘fit for purpose’ after all.
Stijn van Voorst (right) is a Phd student at Tilburg Law School (Tilburg University) and part time junior lecturer at Radboud University.
- Fit for purpose? Scrutinizing the EU’s system for ex-post evaluation - November 30, 2017
