kom (2018) 0507 - Ingen titel

Europaudvalget 2018
KOM (2018) 0507
Offentligt

EUROPEAN

COMMISSION

Brussels, 29.6.2018

SWD(2018) 356 final

COMMISSION STAFF WORKING DOCUMENT

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE

COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE

COMMITTEE OF THE REGIONS

on the interim evaluation of the implementation of the Justice Programme 2014-2020

{COM(2018) 507 final} - {SWD(2018) 357 final}

kom (2018) 0507 - Ingen titel

Table of contents

INTRODUCTION ................................................................................................................................ 3

BACKGROUND TO THE INITIATIVE ............................................................................................. 4

IMPLEMENTATION / STATE OF PLAY ........................................................................................ 10

METHOD ........................................................................................................................................... 17

ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS .......................................... 20

CONCLUSIONS AND WAY FORWARD ........................................................................................ 25

ANNEX 1: PROCEDURAL INFORMATION ............................................................................................ 28

ANNEX 2: STAKEHOLDERS CONSULTATION .................................................................................... 31

ANNEX 3: METHODS ................................................................................................................................ 33

kom (2018) 0507 - Ingen titel

NTRODUCTION

Purpose and scope of the evaluation

This staff working document sets out the results of the interim evaluation of the Justice

Programme for the period 2014-2020.

The evaluation was carried out under the provisions of Article 14(2)(b) of the European

Parliament and Council Regulation establishing the Justice Programme for the period

2014-2020

It aimed mainly to assess the Programme’s outputs and results compared to its objectives

and to assess qualitative and quantitative aspects of its implementation. It also assessed

whether the current Programme is on track to achieving its objectives and the extent to

which the recommendations of the previous 2007-2013 ex-post evaluation have been

followed. The ex-post evaluation carried out in 2015 concerned the three predecessor

programmes which were merged in the current Justice Programme (namely the Civil

justice programme, the Criminal justice programme and the Drug prevention and

information programme)

The current interim evaluation informed the Report from the Commission to the

European Parliament, the Council of the European Union, the European Economic and

Social Committee and the Committee of Regions which this document accompanies. It

also helped the reflection on the Programme's future by providing evidence-based

information.

Under the Regulation, the Commission is requested to present an ex-post evaluation

report for the Justice Programme by 31 December 2021.

The Programme was assessed based on the following main

evaluation criteria:

1. Relevance:

whether and to what extent the Justice Programme addresses needs and

problems of the target groups identified in the 2011 Impact Assessment

and in the legal

basis of the Programme (as well as emergent needs related to the creation of a European

Area of Justice) and whether its objectives are still relevant for the needs and problems of

the beneficiaries;

2. Efficiency:

whether and to what extent the costs of the Programme were proportionate

given the benefits achieved and which parameters/factors participated in these results;

3. Effectiveness:

whether and to what extent the Justice Programme has achieved its

general objective, as well as its four specific objectives, and which are the factors that

have contributed to these achievements;

Regulation (EU) No 1382/2013 of the European Parliament and of the Council of 17 December 2013 establishing a Justice

Programme for the period 2014 to 2020 (OJ L 354 of 28.12.2013), available online at:

http://eur-lex.europa.eu/legal-

content/EN/TXT/PDF/?uri=CELEX:32013R1381&from=EN.

Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective

–

Final Report. Specific programme

evaluation:

Civil

Justice,

available

here

http://ec.europa.eu/justice/grants1/files/expost_evaluations_reports_2007_2013/civil_justice_programme.pdf

. Ex-post evaluation of

five programmes implemented under the 2007-2013 financial perspective - specific programme evaluation: Criminal Justice Support

Programme,

available

here

http://ec.europa.eu/justice/grants1/files/expost_evaluations_reports_2007_2013/criminal_justice_programme.pdf.

Ex-post evaluation

of five programmes implemented under the 2007-2013 financial perspective Specific programme evaluation: Drug Prevention and

Information

Programme,

available

here

http://ec.europa.eu/justice/grants1/files/expost_evaluations_reports_2007_2013/dpip_programme.pdf

SEC (2011) 1364 Final.

kom (2018) 0507 - Ingen titel

4. Coherence/Complementarity/Synergies:

whether and to what extent the Programme

is coherent with other interventions at the EU and international level, such as with the

predecessor EU programmes in the field

, with activities supported by other Union

instruments and, in general, with the European priorities in the fields covered by the

Programme;

5. EU added-value:

to what extent the effects from the EU action are additional to the

value that would have resulted from action at the national level only;

6. Equity:

whether and to what extent the Justice Programme has distributed the

available resources fairly among beneficiaries in different Member States, took into

consideration the needs of target groups, promoted gender mainstreaming, the rights of

the child and the rights of people with disabilities

;

7. Scope for simplification:

whether and to what extent the management of the Justice

Programme could be further simplified.

Geographically,

the Programme is open to all EU Member States

, but also to the

European Free Trade Association States that are party to the European Economic Area,

candidate countries, potential candidates and countries acceding to the Union, provided

that they conclude an agreement with the Union laying down the details of their

respective participation in the programme. Albania joined the Programme as of 2017.

The

reference period

for this interim evaluation is the first half time of the Programme’s

implementation from 2014 to mid-2017

ACKGROUND TO THE INITIATIVE

The Treaty on the Functioning of the European Union provides for the creation of a

European area of freedom, security and justice based on mutual recognition of judicial

decisions and mutual trust among Member States, in which persons are free to move and

can rely on the respect of fundamental rights as well as of common principles, such as

non-discrimination, gender equality, effective access to justice for all, the rule of law and

well-functioning independent judicial systems.

These ambitious goals, set by the Treaty, have also been reaffirmed by the European

Council in the Stockholm Programme

. The achievement of a Europe of law and justice

is one of the political priorities of the EU and the 2014-2020 Justice Programme is one of

the EU instruments that contribute achieving this objective.

The Programme was designed to overcome the obstacles in the functioning of an

effective European Area of Justice

and to encourage national judicial systems to have

faith in each other’s standards of fairness and justice,

as prerequisite for the realisation of

a European Area of Justice. The Programme also contributes directly to the Juncker

As already mentioned, the Justice Programme has replaced three programmes that were in force during the 2007-2013 Multi-annual

Financial Framework.

In this context, “equity” does not mean necessarily equality (i.e. that target groups and Member States receive the same level

support in terms of projects funded and resources allocated). Instead, the concept of equity relates to the allocation of resources

mirroring the different intensity of, and urgency for, support of each target group/Member States rather than for example their

capacity for advocacy or in accessing the Programme funds/managing projects.

The United Kingdom and Denmark do not participate in the Programme.

Given its relatively early stage of implementation, the 2017 Annual Working Programme has been analysed in terms of design and

structure, not in terms of execution.

OJ C 115, 4.5.2010, p. 1.

Such as, in particular, insufficient judicial cooperation in civil and criminal matters, wide differences in national procedures and laws

across the EU, lack of trust from EU citizens in the functioning of judicial systems of other Member States, lack of awareness and

difficulties in the exercise of EU citizenship rights by people, limited improvements in preventing and combating drug-related crimes.

kom (2018) 0507 - Ingen titel

priority aimed at establishing an area of justice built on respect of fundamental rights and

mutual trust, and digital single market.

Description of the programme

The general objective

of the Justice Programme is to contribute to the development of a

European area of justice based on mutual recognition and mutual trust, in particular by

promoting judicial cooperation in civil and criminal matters.

In order to do so, the Justice Programme provides for four

specific objectives

which aim

to:



facilitate and support

judicial cooperation

in civil and criminal matters. Through this

objective, the Programme supports activities that contribute to:



the effective and coherent application of the EU acquis relating to judicial

cooperation in civil and criminal matters also by building-up and/or

improving data collection and statistics on the application of the EU acquis;

the enforcement of EU instruments and judicial decisions, in particular

resulting from cross-border disputes;

the improvement of the exchange of information among professionals in order

to enhance the operational cooperation and mutual trust in the EU.



Under this specific objective, the Programme supports the participation of the EU to

the Hague Conference of Private International Law

and activities related to the

setting up and strengthening of national networks under the European Judicial

Network in civil and commercial areas. Moreover, specific calls for proposals to

prevent radicalisation leading to terrorism and violent extremism have also been

launched in the wake of terrorism episodes in the EU. In the context of this specific

objective, the EU carries-out its cooperation with the Council of Europe on two

particular issues: the SPACE report on prison statistics and the setting up of the EU

network of prison monitoring bodies in the Member States.



support and promote

judicial training

of professionals such as judges,

prosecutors, notaries, prison staff and lawyers on civil and criminal law EU

instruments, fundamental rights, judicial ethics and the rule of law, including language

training on legal terminology, with a view to fostering a common legal and judicial

culture. Judicial

training can involve “basic” components, such linguistic skills and

terminology, and more specialised aspects, such as seminars on specific aspects of

both civil and criminal law, e-learning and exchanges of staff and experience.

For

example, specialised training is made available in the field of competition law, which

typically requires a significant amount of specialisation. The Programme supports, in

particular, the training of the members of the judiciary and judicial staff, but also other

legal practitioners associated with the judiciary. It also funds the development of tools

for training providers and co-finances the expenditure associated with the work of the

European Judicial Training Network.

Certain areas represent priorities for the training activities: EU civil, criminal and

fundamental rights law, legal systems of the Member States, judicial ethics and the

rule of law, knowledge of cross-border IT tools and linguistic skills.

The EU is party to The Hague Conference on Private International Law since in 2007. The purpose of this international

intergovernmental organisation is to work for the progressive unification of the rules of private international law in the participating

countries. Most of the conventions developed by The Hague Conference fall within exclusive or partial external competence of the

EU and are part of the EU

acquis.

European Commission, Directorate-General for Justice and Consumers, European Judicial Training 2016. Available online at the

following address: http://ec.europa.eu/justice/criminal/files/final_report_2015_en.pdf.

kom (2018) 0507 - Ingen titel

Moreover, the Programme finances projects which aim at achieving the objectives of

the European e-Justice Strategy 2014-2018

and that contribute to the effective and

coherent application of EU criminal law in the areas of the rights of persons suspected

or accused of crime (procedural rights priority) and of the rights of victims of crime;



facilitate effective

access to justice

for all, including to promote and support the rights

of victims of crime as well as the procedural rights of suspects and accused persons in

criminal proceedings. Actions financed in this area by the Programme aim at

providing EU citizens with effective remedies in case of violation of EU law,

especially where national procedures are too difficult for citizens to be enabled. In

particular, the Programme aims to promote the use of other types of remedies and

non-remedies developed in the EU that can provide a quick, efficient and less costly

solution to disputes, as supported, for example, by the e-Justice Portal.

It also aims

to encourage a close cooperation between national authorities or administrative

bodies, which is particularly important for the effectiveness of certain EU rights.



promote

initiatives in the field of drug policy

as regards judicial cooperation and

crime prevention aspects

. In this field, the Justice Programme promotes initiatives

which focus on judicial cooperation and crime prevention. The activities under the

Programme mainly aim to promote practical application of drug-related research, to

support civil society organisations and key stakeholders and to expand the knowledge

base and develop innovative methods of addressing the phenomenon of new

psychoactive substances.

Budget

The initial total Justice programme's budget for the period 2014-2020 is

EUR

377 604 000.

According to available sources, the

total requested and committed EU

contribution still not reached the amount planned, despite the improvements made

in 2016

(see Table 1). In general, most resources allocated to action grants and operating

grants have been committed (with a commitment rate of close to 90%), but the

commitment rate for procurement activities was much lower, hovering around 60%-70%.

Table 1: Total amount planned and committed in the Justice Programme 2014-2016

Budget year

2014

2015

2016

Total 2014-2016

Annual amount

planned (in euro)

45 812 000.00

47 226 649.00

50 250 000.00

143 288 649.00

Annual amount

committed (in euro)

36 671 240.16

39 675 719.11

47 535 032.81

123 881 992.08

Ratio

committed/planned

80.05%

84.01%

94.60%

86.46%

Council of the European Union,

Multiannual European e-Justice Action Plan 2014-2018,

2014/C 182/02, available here

https://publications.europa.eu/en/publication-detail/-/publication/2a2f84b9-f3a4-11e3-831f-01aa75ed71a1/language-en.

Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and

the Committee of the Regions, The EU Justice Agenda for 2020 - Strengthening Trust, Mobility and Growth within the Union,

Strasbourg, 11.3.2014, COM(2014) 144 final.

Insofar as this kind of initiatives are not covered by the Internal Security Fund for financial support for police cooperation,

preventing and combating crime and crisis management (for more info, see

https://ec.europa.eu/home-

affairs/financing/fundings/security-and-safeguarding-liberties/internal-security-fund-police_en

), or by the Health for Growth

Programme (for more info, see

https://ec.europa.eu/health/funding/programme/2014-2020_en

kom (2018) 0507 - Ingen titel

Sources: Annual monitoring reports (reports on the implementation of the Annual working programmes and data extracted from

Sygma for 2016)

In order to allow the achievement of its objectives, the Programme identifies a wide

range of activities to be implemented. In particular, according to Article 6 of the

Regulation establishing the Justice Programme for the period 2014 to 2020, the

Programme can finance:



Analytical activities,

in particular: collection of data and statistics; development of

common methodologies and, where appropriate, indicators or benchmarks; studies,

researches, analyses and surveys; evaluations; elaboration and publication of guides,

reports and educational material; workshops, seminars, experts meetings and

conferences;



Mutual learning, cooperation, awareness raising and dissemination activities,

particular: identification of, and exchanges concerning good practices, innovative

approaches and experiences; organisation of peer reviews and mutual learning;

organisation of conferences, seminars, information campaigns; compilation and

publication of materials to disseminate information about the Programme and its

results; development, operation and maintenance of systems and tools, using

information and communication technologies, including the further development of

the European e-Justice portal as a tool to improve citizens' access to justice;



Training activities,

for instance: staff exchanges, workshops, seminars, train-the-

trainer events, including language training on legal terminology, and the development

of online training tools or other training modules for members of the judiciary and

judicial staff;



Actions to support the main actors

whose activities contribute to the

implementation of the objectives of the Programme, such as: support for Member

States in the implementation of Union law and policies; support for key European

actors and European-level networks, including in the field of judicial training; support

for networking activities at European level among specialised bodies and entities as

well as national, regional and local authorities and non-governmental organisations

The main

stakeholders

eligible for support are: European networks active in the area of

judicial cooperation; justice professionals; public or private organisations (usually non-

profit-oriented, but in particular cases also profit-oriented organisations

); national,

regional and local authorities in EU Member States; non-governmental organisations;

national judicial schools of individual Member States; universities and research

institutions as well as international organisations active in the areas covered by the

Programme.

In terms of

target groups,

intended as the groups that can benefit directly or indirectly

from the Programme, potentially all EU citizens are included, since the Programme aims

at creating a European Area of Justice where all citizens are aware of and can exercise

their rights. However, in particular, the Justice Programme covers: judicial practitioners,

public authorities, universities, international organisations, non-governmental

organisations and other research entities; companies undergoing insolvency or pre-

insolvency proceedings; citizens accused or victim of a crime, families, consumers,

minors, groups in need of specialised legal protection and subjects at risk of

To see the types of activities funded by the Justice Programme in 2014, 2015 and 2016, see the Mid-term evaluation of the Justice

Programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018, p. 71-72.

Bodies and entities which are profit-oriented can be funded only in conjunction with non-profit or public organisations.

kom (2018) 0507 - Ingen titel

radicalisation. Additionally, specific calls support entrepreneurs in the single market and

national drug awareness and prevention agencies

The Justice Programme has

three main funding mechanisms:

action grants, operating

grants and procurement actions



Action grants

are addressed to civil society organizations, to Member States

authorities and universities;



Operating grants

("support to Networks") fund mainly European networks active in

the area of facilitating and supporting judicial cooperation in civil and/or criminal

matters that have signed Framework Partnership Agreements with the Commission;



Procurement actions

("Commission initiatives") fund mostly conferences, expert

meetings, seminars, awareness-raising activities and studies, but also translation

services and IT projects.

The Programme intervention logic is outlined in the Figure 1.

External factors could influence the Programme's outcomes

and make challenging to

identify changes as a result of its intervention. In particular, the economic crisis has led to

a lack of interest on the part of public sector stakeholders in the areas covered by the

Programme (particularly in the field of access to justice) and, therefore, to a related

reduction of national funding for the justice systems. This is more accentuated in certain

Member States where the political climate is less supportive

. Also the uneven pace at

which Member States incorporate the

EU acquis

in their national legislations is a factor

influencing the performance of the Programme

Baseline and points of comparison

For more information, see below p. 15 and see also section 5.4 on "Target groups and beneficiaries" in the Interim report, p. 77 et

seq.

For more information on the number of action grants and operating grants awarded per year and on the number of procurement

activities, see section 5 "Implementation state of play" in the Interim report, in particular p. 69 et seq.

Mid-term evaluation of the Justice Programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018,

published.

Ibid.

kom (2018) 0507 - Ingen titel

The interim evaluation assessed the Justice Programme performance starting from the

situation described in the 2011 Impact Assessment

and in the ex-post evaluation of

2007-2013, concerning the three predecessors programmes (Civil justice programme, the

Criminal justice programme and the Drug prevention and information programme),

carried out in 2015

These documents contain the baselines and main points of comparison for measuring the

Programme’s achievements during the reporting period.

The baseline analysis showed that, if no changes had been made between the 2007-13

and 2014-20 programming period, the three previous programmes would have continued

to be successfully implemented, but at a reduced potential in particular due to:

The

lack of flexibility in the funding instruments which did not reflect the pace of change and

reform in this policy area;

The fragmentation of funding that reduced the capacity of

the programmes to deliver results in horizontal and cross-cutting issues; and

The

elevated number of different funding instruments which increased the administrative

burden.

Given these difficulties, the Impact Assessment suggested the option to consolidate the

three programmes in the current Justice Programme.

This possibility was assessed in terms of its relevance/scope, effectiveness,

complementarity, European added value, efficiency and potential for simplification.

Since the option to proceed with only one programme was undertaken, the findings of the

Impact Assessment, together with the recommendations of the ex-post evaluations of

2007-2013, provide a useful baseline to verify whether the current Justice Programme

achieved its expected goals.

Follow-up on the ex-post evaluation report of the period 2007-2013

All the ex-post evaluation reports concerning the previous funding programmes

confirmed their overall effectiveness and highlighted that their specific objectives and

priorities were largely specific, attainable and realistic, but not always measurable.

Indicators, allowing the measuring of progress toward the attainment of the specific

objectives, were not included.

Moreover, the reports pointed out the need to make improvements on other issues which

could have increased the impact, added value, effectiveness and efficiency in delivering

result of the programmes. In particular, the reports have identified the following

needs

common to the three programmes:

1. Better definition of the priorities

in order to ensure that they can be adequately

achieved within an earmarked budget;

2. Realistic assessments of project risks and better risk mitigation strategies

throughout the projects duration;

3. Increase focus on assessment of impacts at all levels

and not merely on outputs, as

regards monitoring and evaluation;

4. Explore ways of enhancing the uptake of project outputs, results and best

practices

by other organisations, including in other Member States;

5. More dissemination/use of results and outputs

of the funded activities.

European Commission, 'Commission Staff Working Paper

–

Impact Assessment - Accompanying the document Proposal for a

Regulation of the European Parliament and of the Council establishing for the Period 2014-2020 the Justice Programme. Impact

Assessment’. SEC

(2011) 1364 Final.

Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective

–

Final Report, ibid.

kom (2018) 0507 - Ingen titel

The ex-post evaluation reports of the period 2007-2013 have also highlighted the

following

problems: 1. The dilution of funds

amongst many small-scale projects with

limited impact and EU dimension;

2. No balanced geographical spread among the

organisations

which receive funding;

The

complex and bureaucratic procedures

for

the applicants;

The

high administrative burden

on the Commission and an increase

of the

length of procedures

due to the multiplication of procedures for the different

programmes.

These issues were taken into consideration and were either integrated into the Regulation

establishing the current Justice Programme and in the Annual Work Programmes or

translated into technical implementation measures and included in the Guides for

applicants to projects calls.

In particular,

the merger of the three previous programmes

has provided a

significant

positive impact on the identified problems of scope, effectiveness, fragmentation and

efficiency

(see further details in section 5 “Analysis and

answer to the evaluation

questions”).

MPLEMENTATION

STATE OF

LAY

Programme management

The Justice Programme is implemented via direct centralised management by the

Commission. This was meant to ensure a close relationship between the Programme

management and EU policy-making and to contribute to the alignment of budget

implementation with EU policy priorities in line with the "budget for result" approach.

This management mode allows the Commission to tailor funded activities to policy

priorities and policy needs and to target directly the relevant groups of stakeholders.

According to the evaluation carried out, it also allows a close contact with the

programmes' beneficiaries and better knowledge of the needs on the ground. Indeed, all

beneficiaries interviewed, who also participated in predecessor programmes, agreed that

the current Justice Programme represented an improvement in terms of capacity to

engage with European Commission officials (i.e. having a reliable point of contact to

answer their questions and help them address any project-related issues that may arise

during the implementation phase)

State of play

The purpose of this section is to present the state of play of the Programme’s

implementation and its key initiatives in the period 2014-2017. It will also provide the

qualitative and quantitative results achieved in the same period, in general terms and for

each specific objective.

Although, the beneficiaries interviewed were not specifically asked to describe their experience with other kinds of Programme

management.

kom (2018) 0507 - Ingen titel

In order to measure the progress from the baseline situation, a series of Programme

indicators have been devised for the current Justice Programme. The introduction of a

system of indicators has proved to be adequate for measuring the achievements of the

Programme since the selected indicators show if the targets are close to being achieved or

have already been achieved and this contributes to better focus the Programme's

outcomes.



General objective:

The development of a European area of justice based on

mutual recognition and mutual trust

The Regulation establishing the Justice Programme does not provide for any indicator

concerning its general objective. However, the Commission reports via the Programme

Statements, accompanying the Draft General Budget of the EU

, use the percentage of

legal practitioners trained (not only through this Programme) as the main indicator for

measuring the Justice Programme’s impact, as shown in Table 2.

Table 2: Result indicator, target and actual value

Result indicator

1. Cumulative number of legal professionals

receiving training (not only through the

Programme) on EU law or law of another

Member State, including Civil Justice,

Criminal Justice and Fundamental Rights

Baseline

2013

239 000

2016

638 000

Targets 2020

700 000 legal

practitioners by 2020

Source: Annual reports on European Judicial Training

The 2020 target was almost reached already in 2017

. The graphic below (Figure 2)

shows clearly that the number of legal practitioners trained has constantly increased

between 2013 and 2016.

Figure 2: number of legal practitioners trained

800000

700000

600000

500000

400000

300000

200000

100000

2013 (baseline)

2014

2015

2016

2020 (target)

COM (2017) 400 - May 2017, Draft Budget of the European Union 2017. Information on indicators related to the general objectives

of the Programmes and indicators related to the specific objectives of the Programmes are provided by this document.

Moreover, gaps in data described in the training reports lead to surmise that the number of trained judicial staff is somewhat

underreported.

kom (2018) 0507 - Ingen titel

Concerning the Specific objectives of the Programme,

the following indicators have

been used to evaluate its performance:



Specific objective 1: Judicial cooperation in civil and criminal matters

The Programme's performance under this specific objective is measured through two

main indicators (see Table 3): the average time of the surrender procedure under the

European Arrest Warrant in cases where the person consents to the surrender and the

number of exchanges of information in the European Criminal Records Information

System. The European Arrest Warrant is the most important EU legal instrument

developed in the area of mutual recognition of judicial decisions. It consists in a

simplified cross-border judicial surrender procedure for the purpose of prosecuting or

executing a custodial sentence or detention order (a warrant issued by one EU Member

State's judicial authority is valid in the entire territory of the EU)

The European Criminal Records Information System is a database established to improve

the exchange of information on criminal records throughout the EU. All EU Member

States are currently connected to this system .

Concerning the

first indicator,

the data appears to be highly influenced by exogenous

factors (such as the different degree of incorporation of EU directives at national level, as

well as the different degree of judicial system reforms within EU Member States that

have a significant impact on the level of enforcement of the EU

acquis

) that make

estimating the contribution of the Justice Programme to the realization of this indicator

difficult. Concerning the

second indicator,

after a rapid increase post the 2012 baseline,

the number of exchanges appears to have stabilised and it is likely that this objective will

be reached.

Table 3: Result indicators, target and actual values

Result indicators

Average time of the surrender

procedure (time between the

arrest and the decision on the

surrender of the person sought)

under the European Arrest

Warrant in cases where the

person consents to the surrender

Number of exchanges of

information in the European

Criminal Records Information

System

Baseline 2013

14-20 days (according to DG

Justice

aggregations

national reports of Member

States to the Council)

2015-2017

14 days (2015)

Targets 2020

10 days by 2020

300 000 exchanges (2012)

2 571 991 (2017)

3 500 000

exchanges by

2020



Specific objective 2: Judicial training

The Programme's performance concerning this specific objective is measured through

one main indicator: the number and percentage of members of the judiciary and judicial

The European arrest warrant has been operational since 1 January 2004. It has replaced the extradition procedure that used to exist

between EU countries. At present, a person who has committed a serious crime in a Member State, but who lives in another can be

returned to the first country to face justice quickly and with little administrative burden. For more info, see Council Framework

Decision of 13 June 2002 on the European arrest warrant and the surrender procedures between Member States (2002/584/JHA).

This database ensures that information on convictions is exchanged between EU Member States in a uniform, fast and compatible

way, provides judges and prosecutors with easy access to comprehensive information on the criminal history of persons concerned,

including in which EU Member States that person has previously been convicted and removes the possibility for offenders to escape

convictions by moving from one Member State to another. On 28 June 2017, the European Commission published its first report on

the use of this database by EU countries.

These obstacles were mentioned by several projects and shown also by the "Justice Scorecards".

kom (2018) 0507 - Ingen titel

staff that participated in training activities, staff exchanges, study visits, workshops and

seminars funded by the Programme.

This indicator appears well-adapted to the goals of the specific objective and of the

Programme, as it helps track both the overall contribution of the EU funding instruments

of the European Commission and the specific contribution of the Justice Programme. It is

clearly measurable and it is showing that, looking at the entire Commission, the goal of

training 20 000 practitioners yearly by 2020 has been already achieved (see Table 4). The

drop in 2016 below this threshold is likely misleading, as data from the European Social

Fund-funded judicial training is currently missing

. As ca. 3 000 legal practitioners were

trained in 2015 with European Social Fund funding, it is quite likely that once this data

are reported, the objective will have been achieved for 2016 as well.

Table 4: Result indicator, target and actual value

Result indicators

Number and percentage of

members of the judiciary and

judicial staff that participated in

training

activities,

staff

exchanges,

study

visits,

workshops and seminars funded

by the Programme

Baseline 2011

For the whole Commission:

8 639 in 2011

2015

25 680

2016

18 444

Targets 2020

20 000 by 2020

(for the whole

Commission)

For DG Justice

Consumers:

6 681

and

16 723

13 930



Specific objective 3: Access to justice

The Programme's performance concerning this specific objective is measured through

two main indicators: the number of hits on the e-justice portal and the number of Victim

Support Organisations with national coverage (see Table 5). The e-justice portal provides

information on justice systems and improves and facilitates the access to justice

throughout the EU, in 23 languages

. The number of Victim Support Organisations is an

important indicator to assess the implementation of Directive 2012/29/EU establishing

minimum standards for the rights, support and protection of victims of crime. This

directive contains important procedural provisions for victims of crime regarding, in

particular, their right to be heard and have access to support services (i.e. Victim Support

Organisations).

Concerning the

first indicator,

it has already exceeded the stated goals for 2020.

Concerning the

second indicator,

it seems likely that it will be reached, though it partly

depends on whether Member States/Victim Support Organisations will implement

quality standards, which is partially out of the control of the Justice Programme and

could be influenced by exogenous factors, such as the uneven pace at which Member

States implement and enforce the EU

acquis

or other political priorities or emergencies in

the Member States (in particular the economic crises has weakened the civil society

engagement for this specific objective).

Table 5: Result indicators, target and actual values

Result indicators

Number of hits on the e-justice

portal

Baseline 2013

630 000 (2012)

2017

2 690 574

Targets 2020

+ 20% per year

For more info on the European Social Fund, see

http://ec.europa.eu/social/main.jsp?catId=325&langId=en.

More info available at the following link:

https://e-justice.europa.eu/home.

Source of data: DG Justice and Consumers.

kom (2018) 0507 - Ingen titel

Number of Victim Support

Organisations with national coverage

(implementation

Directive

2012/29/EU)

10 Victim Support

Organisations

22 Victim Support

Organisations

Victim

Support

Organisations

–

at least one

Victim Support Organisation

in each Member State

fulfilling

the

quality

standards/indicators set by

Member States or developed

individually by the Victim

Support Organisations (by

2020)



Specific objective 4: Drugs policy

The Programme's performance concerning this specific objective is measured through

two main indicators: the number of new psychoactive substances assessed (including

through testing, if necessary) to enable the EU or the Member States to take appropriate

action to protect consumers, depending on the type and level of risk that they may pose

when consumed by humans and the % of problem opioid users that are in drug treatment.

Given the stability observed over time of

indicator 2

(which has remained at the 50% for

the last five years), it seems difficult that this percentage could reach 60% in 2020.

Concerning the

first indicator,

the reverse is true, as figures appear to fluctuate

significantly. The 2020 yearly goal was reached in 2015, but in 2016 the number returned

near to the 2012 baseline (see Table 6). This indicator can be influenced by different

exogenous factors, such as macroeconomic conditions, rate of development of new

psychoactive substances, etc.

Table 6: Result indicator, target and actual value

Result indicators

Number of new psychoactive substances

assessed (including through testing, if

necessary) to enable the EU or the Member

States to take appropriate action to protect

consumers, depending on the type and level

of risk that they may pose when consumed

by humans

% of problem opioid users that are in

drug treatment

Baseline

68 (2012)

2014/2015/2016

100 (2015)

66 (2016)

Targets

2020

95 (by 2020)

50% (2011)

50% (2014, 2015

and 2016)

60% (by 2020)

Moreover, a series of Programme-related indicators are also mentioned in the legal basis

of the Justice Programme

. They provide a basis for monitoring and evaluating the

extent to which each of the Programme's specific objectives has been achieved. These

are:

The number of persons reached by awareness-raising activities funded by the

Programme;

The improvement in the level of knowledge of Union law and policies in

the groups participating in activities funded by the Programme compared to the entire

target group;

The number of cases, activities and outputs of cross-border cooperation,

including cooperation by means of information technology tools and procedures

established at Union level;

Participants' assessment of the activities in which they

participated and of their (expected) sustainability; and

5.The

geographical coverage of

the activities funded by the Programme.

Art. 15 of the Regulation (EU) 1382/2013 of 17 December 2013 establishing a Justice Programme for the period 2014 to 2020.

kom (2018) 0507 - Ingen titel

Programme indicators 1 and 3

relate to the outputs and outcomes of funded activities

and can be calculated through the final reports of the projects. At the moment, there are

not sufficient available data to come to consolidated conclusion on these indicators.

Programme indicators 2 and 4

relate to the perception of target groups concerning the

activities they participated in and the level of knowledge they acquired. The indicator 4 is

not measurable at this stage, instead for the indicator 2, no surveys on the perception of

target groups were deployed and, therefore, it cannot be measured.

Programme

Indicator 5

speaks to the equity in distribution of activities funded by the Programme

amongst beneficiaries in the 26 participating Member States. For projects financed by

means of applications connected to the Annual Working Programmes of 2014 and 2015,

beneficiaries from all Member States benefitted in some capacity (either as coordinators

or as participants), while for 2016 no beneficiaries from Estonia received funding. The

goal is to ensure 100% coverage through 2020 and it has been largely respected so far

(Member States coverage of participating organisations in 2016 was of 96%). However,

this indicator does not capture two essential elements connected to the equity dimensions,

namely: (a) the unfulfilled ‘demand’ of applications (i.e. ratio of beneficiaries /

applications for each Member State) and (b) the distribution of programme resources.

In general, there is no baseline available for all these indicators, as they relate directly to

the specific actions of the Justice Programme, and there are also no defined targets,

except for the geographical coverage.

Other data, collected during the interim evaluation, show the current state of play of the

Programme’s implementation and its key initiatives in the period 2014-2017.

As regards

the type of target groups,

the Justice Programme focuses mainly on research

institutions/universities, victims, young people and families (see Figure 3).

Figure 3: Text mining of target groups of the Justice Programme, by Specific objective

Source: Evaluation team’s elaboration based on the text mining exercise conducted on Grant Application Forms of the Justice

Programme

In relation to

the types of partnership,

the data collected in the interim evaluation

suggest that the Programme is successful in attracting more transnational partnerships

Indeed, on average the number of participating organisation per project is equal to six-

eight, a slight increase compared to the average of four-five observed in the predecessor

programmes. Within the Justice Programme network there are 292 organisations

connected out of 336 organisation reported in the database, i.e. 45 are isolated, meaning

that only one organisation out of five within the Justice Programme collaboration

Interim evaluation of the Justice Programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018,

published.

kom (2018) 0507 - Ingen titel

network is isolated. Public bodies (mainly judicial training organisations) and higher

education institutions (which also provide judicial training) are particularly well

represented within the networks of the Justice Programme, reflecting the judicial training

priority of the Programme.

The evaluation showed that one of the key vectors through which the Programme

supports the development of an area of justice in Europe is by catalysing the

development of transnational partnerships. Transnational partnerships are essential

contributors to the effectiveness of the Programme. According to the 78% of

respondents, this has increased their capacity to implement the projects, 70% reported an

increase in sustainability of results and 75% mentioned increased capacities.

Moreover, these partnerships appear to be quite robust, with many interlinkages amongst

participants

In addition, 50 organisations participating in the Justice Programme are also active in the

Rights, Equality and Citizenship Programme. It thus appears that the Programme has

facilitated the growth of durable European networks in the area of justice policy and that

structured networks of collaborations exist also in relation to other EU funding

programmes.

However, taking into account the involvement of partner organisations,

it appears that

the bulk of participants to the Programme come only from few Member States and,

therefore, a balanced geographical spread has still to be achieved

(as was the case

with the predecessor programmes). For example, about 25% of all the beneficiaries come

from either Italy or Belgium and about half of the partner organisations come from five

countries and the remaining 50% are organisations from the remaining 21 Member

States. Consequently,

this affected the allocation of funds.

For example, the largest

share of available resources for action grants is attributed to Italian organisations,

followed by organisations in the Netherlands, France, Belgium and Germany. Instead, the

vast majority of resources available for Annual Work Programmes allocated to main

actors via operating grants is distributed to organisations based in Belgium

With regard to the

kind of projects/activities funded,

the two specific objectives with

the highest number of awarded

grants

are the ones on "Judicial Training" and "Access to

Justice". Through them, the Programme mainly financed grants focused on training

activities, but also analytical activities and actions to support main actors. Instead, in

relation to the

implementation of procurements,

the Programme finances activities

focused mostly on the specific objective "Judicial Cooperation", followed by the specific

objective on "Access to Justice". In 2014, most funded activities concerned the

organisation of conferences and studies, whereas in 2015 activities focused more on

Information Technology interventions.

The

awarding rate of calls for proposal

(in 2014 and 2015

), of almost all specific

objectives, ranged between around 16% and 47% . With respect to the specific objectives

"Judicial Cooperation" and "Judicial Training", almost half of the applications submitted

were awarded; instead, in the field of drug policies, where very few calls were launched

in the period covered by the 2014-2016 Annual Work Programmes (5%), the demand for

funding outstripped the supply significantly.

See the results of the Social Network analysis of the networks of beneficiaries within the Programme in the Interim report, ibid, p.

56.

This is due to the fact that operating grants resources are made available to organisations representing European networks, which

tend to be headquartered in Belgium (thanks to proximity with other EU institutions ).

This information is not available for 2016 yet.

kom (2018) 0507 - Ingen titel

In 2016, a drop in the grant applications received, coinciding with the rollout of the

Participant Portal

, was registered. This drop in applications notwithstanding, in general,

the number of awarded projects remained stable across the four specific objectives and

even increased significantly in the case of judicial cooperation projects. The drop also led

to an increase of the awarding rate.

ETHOD

Short description of methodology

The evaluation process was supported by an external evaluator (Ernst & Young

Financial-Business Advisors)

, under the Commission's responsibility. The criteria used

for the evaluation include: relevance, effectiveness, efficiency, coherence, equity, scope

of simplification of the Programme and EU added value (see Table 7).

Table 7: Evaluation criteria and corresponding evaluation questions

Evaluation Criteria

Evaluation questions

EQ1. To what extent have the objectives been achieved considering the set indicators?

What progress has been achieved so far by the Justice Programme in meeting indicators

listed in Article 15 of the Justice Regulation

ii) Is the Justice Programme meeting its general objective at the Programme level?

EQ2. How adequate were the actions funded under the Programme to the objectives of the

Programme?

EQ3. What factors influenced the achievements observed?

Have any specific factors favoured or deterred the achievement of the SOs of the

Programme?

ii) Have any unintended effects influenced the effectiveness of the Programme?

EQ4: How have the eligibility criteria influenced the formation of partnerships and the scope of

proposals?

EQ5. Compared to the 2007-2013 six predecessor programmes, how did the Justice Programme

perform in terms of better policy targeting, and targeting of the right groups of beneficiaries,

Programme management and economies of scale?

How has the consolidation of predecessor programmes influenced the management of the

Justice Programme in terms reduction of administrative burden and the achievement of

economies of scale?

ii) Has the consolidation of predecessor programmes resulted in better policy targeting, in

terms of a reduction of dispersion/proliferation of calls with similar or overlapping

objectives?

iii) Have funded beneficiaries been able to enhance their capacity to support relevant target

groups?

EQ6: How effective have been the communication activities in informing the potential applicants

about upcoming calls and in increasing the visibility of funded projects?

EQ7. Will the results of the implemented actions be sustainable in the long-term?

Do funded actions achieve results that last beyond the duration of individual projects?

ii) Do partnerships among beneficiaries of the Justice and REC Programme endure beyond

participation in the individual calls?

EQ8: Which are the costs and benefits of the intervention?

EQ9: To what extent has the intervention been cost-effective?

Effectiveness

Efficiency

10.

EQ10: What factors influenced the efficiency with which the achievements observed were attained?

11.

EQ11: Is there any scope for using alternative implementing measures other than action grants

and operating grants, e.g. innovative financial instruments?

12.

EQ12: How relevant were the actions funded under the Justice and REC Programme to the needs

Relevance

of the different stakeholders?

Did the Programmes address stakeholders’ needs, as identified in the IL of the

Programmes?

ii) Has the Justice Programme provided relevant support to beneficiaries, in terms of

See ‘The Justice Programme

implementation in 2016 & 2017’, Directorate-General

for Justice and Consumers, 2017. The

Participant Portal is an electronic platform used to manage the applications received for calls published in the framework of the

Justice Programme. According to the evaluation results, the new Participant Portal is not adapted to typical Justice Programme

applicants (such as training organisations), as the Portal was initially designed with research institutions and voluminous grants in

mind.

Interim evaluation of the Justice Programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018,

published.

kom (2018) 0507 - Ingen titel

Evaluation Criteria

Evaluation questions

capacity to address the needs of target groups?

13.

EQ13: How well do the (original) objectives (still) correspond to the needs within the EU?

14.

EQ14: How relevant is the EU intervention to EU citizens?

15.

EQ15: How relevant were the groups targeted by the intervention?

16.

EQ16: To what extent is the Justice Programme coherent / complementary with other EU and/or

national policies and funding Programmes that have similar objectives to the Union bodies' work

(external coherence)?

Coherence,

Complementarity,

Synergies

17.

EQ17: What synergies exist within the Programme and with other EU Programmes?

18.

EQ18: To what extent the different actions and interventions lead to a coherent approach within

the Programme (internal coherence)?

19.

EQ19: To what extent is the intervention coherent with international obligations, including the

2030 Agenda for sustainable development?

20.

EQ20: What is the additional value resulting from the EU intervention(s), compared to what could

be achieved by Member States?

21.

EQ21: To what extent do the issues addressed by the intervention continue to require action at EU

level?

22.

EQ22: What would be the most likely consequences of limiting the level of the existing EU

EU added value

intervention or completely stopping/withdrawing from it?

How have policy areas relevant to the Justice Programme evolved since 2014, and how

does this affect the added value of the Programme?

ii) How has the Justice Programme influenced national policies in its respective fields?

23.

EQ23: How much EU added value resulted from national projects compared to transnational

projects?

ii)

Did any national projects overlap with, and address the same needs as, projects funded by

the Justice Programme?

Did projects funded under the Justice Programme foster the development of effective and

durable cross-border networks?

24.

EQ24: How fairly are the different activities distributed across the different target groups and EU

Member States?

Were resources of the Justice and Programme allocated with the differentiated needs of

target groups from different Member States in mind?

ii) Were resources allocated also in consideration of the different needs of beneficiaries

across Member States?

Equity

25.

EQ25: How has gender mainstreaming been promoted?

Was gender mainstreaming (intended as a cross-cutting theme) included in the

programming and implementation of the Justice Programme, including calls for

proposals?

How was the principle of gender mainstreaming applied in practice by beneficiaries?

ii)

26.

EQ26: How have the rights of the child been promoted?

27.

EQ27: How have the rights of people with disabilities been promoted?

28.

EQ28: How can the Programme management with focus in particular on the grant management,

Scope for

simplification

be further simplified to alleviate administrative burden of the Commission and of the applicants?

29.

EQ29. Is there any scope for further simplification through changes in the management mode of

the Programme?

The evaluation deployed a series of quantitative and qualitative tools, which were

devised to provide detailed responses to specific evaluation questions associated with

each evaluation criteria. The evaluation methodology relied on a

Mixed Methods

approach

which integrates and compares quantitative and qualitative approaches, data

collection, data analysis and interpretation in order to strengthen the reliability of data,

the validity of the findings and recommendations by triangulating multiple sources of

information.

The analysis covered the call for proposals and procurement activities financed in 2014,

2015 and 2016 Annual Work Programmes. The 2017 Work Programme on the other

hand has been analysed chiefly in terms of design and structure, not in terms of

execution.

One key method used to analyse the Programme was

desk research,

which focused on

available documents at programme, policy and project level. The desk research was

kom (2018) 0507 - Ingen titel

complemented by

fieldwork,

which was leveraged to collect relevant data and input from

stakeholders. This helped to fill knowledge gaps and validate information which was

retrieved during the course of the desk research. The following fieldwork activities were

performed:

web-based survey

was launched to gauge and compare the views of

relevant stakeholders (i.e. project applicants, project beneficiaries, Programme

Committee Members and additional relevant stakeholders);

2. Some interviews

were

conducted with project beneficiaries and Commission officials;

Finally,

a webinar

was held, which provided a chance to discuss preliminary findings with a number of

beneficiaries.

The collected data was then used to perform a series of quantitative and qualitative

analyses. On the quantitative side,

the analysis performed

helped provide perspective

both at the Programme level, assessing for instance how and where the Programme has

deployed the available financial resources, and at the project level, giving a granular view

of quantitative results indicators provided by individual projects. Also, a

Social Network

Analysis

was performed. This analysis helped achieve a “bird’s-eye” view of the

collaborations and transnational networks that were developed as a result of projects

financed by the 2014-2020 Justice Programme. Finally, concerning the qualitative

analysis,

automated text mining techniques reinforced with machine learning

algorithms

were deployed to help sift through the large amounts of data available on

individual projects. This was done in light of the need to give structure and find meaning

within a large and mostly unstructured dataset (namely project applications and final

reports), so as to create taxonomies of the information present in free text fields.

Crucially, text mining helped in the analysis to find commonalities and correlations in the

data that might not been found had solely manual techniques been deployed.

Robustness and limitations of findings

The evaluation’s findings and conclusions are deemed as credible and consistent

according to the methodology. There were, however,

some challenges

which by their

nature could not always be mitigated.

First of all, only 8 projects had both monitoring and survey data available which means

that no statistical inference has proved feasible and that the triangulation of quantitative

data at the project level has been limited. Nevertheless, the analysis of a series of output

indicators allowed to have a description

of the actions’ features as well as progress in

terms of target groups reached and activities carried out.

Moreover, another challenge was that in evaluations of EU legislation “it can be difficult

to identify a robust counterfactual situation (i.e. what the situation would be if EU laws

had not been adopted)”, which made purely quantitative analysis problematic

In view of these limitations, a counterfactual analysis, built on qualitative data, has been

considered to be the more suitable method. The analysis of direct questions on

behavioural change assumes that the respondents are able to reflect on their behaviour in

hypothetical, counterfactual situations and that they are telling the truth to the best of

their knowledge. However, as respondents have an interest in the continuation of public

support, they might be tempted to over-emphasize the merits thereof by answering

strategically

. To address the positive bias in answers by beneficiaries and the self-

European Commission, Better Regulation Toolbox, p. 344,

https://ec.europa.eu/info/sites/info/files/better-regulation-toolbox_0.pdf.

In interpreting the results, it should be considered that there may be two sources of bias to the answers: 1.

Strategic behaviour

(beneficiaries may tend to overestimate the effect of the treatment to further legitimise their receipt of funding, while applicants may

tend to say to be well off also in absence of funding

–

also to qualify for the next round); 2.

Estimation bias:

both beneficiaries and

applicants don’t know exactly what the counterfactual situation would be like, so answers are based on their educated

guess. This

kom (2018) 0507 - Ingen titel

selection bias in answers from applicants who have not received funding, a similar set of

questions was asked to applicants having had their proposal rejected. As the features of

all respondents (both successful and unsuccessful applicants) are known, this helped

triangulate findings gathered from beneficiaries.

In addition, even though the four case studies proved to be a useful tool to get a better

picture of the Programme’s activities, their representativeness of the Programme as a

whole was somewhat limited.

Despite these limitations, the evaluation carried out is supported by a robust evidence

base.

NALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS

The Justice Programme is performing generally well at midterm with regard to its

specific

objectives,

terms

effectiveness,

efficiency,

relevance,

coherence/complementarity/synergies and EU added value, but should do better in

particular in terms of equity

5.1 Effectiveness

The analysis of indicators has shown that significant progress has been achieved on

several fronts since some targets are close to being achieved and others have already

been achieved. Therefore,

the Justice Programme contributes positively to the

general objective of the Programme through the progress under its specific

objectives.

The Programme-specific indicators are adequate to monitor progress towards the

objectives of the Programme, but sometimes they are difficult to measure

due to a

lack of adequate tools (e.g. there is no satisfaction survey to help gauge the perception of

participants that took part in training activities) or to the influence of exogenous factors

(such as macroeconomic conditions, rate of development of new psychoactive

substances). Moreover, some indicators, concerning for instance the geographical

coverage of the Programme, do not capture certain underlying dynamics, such as the fact

that the Programme, so far, has been dominated by beneficiaries from a small number of

Member States.

Overall, the 2014-2020 Justice Programme is broadly perceived by applicants,

beneficiaries and stakeholders as an improvement compared to the three

predecessor programmes (namely

the Civil justice programme, the Criminal justice

programme and the Drug prevention and information programme), both in terms of better

policy targeting and better involvement of the right stakeholder groups. A better targeting

is observed in the sense that calls tend to tackle specific issues which are considered

priorities

. The merging of predecessor programmes has been particularly effective in

the field of European judicial training since this has reduced the overlaps and enhanced

the capacity of training providers by reducing the complications associated with having

to deliver training across multiple programmes with different rules.

Given the wide scope of the general objective of the Programme, a large number of

external factors influenced its effectiveness, such as the migration crisis (concerning, in

particular, the specific objective on “access to justice”) and, in general, the uneven pace

applies especially to applicants who can base their judgment only on the knowledge they acquired in the application phase with no

actual exposure to the implementation of the measures.

For more information and evidence, see section 6 "Responses to evaluation questions" in the Interim report, p. 94 et seq.

However, it should be noted that better targeting makes sense as long as there is enough demand. If the targeting is excessively

specific the risk is that of receiving a small number of high quality proposals, with the risk of not allocating the entire available

budget.

kom (2018) 0507 - Ingen titel

at which Member States transpose and enforce the

EU acquis.

This, however, did not

undermine its intervention logic and its

operational flexibility:

the annual work

programmes can be easily adapted to emerging needs in the area of justice (see more

under “Relevance”).

Finally, compared to the 2007-2013 period,

the sustainability of projects (with their

results and outputs

) beyond the end of their life-cycle has become an increasingly

important factor

to consider in the evaluation process. However, projects focused on

creating tools/outputs might have a more difficult sustainability landscape since, in these

cases, sustainability depends on whether the organisations that implemented the projects

can receive sufficient resources to maintain the tools once the EU funding is over.

5.2 Efficiency

The Justice Programme has been cost-effective relative to the actions it has funded

so far.

Indeed, the results of the evaluation have shown that the beneficiaries’ perception

of the efficiency of the Programme is positive. 60% of respondents to the relevant survey

question consider the benefits of the intervention greater than costs. This is true for the

Programme as a whole, but, in particular,

for the specific objective on judicial training

where more projects than expected have reported to have carried out training events and

activities and more people than expected have been involved in the activities, both within

the categories foreseen and beyond. The increase is quite significant also in terms of the

types of modules of training provided (live events/remote training/tools and guidelines).

Moreover, the average expenditure on judicial training is on average lower than expected

for the projects identified.

A key achievement of the Programme, compared to its predecessors, has been the

lower

burden on beneficiaries in terms of time and financial resources.

Nevertheless, there

is still room for improvement in terms of easing requirements and obligations to make

the Programme even more efficient in its implementation (see more under “Scope for

simplification”).

According to beneficiaries,

the current instruments (action grants, operating grants

and procurement activities) are adequate for the needs of the Programme

and,

therefore, using alternative and innovative funding instruments is not necessary.

However, the efficiency of their implementation should continue to be improved, notably

with regard to procurement actions, to ensure that the allocated funding is indeed used.

Indeed, since the forward planning for procurement actions occurs, usually, many months

before the action itself, this leads to a tendency to overestimate procurement needs (to

ensure that sufficient budget is allocated) and to a subsequent low implementation rate of

the allocated funding. The efficiency of such actions must be increased by ensuring a

greater degree of connection between the planning of procurement actions and their

implementation.

5.3 Relevance

According to all interviewed beneficiaries, the Programme is highly relevant to

address the needs of selected target groups.

Indeed, one of the key features of the

Programme has been its ability to adapt and modify the priorities in light of emerging

needs. For instance, following a string of terrorist attacks in Europe, two calls were

Sustainability of results and outputs, achieved through the implementation of activities that generate increased awareness,

knowledge and skills and through the development of tools that can be used also after the project ends, by beneficiaries and target

groups alike, (including transferred to other organisations).

kom (2018) 0507 - Ingen titel

issued to combat the radicalisation of inmates in prison. This can be achieved thanks to

the unique structure of the Justice Programme, as the Programme has been devised with

broad specific objectives, due to the fact that it has aggregated the three programmes

from the previous programming period. The general and specific objectives allowed the

Commission to adapt the Programme to evolving needs within the EU, especially in

terms of the judicial cooperation.

However, the specific objective concerning the

initiatives in the field of drug policies is sometimes difficult to reconcile with other

Programme priorities,

such as judicial cooperation and access to justice, as drug-related

prevention policies tend to be broader in scope.

In general, however, the

needs identified at the time of the Programme’s adoption are

still actual and relevant,

in particular the general objective of further developing a

European Area of Justice based on mutual recognition and mutual trust.

With respect to stakeholder needs, there is still scope to further increase the relevance of

the Programme through systematic analyses per main stakeholder type, as well as per

specific objective and Member States, so as to ensure that priorities related to each

specific objective for each call, as set out in the annual work programmes, are in line

with the current key needs of the stakeholders. Moreover, the Programme could include

additional target groups that are relevant to the achievement of its general objective.

These groups include staff in regulatory agencies, young judicial professionals and

students and judicial professionals in candidate countries and countries within the scope

of the European Neighbourhood policy. This is however currently not possible under the

legal basis of the Justice Programme.

5.4 Coherence, Complementarity, Synergies

The Programme presents a good level of coherence and complementarity with other

EU instruments, programmes and actions (for

instance, there is a very high coherence

with the European Agenda for Justice 2020

) and the risk of duplications or

incoherence is very low.

Especially in the case of judicial training, the merging of the

predecessor programmes has increased the coherence with other EU initiatives and

among different training objectives and reduced the possibility of duplications, both in

scope and in funding. A strong coherence and complementarity in the access to justice

specific objective exist with Connecting Europe Facility Telecommunications

Programme that also contributed to the development of e-Justice Portal and e-Codex.

Some exceptional overlaps in terms of objectives, target groups and actions still exist as a

natural consequence of the broad objectives and target groups covered by the

Programme, ranging from judicial cooperation in civil and criminal matters (thus

essentially the full spectrum of judicial activity) to drug policy and judicial training, as

well as since it targets all EU citizens.

In any case, there is still potential for strengthening the synergies with other EU funding

programmes and initiatives. This is the case, for instance, of the drugs policy area, where

the coordination with the Health for Growth Programme could be increased.

According to interviewed stakeholders,

the coherence with national policies and

initiatives with similar objectives and/or targeting the same areas is high. The

Programme fills the gaps left by national actions,

while existing national projects and

initiatives complement the Programme rather than being in contrast with it or a mere

duplication. Indeed, when both the Justice Programme and national initiatives have the

same (or similar) objectives and target groups, there are always differences between

The objectives and addressed areas of the European Agenda for Justice for 2020 are very much in line with the Justice Programme,

especially in the area of judicial cooperation, even if the Agenda prioritises areas such as terrorism and cybercrime that are less

present under the Justice Programme.

kom (2018) 0507 - Ingen titel

them, in terms of size of geographical scope (with enlarged scope in terms of target

groups), available resources and number of projects funded.

The Justice Programme is coherent also with international obligations,

such as the

UN 2030 Sustainable Development Agenda. Indeed, some general principles of the UN

agenda can be linked to objectives and areas targeted by the Justice Programme. This is

the case of the UN objective of ensuring peace and security, to be achieved through just

and inclusive societies that provide equal access to justice, effective rule of law, as well

as transparent and effective judicial institutions. All these elements can be found as

integrative parts of the Justice Programme, both in terms of access to justice and judicial

training. Moreover, the EU is party to The Hague Conference on Private International

Law and pursues its international action in relation to civil justice mainly through this

international organisation.

5.5 EU added value

All evidence collected confirms the high added value of the Programme

intervention,

which is considered

de facto

instrumental for a good and effective

achievement of objectives in the area of justice.

The results of the evaluation show that the survey respondents agree in saying that not

only the funded activities would have not been possible without the EU intervention, but

also confirmed that the same results would have not been achieved with Member States

intervention only, in terms of creation of partnerships, realisation of outputs, funding of

innovative actions, sustainability of results, and, especially, implementation of projects of

transnational size and scale. Indeed, the EU added value of the Justice Programme is

evident, above all, in the promotion of transnational projects with a European dimension

to tackle cross-border issues and in the provision of financial resources to fund activities

in key areas that are not necessarily high on the agenda of Member States, due to lack of

political will (according to stakeholders, this is particularly true for the specific objective

on access to justice

According to beneficiaries interviewed, the Programme can influence and align, at least

to a moderate extent, national actions in the targeted areas.

The Programme makes also beneficiaries able to work with partners in other Member

States, a possibility that increased their knowledge and understanding of the issues

covered by the Programme, widened their approach and range of skills, and provided

them with access to good practice examples and tools developed in other Member States.

As already mentioned above, partnerships are a key element for the success of the Justice

Programme and are present in the majority of projects.

The lack of national funding is one of the main reasons why the Programmes’ funded

activities would not have been possible through single Member States action. This is

particularly true for the initiatives in the field of drug policies, where national initiatives

are often missing a crucial transnational dimension that is pivotal in this field, and for

EU-level judicial training that, usually, is not provided at national level. Moreover, the

Justice Programme ensures the continued existence of European networks such as the

European Judicial Training Network.

The perceived importance of the Programme can be explained also by the high number of

projects awarded after just three years, compared to the three predecessor programmes

and the increase in the number of yearly applications received from potential

See Interim report, ibid.

The Civil Justice Programme, the Criminal Justice Programme and the Drug Prevention and Information Programme awarded 806

projects over seven years (2007-2013). The Justice Programme awarded 418 just in the three-year period 2014-2016.

kom (2018) 0507 - Ingen titel

beneficiaries indicates that the Programme remains a key source of funding across the

EU.

Moreover,

judicial training figures more prominently

in the intervention logic of the

Justice Programme, compared to the predecessor programmes. This is an important step

forward for the EU added value of the Programme since judicial training is central to

build mutual trust, improve cooperation between judicial authorities and practitioners in

the Member States and increase the coherence in the application of the EU legislation.

The results of the evaluation show that the

issues and areas addressed by the Justice

Programme would require further action and involvement at EU level.

Indeed, the

demand of EU action in these fields and the fact that the number of applications to the

Programme are still higher than the number of awarded grants demonstrate a clear

interest in the priorities addressed by the Programme.

5.6 Equity

The promotion of the cross-cutting priorities of gender equality, rights of child and

rights of people with disabilities is highly significant for the Justice Programme and

it’s enshrined in its legal basis.

In particular, the principles of gender and child rights

mainstreaming are evaluated during the evaluation process under the quality of

proposals.

The issue of gender mainstreaming is specifically mentioned in the “part B” of

the application form. However, the evaluation showed that, in terms of projects

developed, gender issues and equality are hardly a major theme in most of them.

Reference to women and gender equality are nonetheless found in five-six projects,

which, still not focusing directly on the matter, take into account such element in their

implementation.

The Justice Programme supports the rights of the child both in the programming phase

(design of the calls for proposals) and by means of the activities of projects selected for

funding. The respect of the rights of the child is therefore enhanced by the fact that

organisations applying for funding (and any of their partners), which will work directly

with children during the project implementation, must provide the Commission with a

description of their child protection policy. Moreover, calls for proposals funded under

the Justice Programme contain topics that are relevant to the subject. The results of the

evaluation show that the majority of respondents agree to further mainstream and

promote the rights of the child in the Programme.

Concerning the priority of rights of people with disabilities, it seems more in the

background compared to the previous two. Indeed, among the projects funded, only one

directly targeted at people with disabilities, under the specific objective access to

justice

. In terms of the need to further promote the rights of people with disability in the

Programme, stakeholders have quite heterogeneous opinions, but all categories tend to

think there is at least a “moderate” need to do it.

To understand how the Programme promotes equity through the funded activities,

participants' data broken down by sex, disability status or age, as required by the

Regulation, shall be collected. This is however not yet done.

Finally, as already mentioned, in the future, the Programme should try to distribute its

resources in a more balanced manner across the different target groups of beneficiaries

and Member States.

5.7 Scope for simplification

Project “Enhancing Procedural Rights of Persons with Intellectual

and/or Psychiatric Impairments in Criminal Proceedings:

Exploring the Need for Actions”.

kom (2018) 0507 - Ingen titel

The evaluation did not identify clear scope for further simplifications concerning

the management mode of the Programme.

The current direct management mode

appears adequate given the size of the Programme.

However, notwithstanding the improvements made with the current Programme, both

Commission officials and beneficiaries interviewed expressed critical opinions on the

implementation process, with specific regard to budget management and reporting duties.

Among the reasons provided, beneficiaries indicate that the financial reporting is too

detailed and inflexible when compared to the ones applied within other EU Programmes

(e.g. Horizon 2020 and Erasmus+).

In terms of administrative burden, nearly 70% of beneficiaries and applicants perceived

as still burdensome the proposal drafting, the provisions of necessary administrative and

financial information to participate in the calls and the monitoring and reporting

requirements. In general, beneficiaries agreed that the application and reporting

procedures were complex and long, but recognised that it was probably hard to simplify

them further, given the need to ensure accountability of European financial resources.

According to beneficiaries, the duration of projects financed by the Programme could be

extended, especially with regards to action grants. In particular, an extension to three

years was considered ideal, as it would be a timeframe that is aligned with the average

duration of scientific partnerships between higher education institutions. Moreover,

beneficiaries mentioned that also operating grants could be extended to cover at least two

years (instead of one) in order to reduce the administrative burden, in terms of

application and reporting. However, the annual operating grants allow the Commission to

have oversight of the activities of the funded organisations and allow for more flexibility

to adapt the work of beneficiaries to the emerging needs in their respective fields of

expertise.

Another key issue, according to small civil society organisations, is the difficulty to find

the co-financing amounts required; therefore, small non-governmental organisations

could be more supported by the Justice Programme

Even if, at first, the rollout of the new Participant Portal created some difficulties,

currently stakeholders consider the submission of proposals via the Participant Portal an

improvement, compared to the previous IT system, since the number of documents

required for the eligibility check has decreased and, therefore, also the related

administrative burden. In addition, once registered, these documents will not be required

any more, only in case of changes. However, since the Participant Portal has been

designed with research institutions and voluminous grants in mind, in its current form,

the Portal appears ill-adapted to typical Justice Programme applicants. There is,

therefore, scope for some improvements in this regard.

Finally, monitoring requirements and indicators, both at Programme and project level,

could be simplified and streamlined.

ONCLUSIONS AND WAY FORWARD

Without the Programme's support, there is broad agreement among beneficiaries, legal

professionals and Commission officials that a lot of activities, above all cross-border

activities (such as the judicial training and the judicial cooperation in both civil and

criminal matter) would not be provided by the Member States and the existence of

relevant European networks, such as the European Judicial Training Network (a key

player in the field of judicial training), would be at risk. Indeed, the areas covered by the

It is to be noticed that the average amount of grants has been increased to the current level following the ex-post evaluation of the

predecessor programmes that, as already said above, found that the dilution of funds amongst many small-scale projects had had

limited impact and EU dimension.

kom (2018) 0507 - Ingen titel

Justice Programme are often not prioritised at national level and this makes the

Programme unique since its positive impacts are not likely to be felt by other means or

by single Member States' actions. In this sense, the EU added value of the Justice

programme is undeniable and this is true for both its general objective and specific

objectives. The activities financed help to tackle cross-border issues by creating

partnerships, networking opportunities and occasions to build trust among judicial staff

of different Member States. This fosters the acquisition of durable legal knowledge and

competencies. Moreover, the broader scope of the current Justice Programme makes

possible to identify each year the adequate priorities and to adapt them to emerging needs

and threats.

The Programme's interim evaluation confirms its

relevance

for:



contributing to upholding EU values such as the rule of law, the independence of the

judiciary, the effectiveness of the justice system and its equal access to all;



supporting Member States to achieve more effective justice systems, where legal

practitioners know how and when to apply EU acquis, where cross-border cooperation

is thriving and the rights of victims and persons accused of crime are respected;



supporting to collect comparative information on the efficiency, quality and

independence of national justice systems and national rule of law safeguards.

Some new priorities emerged and improvement could be made in specific areas of the

Programme. In particular, the basin of potential recipients of the actions could be

expanded to increase its relevance and the achievement of its general objective.

Moreover, the geographic balance is still not good overall, since the Justice Programme

is dominated by beneficiaries from only a small number of Member States.

Regarding the

efficiency

of the Programme, the administrative burden on beneficiaries

has been reduced but there is still room for improvements to make the Programme even

more efficient in its implementation.

The

simplification

carried out, thanks to the merger of the three previous programmes,

has produced improvements in the quality of the evaluation and monitoring process.

However, the development of workable indicators, allowing for precise monitoring of the

quality and the impact of projects, should be better taken into consideration.

The Programme’s current structure works well by reducing the risks of duplications and

allowing the Programme to be sufficiently flexible to adapt to emerging needs, even

though coordination and information exchange between EU programmes and projects

could be strengthened in order to guarantee a more coherent and efficient allocation of

resources according to the most relevant priorities.

At EU level, the Justice Programme is

coherent and complementary

with other EU

instruments, programmes and actions. However, in the future, there could be scope for

further centralisation in terms of Judicial Training for legal practitioners

Lessons learned and way forward

for the next programming period

However, it should be noted that each Directorate-General is responsible for the enforcement of the legislation it is in charge of and

the centralisation of training on all EU topics would require additional financial resources.

For additional information on the conclusions, see section 7 "Conclusions" in the Interim report, p. 161 et seq.

For more information, see section 8 "Recommendations" in the Interim report, p. 167 et seq.

kom (2018) 0507 - Ingen titel

Effectiveness



The Commission will consider the possibility to separate the specific objective on drug policy

from the Justice Programme and move it to a dedicated fund addressing drug policies.

Efficiency



The Programme should increase its capacity to make an efficient use of resources allocated to

procurement actions, by re-assessing the way these actions are planned. For example, needs

assessments should be performed to gauge the amount of resources required to finance

procurement actions.

Relevance



The relevance of the Programme could be improved by engaging in a more systematic manner

with stakeholders concerning the definition of the annual work programmes. This could be

achieved, for instance, by means of open public consultations or yearly surveys. The Commission

will also consider the possibility to target additional groups that are currently outside the scope of

the Programme.

Coherence, Complementarity, Synergies



The Commission will consider the possibility to ensure that projects with similar objectives are

aware of each other and can build on their achievements. Commission could also help projects

funded across specific objectives to get in touch with each other.

EU added value



The Commission will consider the possibility to set up analytical activities and needs assessments

that identify systematically the critical conditions for the successful implementation of tools and

policies at national level before the launch of calls for projects.

Equity



The Commission will consider to instruct the applicants more concretely on how to include the

gender mainstreaming and the promotion of the rights of the child and of people with disabilities

in the different stages of a project. The Programme should also stress more the importance of

collecting data disaggregated by age and sex. The Commission will also consider to what extent

the participation of non-governmental organisations (for which co-financing might represent an

issue), should be supported. Actions should also be taken to foster a more equitable distribution of

resources among organisations in different Member States and to foster development of

Framework Partnership Agreements with a broader set of organisations.

Scope for simplification



The Commission will consider to provide more guidance to the applicants in the template Part B.

Each item should be better explained and complemented with examples. In addition, the Guide for

Evaluators could be published, to make the applicants aware of the evaluation methodology, and

the Participant Portal should be more adapted to cater to the needs of typical Justice Programme

applicants. Moreover, indicators contained in the Regulation should be separated conceptually

into output and outcome/impact indicators, indicate how the Programme contributes to the

fulfilment of said indicators and identify the main data sources for each indicator as well as their

availability. Finally, the monitoring system should envisage the possibility for the beneficiary to

update the values of the indicators.

kom (2018) 0507 - Ingen titel

NNEX

1: P

ROCEDURAL INFORMATION

EAD

DG, D

ECIDE

LANNING

/CWP R

EFERENCES

Lead Directorate-General:

Directorate-General for Justice and Consumers (DG JUST)

Decide planning reference:

RGANISATION AND TIMING

The evaluation was carried out between October 2017 and April 2018.

An external contractor, Ernst & Young Financial-Business Advisors, was commissioned

under the Framework Contract for evaluation, Impact Assessment and Related Services

(JUST/2016/JCOO/FW/JU04/0178) to carry out a interim evaluation of the Justice

programme (2014-2020). The final report was completed in April 2018. An Inter-service

Steering Group led by DG JUST was set up specifically for this evaluation. The

representatives of the following Commission services were members of the group:

Secretariat General, Directorate-General for Budget, Directorate-General for

International Cooperation and Development, Legal Service, Directorate-General for

Communications Networks, Content and Technology, Directorate-General for Migration

and Home Affairs, Directorate-General for Neighbourhood and Enlargement

Negotiations, Directorate-General for Health and Food Safety, and European Anti-Fraud

Office.

Chronology of the interim evaluation:

Date

January 2017

December 2017

April 2018

Meetings of Inter-service Steering Group

Kick-off meeting

of the mid-term evaluation Inter-Service

Steering group

Second meeting

of the mid-term evaluation Inter-Service

Steering group

to discuss the interim report

Third meeting

of the mid-term evaluation Inter-Service

Steering group

to discuss the draft final report and ask for

several improvements

XCEPTIONS TO THE BETTER REGULATION GUIDELINES

A public consultation was not carried out specifically under this evaluation, but under the

Impact Assessment for the post-2020 Multiannual Financial Framework.

N/A

VIDENCE

SOURCES AND QUALITY

The evaluation was based on evidence from different sources. The complete set of

documents that were consulted for this evaluation is listed below:

Legal basis of the Justice Programme:

ONSULTATION OF THE

RSB (

IF APPLICABLE

)

kom (2018) 0507 - Ingen titel



Arts. 81(1) and 82(1) TFUE;



Regulation (EU) No 1382/2013 of the European Parliament and of The Council of 17

December 2013 establishing a Justice Programme for the period 2014 to 2020.

Commission Implementing Decisions 2014, 2015, 2016:



Commission Implementing Decision C(2014)2556 concerning the adoption of the

work programme for 2014 and the financing for the implementation of the Justice

Programme;



Commission Implementing Decision C(2015)1997 concerning the adoption of the

work programme for 2015 and the financing for the implementation of the Justice

Programme;



Commission Implementing Decision C(2016)1677 concerning the adoption of the

work programme for 2016 and the financing for the implementation of the Justice

Programme.

EU documents:



Council conclusions on the European Judicial Network in civil and commercial

matters (15349/16) on 8 and 9 December 2016;



Conclusions of the European Council on 26-27 June 2014 in the area of freedom,

security and justice (including on judicial cooperation in civil matters)



The European Agenda for Justice for 2020, adopted in 2014;



The European Agenda for Security, adopted in 2015;



The 2030 Sustainable Development Agenda;



The “Juncker Priorities” for 2015-2019;



Conclusions of the European Council on 26-27 June 2014 in the area of freedom,

security and justice;



Multiannual European e-Justice Action Plan 2014-2018.

Justice Programme documentation:



Ex-post evaluation report of Civil Justice Programme (2007-2013)

http://eur-lex.europa.eu/legal-content/en/TXT/?uri=CELEX:52017DC0059



Ex-post evaluation report of Criminal Justice Programme (2007-2013)

http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52017DC0115



Ex-post evaluation report of Drugs Prevention and Information Programme (2007-

2013)



The mid-term evaluation report of Civil Justice Programme, 2011

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2011:0351:FIN:EN:PDF



The mid-term evaluation report of Criminal Justice Programme, 2011

http://ec.europa.eu/justice/funding/jpen/interim_evaluation_report_2011_en.pdf



The mid-term evaluation report of Drugs Prevention and Information Programme,

2011

http://eur-lex.europa.eu/legal

content/EN/TXT/PDF/?uri=CELEX:52011DC0246&qid=1467122450426&from=EN

kom (2018) 0507 - Ingen titel



Policy-specific reports published by DG Justice and Consumers, such as the annual

report on European judicial training

https://e-justice.europa.eu/fileDownload.do?id=5799faaa-b4ce-498c-aeb1-bba4b81d900e,

https://e-justice.europa.eu/fileDownload.do?id=ae7d84cc-2bd0-4eb3-9c31-94b753a20bdb



EU Justice Scoreboard COM (2017) 167 final

https://ec.europa.eu/info/strategy/justice-and-fundamental-rights/effective-justice/eu-

justice-scoreboard_en

kom (2018) 0507 - Ingen titel

NNEX

2: S

TAKEHOLDERS CONSULTATION

Consultations with the main stakeholders of the ‘Justice Programme’ were conducted

over several steps:



web-based survey

targeting the following stakeholder groups:

projects applicants (also unsuccessful applicants )

;

projects beneficiaries

;

EU26 Member State Programme Committee Members

;

Other relevant stakeholders involved directly by the Commission;

This consultation was launched by the external consultant via the EU Survey tool and

was open from 23 December 2017 until 6 March 2018. It received

132 responses



33 interviews

with 12 Commission officials and 21 selected projects beneficiaries and

representatives of target groups were performed between the 8

of January 2018 and

the 26

of March 2018. Interviews had a twofold objective: 1. Collecting additional

information (from the survey and the desk research) to feed the analysis alongside the

evaluation criteria, filling gaps (especially in terms of quantification of costs and

benefits) and going more in-depth with specific aspects, such as the identification of

best practices; 3. Gathering insight and input to draw conclusions and

recommendations on how to improve the design and functioning of the Programmes.

Interviews consisted of semi-structured phone interviews. Questions were customised

according to the different categories of stakeholders targeted, taking into account their

different contribution to the evaluation questions;



1 webinar,

organized in March 2018, to discuss preliminary findings with few

selected project beneficiaries. The webinar aimed at assessing at a higher level results

achieved by the Programme, testing and discussing preliminary findings and

conclusions, also identifying key success factors and areas for improvement in view of

the design of operational recommendations. It was organised using Zoom (an online

tool);



A broad

public consultation

undertook by the Commission

in the area of "values

and mobility"

was available online in 23 official EU languages for a mandatory

period between 10 January 2018 and 9 March 2018. The purpose of this consultation

was to collect the views of all interested parties on how to make the most of every

euro of the EU budget. Consultations have taken place in the context of evaluations of

existing EU financial programmes covering several policy areas, including the Justice

Programme. In total, the public consultation received

1839 replies from all over

Europe.

The respondents had experience with the following EU programmes: 1.

Europe for Citizens Programme and /or 2. Rights, Equality and Citizenship

Programme and /or 3. Creative Europe Programme and /or 4. Justice Programme. The

results of this public consultation have shown that, according to the majority of

respondents, "promote

European identity and common values",

as well as

From 16 Member States.

From 8 Member States.

For more information on the outcomes of the web survey, see Annex 2 "Analysis of surveys and quantitative analysis" in the

Interim report, ibid, p. 172 et seq.

kom (2018) 0507 - Ingen titel

"promote

rights and equality",

are important common policy challenges to be

addressed in each of the four programmes. "Support

active citizenship, democratic

participation in society and the rule of law"

and "promote

social inclusion and

fairness"

appear to be important challenges to be addressed in the concerned

programmes as well. Instead,

"support innovation", "foster European cultural

diversity and cultural heritage", "promote European identity and common

values"

are considered as policies that fully or fairly well address the challenges by

half or more of the respondents. Around 80% the respondents agree that these

programmes add value to a large extent or to a fairly good extent to what Member

States could achieve at national, regional and/or local levels. The main obstacles

identified by the respondents, that could prevent the current programmes/funds from

achieving their objectives, are very similar regardless of the programme concerned:

"lack of budget of the programmes to satisfy demand", "insufficient support

provided to small-scale stakeholders"

and

"lack of support to first-time

applicants"

are identified as the main three obstacles. Finally, the respondents agree

that

"the use of more simplified application forms", "facilitating structured

network and partnerships", "facilitating funding for actions cutting across the

sectors of action",

as well as

"better coordination between different

programmes/funds",

are the main steps to be taken to simplify and reduce the

administrative burdens for beneficiaries.

kom (2018) 0507 - Ingen titel

NNEX

3: M

ETHODS

The following methodological tools have been used:

1. Data collection

;



desk research

of relevant documents (at programme, policy and project level);



field research

relied on a

web-based survey

with relevant stakeholders, i.e.

(unsuccessful) project applicants, project beneficiaries, Programme Committee

members, Commission staff and additional relevant stakeholders,

33 interviews

with

21 selected project beneficiaries and representatives of target groups and 12 with

Commission officials and

one webinar

to test and discuss preliminary findings and

conclusions with few selected project beneficiaries.



four case studies

of projects connected to the specific objectives

;



analysis of the results of a

broad public consultation

undertaken by the Commission

between 10 January and 8 March 2018 in the framework of the proposal for the post

2020 Multiannual Financial Framework in the area of "European values and

mobility".

2. Data analysis:



quantitative data analysis

: in addition to monitoring and survey data,

descriptive

statistics

was used to analyse the effectiveness, efficiency and EU added value of the

Programme;

self-reported counterfactual analysis

have been used for the analysis of

its effectiveness and added value

;

exploratory data analysis and econometric

modelling

were used to analyse its effectiveness and EU added value

;

cost-benefit

analysis

to analyse its efficiency

and

Social Network Analysis

to analyse its

effectiveness and equity

;



qualitative data analysis

: a "text

mining" methodology

was used

Based on this methodology, output indicators (as monitoring data at project level) and

results indicators (based on Eurobarometer and other sources) were used. The analysis of

the performance at the project level has mainly relied on the answers to the survey.

For more information, see section 4.1.1 "Data collection tools and activities" in the Interim report, p. 49 et seq.

For more information, see the Annexe 4 of the Interim report, ibid.

For more information, see section 4.2.1 "Quantitative data analysis" in the Interim report, p. 51 et seq.

A counterfactual analysis is a tool that can be used to distil changes that are caused by a specific policy intervention from changes

that would have occurred even in absence of the given intervention (see Interim report, ibid).

Different tools of econometric analysis have been employed in order to examine the results of the self-reported counterfactual

questions, as well as to test the responses of different stakeholders to the main question of the survey linked to effectiveness/relevance

(see Interim report, ibid).

The Cost-Benefit

analysis is as an economic tool applied to public decision−making to quantify the advantages (in terms of

benefits) and disadvantages (in terms of costs) associated with a particular project or policy (see Interim report, ibid).

The Social Network analysis enabled to better understand the explicit and implicit links and interrelations among the different

beneficiaries and hence the value of the partnership approach that is a distinguishing feature of the (see Interim report, ibid).

For more information, see section 4.2.2 "Qualitative data analysis through text mining" in the Interim report, p. 58 et seq.

Text mining is an analytical methodology that has the aim of structuring large amounts of unstructured texts into categories or

clusters of information by broad themes (e.g. project objectives, weaknesses) (see Interim report, ibid).