kom (2018) 0003 - Ingen titel

Europaudvalget 2018
KOM (2018) 0003
Offentligt

EUROPEAN

COMMISSION

Brussels, 10.1.2018

SWD(2018) 3 final

COMMISSION STAFF WORKING DOCUMENT

MID-TERM EVALUATION

of the Regulation (EU) No 250/2014 of the European Parliament and of the Council of 26

February 2014 establishing a programme to promote activities in the field of the

protection of the financial interests of the European Union (Hercule III programme) and

repealing Decision No 804/2004/EC

Accompanying the document

Commission report to the European Parliament and the Council

{COM(2018) 3 final}

kom (2018) 0003 - Ingen titel

Table of contents

INTRODUCTION ................................................................................................................................ 3

BACKGROUND TO THE INTERVENTION ..................................................................................... 4

IMPLEMENTATION / STATE OF PLAY .......................................................................................... 9

METHOD ........................................................................................................................................... 10

ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS .......................................... 13

CONCLUSIONS ................................................................................................................................. 29

ANNEX 1: PROCEDURAL INFORMATION ............................................................................................ 32

ANNEX 2: SYNOPSIS REPORT OF STAKEHOLDER CONSULTATION IN THE CONTEXT

OF THE STUDY ................................................................................................................................ 34

ANNEX 3: METHODS AND ANALYTICAL MODELS .......................................................................... 42

kom (2018) 0003 - Ingen titel

NTRODUCTION

Purpose and scope

This report is a Commission Staff Working document (SWD) that accompanies the mid-

term evaluation of the Hercule III programme. The evaluation of the Commission was

supported by an external study

The mid-term evaluation is required by Article 13, paragraph 2(a) of the regulation that

establishes the Hercule III programme

(‘the Regulation') which provides that “The

Commission shall carry out a thorough evaluation of the programme and present to the

European Parliament and to the Council by 31 December 2017, an independent mid-term

evaluation report on the achievement of the objectives of all the actions, results and

impacts, the effectiveness and efficiency of the use of resources and its added value to

the Union, in view of a decision on the renewal, modification or suspension of the

actions; the mid-term evaluation report shall additionally address the scope for

simplification, internal and external coherence of the programme, the continued

relevance of all objectives of the programme, as well as the contribution of the actions to

the Union’s priorities of smart, sustainable and inclusive growth; it shall also take into

account evaluation results on the achievements of the objectives of the Hercule II

programme".

The scope of the mid-term evaluation, as set out in the evaluation roadmap

, is

represented by all interventions supported under the Hercule III programme up to June

2017 as well as the preparatory and implementing activities undertaken by the

stakeholders for these interventions. It also covers those applications submitted by

potential beneficiaries which remained unsuccessful. The evaluation assesses six

evaluation criteria – relevance of all the objectives of the programme; internal and

external coherence of the programme; effectiveness in attaining it objectives and

expected results; the efficiency of the use of resources; the programme's added value to

the EU; and sustainability i.e. the effects of Hercule III actions after their completion, as

well as the contribution of the actions to the Union’s priorities of smart, sustainable and

inclusive growth.

Mid-term Evaluation of the Hercule III programme, Final Report, CEPS, Economisti Associati, CASE,

wedoIT, 2017 (hereafter "external study").

https://ec.europa.eu/anti-fraud/sites/antifraud/files/herculeiii_midterm_evaluation_en.pdf

Regulation (EU) No 250/2014 of the European Parliament and of the Council of 26 February 2014

establishing a programme to promote activities in the field of the protection of the financial interests of

the European Union (Hercule III programme) and repealing Decision No 804/2004/EC, OJ L 84 of

20 March 2014.

The roadmap for the mid-term evaluation of Hercule III was open to stakeholders' feedback during the

life time of the evaluation on the following website: http://ec.europa.eu/smart-

regulation/roadmaps/docs/2017_olaf_002_midterm_evaluation_en.pdf

kom (2018) 0003 - Ingen titel

The time period covered by the evaluation is the first half of the seven year period that

Hercule III programme lasts for, starting on 1 January 2014 (when the Regulation entered

into force) until June 2017.

ACKGROUND TO THE INTERVENTION

2.1 Description of the intervention

The Hercule programme is an instrument specifically dedicated to supporting the

protection of the financial interests of the European Union by fighting irregularities,

fraud and corruption. It is administered by the European Anti-Fraud Office (OLAF)

within the Commission. The first Hercule programme

was established in 2004 and was

succeeded by Hercule II (2007-2013)

The current Hercule III programme (2014-2020) was adopted in 2014 on the basis of a

proposal tabled by the Commission in 2011.

The regular reporting by Member States

on irregularities and suspected fraud cases,

detected during the implementation of the budget on the basis of Article 325 TFEU,

shows that the financial impact throughout the years remains at levels that require the

Union to keep developing activities in order to strengthen the protection of the Union's

financial interests.

The general objective of the Hercule III programme is 'to

protect the financial interests of

the Union thus enhancing the competitiveness of the Union's economy and ensuring the

protection of taxpayers' money'

, whereas the programme's specific objective is 'to

prevent and combat fraud, corruption and any other illegal activities affecting the

financial interests of the Union'

. In addition, the programme lists five operational

objectives centred around:



strengthening cross-border and multi-disciplinary cooperation;

facilitating the exchange of information, experiences and best practices;

providing technical and operational support to national authorities;

fighting specifically organised fraud, especially cigarette smuggling and

counterfeiting;

Decision 804/2004/EC of the European Parliament an of the Council of 21 April 2004, establishing a

Community action programme to promote activities in the field of the protection of the Community's

financial interests (Hercule programme), OJ L 143 of 30 April 2004.

Decision No 878/2007/EC of the European Parliament and of the Council of 23 July 2007 amending

and extending Decision No 804/2004/EC establishing a Community action programme to promote

activities in the field of the protection of the Community's financial interests (Hercule II programme),

OJ L 193 of 25 July 2007.

The annual reports on the protection of the Union's financial interests Article 325 reports) are available

at:

https://ec.europa.eu/anti-fraud/reports_en.

Article 3 of Regulation (EU) No 250/2014.

Article 4 of Regulation (EU) No 250/2014.

kom (2018) 0003 - Ingen titel



promoting comparative law analysis

The Commission manages a budgetary envelope of EUR 104.9 million over the period

2014-2020 for the Hercule III programme, with an annual average of around EUR 15

million. The eligible beneficiaries

of the programme are national or regional

administrations of Member States and research bodies, as well as educational institutes

and non-profit-making entities that have been established and operate in a Member State

for at least a year. The programme provides financial support for three types of actions

1. "Technical Assistance"

(at least 70% of the programme's budget): support aimed at

providing specific knowledge, equipment and information technology tools to national

authorities as well as providing specific databases and IT tools facilitating data access

and analysis.

2. "Training" (maximum 25% of the programme's budget): support to the organisation of

targeted specialised training, risk analysis workshops, conferences and legal studies

focused on the protection of the Union's financial interests.

3. "Any other action" (not more than 5% of the budget).

The programme is implemented by means of grants following calls for proposals, public

procurement contracts following calls for tender and administrative arrangements

concluded with the Joint Research Centre (JRC) for the development of specific tools.

The programme also provides for the reimbursement of costs incurred by representatives

from a limited number of non-EU countries participating in training actions such as

conferences or seminars. The co-financing rate for grants does in principle not exceed

80% of eligible costs

2.2 Intervention logic

Annual Work Programmes (AWP) ensure that the general, specific and operational

objectives of the Hercule III programme are implemented in a consistent manner. They

outline the expected results, the methods of implementation and their total amount. With

regard to grants, the AWPs include the actions financed, the selection and award criteria

and the maximum co-financing rate.

AWPs are adopted by a Financing Decision of the Commission. The preparation of an

AWP involves a consultation of the main stakeholders within OLAF to ensure that new

requirements of OLAF's partners are taken into account. A draft version of the AWP is

Article 5 of Regulation (EU) No 250/2014.

Article 6 of Regulation (EU) No 250/2014.

Article 8 of Regulation (EU) No 250/2014.

The term ‘technical assistance’ in the context of the Hercule III programme is construed differently

from the definition of Article 186 of Commission Delegated Regulation 2012/1268, as amended, on

the Rules of Application of Regulation 966/2012 of 29 October 2012.

It may be increased to up to 90% in exceptional and duly justified cases.

kom (2018) 0003 - Ingen titel

subsequently sent to the members of the COCOLAF

and the AFCOS

, for consultation

and additional observations. On the basis of these consultations, modifications may be

made. The next step in the adoption of the AWP is an internal consultation within the

Commission in order to ensure that the proposed activities do not overlap with activities

funded under other Union programmes, such as Customs 2020 or the Internal Security

Fund (ISF) activities.

The Commission starts the implementation by preparing the call for proposals (e.g. for

technical assistance, conferences, legal training and studies) once the Financing Decision

is adopted and the appropriate financial resources are made available. Subsequently, the

submitted proposals are evaluated according to the selection and award criteria. The

successful proposals receive appropriate financial support for the relevant actions.

OLAF submits an annual Implementation Report to the Parliament and the Council with

the main results achieved and the relevant insights in terms of consistency and

complementarity with other EU programmes, as required by Article 13(1). This report is

annexed as a Commission Staff Working Document to the annual Article 325 report. The

Committee on Budgetary Control (CONT) of the European Parliament drafts its opinion

on the Article 325 report by the end of the year and the Commission takes account of

these observations in the elaboration of the next AWP.

Within OLAF, the programme is managed by a small sector and staff in the budget Unit

(8 FTE in total).

Advisory Committee for the Coordination of Fraud Prevention (COCOLAF) - aims to build

cooperation between EU countries and the European Commission to prevent and ensure the

prosecution of fraud.

Anti-Fraud Coordination Service (AFCOS) - a national service in each Member State designated to

facilitate effective cooperation and exchange of information with OLAF according to Article 3(4) of

Regulation 883/2013.

kom (2018) 0003 - Ingen titel

Figure 1: Intervention logic

Source: external study

kom (2018) 0003 - Ingen titel

2.3 Baseline and points of comparison

Hercule III was based on an Impact Assessment (IA)

that identified five key drivers

linked to nine problems that should be addressed under the programme. The problems

and the drivers are interconnected as can be seen in Figure 2.

Figure 2: Hercule III drivers and problems

The IA also found that the positive impact of the Hercule II programme was not fully

reached at the time, e.g. because some Member States were not able to co-finance

Hercule II projects, especially where procurement of technical equipment was concerned.

Notably on the eastern and southern fringes of the EU, the areas affected were also those

potential or actual weak points where there is the greatest common interest in

strengthening the control of the EU’s borders against smuggling. In response, an

important change introduced by the Hercule III programme was to increase the maximum

rate of funding in co-financed activities from 50% to 80%

Impact Assessment accompanying document to the Proposal for a Regulation of the European

Parliament and of the Council on the Hercule III programme to promote activities in the field of the

protection of the European Union’s financial interests (SEC(2011)1610 final and SEC(2011)1611 final

of 19 December 2011).

Source: SEC(2011)1610 final, Impact Assessment (page 10).

The maximum co-financed rate can be exceptionally raised from 80% to 90% for very specific actions

satisfying at least two of the following criteria: i) taking place at an external EU border; ii) taking

place at the most vulnerable locations; iii) reflecting the results of the Eurobarometer survey of

citizens’ attitudes to counterfeited, smuggled cigarettes and “cheap whites”; and iv) reflecting the

findings of the 2014 Article 325 report. as regards the number of cases of smuggled cigarettes reported

and the estimated traditional own resources involved.

kom (2018) 0003 - Ingen titel

The Commission’s evaluation of the Hercule II programme concluded that the

programme achieved its objectives

. However, the evaluation report also identified some

areas for improvement: more staff exchanges, access to databases, information exchange

in the procurement area, beneficiaries' reporting and creating synergies between

Commission services. Some of these have already been taken on board or addressed in

Hercule III. For example, cross-border cooperation, including staff exchanges are

encouraged in calls for proposals for Hercule III subsidies. Regarding synergies between

Commission services please see also page 19.

MPLEMENTATION

STATE OF

LAY

3.1 Administrative implementation

As detailed in section 2.1, Hercule III actions are divided along the following categories:



provision of

technical assistance

for the competent authorities of the Member

States;

organisation of

targeted specialised trainings,

risk analysis training workshops,

and conferences;

as well as an open-ended

other actions

cluster not falling into the previous two

categories

The actions are funded under grant agreements or procurement.

Over the period 2014 to 2016, the overall committed budget amounted to EUR 41

million, of which 75% was distributed via grant agreements. 391 applications were

submitted over that period and 125 grants were awarded, while 1700 participants took

annually part, on average, in events funded under Hercule III.

As explained in section 2.2, the programme is implemented on the basis of AWP that are

adopted by a Financing Decision of the Commission

. The preparation of an AWP for a

given year (n) starts in the autumn of the preceding year (n-1) by an internal consultation

within OLAF as well as consulting the main stakeholders. Following these consultations

the AWP is adopted. Once the Financing Decision is adopted and the appropriate

financial resources are made available, the Commission starts the implementation

activities as announced in the AWP. During the spring of the following year (n+1), the

Commission drafts the annual overview mentioned in Section 2.2

to be annexed to the

Article 325 report. The European Parliament adopts its resolution in spring of the year

COM(2015) 221 of 27 May 2015.

An example of "any other action" is the OLAF commissioned Eurobarometer survey. This survey

sought to uncover and explore the attitudes and opinions of Europeans in regards to cigarette black

market. It is entitled "Public perception of illicit tobacco trade", pooled the answers of 27,672

respondents from different social and demographic groups in 28 Member States. The findings of the

Eurobarometer should help Member States better target awareness-raising campaigns to fight cigarette

smuggling.

C(2014)3391 final; C(2015) 2234 final; C(2016) 868 final; C(2017) 1120 final.

Article 13(1) of Regulation 250/2014.

kom (2018) 0003 - Ingen titel

after (n+2). Therefore, a full cycle for the preparation of, implementation and reporting

on an AWP lasts almost 2.5 years.

3.2

Monitoring

The Regulation establishing Hercule III provides that the achievement of the specific

objective of the programme

, "to

prevent and combat fraud, corruption and any other

illegal activities affecting the financial interests of the Union"

has to be measured by four

key performance indicators (KPI)

. Beneficiaries of the actions have to report on the

results achieved. The modalities of this reporting differ according to the type of action

(technical assistance and training)

. In the case of contracts for the purchase of access to

databases, the control over the results of the actions takes a different form: the

Commission receives detailed information on the use of the databases by the different

users. The findings based on the beneficiaries' reports in both the technical assistance and

the training reports are included in the annual overviews on the implementation of the

programme and are used as input for the estimation of the KPI

ETHOD

4.1. Procedural aspects

In line with Article 13(2) of Regulation 250/2014 which mandates an independent mid-

term evaluation, OLAF used an outside contractor to perform a study in support of the

Commission evaluation (the ‘external study’)

. The external study was carried out

between February and June 2017. This study is a key contribution to this SWD, in

combination with other sources (they are referenced as appropriate in the relevant

sections). The main elements used for the external study were desk research and targeted

consultations of stakeholders through interviews and surveys. Further information on the

external study methodology can be found in Annex 3. The external study expresses the

views of the external contractor and, unless stated otherwise in this staff working

Article 4 of Regulation 250/2014.

(a) information on seizures carried out by joint actions and cross-border operations, (b) added value of

technical equipment funded under the programme, (c) information exchanges among Member States,

and (d) the number and type of (specialised) training.

For

technical assistance grants,

the results are reported in a "final technical report" once the request

for final payment of the grant is made as well as in a "final implementation report" that is submitted

one year after the closing date of a grant agreement. The reporting condition is set out in the AWP,

further specified in the calls for proposals and required by the grant agreement. Beneficiaries of

training grants

have to submit a "final technical report" with the main results of surveys held among

the participants to assess the relevance of the event and user satisfaction, as set out in the calls for

proposals and required by the grant agreement. As of 2017, grant beneficiaries have to organise a

"post-event survey" six months after the event took place to measure the mid-term impact of the event,

in particular in relation to the use of skills acquired during a training event and the sustainability of

networking activities. For procurement-based activities, conferences as well as training activities, the

beneficiaries need to organise user satisfaction surveys following these events.

Article 4 of Regulation 250/2014.

The procurement process for the external study started four weeks after the publication of the

evaluation roadmap. The contract was signed on 26 October 2016, implementing framework contract

No TAXUD/2015/CC/132.

kom (2018) 0003 - Ingen titel

document, the Commission neither endorses these findings nor can it be held responsible

for its content.

An Inter-Service Steering Group (ISSG) of Commission services

was set up to follow

the evaluation process. The ISSG reviewed the roadmap, the terms of reference for the

external contract, the inception and the end-of–fieldwork reports. The ISSG was also

closely involved in the preparation of this SWD.

4.2. Data collection

This SWD answers the questions identified in the evaluation roadmap. These questions

were further detailed in the terms of reference for the external contract and

operationalised by the evaluator using judgement criteria and indicators

. The main data

sources it relied upon to reply to the evaluation questions are:

Primary data;

data that were collected specifically for the purpose of the evaluation via:



interviews

with Commission staff; national institutions active in fraud prevention

and the protection of the EU's financial interests, as well as beneficiaries of actions

funded under the programme;

online surveys

with beneficiaries of actions funded during the first two years of the

programme; unsuccessful applicants; participants in events funded under the

programme, as well as users of services procured under the programme, in particular

database users.



Secondary data;

data gathered by examining:



public sources,

such as the text of the Regulation and supporting documents (the

Commission proposal as adopted in 2011, together with the Impact Assessment);

Hercule III AWPs; Hercule III Annual Implementation Reports; Article 325 Reports;

the final evaluation report and interim reports of the Hercule II programme, and legal

acts in relation to other EU programmes, such as Customs 2020, Fiscalis 2020 or the

Internal Security Fund;

documentary evidence

on submitted applications for grants (application forms) and

awarded contracts and grants (Final Technical Reports, Final Financial Reports and

Final Implementation Reports).



The evaluator consulted 574 stakeholders of the programme. In-depth interviews were

held with 49 civil servants (16 staff members of the Commission and 33 from national

administrations). In addition, 56 beneficiaries of the programme were interviewed (71%

of all beneficiaries), 67 unsuccessful applicants (25% of all rejected applicants), 321

participants in events (27% of participants to whom a questionnaire was sent) and 112

users of databases and services (31% of users to whom a questionnaire was sent).

The ISSG was composed of representatives of the Commission Directorates General for Budget, for

Taxation and Customs Union, for Migration and Home Affairs and for Justice and Consumers, and by

the Secretariat General and OLAF.

The evaluator gave an extensive description of its data collection activities in Annex D to its report.

kom (2018) 0003 - Ingen titel

Quantitative information on the implementation of the AWPs, such as the budget and the

number of grants/contracts, broken down by type of action and Member State were

collected from the overviews on the implementation of the AWPs.

4.3. Limitations

The life cycle of a programme like Hercule III is such that it takes some time, at the

beginning of the programme, to produce results. This generates some limitation to the

input available for a mid-term evaluation. The data collection and the interviews were

mostly carried out in 2017. By that time, only partial results of actions funded under the

new programme were available as these actions were funded in the first two years of the

programme (2014 and 2015)

This limitation was particularly felt for Technical Assistance (TA) actions. 13 out of 25

TA actions surveyed for the evaluation were still ongoing at the moment of gathering

data. In addition, for the 13 completed actions, most beneficiaries reported having only

recently received the equipment, thus putting them in no position to provide meaningful

feedback on the outcome and impact of the actions. In other words, at the moment of

performing the evaluation, the independent evaluator had access only to a limited number

of reports with information on results and the contribution the results made to the

achievement of the programme's general, specific and operational objectives.

The timing for the external study therefore resulted in some limitations to the tools that

could be used and led the evaluator to rely to a significant extent on the opinions

expressed by beneficiaries, applicants, participants, and Commission staff. However, the

interviewees arguably account for a representative sample of the programme's

stakeholders, as described above in section 4.2.

It cannot be excluded that there is a bias in the opinions expressed by the stakeholders, be

it positive for those whose application was successful and carried out some Hercule-

funded actions, or negative for those stakeholders whose application was rejected.

Additionally, the evaluation of some training activities under grants, such as conferences

or training sessions, could only be carried out

via

the grant beneficiary who organised the

event and who had access to the personal data of the participants required for the

transmission of the survey questionnaire(s). As not all these grant beneficiaries kept

records of the personal data, it was not possible to contact all the participants to training

and conferences under grants. However, this had only a minor impact on the overall

response rates since it related to events whose aggregated budget amounted to less than

10% of the financial support made available for training actions.

The first AWP was adopted in May 2014 (three months after the adoption of the programme in

February 2014) and the first calls for proposals were launched by mid-2014. The first grant agreements

were concluded at the beginning of 2015 and reports with the first results received by mid-2016. For

technical assistance actions, the first final technical reports only started becoming available by the

autumn of 2016. Final implementation reports that technical assistance beneficiaries have to submit

one year after the closing date of the action were not available yet. As regards 2016, the analysis was

limited to applications received in the context of calls for proposals, as no action co-financed by

Hercule III grants for 2016 calls had been completed by the time fieldwork activities were being

conducted.

kom (2018) 0003 - Ingen titel

The external study found that beneficiaries face difficulties providing the data to measure

the KPI a), b) and c) of Article 4 of the Regulation. The collection of data to measure

these indicators is reportedly the most burdensome aspect of the reporting phase. The

reports to feed the KPIs therefore provided only a relatively limited input into the mid-

term evaluation of the programme. The definition of the KPIs in the Regulation allows

only for results that, in practice, are too general and would require to be further broken

down to be fully used. This should however be addressed by the successor programme in

the next MFF period.

There is no indication, however, that the above limitations would invalidate the result of

the external study.

NALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS

5.1 Relevance

This section focuses on the relevance of the programme for the protection of the financial

interests of the Union. The questions are based on the Better Regulation Guidelines

examples and were adapted for the purposes of the evaluation by the ISSG.

EQ1: To what extent have the specific and operational objectives of the Hercule III

programme proven to be relevant for the general objective of the protection of the

financial interests of the EU?

On the basis of the external study

, it appears that the achievements of the specific and

operational objectives of the programme were relevant to achieving its general objective.

The latest Article 325 report

shows that the estimated financial impact of irregularities

and fraud is hardly declining over the last years (see Figure 3).

The Article 325 report

reports on all (fraudulent) irregularities detected by the Member States.

This decline is not

exclusively linked to the Hercule programmes.

The fluctuations in reported irregularities

and related financial impact can be also due to the level of reporting and (lack of) measures

taken to address the irregularities. Hercule III helps to boost the operational and

administrative capacity of Member States’ customs and other authorities. In that regard

substantial successes were reported in relation to smuggled and counterfeit cigarettes and

tobacco: seizures were made with the help of equipment and training funded under the

programme

External study, page 36.

COM(2017)383 final of 20 July 2017.

SWD(2017) 269 final, Commission Staff Working Document; Annual overview with information on

the results of the Hercule III programme in 2016.

kom (2018) 0003 - Ingen titel

Figure 3: Number of reported irregularities and amount: expenditures

number of irregularities

3000

2500

€ millions

2000

1500

1000

500

2008

2009

2010

2011

2012

2013

2014

2015

2016

Estimated financial impact - Non-fraudulent

Estimated financial impact - Fraudulent

Number of irregularities

20000

15000

10000

5000

Source: External study, contractor's elaboration on Article 325 reports.

The policy problem in connection with fraud and irregularities to the EU's financial

interests that was prevailing at the time of the preparation of the Regulation, as reflected

in the IA, is still prominent. Therefore, considering that the Regulation deviates from the

IA findings only on procedural matters

, the specific and operational objectives of the

programme could be expected to be still relevant for the general objective of the

protection of the financial interests of the EU.

This is confirmed by the stakeholders’ perception as presented in the bar chart below.

The same is corroborated by the conclusions in final technical reports of actions funded

under grant agreements, where all beneficiaries, who submitted the final technical

reports, reported on how the results contributed to the specific and general objectives (see

the answers to the next evaluation question).

For instance the possibility for the Commission to adopt delegated acts for the implementation of the

programme.

kom (2018) 0003 - Ingen titel

Figure 4: To what extent does each operational objective contribute to the PFI and,

more specifically, to preventing and combating fraud, corruption and other illegal

activities against EU financial interests, including cigarette smuggling and

counterfeiting? (Average evaluation; number of respondents)

1. Enhancing transnational and multidisciplinary

cooperation between Member State authorities,

the European Commission and OLAF to prevent

and investigate fraud

2. Facilitating the exchange of information,

experience and best practices related to the

protection of EU financial interests, including staff

exchange

4,19

3,97

4,58

4,49

3,98

4,33

4,46

4,22

4,67

3,98

3,88

4,45

3,98

3,45

3,50

3. Providing technical and operational support to

competent authorities of Member States in their

fight against fraud and other illegal activities

4. Reducing the development of an illegal

economy in key risk areas such as organised

fraud, with special emphasis on actions aimed to

fight cigarette smuggling and counterfeiting

5. Promoting comparative law analysis and

supporting academic analysis of strategic legal

issues with a view to developing a broad

consensus on how to better use legal resources in

the protection of EU financial interests

Beneficiaries

Unsuccessful Applicants

Institutions

Scale: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest

extent. Respondents who replied “Do not know/No opinion” or “Not Applicable” are not included.

Source: On-line survey and interviews with institutions, beneficiaries and unsuccessful applicants.

External study.

This bar chart also shows that the stakeholders perceived that the operational objective

"promoting

comparative law analysis and supporting academic analysis of strategic

legal issues"

was relevant, yet less than the other operational objectives. It should be

observed that this action is very specific to a small group of beneficiaries (research and

educational institutes and not for profit organisations), which explains why most other

stakeholders have seen this operational objective as less relevant.

EQ2: To what extent have the activities of the Hercule III programme proven to be

relevant for achieving its operational and specific objectives?

The activities of Hercule III have been relevant to the achievement of the programme’s

operational and specific objectives.

During the course of the external study, the stakeholders largely agreed to this as also

corroborated by findings in the final technical reports submitted by grant beneficiaries.

One important explanation for this result is that this parameter is assessed during the

evaluation of applications for grant. Applicants must demonstrate to the Commission

kom (2018) 0003 - Ingen titel

how the action is expected to contribute to the programme's general, specific and

operational objectives.

Nonetheless, two actions were largely perceived as less relevant to the achievement of

the general objective. These are: "purchase

of services to store and destroy seized

cigarettes and other counterfeit goods"

and "purchase

of services to carry out chemical

analysis of samples from tobacco and cigarette seizures."

The Commission received only

one application in 2014 and 2015 for grants for the destruction of seizures. Regarding the

latter objective, the Commission asked, under the programme, the Joint Research Centre

(JRC) to set up laboratory facilities for the chemical analysis of samples from seized

tobacco and cigarettes. The overall use of this facility has so far been low, possibly

because it is not yet sufficiently known amongst the operational staff responsible for

processing the seizures. To address this issue, the Commission is organising in 2017-

2018 training sessions in the Member States for operational staff, financed under

Hercule III.

For the same reasons identified under EQ1 above some stakeholders did not find of

particular relevance the actions geared towards building and disseminating scientific and

legal knowledge in the field of the protection of the financial interests of the EU, as this

action is specific to the actual small group of beneficiaries.

While emphasising the need to streamline the number of eligible actions to avoid

dispersion of funds on too many activities, some stakeholders

, suggested additional

activities that would contribute to the relevance of the programme. These include

trainings on big data analysis, activities in connection with detecting and analysing

digital evidence, research in the field of digital forensics, multidisciplinary research into

determinants of fraud affecting the EU financial interests, and risk analysis and profiling

of passengers to enhance detection of smuggled goods.

The external study also found that, in view of the parallel increase of irregularities on the

expenditure side and the decrease on the income side

, more emphasis could be put on

activities in connection with the expenditure side.

5.2 Coherence

This section examines to what extent the activities within the programme are coherent

(internal coherence) and to what extent the programme's activities are coherent with

activities undertaken by other EU programmes (external coherence) in order to detect any

overlaps and to identify synergies.

EQ3: What are the synergies between and within the different types of actions under

the programme and with other EU supported measures, programmes and actions, such

as Customs 2020 or Fiscalis 2020?

The internal coherence of the programme is ensured by (i) the allocation of the budget for

the period 2014-2020, (ii) the control performed by the Commission on possible overlaps

and repetitions on the basis of information provided in the applications, (iii) the

The external study did not explore whether these suggestions were shared by a larger group of

stakeholders. These suggestions therefore have only a limited value.

External study, Annex F, Evolution of key indicators on irregularities and fraud.

kom (2018) 0003 - Ingen titel

Regulation’s requirement that applications for equipment be accompanied by other

activities, e.g., trainings, and (iv) discussions and coordination between Member States

on the occasion of high level conferences financed by Hercule ensuring some actions

which are particularly relevant to the protection of the financial interests of the EU (PFI)

are performed irrespective of grant applications.

There is an internal coherence element in the programme between the different

categories of actions provided for in the Regulation (see Section 2.3). The

financial limits that are set for each category (at least 70% of the budget for

technical assistance actions, maximum 25% for training actions, both anti-fraud

and legal training actions, and up to 5% for other actions) are a guarantee that this

coherence will be respected, by balancing the actions where there is more demand

and other actions that are particularly relevant to the PFI (e.g., conferences on

specific PFI topics or forensics training). The financial implementation of the

AWPs shows that this indicative allocation has been complied with

Applicants for grants have to indicate in their applications whether or not similar

requests have been made in the past and during the evaluation of the grant

applications, allowing the Commission to closely monitor that actions do not

overlap: conferences proposed in training applications should not coincide or

address overlapping topics, whereas requests for funding for expensive technical

equipment are checked to ensure that similar equipment is not already in use by

the same services or in the same geographical area. The accepted applications for

grants do indeed comply with this obligation.

The Regulation provides in Article 10(3) that the purchase of equipment shall not

be the sole component of a grant agreement. This provision is implemented by

requesting applicants to ensure that technical assistance actions contain training

for staff to operate the equipment or to include activities enabling the exchange of

relevant information and best practices with other law enforcement services or

peers in neighbouring Member States.

High-level conferences arranged by OLAF ensure that some key topics for the

PFI (e.g. fight against cigarette smuggling, communication activities on fraud

prevention, fraud in structural funds, cooperation in anti-fraud activities) are

discussed and coordinated at the EU level by relevant national authorities. This

allows a coherent approach to the PFI in general and the programme in particular

across Member States, irrespective of the specific actions for which national

authorities decide to apply for.

ii.

iii.

iv.

The contractor carried out an analysis of the regulations establishing the programme and

Customs 2020, Fiscalis 2020, ISF Police and the Justice programme and the most recent

AWPs. Although each programme pursues very different general objectives, interactions

can be detected between specific and operational objectives. However as regards external

Over the period 2014-2016, on the basis of Financing Decision, around 70% of the budget was spent

on grant agreements and the remaining 30% on procured services, in particular digital forensic training

and conferences. At the end of the year, unspent Hercule III commitments are spent on grants, thus

increasing slightly overall percentage spent on grants.

kom (2018) 0003 - Ingen titel

coherence, the external study found that the overlap between the Hercule programme and

other mentioned programmes is very limited, essentially thanks to the Hercule

programme’s requirements and the Commission internal consultation mechanisms.

As explained above, the selection process allows detecting overlaps, on the basis that

applicants have to indicate whether or not they received grants for similar actions from

other EU programmes.

Some equipment procured under Hercule III can be used for purposes other than the

protection of the EU financial interests. A scanner installed at a border will, for instance,

not only detect smuggled cigarettes but also illicit products such as drugs or trafficked

persons as these might be transported in vehicles that are checked by the scanner

procured under Hercule III. This would not however weaken the coherence of the

programme, and could even be seen as complementing the specific objective of the ISF

programme to achieve "a uniform and high level of control of the external borders" and

thus creating synergies.

The Commission internal coordination process allows preventing overlaps at several

levels. As testified during the interviews with EU staff, few overlaps exist between EU

programmes, such as Customs/Fiscalis 2020, or the ISF (see the bar chart below,

Figure 5). The Commission inter-service consultation process to which draft AWPs are

submitted ensures complementarity of the programmes. Additionally, during the

preparation of the AWP of Hercule, Commission services responsible for the

management of Customs/Fiscalis 2020 and the ISF are closely involved in order to

prevent any overlaps from occurring and to ensure that existing synergies can be

strengthened or that new synergies can be identified. The AWP is also submitted to

different working groups of Member States' representatives to collect information and

feed-back to ensure that the proposed AWP addresses existing or new needs and to avoid

overlaps with other EU and national programmes. Lastly, at the stage of launching calls

for proposal, representatives of other Commission services

are closely involved in the

elaboration of the call documents as well as in the examination of applications in order to

ensure, as far as possible, that overlaps do not occur and that synergies with other EU

programmes are strengthened.

This is supported by the views of the participants to the online survey who mostly replied

that they, for the most part, were unaware of other programmes and that those who were,

indeed detected synergies, and only limited overlaps, between Hercule and other EU

programmes (Figure 5). Although more synergies between Commission services could be

achieved, this needs to be carefully balanced against inflating workload, creating extra

layers and generating delays. If fighting against corruption and VAT fraud would become

more central in Hercule III, the external study advises to maximise synergies with other

programmes managed by DG TAXUD and DG HOME, which touch upon these two

issues that can harm the EU's financial interests.

Directorates-Generals for Taxation and Customs Union (TAXUD), Justice and Consumers (JUST) and

Migration and Home Affairs (HOME).

kom (2018) 0003 - Ingen titel

Figure 5: Replies to the question: "To

what extent does the programme have

synergies and/or overlaps with other EU-funded programmes (you are familiar with)?"

[VALUE] (5)

[VALUE] (4)

[VALUE] (5)

[VALUE](4)

[VALUE](7)

[VALUE](3)

[VALUE](4)

[VALUE](3)

[VALUE](4)

Overlaps

Synergies

Customs

2020/Customs

2013 (DG

TAXUD)

Fiscalis 2020

/Fiscalis 2013

(DG TAXUD)

Internal Security

Justice

Horizon 2020/FP7

Fund – Police / Programme (DG

ISEC (DG HOME)

JUST)

Scale: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest

extent. Respondents who replied “Do not know/No opinion” or “Not Applicable” are not included.

Source: On-line survey and interviews with institutions, beneficiaries and unsuccessful applicants.

External study.

5.3 Effectiveness

This section focuses on the extent to which the programme has attained its objectives and

its expected results.

EQ4: To what extent have the overall intervention logic/strategy of the programme and

the actions contributed to the achievement of the objectives of the Hercule III

programme?

Over the period covered by the mid-term evaluation, the programme financed actions

which, because of their nature or objectives clearly contributed to achieving the

objectives of the programme (e.g. providing co-financing for a scanner purchase in order

to identify smuggled tobacco products). In terms of measuring this contribution, this is

indeed corroborated by the external study as far as anti-fraud and legal training actions

are concerned, as well as by the analysis of the final reports for grants. Because of the

data limitations highlighted in Section 4.3, the external study is inconclusive on this point

as far as technical assistance is concerned.

Under training actions, 65 events were organised for training, conferences or seminars.

The events covered a wide range of different topics, such as training for customs staff,

digital forensic training and training on fraud prevention or general issues in relation to

the protection of the financial interests of the Union. On the basis of the final reports and

in particular of the results of the user satisfaction survey

, the external study (see

As explained in section 4.3, in order to ensure data protection of the participants of the events and

users of databases, the satisfaction surveys were administered by the grant beneficiaries to the

Synergies

Overlaps

kom (2018) 0003 - Ingen titel

Figure 6 below) showed that the expected outcomes, such as networking possibilities or

the acquisition of new skills and competences have been largely achieved. However, only

a few actions involved staff exchanges between national administrations, although the

evaluation of the Hercule II programme recommended that more personal contacts

between administrations, e.g. through staff exchanges, would improve the impact of the

programme.

Figure 6: To what extent did the event funded by the programme contribute to the

achievement of the following outcomes? (Average evaluation; number of

respondents)

1. Improved investigative performance/abilities of law

enforcement officials

2. Increased awareness of fraud risk indicators and EU

anti-fraud policy

3. Enhanced knowledge of specialised methodologies,

tools and techniques to fight fraud

4. Exchange of information and sharing of best practices

between law enforcement agencies in relation to digital

forensic hardware and software to secure evidence…

5. Improved knowledge in the field of comparative law

with regard to the protection of EU financial interests

6. Improved cooperation between practitioners and

academics

7. Increased awareness of the judiciary and other

branches of the legal profession with regard to the

protection of EU financial interests

3.68 (320)

4.02 (259)

3.87 (310)

3.67 (306)

3.68 (253)

3.82 (279)

3.63 (281)

Participants

Scale: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Number of informed respondents in brackets. Differences in number of respondents across dimensions are due to

respondents selecting the “Do not know/No opinion” or “Not Applicable” answer.

Source: Online survey with participants in events. External study.

Technical assistance grants were awarded to applications covering four different topics:

investigation tools (cameras, monitoring devices for vehicles or mobile communications,

adapted vehicles, etc.), detection equipment (scanners, sniffer dogs), Automated Number

Plate Recognition Systems (ANPRS) and the destruction of seizures. Most of the budget

was spent on investigation tools (54%), whereas 28% was spent on detection tools.

ANPRS actions received 15% of the budget and the destruction of seizures (one grant

only) amounted to less than 2%. Taken account of the relatively recent signature of the

first grant agreements at the beginning of 2015, only few results were available at the

start of the evaluation

. Technical assistance actions also include procured access to

participants and by OLAF to the users of databases. The external contractor would receive the results

of the surveys directly as these were filled in.

The interviews with five of the first beneficiaries of grants awarded under the 2014 budget as well as

the survey generated some interesting, yet anecdotal, findings from the first months of operation of the

kom (2018) 0003 - Ingen titel

databases for use by law enforcement agencies during PFI related operations, laboratory

facilities for the examination of samples from tobacco seizures as well as the

development of specific IT-tools for data-mining and data analysis purposes. The latter

provides substantial input to Member States' customs for risk analyses to identify

shipments at risk. The external study found that the users of these services considered

them as effective, as shown in the bar chart below.

The examination of the available 13 Final Technical Reports on technical assistance

actions did not provide sufficient input for the outcome indicators, at the level of results.

As indicated earlier, this is due to more than half of sampled actions being still ongoing

during the evaluation period while many others were concluded very recently, thus

unable to provide more outcome indicators. Yet, they provided anecdotal examples of

achievement. For instance, the deployment of a wide range of investigation and detection

equipment enabled the seizure of more than 21 tonnes of cigarettes during more than 150

different operations. In addition, drugs, weapons and alcohol were seized as well. The

use of the ANPRS led to the discovery of 10 tonnes of drugs. While such equipment

contributes to successful operations, it must be borne in mind that it is used in

combination with other tools and does not allow attributing all the results only to the

programme.

During the mid-term evaluation, the Commission continued receiving final technical

reports and final implementation reports related to actions funded under Hercule II,

which are comparable to the reports required by Hercule III. These reports also contained

interesting information on the results achieved with specific technical equipment. The

aggregated prevented losses to the national and Union budget in these reports represent

an amount that is substantially higher than the Hercule II budget. It is likely to be even

higher because the beneficiaries are not required to report on a systematic basis. The

reporting period is mostly limited to a period covering 12-15 months and results beyond

that period are not reported to the Commission.

Regarding the evaluation of the perception of the stakeholders, the external study found

that the programmes’ actions and their results indeed contribute to the programme's

objectives. Overall, across all respondents’ groups, even the action that is seen as

contributing the least to the programmes’ objectives, i.e. the promotion of comparative

law analysis and the support to academic analysis of strategic legal issues, still scores

relatively well (between 3 and 4 on a scale from 1 to 5 - Figure 7). However, the external

study did also note that more should be done on the expenditure side of the EU budget,

which is an area currently receiving fewer grants.

purchased equipment. One of the interviewed beneficiaries purchased an ANPRS used to check more

than 16 million licence plates that generated more than 3 000 hits (i.e.: licence plates that were flagged

in databases for different types of warnings, including warnings related to PFI-investigations). Another

beneficiary purchased investigation tools that were used in more than 500 operations leading to 37

arrests and 22 convictions.

kom (2018) 0003 - Ingen titel

Figure 7: Replies to the question: "To

what extent did the action funded by the

programme contribute to the achievement of the following objectives?"

1. Protect the EU financial interests

4.39 (49)

3.92 (12)

2. Prevent and combat fraud, corruption and

other illegal activities against EU financial

interests, including cigarette smuggling and

counterfeiting

3. Enhancing transnational and multidisciplinary

cooperation between Member State authorities,

the European Commission and OLAF to prevent

and investigate fraud

4. Facilitating the exchange of information,

experience and best practices related to the

protection of EU financial interests, including staff

exchange

5. Providing technical and operational support to

competent authorities of Member States in their

fight against fraud and other illegal activities

6. Reducing the development of an illegal

economy in key risk areas such as organised

fraud, with special emphasis on actions aimed to

fight cigarette smuggling and counterfeiting

7. Promoting comparative law analysis and

supporting academic analysis of strategic legal

issues with a view to developing a broad

consensus on how to better use legal resources…

Beneficiaries

Institutions

4.50 (46)

3.91 (11)

4.07 (45)

4.00 (12)

4.22 (49)

3.67 (12)

4.10 (39)

3.83 (12)

3.91 (34)

3.27 (11)

3.93 (30)

3.11 (9)

Scale: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest

extent. Respondents who replied “Do not know/No opinion” or “Not Applicable” are not included.

Source: On-line survey and interviews with institutions, beneficiaries and unsuccessful applicants.

External study.

EQ5: To what extent have these objectives been achieved through the Hercule III

programme’s interventions and to what extent have other factors played a role?

Fraud, corruption and irregularities to the EU budget unfold in a multi-factor

environment where the EU and national regulatory frameworks, political will, judicial

capacity, and criminal behaviours inter-play, among others. The achievement of the

objectives of the programme is therefore heavily influenced by these factors. As an

illustration, the positive impact of the procurement of scanners to identify smuggled

tobacco products will be usually stronger in a country where the judiciary fight against

the criminal groups organising such smuggling is a priority and vice versa. However, it is

impossible to measure the interplay between these other factors and Hercule's

interventions in the achievement of the objectives of Hercule III.

The programme's stakeholders have identified a number of specific factors as influencing

the achievement of the programme's objectives. The most important factor hindering the

achievement of the objectives, with no significant differences in ranking among all

stakeholder categories, is the ability of fraudsters to “adapt quickly to new circumstances

kom (2018) 0003 - Ingen titel

at the EU level, operate cross-border and exploit the weakest points in the external

borders”. The other identified factors are:

“the large number and varied nature of competent national and regional

authorities” leading to an uneven PFI;

“differences in incentives and capacities between MS”;

“divergent application of criminal law and penal sanctions” across EU MS;

“pressure on public finance” requiring more cost-efficient solutions.

A few stakeholders, not statistically representative, have identified some other obstacles,

including: weak political willingness to protect the EU financial interests; rising

scepticism vis-à-vis EU institutions; poor knowledge about fraud affecting the EU

financial interests; limited financial envelope of the programme and limited financial

resources available at MS level to protect the EU financial interests; and lack of skilled

personnel and modern technical tools to combat and prevent fraud. Consideration might

be given to the impact of these factors during the final evaluation of the Hercule III

programme.

5.4 Efficiency

This section examines whether the results of the actions have been achieved at a

reasonable cost.

EQ6: To what extent have the desired effects been achieved at reasonable costs?

Cost-effectiveness is inherent to the implementation of

technical assistance

actions: they

mostly consist of the acquisition of equipment that has to be purchased by public

procurement procedures where the "best value for money" principle is used to select

equipment. Nevertheless, due to the data limitation (see section 4.3), further

quantification was not possible. The external study found that the costs per participant for

trainings

are broadly aligned with those recorded by other EU-funded programmes.

The average beneficiary of the programme spent approximately 18 person-days to

complete the application phase, 4.4 person-days for the grant-awarding phase and 13

person-days for the reporting phase. The external study found that this is in line with

other EU funded programmes. This translates into the average applicant spending

approximately EUR 4 500 for transaction costs, i.e.: all costs incurred for the acquisition,

monitoring and reporting (for more detail per type of action, see Figure 8 below).

The external study also established the average cost per participant to anti-fraud training

at EUR 467 and that to legal training at EUR 694. This cost is broadly in line with other

EU-funded programmes. No such analysis could be done for technical assistance because

of limited availability of data.

The external study demonstrated that regulatory costs linked to the application phase

yield “value for money”. The beneficiaries of anti-fraud training activities were expected

to get about EUR 12 for each euro spent in preparing a proposal (the expected return on

investment). For legal training proposals this amount was around EUR 7 for each euro

spent. While beneficiaries of technical assistance grants were expected to get almost

EUR 67, as the average grant requested was much greater (Figure 8).

kom (2018) 0003 - Ingen titel

Figure 8: Costs/benefits analysis

Administrations / Research and educational institutes and non-profit-

making entities

Regulatory costs

Benefits / Return on investment

for

each euro invested in the application

process if grant is awarded

EUR 12.4

Anti-fraud training

activities

EUR 2 195 - Application phase

EUR 467 - Grant award and

contracting phase

EUR 1 830 - Reporting phase

EUR 4 492 - Total

Legal training

activities

EUR 3 235 - Application phase EUR 7.3

EUR 886 - Grant award and

contracting phase

EUR 2 900 - Reporting phase

EUR 7 021- Total

Technical assistance

EUR 1 839 - Application phase

EUR 486 - Grant award and

contracting phase

EUR 1 312 - Reporting phase

EUR 3 637 - Total

EUR 66.8

Source: External study

EQ7: Could the same effects have been achieved with lower costs if procedures had

been simpler, involving less administrative burden and/or efficient implementation

mechanisms had been applied?

On the basis of the external study’s findings, the application process does not appear to

be too cumbersome for the majority of stakeholders (approximately 70%). 30% of

applicants expressed reservations on the application process and 33% of beneficiaries

found the reporting process to be cumbersome. Among those, some made suggestions for

improvement which, according to the external study, would generate some substantial

savings (see footnotes 56, 57 and 58). If their suggestions were accepted, the external

study estimates that savings of about EUR 2 000 per action would materialise (see

Figure 9 below). Some regulatory obligations are, however, inherent to EU procurement

Regulatory costs incurred when dealing with administrative activities are linked to three specific

phases: i) drafting grant proposals; ii) completing the grant award and contracting phase; and iii)

complying with reporting obligations.

The expected value of a proposal is calculated as the grant requested multiplied by the likelihood of

getting the grant (calculated by dividing the number of successful applications by the total number of

applications). This value is then divided by the regulatory costs incurred in the application phase, thus

measuring the expected “return on investment” at the moment of applying for a grant.

kom (2018) 0003 - Ingen titel

and cannot be simplified given the current legal framework. Therefore, only some of the

suggestions of the applicants could be implemented. Almost all respondents who

considered the grant awarding and contracting phase cumbersome suggested what could

be done for making the application phases less cumbersome. However, each suggestion

was provided by a limited number of consulted stakeholders.

Figure 9: Potential savings

Administrations / Research and educational institutes and

non-profit-making entities

Potential Savings

Application

Grant award and contracting

Reporting

Total

Source: External study

EUR 1 100

EUR 350

EUR 700

EUR 2 150

In 2017, the Commission services have started using an electronic management system

for the submission, processing and management of grant applications under the

programme (‘e-grant system’). The objective of this system is to streamline the process

and alleviate the burden of the applicants. The improvements brought about by the e-

grant system will address a large number of these suggestions, for instance by digitising

the whole process and introducing the use of digital signatures. The e-grant system is

expected to noticeably alleviate the administrative burden and related time and resources

spent on the applications in particular.

Avoiding redundancies in the information requested; improving the readability of the application

forms; providing more guidance on the application process, e.g. via webinars or tutorials; reducing the

amount and detail of information requested, e.g. by postponing some information obligations to the

grant award phase; translating calls for proposals and guidance documents into all EU languages;

digitising the entire application, including the use of digital signatures.

Digitising the process; providing assistance in all EU languages and reporting in all EU languages;

shortening the time-gap between the submission of the application and the signature of the contract;

allowing for subcontracting an organisation rather than only physical persons.

Digitising the reporting process, e.g. by creating a centralised platform to upload relevant data;

consideration of the specific needs of research and educational institutes (which, for instance, may not

be able to prepare timesheets); allowing the inclusion of person-days spent on preparing reporting

documents in the budget of the action; and devoting a specific time window to reporting activities,

after the completion of the action.

kom (2018) 0003 - Ingen titel

5.5 EU added value

This section covers the programme's added value at the level of the European Union,

compared with no intervention at EU level in this field.

EQ8: Has the programme allowed delivering results that could not, or to a lesser

extent, be achieved by interventions undertaken at national or regional level?

The external study found that the stakeholders almost unanimously recognised the added

value of the programme (see in particular Figure 10), in all its components, and its

capacity to generate unique results and benefits that could not be generated at national or

regional levels.

The primary explanation is the lack of budget to fund similar actions at national or

regional levels. Absent the programme, the actions provided for by Hercule would

probably not exist. Additionally, from the point of view of the Commission and Member

States’ services, Hercule III funded interventions generate better results and more

benefits than comparable national/regional interventions would

Figure 10: Would the results and benefits of actions funded by the programme also

be generated in the absence of the programme? (% of respondents)

Definitely would not Probably would not

Probably would

Do not know/No

opinion

Definitely would

beneficiaries

institutions

Source:

Online

survey and interviews with institutions and beneficiaries. External study.

Importantly also, the programme contributes to a better protection of EU external borders

and to strengthening long-lasting cross-border cooperation, cross-border exchange of

information and cross-border exchange of best practices (see Figure 11 below). This

confirms the added-value of a programme at the level of the EU. Some beneficiaries,

although not a statistically significant number of them, suggested that there is more

potential in centralised (common) purchases of technical equipment as this would reduce

costs for national authorities and would ensure cross-border use of that equipment.

External study, page 90.

kom (2018) 0003 - Ingen titel

Figure 11: To what extent did the actions funded by the programme lead to cross-

border cooperation, exchange of information and/or best practices and/or common

use of equipment? (Average evaluation; number of respondents)

4.11 (38)

4.00 (13)

4.22 (41)

3.85 (13)

4.17 (42

)

4.00 (13)

3.00 (25)

3.44 (9)

institutions

1. Cross-border cooperation

2. Cross-border exchange of information

3. Cross-border exchange of best practices

4. Common use of equipment across EU

countries

beneficiaries

Scale: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest

extent.

Source: Online survey and interviews with institutions and beneficiaries. External study.

The majority of the funded actions are expected to generate benefits in terms of exchange

of information and good practices between Member States as well as cross-border

cooperation, although to a lesser extent in the case of technical assistance actions (see

Figure 12).

Figure 12: Percentage of actions enabling cross-border cooperation

Anti-fraud

training

Exchange of information and best

practices

Cooperation with other MS

100%

Legal

training

88%

Technical

assistance

45%

Total

71%

76%

94%

24%

54%

Note: For technical assistance, results are based on information sourced from application forms. For legal

training, 37.5% of the results are based on information sourced from application forms. For anti-fraud

training, 4% of the results are based on information sourced from application forms.

Source: External evaluator’s elaboration on application forms and available reporting documents.

Overall, these results are coherent with the IA which had identified that there were

inadequate national solutions to a number of problems within the scope of the

programme (e.g. insufficient information sharing between authorities or inadequate

sharing of best practices).

EQ9: Does the intervention at the EU level provide added value in terms of the

efficient use of financial resources as compared to a possible intervention at national

level?

The external study found that the financial resources are more efficiently used through

the programme than national interventions would, for instance by generating cost savings

or securing better commercial terms (e.g. lower searching costs to find the most adequate

equipment and lower purchasing costs thanks to volume discount). Furthermore, in their

feedback 67% of beneficiaries and 85% of relevant stakeholders from the Commission

and national services mentioned that Hercule III enables more efficient use of resources,

kom (2018) 0003 - Ingen titel

as well as improved planning, monitoring and quality standards due to transparency and

reporting requirements.

5.6 Sustainability

This section examines the likelihood that the programme's actions will continue

generating positive effects after their completion.

EQ10: To what extent are the (positive) effects of the intervention likely to last after

the intervention has ended?

Overall, it appears that the effects of the programme last after the intervention has ended.

The external study found that the beneficiaries of the programme are of the view that the

Hercule III funded actions will, or are likely to, continue producing benefits after their

completion. Equipment purchased under technical assistance are used for several years

after their acquisition, while training actions contribute to acquiring skills and building

networks likely to be used over a longer period of time. The beneficiaries of training

actions are asked whether they will keep on performing the action funded by the

programme after the EU’s financial support has come to an end for the purposes of the

Final Technical Report. The external study analysed 30 out of 41 of available reporting

documents for training actions funded by grants in 2014 and 2016. The analysis showed

that more than 50% of the reviewed training actions will not continue in the absence of

EU funding. However, this seems to address the sustainability of the action itself, rather

than that of its effects.

The experience with Hercule II also provides some interesting input into the

sustainability question that is also relevant to the Hercule III programme. The

sustainability of the results under grants awarded under the Hercule II programme was

promoted by covering maintenance as well as training costs to enable staff to operate the

equipment acquired under technical assistance grants. Beneficiaries were asked to submit

a final implementation report one year after the closing date of the project to demonstrate

its effectiveness and the sustainability of the outcomes. As regards training, seminars and

conferences, the participants were generally confident that the acquired skills and

competences can be put into practice over time: sustainability was achieved in this

respect too. This was particularly true for participants in digital forensic training.

5.7 Contribution to smart, sustainable and inclusive growth

The external study found that the contribution of Hercule III to the five headline targets

identified by the Europe 2020 strategy for smart, sustainable and inclusive growth is not

direct. However, protection of EU financial interests allows for: i) securing a larger

amount of financial resources to pursue EU initiatives that aim to achieve the headline

targets (revenues); and ii) ensuring that such resources are spent to foster smart,

sustainable and inclusive growth rather than illegally diverted to other ends

(expenditures). In addition, the increased protection of EU external borders stemming

from Hercule III actions contributes to creating a level playing field between EU

businesses and their competitors based in third countries, which is considered another

key aspect to ensuring the timely achievement of Europe 2020’s targets.

kom (2018) 0003 - Ingen titel

ONCLUSIONS

On the basis of the answers to the evaluation questions, together with the Commission's

findings in relation to the results of the Hercule III programme, the following conclusions

can be drawn with regard to the evaluation criteria.

Relevance

Against the background that the key drivers and problems identified in the impact

assessment (see Figure 2) for the Commission's proposal for the Hercule III programme

are still prevalent today, in particular the overarching fraud phenomenon, the external

study demonstrated that the specific and operational objectives continue to be relevant

for the protection of the EU’s financial interests.

A few specific actions listed under Article 8 of the Regulation (e.g. “purchase of services

to carry out chemical analysis of samples from tobacco and cigarette seizures” and

“purchase of services to store and destroy seized cigarettes and other counterfeit goods”)

were found to be less relevant to fulfilling their mission in relation to the PFI. This

finding can largely be explained by the fact that these actions target a narrow group of

possible participants and, consequently, the bulk of the participants do not see their

relevance. To address some of these issues, the Commission is organising in 2017-2018

training sessions in the Member States for operational staff, financed under Hercule III,

to increase awareness of the laboratory facilities for the chemical analysis of samples

from seized tobacco and cigarettes.

Some beneficiaries suggested a few ideas of possible additional operational objectives

which would improve the relevance of the programme, in particular in the areas of cross-

border cooperation between Member States, cooperation with non EU partners, as well as

new technological developments. To what extent these ideas are shared by the other

stakeholders was not tested within the external study. Their value is therefore limited.

However, they are coherent with the Commission’s experience in implementing the

programme and the direction towards which the Commission has started steering the

programme over the last few years.

Coherence

The evaluation found that overall both the internal and external coherence of the

programme were ensured. The programme’s internal coherence between its different

components is guaranteed by the financial limits which the Regulation sets for each of

them, by balancing the actions where there is more demand and other actions that are

particularly relevant to the PFI (e.g., conferences on specific PFI topics or forensics

training). Internal coherence is also guaranteed by the possibility to detect overlaps

between applications at the selection stage, as well as the requirement that the purchase

of equipment be complemented by technical assistance activities (eg, trainings to use the

equipment). External coherence of the programme is ensured by the Commission internal

consultation mechanisms that allow preventing duplications and overlaps with other EU

programmes. Such consultation takes place essentially not only at the stage of elaboration

of the AWP, but also during the selection of applications for grants. These conclusions

are supported by the overall perception of coherence by stakeholders, as evidenced by the

external study. Although more synergies with other Commission programmes might be

possible, this needs to be carefully balanced against generating delays and inflating

workload.

kom (2018) 0003 - Ingen titel

Effectiveness

The actions financed under Hercule III contributed to the achievement of the

programme’s general, specific and operational objectives. The programme financed

actions which, because of their features, clearly contributed to achieving the objectives of

the programme, for instance when funding the procurement of investigation and

detection tools (eg, devices and animals to carry out inspections, monitoring devices,

scanners) or supporting trainings in, for example, the digital forensic area.

A large number of training events were funded and these events covered a wide range of

topics relevant to the PFI. The external study concludes, on the basis of the final reports

of the actions and beneficiaries’ surveys, that those actions generated results that were

largely in line with the expected outputs. Nonetheless, only few actions involved staff

exchanges between national administrations and few actions involved international

participation. The external study did also note that more should be done on the

expenditure side of the EU budget, which is an area that has received relatively fewer

grants.

With regards to technical assistance, the timing for the evaluation, i.e. early in the life of

the programme, made it challenging to further consolidate the above findings about the

effectiveness of the programme.

Among the many external factors interfering with the achievement of the programme’s

objectives, the ability of fraudsters to “adapt

quickly to new circumstances at the EU

level, operate cross-border and exploit the weakest points in the external borders”

was

perceived as the most important. Such external factors tend to harm the PFI and require

continued efforts by national authorities to combat and prevent fraud.

Efficiency

The external study shows that the costs generated by the programmes’ training actions

are in line with other EU programmes, be it in terms of the quantity of person-days

necessary at the different stages (preparation of an application, signing of the grant

agreements, and reporting), of the transaction costs for the acquisition and

implementation of actions under grant agreements, or the average costs per participant in

training actions.

Cost-effectiveness is inherent to the implementation of technical assistance actions as

they mostly consist of the acquisition of equipment that has to be purchased by public

procurement procedures where the "best value for money" principle is used to select

equipment. However, the results obtained following the deployment of the equipment

during PFI related operations is difficult to assess for lack of data.

On the basis of the external study’s findings, the application and reporting stages do not

appear to be too cumbersome for a large majority of stakeholders. Some of the critical

applicants made a number of suggestions that are currently in the process of being

addressed. However, only part of these suggestions could be addressed, as some

regulatory obligations are inherent to EU financial and procurement rules and cannot be

simplified given the current legal framework. The Commission has started using a

system for the electronic submission of applications for grants and for the management of

grant agreements. The improvements brought about by the electronic management

system will address a large number of the applicants’ suggestions, for instance by

digitising the whole process and introducing the use of digital signatures. The electronic

kom (2018) 0003 - Ingen titel

system is expected to noticeably alleviate the administrative burden and related time and

resources spent on the applications in particular.

EU added value

The external study’s results demonstrated that the benefits from the programme would

not materialise without the support of Hercule III or equivalent EU funding. The actions

allowed for a better protection of the Union's external borders. In addition, the

programme is an enabler for cross-border cooperation, in particular cross-border

exchanges of information and best practices. The training actions facilitated networking

opportunities between the participants to the events thus contributing to strengthening

cross-border cooperation. The stakeholders almost unanimously recognised the added

value of the programme and its essential contribution to generating benefits.

The external study also revealed that the programme's transparency and reporting

requirements contribute to the improvement of the beneficiaries’ planning, monitoring

and quality standards and to the quality of the beneficiaries' financial management in

general.

Sustainability

There is a consensus among beneficiaries that the actions funded under the programme

will continue producing benefits, even after their completion. This was particularly

relevant for technical assistance actions where the deployment of equipment will

continue generating results beyond the closing date of a grant agreement, but also applied

to training actions, where the beneficiaries were generally positive about the lasting

effect of networking and exchanges of information and best practices. Nonetheless, a

substantial number of beneficiaries indicated that they would discontinue the activities

funded under the programme, if it were to be terminated. This finding suggests that the

discontinuation of the programme would negatively impact the overall protection of the

financial interests of the Union.

Contribution to smart, sustainable and inclusive growth

The evaluation found that the programme contributes indirectly to EU’s strategic goal for

smart, sustainable and inclusive growth, in particular by protecting the financial

resources that are used to achieve the priority’s goals (revenues) and ensuring that such

resources are spent to foster smart, sustainable and inclusive growth rather than illegally

diverted to other ends (expenditures). The increased protection of EU external borders

supported by the programme also contributes to creating a level playing field between

EU businesses and their competitors based in third countries.

kom (2018) 0003 - Ingen titel

Annex 1: Procedural information

This annex provides procedural information. It explains how the evaluation was

managed both in terms of organisation and time. It provides information about external

expertise used.

A1.1

Lead DG:

European Commission, European Anti-Fraud Office (OLAF).

A1.2

Organisation:

The evaluation roadmap

was published in August 2016 and sets out the purpose and

scope of the exercise. The roadmap presented the questions to be addressed under the

five categories of effectiveness, efficiency, relevance, coherence and EU added value.

An Inter-Service Steering Group (ISSG) was set up to support the evaluation. The tasks

of the ISSG were to check the key elements of the external study, to support and monitor

the evidence gathering and stakeholder consultation process, to review all external

deliverables for the evaluation as well as the SWD and to assist with the quality

assessment of the consultant's evaluation report.

The ISSG was composed of the Secretariat-General of the Commission, the Legal

Service and DGs TAXUD, JUST, HOME and BUDG.

The Commission contracted out an external study in support to the evaluation (see

below), which provided the basis for this SWD.

A1.3

Timetable:

Description

Establishment of the Steering Group

Steering Group Meeting: Evaluation roadmap and ToR

Specifications for study contract to support the evaluation sent to

the framework contractor

Steering Group Meeting: Kick off meeting with the external

contractor

Award of a study contract and signature of the contract with

CEPS on behalf of the consortium

Steering Group Meeting: Inception report meeting

Targeted stakeholder consultation –surveys and interviews

Steering Group Meeting: End-of-fieldwork meeting

Steering Group Meeting: External Draft final report meeting

Steering Group Meeting: 1

draft Staff Working Document

Adoption of the report accompanied by the SWD

Date

June 2016

5 July 2016

09 September 2016

25 October 2016

12 December 2016

February - June 2017

30 June 2017

4 September 2017

24 October 2017

January 2018

Source: OLAF

http://ec.europa.eu/smart-regulation/roadmaps/docs/2017_olaf_002_midterm_evaluation_en.pdf

kom (2018) 0003 - Ingen titel

A1.4

Exceptions to the Better Regulation Guidelines:

Given the scope of the Regulation and the well-defined group of stakeholders, targeted

consultations were considered as an appropriate and proportional means of consultation

rather than an open public consultation. Considering that the public at large could not be

considered as directly impacted by the provisions of the Regulation, responsible for their

application, or possessing specific evidence needed for the evaluation, an open public

consultation was not carried out.

A1.5

Consultation of the Regulatory Scrutiny Board:

The evaluation has not been selected for scrutiny by the Regulatory Scrutiny Board.

A1.6

External Expertise:

An external independent study is the basis for the conclusions presented in this

document. A contract was signed on 25 October 2016 with Economisti Associati-led

consortium (external contractor)

The external contractor carried out targeted consultations for the evaluation as explained

in Annex 2.

Using the framework contract n° 2015/CC/132 concluded by DG TAXUD for the "provision of

evaluation and Impact Assessment related services".

kom (2018) 0003 - Ingen titel

Annex 2: Synopsis report of stakeholder consultation in the

context of the Study

This annex provides a synopsis of the stakeholder consultation that was carried out for

this evaluation by an external contractor as part of the external support study. It presents

the main steps and findings of the consultation of interested parties and stakeholders. It

shows that the stakeholder consultation process

complied with the Commission's

stakeholder consultation principles and minimum standards taking into account

proportionality

as required by the Better Regulation Guidelines

A1.1

Feedback on evaluation roadmap

The evaluation roadmap was opened for feedback during the course of the evaluation.

feedback was received

in this period.

A1.2

Open public consultation

Regulation 250/2014 established the Hercule III programme to promote activities in the

field of the protection of the financial interests of the European Union. The scope of this

mid-term evaluation covers the interventions supported under the Hercule III programme

to date as well as the preparatory and implementing activities undertaken by the

stakeholders for these interventions.

The groups of stakeholders it targets and impacts are well-defined, i.e. for the most part

administrators and law enforcement officers at national and regional level. The public at

large cannot be considered as directly impacted by the provisions of the Regulation or

responsible for their application or possessing specific evidence that would usefully

contribute to the evaluation of the application of the Regulation. Therefore, an

open

public consultation was not carried out and stakeholders were consulted in targeted

consultations

as explained below.

A1.3

Targeted consultations – overview of stakeholders consulted

This section presents an overview of different stakeholder groups consulted via in-depth

interviews and a survey.

A total of 574 stakeholders

were consulted through interviews and survey, comprising

16 officials of Commission or national services, 56 beneficiaries

(i.e. 71% of all

beneficiaries),

67 unsuccessful applicants

(i.e. 25% of all unsuccessful applicants),

321

participants in events

(i.e. 27% of all participants invited to reply) and

112 users of

services

(i.e. 31% of all users invited to reply).

Whereas 49 stakeholders

(i.e. all

officials from institutions and 33 beneficiaries) were interviewed, the remaining

stakeholders participated in the online surveys.

Better Regulation Guidelines, p. 70.

kom (2018) 0003 - Ingen titel

European Commission and national officials

Furthermore, exploratory interviews were conducted with several OLAF officials and

with officials from DG BUDG, DG JUST, DG TAXUD. These interviews were

additional to the 16 interviews with the Commission and national services conducted

during the Fieldwork phase, which were divided as follows:

five interviews with OLAF

officials

(Hercule Sector, Internal Auditor, units B1, C3, and D4);

two interviews with

officials from other DGs

(DG JUST and DG TAXUD);

nine interviews with officials

from national institutions

(eight AFCOS which are also members of the Advisory

Committee for the Coordination of Fraud Prevention (COCOLAF)

; and one central

customs authority) in Austria, Bulgaria, France, Germany, Italy, Lithuania, Poland,

Romania and Slovenia.

Beneficiaries

In total,

56 beneficiaries from 18 Member States were consulted

i.e. 71% of those

that were awarded a grant either in 2014 or 2015 (Figure 1).

Figure 1. Consulted beneficiaries by Member States and category of action (number

of respondents)

Anti-fraud training

Legal training and studies

Technical Assistance

Unsuccessful applicants

All entities that applied without success to the Hercule III calls for proposals in 2014,

2015 and 2016 were invited to participate in the online consultation.

67 respondents

COCOLAF’s mission is “to

advise the Commission on any matter relating to the prevention and

prosecution of fraud and all other illegal activities adversely affecting the financial interests of the

Community, and on any matter relating to cooperation between the competent authorities of the

Member States or between Member States and the Commission to protect the financial interests of the

Community, in order to organise more effectively close and regular cooperation between the competent

authorities

counter

fraud”.

For

further

details

see:

http://ec.europa.eu/transparency/regexpert/index.cfm?do=groupDetail.groupDetail&groupID=656

Interviews were conducted with beneficiaries from Belgium, Bulgaria, France, Italy, Lithuania, the

Netherlands, Poland and Romania.

kom (2018) 0003 - Ingen titel

from 21 Member State completed the survey,

i.e. 25% of all unsuccessful applicants

during the first three years of the programme (Figure 2)

. Almost 50% of respondents

were located in Poland and

Romania;

this is in line with data registered in the total

population, as 121 out of 267 (45%) unsuccessful applications during the first three years

of the programme were submitted by Polish and Romanian entities.

Figure 2. Consulted unsuccessful applicants by Member States and category of

action (number of respondents)

PL RO BG ES HU EE

LV CZ

BE CY DE DK EL HR

MT SE UK

Anti-fraud training

Legal training and studies

Technical Assistance

Events participants

312 events participants from 25 Member States and nine from third countries completed

the online survey (Figure 3).

Six invitations to participate in the survey were bounced, as the email address indicated in the

application form was either incorrect or obsolete.

kom (2018) 0003 - Ingen titel

Figure 3. Consulted participants by country of origin (number of respondents)

24 23

10 10 9

Unit: Number of respondents.

Note: 321 total respondents.

Source: Authors’ elaboration on online survey with participants in events.

About one-third of the respondents took part in events arranged in a different MS from

their country of residence (Figure 4). The three largest groups of participants by type of

organisation included customs officials, police officials and academic audiences

(Figure 5). Interestingly, many respondents had participated in more than one type of

event in the context of the same action funded by Hercule III (Figure 6). The reason is

that several actions combined different types of event to achieve the expected results.

Respondents were aged mainly between 25 and 54, with women representing less than

40% of the sample (Table 1).

Figure 4. Participants in events held in the same Member State and in a different

Member State

33%

67%

National Participation

Note: 321 respondents.

Cross Border Participation

kom (2018) 0003 - Ingen titel

Figure 5. Consulted participants by type of organisation

Law enforcement: Customs

Academic audience

Law enforcement: Police

State revenue service

Ministry of Finance

Local/regional authorities

NGO representatives

Legal practitioners: Lawyers,

solicitors, barristers

Ministry of Interior/Home affairs

Other

12%

20%

27%

Note: 321 respondents.

Figure 6. Consulted participants by type of event attended

147

Conference

Training

course

Seminar

Round-table

Working

group

Meeting

Other

Note: 321 respondents; each respondent may have participated in more than one event in the

context of the same action funded by Hercule III.

Table 1. Consulted participants by age category and gender

Age category

Below 25

25-34

35-44

45-54

55-64

Total

Source: External study

Gender

Female

1.3%

9.1%

14.4%

9.1%

4.4%

0.0%

38.1%

Male

1.3%

9.7%

22.8%

18.4%

7.2%

1.3%

60.6%

Prefer not to disclose

0.0%

0.6%

0.3%

0.0%

1.3%

Total

2.5%

19.4%

37.5%

27.8%

11.6%

1.3%

100.0%

kom (2018) 0003 - Ingen titel

For confidentiality reasons, relevant contact details for

users of services

procured by

Hercule III were not provided to the external contractor; hence, OLAF sent the

invitations to complete the online survey to users. Against this background, based on

information provided by OLAF,

352 users were invited to complete

the

survey, which

was eventually completed by 112 respondents

(32% response rate; Table 2). Users

were mainly based in Germany, the UK, Belgium and Italy (Figure 7). Almost 70% of

respondents are users of statistics and IT tools; 30% are users of databases (e.g. NTELX,

GTI, GRS, D&B reporting system, SEASEARCHER, GTA); only 6% are users of

services to carry out chemical analysis of samples from tobacco and/or cigarette seizures

(Figure 8).

Table 2. Share of total population represented by consulted beneficiaries,

unsuccessful applicants, participants in events and users of services

Type of stakeholder

Total beneficiaries

Beneficiaries TA

(2014-15)

Beneficiaries AFT

(2014-15)

Beneficiaries LTS

(2014-15)

Total applicants

Applicants TA

(2014-16)

Applicants AFT

(2014-16)

Applicants LTS

(2014-16)

Participants

(2014-15)

Users

(2014-16)

Total sample

321

112

Total population

267

174

1,194*

(3,400)**

352*

Sample / population

71%

66%

84%

63%

25%

32%

19%

27%*

(9%)**

32%*

Note: *Participants/users invited to complete the online survey. **Participants in events

funded during the first two years of the programme according to the Annual Implementation

Reports.

As regards the type of organisation, the lion’s share of respondents belongs to the

customs category (Figure 9). No information is available with regard to the total

population of users; hence, no conclusion can be drawn about the share of the population

represented by the sample.

kom (2018) 0003 - Ingen titel

Figure 7. Consulted users by country and type of service (number of respondents)

DE UK BE IT SI PL RO SK HU IE NL SE BG CZ DK FR CY ES PT FI EL MT EE HR LT AT

User of statistics and IT tools

User of chemical analysis of tobacco and cigarettes

User of databases

Note: 112 total respondents.

Figure 8. Consulted users by type of service

26%

68%

User of databases

User of chemical analysis of tobacco and cigarettes

User of statistics and IT tools

Note: 112 respondents.

kom (2018) 0003 - Ingen titel

Figure 9. Consulted users by type of organisation

88%

Law enforcement: Customs

National Authorities

Law enforcement: Police

Other

Local/regional authorities

Note: 112 respondents.

kom (2018) 0003 - Ingen titel

Annex 3: Methods and analytical models

This annex provides a description of the methodological approach to the evaluation taken

by the external contractor that supported this SWD. It summarises the main

methodological elements and cross-references to the more detailed methodological annex

of the external report supporting this SWD.

A1.4

Logic of intervention

The intervention logic developed in the context of the evaluation includes a detailed

description of the needs and problems that the programme is supposed to address, the

objectives that the programme is expected to achieve, the activities that the programme

intends to implement, the expected results of the programme and the logical links

between these various components. A comparison between the expected results (i.e. how

the programme was intended to work) and the actual results stemming from the

intervention under analysis (i.e. how the programme worked in reality) enables a mid-

term evaluation based on evidence. Against this background and in line with the Better

Regulation Guidelines, the following components of the intervention logic were

identified and described:



The rationale for the intervention



Needs and problems



Objectives (general, specific, operational)



The intervention



Inputs/activities

The expected results of the intervention



Outputs (expected)



Outcomes (expected)



Impacts (expected)

A1.5

Evaluation Framework

The evaluation framework serves the purpose of guiding the mid-term evaluation and

includes the following items:



evaluation criteria;

evaluation questions;

kom (2018) 0003 - Ingen titel



judgment criteria;

indicators;

data sources;

data collection methods.

The six evaluation criteria are generally used to evaluate EU policies and are enshrined in

the 2015 “Better Regulation Guidelines”

and the “Better Regulation Toolbox”

. These

criteria are made more specific through evaluation questions. Judgment criteria,

expressed in the form of statements are used to answer the various evaluation questions.

They can be then confirmed and/or rejected by the research. A comprehensive set of

indicators serve the purpose of applying judgment criteria and have been chosen

according to the RACER framework

. Data sources are the sources of evidence on

which the external study relies to apply the judgment criteria and answer the evaluation

questions. Data collection method is the approach adopted to gather the required

evidence.

A1.6

Data collection

Primary data,

i.e. data that were collected specifically for the purpose of the

Assignment via:



interviews

with: i) Commission staff; ii) national institutions active in fraud

prevention and PFI; and iii) beneficiaries of actions funded by the programme; and



online surveys

with: i) beneficiaries of actions funded during the first two years

of the programme; ii) unsuccessful applicants; iii) participants in events co-

financed by Hercule III; and iv) users of services procured by Hercule III.

Secondary data,

i.e. data gathered by consulting:



public available sources

including,

inter alia,

the Regulation and accompanying

material (e.g. the IA), AWPs, Annual Implementation Reports, Article 325

reports, final evaluation reports and interim reports of the Hercule II programme

and legal texts establishing other programmes (e.g. Customs 2020, Fiscalis 2020,

Internal Security Fund); and

European Commission,

Better Regulation Guidelines,

19 May 2015,

http://ec.europa.eu/smart-regulation/guidelines/docs/swd_br_guidelines_en.pdf.

European Commission,

Better Regulation “Toolbox”,

19 May

http://ec.europa.eu/smart-regulation/guidelines/docs/br_toolbox_en.pdf.

2015,

available

at:

available

at:

The RACER framework (see European Commission,

Better Regulation “Toolbox”,

19 May 2015)

requires adopting indicators that are: i) relevant, i.e. closely linked to the EQ as operationalised

through the judgment criteria; ii) accepted, i.e. retrieved from relevant literature or best evaluation

practices; iii) credible, i.e. easy to interpret and unambiguous, especially in view of drawing evidence

for policy-making; iv) easy to monitor, i.e. measurable at a low cost; and v) robust, which is equivalent

to saying they cannot be manipulated by the regulators or regulated subjects.

kom (2018) 0003 - Ingen titel



official documentary evidence

on both submitted proposals (i.e.

application forms) and awarded contracts and grants (i.e. feedback from

grant beneficiaries in the form of Final Technical Reports, Final Financial

Reports and Final Implementation Reports (limited to technical assistance

actions)).

A1.7

Evaluation challenges and limitations

This section describes the main challenges encountered and the associated limitations of

the evaluation results and mitigation measures put in place. In doing so it provides an

overall assessment of the robustness of the methodology applied and the reliability of the

available data.



The population coverage ensured by the consulted national institutions (Member

States receiving more than 60% of grants awarded in 2014 and 2015, plus two

MS that did not receive any grant) and beneficiaries (71% of the total population)

allows to draw robust conclusions for all evaluation questions.

However, the shares of total population represented by unsuccessful applicants

(25%), participants in events (27% of invited participants, 9% of total population)

and users of services (32% of invited users) are lower. Therefore, more caution is

required when interpreting conclusions based on these surveys. In order to ensure

the highest quality of the findings: i) all evaluation questions are addressed by

combining feedback from more than one stakeholder category; ii) whenever

possible, data and information collected from stakeholders are compared with

evidence retrieved from application forms and reporting documents in order to

ascertain consistency across data sources consulted to perform the evaluation.

The

timing of the mid-term evaluation

is compliant with Article 13 of the

Regulation establishing the Hercule III programme, which requires the

Commission to present an Independent Mid-Term Evaluation Report to the

European Parliament and the Council by 31 December 2017. Against this

background,

the evaluation is confined to actions funded during the first two

years of the programme

(2014 and 2015). As regards 2016, the analysis is

limited to applications received in the context of calls for proposals; in fact, no

action co-financed by Hercule III grants for 2016 calls was completed at the

moment of conducting fieldwork activities. Some actions covered by the 2014

and 2015 budget were still ongoing during fieldwork; hence, their impact can

only be recorded to a limited extent. Due to the unexpected high number of

applications, some delays in awarding the grants and finalising the funded actions

were registered. In addition, based on information retrieved from application

forms, half of the technical assistance actions funded in 2014 and 2015 were

expected to last more than 12 months.

These data limitations impinge on the

assessment of both outcomes and impacts:



With regard to outcomes,

13 out of 25 technical assistance actions

surveyed for this Assignment were still ongoing at the moment of

gathering data and information to assess the effectiveness of the

programme.

In addition, for completed actions, most beneficiaries argued

that they only recently received the devices, tools and systems co-financed



kom (2018) 0003 - Ingen titel

by Hercule III. As a consequence, whereas most of the beneficiaries of

technical assistance actions provided output indicators (e.g. type and

number of items purchased with Hercule III funds), evidence on outcome

indicators (e.g. successful operations, number of arrests, convictions

seizures, etc.) is scant. Therefore, limited conclusions can be drawn about

the alignment between actual and expected outcomes of technical

assistance actions.



Impacts occur in the long-term and cannot be captured by a mid-

term evaluation.

In principle, in the long-run, the Hercule III programme

is expected to improve the PFI, thus ensuring the competitiveness of the

European economy and the protection of taxpayers’ money. It is too early

to measure such impacts, especially considering that the first actions

funded by the programme were only implemented in 2015.



Available primary and secondary data and information mainly allow

evaluating actions funded via grant agreements,

which represent 75% of

the overall commitments in 2014 and 2015. The evaluation of databases, IT

tools and analyses of cigarettes and tobacco samples funded under

procurement is performed on the basis of feedback provided by respondents

to the online survey with users; hence, all TA actions

have been assessed. It

was not possible to collect evidence from the contractors directly due to

confidentiality, data protection and contractual reasons in case of the

evaluation of conferences, digital forensics training, studies and other actions

procured by Hercule III. Therefore, OLAF contacted the contractor to

distribute the links to the online surveys and to request additional information

where necessary (e.g. satisfaction surveys). Such actions represent less than

10% of the overall commitments in the first two years of the programme.

Therefore, the Assignment covers the bulk of the programme.



The external evaluator did not receive the contact details for event

participants and users of services procured by Hercule III due to

confidentiality and data protection reasons.

With regard to the survey with participants,

OLAF requested beneficiaries of anti-

fraud and legal training grants to invite participants in the events they arranged to

complete the online survey. This approach may lead to biased results

as beneficiaries

might have distorted incentives, e.g. to share introductory emails only with a limited set

of “satisfied” participants or not to support the survey at all, especially in case they

believe there is a risk of negative feedback. It was not possible to control for this type of

bias. Whenever possible, a consistency check

was performed between data collected via

the online survey with participants and data retrieved from Final Technical Reports of

TA actions funded via grant agreements and procurement contracts represent 80% of the commitments

in 2014 and 2015.

A consistency check aims to detect whether the value of two or more data items are not in

contradiction (UN Statistical Commission, UNECE, 2000. Glossary of Terms on Statistical Data

Editing).

kom (2018) 0003 - Ingen titel

anti-fraud training actions. These include, inter alia, the participants’ assessment of each

event with replies provided by about 3,000 participants (i.e. almost 90% of all

participants Table 1)

. In addition, to facilitate future evaluations, the external study

suggests including additional reporting requirements for beneficiaries of actions

involving the organisation of events: beneficiaries should provide OLAF with email

addresses of all participants in events, accompanied by a data protection waiver allowing

OLAF and OLAF’s contractors to contact the participants for evaluation purposes.

Similarly, OLAF was in charge of sending the invitations to complete the online survey

to users of the services funded by Hercule III. While this approach did not lead to any

potential bias, information about the total population of users is unavailable.

Overall, the contractor declared that the planned data collection and analytical exercises

could be implemented as planned. Most of the challenges were overcome and/or

mitigated against. The robustness of the evidence base gathered was judged satisfactory

by the evaluators.

Whereas the survey with participants allows gathering the stakeholders’ perceptions a few months

after the event was held, participants’ assessments included in Financial Technical Reports are based

on data gathered during the event. Therefore, the latter does not allow capturing whether participants,

for example, used the skills acquired during the event in their work activities or interacted with people

they met at the event after the event ended.