Europaudvalget 2017
KOM (2017) 0477
Offentligt
1788419_0001.png
EUROPEAN
COMMISSION
Brussels, 13.9.2017
SWD(2017) 502 final
COMMISSION STAFF WORKING DOCUMENT
on the evaluation of the European Union Agency for Network and Information Security
(ENISA)
Accompanying the document
Proposal for a regulation
on ENISA, the "EU Cybersecurity Agency", and repealing Regulation (EU) 526/2013,
and on Information and Communication Technology cybersecurity certification
(''Cybersecurity Act'')
{COM(2017) 477 final}
{SWD(2017) 500 final}
{SWD(2017) 501 final}
EN
EN
 kom (2017) 0477 - Ingen titel
Table of Contents
1.
INTRODUCTION ....................................................................................................... 3
1.1. Purpose of the Evaluation .................................................................................. 4
1.2. Scope of the Evaluation ..................................................................................... 4
2.
BACKGROUND TO ENISA ...................................................................................... 5
2.1. Description of the ENISA Mandate .................................................................. 5
2.2. Baseline ............................................................................................................. 7
3.
4.
5.
6.
IMPLEMENTATION STATE OF PLAY .................................................................. 8
METHOD .................................................................................................................... 9
RESPONSES TO THE EVALUATION QUESTIONS ........................................... 11
CONCLUSIONS ....................................................................................................... 18
ORGANISATION AND TIMING
................................................................................. 22
2
 kom (2017) 0477 - Ingen titel
1788419_0003.png
1.
I
NTRODUCTION
The European Agency for Network and Information Security (ENISA) was originally
established in 2004 and had its mandate renewed periodically. The current ENISA
mandate is set out in Regulation EU No. 526/2013
1
(the 'ENISA Regulation') and is due
to expire in June 2020.
Article 32 of the ENISA Regulation requires the Commission to conduct an evaluation to
assess in particular the impact, efficiency and effectiveness of the Agency and its
working practices by 20
th
June 2018. Furthermore the evaluation results are to inform
the Commission as to whether it is to propose that the duration of the current mandate be
extended.
In its 2016 Communication "Strengthening Europe's cyber resilience system and
Fostering a Competitive and Innovative Cybersecurity industry"
2
, the Commission
announced that, taking also into account the reinforced role that the NIS Directive
attributes to the Agency, it will advance its evaluation and, subject to the results of such
evaluation, it would present a proposal for a possible new mandate as soon as possible. In
its Communication on the DSM Strategy Mid-term Review of May 2017
3
, the
Commission has further specified that, it would review the mandate of ENISA, included
in its Work Programme, by September 2017 in order to define its role in the changed
cybersecurity ecosystem.
The evaluation falls under the Commission’s Regulatory
Fitness and Performance
Programme (REFIT). It has been conducted according to an evaluation roadmap
4
that
was made public in July 2016.
The Commission concluded a tender with a consortium led by CARSA5, to provide an
independent evaluation of ENISA in autumn 2016 (Annex 4). The present Staff Working
Document is largely based on the results and conclusions of that evaluation study.
http://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1495472820549&uri=CELEX:32013R0526
COMM (2016)410 final.
3
Commission Communication on the Mid-Term Review on the implementation of the Digital Single
Market Strategy - COM(2017) 228.
4
http://ec.europa.eu/smart-regulation/roadmaps/docs/2017_cnect_002_evaluation_enisa_en.pdf
5
Consortium includes CARSA (Consultores de Automatización y Robótica S.A. - lead partner), Logotech
(partner), Ramboll Management Consulting S/A (partner), AIT Austrian Institute of Technology
GmbH (partner), ZSI
–
Zentrum für Soziale Innovation (partner) and Agilis S.A. (partner).
2
1
3
 kom (2017) 0477 - Ingen titel
1.1.
Purpose of the Evaluation
The purpose of the evaluation of ENISA was to assess the performance of the Agency in
achieving its objectives, mandate and tasks, as laid down in the Regulation No 526/2013
and to provide the basis for a possible revision of the current mandate. Its results fed the
impact assessment regarding the future of ENISA.
In compliance with the Better Regulation Guidelines, the evaluation has assessed the
effectiveness, efficiency, coherence, relevance and EU added value of the Agency,
having regard to its performance, governance, internal organisational structure and
working practices.
The analysis also took account of the evolved context where the Agency now operates,
with regard in particular to: the new EU regulatory and policy framework (e.g. the NIS
Directive, the Review of the EU Cybersecurity Strategy); the evolving needs of the
Agency's stakeholders' community; and the complementarity and possible synergies with
the work conducted by other EU and national institutions, agencies and bodies, such as
CERT-EU and the European Cybercrime Centre (EC3) at Europol.
The present Staff Working Document will accompany the Commission Report to the
European Parliament and the Council to allow for a decision how to pursue the
recommendations made.
Scope of the Evaluation
1.2.
The legal basis for this evaluation is set out in article 32 of ENISA's Regulation.
In particular, the main objectives of the evaluation have been:
1. to assess the effectiveness, efficiency, relevance, coherence and EU value added
of the work undertaken by the Agency and its working practices. The assessment
has sought to evaluate, but not be limited to, the implementation of the work
programme as well as how the whole set of activities run by ENISA (including
opinions, guidelines, trainings, recommendations or reports) have contributed to
fulfil its role, objectives, mandate and tasks.
2. to assess how effectively the current governance as well as the internal
organisational structure of ENISA (Management Board-MB, Executive Board,
Executive Director and staff and Permanent Stakeholders Group -PSG) have
contributed to efficiency and effectiveness in the work of ENISA. The assessment
of the organisational structure has also included an evaluation of the efficiency
and effectiveness of the current arrangements related to the location of ENISA's
offices.
3. to assess how successfully ENISA, within its mandate, has met the needs of its
constituency in comparison to other EU and national bodies working on
cybersecurity.
4. to assess the possible need for a revision or extension of the mandate entrusted to
ENISA, also taking into account the evolution of the cybersecurity and digital
privacy landscape, including the regulatory and policy framework (in particular
the adoption of the NIS Directive and the current review of the Cybersecurity
Strategy).
4
 kom (2017) 0477 - Ingen titel
The time period covered by the evaluation is 2013
–
2016 but data and information were
collected and analysed also in 2017. The analysis in fact starts with the entry into force of
the Regulation No 526/2013, which set the new mandate for ENISA and concludes with
issuing of the DSM Strategy Mid-term Review of May 2017, where the Commission has
further specified that it would review the mandate of ENISA by September 2017 in order
to define its role in the changed cybersecurity ecosystem. In terms of geographical scope,
the evaluation assessed the impact of ENISA on all 28 Member States.
2.
B
ACKGROUND TO
ENISA
2.1.
Description of the ENISA Mandate
ENISA was established in 2004 (Regulation (EC) No 460/2004) as the European Union
Agency for Network and Information Security with the objective of facilitating a high
level of network and information security within the EU. The Agency was established in
the context of the new emerging digital economy and the need to safeguard its
development. The initial foreseen duration for the Agency's mandate was 5 years but it
was extended twice (in 2009 and 2011).
The current mandate of the Agency is set out in Article 1 of Regulation EU n. 526/ 2013
(which repealed the 2004 Regulation and represents the new basic Act for ENISA):
"to
undertake the tasks assigned to it for the purpose of contributing to a high level of
network and information security within the Union and in order to raise awareness of
network and information security and to develop and promote a culture of network and
information security in society for the benefit of citizens, consumers, enterprises and
public sector organisations in the Union, thus contributing to the establishment and
proper functioning of the internal market".
ENISA can only intervene where it does not
impact on the national competence of Member States in regard to network and
information security matters such as national security, defence, public security and
criminal law matters.
ENISA's objectives are defined by article 2 of its Regulation, namely:
1. The Agency shall develop and maintain a high level of
expertise.
2. The Agency shall assist the Union institutions, bodies, offices and agencies in
developing policies
in network and information security.
3. The Agency shall assist the Union institutions, bodies, offices and agencies and the
Member States in
implementing the policies
necessary to meet the legal and regulatory
requirements of network and information security under existing and future legal acts of
the Union, thus contributing to the proper functioning of the internal market.
4. The Agency shall assist the Union and the Member States in enhancing and
strengthening their capability
and preparedness to prevent, detect and respond to
network and information security problems and incidents.
5. The Agency shall use its expertise to
stimulate broad cooperation
between actors
from the public and private sectors.
ENISA's understanding of these objectives is set out in the diagram overleaf.
5
 kom (2017) 0477 - Ingen titel
1788419_0006.png
Figure 1 ENISA's activities based on its objectives
Developing and
maintaining a high
level of expertise
of
EU actors taking into
account evolutions in
NIS
Assisting
Member
States and the
Commission in
enhancing capacity
building
throughout
the EU
Assisting
MS and the
EC in
developing
policies necessary to
meet the legal and
regulatory
requirements
of NIS
Enhancing
cooperation
both
between MS of the EU
and between related
NIS communities
Source: Ramboll Management Consulting based on ENISA Website
For ENISA to achieve its objectives, the Regulation identified a list of tasks, as well as
operational, governance, organisational and financial provisions. The problems ENISA is
intended to address are set out in detail in the baseline section. The diagram below
(figure 2) summaries how ENISA as an Agency is intended to address the core issues
such as lack of knowledge, expertise, trust and the resulting limited co-operation.
Figure 2 ENISA's Intervention Logic Diagram
Source: Ramboll Management Consulting
ENISA is governed by a Management Board (MB) consisting of representatives from
each of the 28 Member States and the Commission. A subgroup of the MB forms the
Executive Board and it is tasked with routine decision making in the interests of
administrative efficiency. An Executive Director appointed by the Management Board is
responsible for managing the Agency. A Permanent Stakeholders Group (PSG) of
6
 kom (2017) 0477 - Ingen titel
1788419_0007.png
experts from industry, academia and consumer organisations advises the Agency in
regard to the performance of its activities. The Agency also maintains an informal
network of National Liaison Officers from the Member States to promote the Agency and
facilitate outreach.
2.2.
Baseline
The current ENISA Regulation has repealed the 2004 founding act of the Agency and the
situation before its adoption is considered as baseline for the purpose of the evaluation.
Before 2004, prior to the existence of ENISA, there was no European focal point
regarding network and information security. Furthermore, some Member States were not
only devoting limited resources to network and information security capacity building,
but were not sufficiently aware as to the importance of security and resilience of
information communications technologies. There was no EU level advice or support
mechanism to assist the Member States or the EU institutions, agencies and bodies
become more aware of the critical issues related to secure and resilient digital
infrastructure and services.
In the period 2004
–
2013, the EU had benefited from the establishment of an agency
mandated to contribute to network and information security, helping Member States and
the business community to prevent, to address and to respond to major network and
information security risks. The mandate had not been substantially changed during this
time frame but simply extended with regard to its duration.
However, the fast evolving nature of the cybersecurity domain, which was slowing
becoming also a policy priority at EU and national level, made the previous mandate of
ENISA outdated. In fact, by 2013, the centrality of the smartphone, the proliferation of
mobile apps and the increasing reliance of public services (e.g. energy generation,
transportation, etc.) on digital technologies required a renewed focus on network and
information security. That year saw the publication of the first EU Cybersecurity
Strategy
6
, accompanied with a proposal for a Directive on Network and Information
Security (the 'NIS Directive').
As reported in the impact assessment accompanying the 2013 legislative proposal,
throughout the debate on the future NIS policy in Europe the Member States and various
stakeholders repeatedly expressed the view that a modernised NIS agency was needed to
best serve the goals of a renewed NIS strategy. In particular, the tasks defined by the
previous ENISA Regulation were considered insufficient to provide the Agency with the
necessary flexibility and adaptability to respond to the challenges of the continuously
evolving NIS environment.
The renewal of the ENISA mandate in 2013 addressed also the internal governance and
operations of the Agency (i.e. an Executive Board to aid decision making, a requirement
for a branch office in Athens to increase operational efficiency). An effort was made to
make the mandate flexible enough to respond to the evolving threat landscape but no
substantive changes were made to the key objectives and the Agency was again entrusted
with a fixed-term mandate (until 2020). This followed the rapid development of the
6
http://ec.europa.eu/newsroom/dae/document.cfm?doc_id=1667
Cybersecurity Strategy for the European
Union: An Open Safe and Secure Cyberspace JOIN 2013 (1) final 7 Feb. 2013
7
 kom (2017) 0477 - Ingen titel
1788419_0008.png
Agency from a start-up situation in 2004. By the end of 2012, ENISA had 58 staff
members and an annual operating budget of approx. € 8.2
million
7
.
3.
I
MPLEMENTATION STATE OF PLAY
Based on the mandate entrusted to it, ENISA supports the European Institutions, the
Member States and the business community in
addressing, responding and especially
preventing network and information security problems.
It does so through a series of
activities across five areas identified in its strategy
8
, adopted by the Management Board
in 2016:
expertise: provision of information and expertise on key network and information
security issues.
policy: support to policy making and implementation in the Union.
capacity: support to capacity building across the Union (e.g. through trainings,
recommendations, awareness raising).
community: foster the network and information security community [e.g. support
to the Computer Emergency Response Teams (CERTs), coordination of cyber
exercises].
enabling (e.g. engagement with the stakeholders and international relations).
ENISA carries out its activities according to an annual and multiannual work
programme
9
. Every year, the Executive Director puts forward a proposal for the annual
work programme, including a multiannual outlook on the strategic objectives and the
resources. The resulting programming document is discussed by the Management Board
and the Permanent Stakeholder Group provides an opinion to the Executive Director. The
Commission also provides an official opinion based on the draft approved by the
Management Board.
ENISA regularly executes its work programmes according to the planning. The latest
Work Programme (WP 2016) has resulted in 64 deliverables executing in full the
planning. The results of each work programme are presented by the Executive Director in
annual activity reports, which in the period under consideration (2013-2016) have been
adopted unanimously and on time by the Management Board.
The Executive Director also commissioned annual external evaluations
10
of the Agency,
the conclusions of which fed follow-up action plans. The progress on the follow-up
action plans has been reported annually to the Management Board and the Commission,
indicating where no specific action had been taken due to management decision or lack
of resources.
7
8
ENISA General Report 2012, position on 31/12/2012
https://www.enisa.europa.eu/publications/corporate/enisa-strategy
9
https://www.enisa.europa.eu/publications/corporate/enisa-work-programmes-general-reports
10
https://www.enisa.europa.eu/about-enisa/annual-ex-post-evaluation-of-enisa-activities
8
 kom (2017) 0477 - Ingen titel
1788419_0009.png
In recent years, ENISA has gained responsibilities involving the Regulation on electronic
identification and trust services for electronic transactions in the internal market (eIDAS
Regulation), the Telecommunications Framework Directive and the ePrivacy Directive
around security and reporting requirements.
Furthermore, under the Directive on Security of Network Information Systems (the 'NIS
Directive', adopted in 2016), the Agency has gained significant new responsibilities
involving assistance to Member States and its involvement in both the NIS Cooperation
Group and as secretariat of the CSIRT Network, the two fora being responsible for EU
cybersecurity cooperation at strategic and operational level respectively (articles 7, 9, 11,
12 and 19 of the NIS Directive).
ENISA has today 84 staff members, of which 48 Temporary Agents, 30 Contract Agents
and 5 Seconded National Experts. Its offices are located in Greece, notably the
administrative seat in Heraklion (Crete) and the core operations department in Athens.
ENISA's annual budget increased by 16% over the period 2013 to 2016, amounting to
10.5m€ in 2016; in 2017 its budget was raised to 11.25 m€. The revenue of the Agency
derives mostly from the EU budget, to which an annual contribution is added from the
Greek authorities for the rental costs of its offices and a contribution from EFTA
countries.
4.
M
ETHOD
The evaluation process was assisted by a
Steering Group
composed of the
representatives of the European External Action Service and selected Commission
Directorates General (DGs) including DG CNECT, DG HOME, DG JUST, DG JRC, DG
DIGIT, DG HR, DG BUDG, together with the Secretariat-General and the Legal
Service.
The Group steered and monitored the progress of the exercise, ensuring the necessary
quality, impartiality and usefulness of the evaluation.
The evaluation was supported by an external comprehensive study. The contractors were
given the task of collecting data and evidence in order to answer the evaluation questions
set out in annex 3.
During the preparation phase of the study, familiarisation interviews took place with the
Commission services
11
and CERT-EU at the commencement of the study. This was
supplemented by preliminary desk research covering legal, policy and academic
documents of relevance to the study.
Extensive desk research, as part of the data collection phase, on both primary and
secondary sources was then conducted to capture the legal and regulatory landscape
involving Commission Communications, Regulations and Directives, the work outputs of
the Agency (i.e. workshop reports, conference publications, study findings) reports from
the cyber security stakeholders (industry white papers, indices, public policy papers,
expert groups etc) and websites, blogs and databases. The study then conducted in-depth
interviews of up to 90 minutes each with 49 persons drawn from the cybersecurity
stakeholders. This involved ENISA staff and management, Member State representatives
(including some involved in the governance of ENISA), industry representatives, staff of
11
DG-CNECT, DG-DIGIT
9
 kom (2017) 0477 - Ingen titel
1788419_0010.png
the Commission, other EU Agencies, Members of the European Parliament and CSIRTs.
Industry, consumer representatives and civil society also participated. A stakeholder
online survey was distributed to CSIRTs in all 28 Member States and CERT-EU to
gather views and input on ENISA.
The study also carried out a benchmarking excercise of ENISA relative to other EU
Agencies such as CEPOL, BEREC etc
12
, a positioning exercise including some national
cyber security agencies from large Member States
13
and an assessment of strengths,
weaknesses, opportunities and threats (SWOT), validation of collected interview and
survey data with secondary sources and a stakeholder workshop.
The study was supplemented by qualitative analysis carried out by the Commission
through discussions and targeted consultations of key stakeholders. The Commission also
carried out an open public consultation, in accordance with the requirements of the Better
Regulation Guidelines.
The public consultation opened on 18
th
January and closed on 12
th
April 2017. 90
responses were received, from 19 Member States. A copy of the questionnaire used in the
public consultation and a synopsis of the online public consultation is included in Annex
2.
The evaluation faced some
limitations
with regard to data collection, which were
mitigated to the possible extent.
Among those limitations, the key impact indicators (KIIs) of the Agency set in the annual
work programmes and reported upon in the annual activity reports change from one year
to the next, limiting the possibility to implement a comparison of the Agency’s outputs
and results over the entire period of 2013-2016.
With a total of 90 responses, the results of the public consultation cannot be considered
to be completely representative of all stakeholders concerned
14
. To overcome this
weakness, further inputs from stakeholders were collected by the Commission. For
example a roundtable organised by Commission Vice President Ansip in the context of
the review of the cybersecurity strategy and discussion at the Council Horizontal
Working Party on Cybersecurity were used to collect views from Member States.
Another limiting factor with the evaluation methodology was its reliance on stakeholder
contribution. Some of them were part of ENISA's governance and organisation
structures, that is to say staff, members of the Management Board, National Liaison
Officers and Permanent Stakeholder Group representatives. The risk of possible bias was
mitigated through triangulation of the data across different stakeholder groups and across
the data collection tools. For example, the surveys and the interviews which primarily
covered views from ENISA’s staff, management and direct stakeholders were considered
12
13
14
Full list includes Europol
–European
Cybercrime Centre (EC3),EU Agency for Fundamental Rights
(FRA), Office of the Body of European Regulators for Electronic Communications (BEREC),
European Monitoring Centre for Drugs and Drug Addiction (EMCDDA), EU Agency for Law
Enforcement Training (CEPOL) and European Fisheries Control Agency (EFCA).
CERT-EU, Joint Research Centre of the EC (DG-JRC), Europol-EC3, Netherlands National Cyber
Security Centre, Frency National Cybersecurity Agency (ANSSI) and Spanish National Institute for
Cybersecurity (INCIBE).
However, in the public consultation the views of national authorities of 15 Member States are
represented. The private sector is represented by 27 respondents which include eight umbrella
organisations, thus representing a significant number of European enterprises whose activities are
linked with cybersecurity.
10
 kom (2017) 0477 - Ingen titel
1788419_0011.png
against the public consultation results and the workshop where a broader scope of
stakeholders have been reached.
There was limited input from some cyber security stakeholders in industry and in public
authorities who were not directly part of ENISA's outreach
15
. The study responded by
undertaking in-depth interviews with 4 industry representatives, having 9 such interests
attend the stakeholder workshop and taking account of the online public consultation (see
above).
Based on the elements above, this evaluation has been carried out on the basis of the best
available data.
5.
R
ESPONSES TO THE
E
VALUATION
Q
UESTIONS
The responses are grouped around the 5 key evaluation criteria as set out in the Better
Regulation Evaluation Guidelines. A list of the individual evaluation questions is
provided in Annex 3. The summary results are presented in table 1 below.
Table 1 Summary of results of the evaluation according to the criteria
Evaluation criterion
Relevance
Effectiveness
Efficiency
Coherence
EU-added value
Overall assessment
Achieved to a large extent
Partially achieved
Achieved to a large extent
Partially achieved
Partially achieved
Effectiveness
Summary:
ENISA overall met its objectives and implemented its tasks. It made
a contribution to increased NIS in Europe through its main activities (capacity
building, provision of expertise, community building, support to policy). It
showed potential for improvement in relation to each. However, ENISA faced
difficulties to make a big impact in the vast field of NIS. This was also due to the
fact it had fairly limited human and financial resources to meet a broad mandate.
Effectiveness is about whether a particular EU Action, in this case the existence of
ENISA, has met its objectives and supporting tasks as set out in articles 2 and 3 of the
Regulation, i.e. it is outcome focused. In response to the questions about effectiveness,
the evaluation concluded that ENISA met its 5 objectives but not all to the same extent.
In particular, the need to prioritize its activities according to the annual work programme
set by the Management Board, led the Agency to focus its effort more on the needs of
Member States and EU institutions than of the industry. In the same way, ENISA has
achieved to a less extent the objectives linked to the development and maintenance of
expertise and the support to policy development and implementation.
15
Previous evaluations of ENISA in 2014 and in 2015 along with interviews in the current study affirmed that ENISA does not have
sufficient outreach to industry and academia.
11
 kom (2017) 0477 - Ingen titel
1788419_0012.png
The evaluation concludes that ENISA has effectively created strong and trustful
relationships with some of its stakeholders, notably with the Member States and the
CSIRT community.
ENISA was most effective in strengthening capabilities, in particular in providing
support to develop national CSIRTs, as also evidenced by responses to the survey of
CSIRTs (see below), and national cybersecurity strategies
16
. Interventions in the area of
capacity building were perceived as important in particular for less resourced Member
States. Furthermore, the majority of Article 14 requests for assistance concerned Member
States request for training supports
17
. The findings of the evaluation further suggest that
the support provided by ENISA was perceived as complementary to that of other public
interventions, clearly pointing out to a role for ENISA in the area.
Stimulating broad cooperation has been one of the highlights, with stakeholders widely
agreeing on the positive role ENISA plays in bringing people together. The evaluation
also concluded that there was comprehensive implementation of tasks supporting
voluntary cooperation with stakeholders within the Union. In particular the Cyber
Europe exercises
18
, support to the CSIRT community, ENISA's publications and the
European Cyber Security Month initiative are to be considered as key achievements. The
Commission acknowledges that these specific initiatives are highly valued by Member
States, CSIRTs and industry stakeholders.
It can be concluded that ENISA’s activities have made an important
contribution to
enhance cooperation
between Member States and related NIS stakeholders. This is
further evidenced
19
by 82 of 88 respondents to a survey of ENISA direct stakeholders
20
asserting that ENISA had built strong and trustful relationships. This has further been
confirmed by the results of the public consultation, where 79% of respondents affirmed
that ENISA has achieved to some or great extent the objective of supporting cooperation
in the cybersecurity community. Community building has been enhanced across Member
States and in particular the cooperation between CSIRTs has increased.
As stated above, ENISA partially met the objective of providing
expertise.
Its guidelines
and reports are used by many stakeholders but are more appreciated for their availability
and the fact that they are coming from an EU Agency rather than for the outstanding
quality of expertise. Some Member States (particularly those with significant cyber
security capabilities
21
), some EU institutions and industry representatives would expect
more from ENISA in terms of expertise. The findings show that ENISA struggles to hire
experts, which can be explained by a combination of factors: the general difficulties
across the public sector to compete with the private sector when trying to hire highly
specialised experts; the low level of development of ENISA's human resources policies
(HR department formally established only in 2016) and somewhat low level of
In relation to national strategies, since 2013 ENISA has produced good practice guides on how to create
and evaluate a strategy and it has run an experts group with the goal of information exchange on
strategies lifecycle phases. It has furthermore directly supported 5 Member States in creating their
strategy.
17
9 of the 13 Member States who made Article 14 requests for processing in 2015 involved training
requests. Section 1.5.7. of the ENISA Annual Activity Report 2015 refers.
18
ENISA developed a cyber-exercise capability that is able to train the EU cyber response teams to deal
with crisis scenarios. Cyber Europe is the main cyber exercises of the European Union, engaging more
than one thousand participants from the public and the private sector, taking place every 2 years since
2010.
19
Supplemented by 51 of 65 respondents to the online public consultation.
20
Management Board, Permanent Stakeholder Group
21
ANSSI in France employs in excess of 600 people and over 700 staff are estimated in the UK's NCSC. .
12
16
 kom (2017) 0477 - Ingen titel
1788419_0013.png
attractiveness related to ENISA's location, for example linked to difficulties encountered
by spouses to find work. As a consequence, ENISA heavily relied on the procurement of
external expertise, involving 80% of its operational budget, in the implementation of
tasks is a consequence.
All of this has to be balanced with the scope of ENISA's mandate as a horizontal Agency
being very broad covering cyber security matters in energy, transport, finance, health,
water, public administrations etc. The evaluation also found that ENISA provides an
independent source of information and its capacity building role is highly valued by
Member States, particularly by those with limited cyber security capacity.
ENISA has assisted the Member States and the Commission in developing and
implementing the
policies
necessary to meet the legal and regulatory requirements of
NIS.
The evaluation concluded that ENISA enhances cooperation and ensures capacity
building but the development and maintenance of expertise and support of the
development and implementation of policy is somehow limited. The key constraining
factor is limited resources in an Agency with a very broad mandate and high expectations
of its stakeholders. This is not a new situation, as a previous evaluation from 2007
22
recommended the Agency size should be increased to at least 100 staff. Moreover
evaluations and impact assessments from 2009
23
and 2010
24
highlighted concerns about
the Agency's ability to achieve planned impacts.
Efficiency
Summary:
Despite its small budget
–
among the lowest compared to other EU
agencies
–
the Agency has been able to contribute to targeted objectives, showing
overall efficiency in the use of its resources. A location split between Athens and
Heraklion required additional efforts of coordination and generating additional costs
but the move to Athens in 2013 of the core operations department increased the
agency's operational efficiency.
Efficiency is the extent to which outputs are maximised relative to inputs. Efficiency
considers the relationship between the resources consumed by an intervention and the
changes generated by it (which may be positive or negative). The assessment of the
efficiency
of ENISA considers the relationship between the resources used by the Agency
and the output generated by its activities. Since this initiative does not present significant
(direct, indirect, enforcement) regulatory costs, they have not been part of the assessment
of ENISA efficiency.
ENISA ensures full budget execution and it demonstrates high efficiency in the
implementation of its tasks as evidenced, among the others, by the volume of outputs. All
deliverables planned in the annual work programmes were regularly executed. Statistics
also show that downloads of publications have been consistent over the period under
Communication from the Commission to the European Parliament and the Council on the evaluation of
the European Network and Information Security Agency (ENISA) Brussels, 1.6.2007 COM(2007) 285
final
23
Ramboll, Euréval, Matrix insight (2009): Evaluation of the EU decentralised agencies in 2009, Final
Report Volume III
–Agency
level findings
24
SEC(2010)1126
13
22
 kom (2017) 0477 - Ingen titel
1788419_0014.png
review (approx. 900,000 downloads yearly), while the participants to the Cyber Europe
exercise have increased from about 600 in 2014 to about 1000 in 2016.
The current governance structure is also considered as conducive to the efficient
functioning of the Agency by 76 out of 88 respondents to the survey of the inner circle of
ENISA stakeholders (Management Board, Executive Board etc.) but with more
flexibility sought in the planning cycle. ENISA's working practices were found overall
efficient by the external contractor. Some of the tools in place in the Agency are
advanced in comparison to those used by other agencies and favour efficiency, for
example the Agency's workflow paperless management system with the use of e-
signatures.
The budget size forces the Agency to prioritise its work in undertaking the various tasks
set out in its mandate. The Agency develops 50-60 publications every year, with approx.
900,000 downloads. As indicated above the Agency has indeed very limited resources:
one of the smallest annual budgets (circa 10.4m€) and level of human
resources
(presently 84 staff) compared to 40 agencies covered by European Court of Auditors
report
25
on agencies in 2016.
Almost all stakeholders agree that ENISA’s resources are too low to implement all the
given tasks. 38 of 54 respondents to the online public consultation expressed the view
that the size of the Agency was inadequate. Under these conditions, ENISA has to
prioritise its work to ensure resources are spent efficiently. At the same time, the Agency
has to fulfil a number of administrative requirements as set by the Commission as an EU
body. These requirements are the same for all EU agencies but weigh more heavily on
smaller agencies due to significant fixed costs and inability to take advantage of
economies of scale.
With regard to the Human Resources function, in 2015 ENISA spent 2.5% of its budget
on staff recruitment, which is considerably higher than that in comparable Agencies in
the benchmarking exercise. The focus on recruitment reflects the challenges of hiring
and retaining cyber security professionals in challenging marketplace.
The set-up of an office in Athens (in 2013) to host the department of core operations
contributed to efficiency gains, as it improved accessibility of the Agency. However, the
split location of the Agency within Greece, with offices in both Athens and Heraklion
means that ENISA has duplicate office accommodation costs and has to implement
additional efforts to ensure coordination between the offices and bear the extra travel
costs. In this context, its administrative expenditure is higher, at 14.8% of its budget,
relative to other EU Agencies in the benchmarking exercise. In view of this situation no
significant measures were identified to increase efficiency.
25
By comparison, the European Agency for the Operational Management of Large-Scale IT Systems in the
Area of Freedom, Security and Justice - EU-LISA
had a budget of 71.7m€ and 134 staff, Europol had
a budget of 95m€ and 666 staff and the European Agency for the Management of Operational
Cooperation at the External Borders
– FRONTEX had a budget of 143.3m€ and 309 staff.
14
 kom (2017) 0477 - Ingen titel
1788419_0015.png
Relevance
Summary:
In a context of technological developments and evolving threats and of
significant need for increased network and information security (NIS) in the EU,
ENISA's objectives proved to be relevant. In fact, Member States and EU bodies rely
on technical expertise on the evolution of network and information security issues;
capacities need to be built in the Member States to understand and respond to threats,
and stakeholders need to cooperate across thematic fields and across institutions. NIS
continues to be a key political priority of the EU to which ENISA is expected to
respond.
Relevance looks at the relationship between the needs and the problems in the society
and the objectives of a given intervention.
In a context of technological developments and evolving threats and of significant need
for increased network and information security (NIS) in the EU, ENISA's objectives
proved to be relevant. Changes in the activities of ENISA based on the annual work
programme show that the way the objectives have been defined allows for flexibility to
focus on different needs from one year to another.
Most interviewees considered all of ENISA's objectives to be of continued relevance.
Some stakeholders, including Member States wanted ENISA to have a role as an
analytical centre, analyzing threats and incidents in detail to provide more informed
advice while others sought further cooperation with Europol.
Direct stakeholders interviewed asserted that ENISA was well aligned with the priorities
of the stakeholders, particularly with the
NIS Directive and the Commission’s
communication from July 2016
26
. Member States and EU bodies rely on expertise on the
evolution of NIS, capacities need to be built in the Member States to understand and
respond to threats, assistance in the development of new policies on NIS is required and
stakeholders need to cooperate.
Enhancing capabilities can be considered a highly relevant objective, in particular given
the need for an agency to help less resourced Member States. This is also instrumental, in
the context of increased cyber threats, to achieve higher degree of information sharing by
ensuring the counter-parts have to some extent similar capacity levels.
Half of the respondents (31 of 62 responses) to the online public consultation considered
all of ENISA's services, encompassing guidelines and recommendations, training
materials and events, reports, the Cyber Europe Exercise, technical advice, events and
Article 14 requests for assistance to be very relevant, relevant or somewhat relevant.
Community building followed by capacity building were the 2 key themes of most
relevance to the different categories of stakeholders. It emerged that ENISA's key
strength lays in brokering and facilitating cooperation between Member States and in
particular the CSIRTs and is essential for delivery of EU political priorities.
26
European Commission: Communication from the Commission to the European Parliament and the
Council, the European Economic and Social Committee and the Committee of the Regions -
Strengthening Europe's Cyber Resilience System and Fostering a Competitive and Innovative
Cybersecurity Industry COM(2016) 410 final.
15
 kom (2017) 0477 - Ingen titel
1788419_0016.png
In conclusion ENISA's objectives have been so far highly relevant in regard to
cybersecurity matters. Cybersecurity is cross border in nature and impacts on the
resilience of the Digital Single Market. On matters of digital privacy, stakeholders took
the view that ENISA was not best suited to addressing it given the distinct interests and
conflicts that can arise between digital privacy and cybersecurity.
Coherence
Summary:
ENISA’s activities have been generally coherent with the policies and
activities of its stakeholders, at national and EU level, but there is a need for a more
coordinated approach to cybersecurity at EU level. The potential for cooperation
between ENISA and other EU bodies has not been fully utilised. The evolution in the
EU legal and policy landscape make the current mandate less coherent today.
The evaluation concluded that
ENISA’s activities have been generally coherent with the
policies and activities of its stakeholders but there is a need for a more coordinated
approach to cyber security at EU level.
ENISA’s activities have been coherent with the activities of the Member
States. In
particular, while ENISA itself is not a CSIRT, it provides training and networking
support measures for the Member State based CSIRTs. There is a strong coherence
between ENISA’s activities and those of the national CSIRTs with respondents
to the
survey of CSIRTs highlighting ENISA's role in organising workshops and conferences
and facilitating co-operation via the CSIRT Network.
Member States have very diverse stakeholder needs. Some Member States have well
advanced capabilities in cybersecurity (for example encompassing national laws,
strategies, funding, partnerships with the private sector and well-resourced public
authorities with a dedicated cybersecurity remit, proactive CSIRTs) whereas others have
very limited capabilities and resources (for example absence of laws, public bodies,
funding, reactive CSIRTs etc). While there is some duplication of effort with respect to
some Member States’ national cyber security authorities (e.g. in terms of expertise,
extent of preparedness etc) as set out in the positioning exercise, stakeholder interviews
indicated that other Member States are in need of capacity building support and are
reliant on ENISA.
The evaluation also concluded that ENISA activities were highly coherent with the
policies at EU level. In particular, the evaluation found them in line with the objectives of
the EU Cybersecurity Strategy, the NIS Directive and the security provisions of related
policies, such as the ePrivacy Directive and the General Data Protection Regulation.
Some of the activities performed by ENISA in the period under consideration, for
example the pan-European cyber exercises and the European Cybersecurity Month
campaign, were stemming from the Cybersecurity Strategy itself. Furthermore, in the
course of 2016, ENISA has amended its annual work programme for that year in order to
ensure coherence with the provisions of the NIS Directive.
At organisational level, ENISA activities were found to be coherent with those of the
European Commission and other EU bodies. However the potential for cooperation
between ENISA and the European Commission as well as other EU bodies is not fully
utilised. There is general complementarity of the work between ENISA and the Joint
16
 kom (2017) 0477 - Ingen titel
1788419_0017.png
Research Centre (JRC)
27
of the Commission as the organisations vary in the
stakeholders they target and approach issues from a different perspective. However, there
is a risk of duplication of efforts as there is no systematic direct coordination (it mostly
happen through the DG CONNECT).
Good levels of cooperation and coordination have been achieved between ENISA and
EC3: little to no overlap was identified between the two organisations, which overall
seem to cooperate well. However, there is room for more multilateral coordination to
ensure better coherence and complementarity i.e. in the case of ENISA, EC3, CERT-EU
and sectoral EU authorities who are developing competence in cyber, in order to attain
increased NIS in Europe.
In particular, from the evaluation it emerged a risk of overlap between CERT-EU and
ENISA with a risk of duplication of services to national CSIRTs. This would appear to
be linked to direct support and assistance to Member States' CSIRTs and cross-border
operational cooperation. CERT-EU is the computer emergency incident response team of
the EU institutions and as a CSIRT it has peer to peer relationships with the Member
State CSIRTs. ENISA is not a CSIRT but provides capacity building supports for
CSIRTs.
EU Added Value
Summary:
EU-added
value: ENISA’s added value lied primarily in the Agency’s
ability to enhance cooperation, mainly between Member States but also with related
NIS communities. There is no other actor at EU level that supports the cooperation of
the same variety of stakeholders on NIS. The added value provided by the agency
varied according to the diverging needs and resources of its stakeholders (e.g. big
versus small Member States; Member States versus industry) and the need for the
agency to prioritize its activities according to the work programme.
EU added value is about the benefits of initiatives taken at European level relative to
solely national domestic approaches.
The evaluation found that ENISA added value primarily in enhancing cooperation,
mainly between Member States but also with related NIS communities. This has been a
key achievement in an area with a strong cross-border dimension. ENISA's activities
could have been to some extent replaced nationally or through regional/bilateral
cooperation but it would have been difficult to ensure the same community building
across the Member States without a decentralised EU agency for cybersecurity. Prior to
the establishment of ENISA, key initiatives such as the Cyber Europe exercises and
CSIRT capacity and trust building with Member States did not exist. In absence of an EU
agency, there may have been a reduced focus on cyber security by those Member States
with fewer resources who, to date, have made very limited investments in cyber security
capabilities.
27
See
https://ec.europa.eu/jrc/en/research-topic/cybersecurity
17
 kom (2017) 0477 - Ingen titel
1788419_0018.png
Highlights of ENISA's added value
Certification -
as a neutral third party ENISA supports work related to establishing a possible EU ICT
certification framework. It conducted consultation with more than 18 Member States, carried out an
EU-wide survey on set of policy options and an analysis of EU certification laboratories landscape;
ENISA also elaborated sector specific needs for ICT security certification (e.g. semiconductor
industry). ENISA is regularly invited to participate in the Management Team meetings of the SOGIS
group to offer policy advice to support the SOGIS Mutual Recognition Agreement (MRA).
ENISA Threat Landscape
–an
annual report providing an overview of threats, current and emerging
trends to inform threat assessments and policy making became a reference point for cybersecurity
community cited in a large number of prestigious sources and used as education material in universities
and industry training courses.
ENISA has added value as facilitator for the essential trust building and cooperation
between the various stakeholders (i.e. Member States, industry, users etc) in cyber
security, through joint exercises such as Cyber Europe, is role as networking facilitator
for Member State CSIRTs, provision of independent and neutral guidance and advice, the
supports for policy implementation. These activities have positively impacted on the
trust and confidence in the Digital Single Market. The evaluation found that while all
categories from stakeholders
–
Member States, industry, research community
–
benefitted by an action at EU level, their needs have not equally been met due to the
small size of the Agency compared to the challenge of bringing added value in such a
vast field like cybersecurity.
Highlights of ENISA's added value
CSIRTs support:
ENISA provides continuous capacity building support (2014 - 2017: 114 training
courses, assistance under the Article 14 provided over 23 times). ENISA is the Secretariat for CSIRT
Network established by the NIS Directive.
Cyber Europe Exercises
–
this largest and most comprehensive bi-annual EU cyber-security exercise
has given the opportunity to around 4000 cybersecurity experts from over 2000 different organisations
to be trained to deal with difficult and complex cybersecurity incidents. The level of satisfaction with
the exercise is high to very high for over 99,9% of the participants.
European Cybersecurity Month (ECSM)
–
the cybersecurity awareness raising campaign running
each year for the entire month of October mobilised so far over 30 countries all over Europe to
organise activities, conduct media and social media relations.
The evaluation noted that most stakeholders think that in the future ENISA could take on
a more important role in the EU cyber security landscape. This potential of the Agency
would be lost in case of a discontinuation. According to some of the interviewees, the
division of ENISA’s activities across different organisations could lead to further
fragmentation in the cyber security field in Europe as sector specific cyber security 'silos'
of competence emerge. Such an approach would be at odds with increased importance of
cyber security on the EU policy development agenda.
6.
C
ONCLUSIONS
ENISA was entrusted with a broad mandate and its objectives proved to remain
relevant
today. In a context of technological developments and evolving threats and of significant
need for increased network and information security (NIS) in the EU, there is a need for
technical expertise on the evolution of network and information security issues.
18
 kom (2017) 0477 - Ingen titel
1788419_0019.png
Capacities need to be built in the Member States to understand and respond to threats,
and stakeholders need to cooperate across thematic fields and across institutions.
Despite its small budget, the Agency has been operationally
efficient
in the use of its
resources and implementation of its tasks. The location split between Athens and
Heraklion, however, generated administrative costs.
In terms of
effectiveness,
ENISA partially met its objectives. The agency successfully
contributed to increased NIS in Europe by offering capacity building in 28 Member
States
28
, enhancing cooperation between Member States and NIS stakeholders; provision
of expertise, community building and support to policy. Overall, ENISA diligently
focused on the implementation of its work programme and acted as trusted partner for its
stakeholders in a field which only recently has been recognised to have such strong
cross-border relevance. For long, cybersecurity has primarily been seen as an area of
national competence, where EU intervention was only partially accepted.
ENISA managed to make an impact, at least to some extent, in the vast field of NIS but it
has not fully succeeded in developing a strong brand name and gaining sufficient
visibility to become recognised as a "the" centre of expertise in Europe. The explanation
for this lies with the broad mandate of ENISA, which was not met with proportionally
big resources. Furthermore, ENISA remains the only EU agency with a fixed-term
mandate which limits its ability to develop a long term vision and support its
stakeholders in a sustainable manner. This is also in contrast with the provisions of the
NIS Directive, which entrust ENISA with tasks with no end date. Finally, the assessment
found that this partial effectiveness can partly be explained to the high reliance on
external expertise over in-house expertise, and the difficulties in recruiting and retaining
specialised staff.
ENISA’s
added value
lies primarily in the Agency’s ability to enhance cooperation,
mainly between Member States but also with related NIS communities (in particular
between CSIRTs). There is no other actor at EU level that supports the cooperation of the
same variety of stakeholders on NIS. However, due to the need to strictly prioritize its
activities, ENISA’s work programme is
mostly guided by the needs of Member States.
As a result, it does not sufficiently address the needs of other stakeholders, in particular
the industry. It also made the Agency reactive to fulfilling the needs of its key
stakeholders, preventing it from achieving a bigger impact. Therefore, the added value
provided by the Agency varied according to the diverging needs of its stakeholders and
the extent to which the Agency was able to respond to them (e.g. big versus small
Member States; Member States versus industry).
Recommendations
The cybersecurity threat landscape is evolving fast with new threats emerging as Europe
becomes ever more reliant on digital infrastructure and services through not only
28
Respondents to the public consultation were asked to comment on what they perceived as ENISA’s main achievements over 2013-
2016. Respondents from all groups (in total 55, including 13 from national authorities, 20 from private sector and 22 from
"other") perceived the following as ENISA’s main achievements: !) The coordination of the Cyber Europe exercises; 2) The
provision of support to CERTs/CSIRTs through training and workshops fostering coordination
and exchange; 3) ENISA’s
publications (guidelines and recommendations, threat landscape reports, strategies for incident reporting and crisis management
etc.) that were considered as useful to create and update national security frameworks, as well as for reference to policy makers
and cyber practitioners; 4) Assisting with the promotion of the NIS Directive; 5) Efforts to increase awareness on cybersecurity
via the cybersecurity month.
19
 kom (2017) 0477 - Ingen titel
connected devices but now omnipresent connectivity. The Internet of Things creates new
opportunities related to energy efficiency, environmental protection, connected mobility,
real time health monitoring and smart and seamless financial transactions in the digital
economy and society. However in tandem with these business drivers are new
vulnerabilities and exploits enabling compromised devices to disrupt the Digital Single
Market.
Europe needs a focal point to address these new threats which are horizontal in nature
impacting on multiple industrial sectors. The findings of this evaluation suggest that
there could be a need for an EU Agency organised on a cross sectoral/horizontal basis
with a strong mandate. The evaluation also found that there is also a need for cooperation
and coordination across different stakeholders. The need for a coordinating entity at EU
level to facilitate information flows, minimise gaps and avoid overlapping of roles and
responsibilities becomes ever more acute. A decentralised EU agency and a neutral
broker, could ensure a coordinated approach to cyber threats in the EU.
20
 kom (2017) 0477 - Ingen titel
List of Annexes
Annex 1: Procedural Information
Annex 2: Stakeholder Consultation
–
See Annex 2 of Impact Assessment.
Annex 3: Evaluation questions
Annex 4: Contractor's Report
–
See Annex 6 of the Impact Assessment.
21
 kom (2017) 0477 - Ingen titel
1788419_0022.png
Annex 1: Procedural Information
Lead DG, Decide Planning
This evaluation Staff Working Document was prepared by Directorate H "Digital Society, Trust
and Cybersecurity" of Directorate General "Communications Networks, Content and
Technology".
The Agenda Planning reference of the initiative "Evaluation of the European Union Agency for
Network and Information Security (ENISA), is 2017/CNECT/002.
Organisation and Timing
Several other services of the Commission with a policy interest in the assessment of the initiative
have been associated in the development of this analysis.
An Inter-Service Steering Group (ISG), consisting of representatives from various Directorates-
General of the Commission and the European External Action Service (EEAS), was set up in
2016 to steer the evaluation of ENISA during all key phases.
The ISG on the evaluation of ENISA met twice, on 24 June and 9 December 2016. DG CNECT,
DG HOME, DG JRC, DG JUST, EEAS, and Secretariat General (SG) participated in the
meetings. The Steering Group approved the evaluation roadmap, the terms of reference of the
external study and the questionnaire of the public consultation. Further consultations, in
particular with regard to the study, were carried out by written procedure.
The ISG was further expanded in the context of the review of ENISA and the set-up of an EU
cybersecurity certification and labelling framework.
The Commission Directorate General for Communications Networks, Content & Technology
(DG-CNECT) concluded a tender with a consortium led by CARSA
29
, to provide an
independent evaluation of ENISA in November 2016.
Evidence, Sources and Quality
The Commission gathered qualitative and quantitative evidence from various sources (a
summary of which is attached to Annex 2 of the Impact Assessment report):
(1) Four weeks online public consultations regarding the evaluation and review of
ENISA (19 January- 12 April 2017);
(2) A stakeholder workshops with Member States, industry and academia
representatives;
(3) A study conducted by the above mentioned contractor;
(4) Fifty expert interviews conducted by the external contractor;
(5) A survey on the ENISA review to the Computer Security Incident Response
Teams Network;
29
Consortium includes CARSA (Consultores de Automatización y Robótica S.A. - lead partner), Logotech
(partner), Ramboll Management Consulting S/A (partner), AIT Austrian Institute of Technology
GmbH (partner), ZSI
–
Zentrum für Soziale Innovation (partner) and Agilis S.A. (partner).
22
 kom (2017) 0477 - Ingen titel
(6) A survey to ENISA Management Board, Executive Board, Permanent
Stakeholder Group, and ENISA staff;
(7) Direct dialogue with stakeholders;
(8) A roundtable with European Commission Vice-President for the Digital Single
Market, Andrus Ansip, on 25 April 2017; and
(9) Desk research and literature review done in-house by DG CONNECT.
With regard to the quality of the evidence, the following points must be noted:
There are limitations with regard to gathering data. For instance, the public
consultation on the ENISA review received 90 submissions. With a total of 90
responses, the results of the public consultation cannot be considered to be fully
representative of all stakeholders concerned. However, the views of national
authorities of 15 Member States (including the position paper provided by
France) are represented. The private sector is represented by 27 respondents
which include eight umbrella organisations, thus representing a significant
number of European enterprises whose activities are linked with cybersecurity;
As regards the survey on ENISA, which was addressed to CERTs and CSIRTs,
the answers in both surveys were anonymous. Thus, it is not possible to know
whether some of the respondents might have started the survey and only partially
completed and might have reopened it using a different browser or device and
completed the survey then. This would result in answers that are double counted.
23
 kom (2017) 0477 - Ingen titel
Annex 3: Evaluation questions
The evaluation roadmap set out the key questions to be addressed by the evaluation.
Effectiveness:
To what extent has the Agency achieved its objectives and implemented the tasks
set out in its mandate? What are the key factors influencing/restricting progress
and how do they link to the agency (if at all)?
What have been the benefits of acting at Agency level both from the operational
and strategic perspective?
To what extent has ENISA contributed to the overall EU goal of increasing
network and information security in Europe?
How appropriate is the balance of activities in relation to different cybersecurity
and digital privacy topics considering the evolving needs of the main
stakeholders?
To what extent ENISA became an EU-wide centre of expertise and a reference
point for EU institutions, Members States and the wider stakeholders community,
in providing guidance, advice and assistance on issues related to network and
information security?
How effectively the Agency manages to set its work priorities?
How effectively does the Agency tackle important upcoming, unplanned issues
deriving by demands of its constituencies and/or EU policy priorities?
Does the Agency consistently perform the same tasks with the same quality level
over the time?
How does ENISA compare to the other EU and national bodies offering similar
services in relation to their capability to satisfy the cybersecurity and digital
privacy needs of ENISA's constituency?
To what extent has ENISA been more effective in achieving its results compared
to other past, existing or alternative national or EU level arrangements?
How do the current governance, the internal organisational structure and the
human resources policies and practices of ENISA contribute to efficiencies and
effectiveness in the work of the agency?
How effective has ENISA been in building a strong and trustful relationship with
its stakeholders when executing its mandate?
What is the impact of the current arrangements related to the location of ENISA's
offices on the overall capability of the Agency of meeting its objectives?
24
 kom (2017) 0477 - Ingen titel
Efficiency:
To what extent has ENISA been efficient in implementing the tasks set out in its
mandate as laid down in its Regulation? To assess this question, elements relating
to internal structure, operation, programming of activities and resources,
accountability and controls, etc. will be analysed.
Were the annual budgets of the Agency implemented in an efficient way with a
view on achieved results?
Have the resources allocated to the agency been sufficient for the pursuing of its
tasks (input/output analysis)?
To what extent are the organisational solutions and procedures of ENISA
adequate to the work entrusted to it and to the actual workload? Is the planning
cycle of the agency (work programme and budget) in line with the objective of
achieving efficient results?
To what extent have ENISA's governance, organisational structure, locations and
operations as set in its Regulation and the arrangements related to the location of
its offices been conducive to efficiency and to achieving economies of scale?
To what extent are the internal mechanisms for programming, monitoring,
reporting and evaluating ENISA adequate for ensuring accountability and
appropriate assessment of the overall performance of the Agency while
minimising the administrative burden of the Agency and its stakeholders
(established procedures, layers of hierarchy, division of work between teams or
units, IT systems, etc)?
To what extent has ENISA succeeded in building up the in-house capacities for
handling various tasks entrusted to it? Are the "make or buy" choices made
according to efficiency criteria?
To what extent and how have external factors influenced the efficiency of
ENISA?
Coherence:
To what extent is ENISA acting in cooperation with the European Commission
and other EU bodies, to ensure complementarity and avoid duplication of efforts?
To what extent is ENISA acting in cooperation with the Member States to ensure
complementarity and avoid duplication of efforts?
To what extent are ENISA activities coherent with the strategy documents
adopted in this policy field?
Are the procedures put in place effective to ensure that ENISA's cooperation
activities are coherent with the policies and activities of its stakeholders?
What are the risks/sources of overlaps/conflict of interests?
25
 kom (2017) 0477 - Ingen titel
Relevance and EU added value:
What would be the most likely consequences at the EU level of stopping ENISA?
How could ENISA increase its added value and its contribution towards the EU,
the Member States and the private sector in the future, using the capabilities and
competences already in place?
How far are the Agency's tasks and resources aligned with key EU political
priorities?
Which Agency tasks are absolutely essential to deliver on these priorities?
Which Agency tasks are necessary to continue implementing existing and
evolving obligations under the Treaties and EU legislative framework?
Are there some Agency tasks that have become redundant / negative priorities? If
so, which are they?
Are the objectives set out in the mandate of ENISA still appropriate given the
current cybersecurity and digital privacy needs, regulatory and policy framework
and needs?
Have some of the initially non-core activities of the Agency become part of its
core-business? What was the rationale in such cases?
What would be the most likely consequences at the EU level of stopping ENISA's
activities?
Other questions:
Does the new scenario with increased frequency, sophistication and potential
impact of cyber-threat trigger new needs from ENISA's constituency? To what
extent could ENISA’s current mandate, tasks and/or capabilities address these
needs?
How does the new policy and regulatory landscape, having regard to the recently
adopted Network and Information Security Directive and the priorities set by the
Digital Single Market Strategy, impact on ENISA's activities?
What are the main strengths and weaknesses of ENISA, within its current
mandate and organisational set-up and capacity, in taking up the new challenges?
Is a fixed-term mandate coherent with the new challenges and tasks ENISA will
have to take on?
Which are the concrete needs and opportunities for further increased practical
cooperation with Member States and EU bodies?
26
 kom (2017) 0477 - Ingen titel
Which are the concrete needs and opportunities for cooperation and synergies
with international bodies working in adjacent fields, like the NATO Cooperative
Cyber Defence Centre of Excellence?
How could ENISA’s mission, tasks, working practices or activities be further
developed in order to better respond to the new cybersecurity landscape?
What would be the financial implications associated to each of the possible
options for modifying the mandate as they emerge from the evaluation?
27
 kom (2017) 0477 - Ingen titel
Annex4 : Contractor's Reports
(See annex 5 of the Impact Assessment Report).
"Study
on the Evaluation of the European Union Agency for Network and Information
Security"
by Ramboll and Carsa for the European Commission Directorate-General of
Communications Networks, Content & Technology
"Evaluation
of ENISA Public consultation on the evaluation and review of the European
Union Agency for Network and Information Security (ENISA) Synopsis report"
A study
prepared for the European Commission DG Communications Networks, Content &
Technology by Ramboll and Carsa.
28