kom (2018) 0050 - Ingen titel

Europaudvalget 2018
KOM (2018) 0050
Offentligt

EUROPEAN

COMMISSION

Brussels, 31.1.2018

SWD(2018) 40 final

COMMISSION STAFF WORKING DOCUMENT

Mid-term evaluation of the Erasmus+ programme (2014-2020)

Accompanying the document

REPORT FROM THE COMMISION TO THE EUROPEAN PARLIAMENT, THE

COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE

COMMITTEE OF REGIONS

Mid-term evaluation of the Erasmus+ programme (2014-2020)

{COM(2018) 50 final}

kom (2018) 0050 - Ingen titel

Table of Contents

INTRODUCTION ............................................................................................................................

BACKGROUND TO ERASMUS+ AND ITS PREDECESSOR PROGRAMMES ........................

IMPLEMENTATION OF ERASMUS+ UNTIL 2016 ..................................................................

METHOD OF THE EVALUATION.............................................................................................

FINDINGS.....................................................................................................................................

5.1. Effectiveness

............................................................................................... 20

5.1.1.Outcomes for learners and practinioners………………………………........20

5.1.2.Outcomes for organisations and systems……………………………….......31

5.1.3. Transversal questions about effectiveness………………………………….40

5.2. Relevance

.................................................................................................... 44

5.3. Coherence

................................................................................................... 54

5.4. Efficiency and simplification

..................................................................... 63

5.5. European added value

............................................................................... 72

CONCLUSIONS ...........................................................................................................................

ANNEX 1: PROCEDURAL INFORMATION ............................................................... 78

ANNEX 2: STAKEHOLDER CONSULTATION .......................................................... 82

ANNEX 3: METHODS AND ANALYTICAL MODELS............................................... 91

ANNEX 4: EVALUATION: CRITERIA AND QUESTIONS ...................................... 103

ANNEX 5A: INTERVENTION LOGIC OF ERASMUS+ AND ITS PREDECESSOR

PROGRAMMES .......................................................................................... 107

ANNEX 5B: FACTSHEET ON ERASMUS+................................................................ 113

ANNEX 5C: SPECIFIC OBJECTIVES OF ERASMUS+ ............................................. 115

ANNEX 5D: FACTSHEETS ON PREDECESSOR PROGRAMMES ......................... 117

ANNEX 5E: LEGAL BASIS INDICATORS ................................................................. 127

ANNEX 6: GLOSSARY ................................................................................................ 135

kom (2018) 0050 - Ingen titel

Executive Summary

The 2014-2020 Erasmus+ programme and its predecessor programmes (2007-2013)

have provided millions of people in Europe and beyond with opportunities to learn,

volunteer or teach abroad. It is one of the EU's best-known successes. Its impact goes

beyond individuals, as it also has a positive impact on education, training, youth and

sport organisations or related systems and policies in developing cross-country

cooperation.

This mid-term evaluation of Erasmus+ looks at the current programme until the end of

2016. It also includes an evaluation of the long-term effects of the predecessor

programmes. It is based on a very reliable methodology (see 5.3 for its strengths and

limitations) and over a million responses from interested parties. It shows that at mid-

term, the programme is on track to

achieve or exceed the vast majority of the targets

set in the Erasmus+ Regulation (EU) No 1288/2013.

Effectiveness and European added value

1) The evaluation finds that the

Erasmus+ programme is highly valued

by the general

public as well as by its stakeholders. This finding is linked to the available evidence that

the programmes under evaluation – both under their intra-EU and external dimensions -

achieve a broad range of concrete and positive impacts on their beneficiaries:

learners

(students, apprentices, volunteers, young people, etc.);

practitioners

(teachers, trainers,

youth workers, staff, etc.);

participating organisations

(schools, universities, youth and

sport organisations, providers of vocational education and training and of adult

education, etc.).

2) Though less visible, the evaluation confirms the

systemic effect

of the evaluated

programmes

on education, training, youth and sport policies and systems,

directly

through the critical mass reached at least in the higher education sector or indirectly in

funding policy cooperation (Open Method of Coordination). This systemic effect goes

together with partial progress made in the area of

dissemination

of results of the

programme. However, the evidence of the exploitation of project results by policy

makers and the effective engagement of the latter when they are not included in the

project itself is not always clear. In this sense, the evaluation found that the dissemination

of results is one of the aspects of Erasmus+ where there is room for further improvement.

The evaluation also noted that the impact of funded projects on national systems could be

more systematic if there were more cooperation projects fit for mainstreaming, focussed

on fewer priorities at EU level and further efforts made for mainstreaming these at

national level. The systemic impact of the actions of a new kind introduced only in 2014

(KA3, alliances, etc) is meant to be evaluated at final stage, after 2020.

3) The evaluation considers there is potential for

better definition of actions

maximise the programme's impact in Adult Education, sport, Jean Monnet activities and

the Student Loan Guarantee Facility. Considering the funding available at EU level,

evidence shows that: the contribution in the

adult learning

sector is diluted due to the

wide size of the target population and the fragmented and diverse nature of the sector; in

the field of

sport,

resources should not be spread too thinly to have meaningful result;

the

Student Loan Guarantee Facility

has not yet lived up to volume expectations partly

due to delays in its launch. Regarding

Jean Monnet activities,

there is a need to

strengthen the youngest generation's (notably school pupils') awareness and

understanding of European integration.

4) In light of the impacts achieved in all other areas, the evaluation highlights the

strong

European added value of Erasmus+ and its predecessor programmes,

compared to

kom (2018) 0050 - Ingen titel

what could have been achieved with similar programmes focused on separate

geographical areas. This added value is the result of greater impact due to a much higher

volume and wider scope of funded activities, fairer access to learning mobility, deeper

EU integration, a clearer international dimension and mainstreamed best practices.

However, the programme's potential for stimulating learning innovation seems to be

lagging behind and could be further exploited in the future.

5) The evaluation concludes that,

in the absence of Erasmus+ and its predecessors,

there would be clear negative effects

on learning mobility abroad, transnational

cooperation among organisations, including with partner countries, integration between

European countries as well as the attitude of participants towards the EU.

Erasmus+ is more coherent, more relevant and only partly more efficient than its

predecessors.

1) The main structural change of the Erasmus+ programme, compared to its

predecessors, is its

integrated nature

which has contributed to enhance the programme's

internal coherence.

Erasmus+ covers learning in all its contexts

– whether formal or

non-formal, including youth work and sport – and

at all levels of the lifelong learning

continuum:

from early childhood education and schools, vocational education and

training (VET), adult learning to higher education, including its international dimension.

The evaluation highlighted the positive effects of this integrated approach underpinned

by the lifelong learning logic.

The evaluation highlights the following positive consequences of the Erasmus+ design,

which has resulted in:



a sharp increase of

cross-sectoral cooperation

between education and training

sectors, youth and sport;



an improved

geographical balance

with small countries and countries from

Central and Eastern Europe being better integrated;



simplified architecture

in three key actions;



single brand name

which has contributed to the programme's

increased

visibility

and a progressively strong adherence by the sectors covered.

2) In terms of

policy relevance,

the evaluation shows that stakeholders see Erasmus+ as

being more

clearly aligned with EU policies and priorities

than predecessor

programmes. However, a majority of programme countries call for more flexibility at

national level and the evaluation has shown that to maximize the impact of the

programme priorities could be reduced and better focused.

3) The evaluation found a

high complementary between

Erasmus+ and other EU

policies and programmes relevant to education, training, youth and sport (e.g. Europeran

Social Fund, Horizon 2020). Although the level of synergies differs, it is notable that the

evaluation detected

very few overlaps.

4) When it comes to

budget,

the evaluation concludes that

more is needed for the

programme to reach a critical mass

in sectors other than higher education. The demand

largely exceeds the funding available including in higher education. The evaluation,

including the public consultation, suggests that a further reconsideration of the

programme's financial envelope is needed. Without prejudice to negotiations on the next

Multi Financial Framework, the evaluation shows that the budget could be differently

shared between the programme sectors, in particular at the advantage of sectors showing

the highest performance, but which have received relatively less funding up until now,

such as school education and vocational education and training.

kom (2018) 0050 - Ingen titel

5) The evaluation shows that Erasmus+

mobility actions are clearly cost-effective,

especially learners' mobility (with a cost for the EU of 15€ per day/learner). Cost-

effectiveness of other actions remains harder to quantify. The

management costs

(6% of

Erasmus+ administrative and operational budget)

are deemed reasonable,

especially

when compared to similar national schemes (14% in average). The overall efficiency

stemming from the merge of 7 predecessor programmes is not yet clear. More efficiency

gains are expected to materialise during the growing phase foreseen by the budget profile

of Erasmus+ until 2020. This will have to be evaluated at final stage.

6) In terms of

programme management,

the division of responsibilities, as inherited

from predecessor programmes, between the Commission, National Authorities, National

Agencies and EACEA, is overall clear and fit for purpose. A majority of programme

countries wish more flexibility in implementing the budget. A truly performance-based

approach has been adopted, though some indicators need to be fine-tuned and less

information collected while being better exploited.

7) However, there is clearly a

repeated call for further simplification.

Following a

difficult transitional period/learning process, there is broad agreement that Erasmus+ has

brought major improvements (e.g. simplified grants, digitalisation, VET Charter,

linguistic support, etc.) but that procedures of application and reporting could be further

simplified to reduce the administrative burden on beneficiaries. Applicants for small

projects are too often expected to meet the same requirements as applicants for large

ones. IT tools are not inter-operable and enough user-friendly. The application process

could more clearly focus on those criteria that matter most for effectiveness. A specific

challenge is to improve the efficiency of the decentralised international credit mobility

action which includes more than 12 different budget envelopes for partner countries.

Overall, the evaluation found that

all

the evaluated predecessor programmes were/are

highly effective,

whereas Erasmus+ is

more coherent,

relevant

and only

partly

more efficient

than its predecessors.

kom (2018) 0050 - Ingen titel

NTRODUCTION

The Union

Programme for Education, Training, Youth and Sport

for the period 2014-

2020 'Erasmus+', and its predecessor programmes for the years 2007-2013, support learning

mobility of individuals worldwide, transnational cooperation between organisations and

promote Member States' reforms in the education, training, youth and sport fields. This is how

the EU invests in people with a view to unlocking individuals' potential regardless of age or

background, in support of Member States' efforts to develop human resources in Europe and

beyond.

Erasmus+ contributes to the objectives of the Europe 2020

Strategy

and more specifically to

the Strategic Framework for European Cooperation in Education and Training (ET 2020)

, the

European Youth Strategy

and the EU policy in the field of sport

. Erasmus+ also contributes

to the EU's more recent overall political objectives, such as the European Pillar of Social

Rights and the EU Global Strategy on Foreign and Security Policy.

The Erasmus+ programme (2014-2020)

has integrated all previously existing EU

programmes

in the domains of Education, Training, Youth and Sport, and includes an

international dimension that is funded by different external action instruments. In the fields of

Education, Training and Youth, the programme pursues its objectives through three types of

actions: learning mobility abroad, transnational cooperation projects and policy support.

Separately, the Jean Monnet activities promote teaching and research on the European

integration

, while the Erasmus+ supports transnational cooperation activities in the field of

sport, focusing in particular on grassroots level sport.

The Regulation establishing the Erasmus+ Programme stipulates that a mid-term evaluation

report accompanied, if appropriate, by a legislative proposal to amend the Regulation, shall be

submitted by the Commission to the European Parliament, the Council, the European

Economic and Social Committee and the Committee of the Regions by 31 December 2017.

Furthermore, the mid-term evaluation shall include the ex-post evaluation of the Erasmus+

predecessor programmes over the period 2007-2013 i.e. Lifelong Learning, Youth in Action,

Erasmus Mundus, ALFA, Tempus, Edulink and sport preparatory actions, taking into account

their long-term results and impact. Therefore, this report covers actions for the

period 2007-

2016 in all programme countries

(Member States of the European Union, the Former

Yugoslav Republic of Macedonia, Iceland, Liechtenstein, Norway, Turkey) and

partner

countries

(neighbouring the European Union and other partner countries). The

baseline for

this evaluation is the period 2007-2013 unless otherwise specified.

In total, the period

under evaluation corresponds to a total budget of

over EUR 15 billion.

The objectives of the mid-term evaluation are to assess

five evaluation criteria

a) the

effectiveness of the measures taken to achieve the Erasmus+ programme's objectives,

including the contribution made to the realisation of the Europe 2020 strategy; b) the

continued relevance of all of its objectives; c) the Programme's internal and external

http://ec.europa.eu/europe2020/index_en.htm

http://ec.europa.eu/education/policy/strategic-framework/index_en.htm

http://ec.europa.eu/youth/policy/youth_strategy/index_en.htm

http://ec.europa.eu/sport/policy/index_en.htm

Regulation (EU) No 1288/2013

of the European Parliament and of the Council of 11 December 2013 establishing

'Erasmus+': the Union programme for education, training, youth and sport and repealing Decisions No 1719/2006/EC, No

1720/2006/EC and No 1298/2008/EC

http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32013R1288

Article 10 of the Regulation (EU) No 1288/2013; see ICF, stand-alone report on

Jean Monnet actions

("ICF/JMO"

hereinafter), ICF, vol. 2. The relevance of the institutions designated in Article 10(c) of Regulation (EU) No 1288/2013 and

pursuing an aim of European interest and funded under Jean Monnet activities (about €29 million in 2016) has not been

evaluated due to its discretionary nature.

kom (2018) 0050 - Ingen titel

coherence; d) the efficiency of the Programme and the scope for simplification; and lastly e)

its European added value. The results of the evaluation will be used to improve the

implementation of the current programme until 2020 where possible. They will also be used

to feed into the impact assessment for its successor programme.

The mid-term evaluation assesses the performance of the different (Key)

actions

included in

Erasmus+ and evaluates to which extent findings differ across

fields and sectors

included in

the programme and across the different

target levels

(individuals, organisations and systems).

It also assesses the extent to which the programmes have contributed to policy development

and implementation in the participating countries. The evaluation does not cover actions that

were discontinued during the previous programming period (2007-2013)

as these were not to

be evaluated, under the Terms of Reference. However, feedback from stakeholders on the

interruption of actions discontinued with Erasmus+ was collected and analysed.

This Staff Working Document on the mid-term evaluation of Erasmus+ and predecessor

programmes has been carried out according to a

Roadmap

published in January 2016

. It

draws mainly, among other sources

, on the National Reports submitted by the programme

countries in accordance with Article 21(4) of Regulation (EU) No 1288/2013, previous

evaluations and other studies

and the final deliverables prepared by

ICF Consulting

Services

Ltd, (hereinafter ICF), under contract to European Commission (DG Education,

Youth, Sport and Culture)

. The final report delivered by ICF provides answers to all

evaluation questions

defined in the Terms of Reference and related to the aforementioned

five evaluation criteria. The final report of the contractor contains also recommendations

addressed to the Commission.

The

actions

of the programme are defined in Articles 6-10 (education and training), 12-15 (youth) and 17 (sport) of

Regulation (EU) No 1288/2013. They include Key Actions 1 (mobility for learners and practitioners), 2 (cooperation among

organisations) and 3 (policy support), Jean Monnet activities, sport activities, and their respective specific actions.

The

fields

of the programme are defined in Article 1.3 of the legal base. They consist of education and training (including

five

sectors:

school education, vocational education and training, higher education (HE), international higher education and

adult learning), youth, and sport.

The external cooperation agreements in HE, training and youth with the USA and Canada were discontinued in 2011

following unilateral decisions by these two countries to interrupt co-funding in the aftermath of the economic crisis.

http://ec.europa.eu/smart-regulation/roadmaps/docs/2015_eac_014_evaluation_erasmus_en.pdf

See annex 3 for sources, data collection and method of analysis

The

Evaluation of the EU Development Co-operation Support to Higher Education in Partner Countries (2007-2014),

August 2017, Particip, LeA, ECDPM, Ecorys for the Directorate-General for International Cooperation and Development

(European Commission) includes the ex post evaluation of the four predecessor programmes in the field of international

higher education.

See annex 1 for evaluation process

See annex 4 for evaluation questions

kom (2018) 0050 - Ingen titel

ACKGROUND TO

RASMUS

AND ITS PREDECESSOR

ROGRAMMES

Erasmus+ is the

EU Programme in the field of education, training, youth and sport,

with

a budget of

EUR 16.45 billion

for the period

2014-2020

. It provides opportunities for

people of all ages (university students but also school pupils, trainees, apprentices, etc.) to

study, be trained, volunteer and learn in other countries. It also fosters the professional

development of practitioners and supports cooperation on tangible results, networking and

share of knowledge among organisation and institutions in the fields covered by the

programme.

Through cooperation in formal, informal and non-formal learning, the Programme aims to

address the following challenges: economic recovery and high youth unemployment; skills'

mismatches, low employability and education poverty; global competition for talents;

Information and Communication Technology potential and digital divide; social exclusion

and intolerance; lack of trust in the EU and low participation in democratic life; threats to the

integrity of sport and, more generally, to common European values.

The

general objectives

of Erasmus+ are to contribute to the

Europe 2020

strategy for

growth and jobs

, including the headline education targets

, as well as the strategic

framework for European cooperation in education and training ('ET

2020'),

including related

benchmarks. Erasmus+ also aims to contribute to achieving the objectives of the

EU Youth

Strategy

and the EU

Work Plan for Sport,

to promote the sustainable development of

partner countries in the field of higher education and youth, as well as to foster European

values

Specific objectives

tackled by the programme include the improvement of the level of key

competences and skills, with particular regard to their relevance for the labour market and

their contribution to a cohesive society; the promotion of solidarity and participation in

democratic life in Europe and the labour market; the improvement of quality, innovation,

excellence (including in European studies) and internationalisation at the level of

organisations and practitioners; support to the modernisation of education and training

systems, in particular through evidence-based policy cooperation; the enhancement of the

European/international dimension of its sectors, including with partner countries in

complementarity with the Union's external action; the promotion of the Union's linguistic

diversity and intercultural awareness; cross-border threats to the integrity of sport; support to

good governance in sport and dual careers of athletes; and the promotion of voluntary

activities in sport.

Erasmus+ results from the integration of the following

predecessor European

interventions

implemented during the period 2007-2013:

Lifelong Learning

(LLP)

Youth

The Programme has an overall indicative financial envelope of 14.774 billion EUR under Heading 1 "Smart and Inclusive

Growth" and of 1.680 billion EUR under Heading 4 "Global Europe" of the EU Budget for the seven years (2014-2020), EU-

appropriations,

well

million

EUR

from

the

European

Development

Fund

https://ec.europa.eu/programmes/erasmus-plus/programme-guide/part-a/what-is-the-budget_en

Article 4 of the Regulation (EU) No 1288/2013

https://ec.europa.eu/info/strategy/european-semester/framework/europe-2020-strategy_en

In EU average: rate of early school leavers below 10%; at least 40% of people aged 30–34 having completed HE.

In accordance with

Article 2 of the Treaty on European Union

Articles 5, 11 and 16 of the Regulation (EU) No 1288/2013

See Annex 5b and 5d for more detailed

factsheets on each programme;

see also annex 1 of ICF's final report for a

comparative tables of Erasmus+ and predecessor programmes

LLP accounting for 84% of the budget for education and training 2007-2013 comprised 4 sector-based programmes:

Comenius

(SE);

Leonardo da Vinci

(VET),

Erasmus

(HE),

Grundtvig

(AE), transversal programme (policy cooperation;

language learning; ICT-based innovation, dissemination of results) and the

Jean Monnet programme.

kom (2018) 0050 - Ingen titel

in Action

(YiA),

Preparatory Actions in sport

(as of 2009), as well as in the sector of

international higher education:

Erasmus Mundus II

(as of 2009, with the rest of the world),

Edulink

(as of 2008, with African, Caribbean and Pacific Group of States),

Tempus IV

(with

neighbouring countries),

Alfa III

(with Latin America) and programmes of cooperation with

certain industrialised countries.

The current Erasmus+ programme has the following architecture:

Key Action 1: Learning mobility of individuals (opportunities

for students, trainees,

apprentices, young people and volunteers, as well as for professors, teachers, trainers, youth

workers, staff of educational institutions and civil society organisations to undertake a

learning and/or professional experience in another country)

Key Action 2: Cooperation for innovation and the exchange of good practices

(transnational or international projects promoting cooperation, innovation, exchange of

experience and know-how between different types of organisations and institutions involved

in education, training and youth or in other relevant fields)

Key Action 3: Support for policy reform

(actions supporting national authorities and

stakeholders in defining and implementing new and better coordinated policies in the field of

education, training and youth).

Jean Monnet activities

(actions aimed at improving the quality of teaching on European

integration studies, as well as projects and operating grants aimed at promoting discussion,

reflection on EU issues and enhancing knowledge about the EU and its functioning)

Sport

(cooperation projects, events, studies and other initiatives aimed at implementing EU

strategies and priorities in the field of sport)

Erasmus+ relies on

management modes

inherited from predecessor programmes.

The

European Commission

bears the overall responsibility for the supervision and

coordination of the agencies in charge of implementing the Programme at national level. The

European Commission manages the budget and sets priorities, targets and criteria for the

Programme. Furthermore, it guides and monitors the overall implementation, and evaluates

the Programme at European level after having received the National Reports from

participating countries, as described under "method". It also manages directly few actions of

the programme

At European level, the

European Commission's Education, Audiovisual and Culture

Executive Agency

(EACEA)

is responsible for the implementation of centralised actions of

the programmes

(direct management) which account for a small share of the total budget.

The largest share of the budget is implemented through indirect management. The European

Commission entrusts implementation and promotion tasks to

National Agencies

(NAs)

established in each of the 33 countries participating to the Erasmus+ programme (named

hereafter "programme countries"

) which implement those actions of the programme with the

These actions consist mainly of administrative expenditure (studies, external communication and dissemination, IT

systems, etc.), policy coordination and support actions, politically sensitive and new actions, pilot projects and preparatory

actions.

https://eacea.ec.europa.eu/homepage_en

Concerning Erasmus+ these actions are: Joint Master Degrees, large-scale European Voluntary Service under KA1,

Knowledge Alliances and Sector Skills Alliances under KA2, most of the KA3 actions, Jean Monnet activities and sport

actions.

Currently, 57 NAs appointed and supervised by the National Authorities (NAUs). Since 2014, their performance is also

controlled by Independent Audit Bodies identified in each country.

EU Member States, EFTA/EEA countries, the Former Yugoslav Republic of Macedonia and Turkey.

kom (2018) 0050 - Ingen titel

highest volume

so as to bring the programme as close as possible to its beneficiaries and to

adapt to the diversity of national education, training and youth systems.

No direct support is given to individual beneficiaries

; all support is channelled through

participating organisations, which distribute it to individual learners or practitioners.

2.1.

Main changes in the programmes over the period under evaluation

In line with the

subsidiarity principle

and Articles 165 and 166 of the Treaty on the

Functioning of the European Union, the EU intervention aims at supporting and

supplementing the actions of Member States, while fully respecting their responsibility for the

content of education, training, youth and sport policies. Meanwhile, more than its

predecessors, Erasmus+ has been designed with a view to better aligning spending actions

with the

priorities set at EU level

in each policy area.

The integrated structure of the programme is the most noticeable change in scope brought by

Erasmus+. Other important developments between the two programming periods have taken

the form of an increased budget allocation (+40%); a simplified structure in 3 Key Actions;

streamlined ways of implementation and systematic monitoring, as evaluated under

"effectiveness", "coherence" and "efficiency".

a) Erasmus+ does not only draw on the legacy of its predecessors but also brings them

altogether into a

single integrated programme,

including its international dimension funded

by the EU external action instruments (2% of all Erasmus+ student mobility activities have

involved partner countries)

. The suppression of (sub-) programmes per field (sector)

contributes to the "effectiveness", "efficiency" and "internal coherence" of the programme, as

evaluated below.

b) Erasmus+ benefits from

increased resources

with over EUR 16.4 billion spread over

seven years as opposed to a total budget of slightly more than a total of EUR 9 billion

allocated to its predecessors over 2007-2013.

Examples of changes with Erasmus+

• Opportunities for

over 4 million people

to study, train, volunteer or gain professional experience

abroad (1.8 million learners and practitioners over 2014-2016)

• A stronger focus on improving

young people’s job prospects

to tackle youth unemployment

(more and shorter traineeships since 2014)

• A

more inclusive

programme supporting people with fewer opportunities (11.5% of learner

participants)

• Opportunities for participants to study

worldwide

(12% of all higher education participating

organisations)

•

Language learning support

for mobility participants (14% of learners in long term mobility)

• New funding for actions in the field of

sport

(1,8 % of the budget – heading 1)

NAs manage KA1 mobility (except Erasmus Mundus Joint Master Degrees), KA2 strategic partnerships, structured

dialogue between young people and decision-makers under KA3

Students, trainees, apprentices, pupils, adult learners, young people, volunteers, professors, teachers, trainers, youth

workers, professionals of organisations active in the fields of education, training and youth

Development Cooperation Instrument

(DCI),

European Neighbourhood Instrument

(ENI),

Partnership Instrument for

cooperation with 3

countries

(PI),

Instrument for Pre-accession Assistance

(IPA),

European Development Fund

(EDF). The

Youth in Action

programme had already integrated its international component with the

European Voluntary Service

and

Youth in the World

opened to partner countries, with an emphasis on neighbouring countries.

See Annex 5b for more detailed

factsheet

kom (2018) 0050 - Ingen titel

c) For simplification purposes, following the conclusions of the Impact Assessment for

Erasmus+

, the current programme is based on

three cross-cutting Key Actions

which

apply to all education and training sectors and to the youth field: Key Action 1 (KA1)

supports learning mobility of individuals; Key Action 2 (KA2) fosters cooperation for

innovation and the exchange of good practices, including collaborative platforms, and Key

Action 3 (KA3) supports policy reforms. This integrated approach does not apply to the two

stand-alone strands for sport and for European integration studies ("Jean Monnet"). The fact

that the nature of the funded activities has not fundamentally changed with Erasmus+, as

reflected below, makes it possible, however, to draw conclusions about the likely impact of

the current programme based on what can be observed ex post from the impact of its

predecessors.

Main type of actions per Erasmus+ Key Actions

Learning mobility of

individuals (KA 1)

Mobility of learners

and practitioners

Erasmus Mundus

Joint Master Degrees

Student Loan

Guarantee Facility

Cooperation for innovation and Support for policy reform

exchange of good practices (KA2) (KA3)

Strategic Partnerships

IT support platforms

Knowledge Alliances

(HE)

Sector Skills Alliances

(VET)

Capacity Building (HE and

Youth)

Open method of Coordination

(OMC)

Prospective initiatives

EU transparency and

recognition tools

Dissemination & exploitation

Policy dialogue with

stakeholders, third countries

and international organisations

In bold: most significant novelties of Erasmus+

d) The integrated nature of the programme means that

cross-field approaches or tools

can be

used for the purpose of grant application, monitoring, audit or dissemination.

This has materialised through on-line application, single audit system, generalisation of

simplified grants (lump sums, unit costs), performance-based monitoring system, a single

Project results platform, etc. In designing Erasmus+, specific attention has been paid to make

more result-oriented

than its predecessors and more agile to adapt to policy evolutions

notably in the light of the socio-economic challenges mentioned above.

2.2.

Intervention logic

The EU mandate in the education, training, youth and sport areas sets the scope for the

intervention logic

. An overarching intervention logic was designed to cater for all the

programmes in scope of the evaluation, irrespective of the programming period. This

approach was followed in order to assess and compare more swiftly results across

programmes in order to make a more comprehensive assessment of the impacts achieved

during the baseline period. As a matter of fact, although each evaluated programme had

different architecture, most of the supported actions (mobility, cooperation and policy reform

support) were transversally implemented – although with a different level of intensity - under

Erasmus+ Impact Assessment,

Commission Staff Working Paper, SEC(2011) 1402 of 23.11.2011

For instance, the number of activities has been reduced from 75 to 11 in the sole case of LLP.

See Annex 5a for a more detailed visualisation of the intervention logic

kom (2018) 0050 - Ingen titel

all programmes. For that purpose, the programme's specific objectives and actions have been

grouped according to the level at which the results are expected

Furthermore, considering the overwhelming dominance of the external factors of Member

States' policy making and spending in these areas, it is not easy to attribute and quantify the

specific effects of the EU intervention.

Spill-over between intervention levels

is expected, as shown in the graphic below,

explaining why most of the specific objectives are expected to deliver results at more than one

level. For instance, the mobility of learners and practitioners (KA1) can – in addition to

individual-level results – improve the performance of the organisations and have an impact on

systems, especially in terms of outcome recognition. Also, the performance of individual

organisations (KA2) is expected to benefit from European cooperation in the fields of the

programme, including through its modernising effects on national systems and reforms

prompted by the open method of coordination and KA3.



Systems,

to improve policies

in programme and partner countries;

Organisations,

to promote cooperation

and positive changes in work methods;

Individuals,

to increase

participants’ competences

as well as change their attitudes,

practices and perceptions;

Source: Roadmap for the evaluation, January 2016

See Annexes 5a and 5c showing respectively at which level actions and objectives aim to produce results

kom (2018) 0050 - Ingen titel

MPLEMENTATION OF

RASMUS

AND PREDECESSOR PROGRAMMES UNTIL

2016

In the period 2007–2016, the programmes have funded mobility for more than

4.3 million

learners and more than 880,000 practitioners.

Many more have benefited from short-term

blended mobility and/or other forms of transnational exchanges as part of cooperation

projects. In the current programming period (2014–2016) alone, at the time of the mid-term

evaluation (i.e. without taking into account fully 2016 data), the programme had already

benefited over 1.4 million learners and 400,000 practitioners. Approximately 67 000 projects

have been contracted in the same period, which represent a reduction in yearly number by

around 50% compared to 2007-2013. Erasmus+ has indeed been designed to fund

fewer, but

more multifaceted projects with larger average numbers of participants.

Comprehensive

information is available from the

Erasmus+ programme – Annual Report

(taking into account

fully 2016 data)

The Erasmus+ programme has been

fully implemented as from 2015

without major

difficulties

. The international higher education strand and the

Student Loan Guarantee

Facility

were launched only in the second year of the programme, when arrangements were

ultimately made respectively with the External Relations instruments and the European

Investment Fund. The implications of the newly created

European Solidarity Corps

Erasmus+, more specifically on the implementation of the

European Voluntary Service

under

the Youth Chapter, only materialised in 2017; it is, for that reason, out of the scope of this

evaluation.

Erasmus+ was allocated EUR 16.4 billion for the 2014-2020 period

. Despite the overall

significant budget increase compared to the previous programming period (+40%), Erasmus+

has only experienced a limited budget increase in 2014-2016 according to the programme's

budgetary profile which concentrates the funding increase over 2017-2020

, as shown below.

It is therefore

too early at mid-term to evaluate the full impact of the additional resources

allocated to the current programme.

http://ec.europa.eu/programmes/erasmus-plus/about/statistics_en

The projects managed by the Greek National Agency for youth (0.22% of the Erasmus+ budget in 2015) were suspended

from December 2014 to April 2016 due to an insufficient management assurance. Certain OECD-related activities could

partially not be funded (€ 1 million) until the general framework for the cooperation between the Commission and the OECD

was finalised on time. See DG EAC's AAR 2015.

The Student or

Master Loan Guarantee Facility

(SLGF) is a new scheme under Erasmus+ (max. EUR 520 million

budgeted), aimed at fostering HE

degree-mobility

by easing access to student loans, irrespective of socio-economic

background. The guarantee managed by the European Investment Fund (EIF) enables students enrolled in a full Master’s

programme abroad to apply for a comparatively favourable Erasmus+ backed loan provided by participating financial

intermediaries. ICF, stand-alone report on the Evaluation of the SLGF ("ICF/SLGF" hereinafter), ICF, vol. 3

The Programme has an overall indicative financial envelope of 14.774 billion EUR under Heading 1 of the EU Budget and

of 1.680 billion EUR under Heading 4 and the EDF for the seven years (2014-2020), EU-28 appropriations.

Consequently, the final budget available to cover Erasmus+ actions in 2015 - EUR 2.115 billion – represents only a slight

increase compared to 2014 (+2.1%).

kom (2018) 0050 - Ingen titel

Source: Erasmus+ activity report 2015

Distribution of Erasmus+ budget across sectors (2014-2016)

Source: ICF calculations based on EplusLink

The most substantial part of the Erasmus+ budget is allocated to education and training and,

within this field, to higher education, as shown in the graph above

. In terms of actions,

mobility has the largest share of the budget, as displayed below.

Key data, 2014 - 2016

KA1

Projects received

Contracted projects

Amounts allocated (in

million EUR) *

Participants

108,904

49,073

3,798.9

2,049,140

KA2

34,298

6,936

1,469.4

1,231,267

KA3

4,456

2,127

205.9

184,403

Jean Monnet

2,425

762

126.2

944,245

Sport

1,325

238

67.3

Source: Erasmus+ Programme Annual Reports 2014-2016

(

taking into account fully 2016 data)

*Amounts not only for project grants but including also operating grants

"Other" is compound of operating grants to national agencies and administrative expenditure

kom (2018) 0050 - Ingen titel

ETHOD OF THE EVALUATION

Considering the high quality of data collected through several complementary techniques

from various sources, the sophisticated analysis and triangulation of evidence, the positive

opinion of the Regulatory Scrutiny Board and the improvement stemming from its comments,

as well as the significant feedback received (with more than a million responses processed in

total), this

evaluation can be considered very reliable and valid.

4.1.

Evaluation framework

To undertake the combined mid-term evaluation of Erasmus+ and the final evaluation of its

predecessor programmes, as outlined above in section 1, the Commission has drawn upon a

set of

evaluation questions relating to five main criteria:

effectiveness, relevance,

coherence, efficiency and EU added value, as per the evaluation Roadmap

To ensure comparable approaches, the evaluation framework developed

by the external

evaluator

classifies actions across the three

levels of intervention

of the programmes and

uses that as the basis of comparison over 10 years, across all sectors and various target

groups, namely:



System

level

Organisation

level

Individual

level, divided into actions for

learners

and

practitioners

across all

different

areas

(education and training, youth and sport) and – within education and

training – different

sectors

(school education (SE), vocational education and training

(VET), higher education (HE) and adult education (AE)). For the purpose of the

evaluation, the overarching terms of

Learners

and

Practitioners

are used

In addition, the analysis of effectiveness has been based on the intervention logic represented

graphically in annex 5a accompanied by a detailed elaboration of expected

outcomes

(results

directly for individual beneficiaries;

impact

indirectly for a wider population, participating

organisations or education, training, youth and sport systems)

. All specific objectives of the

programme (annex 5c) are reflected in the outcomes or impacts of the intervention logic.

The purpose of the evaluation is to evaluate the outcome of the programme on target groups,

not to audit each type of activities. Therefore this intervention logic is a powerful simplified

model to develop questionnaires based on the most relevant levels of outcomes. It is worth

noting that these three levels of intervention are not equivalent to the three Key Actions, as

explained above

. A given Key Action can indeed have an effect on more than one level of

intervention (see 2.2). Key Actions have been set within Erasmus+ but these three levels of

interventions apply to both programming periods

http://ec.europa.eu/smart-regulation/roadmaps/docs/2015_eac_014_evaluation_erasmus_en.pdf

ICF, Annex 14 (not published), where a matrix maps and compares the types of actions in the predecessor programmes and

the corresponding actions in the Erasmus+ programme

Learners

refer to all individuals involved in formal, non-formal and informal education as pupils, students, apprentices,

volunteers, young people, etc.

Practitioners

refer to those involved in the same respect as teachers (including prospective

teachers), trainers, youth workers, educators, coaches, organisation leaders or staff, etc.

See annex 5a. This framework of outcomes has resulted from a combination of review of programme documentation,

literature on results and impacts of similar types of interventions and scoping interviews with policy officers in charge of the

programme. ICF, 3.3

For instance, individuals are also targeted within KA2 activities in the case of school pupil or adult mobility.

The predecessor programmes in international higher education had a variety of intervention logics of which only some

elements are reflected in the evaluation intervention logic. Doing an evaluation programme by programme would have

fragmented the data collection and would not have allowed to use some of the more advanced analytical techniques.

kom (2018) 0050 - Ingen titel

This unprecedented

combination of the ex post evaluation of predecessor programmes

and the mid-term evaluation

of the current programme is a legal requirement. It implies to

evaluate differently the two periods, as laid down in article 21(2) of the Erasmus+ Regulation:

[The mid-term evaluation report] "shall

also take into account the results of an evaluation of

the long-term impact of the predecessor programmes (Lifelong Learning, Youth in Action,

Erasmus Mundus and other international higher education programmes)."

Whereas

Erasmus+ is evaluated against all aforementioned criteria, its predecessors are mainly

evaluated in relation to effectiveness and relevance

. However, since

the predecessors are

systematically taken as a baseline for comparison with Erasmus+,

they have been

evaluated also for comparison purposes in terms of coherence, efficiency and European added

value.

Systemic impact as well as long term effects of projects should mainly be attributed to

previous programmes, whilst short-term effects on individuals have rather been evaluated in

the context of the current programme. The results at the level of individuals are unlikely to be

different between the two programming periods as the types of actions are highly similar.

What has mainly changed in that respect is the scale and reach of the programme as of 2014.

The programme is designed so that the systemic impact of projects (Key Actions 2 and 3)

should only be evaluated "by

the end of the programme"

as per article 21(1)b of the legal

basis. At mid-term,

systemic and long term effects can also be attributed by extrapolation

to Erasmus+

due to the continued nature of basic activities (mobility; cooperation).

4.2.

Data collection and analysis

Evaluation findings have been based on

a wide range of data sources

systematically

triangulated i.e. cross-checked against each other: programme documentation, programme

data and views of the managing bodies, views of direct beneficiaries as well as non-

beneficiaries, EU and national stakeholders, policy makers, programme countries, bodies in

charge of implementation of other comparable programmes and the general public

The evaluation combined a number of

techniques for data collection and analysis.

Most

results are reached by a mix of evidence combining a quasi-counterfactual assessment based

on beneficiaries’ surveys (i.e. comparisons between beneficiaries and a control group), self-

reported feedback of beneficiaries (i.e. monitoring surveys), as well as qualitative interviews

and case studies. While monitoring surveys could only give findings as from 2014, evaluation

surveys covered 2007-2016 and enabled the evaluators to compare the two programme

generations. Monitoring data have been fully used and supplemented with surveys carried out

by the external evaluator, providing more objective findings (with questions not based on self-

perception), more comprehensive data (covering as well the 2007-2013 period and KA2

participants) and more comparable responses (as questions were harmonised across sectors

and periods for comparison purpose).

Collection tools presented below covered all programme countries, sectors and main types of

actions (except where mentioned otherwise). The

quality of the one million responses

processed

was in most cases exceptionally good. All quantitative data was statistically

The Terms of reference of this evaluation specify that "The

questions are in many cases phrased in a general way, however

the intention is that they cover the entire Erasmus+ programme (where applicable also the predecessor programme)."

For

instance: "What

are the long-term impacts of the predecessor programmes? We are interested in the impact of all

actions/elements of the predecessor's programmes, in particular those actions/elements that are continued in new Erasmus+

programme. We are also interested in impact of actions/elements that have discontinued to the extent that it might help to

design the future programme."or

"What

conclusions can be drawn on the likely impact of Erasmus+ programme given the

fact that significant parts of their actions are continuation of predecessor's programmes?"

Primary data were mainly collected from November 2016 to May 2017.

kom (2018) 0050 - Ingen titel

processed and presented using visualisation tools, while all qualitative data was systematically

coded.

As required by the Erasmus+ Regulation, programme countries have submitted their own

evaluation reports on the implementation and impact of decentralised actions of Erasmus+ in

their respective territories. Their findings are summarised in the

National reports synthesis

(NRS)

annexed to the external evaluator's report

In line with Better Regulation rules, an

Open Public Consultation

(OPC) gathered the

opinions of the general public and all interested groups on all evaluation criteria.

Different

questions were asked depending on the level and scope of knowledge of the programmes

declared by each respondent. Consultation findings are presented separately in the synopsis

drawn by the external evaluator

. More widely, responses from all sources are presented in

Annex 2.

Box 1 provides an overview of the data sources from which the evidence was drawn. A

detailed description of the individual methods is provided in Annex 3.



Literature review

(131 sources about 2007-2016)

Key informant

interviews

(190 interviews about 2007-2016)

National reports

(34 countries mainly about 2014-2016)

Open Public Consultation

(covering 2007-2016 with 4,786 responses of which 1,800 responses

were fully exploitable; 24 position papers submitted)

Social Media

analysis (725,678 posts over 12 months)

Benchmarking

(18 comparable national/transnational programmes)

Programme database

analysis (all beneficiaries 2007-2016) including network analysis

Monitoring surveys

(over 955,000 respondents - KA1 since 2014)

Online Linguistic Support

(523,238 participants since 2014)

Beneficiary surveys

and

control groups

(over 47,000 respondents over 2007-2016)

Survey of socio-economic organisations

(947 responses over 2007-2016)

Agencies survey

(130 responses from National Agencies or the Executive Agency EACEA)

Experts survey

(1,122 responses from project assessors over 2007-2016)

Case studies

(38 organisations or policy cases; 233 respondents over 2007-2016)

Review of selected

projects’ outputs

(386 project reports over 2007-2016)

Expert panel assessment

of projects’ outputs (100 projects)

Jean Monnet activities (2,350 survey respondents; 5 interviews)

Student Loan Guarantee Facility (SLGF) (219 survey respondents; 33 interviews)

National reports synthesis

(NRS hereinafter), ICF, vol. 6

https://ec.europa.eu/programmes/erasmus-plus/resources/consultations/erasmus-plus-mid-term-evaluation-2017_en

Synopsis of the open Public Consultation

(OPC hereinafter), ICF, vol. 5

kom (2018) 0050 - Ingen titel

4.3.

Method strengths and limitations

Overall,

the quality of available data for the predecessor programmes was good, while

for Erasmus+ it was very good.

This reflects improvements made in terms of monitoring

since 2014. The evaluation used the wealth of the programme data available from the

Erasmus+ management or monitoring tools, including some data on results (e.g. satisfaction-

rate) since 2014. Although the period within the scope of the evaluation was 2007-2016,

programme data was not fully available for 2007

The main issues that affected the analysis of participation patterns include the following:

i) the absence of a harmonised definition, across sectors, of people with fewer opportunities

and from

disadvantaged backgrounds

(programme data were completed with other

findings)

;

ii) the lack of a complete picture on the type of participating

organisations

(some 40% of

organisations were classified as "other" in the programme dataset limiting the analysis of

participation per type of organisation).

It should also be noted that the analysis over time of networking among participating

organisations or repeated participation of individuals in the programme could not be fully

pursued, as prior to 2014 the same organisations could be recorded under multiple names,

which made such an analysis difficult to execute. These

limitations have however remained

marginal

or have been overcome through the use of other sources.

The

monitoring surveys of beneficiaries

carried out by the Commission concern all learners

or practitioners taking part in mobility under KA1 (since 2014 only)

. All are requested to fill

in an online survey on completion of their mobility. Therefore the

reliability of the data is

strong.

Given the sample sizes (first two years of the programme) the data is considered to be

strongly reliable even though it was too early to include 2016 data. This being said the link

between the monitoring questions and the indicators they are supposed to measure can be

weak in some cases. Above all,

most questions are based on the self-perception of

beneficiaries

in contrast to more objective measurements such as the assessment of language

proficiency after mobility (OLS

). This is why monitoring data were supplemented with

primary data collection, such as surveys carried out by the external evaluator, providing more

objective and comparable responses.

Series of

surveys of beneficiary learners and staff

and related control groups covered all

programme target groups and predecessor programmes as well (2008-2016 in total), both for

mobility and cooperation, over 2007-2016. Overall,

findings could be generalised

to the

whole programme with a

sufficient degree of confidence

because of the

sample sizes of all

surveys

and the distribution of various background variables within the survey samples.

Given the limited data available on the background of beneficiaries, it nevertheless remains

impossible to quantify the degree of comparability of the samples to the programme

population

Monitoring data for 2016, although not yet confirmed, were also analysed for outputs and participation patterns.

See sections on effectiveness and relevance below for encouraging results in terms of social inclusiveness

See annex 5e reporting on all output and result indicators from the legal basis against targets set in DG EAC's Strategic

Plan. As these legal basis indicators are based primarily on direct beneficiaries’ perceptions of the programme contribution,

they are only meaningful when asked shortly after the mobility experience. It would not have been possible to reconstruct

them even if the evaluation surveys had asked comparable questions to the beneficiaries of 2007-2013.

The

Online Linguistic Support

(OLS) was introduced with Erasmus+. Before and at the end of their mobility, participants

(HE, VET or EVS) take the language assessment to measure their level and progress in the language. Language courses are

also provided to them on-line.

There is no database that would provide data on the overall target population. Moreover for several surveys (pupils, young

people, sport staff) or certain actions (KA2, predecessor programmes) there is often no contact database of direct

kom (2018) 0050 - Ingen titel

partially counterfactual approach

("quasi-experimental") was also used to quantify the

contribution of the programme to the results measured. This was done by assessing the

difference in results between Erasmus+ beneficiaries and their "peers" who did not take part.

The challenge is to attribute the difference in outcomes between both groups to the

programme isolating any other underlying factor (attribution).

Control groups of non-

beneficiaries

were therefore set up. In an ideal world, these should be selected on random

basis. However, a random allocation is neither feasible in practical terms nor desirable for

ethical reasons. By default, a ‘matched sample’ was sought where individuals are similar in

some background characteristics such as gender, age, etc. There were no significant

differences between the control group and the treatment group for most variables, and where

there were, they concerned variables that were not likely to strongly influence the findings.

Most control groups reached a

significant size allowing comparison with beneficiaries,

except in sport and Adult Education sectors

where the control groups were too small. As a

consequence, the control group of VET staff was also used for Adult Education and the

control group of Youth staff was also used for sport, as these were found to be sufficiently

comparable against a range of background characteristics. However this counterfactual

approach cannot exclude the possibility that the difference in a given variable between

beneficiaries and non-beneficiaries could be partially due to the selection into the programme

rather than the effect of the programme.

In addition to the above ‘post’ surveys (i.e. beneficiaries were surveyed after they took part in

the programme), the external evaluator carried out pre-surveys in two sectors where short

term mobility exchanges apply (i.e. school pupils and Youth exchanges). This was done as far

as possible within the time frame of the evaluation assignment for the purpose of

pre/post

comparison.

In spite of repeated recruitment efforts, the sample sizes suffered from

a high

attrition from the pre to the post survey,

especially for the Youth sector where comparison

of profiles prior to Erasmus+ mobility could not be done. Pre-post results for the

Youth

sector

have therefore not been interpreted considering the

high likelihood of sample bias.

This allowed nevertheless for the school sector to draw conclusions on selection into the

programme and to observe changes during the time of the mobility experience.

To get a better understanding of their quality and dissemination potential, a

review of

selected project outputs,

mainly collected from National Agencies, was carried out. Given

that only relatively few Erasmus+ funded projects were completed at the time of this

evaluation, the sample remained inevitably

somewhat biased towards predecessor

programmes.

As for the

Open Public Consultation

(OPC), a large number of partial responses only

contained very basic background information on respondents and had therefore to be marked

as unsuitable for analysis. In any case, in contrast to surveys, public consultation findings can

never be representative due to the selection bias inherent to any open recruitment.

Lastly, some programme countries have delivered their

national reports

with a four month

delay. This, in turn, has delayed the adoption of the Commission report to take into account

all programme countries' views. The period of time between the deadlines set by the legislator

respectively for the delivery of the national reports and the Commission report was reduced

from 9 months (LLP) to 6 months (Erasmus+), which has not allowed the external evaluator

to make a full synthesis of all findings

beneficiaries, or not standardised ones, hence there is no background data on the total population of beneficiaries to compare

with respondents' profile. This is why respondents have been recruited through organisations.

ICF has delivered separate reports, respectively volumes 1 (main report) and 6 (separate synthesis of national reports).

kom (2018) 0050 - Ingen titel

4.4.

Judgement on the validity and reliability of the findings

The

external evaluator (ICF)

contracted for this assignment

has carried out since May

2016 all tasks as required

under the scrutiny of an inter-service group (ISG)

and the

daily steer of DG Education, Youth, Sport and Culture (EAC). The only significant change

compared to the initial work plan was a two-month delay of the (sub)contractor in launching

the beneficiary surveys

. The ISG was consulted at each stage of the evaluation process and

reviewed each deliverable produced by the contractor as well as this Staff Working

Document

The Regulatory Scrutiny Board, responsible for the independent quality control of this

evaluation, acknowledged in its

positive opinion

with comments the significant efforts of

data and evidence collection and noted the good methodology used

. This document has been

improved following its comments.

Based on the above elements of the evaluation method and as further described in Annex 3,

the reliability and validity of the evaluation can be regarded as strong

Elements of the method

Validity of overall judgements

Definition of results and impacts

Validity of measurement tools to collect data on results and impacts

Reliability of overall evaluation design

Collection of qualitative data

Interpretation of qualitative data

Generalisation of findings to the whole programme

Counterfactual assessment

Surveys

Programme data

Source: ICF, 3.5; see section above for assessment grounds

Assessment of reliability and

validity

Strong

Medium to strong

Medium

ICF Consulting Services Ltd under specific contract – EAC-2016-0219 implementing Framework contract EAC/22/2013

Commission services (BUDG, DEVCO, EAC, EACEA, EEAS, EMPL, FPI, HOME, JRC, NEAR, SG), and EESC

DG EAC wrote to ICF to mitigate that issue in March 2017. The delay had no consequence on the final report.

ISG meetings on 9 September 2015 (Kick-off), 18 July 2016 (Inception report); 21 November 2016 (Interim report 1); 24

March 2017 (Interim report 2); 10 July 2017 (Draft final report); 11 September 2017 (Final report). The ISG was thereafter

consulted on the drafting of this Staff Working Document (29 September 2017) and informed of the Regulatory Scrutiny

Board's positive opinion (27 November 2017).

Ares(2017)5629740 - 17/11/2017

ICF, 3.7

kom (2018) 0050 - Ingen titel

INDINGS

5.1.

Effectiveness

The assessment of the effectiveness

of Erasmus+ is

overall very positive,

especially when

compared to the situation before the integration of predecessor programmes

. It has been

informed by 15 evaluation questions that looked into the extent to which:

outcomes (i.e. outputs, results and impacts) were delivered with particular attention to

disadvantaged groups



spill-over, sustainable or unintended effects took place



certain fields or actions were more effective than others (including discontinued

actions)



results were disseminated and exploited.

The programme

reaches or exceeds the vast majority of its indicator targets set in the

legal basis

except in the case of the Student Loan

Guarantee Facility (SLGF) which is

missing the target by a high margin, as detailed below

. However, some output targets might

have been set at a low level initially (sometimes below predecessor programme levels). The

quality of the data on participation of disadvantaged groups although on target is unclear due

to various definitions and practices (see below). Lastly, for Jean Monnet activities there is no

exact data on numbers of students reached and this indicator is likely to be very

overestimated. These limitations concern only a few indicators and the vast majority are

without doubt on target.

Incoming mobility of students from

higher education international strand

is also below

target but the action started in 2015 only and the

overall trend over ten years is positive.

Evaluated programmes have been important for the EU's global outreach, as elaborated

hereafter

The

Open Public Consultation confirms that the programme is achieving its objectives

a (very) large extent. OPC respondents highlighted two elements as particularly helpful in

terms of effectiveness: the new integrated structure of the programme

and the overall

increase in funding

. One sustainable result of the programme and its predecessors is the

cultural shift in the perception of mobility and its positive image, at least in the higher

education sector and progressively in VET. These long-term changes are yet to fully

materialise in the other sectors.

5.1.1. Outcomes for learners and practitioners

Compared to the last three years of predecessor programmes, Erasmus+ has supported during

the first two years

much higher volume of mobility

within and outside the EU in higher

education (the sector that has most beneficiaries overall), but also a higher number of

practitioners

in VET and schools

. The repeated participation of individuals is kept



Effectiveness

analysis considers to which extent a programme has achieved its objectives, using appropriate points of

comparison and elaborating an opinion on the role of this programme in delivering the observed changes.

ICF, 6; NRS 5; OPC 3.3 and 5.1.2

Erasmus+ annual reports:

http://ec.europa.eu/programmes/erasmus-plus/about/statistics_en

ICF, 6.2.1; tab. 6.3 and annex 3 JMO, 1.2.2

Evaluation of the EU Development Co-operation Support to Higher Education in Partner Countries (2007-2014),

(2017),

Particip, LeA, ECDPM, Ecorys for the Directorate-General for International Cooperation and Development (European

Commission)

One EU level organisation representing over 30 HEIs; one sub-national organisation representing a region; and one

national agency for HE cooperation.

One EU-level organisation representing over 30 HEIs.

See section 3 for increasing budget profile as of 2017

OPC: 84% agree that Erasmus+ provides more opportunities for mobility than predecessor programmes (n = 552).

kom (2018) 0050 - Ingen titel

reasonably low, indicating a continuous impact on renewed cohorts. In contrast, as intended

with the design of Erasmus+

there has been

a significant decrease in mobility for pupils

due to the fact that individual mobility of pupils was not embedded in the programme as such

any more - and a

slight decrease for VET learners

5.1.1.1. Outcomes for learners

Beyond contrasted evolutions in outputs across sectors, drawing on a variety of sources

, the

evaluation found

positive and sustainable results for all individual learners

taking part in

mobility

. Firstly, monitoring surveys outline that most learners themselves are satisfied with

their mobility experience

. The rate of formal recognition of participation in Erasmus+ is

high but, more importantly for those concerned; the rate of recognition of learning outcomes

is also rather high

. The vast majority agree that the programme helped them improve more

than 6 competences out of a list of 9 (see fig.)

. To be noted, higher education students are

comparatively least positive in their self-assessment

Share of learners who (strongly) agree that they developed transversal competence

Erasmus+ no longer provides for (KA1) individual pupil mobility, which is now only limited to mobility activities

embedded in (KA2) cooperation projects.

However as mobility in VET is more dominated by traineeships in companies, it is likely that the programme now reaches

more typical VET learners. The number of adults reached through mobility remained besides stable.

Sources: monitoring surveys, evaluation surveys including control groups, case studies, literature review. For instance, the

Erasmus impact study (2014) outlines how the programme has improved prospects for a successful career by giving HE

students skills that are sought after by employers:

http://ec.europa.eu/dgs/education_culture/repository/education/library/study/2014/erasmus-impact_en.pdf

For mobile learners but also for those taking part in KA2 blended learning mobility and its equivalent in the past

93% of learners were very satisfied or rather satisfied with their mobility experience; ICF, 6.4.3 per sector

ICF, tab. 5.11: Over 2014-2016, formal recognition of participation in Erasmus + ranges from 80% (VET with Europass)

to 87% EVS and 88% youth exchanges (Youthpass). Academic recognition is at 83% for HE.

More than 80% learners declare having improved six or more competences from the proposed set of nine competences,

except in HE (71%); ICF, fig. 6.4

This could be explained because their competences are already at higher level when entering the programme, if it was not

for the fact that those HE students in traineeships show a different pattern. ICF, fig. 6.3 and 6.4). Moreover, HE learners

within Europe are less satisfied than others; e.g. with mentoring (68% (rather) satisfied against 81%). IFC, fig.6.7 to 6.9

kom (2018) 0050 - Ingen titel

Source: ICF calculations based on Erasmus+ participant monitoring surveys. Multiple answer question. N = 730,254

In line with the

intervention logic

, all programmes under review are expected to contribute

to changes in learners’ skills and competences as well as attitudes. In the longer term, they are

expected to result in better completion rates and more positive employment outcomes. Both

have materialised as detailed below. As far as the general indicators of Erasmus+ are

concerned, trends are also positive and the

Education and Training Monitor 2017

highlights a

further decrease in the rate of early school leavers, and a tertiary attainment target that is

within reach

Most

result

indicators set in the legal basis of Erasmus+ are on or above target

. The

evaluation found, using a variety of sources, clear evidence of results for mobile learners in:

foreign language skills development, especially for those with low entry levels; feeling of

belonging to the EU; willingness to be mobile and to work abroad; shorter transition from

education to work

; positive perception of the value of education and positive feeling towards

the sending organisation which are key precursors for retention and attainment in education

and training. As evidenced for those benefiting from KA1 mobility, the programme is

associated with stronger rate of completion of studies

However the result indicators set in the programme legal basis and related targets set in DG

EAC's strategic plan have some limitations in that they are mostly based on the self-perceived

contribution of KA1 reported by beneficiaries. Some of the targets were exceeded and others

not reached (e.g. participants declaring that they have increased their language skills; young

participants declaring being better prepared to participate in social and political life).

However, there are concerns about the plausibility of the target values in both cases

The evaluation does not permit isolating how many of these results are the direct effect

of the

selection of the most motivated

into the programme. For instance, Jean Monnet students do

show better results than other higher education students when it comes to knowledge of the

EU. This however can be expected given that the programme reaches mainly those who have

shown an interest in the EU topics

The most successful objectives of the Programme, which are also the most relevant, according

to OPC respondents, have been a)

developing the skills and competences of learners;

and

b) promoting the European dimension of education, training and youth activities

. Other

studies also confirm this in the relevant sectors

. Higher education who have participated in

the programme are not only more likely to be employed, but also more likely to secure

management positions. On average, 64% of Erasmus students, compared to 55% of their non-

mobile peers hold such positions within 5-10 years from graduation. This holds even more

true for Erasmus students from Central and Eastern Europe, where around 70% of them end

up in managerial jobs

. The same can be said for the international dimension of higher

Section 2.2 above and Annex 4a (intervention logic)

http://ec.europa.eu/education/policy/strategic-framework/et-monitor_en

See annex 5e reporting on all output and result indicators from the legal basis.

For mobile learners in KA1 only; this has not been demonstrated for other types of actions

Measured as a proxy by the absence of repetition of a class within the typical duration of mobility

ICF, 6.4.2 and tab. 6.10. As there was no comparable data from the predecessor programmes (no baseline data) for many

indicators, “it seems that the targets were defined somewhat arbitrarily”.

Respectively 86% and 80% of OPC respondents (n > 1400) thought that these two objectives were achieved to a ‘very

large’ or ‘large’ extent.

ICF, tab 6.13 and 6.14 for analysis per sector; there is less evidence of positive impact of mobility in the Adult Learning

sector, but it could be simply not researched.

Follow-up to the 2014 Erasmus Impact Study focusing on

regional analysis

of the benefits of the Erasmus programme,

2016

kom (2018) 0050 - Ingen titel

education, with clear evidence of strengthened capacities and employability of individual

students and scholars who have participated in mobility

Albeit mostly based on the self-reported perceptions of participants, most National Reports

refer also to improvement of the level of key competences and skills of learners with a high

level of similarity with the ones listed above. On top of technical/professional knowledge and

skills

, the impact of Erasmus+ on employability is underlined by several countries mostly in

the case of HE and VET

, transversal skills

or soft skills are also frequently identified.

These include communication skills

, social and civic skills

, and more widely personal

development

The case studies confirmed the

strongly positive influence

the programme plays

on personal

development and maturity

of young people. Both sources frequently outline that learners

gain in confidence, independence, ability to cope with new environments and open-

mindedness through their mobility. Participation in the programme also leads to development

of learners’ social capital

. In contrast to very positive effects of all forms of mobility, there

is less clear evidence of impact of transnational partnerships on learners (excluding their

mobility component) except in the school sector

100

. For example, a survey conducted as part

of the eTwinning

101

evaluation, indicated that 55% of teachers believe that the platform is

helpful in motivating their pupils

102

At programme level, the areas where learner surveys have identified

strongest positive

difference between the results of participants and non-participants

103

include: willingness

to move abroad permanently (+31% likeliness compared to non-participants);

stronger

feeling of being an EU citizen

(+19%);

shorter transition to employment

(+13%)

104

; value

attached to education (+8%) and also openness to immigration and minorities (+2.6%). The

figure below gives an overview of all areas evaluated according to the degree of difference

found

105

Evaluation of the EU Development Co-operation Support to Higher Education in Partner Countries (2007-2014), (2017),

Particip, LeA, ECDPM, Ecorys, already mentioned

BEde, DK, LU, HU, MT, NL, AT, PT, NO

BEnl, FR, MT, NL, FI, SE, IS, NO

CY, HU, NL, NO

LT, PL, UK, IS

DE, LT, LU, MT, NL, SK, UK

BEnl, DK, DE, IE, EL, ES, CY, LT, LU, MT, NL, PL, PT, FI, SE, UK, IS, TR, (e.g. for learners and youth:, self-esteem,

self-awareness)

55.7% of beneficiaries were in frequent contact with persons they met during participation in the programme (at least

monthly basis) and another 30.8% were occasionally in contact (at least once a year). VET beneficiaries have more

occasional or rare contact with people they met through the programme.

100

For instance, pupils who benefitted from partnerships have higher degree of independence, motivation and self-

confidence, better communication skills, increased enthusiasm for learning European languages.

101

eTwinning

offers to the school practitioner community a platform to develop and share on-line their transnational projects

(involving more than 183,000 schools in September 2017):

https://www.etwinning.net/en/pub/index.htm

102

Education for Change (2013)

Study of the impact of e-Twinning on participating pupils, teachers and schools

http://bookshop.europa.eu/en/study-of-the-impact-of-etwinning-on-participating-pupils-teachers-and-schools-pbNC3112371/

103

ICF, tab. 6.12 for estimations of treatment effect and statistical significance

104

Measured for HE and VET beneficiaries only; not measured for school pupils as not yet relevant

105

ICF, 6.4.4.4 - 7 and Annex 3 for detailed analysis per sector

kom (2018) 0050 - Ingen titel

Comparison of results for learner beneficiaries and control group

Large positive

difference

Short transition to

employment (0-3

months)

Feeling of belonging to

the EU

Medium positive

difference

Willingness to work

abroad

Positive perception of

the value of education

Statistically significant

yet small positive

difference

Openness to other

cultures and minorities

Problem solving

behaviours

Medium negative

difference

Active civic engagement

in an organisation (in the

past)

Civic competence -

perceived importance of

community engagement

activities

Completion of studies

(in a typical duration)

Positive feelings

towards their school/

education org.

Sense of initiative

Civic competence -

perceived importance of

civic behaviours and

activities

Digital competence - use Learning to learn

of various online

resources

Positive feelings

towards the EU

Source: ICF Beneficiary surveys: only areas where the difference is statistically significant i.e. unlikely due to a sampling interference

The share of beneficiaries who took

less than three months to find a job

stood at 68.5%,

while the percentage of the overall control group was at 59.2%. This trend mirrors the positive

expectation of most beneficiaries towards Erasmus+

106

. This is particularly strong for the

higher education

107

and VET beneficiaries, in contrast with the youth sector

108

. Erasmus+ also

leaves a positive effect on entrepreneurship

109

. Overall, this confirms the findings of several

previous studies about the effects of learners’ mobility on employability

110

The

pre/post analysis for pupils

confirms the positive contribution of Erasmus+ in five

similar result areas, the order of which appears nevertheless specific to the school sector.

There are statistically significant differences in results across pre- and post-survey of school

pupils in EU citizenship (+7 score points), digital competence (+4), racism and xenophobia

(+3), self-confidence in education (+4), civic participation and volunteering (negative

difference).

However, there is

likely selection of already more performant pupils into the programme

at the level of schools,

since the pre-post survey also shows already a significant difference

between participants and the control group at entry into the programme. Pupils participating

in short-term mobility speak a foreign language more often than the control group

(+13 percentage points or p.p.) and feel more often as EU citizens already before going on

mobility (+9 p.p.). Cultural awareness (+7 points in index score or p.i.s.), positive attitudes

towards immigration and minorities (+6 p.i.s.), digital competence (+5 p.i.s.) and perceptions

106

107

They are 92% in general to believe so according to the beneficiary survey.

70% of Erasmus Mundus alumni found their first job within three months after graduation according to the tracer study in

Evaluation of the EU Development Co-operation Support to Higher Education in Partner Countries (2007-2014), (2017),

Particip, LeA, ECDPM, Ecorys, already mentioned

108

Beneficiaries in the youth sector experience longer transition than the control group, which could be due to the specific

profile of this target group.

109

The interim evaluation of the EIT indicates a self-reported rate of adoption of entrepreneurial skills of 69% for Erasmus

mobile students compared to 83% for EIT-KICs. The foundation rate for new start-ups is 7% with Erasmus+ (6% with EIT-

KICs).

110

ICF, 6.4.5 for findings of the literature review; ICF, annex 11 (literature reviewed)

kom (2018) 0050 - Ingen titel

of the value of their education (+4 p.i.s.) were also higher. This is confirmed by the case

studies where some respondents claim that participation is seen as a certain form of reward.

Foreign language skills development is more important in those sectors where learners tend to

have lower entry levels (i.e. in VET and youth more than in higher education). The analysis of

OLS foreign language tests (pre/post mobility) proved that

the weaker the proficiency entry

level, the stronger the linguistic improvement

due to mobility

111

Comparison of levels of proficiency before and after mobility, by sector

Legend: share of learners in each sector at a given level of proficiency; blue: before mobility (first language assessment), red: at the end of

the mobility (final language assessment)

Source: ICF calculations based on OLS data as of 31/08/2016. Higher education: n= 500,410; VET: n=19,441; EVS: n=3,387

Beneficiaries were asked in case studies to describe the most important contribution of the

programme. The word-cloud below visualises the most cited testimonies.

What do you see as the most important contribution of the programme for yourself?

Source: ICF case studies –interviews with learners: 200 mentions or expressions were collected; the size of the font captures the frequency

with which a given result was mentioned.

111

ICF, fig. 6.5

kom (2018) 0050 - Ingen titel

When asked about reasons hindering the achievement of programme objectives

112

respondents identified the following barriers during the public consultation, in descending

order of frequency:

linguistic barriers; financial difficulties

and

complex application

process

113

. The factors that enable positive results at the level of individual learners are related

to their

motivation

to participate as well as the quality of the learning experience including

foreign language learning support, the

support

received in the host organisation and their

integration

among other students and local communities

114

. Lastly, levels of recognition of

learning outcomes vary according to types of actions

115

, but signing a learning agreement

remains a strong predictive indicator for later recognition. Only a small share of learners does

not sign one

116

Although financial barriers are a major obstacle to mobility

117

, the

Student Loan Guarantee

Facility,

which was an innovation in Erasmus+,

has so far failed to attract

financial

intermediaries in sufficient numbers

118

– especially for the incoming student segment

119

. First

beneficiaries are, however, satisfied. The Student Loan Guarantee Facility already shows

signs of social fairness

120

, but its visibility at this early stage is far from being sufficient

throughout the supply chain

121

5.1.1.2. Outcomes for disadvantaged learners

The current programme is rightly perceived as

paying more attention to the participation of

disadvantaged people

122

. To respond to the criticisms regarding the predecessor programmes

– notably in the field of higher education – of being elite programmes, Erasmus+ has put in

place actions to strengthen the participation of disadvantaged groups compared to predecessor

programmes

123

. In the period under review, 11.5% of participants under KA1 had special

needs or came from a disadvantaged background. This share rose to 30% for learners in the

Youth field between 2014 and 2016

124

. The number of participants with special needs or fewer

opportunities has more than doubled since the predecessor programmes, i.e. more than

proportionally compared to the overall budget increase. The indicator targets have even been

112

This question was only addressed to OPC respondents who indicated that they have detailed or some knowledge of the

Erasmus+ objectives and actions’.

113

Other barriers mentioned in the OPC: lack of cooperation between actors involved at their level; differences in educational

systems hindering their cooperation; lack of information about the Programme and poor quality of courses.

114

Monitoring surveys show that the integration of learners in local communities is more difficult than in the host institution.

HE mobile learners (within programme countries) feel least well integrated to both; ICF, fig. 6.9

115

Recognition of learning outcomes is not compulsory. Europass is a tool helping to document and to make learning

outcomes visible. In VET, where the most common form of documentation is Europass, recognition using ECVET credit

remain small by nature (credits points are not commonly used in VET systems) but is progressing (3 percentage points

between 2014 and 2015). The degree of recognition is highest in HE thanks to ECTS, particularly within programme

countries and among students in traineeships. The degree of recognition within the youth sector is higher for those in youth

exchanges, albeit of shorter duration, than for those in EVS.

116

In VET, 12% of learners did not sign a learning agreement prior to departure on mobility.

117

E.g. according to Portugal’s recent estimations the contribution of families to HE mobility can reach 50%; NRS, 3.4

118

After two years, 6 intermediaries in 5 countries and 162 students supported in 2016, well below initial expectations

119

ICF, stand-alone report on the SLGF, 4.4 for reasons and flaws in the design of the loan facility

120

71% of the first loan beneficiaries (n=65) said in 2016 they would not have been able to study abroad without the loan.

43% of SLGF beneficiaries (mostly from Spain) in 2015 or 2016 had parents not holding a university degree

; ICF

standalone report, 4.4.3 and 4.4.4 respectively

121

93% of respondents met at student fairs (80% for the exhibitors) had no prior knowledge of the loan facility.

122

Several case study interviewees; NRS, 3.4 for national examples of promoting the participation of vulnerable groups

123

Top up grants and grants to fund accompanying persons are now available for those who may need such assistance. A

youth inclusion strategy entails financial incentives and specific project format to foster the participation of young people

with fewer opportunities and from disadvantaged backgrounds.

124

ICF, annex 2, section 2.2.3. However there are concerns about the reliability of data given the collection method and

differences in the breadth of definitions used across sectors. Survey data suggests that the differences between sectors are

much smaller. ICF, 6.3

kom (2018) 0050 - Ingen titel

exceeded

125

, though data reliability is weak

126

. This is why some countries call for more

consistent definitions and rules for disadvantaged participants

127

According to case studies, the programme

reaches out to those who are easier to reach

among the disadvantaged,

and not to those who are disengaged or at risk of marginalisation.

This is not specific to Erasmus+ as other benchmarked programmes struggle with the same

challenge. This is partly due to the fact that the participating organisations, even under an

action as inclusive as the European Voluntary Service (EVS)

128

, tend to select the most

motivated and performing individuals. In spite of the specific measures mentioned above to

foster social inclusiveness, the evaluation found that more specific support and adequate

funding would be needed

129

. The lack of spending flexibility limits the capacity to face

unforeseen situations inherent to vulnerable groups (see Efficiency)

130

72% of respondents to

the consultation indicated that there has not been sufficient progress in this area under the

current Programme and that this aspect

deserves more attention and funding

131

. This was

also flagged by the majority of countries in their National Reports

132

To mitigate the limitation of the related programme indicator mentioned above, beneficiaries

were personally asked about their social background

133

. Evaluation survey responses indicate

that the self-reported participation of people from a minority background or with learning

difficulties is rather small. Yet, responses highlight that

disadvantaged learners show more

positive results than the others,

in particular in completing formal learning and in boosting

their self-confidence in their education capacities. However, no significant difference was

noted for them in terms of benefiting from shorter transition periods to employment following

their mobility experience.

These survey results are statistically significant

134

, but the pre/post surveys of school pupils

and the case studies indicate that they are likely due to selection into the programme, at least

for a part

135

and cannot be confirmed with strong evidence from data collected or the literature

reviewed

136

125

The numbers of participants with special needs or fewer opportunities supported were for education and training 8,000 in

2013 (LLP) and 15,000 in 2014 (Erasmus+) with a target of 40,000 in 2020 ; for Youth 18,700 in 2013 (YiA) and 43,000 in

2016 (Erasmus+) with a target of 37,000 in 2020 (Annual Activity Report 2016 of DG EAC).

126

When compared with other sources of evidence (surveys, participation of disadvantaged schools) it seems that the

programme data may have overestimated the participation of disadvantaged. Moreover the data from predecessor

programmes is incomplete, hindering comparison over a longer period of time. See ICF, 6.2.2 about data limitations and 6.3

on other measurements ; ICF, annex 2 for details

127

FR, IT, NL

128

The persons reached through youth exchanges might be more varied but there is less data on their profile. NRS, 3.4

129

OPC: According to the position papers issued by two national level organisations and one EU level organisation

(representing 50 regional and local stakeholders in the field of education and training), the lack of funding has a particularly

negative impact on beneficiaries from disadvantaged backgrounds. Disadvantaged people are less likely to participate on

their own initiative in activities on offer without receiving pro-active guidance. Current traveling grants hinder the

participation of certain targets groups (e.g. youth coming from rural areas) according to: one national level organisation

representing 8 members in the youth and social work sector; one organisation representing 34 NGOs active in the youth and

social sector; and one national organisation representing 22 religious organisations. NRS, 3.4

130

OPC: positions of a non-EU ministry and 34 NGOs from 18 Member States active in the youth /social sector

131

OPC, fig 6.2: A few responses suggested to improve the communication towards people from disadvantaged backgrounds,

especially in rural areas. However, around one-third of the respondents stated that Erasmus+ already does enough in this area.

132

NRS, 3.4: BEde, BEnl, BG, CZ, DE, EE, IE, EL, HR, IT, CY, LV, LT, LU, HU, MT, NL, AT, PT, RO, SI, SK, SE, UK,

IS, FY, TR

133

Learners were asked questions about possible disadvantage such as having a disability, the extent to which their parents

were unemployed or families received social support or whether they repeated a class in the past.

134

ICF, 6.8.2 for detailed survey findings

135

If the grant does not enable the full group of learners to join, it is unlikely that those at the margin would participate.

136

Most of evidence from these two sources is about participation rather than results of disadvantaged groups. In the rare

cases where their results were compared with other beneficiaries the results were positive. A particularly under-researched

area is the multiplier effect from staff to disadvantaged groups.

kom (2018) 0050 - Ingen titel

Most interviewees in case studies believe in the potential of the programme to provide

powerful results for disadvantaged groups. Several of them were clear that more than other

groups,

disadvantaged learners would not have gone abroad to study had it not been for

the programme.

Meanwhile, the participation of organisations with high shares of

disadvantaged learners remains low. The case studies suggest that specific top-up points for

organisations with high share of disadvantaged people should not have been discontinued in

the current programme and that accompanying disadvantaged learners (e.g. in European

Voluntary Service) requires a particular care and specific measures.

On the external dimension, the picture is somewhat similar. All the predecessor programmes

promoted inclusiveness in higher education from a gender perspective. Some promoted other

under-represented, disadvantaged groups. However their specific evaluation noted that the

mobility programmes only partly succeeded in achieving equitable participation

137

Beyond mobility, the other types of action have also

inspired new ways of working with

people from a disadvantaged background, especially in sectors other than higher

education

138

. This was reported by 44% of practitioner respondents

139

, although only 19%

stated that the activity they took part focused on integration of disadvantaged groups. This

implies that there are

significant spill over effects

even where it is not the main purpose to

target these groups (i.e. impacts beyond results).

Lastly,

certain actions perform better in terms of social inclusiveness.

Studies in the Youth

sector found that international youth projects or mobility

140

have had significantly more

impact on young people with fewer opportunities, especially in learning to learn, developing

cultural awareness and building self-confidence (+3.6%), likely due to the lower starting

points

141

. The collaborative platform eTwinning seem slightly more able to reach

disadvantaged groups than the rest of the programme

142

. Some of the adult education actions

are succeeding in engaging with hard to reach groups, but given the low budget share the

scale of these actions have only been small.

5.1.1.3. Outcomes for practitioners

Within Erasmus+,

practitioner mobility has overall seen a significant increase

compared

to the past

143

. It remains

more common in higher education

than in all other education and

training sectors put together, although higher education practitioners have comparatively more

opportunities abroad through other schemes. It should be noted that an important share of

practitioners in the adult education sector teach primarily young people (aged below 24)

144

This could also be related to the limited effectiveness of the programme in reaching out to

practitioners who are not employed in large organisations.

137

Evaluation of the EU Development Co-operation Support to Higher Education in Partner Countries (2007-2014), (2017),

Particip, LeA, ECDPM, Ecorys, already mentioned

138

Practitioner beneficiaries show greater use of strategies to enhance education attainment of students than their control

group. Staff from organisations with high share of learners with disadvantaged background more frequently took part in

activities focusing on disadvantaged learners. A majority of sport projects focus on social inclusion and physical

enhancement activities.

139

Evaluation survey of practitioners: even more true for sport (68%) and youth organisations (71%), but also for schools

(57%) and AE (52%); less for HE (32%); ICF, 6.5.2.2 and fig. 6.32 to 6.34 for breakdown by sector

140

RAY, 2015; Schroer, 2003; Sherraden et al., 2008

141

RAY, 2015a

142

ICF, 6.4.8.2

143

Increase in all sectors where reliable data is available except the sector of adult learning where no additional funding was

available in 2014. There is no reliable data on practitioner mobility for predecessor programmes in the sector of youth and

higher education.

144

ICF, 6.3 for highlights about the profile of practitioners: The teaching staff taking part in VET and HE sectors represents a

more balanced sample of subjects than school staff, where nearly half are foreign language teachers.

kom (2018) 0050 - Ingen titel

In addition,

collaborative platforms expand participation in the programme

(eTwinning,

School Education Gateway

145

, EPALE

146

) and are associated with positive results for

beneficiary practitioners

147

. In particular eTwinning, the platform for schools, attracts a very

large number of teachers and pupils, in the EU and beyond, as mentioned previously. In 11

countries, eTwinning seems to be reaching to more than 10% of all school teachers

148

. As a

stepping stone into the programme, as noted by some interviewees, it does reach to a broader

audience than other types of action

149

. After stagnation in 2011-2013, its audience has been

sharply increasing under Erasmus+, pointing at a strong potential of this kind of intervention

for the future.

Practitioners are even more satisfied with their mobility experience (98%

compared with

93% for learners) and very positive about the contribution to their competence development

according to the monitoring surveys. For most of the competences the differences across

sector are minor

150

Practitioners who (strongly) agree that they developed a given key competence or

skill

Source: ICF calculations based on Erasmus+ participant monitoring surveys. This is a multiple answer question therefore the data does not

add to 100%. N = 227,319

Most National Reports refer to results for practitioners with a high level of similarity with the

benefits listed above, in particular for education and training staff: language skills

151

, new

145

The

School Education Gateway

offers collaborative space and on-line resources to all professionals in the field of school

education:

https://www.schooleducationgateway.eu/en/pub/index.htm

146

Electronic Platform for Adult Learning in Europe

launched under the LLP and animated by National Support Services

for stakeholders to exchange and develop their professional practice:

https://ec.europa.eu/epale/

147

Based on extrapolation from eTwinning findings since primary data was not collected specifically about the School

Education Gateway and EPALE platforms for this programme-level evaluation and even though it is difficult to differentiate

their contribution from other actions of the programme

148

BG, EE, EL, HR, CY, LV, LT, MT, RO, SI, SK, by increasing order; from 2,2% in Germany to more than 35% in Malta;

ICF, fig. 6.24: CSS data on registered teachers per country and Eurostat data on the overall population

149

More than 183,000 schools registered by September 2017 based on CSS Monitoring data. It does not mean that the

registered users actually take part in a project every year. Only in 2016, 29,752 teachers took part in at least one project. ICF,

6.4.9 for profile of users compared to other participants

150

ICF, 6.4.7.5 and fig. 6.17 for a breakdown per sector

151

DE, LV, FI, UK, IS, TR

kom (2018) 0050 - Ingen titel

teaching methods

152

and job satisfaction

153

; as well as for youth organisations’ staff:

professional development

154

and new professional methods

155

Evaluation surveys found in addition, when comparing with a control group, that participation

in the programme is associated with

wider networking

and cooperation,

stronger

attachment to Europe,

and

greater use of digital resources,

as shown below

156

Main counterfactual results for practitioners

Large positive

difference

Medium

positive

difference

Statistically

Medium

significant yet small negative

positive difference difference

Job satisfaction

Networking - frequent Feeling

contact with foreign European

counterparties

Cooperation with

Involvement in

international partners - volunteering or

exchange on topics

community

and methods of

activities

professional interest

Use of digital

resources and

media

Perception of the

importance of

political and civic

participation

Source: ICF beneficiary survey for practitioners

From

virtual collaboration on Erasmus+ funded platforms,

beneficiary practitioners also

observe positive results, though admittedly with smaller intensity than in the case of mobility

or partnerships. Even though it is difficult to differentiate its contribution from other

actions

157

, in certain areas practitioners associate eTwinning with more positive results than

the control group, such as international networking, but also digital skills (the collaborative

platform might attract practitioners that are already digitally skilled, but can also enhance

such skills). Findings of other sources support the positive results from ICF’s general surveys:

another distinct evaluation of eTwinning revealed that a vast majority of teachers find the

platform helpful to network across Europe (64%) or to improve relationships between

teachers and learners (62%)

158

More generally, all practitioners interviewed cited

positive results for themselves as well as

their learners and organisations

159

. They found their Erasmus+ experience as an opportunity

to go ‘outside of the ordinary’ which can be important for retention in the teaching profession,

a crucial issue in many countries

160

. As practitioners appear mostly motivated by individual

improvement, it is not surprising to find a clearer contribution to development of staff than

152

153

BEnl, FR, CY, LV, LT, MT, FI, UK

BEnl, FI, UK, IS

154

BEnl, EL, CY, LT, PT

155

BEnl, CY, LT

156

Involvement in volunteering is also higher but can result from the programme selecting the most engaged staff.

157

ICF’s beneficiary survey covered also eTwinning participants. However most of respondents who took part in eTwinning

also took part in other activities funded by Erasmus+. It is therefore not possible to judge whether the difference is due

specifically to eTwinning. ICF, fig 6.26

158

Education for Change (2013), mentioned above

159

Professional development, motivation, open-mindedness were commonly cited in the context of case studies (most

significant change approach) next to a range of skills. The surveys also found an association (though a weaker one) with job

satisfaction and, in the school sector, wider use of strategies to improve pupil attainment.

160

This is particularly true in schools and VET; less in the adult education, youth and sport sectors where "out of curriculum"

freedom is more easily available by other means.

kom (2018) 0050 - Ingen titel

hard evidence of evolution of pedagogical or institutional practices. Other studies point to

positive impact on teaching practices, social competences and motivation for professional

cooperation

161

Recognition is meanwhile less formalised for practitioners

than for learners by nature.

Only around half of the practitioner survey respondents recorded having received some form

of recognition

162

. While a majority of practitioners noted informal recognition by peers,

effective recognition by the hierarchy is less common. At the same time, nearly half of

respondents state that participation in the programme helped them

achieve new roles or

positions,

which is very positive.

It should also be noted that several of the results discussed in this section also have a spill-

over effect from individual level to organisational level, as elaborated below.

5.1.2. Outcomes for organisations and systems

As for mobility (KA1), the evaluated programmes reached or exceeded the vast majority of

the targets when it comes to the number of projects funded (KA2 and KA3). Compared to

predecessor programmes, this represents a

strong increase in the number of higher

education projects

163

, but also a strong decrease in all other sectors. This is due to the lower

budget available at the start of the programme but also to the fact that

Erasmus+ aims to

focus on funding fewer large-scale projects

164

. Erasmus+ is nonetheless perceived as

providing more opportunities for cooperation than it was previously possible

165

However, the fact that Erasmus+ and its predecessor programmes have produced a high

volume of intellectual outputs is

not necessarily an indicator of project quality

166

. The

panel experts have pointed out several issues with the selection process. They consider that

the quality of applications selected is in general quite average. According to monitoring data,

applications are frequently scored higher than the final outputs. Many projects (other than

KA2 Alliances) lack a continuous quality review, external support, or solid needs analysis.

National Reports refer to some enhancing-effectiveness factors (e.g. experienced staff on the

applicants' side - a condition more often met by large organisations and therefore in higher

education, and good cooperation between partners); they also outline challenges which hinder

effectiveness (e.g. high-quality projects rejected due to lack of funding).

On the contrary, the evaluators find that fewer outputs with clearer added value might

possibly meet more clearly the innovation objective of the programme. According to the

qualitative assessment of outputs carried out by the expert panel, an area for further

improvement of Erasmus+ is the quality of KA2 (and to a more limited extent KA3 which

cannot be fully evaluated at mid-term) outputs and especially their effective

potential for

mainstreaming

beyond participating organisations. Although the programme often produces

something new for participating organisations

167

, this innovation does not enough follow the

state of the art developments in a given sector. While the programme does fund good and

161

In the literature evidence of impact on adult education is somewhat scarce and absent for sport and Jean Monnet; ICF,

6.4.7.

162

85% of practitioners got international experience accepted as training; 58% state it was acknowledge by hierarchy or

peers; 45% state that it helped them attain a new function/ level of seniority; 26% received a financial reward.

163

Prior to 2014, the higher education sector was rather under-represented in the volume of cooperation projects compared to

its share of mobility funding.

164

See above section 3

165

OPC: 80% (n = 558) strongly agreed or agreed

166

Sources to assess project quality: monitoring surveys, case studies, expert panel assessment of KA2/KA3 outputs

167

European cooperation in education and training: added value and impact, L’Observatoire Erasmus+, Pluricité and

Synoptic.Pro, note 6, November 2017

kom (2018) 0050 - Ingen titel

innovative activities

168

, these are relatively hidden among many standard projects. The vast

majority of case studies identified some form of innovation or change in the organisations

reviewed (e.g. use of materials or methods shared during the projects)

169

. Yet often these

innovations remain small scale or light touch and limited to a specific part of the organisation.

For many funded projects, it appears that the main benefits arise from the participation

process rather than the output itself. However, where there are instances of high quality

outputs, these could be better supported upstream and thereafter mainstreamed building upon

the appreciated precedent of the 2016 specific call in the area of social inclusion

170

5.1.2.1. Outcomes for organisations

171

Over 2007-2016,

more than 940,000 organisations

were contracted, at an average of

approximately 115,000 organisations per year for the predecessor programmes but 80,000

under Erasmus+

172

. During the public consultation, 74% of respondents thought that the

Programme improves the quality, innovation and internationalisation in education, training

and youth organisations

173

. When asked in the monitoring survey about

expected changes to

their institutions,

more than 55% (up to 85%) of practitioner respondents stated that most

such changes (10 out of 11) had indeed taken or were taking place

174

. Later, when asked in the

evaluation survey about changes that had happened after their participation

175

, a vast majority

of practitioners once again agreed with a high number of types of changes listed, such as the

introduction of new teaching materials

176

or new assessment methods

177

. In the youth sector,

75% of practitioners agreed with most statements about organisational results. Lastly, most of

other socio-economic actors (companies, public authorities, civil society bodies) stated that

their objectives were (fully) met

178

. For them, the most commonly cited barriers concern the

complexity of the application procedure and the low success rates in applications, which is

due to budget availabilities

179

Furthermore, the

level of cross-sectoral cooperation increased

under the current programme

by 23 percentage points

180

. Paradoxically, interviewees are often sceptical about the reality of

this key evolution. This contrasts with OPC or agency respondents, whom, to a very high

168

74% of higher education practitioners in the higher education sector reported the creation of new research projects and

37% the creation of new spin offs.

169

Case studies were selected at random using a rather broad set of criteria; ICF, 6.5.2.4 and Annex 9

170

An envelope of EUR 13 million was set aside to support the upscaling of good practices from grass-roots level in the area

of social inclusion; Official Journal C 99/05 of 15/03/2016.

171

Results at organisation level were assessed mainly through monitoring survey, practitioner survey (staff were asked not

only about their own development but also about changes in their organisation), survey of socio-economic actors and case

studies, supplemented with more transversal sources (OPC, NRS, literature, etc)

172

ICF, Annex 2; multiple participation of organisations remains unclear, as detailed below in section 4.3 and annex 3

173

n = 1333. Of significance for the Jean Monnet strand, 70% of respondents thought that promoting excellence in teaching

and research in European integration activities had been effective to a (very) large extent (n = 888).

174

The indicator of DG EAC's Strategic Plan is “% of organisations that have developed/adopted innovative methods and/or

materials, improved capacity; outreach methodologies, etc.” However, all findings are based on surveys of staff, not

organisations. Hence there is only a partial match between the official indicator and available data.

175

In contrast to evaluation surveys, monitoring survey ask shortly after participation about what mobile staff intends to do

on return to their organisations. The findings are very positive but they can capture the intention rather than the actual effect.

176

81% SE, 72% VET, 72% HE, 77% AE

177

68% SE, 63% VET, 59% HE, 68% AE

178

ICF, 6.5.2.3 and fig. 6.40: survey of participant organisations other than education, training, youth and sport ones

179

Response patterns differ according to the profile of respondent. Companies cite as most important barrier the lack of

knowledge about the programme. Public authorities first cite lack of staff available to take part in the programme.

180

ICF, 6.10: with 47% of projects in the predecessor period involving multiple sectors/fields, compared to 70% of projects

so far under Erasmus+. A similar finding holds for pairs of organisation linked in a same partnership, with one third of pairs

under the predecessor programme being cross-sectoral partnerships, whereas half of pairs under Erasmus+ are cross-sectoral.

kom (2018) 0050 - Ingen titel

extent, recognise the effectiveness of cross-sector cooperation

181

. In the same vein, the

network analysis shows promising results in the form of an increase in the number of sectors

participating per project. There are notably many more higher education institutions taking

part in other strands of the programme than in the past. Cross-sector participation seems more

challenging in other education and training sectors where organisations are smaller in average.

Reciprocally, youth organisations strongly contribute to the international dimension of the

programme and sport organisations take part in other sectors of the programme even more

than in the their own strand of Erasmus+. This contrast between perception and observation

suggests that there is a

need for better communication

to potential applicants about cross-

sectoral opportunities.

In terms of

improved geographical balance,

the centrality of countries in the programme

network is changing. Small countries and countries from central and eastern Europe are better

integrated

182

. The main exception remains the country coverage of Jean Monnet activities,

with about 25% of its grants going to two countries only

183

The surveys also demonstrate a

clear internationalisation of organisations,

in and outside

the EU

184

. In the monitoring survey, practitioners in general

185

most strongly believe that their

mobility will lead (85%) or has already led (74%) to internationalisation of their sending

institution, as shown in the figures below. For instance, 59% of concerned respondents

believe that the Jean Monnet grant helped them to allocate additional funding for teaching or

research about the EU. The fact that practitioners have developed strong networks, as

demonstrated above, corroborates other findings on the internationalisation of their

organisations. Participant organisations are significantly more likely than non-participants to

be engaged in transnational cooperation, except in the sport sector

186

. This holds true to the

largest extent for higher education institutions

187

and to the least for schools

188

, as confirmed

in case studies. Specific findings for the international dimension of the programme underline

the strong contribution of the predecessor programmes to the internationalisation of higher

education institutions in partner countries through curriculum development and the

improvement of management practices of universities

189

. But even in the case of school

education, a vast majority of respondents believe, for example, that Comenius transnational

partnerships strengthened the European dimension of schools

190

and the survey conducted as

part of the eTwinning platform evaluation has observed, as a result, a certain

internationalisation of schools. It is particularly important for organisations in the youth and

181

80% (n = 471) of OPC respondents (strongly) agreed that the integrated nature of Erasmus+ has strengthened cooperation

across sectors. 90% of agency respondents strongly or rather agree with this opinion.

182

ICF, 6.3 for participation patterns and fig. 6.3 as for the evolution of the centrality of the top 20 countries

183

ICF/JMO, 1.2.3 and fig. 1.5: Italy (16% of grants, considering funding from Heading 1 only) and Spain (9%). Countries in

Northern and South-Eastern Europe are not very often among the JM beneficiaries when compared to Western or Southern

Europe. Meanwhile Jean Monnet covered 82 different countries since 2007 (with Heading 4).

184

Internationalisation

is a broad concept that has a variety of layers. It can mean simply the fact of having contacts with an

organisation from another country but also integration of an international dimension into regular activities (for example

through eTwinning), teaching in a foreign language, hosting staff/ learners from other countries, etc

185

Higher education staff is more optimistic about the internationalisation of their institution, higher education international

staff about spin-off effects, while school education staff believes more strongly in the use of new teaching methods and 76%

of youth staff believe in the effect on the quality of the project they develop in their organisations. ICF, fig. 6.18 and 6.20

186

In contrast to other sectors, there was no significant difference in the likelihood of engaging in transnational partnerships

between Erasmus+ sport participants and non-participants but this needs to be taken cautiously as the control group for sport

was reconstructed in combining the control group for VET and the too few non-participants who responded for sport. See

Annex 3.

187

In the HE sector, internationalisation is more advanced than in other sectors (ICF, Annex 11 on literature).

188

ICF, tab.6.33 for share of organisation with an international strategy (79% v. 68% in control group)

189

Evaluation of the EU Development Co-operation Support to Higher Education in Partner Countries (2007-2014),

(2017),

Particip, LeA, ECDPM, Ecorys, already mentioned

190

ICF, Annex 11 (GES and ZSB, 2010)

kom (2018) 0050 - Ingen titel

sport sectors which work on specific issues for which they have few counterparts in their

countries

191

. Given their scale and the number of organisations reached, it can be concluded

that the programmes strongly contribute to strengthening the internationalisation of education,

training, and youth sectors.

Whether mobility has led or will lead to changes in the sending institution

Source: ICF calculations based on Erasmus+ participant monitoring surveys.

In contrast, the internationalisation effect is less clear-cut on sport organisations. Competitive

sport is already much internationalised independently from any EU intervention, whereas

grassroots sport organisations have fewer international opportunities. In the latter case

Erasmus+ has not reached the required critical mass. Yet, sport organisations' knowledge and

usage of the sport guidelines promoted by the EU (on dual careers

192

or health enhancing

physical activity

193

) suggest some

alignment of sport actions with EU policies

(see fig.

below)

194

. A total of 45% of organisations strongly comply

195

with the good governance in

sport principles

196

191

192

ICF, 6.5.2.4 and Annex 9 (case studies)

http://ec.europa.eu/assets/eac/sport/library/documents/dual-career-guidelines-final_en.pdf

193

http://ec.europa.eu/assets/eac/sport/library/policy_documents/eu-physical-activity-guidelines-2008_en.pdf

194

ICF, fig 6.38 and 6.39

195

ICF, tab. 6.23 and 6.24; organisations are the least compliant with the requirement of an appeal procedure

196

http://ec.europa.eu/assets/eac/sport/library/policy_documents/xg-gg-201307-dlvrbl2-sept2013.pdf

kom (2018) 0050 - Ingen titel

Usage of the EU guidelines in the field of sport

Source: ICF beneficiary surveys (among practitioner respondents aware of these guidelines))

Only 4 out of 10 participant organisations have strong measures in place to combat doping

and violence, while almost half have measures to combat discrimination

197

Share of respondents who comply with the anti-doping, violence and

discrimination in sport principles (% of organisations)

Principle

Anti-doping

Violence

Discrimination

Strong measures

40%

41%

48%

Medium measures

27%

38%

33%

Low measures

33%

21%

19%

Note: Index (1 (strongly agree)-4 (strongly disagree), 1-1.5=strong measures, 1.51-2=medium measures, less is low measures

More generally, another very positive monitoring finding is that overall 91% of practitioners

state that they have shared their own knowledge with students and/or other persons after their

participation in Erasmus+

198

. The programme is particularly seen as

transformational for

youth organisations,

where members of staff are significantly more optimistic about the

contribution of the programme to their organisations than in other sectors

199

. Interestingly,

71% of practitioners in the sport sector state that the programme contributed to new

cooperation with civil society

200

. The literature review points as well to strong positive

impact, particularly on: internationalisation (all sectors), teaching capacity (school education,

VET), and staff development (VET, youth)

201

, although evidence of benefits for organisations

on sport, adult education and Jean Monnet activities is somewhat scarce.

Other results are softer and there is

less evidence of deep changes of institutional or

pedagogical practices

202

. The qualitative findings, however, suggest that a key contribution

that was not measured quantitatively is the self-reflection and

soft-benchmarking

that

happens through the opportunities for practitioners to see how other organisations tackle a

given issue. In all case studies, concrete changes at organisational level were indeed cited

203

In many cases these are light adjustments rather than major structural evolutions; however an

accumulation of

light changes over time can result in a step change.

Frequently noted

mentions include not only internationalisation (most commonly cited)

204

, but also soft

benchmarking, implementation of new tools, materials, increased positive image and with that

197

198

ICF, tab. 6.25 to 6.28

ICF, Fig. 6.29; from 97% of HE International staff to 86% of practitioners of Adult Education.

199

ICF, fig. 6.36

200

ICF, fig. 6.37

201

ICF, tab. 6.30 and 6.31 and Annex 11

202

Findings are less robust, in the absence of indicators of concrete change, such as improvement in student outcomes.

203

ICF, 6.5.2.4 and Annex 9 for a summary of case studies and effects identified at organisation level

204

ICF, tab. 6.29

kom (2018) 0050 - Ingen titel

associated empowerment of practitioners and learners. Other effects for organisations include

the development of national partnerships, cooperation with companies

205

or quality

frameworks specifically for youth work. On average, 58% of practitioners reported as an

outcome the creation of quality assurance approaches

206

The National Reports confirm that Erasmus+ has three main impacts on participating

organisations, namely their internationalisation

207

, the

quality of education and training

they

provide

208

and the

professionalisation of youth work

209

. Both mobility actions and Strategic

Partnerships allow for the development of new methods, often innovative at least for

participating organisations, which can have a positive impact on quality. However, the

evidence gathered at EU level in that respect is not sufficiently strong to assess whether the

programme is transforming the practices of practitioners and organisations. Though

practitioners are positive about the fact that they implement the lessons learnt and results from

cooperation projects and mobility exchanges, there is only anecdotal evidence that the

programme is changing institutions’ approaches to pedagogy and their target groups. Deeper

qualitative work of a larger scale would be needed to see these types of results at final

evaluation stage.

5.1.2.2. Outcomes for systems and policies

210

The general literature shows

little evidence of system level impacts

of preceding

interventions, in contrast to the abundant findings about individual impacts. Where such

evidence exists, it is positive. But it is very fragmented in all sectors, although stronger in

higher education, VET and youth.

This is why, since 2014, Erasmus+ focuses through KA3 on funding

actions that are

specifically designed to have system level effects

211

. Comparable actions existed under LLP

and Youth in Action, but some new features were introduced under the current programme

212

enabling the Commission to address more rapidly emerging needs (e.g. the Paris Declaration,

the New Skills Agenda for Europe). The systemic impact of Erasmus+ is only expected "by

the end of the Programme"

213

and therefore can only be evaluated at final stage

214

. In

particular, most KA3 projects were not finalised at mid-term. Nonetheless some National

Reports outline KA3's policy effects although limited by the relatively small funding allocated

to KA3

215

Not all KA3 case studies could identify system level effects.

Although KA3 has

205

62% of VET staff report influence on cooperation with companies

206

Truer for sport (67%) and above all youth organisations (76%) than for education and training ones; ICF, 6.5.2.2 and fig.

6.35 to 6.37 for breakdown by sector

207

BEde, BEnl, CZ, DK, EE, IE, EL, ES, FR, HR, CY, LT, HU, MT, NL, PL, RO, SI, FI, UK, IS, FY, LI, NO, TR

208

BEnl, DE, EE, IE, EL, ES, FR, HR, LT, HU, MT, PL, RO, SK, FI, SE, UK, FY, NO, TR;

209

BEnl, IE, EL, ES, IT, LT, NL, PL, SI, SK, FI, SE, UK, NO

210

Results at organisation level were assessed mainly through monitoring survey, practitioner survey (staff were asked not

only about their own development but also about changes in their organisation), survey of socio-economic actors and case

studies, supplemented with more transversal sources (OPC, NRS, literature, etc)

211

Including support to European networks and tools fostering transparency and recognition of skills and qualifications

212

Under

Key Action 3

(KA3) the programme funds Structured dialogue between young people and policy makers; national

contact points in relation to EU tools (EQF, EQAVET, ECVET, adult learning, etc.) and policies (Eurodesk, ENIC); support

to EU level civil society bodies; actions aimed to strengthen evidence-based policy making (studies, cooperation with the

OECD); activities related to EU-level OMCs; forward looking projects and projects on specific EU-level priorities

(apprenticeships, social inclusion, etc.).

213

Article 21 (1) (b) of Regulation 1288/2013

214

The vast majority of interviewees, who were provided with a list of KA3 projects, were not aware of them and many were

unclear about the difference between KA2 and KA3.

215

e.g. in the youth field: BEnl, IE, FR, LV, NL, PL, PT, NO, TR. However, a too low number of KA3 projects result in

limited policy-effects (e.g. HR, HU, PT); NRS 5.1.6, 5.2 and 5.5

kom (2018) 0050 - Ingen titel

the highest potential to produce systemic impact

216

, further involvement of decision-makers is

required. It seems therefore

too early to conclude on the additional systemic impact

brought specifically by Erasmus+.

However, even if effectiveness varies across instruments and countries (some use them much

more strategically than others

217

some policy networks financed through Erasmus+ have

proven influential since 2007:

the national contact points for EQF

218

or EQAVET

219

, the

Bologna experts

220

or the partnership between the Commission and the European Youth

Forum

221

. Studies and cooperation with OECD

222

funded by Erasmus+ and LLP have provided

Member States with stronger evidence, according to the evaluation of ET2020

223

On the international dimension, the general finding is that the projects funded by previous

programmes have

contributed to policy and institutional reforms in partner countries,

through improved quality assurance, standardisation of higher education and a rapid extension

of EU-supported Bologna principles. This has had an evidenced impact on the harmonisation

of programmes and curricula in partner countries as well as on mobility and the recognition of

studies and qualifications within and between non-European regions and Europe

224

Furthermore, the effectiveness of the programmes evaluated should also be assessed through

their

contribution to policies via the Open Method of Cooperation (OMC),

which they

have substantially supported over the period 2007-2016

225

. The evaluation of the OMC in

education and training (ET2020) noted that according to national stakeholders the outputs of

OMC activities have been used to influence national policies and practices. Therefore the

programme plays a major role in supporting financially the OMC process, which in turn

influences national education and training policies,

including those priorities of ET2020

which are not easy to tackle through mobility and cooperation actions (e.g. policies on

qualifications design or early school leaving policies). The programmes had also a certain

systemic effect

in the field of youth.

The most recent evaluation of the European Youth

Strategy

226

concluded that the OMC was rather positive about the influence on national

policies and structures in the youth field, while the results in the youth field might not always

be measurable in a way that is easily comparable with the area of education and training. At

the same time, the activities funded by Youth in Action and Erasmus+ were identified as

strong pillars of the OMC, regarding the structural dialogue with young people and the

increasing use of the

Youthpass

to promote learning outcomes of volunteers. Sport is too

recent a programme sector to be evaluated at interim stage in relation to systemic impact

227

216

e.g. LT, NO, TR

217

The system level impact of policy instruments, such as the EQF or ECVET, is to a large extent determined by national

policy factors and priorities. The fact that Ersamus+ finances the related networks makes for an indirect causality link.

218

ICF (2012)

Evaluation of the Implementation of the European Qualifications Framework Recommendation

219

ICF (2013)

Evaluation of implementation of the European Quality Assurance Reference Framework for Vocational

Education and Training (EQAVET);

EQAVET national points are only funded through the programme since 2016.

220

ICF (2013)

Study to examine the impact of the national teams of Bologna experts on the implementation of the Bologna

process;

National Reports mention also the support to the Bologna and Copenhagen processes (BEnl, IT, RO, FI, NO, TR)

221

Ecorys (2013)

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements

222

E.g. PISA and PIAAC surveys and their subsequent analysis, both highly influential sources of evidence

223

Ecorys (2014)

Interim Evaluation of the strategic framework for European cooperation in education and training

224

Evaluation of the EU Development Co-operation Support to Higher Education in Partner Countries (2007-2014),

(2017),

Particip, LeA, ECDPM, Ecorys, already mentioned

225

The programmes fund at EU level working groups and research and co-fund at national level the implementation of EU

tools (e.g. National Contact Points for the European Qualifications Framework); ICF, 6.5.3

226

ICF (2016)

Evaluation of the EU Youth Strategy and the Council Recommendation on the mobility of young volunteers

227

There is no formal OMC in the field of sport but similar activities take place under the EU Work Plan for sport. The

Commission adopted

a report on its implementation

in January 2017. The responses of 105 policymakers and stakeholders

suggest that the Work Plan is perceived as influential by the respondents, although no concrete examples of effects on sport

organisations or sport policies were given.

kom (2018) 0050 - Ingen titel

Overall,

88% of respondents to the consultation agreed that Erasmus+ has increased

systemic impact compared to the predecessor programmes

228

, whilst 46% estimated that

the programmes had been effective in modernising education, training and youth systems

229

As shown in the figure below, around

80% of the key informant interviewees saw some

positive influence of the programme on policy,

of which half were able to cite concrete

examples. This can be regarded as a good performance

230

. Interviewees tend to be more

positive about the systemic influence of the programme

in smaller and less wealthy

countries.

Generally, most frequent examples cited dealt with EU tools related to

qualifications. It seems that interviewees no longer think of systemic changes as resulting

from the scale of funded mobility. Lastly, the examples remained particularly rare in the sport

sector

231

Share of key informants interviewed who were able to cite concrete policy/ system changes

linked to Erasmus+ or its predecessor programmes

Legend: As positive were coded responses which named a concrete and influential project or a clear area of influence supported via one of

the programmes. As neutral were coded interviews that thought there was some influence but did not give a concrete example. As negative

were coded interviews which said they are not aware of any such influence.

The interviewees mainly cited

four types of changes at system level

232

. Occasionally, even a

single funded project can be used as a

strategic input

at national policy level (e.g. no

alternative resource to design work-based learning legislation would have been available in

Latvia). More frequently, examples were found where an effort has been made to pool

together the findings of several projects to accompany a

systemic development

inspired by

European priorities (e.g. the Polish reform based on ECVET). Thirdly, the

scale of mobility

actions can trigger a change in national policy to remove barriers (e.g. Erasmus+ beneficiaries

are no longer charged a visa fee when going to Turkey). Lastly, programme funded activities

can

raise awareness

of an issue at policy level (e.g. early development of Lifelong Learning

Platforms at regional level in Spain).

System level influence via individual cooperation projects remains an exception.

This

does not so much depend on the type of action but rather on the own merits of a given project.

Effects at system level are more likely to happen if the project is aligned with country

priorities

233

. Even when small scale projects prove to be innovative, unless they had a high

level influential stakeholder involved from the beginning

234

they typically would not have had

228

229

OPC: n = 424

OPC: n = 1,206; it must be noted nevertheless that 262 respondents rather selected “no opinion” and that, as seen under

5.1, no more than 44% regarded this objective as extremely relevant.

230

The interviews were carried out mainly with key policy-makers who generally were not directly involved in projects.

231

Relatively few projects focus on systemic issues stated in the programme objectives (combating threats to sport,

governance; dual careers of athletes). An high share of sport organisations taking part are besides small local bodies.

232

ICF, 6.5.3.1 for more examples; ICF, 6.6 for effectiveness per sector and type of action

233

National Reports of FI, NO

234

Outlined in several National Reports, be it at local (LT, NL, NO, TR) or national (CY, IS) level

kom (2018) 0050 - Ingen titel

the capacity to reach out effectively to decision makers. It should nonetheless be noted that

Knowledge Alliances

and

Sector Skills Alliances,

two actions of a new kind and more

ambitious within KA2, could not be evaluated at mid-term but might have a certain impact on

systems at the time of the final evaluation of Erasmus+.

The evaluation shows that a better way for the programmes to achieve system level impact is

reaching out to a critical mass of practitioners and organisations.

This has been the

case in higher education where mobility actions, by their volume, led to system level changes

e.g. implementation at national level of European credit systems (ECTS) and joint degrees

(Erasmus Mundus). Erasmus increased the employability advantage for Erasmus alumni over

their non-mobile peers by 45% and half of all European graduates who studied or trained

abroad benefited from it according to ex-post impact research

235

. In several of the other

sectors, the programme is also close to reaching a critical mass. KA1 alone supported in 2014

1.6% of higher education students, close to 1% of VET students and 0.36% of school staff

236

However the impact on VET and adult education systems is more difficult to detect because

of the scale and the heterogeneous nature of these sectors, which would require more targeted

actions considering available funding. In KA2, eTwinning is also instrumental in reaching out

to a critical mass of teachers, as described above. Other types of activity do not reach the

same volume, but can be influential in areas where system level changes are expected

237

. For

the youth sector, although an impact on systems is observed through other channels, it is

difficult to evaluate to what extent the programme succeeds in reaching a critical mass

238

Other sectors such as adult education or sport remain too small to see system level results

through this channel.

Last but not least, Erasmus+ is expected, in line with its legal basis, to contribute to the

Europe 2020 headline target in education

239

, namely (a) higher education attainment and

(b) tackling early school leaving. Though the EU countries seem well on target to deliver on

these two dimensions, the contribution of the programme and its predecessor to their

evolution is

mostly indirect.

The choice of these overarching indicators for the programme

can indeed be questioned in terms of testability and plausibility. Though the programme

recitals cover both targets, this is done together with many other priorities. It is true that, as

seen above, some countries did make policy changes linked to these two benchmarks. The

programme also reaches a high number of learners via KA1 and KA2, whose

attitude

towards education/training is more positive

when compared to control groups. Such a

positive attitude can be analysed as a precursor of education retention and as reducing the risk

of drop-outs. However early school leavers or people not attracted by higher education are

more likely to be found among disadvantaged people. As seen above, both the share of

organisations targeting hard-to-reach groups and the share of individuals particularly at risk of

exclusion are low within the programme. The programme does improve attractiveness of

participating higher education institutions but it cannot be deduced that it attracts people who

would not have otherwise studied at higher education level. Lastly, the review of selected

projects reveals a weak alignment between the transnational partnerships funded and these

two key priorities. Though some potential does exist, the causality link is only indirect.

As regards

other ET2020 European benchmarks

mentioned in the general objectives of the

programme, employability of young people has also improved in recent past (77% are

235

DG EAC's Annual Activity Report 2016; ex post Erasmus Impact Study (2014)

236

ICF, tab. 6.33

237

In the school practitioner survey, 61% stated that they improved governance and quality assurance approaches. In the

evaluation sample, 7% o HE staff who is in departments that teach about the EU has applied for Jean Monnet grant.

238

The overall population of youth staff/organisations concerned is not determined (no comparator).

239

The Europe 2020 strategy aims to reduce early school-leaving rates to a level below 10% (10.7% in 2016) and to enable at

least 40% of 30-34 year-olds to have completed tertiary or equivalent education (39.1% in 2016).

kom (2018) 0050 - Ingen titel

employed i.e. +1 percentage point from 2014 to 2015). In contrast, other impact indicators

more related to social inclusiveness have registered deterioration over the last years, with

worse levels of basic skills and with lower rates of participation in youth out-of-school or

physical activities

240

, which could justify a strengthened focus on social inclusion.

5.1.3. Transversal questions about effectiveness

When considering all objectives together,

71% of respondents to the consultation thought

that the programme is achieving its objectives

to a (very) large extent. Education and

training providers who responded to the consultation claimed that lack of funding and

insufficient language skills are among the barriers hindering the achievement of Erasmus+

objectives.

5.1.3.1. Most effective actions and fields

The majority of respondents and countries

241

believe that mobility is essential

in ensuring the

effectiveness of Erasmus+

242

. Actions cited as most effective by agency respondents are

mobility ones

243

and strategic partnerships, with the latter mentioned with more mixed

views

244

. Moreover, longer term mobility actions are associated with stronger results for

learners in beneficiary surveys and by a few countries

245

. For practitioners, what matters most

seems to be participation itself, irrespective of the type of action

246

. Lastly, KA2 Alliances are

perceived by agencies as being less effective for organisations

247

. It can meanwhile be noted

that respondents tend to consider the actions closest to their interest as more effective.

Comparing the relative effectiveness of Erasmus+ fields cannot be done without keeping in

mind the diversity of needs they address and funding available at EU level. Nevertheless,

30 years of Erasmus with a larger and targeted budget allocation makes of

higher

education

248

the strongest field in effectiveness. In contrast, less funded or more recent fields,

such as adult education or sport, not only miss the critical mass but also a similar strong focus

to guarantee an equivalent level of effectiveness.

School education and VET

fields appear in

an intermediary position, indicating at this stage the

highest potential for improving the

overall effectiveness of an EU programme in the future

249

More generally, the respondents to the consultation have considered

factors which could

maximize the effectiveness

of the programme, as follows:

240

241

See DG EAC's Annual Activity reports:

https://ec.europa.eu/info/publications/annual-activity-reports-2015_en

NRS, 5.2 and OPC, 3.3 for nuances per objective: from developing skills (86%) or promoting the European dimension

(80%) to modernising systems (46%). ‘No opinion’ responses were excluded. Slightly more respondents who had ‘detailed

knowledge’ of the programme thought that objectives were being achieved than those who had ‘some knowledge’.

242

BEnl, IT, CY, HU, MT, SI, FI, UK, TR whereas a few see KA2 as more effective due to long-lasting effects (RO, FI).

Lastly, DK, EE and NO do not found KA1 and KA2 different in effectiveness.

243

Covering KA1 and KA2 mobility activities with the exception of the Student Loan Guarantee Facility

244

ICF, tab. 6.41 to 6.44 for details and breakdown per sector; the key reason why KA2 strategic partnerships are appreciated

in the school and adult learning sector is that they offer mobility options. When asked about system level effects, only ¼ of

respondents cited actions under KA3. The majority cited KA2 types of actions.

245

ICF, tab. 6.40; National reports of CY, NL, FI

246

However two issues of method affect the breakdowns of preference per type of action for practitioners. Staff frequently

participates in multiple types of actions and the sample sizes are rather small for some types of actions.

247

Less clear when it comes to reporting about least effective actions, as few respondents opted to name some.

248

Without including Jean Monnet activities and the Student Loan Guarantee Facility in this judgement

249

Confirmed by several National Reports (BE, CZ, LT, PL, RO, FI, NO, CH)

kom (2018) 0050 - Ingen titel

To what extent do you think the following topics need to be addressed to maximize the impact

of any successor to the Erasmus+ Programme?

Source: Erasmus+ OPC survey, 28 February – 31 May 2017, ICF analysis. Base indicated next to each topic

According to the public consulted,

the programme should be better funded, more inclusive

and

user-friendly

while its opportunities should be better known and its outcomes

widely exploited

and publicised.

5.1.3.2. Visibility and dissemination of results

Erasmus+ is

very well-known across Europe

and visible beyond. A majority of citizens have

heard about Erasmus+ and student exchange programmes come third in the list of the EU's

positive results

250

The dissemination and exploitation of project results are

much more effective and

systematic with Erasmus+ compared to the predecessor programmes,

albeit with nuances

according to the geographical level considered

251

. The monitoring was much less performant

in 2007-2013, when it was scattered over different tools covering only parts of previous

programmes. Good practice examples are used to raise awareness of the general public; to

guide potential applicants; to inform the work of practitioners, researchers, social partners,

organisations and policy makers. This new approach has been recognised in the public

consultation

252

and by the experts

253

. It addresses the objectives of transparency,

accountability and visibility

254

, drawing on the lessons learnt from the interim evaluations of

predecessor programmes.

250

Respectively 53% and 25% of respondents to the standard Eurobarometer. Visibility is evidenced on social media.

251

Sources: Desk research, KI interviews, case studies, expert panel assessment and agency survey; ICF, 6.9

252

Number of OPC respondents mentioning the empowerment of policy-makers through dissemination when asked to

indicate what changes happened for systems in each sector: SE (31); VET(41); AE (34); ICF, Tab. 5.1

253

A conclusion from the expert panel assessment workshop is that the branding enhances the attractiveness of outputs.

254

77% of agency respondents strongly or rather agree that the strategy has contributed to better communication of the

projects results and 58% that it would have met its objective of transparency and accountability.

kom (2018) 0050 - Ingen titel

EU level,

the establishment for the first time of an overarching dissemination and

exploitation

strategy

255

with regularly reviewed indicators and a single project results

platform

256

represent clear improvements in comparison to the past

257

. Specific steps have been

taken both by the Commission and the Executive Agency to ensure that the exploitation of

projects’ results effectively serves to inform policy making

258

The

project results platform

offers a common and accessible space for disseminating most

project results whilst also highlighting good practices and success stories (including from LLP

and Youth in Action). Its value in comparison to the past lies in structured and selected

material. Insights gained from EU interviews suggest that the platform is actually used to

inform the work of policy makers at EU level. Content-wise, the main limitation is the lack of

complementary external reference materials that could inspire users and a lack of success

stories in certain sectors (e.g. adult learning) due to the insufficient quality of reports.

national level,

the assessment is also positive about the strategy, setting clear rules for the

selection of good practice examples. However, the evidence of the

exploitation of project

results by policy makers

259

and the effective engagement of the latter when they are not

included in the project itself is not clear. The vast majority of National Agencies mainly

disseminate examples of good practices to potential applicants or beneficiaries, and only

rarely to inform policy making bodies. Moreover, though valued by over half of respondents

as shown in the graph below, 37% of agency respondents do not agree that the dissemination

platform is user-friendly. This sentiment, more mixed than at EU level, concurs with insights

gained from national interviews and reports

260

Source: ICF experts’ survey

255

256

DG Education, Youth, Sport and Culture, March 2015; ICF, 6.9, table for a description per level of intervention

Initially named VALOR following the LLP interim evaluation and building on predecessors (ADAM, EVE), the

Project

Results Dissemination Platform

acts as a repository for all funded projects under Erasmus+ and some of those funded under

its predecessors, highlighting best practices and success stories in terms of policy relevance, communication potential, impact

or design:

http://ec.europa.eu/programmes/erasmus-plus/projects

257

Under Erasmus+, dissemination channels notably include in addition to the project results platform, DG EAC or EACEA's

websites, social media, collaborative platforms (eTwinning, School Education Gateway, EPALE, Youth Portal), videos,

newsletters, electronic and paper-based publications, etc. A distinct evaluation of this strategy was on-going in October 2017

and preliminary evidence seems to confirm these findings.

258

E.g. the project results platform enabled in 2015 to retrieve 160 projects that supported integration of asylum seekers.

These were used for policy communication.

259

Only 6% of agency respondents actually strongly agree with the statement that the programme results are adequately

exploited for policy purposes.

260

BEnl, IE, MT, FI. The search function of the platform, for example, was described as difficult to use.

kom (2018) 0050 - Ingen titel

Furthermore National Reports refer not only to the dissemination platform, but also to other

dissemination channels (e.g. events, websites, social media). Despite all tangible

dissemination efforts, some observe that the

results of projects are seldom further

exploited

261

. While this can be due to factors that are external to the programme itself (e.g.

professionals’ lack of time) some regret a lack of concentration of resources on the most

interesting projects

262

and an insufficient involvement of national authorities and/or

municipalities in their dissemination

263

project level,

the expert panel confirmed that

dissemination is effective in 90% of the

cases

they assessed

264

. The platform is also of use for people looking for partners.

Interviewees and participants were aware of the dissemination and exploitation strategy

requirements, but often perceived these as

burdensome.

Despite the standardised processes in

place (except in the case of Jean Monnet activities

265

), project leaders often do not have the

capacity to effectively exploit results. This might explain why, according to OPC respondents,

the dissemination of results is one of the aspects of Erasmus+ that

should be improved

in the

future.

261

262

BEnl, DE, IE, HU, MT, NL, PL, PT, RO, SI, SK, FI, FY, NO, TR

BG, CY, HU, FI, SE, FY, TR

263

SI, IS

264

The vast majority of projects reviewed: had all or some projects’ products disseminated online (90 out of 100) and none of

the projects without online outputs were funded under the current programme; 93 had several (55) or at least a few

dissemination activities (38). Outputs were made available fully (57) or partially to the target group(s) and the wider public in

most cases. 85 outputs were assessed as of high to good quality (41 and 39 respectively) and generally considered relevant

and coherent, but nevertheless bringing rather limited innovation.

265

Even though there is a requirement of disseminating the JM results to audiences beyond the stakeholders directly

involved, no specific targets are defined.; ICF/JMO 2.1

kom (2018) 0050 - Ingen titel

5.2.

Relevance

The assessment of

the relevance

266

of Erasmus+ is generally positive

267

in regard to both its

alignment with policy priorities and its adaptability to socio-economic needs. The relevance

of the programme, compared where appropriate with its predecessors, has been assessed

against 8 evaluation questions that overall looked into the extent to which Erasmus+ is:



aligned with EU and national policy priorities

responsive to the needs of stakeholders, sectors and society at large

visible

suitable for attracting different target audiences, including hard-to-reach groups

The vast majority of interviewees and most of OPC respondents

268

consider that the

programme shows a

high degree of alignment with key EU priorities

namely as set in the

Europe 2020 Strategy, ET2020 Joint Report, the Youth Strategy, and to some lesser extent,

the Union's external action, including its development objectives, and the EU Work Plan for

sport

269

Erasmus+ is

strongly relevant to the rationale of the European project:

it fosters the intra-

European integration through learning. For instance, Erasmus+ and its predecessors have

played an important role in supporting the Member States and beyond in implementing the

Bologna process with the introduction of the ECTS. Had it not been for Erasmus+, much

fewer students would have benefited from learning mobility or would have been much more

oriented towards non-EU English-speaking countries, according to interviews

270

. The

consequences of a scenario where Erasmus+ would not exist are elaborated further in the

section on its European added value below.

The international higher education dimension of the evaluated programmes has contributed to

achieving socio-economic development objectives.

The EU programmes for international

cooperation in higher education have contributed to fostering regional integration as a key

objective in the EU’s relations with partner regions and supporting EU policies in the field of

legal migration and mobility

271

266

Relevance

analysis looks at the relationship between the needs of participants and problems in society and the objectives

of a programme and hence touches on aspects of design.

267

Consensus among key informants, at EU and national level, concurring with findings of the agency and experts surveys,

the OPC and the synthesis of the NAUs reports; ICF 4, NRS 3, OPC 3.1 and 5.1.3

For instance, agencies’ respondents perceive a clear alignment between the programme and the EU priorities, e.g. in

improving the quality of education and training (74% of respondents perceiving this as strong alignment), increasing the

number of those who benefit from mobility (70%), opening up education and training to the wider world (68%). Similar

insights are also available for the youth field: support to volunteering and youth organisations (83% of respondents rated it as

‘strong’), engaging participation of young people in policy making (55%), non-discrimination and access to social rights

(52%). Conversely, least alignment is found in the following areas: increasing cognitive skills in reading, math and science

(28% whilst 24% claimed they ‘don’t know’), increasing higher education attainment (44% whilst 33% ‘don’t know’),

capacity building in higher education in partner countries (39%, whilst 39% ‘don’t know’), access to quality employment –

youth strand (28% whilst 14% ‘don’t know’), capacity building of youth sector in Partner countries (34% whilst 14% ‘don’t

know’). ICF, Fig. 4.1 and 4.2

268

96% (n = 606) of OPC respondents agreed that Erasmus+ is well aligned with EU policies and priorities.

269

Sport pilot projects launched in 2016-2017 (dealing with health-enhancing activities or refugees) might help mitigate this

lower perception of relevance.

270

ICF, tab.4.6 - Illustrative examples of perceived (mis-)alignment of Erasmus+ with national policies (KIIs and case

studies)

271

Many partner countries which are not members of the EHEA (European Higher Education Area) are influenced by

Bologna principles through Erasmus+. See,

Evaluation of the EU Development Cooperation Support to Higher Education in

Partner Countries 2007–2014,

Particip, LeA, ECDPM, Ecorys (2017), already mentioned.

kom (2018) 0050 - Ingen titel

Other evidence

272

and the consultation

273

reveal in addition that

the alignment with

European policy priorities is stronger with Erasmus+ than with its predecessors,

shown below.

Share of projects that focus on issues that are high on the European policy agenda

40%

32%

36%

38%

30%

20%

10%

Very high

Erasmus+ (n = 977)

Source: ICF, experts’ survey

Percentage of

respondents

High

Average

10%

Low

Very low

14%

I don't

know

Predecessor programmes, 2007 - 2013 (n = 902)

On the one hand, the

review of topic coverage

of funded projects undertaken for this

evaluation and the expert survey confirm that these align to a high extent with EU-level

general

priorities, such as promoting lifelong learning, equity or social cohesion. For instance,

73% of expert respondents as opposed to 52% in the case of predecessor programmes,

estimate that a high share of the funded projects which they assessed align with EU-level

priorities. This is also consistent with the data analysis of cooperation projects, albeit with

variations in topic prioritised before and after 2014, as shown in the table below:

First 10 topics of cooperation projects under the Erasmus+ and LLP

LLP

Erasmus+

22%

18%

17%

15%

ICT - new technologies - digital

competences

Curricula/teaching methods

Creativity and culture

20%

18%

11%

EU citizenship

Teaching and learning of foreign

languages

Creativity and culture

Environment / sustainable

development

ICT - new technologies - digital

competences

Health and wellbeing

Reinforcing links between education

and working life

Comparing educational systems

Teaching and learning of foreign languages 11%

Skills and labour market issues

EU Citizenship

International cooperation, international

relations, development cooperation

10%

Entrepreneurial learning - entrepreneurship 9%

education

272

Stronger alignment than its predecessors according to agencies' respondents e.g. in opening up education and training to

the wider world (64% of programme agencies’ respondents perceive stronger alignment), enhancing creativity and

innovation, including entrepreneurship at all levels of education (61%), increasing the number of those who benefit from

mobility (56%), or in the youth field strengthening cooperation with policy makers (58%). Results are more balanced e.g. in

improving the quality of education and training (52% as opposed to 44% who feel that alignment is the same), capacity

building of higher education in partner countries (51% as opposed to 43%), increasing employability of young graduates

(51% as opposed to 40%) or in increasing the transparency of qualifications and skills (48% as opposed to 48%). Conversely,

areas where alignment is felt to be the same than in the past by most respondents include increasing cognitive skills in

reading, mathematics and science (60% versus 34%), facilitating access to education and training for all (57% versus 32%),

increasing higher education attainment (58% versus 37%), reducing early school leaving (53% versus 43%), increasing

participation in lifelong learning (22% versus 55%), and in the youth field: non-discrimination and access to social rights

(12% as opposed to 81%), access to quality employment (24% as opposed to 68%), access to quality education for all young

people (27% as opposed to 68%).; ICF, 4.2.1 and ICF, annex 4

273

83% (n = 468) of OPC respondents (strongly) agreed that Erasmus+ has stronger emphasis on promoting youth

employment compared to predecessor programmes.

kom (2018) 0050 - Ingen titel

Natural Sciences

Sport

Intercultural/intergenerational education

and (lifelong)learning

Health and wellbeing

projects)

Source:

ICF

calculations

based

the

Epluslink

and

LLPLink

274

total

decentralised

In red: significant decrease under Erasmus+. NB: most projects of the sport strand are centralised, so do not appear here.

On the other hand - when analysing projects more in-depth, beyond statistical data - some key

challenges are only addressed marginally in practice. The extent to which the projects

reviewed

align with more

specific

priorities (e.g. early school leaving, higher education

attainment, adults’ participation in adult learning, etc.) is not clear enough. For instance,

across the sampled projects in higher education, none focused on the attainment headline

target

275

and in school education, only a very small number dealt with early school leaving

276

There is still room for improvement in the number of activities in topics on key competences

and basic skills, even though these have been a priority in many calls for project applications.

Lastly, in the sport field, respondents at national level often felt a gap between broad EU

priorities and the need for more customised ones

277

Most interviewees also consider that Erasmus+ has proven to be

flexible in adjusting to new

emerging EU-level challenges

278

(e.g. refugee crisis, intolerance or violent radicalisation

were often mentioned), particularly through its annual work programming

279

. Several

National Reports value as well the flexibility of Erasmus+ objectives to address emerging

challenges

280

Examples of flexibility and relevance to the EU policy agenda

Following the

Paris Declaration on promoting citizenship and the common values of freedom,

tolerance and non-discrimination through education

281

, the prevention of radicalisation through

education, youth outreach and sport activities was identified as a priority within the updated EU Agenda

on Security. More than EUR 200 million was allocated to support Erasmus+ projects fostering inclusion

and promoting fundamental values, on top of a specific EUR 13 million call launched in March 2016 to

identify and spread best practices.

In the context of the

refugee crisis

of 2015, Erasmus+ started as of 2016 to provide online language

assessment and courses for newly arrived third country nationals, through the online linguistic support

(OLS) in dedicating 100.000 additional licences for a period of three years.

274

Data included for all actions for which ‘topic’ information was available, including LLP (COM06, COM07, COM13,

GRU06, LEO04) and Erasmus+ (KA200, KA201, KA202, KA203, KA204, KA205, KA219) mobility actions

275

Europe 2020 targets of "reducing the share of early leavers of education and training to less than 10%" and "increasing the

share of the population aged 30 to 34 having completed tertiary or equivalent education to at least 40%" by 2020

276

Nonetheless it is the third most popular topic in the school field in the period 2014-2016: 553 projects were funded in total

under the topic "Early school learning/combating failure in education" for EUR 84 million.

277

ICF, tab. 4.6, quotations from KIIs: "Erasmus+ seems to focus more on scientific issues or general issues in sports"; "At

the grassroots level, there are continuing issues around retaining volunteers to support local clubs and local organisations (not

reflected in Erasmus+ priorities)"

278

Particularly positive among EU level stakeholders in school and higher education sectors, whilst perceptions are more

mixed among EU-level stakeholders in the sport sector and among national stakeholders. For instance, the good alignment of

the programme with the Modernisation of Higher Education communication, the New Skills Agenda or the increased

attention paid to work-based learning (ErasmusPro) were often valued.

279

Although challenges are not always explicitly stated; for instance, the 2017 Programme guide briefly presents the

European Solidarity Corps (part describing EVS) as a means to help "resolve challenging situations" without specifying the

latter.

280

EE, IE, NL, SI, NO, etc

281

http://ec.europa.eu/education/news/20160316-paris-declaration-education_en

kom (2018) 0050 - Ingen titel

Meanwhile, 79% of OPC respondents indicated that in their view there are no

other priorities

or actions that Erasmus+ should address.

For the remaining share of respondents, the main

suggestions

282

included in descending order of frequency: i) promoting European

citizenship

and civic education; ii) fostering inclusion and

diversity,

especially the integration of

migrants; iii) linking education more strongly with

enterprise

and promoting international

internships.

In contrast, the context has changed radically since the Jean Monnet programme was founded.

There is now substantial volume of teaching and research about the EU, particularly in EU

countries, irrespective of Erasmus+ support. Because of their focus on higher education, the

evaluation shows that the

relevance of Jean Monnet activities could be enhanced

focussing more on other segments of the learning continuum such as schools and VET. Given

the low turnout in the European Parliament elections

283

and the lack of trust in the EU

institutions

284

, there is a need to strengthen Europeans’ understanding of the EU. This is the

underpinning challenge that Jean Monnet aims to address. Overall, only 56% of Europeans

state that they understand how the EU works - a number that has nevertheless improved by 9

percentage points since 2004. However, Jean Monnet activities target a population that has

been tested as already more knowledgeable about the EU than non-beneficiaries. One third of

staff respondents dedicate substantial share of their teaching to students outside EU-related

fields of study. However, most of those reached remain students who would still have studied

European integration

285

The extent to which all the ambitious objectives of the programme are achievable over time is

also sometimes questioned. Some stakeholders

call for greater emphasis on the promotion

of social inclusion.

In the aftermath of the 2008-2009 economic crisis, Erasmus+ was

primarily focused on economy-related priorities. Since 2016, a shift towards social inclusion

is observed in its implementation. Responses seem to indicate that the social inclusion

objective could be explicitly defined and go together with a higher and more visible budget to

achieve a more tangible impact

286

. However, views remain mixed on this issue. Many

stakeholders, not only from the youth sector but also from education and training, believe that

too much emphasis is put on employability to the detriment of the cultural value of education

or the youth sector; whereas some other stakeholders insist on the continued relevance of

employability. Some National Reports refer to needs that could be further addressed,

including i) the objectives connected to the

Paris Declaration

and prevention of violent

radicalisation

287

; ii) traineeships to enhance labour-market

inclusion

288

; iii) transversal, social

and personal

skills

– hence aligning well with data collated by the external evaluation at

national level.

282

For more suggestions see OPC 3.1 and NRS 3.1

283

http://www.europarl.europa.eu/elections2014-results/en/turnout.html

284

Standard Eurobarometer 86 – Wave EB86.2

285

ICF/JMO, 2.1 and fig.2.1 and 2.3

286

Several interviewees welcome the KA3 call on social inclusion (13 million EUR in 2016) but a quarter of them see it as

too modest to effectively achieve tangible impact. Other social inclusion-related measures (e.g. social inclusion emphasis

recently put on Erasmus+ KA2 transnational partnerships) were very scarcely mentioned. ICF, 4.6 about the efforts to

strengthen the social inclusion dimension of Erasmus+ since 2016

287

FR, LU, SK

288

BEnl, FI, IT, TR. In particular, VET-higher education permeability was felt insufficiently addressed. There is however no

consensus on labour-market inclusion. CZ, EE, ES, HR, IT, LT, PL call for tighter link with companies. A number of

interviewees expressed the view that Erasmus+ is probably not the best instrument for targeting the labour market.

kom (2018) 0050 - Ingen titel

The

alignment of Erasmus+ objectives with national policies

is almost equally recognised,

by most of respondents

289

. Overall, a vast majority of experts surveyed estimate that a high

share of the projects funded align very highly to fairly with national level priorities

290

. Experts

felt that there has been an improvement on this matter between Erasmus+ and its predecessor

programmes, as illustrated below.

Share of projects that focus on issues that are high on the policy agenda of my country

50%

40%

30%

20%

10%

40%

25%

13%

Very high

Erasmus+ (n = 983)

High

Average

Low

Very low

I don't

know

16%

39%

Percentage of

respondents

Predecessor programmes, 2007 - 2013 (n = 906)

Source: ICF experts’ survey

Nevertheless views from national interviewees are more mixed

291

, with organisations being

more complimentary than policy makers

292

or agencies’ respondents

293

. The most noticeable

difference is found in the sport field where no stakeholders and only one out of 4 policy

makers see Erasmus+ as strongly relevant to solving national challenges.

Most of the national authorities

294

consider as well that Erasmus+ decentralised actions

295

are

relevant to country needs,

although fewer countries refer to national priorities in the adult

education, sport and Jean Monnet sectors. Furthermore some countries ask for more flexibility

to allow the programme to adapt better to local needs

296

Examples of relevance to national needs or need for more flexibility

Relevance to national needs

In Denmark, the internationalisation of education is a

national priority. The Danish higher education

institutions receive an “internationalisation rate” for

every student they receive or send, including under

Erasmus+.

Ireland explains that Erasmus+ growing emphasis on

youth employment has aligned the programme with the

Need for more flexibility

Hungary and Lithuania require more

flexibility to local/national needs in the

youth sector.

Germany considers that call priorities

should be regularly aligned with societal

realities in a more targeted manner. It

requests the possibility to set national

289

86% (n = 527) of OPC respondents (strongly) agreed that Erasmus+ is well aligned with national policies and priorities.

Interviews, agencies and experts’ surveys; for instance most of experts consider that the projects funded generally align well

(from high to average alignment) with both EU and national policy priorities.

290

68% vs. 46% under the predecessor programmes (p.m. Erasmus+ aligned with EU priorities: 73%); ICF, fig.4.11.

291

Examples of alignment between national priorities and the programme were reported by at least half of interviewees and

by at least one type of interviewee per case study. ICF, tab. 4.5 shows that Erasmus+ objectives or actions align well with the

vast majority of policy priorities/needs commonly reported per sectors by KIIs.

292

Except in the school education sector, where the share of policy makers with a strong positively feeling of the alignment

was conversely slightly higher. ICF/Technopolis, fig. 4.5 to 4.10 for breakdown per sector.

293

60% agencies’ respondents disagree or strongly disagree with the statement that Erasmus+ is better aligned with national

priorities than its predecessor programmes.

294

BEde, BEfr, BEnl, BG, CZ, DE, EE, IE, EL, ES, FR, HR, IT, LT, LU, HU, MT, NL, AT, PL, PT, RO, SI, SK, SE, UK,

FY, LI, NO, TR; NRS, tab 3.1 for details per sector

295

National Reports were not required to cover centralised actions (e.g. sport field, Jean Monnet, etc).

296

DK, DE, IE, LV, LT, HU, SI, SK, etc

kom (2018) 0050 - Ingen titel

Irish national agenda allowing important synergies.

priorities to respond better to the needs of

national target groups.

Source: ICF 4.1, NRS 3.1

Most position papers collected during the consultation highlight that Erasmus+ is

one of the

EU’s most successful programmes

and that it remains highly relevant. The majority of the

consultation respondents (59%) thought that the current

Erasmus+ objectives are extremely

relevant to the current challenges and needs,

especially as regards the development of

skills

and competences, the

European dimension

of education and youth activities,

languages

and intercultural awareness. In contrast, fewer respondents saw the relevance of the objectives

regarding the international strand, Jean Monnet, policy support and sports

297

Relevance of the programme can also be assessed in relation to its

capacity to fund

innovative projects,

considering the needs for innovation in a context of global competition

between economies and particularly between education systems. While evidence shows that a

wide range of applicants are well targeted, the innovative capacities are rather limited

298

. The

experts’ survey indicates that the projects are most frequently (42%) moderately innovative

with a relatively small share that was considered as highly or very highly innovative (15% and

2%, respectively). At the same time, similarly to the increased quality of applications

299

, an

improvement can be seen in terms of innovation when comparing Erasmus+ with its

predecessor programmes. This could further improve at the final evaluation stage when KA2

Alliance-type projects are sufficiently advanced or have been finalised to be evaluated.

More generally, key evidence suggests that the programme is

well suited to attract a wide

range of target audiences

across the fields it covers. It is particularly suitable for reaching

out to learners with different profiles

300

with one noticeable exception: disadvantaged

groups,

as elaborated below

301

A deeper analysis shows that the

needs of different stakeholders are generally well

addressed,

albeit with variations according to sector, organisation size and level of

intervention. The majority of the interviewees estimate that the programme strongly

contributes to addressing the socio-economic needs of the

learners

302

. Monitoring survey

highlights that the motivations of learners are fairly well aligned with programme objectives

and related to key competences. Foreign language skills is the most mentioned need in all

sectors. In VET, learners’ motivations for participation are particularly well aligned with

programme objectives on key competences, including technical skills (80%). At least 50% of

agencies’ respondents believe that Erasmus+ meets the needs of target groups including

practitioners

303

better than its predecessor programmes. They all agree that learners’ needs in

higher education and VET are met by both KA1 and KA2, but the needs of school pupils are

partially or weakly met by KA1, despite the fact that teachers take part in the action. On a

297

OPC 3.1; ‘extremely relevant’ objectives for the Programme include: skills and competences of individual learners (73%);

the European dimension of education and youth activities (70%); the quality, innovation and internationalisation in

education, training and youth organisations (68%); the teaching and learning of languages and intercultural awareness (68%)

the skills and competences of practitioners (62%); the Union's external action (54%); excellence in teaching and research in

European integration (48%); policy reforms at national level (44%); sports objectives of Erasmus+ (33%)

298

ICF, 6.13 and 6.18. The share of innovative methods considered low or very low was 24% and 5% respectively

299

ICF, fig.4.16 : according to the expert survey, improvement in application quality is found between Erasmus+ and its

predecessor programmes; currently, almost one third of the applications is considered as of high quality by experts (18% for

predecessor programmes), whereas most frequently (42%) of average quality (37% for predecessors). Caution: 37% of

experts have no opinion as for predecessor programmes.

300

Beneficiary survey, ICF 4.4 and 6.3

301

Drawing on the definition set out in Erasmus+ Programme guide (p.9)

302

ICF, 4.2 and 4.3

303

ICF, 4.3.3 about practitioners' needs

kom (2018) 0050 - Ingen titel

qualitative note, many examples in interviews suggest strong relevance of the programme to

learners’ needs

304

. Lastly, as detailed above in the section on effectiveness, participants’

satisfaction is very high (81% to 100% claiming being very or rather satisfied with their

mobility experience). In contrast, interviewees have more mixed views on the programme’s

ability to meet the needs of

organisations.

The needs of small organisations especially are not

met sufficiently

305

. Lastly, agencies’ staff consider that the needs of schools, VET and youth

systems

are addressed by KA3 (highly relevant), while the relevance for higher education and

adult learning systems would be more moderate.

The

discontinuation of a small number of (Lifelong Learning Programme and Youth in

Action) actions

had only a limited impact on the extent to which stakeholders’ needs were

addressed, except in two areas:

306



many interviewees claimed that pupil mobility (and to a smaller extent adult learner

mobility) should return to a KA1-type of funding, instead of under the more complex

coverage of KA2

307

discontinued local youth initiatives previously funded by Youth in Action were

considered more accessible than current Erasmus+ transnational youth initiatives

308

The former enabled young people to initiate their own projects, whereas the latter are

seen as too complex for informal groups of young people, especially NEETs (young

people not in Education, Employment, or Training), to apply

309

Knowledge about Erasmus(+) 2009 vs 2016

Source: Standard Eurobarometer 86 and Special Eurobarometer survey 316

All findings reveal that the programme is

perceived as increasingly visible and

attractive

310

. Erasmus+ is well known to the audience in general (see fig. above)

311

and

especially to the young generations. Its large visibility also, including outside the EU, stands

out when compared with that of other EU programmes such as Horizon2020, Europe for

Citizens or EU Aid Volunteers

312

. It can also be positively noted that the degree of repeated

participation appears to be low for learners (less than 10%), whilst it is expected for the more

304

ICF, 4.3.2 and 4.3.3, text boxes with examples of responsiveness to needs reported by interviewees (case studies)

305

ICF, 4.3.4 about organisations' needs

306

Interviews at national level and National Reports

307

National reports of DK, FR, IT, CY, FI, CY,UK, etc; ICF, 4.3, including illustrative quotations from KIIs

308

BEnl, HU, MT, CH

309

ICF, 4.3, including illustrative quotations from KIIs

310

Social media analysis, Agencies survey (76%), National reports, Eurobarometer survey and outputs to Erasmus+ 30 years

anniversary campaign. See also section on coherence below.

311

Less visible in the sport community (e.g. HU) or in reaching dual VET (DE, AT)

312

ICF 4.5, 6.9 and annex 19 (not published). See Annex 3 about social analysis performed

kom (2018) 0050 - Ingen titel

permanent population of staff

313

. It is nevertheless found that the extent to which Erasmus+

reaches out to potential organisations varies across the different fields. The visibility of the

Jean Monnet activities could, in particular, be stronger

314

. A few National Reports

315

link the

lower participation of primary education and early childhood education and care in Erasmus+

to a lower awareness of the programme in these sectors. Some National Reports make

suggestions on how to increase visibility

316

Furthermore, a number of

barriers to participation

are identified limiting the extent to

which the programme addresses the needs of stakeholders. At the individual level, financial

barriers

317

and administrative burdens

318

are more often reported than are geographical

disparities

319

. At the organisation level, the programme offers less room for small scale

projects than in the past, reducing the extent to which it reaches grassroots level organisations

in all their diversity

320

. Evidence suggests that smaller sized organisations often have neither

the experience, nor the resource capacity that a successful application requires

321

. Many

stakeholders believe that funding has become too highly competitive, making it only

accessible to the most experienced/largest organisations, preventing access to potential

newcomers, hence reducing the number of those who may benefit from a European

experience

322

Despite the specific attention paid to social inclusion

323

with varied results across sectors

324

and to

widening the participation of disadvantaged target groups

325

, recognised by a

majority of interviewees

326

and 30% of agencies, evidence suggests that there is room for

improvement

327

. The share of disadvantaged groups represents almost 10% of KA1

participants (almost 25% in the youth field as well), as detailed in the section on effectiveness

above, due to the programme’s specific efforts in the area (see fig below). Erasmus+ has

offered new experiences abroad to many people who would have otherwise not been able to

313

Based on the monitoring data, the vast majority of VET learners (95%) and students in international higher education

mobility (91%) are first time participants in EU funded mobility actions (Erasmus+ or LLP). This is a positive finding as the

benefits of mobility on attitudes are likely to diminish with repeated participation. Concerning staff mobility, while the

majority of staff in schools, VET, adult learning and international higher education are first time participants, most staff in

youth and higher education are repeated participants.

314

Even among higher education practitioners who teach about the EU, only 55% have heard about the Jean Monnet

activities; ICF/JMO, 2.5

315

DE, RO, FI, etc

316

NRS 3.3: SI, IS

317

HU, PT

318

Participants expressed concerns about complex administrative procedures related to application and reporting: BEnl, BG,

CZ, DK, DE, EE, EL, ES, FR, HR, IT, CY, LV, LT, LU, HU, MT, NL, AT, RO, SK, FI, SE, UK, IS, FY, LI, NO, PL, PT,

CH, TR; see section on efficiency below.

319

Few countries report issues of unequal participation by regions. FR observes a reduction in the participation of overseas

territories. BG and ES mention the specific needs of small towns and rural areas.

320

EE, IE as regards vulnerable groups; ES, EL, PL, FY concerning newcomers

321

Large number of national interviews; also denoted across the National Reports BE, BG, CZ, DE, FR, HR, CY, LV, LT,

RO, SK, FI, NO, etc

322

Both data programme analysis and the expert survey identifies major unbalances in competition for grants among the

different sectors; ICF, 4.4 and tab. 4.10 for success rates for Erasmus+ per sector

323

Interviewees most often referred to the call on policy experimentation (KA3) on social inclusion across sectors.

324

In the field of adult education, only 15% of agency respondents (lowest share) strongly agree that the programme offers

more opportunities for disadvantaged target groups than in the past. In contrast, the highest share is found in youth (44%).

325

ICF, 4.6: 57% of agency respondents perceive the actual participation of disadvantaged groups in Erasmus+ as higher or

the same compared to other national or international programmes. It should be mentioned, however, that for 36% of agency

respondents, the absence of relevant statistical data was an obstacle to provide any estimates on this issue. IFC, Annex 10 for

the most frequently targeted disadvantaged groups as per the expert panel assessment of Erasmus+ project outputs.

326

KIIs (national level) assessed the relevance of Erasmus+ to hard-to-reach or specific disadvantaged groups positively

almost in all sectors

327

ICF 4.3 and 4.6, similar findings from NRS 3.4 and OPC 3.1

kom (2018) 0050 - Ingen titel

afford it. In countries where there is no mobility programme targeting disadvantaged group,

Erasmus+ is particularly appreciated. Where programmes supporting mobility do exist, they

do not have such a priority or – if they do – they do not promote it to such an extent as

Erasmus+ does

328

Perceived level of participation of disadvantaged groups (Erasmus+ compared to other

comparable national/international programmes/initiatives

40%

30%

20%

10%

27%

36%

Higher level

Equivalent

Lower level

No opinion/data/n.a.

Source: ICF survey with National Agencies (n = 117)

However, according to a vast majority of National Reports

329

and a shared perception among

all groups of respondents,

young people from disadvantaged backgrounds and with

special needs are not sufficiently addressed.

The view prevails that Erasmus+ has even

limited potential to reach those organisations that address specifically this group of people.

Even the European Volunteer Service (EVS) remains strongly competitive. Most National

Reports and findings from the public consultation welcome the additional funding available

for disadvantaged participants

330

but do not consider it sufficient on its own. Some

countries

331

call for clearer definitions and streamlined rules applying to disadvantaged

participants in Erasmus+. According to the expert panel, while a good share of projects

viewed considered disadvantaged groups, this was often not substantiated with outputs

suitable to them. Moreover, the Student Loan Guarantee Facility (SLGF) is commonly found

to be insufficiently tailored to address the needs of the disadvantaged who are risk averse to

go abroad for a full Master programme or to take up loans even if repayments are not income-

contingent

332

, although the first evidence from the beneficiaries surveys rather shows many

beneficiaries are first-generation higher education students or coming from families with

financial difficulties

333

. Despite some progress made in the programme design, it can be

concluded that there is a certain dissonance between its ambition and actual project outputs

specifically addressing the disadvantaged target population. The views whether more can be

done at European level to address these challenges were nevertheless mixed among national

policy-makers.

328

The inclusion of disadvantaged groups is not specifically encouraged by the Baltic-American Freedom Foundation,

Denmark-USA programme, the Fulbright programme, German-Turkish Youth Bridge and Prämienprogramm

329

BEde, BEnl, BG, CZ, DE, EE, IE, CY, EL, ES, FR, HR, IT, LV, LT, LU, HU, MT, NL, AT, PL, PT, RO, SI, SK, FI, SE,

IS, FY, TR

330

In addition to the social inclusion dimension in mobility (e.g. top-ups), in 2016, more than €200 million were prioritised

under KA2 as well as €13 million for dissemination and €14 million for policy experimentation under KA 3.

331

FR, IT, NL

332

According to literature reviewed, interviewed students and 79% of agency respondents

333

ICF, stand-alone report on SLGF and annex 8 on SLGF survey results

kom (2018) 0050 - Ingen titel

Examples of support to disadvantaged individuals or hard-to-reach group

In Poland, the NA implements the ‘Foreign mobility of disabled and disadvantaged students’ project co-

financed under Erasmus+ and the ESF. This project funds scholarships for students with disabilities and

provides additional funds for disadvantaged students receiving grants from their home universities.

In Turkey, as a result of a KA2 project implemented by the Sincan Public Training Centre, adult training is

delivered to inmates at the Prison using a tailored curriculum and method.

Source: NRS, 3.4 (PL, TR)

kom (2018) 0050 - Ingen titel

5.3.

Coherence

The assessment of the coherence of Erasmus+ is

overall very positive,

both internally

(compared to its predecessors) and externally (compared to other existing interventions). It

has been informed by 10 evaluation questions that looked into the extent to which:



the programme’s internal coherence improved compared to its predecessors

the Erasmus+ objectives were consistent, mutually supportive across fields

synergies or duplications between fields and actions existed and were dealt with

the centralised and decentralised actions complemented each other

Erasmus+ design provided appropriate support to the EU policy agenda

Erasmus+ was coherent with relevant policies and programmes and complementary to

other interventions in the same fields at EU or national/international levels

respectively.

5.3.1. Internal coherence

334

The merger of seven predecessor programmes into

Erasmus+ has improved the coherence

of the EU intervention

according to the vast majority of respondents

335

. As mentioned

before,

Erasmus+ now covers learning in all its contexts

– whether formal or non-formal,

including youth work– and

at all levels of the lifelong learning continuum:

from early

childhood education and schools, vocational education and training (VET) through to higher

education and adult learning. The evaluation highlights the positive effects of this integrated

approach underpinned by the lifelong learning logic. It underlines the necessity of the

integrated approach in today's education and training environment, where the borders between

the traditional education and training sectors have become less clear-cut

336

Different sources agreed on the

added value of the integration of several programmes

within a single structure. The

objectives are seen as mutually supportive

across

education/training, youth and sports, as well as across actions

337

. These fields are seen as

naturally fitting in a programme allowing

cross-sector fertilisation,

which - according to

90% agency respondents - was not always possible in the past. Around 3/4 of these

respondents feel that the integrated programme enables increased synergies, while 2/3

estimate it avoids internal overlaps.

All these improvements are confirmed by desk research, national reports and a vast portion of

the interviewees who appreciate that the programme has been designed with better logical

linkages between objectives and actions

338

. It is very often valued for its clearer and

simplified architecture

with only

three Key Actions

(KAs)

339

, cross-cutting priorities (e.g.

social inclusion, prevention of violent radicalisation, etc.) and a much lower fragmentation of

actions than denoted in the past (e.g. Grundtvig, international cooperation

340

), with only a few

334

Source of this section (where not mentioned otherwise): ICF, 5.2; NRS 4.1; OPC 3.2 and 5.1.3. The evaluation of the

coherence in relation to Jean Monnet actions was evaluated based on desk research.

335

Interviews, agency survey; ICF, 5.2.4: no contradiction in objectives noted between JMO and the rest of Erasmus+

336

ICF, Conclusion D: Many VET providers are falling under the definition of 'schools' (in school-based VET) but they can

also be providers of adult education. Similarly, higher education organisations are also providers of adult education in many

cases. The cooperation between education and training (in particular for schools but also beyond) and civil society (youth

sector) is a common reality on the ground.

337

ICF, tab. 5.3

338

ICF, 5.2.1

339

See above section on background. 89% (n = 619) of OPC respondents (strongly) agreed that the structure into the three

Key Actions is working well. 65% of agency respondents (strongly/rather) agree that it increases EU added value.

340

The merger of Tempus, Alfa III and Edulink into the Erasmus+ Capacity Building action allows for a strengthening of co-

operation between different regions, however the focus on intra-regional co-operation for partner countries remains limited,

according to the

Evaluation of the EU Development Co-operation Support to Higher Education in Partner Countries (2007-

2014),

(2017), Particip, LeA, ECDPM, Ecorys, already mentioned.

kom (2018) 0050 - Ingen titel

exceptions

341

. Almost all country reports

call for stability

of the structure and scope of the

programme in the future. The stronger attention given in the design and implementation of

Erasmus+ to budget based on performance

342

, monitoring of individual outcomes

343

and

dissemination of project results

344

is also valued. As detailed under "effectiveness" above,

actions are delivering at more than one level, with spill-over effects (e.g. individual staff

mobility having impact on sending organisations) and common support tools (e.g. widening

scope of transparency and recognition tools).

The

advantages of Erasmus+ programme

listed by the national respondents include its

strong branding

and its greater visibility

345

, especially among policy-makers. This is

particularly true for VET, adult education and even for the youth sector in contrast to

stakeholders' initial fears. European challenges (such as the integration of refugees or the

attention to disadvantaged groups in higher or adult education) are now addressed at least in a

more flexible way

346

. Moreover,

lifelong learning is approached more as a lifecycle

(including non-formal and informal learning) rather than in a segmented way. Even if there

remains residual concern

347

that the brand “Erasmus” is traditionally linked to higher

education, the fact that the Erasmus+ provides

one place to look for funding opportunities

in the related sectors with a standardised and more transparent administration is appreciated.

Meanwhile,

some concerns have been reported at national level.

In two-thirds of the

countries, different NAs continue to deal with distinct sectors, replicating the way they

functioned under the previous programme generation

348

. Some National Reports confirm that

there is no systematic exploitation of potential synergies between different types of actions

349

The disadvantages inherent to any one-size-fits-all approach are naturally pointed out

350

Respondents from various organisations and sectors noted that the

adult education

strand has

shrunk in size and lost its specificity. The

youth

chapter would be less integrated, less visible

for instance within KA2 (or would now link to more abstract EU policy targets than in the

past). Whilst the objectives of the

sport

chapter align well with EU priorities, some potential

applicants do not always see how to make them fit in their local perspective. More generally,

an integrated programme is likely to offer less autonomy to sectors and less room for country-

specific priorities compared to its predecessors. There is for instance the sentiment among

national authorities that the integration of the programme committees has led to discussions

becoming more abstract for each policy area integrated in Erasmus+.

When looking at specific examples, further nuances emerge.

The

increased

decentralisation was positively viewed

by the vast majority of respondents because it brings

the programme closer to its beneficiaries and helps engage local partners, while centralised

actions (e.g. in KA3) complement well other actions of the programme. Meanwhile, policy

341

NRS: mobility in the field of school education and VET is not included under KA1 but can be covered under KA2,

creating some confusion among schools (e.g. DK, FR, IT, CY, LV).

342

Article 18(7) and (8) of Regulation (EU) No 1288/2013 about

performance based budget:

in practice, around 25% of the

funds allocated to National Agencies for KA1 are distributed on the basis of the level of implementation achieved, whilst the

remaining budget is allocated essentially on the basis of the size of the population. For KA2 and 3, qualitative criteria to

increase EU added value apply (e.g. link with OMC; resource commitments of project partners).

343

Article 21 (idem) on monitoring of performance and results

344

Article 22 (idem) on the dissemination of results

345

76% of agency respondents strongly or rather agree that the integrated programme is more visible /better branded than the

predecessor programmes; confirmed by social media analysis

346

ICF, tab. 5.9

347

CY, EE, HU, SI, etc. Besides, for a small portion (estimated less than 10%) of respondents Erasmus+ still remains viewed

as associated with the mobility experience of HE students.

348

Nevertheless certain National Agencies have merged since 2014. Their number has been reduced from 63 in 2013 (with

12 NAs common to education and youth) to 57 (16 common NAs) in June 2017.

349

BEnl, EE, HU, SI, SK, UK, NO

350

ICF, tab. 5.9

kom (2018) 0050 - Ingen titel

makers and NAs were clearly more positive than EU level organisations which look for a

more direct access to funding. The

main critical perceptions relate to KA2

suggesting that

too broad scope of the action and the decentralisation of strategic partnerships may result in

differences in implementation or a prevalence of nationally-driven agenda

351

. Some argue that

this could even push partnerships or cross-country organisations to apply rather in larger

countries. Besides to attract talent to Europe, some suggest it could make more sense to

centralise international credit mobility. However, providing concrete examples of synergies or

overlaps has proven more difficult than anticipated for most interviewees at national level or

agency respondents. The only area where some overlaps have been noticed concerns those

projects in the sport sector which focus on social inclusion and could be youth projects as

well. However,

overlaps might be more often due to misunderstanding

at application stage

than to the design of the programme as such

352

Examples of internal synergies or overlaps

Synergies

Between fields:

"It

is more likely that a

young person who has done an Erasmus will

also go on youth exchanges or EVS

projects."

"An

integrated programme stimulates cross-

sectoral collaboration in a valuable way.

Regional development projects are easier

[…] to communicate and are also

administratively simpler when all the target

groups are part of the same programme."

(SE)

"Our

organisation takes part in an activity

under KA 3 in the youth chapter and uses the

outputs for advocacy in the education field."

(Structured dialogue with young people)

There is evidence of education institutions

successfully applying to implement sport

projects that are also tied to the education

and youth field.

Between KAs:

Synergies are reported at

organisation level between e-Twinning and

KA1 and KA2 in the school sector.

"The

way in which KA1 and KA2

complement each other provides

opportunities for cooperation between long-

term network partners."

(FYROM)

"Within

KA2, there is a close cross-sectorial

cooperation, which benefits all sectors. […]

There are strategic partnership projects that

have led to development of tools and

databases to heighten the quality assurance

"Student

mobility within the field of school

education falls within KA2. This has led to

confusion among the schools given that student

mobility for all other sectors falls under KA1. It is

recommended that student mobility for pupils in the

field of school education be moved to KA1."

(DK)

Mobility opportunities under KA1 and KA2

"for

Erasmus+ Youth in Action the logic is not kept.

[…]. KA1 and KA2 in Erasmus+ seem to be partly

overlapping in practice, […]. This leads to the point

that applicants are partly going for their mobility

projects to KA2 because of the higher funding"

(Youth NAs' input)

"KA1 could benefit from

Overlaps

"Yet,

a significant difficulty has been experienced

with the adult education and VET actions […].

Coverage and scope of these fields mostly overlap

and cannot easily be differentiated by the

beneficiaries."

(TR)

The last example on the right column has a

counterpart, as meanwhile it reduces the

participation of purely sport-focused organisations.

351

352

Agency survey, OPC position papers from 3 EU-level organisations representing together some 70 organisations

Common input for the mid-term evaluation 2017 from the NAs Erasmus+ Youth in Action, August 2016

kom (2018) 0050 - Ingen titel

of VET students’ mobility projects, which

have been implemented by the vocational

colleges working within KA1"

(DK)

focusing on training and capacity building tackling

learners directly and KA2 more on intellectual

outputs that are tangible enough and with long term

impact."

(Youth NAs' input) "There

is a certain

overlap between youth worker mobility and

strategic partnerships for good practices"

(BE-nl)

KA2 (Strategic partnerships) and KA3 (Forward

looking cooperation initiatives) could overlap (ICF).

"KA2

is often used for policy reform too at national

level"

(Youth NAs' input).

"The

inconsistencies arise from differences in

grants given per participant for training activities

versus youth exchanges [two sub-actions in KA1].

The grants for training activities are higher, which

increases the incentive to apply for funding for

training activities."

(DK)

Between actions:

KA3 Policy

experimentations and Forward looking

initiatives: (one is top-down, the other is

bottom-up) are meant in their design to be

the two sides of the same coin. (ICF)

Source (where not mentioned otherwise): ICF, 5.2.2

Another area where findings are more contrasted concerns

cross-sector opportunities.

Whilst

a significant number of respondents

353

see this as a clear improvement under the new

programme (e.g. for youth), many others

354

remain more uncertain (e.g. unclear remit of adult

education

355

) or have concerns about its actual materialisation (e.g. for sports

356

). A majority

of respondents asked are either not interested or not convinced to take part in cross-sectoral

projects (e.g. finding different partners require more efforts). However, as detailed under

5.1.2, the

level of cross-sectoral cooperation increased

under the current programme by 23

percentage points

5.3.2. External coherence

357

At programme level, complementarities

are mainly found with the European Social Fund

358

(ESF) and the Research framework programme Horizon 2020, and to a lesser extent with

INTERREG, Erasmus for Young Entrepreneurs programme (EYE) and in the field of sport

with the Third Health programme. For instance, in Greece, the participation of pupils in

eTwinning was boosted nation-wide by the combined support of European Investment and

Structural Funds to teachers' training (ICT skills) and classroom equipment (digital tablets)

359

Potential

synergies

that were noted by interviewees include: Erasmus+ and the European

Training Foundation (ETF); Europe for Citizens and Culture/Creative Europe programmes;

the EU Programme for Employment and Social Innovation (EaSI) which has a component of

353

High share of agencies’ respondents; DK, EE, NL, PL, PT, SI, NO, etc

354

Over half of interviews (at national level particularly), several OPC position papers, DE, EE, MT, NL, SI, SK, FY, TR…

355

Overlaps are observed between VET and adult education (CY, PT, TR); LLP interim evaluation COM(2011) 413,

section 5

356

The borders between sport and other strands are rather blurred. There are more sport organisations participating in other

strands than in the sport one. Reciprocally, more than 5% of participants in the sport strand are higher education institutions.

Many grassroots sport projects could be funded under the youth strand.

357

Source of this section (where not mentioned otherwise): ICF, 5.3; OPC 3.2; NRS 4.2

358

The ESF legal basis recommends taking up good practices from Erasmus+ to further develop them under ESF. Erasmus+

mobility grants can be topped-up by ESF or followed by placement services financed by ESF. Under the ESF investment

priority “Improving the quality of tertiary education”, for instance, the ESF may support activities to encourage the

development of Erasmus activities in a university.

359

Workshop DG REGIO/KPMG, 18 November 2016, organised in the context of the study mentioned below

kom (2018) 0050 - Ingen titel

entrepreneurship education, etc. Few potential

overlaps

were alleged by some interviewees

(e.g. in the case of strategic partnerships) but without any concrete examples

360

The integration of the programme has also enhanced the external coherence in removing

certain overlaps

361

. Insights on the coherence of the programme against other EU level

programmes were gained from both a majority of EU level interviews and the vast majority of

agency and OPC

362

respondents. Those provided examples of (potential) synergies or

overlaps, confirming desk research and benchmarking.

Examples of external complementarities, synergies or overlaps

Complementarities

Synergies

Overlaps

With the European Social Fund (ESF), Youth Employment initiative

International visits, transfer

of good practice from

abroad, the training of

teachers, etc are supported

by the ESF. Topics which

are dealt with include digital

education, language

training, and recognition of

results from formal and

informal learning,

cooperation between

schools and employers, etc

363

(KPMG )

Former LLP ‘People on the

labour market’ initiative is

now run through ESF,

providing mobility

opportunities for people

after studies.

A number of projects under

the operational programme

‘Employment, Human

Resources and Social

Cohesion’ are co-funded by

the ESF, the Youth

Employment Initiative and

Erasmus+ (CY).

In Latvia, Erasmus+ and the

ESIF in the field of

education and science are

managed by the same

Ministry, helping to increase

synergies. (KPMG)

In Sweden, the ESF is used

to fund measures in

preparation and as a follow-

up on the Erasmus+

mobility. Although

Erasmus+ has a smaller

budget, by combining the

two funds, both objectives

of labour mobility and

labour market activation are

better achieved. (KPMG)

Discussion between

stakeholders of ESIF and

Erasmus+ has increased in

the Czech Republic since

"For

example, why does ESF

funds projects to establish

placement/internship structures if

Erasmus+ has done so for the

past 10 years (and beyond)?"

NB: This might not necessarily

be an overlap, as Erasmus+

supports transnational activities,

whereas the ESF focuses on

activities implemented in a

national context.

360

Existing or potential synergies were most often reported than overlaps across the nine programmes discussed. In the OPC,

only one EU level organisation identified overlaps between Erasmus+ and national funds in education and training, in

particular in the case of Germany (DAAD funds for joint degree programmes) and Norway (national travel support for

outgoing mobility). One organisation was concerned about potential overlaps between Erasmus+ and the European Solidarity

Corps. ICF, NRS 4.2

361

Mobility of doctoral researchers to/from partner countries, which overlapped with Marie Skłodowska-Curie actions

362

80% of respondents (n = 1485) believed that Erasmus+ does not overlap with other funding opportunities, compared to

3% of respondents who believed it ‘fully overlaps’.

363

Study on the co-ordination and harmonisation of ESI Funds and other EU policies, DG REGIO/KPMG/Prognos, to be

published in 2018

kom (2018) 0050 - Ingen titel

2014. The Czech Ministry

of Education has set up a

centre (DZS) which co-

ordinates EISF, Erasmus+,

EQF, EQAVET or

Europass. Beneficiaries find

it easier to apply for

Erasmus+; ESIF is only

used to provide additional

resources and authorities are

trying to avoid financing the

same beneficiaries twice.

(KPMG)

With the European Regional Development Fund (ERDF), INTERREG

- Some INTERREG

projects prepare framework

for mobility activities under

Erasmus+ (e.g. Danish-

German projects)

- For instance, Erasmus+

supports mobility, whereas

the ERDF support

investments in

infrastructure, such as the

renovation of buildings. Due

to these investments, Latvia

became more attractive to

students and academics

from other Member States.

(KPMG)

With the Framewok Programmes for Research and Innovation

On the international

dimension of the previous

programmes

364

, "the

EU’s

approach to supporting

higher education on the one

hand and research and

innovation on the other was

complementary…

…and

has created some

synergies, for example:

- Links between Erasmus

Mundus and Tempus IV with

the Seventh Framework

Programme (FP 7), the

EU's main instrument for

funding research;

- Edulink’s connection with

the ACP Science &

Technology Programme

(ACP S&T)…

"Erasmus+

complements

Horizon 2020 very well

[since the focus of each

programme is rather

364

ERDF is moving towards smart

specialisation strategies for

regions, which is related with

training (ESF, Erasmus+).

…However,

formal and

institutionalised attempts to

connect the major programmes

targeted at universities were very

limited (they existed only in the

case of ACP HE Institutions), and

there is room (and demand) for

creating more synergies."

The share of EU students in

in the EIT Label-

programmes has slightly

declined in recent years,

Horizon 2020 (e.g. priority 6 on

societal challenges) tackles same

issues as Erasmus+, on larger

scale but without synergy. For

Evaluation of the EU Development Co-operation Support to Higher Education in Partner Countries (2007-2014), (2017),

Particip, LeA, ECDPM, Ecorys, already mentioned

kom (2018) 0050 - Ingen titel

complementary and partly

intends to fulfil the same

goals] e.g. Joint Master

Degrees (Erasmus Mundus)

in Erasmus+ and Joint

Doctorates in Marie Curie

Actions"

which could be due to the

alignment of the EIT

programmes with Erasmus

Mundus actions.

(Interim evaluation of EIT)

example, early school leaving

was addressed in 20-30 projects

under Horizon 2020 and with

over 100 projects under

Erasmus+.

With Young Entrepreneurs programme (EYE)

Many EYE participants

have gone through Erasmus

mobility while they were

students. Individuals are

later more open to

international mobility.

Source (where not mentioned otherwise): ICF, 5.3

Despite this promising picture, most of the examples supplied remained rather general. In

particular, few examples were provided where ESF would have funded follow-up of

Erasmus+ projects (KA2). More synergies should be encouraged between ESF and policy

actions (KA3) for designated bodies. Considering the call for action set in the Erasmus+ legal

basis

365

, some respondents suggest that communication between services in charge of given

programmes could be strengthened at EU level. In particular, no clear example of dedicated

coordination tools between EC services was reported.

At policy level,

the topical coverage of the selected projects reviewed reveals a clear and

satisfactory alignment with different EU-level policies

366

. Most interviewees (particularly

at EU level) perceive Erasmus+ as supporting key EU policy agendas

367

, especially in

education and training, similarly to its predecessors but

in a more streamlined manner.

Most of EU interviewees could confirm it with examples of complementarities. Out of the

80% of reviewed projects

368

aimed at contributing to the main EU policies/strategic

documents analysed as part of desk research

369

, two-thirds focused on ET2020 topics, mainly

social inclusion

370

. However, across the sampled projects, only a very small number focused

on the Europe 2020 headline targets, despite these being the two first key indicators of

Erasmus+

371

. Lastly, the Jean Monnet projects reviewed appeared to be the least aligned,

being perceived as too focussed on experts' needs

372

365

366

Articles 3.2 (b) and 25 of Regulation (EU) No 1288/2013 define complementarity

Confirmed across the sources mentioned under this section by a high share of respondents or projects reviewed

367

Europe 2020* (for most interviewees), ET2020*(to a high extent), European Youth Strategy* (although perceived as a bit

less visible), Copenhagen process and tools (often mentioned for VET), Modernisation of higher education agenda (strong

consistency), EC Communications on ‘New Skills Agenda for Europe’* or 'European higher education in the world'* (albeit

both mentioned to a lesser extent), EU Work Plan for sport* (probably consistent although not as clear at national level as it

is at EU level), etc.

(*): analysed through Desk Research

368

ICF, Annex 10 (published) and 17 (not published)

369

Marked with an asterisk (*) in the footnote above. For instance, over half of the sport projects reviewed aligned with

priority objectives listed in the EU Work Plan or sport. ICF, tab. 5.13

370

Only three of the 18 comparator programmes reviewed pay special attention to the participation of disadvantaged groups

e.g. the Causeway programme, the Nordplus Higher Education and Adult programme programmes and the UK Sport's

IDEALS programme in sport. Attention paid under Erasmus+ to widening participation was seen as complementary in most

other cases.

371

Early school leaving and Higher education attainment as per Annex 1 of Regulation (EU) No 1288/2013

372

Project review, JMO beneficiaries and non-beneficiaries survey, desk review

kom (2018) 0050 - Ingen titel

Examples of support to EU policy agenda

ET2020 working groups (WG):

as per the reviewed ET2020 priorities for 2016-2018, the WGs

have now a recurrent mandate to support and to follow-up the Erasmus+ KA3 policy

experimentation. For instance, in 2016, synergies were found for the following topics:

‘Strengthening teacher training and education by using the opportunities of new technologies" ;

‘VET teachers and trainers in work-based learning/apprenticeship (VET)’ ; ‘Employment and

Skills: validation of informal and non-formal learning in Education and Training’, etc.

Building on the legacy of LLP, Erasmus+ support the review and implementation of several

transparency and recognition tools,

by (co-)funding the Secretariats and national bodies for

ECVET and EQAVET, the Euroguidance network, the national Europass centres and EQF

Coordination Points.

Source: ICF, 5.3

Erasmus+ programme is also

coherent, to a high extent, with interventions pursued at

national and international level

which have similar objectives. A vast majority of

interviewees consider it to be overall well aligned with national priorities in all sectors.

Benchmarking of comparable programmes

373

but also National Reports

374

, interviews, agency

survey and desk research did not detect any notable overlaps

375

and provided interesting

examples of complementarities at both levels. The area most often mentioned by VET

interviewees in that respect is school and work-based learning. The higher education

interviewees offered several examples of national-level programmes that show significant

coherence with Erasmus+

376

The benchmarking confirmed a fair

alignment between Erasmus+ and national

comparable programmes

in terms of objectives, target groups

377

and duration of actions

378

Although few overlaps were reported

379

and efforts have already been made to clear them

380

the existence of parallel funding opportunities

381

could likely be better communicated to the

public at national level, considering the aforementioned call for action set in the legal basis.

Indeed, despite the fact that several NAs run some schemes in parallel to Erasmus+, many

beneficiary organisations and NA respondents

382

claimed they were not aware of any similar

national support.

Countries themselves did not identify any specific overlaps

between Erasmus+ and other

national and international programmes,

but rather more synergies.

Erasmus+ is different

373

Out of 58 national or international schemes shortlisted as supporting actions similar to those of Erasmus+, 18 were

benchmarked against the programme. Action-wise, the vast majority support mobility. Several NAs run such programmes in

parallel to Erasmus+ (ICF, Annexe 12).

374

NRS, 4.2

375

Few National reports (e.g. FR) mention cases where synergies could be improved. For instance, to avoid the duplication of

similar tools at local level, the Europass could be made more flexible with a second part for information customised to local

requirements.

376

Nordplus Higher Education (Nordic countries); CEEPUS (various countries); National Science Centre programmes (PL);

Norway/EEA funds instrument (various countries), etc.

377

High alignment in higher education, VET and youth ; lower in school and adult education and in sport

378

ICF, tab. 5.15. For example, some comparator schemes, especially at the school level and in adult learning, offer much

longer average mobility stays for the practitioners.

379

20% of OPC respondents considered that Erasmus+ is ‘fully’ or ‘partially’ overlapping with national funding

opportunities (for student mobility or volunteering activities). However, most of them specified that even if there is a partial

overlap, Erasmus+ is still very much required either due to: a lack of funding even with the overlap, different objectives of

the funding even when targeted at the same beneficiaries or differences in geographical areas covered.

380

54% of agency respondents declared their agency had taken specific actions to ensure complementarity between Erasmus+

and other programmes (e.g. joint events, information sessions)

381

For mobility in education and training (DK, EE, CY, HU, FI, etc) or youth (EE)

382

Case studies for the former; ICF, tab.5.16 for the latter

kom (2018) 0050 - Ingen titel

from (inter)national initiatives in many respects

383

, e.g. wider number of beneficiaries and

projects; wider geographical scope; different target groups or sectors than national initiatives;

mobility for longer periods; no other initiatives in some smaller countries to support certain

sectors (e.g. youth) or policy reform similar to Key Action 3. A few national reports even

mention cases of synergies

384

Although aiming to achieve comparable goals in most cases

385

none of the national schemes

reviewed can match the pan-European scope of Erasmus+,

not to mention its worldwide

dimension, making it the best-placed programme to develop internationalisation. In most

cases it is also more generous than other similar schemes. Furthermore

Erasmus+ offers

some unique actions

that none of its comparators does (or, at least, not to the same extent),

such as electronic support platforms (eTwinning, EPALE etc.), stakeholder dialogue or

knowledge building for evidence-based policymaking. Over half of agency respondents

strongly or rather agree that Erasmus+ is doing better than the latter in all five areas

considered

386

This can be regarded as a real added value of Erasmus+, as developed under the

related section below.

383

384

CY, EE, MT, NL, SI

FI, SE, NO. In Norway a number of VET schools, use Erasmus+ mobility strategically to offer students specialised

training in a sub field that is not available at their own school or even in the country.

385

ICF, tab. 5.15. The level of alignment of goals is particularly high in formal education and relatively lower in the sport

field.

386

Better alignment with EU policy priorities (especially in higher education and youth); broader geographical coverage;

more opportunities for disadvantaged target groups (except in adult education); broader topic coverage; more funding for

comparable activities.

kom (2018) 0050 - Ingen titel

5.4.

Efficiency and simplification

When it comes to assessing the efficiency of the programme

387

, including in comparison to its

predecessors,

the overall picture is positive

388

, stemming from the analysis of 24 evaluation

questions that cover the following areas:



the cost-effectiveness of the main types of actions

the efficiency of implementation and management modes

the efficiency gains through changes in the integrated programme

the efficiency of monitoring arrangements and measures to identify and prevent fraud

and irregularities.

Overall the

costs of management for the EU of the entire programme are reasonable

(6% of the Erasmus+ administrative and operational budget)

389

. This is particularly clear

when compared to other, much smaller, comparable national actions, which appear more

costly (on average, 14% of their respective budget)

390

Most of respondents to the consultation believe that the

user-friendliness of the programme

has improved over time

391

. However, work on simplification (e.g. online application forms)

has already started in 2017. The level of dissatisfaction has decreased as of 2016 after a steep

learning curve.

Nonetheless,

there is still room for improvement

in several features of the programme

392

especially concerning further simplification of procedures and tools (e.g. on-going

development of e-forms), more flexibility on budget allocations, adjustment of the level of

simplified grants as already proposed in 2017, as well as the reversal of the low success

rates

393

in some particular fields and sectors.

5.4.1. Cost-effectiveness

With regard to learning mobility of individuals under KA1, the relationship between the costs

(inputs) and effects (results

and impacts),

indicates a

positive cost-effectiveness

394

especially

for learners but also for staff

395

. Considering the positive outcomes identified above under

"effectiveness", the costs per individual appear clearly reasonable. This is further supported

below by the evidence of European added value of these types of actions, also compared to

similar programmes reviewed at national level. The average cost of a mobility activity is

1,500€ per learner (15€

per day/learner)

396

and between 700€ and 900€ per staff member

387

Efficiency considers the relationship between the resources used and the results of the programme with a view to

maximise yield for given resource or minimise costs for a given output.

388

ICF, 7; NRS, 6; OPC, 3.4

389

This includes the operating grant for National Agencies as well as the administrative expenditure of the Commission and

EACEA. In 2015, these management costs represented 19% of the budget managed directly by the Commission (mainly

covering costs for the development of IT tools for NA as programme stakeholders), 10% for EACEA and 5% of the budget

managed indirectly by NAs, which appears well in line with the different nature, complexity and volume of activities

managed respectively at each level; ICF, 7.4, tab. 7.18

390

Share of administrative costs in comparator programmes, from 10% (EEA) to 20% (Causeway Ireland); ICF, fig. 7.15

391

OPC, 3.4

392

ICF, 7.1 and 7.2; NRS, 6.1; OPC, 3.4 and 5.1.4

393

Funded projects out of the total number of applications

394

Cost-effectiveness analysis judges costs against the benefits achieved.

395

Although generally positive, more positive for learners than for staff: ICF, 7.3.4 and 7.3.5

396

Although the format of the mobility activities is quite standardised, the differences in average grant support are

determined by several factors, such as the average duration of a typical activity, the contribution to higher subsistence costs

in the case of staff mobility as well as the contribution to travel costs, which are normally higher in the case of mobility

activities from or to Partner Countries. The cost-effectiveness of learner mobility is clearly shown when comparing the low

costs to the results identified (see section on effectiveness). The costs per learner vary between roughly 900€ in the youth

kom (2018) 0050 - Ingen titel

(200€

per day/staff)

397

. An

increase in costs per learner has been measured compared to

the predecessor programme

as regards higher education (+9%) and VET (+1%)

398

. An

increase in costs per staff member is similarly observed in higher education (+22%), adult

education (+19%) and VET (+2%), while a decrease is noted in the case of school staff (-

6%)

399

. Although the impact of a mobility activity is more significant by nature for learners

than for staff, the effects on staff also create positive spill-over effects at the level of the

organisations/institutions as well as on the learners who are not necessarily taking part in

mobility activities. Hence,

the overall ratio of costs versus effects can be considered

positive in all cases.

There is also an European added value of the programme actions for

staff as for learners, since the comparison with national programmes is favourable for

Erasmus+, as demonstrated below under "added value".

As the grant amounts mentioned above may appear low, the degree to which these grants

cover expenses incurred has been examined. On average

for 42% of KA1 learners the

Erasmus+ grant covered most of expenses

(76-100%), but as the graph shows below, wide

differences are made between learner categories

400

. Erasmus+ is successful in leveraging

complementary funding from national or regional budgets

401

Share of expenses covered by the grant – by type of learner

80%

70%

60%

50%

40%

30%

20%

10%

HE traineeships HE international

0-25%

HE students

51-75%

VET learners

76-100%

Averages

26-50%

Source: ICF calculations based on Erasmus+ participant monitoring surveys

402

On the other hand, the conclusion is more nuanced as regards the

cost-effectiveness of

cooperation projects.

This is because there are

a variety of multilayer effects

having a

sector and 4700€ in the higher education international sector (average cost: 1,500€ per learner). The costs per day of mobility

vary between 10€ in HE sector (programme countries) and 60€ in VET (15€ on average per mobile learner per day).

397

The cost effectiveness of staff mobility is also positive though the judgement is more nuanced. The average costs per

mobile member of staff is between 700€ and 900€. The costs are lowest in HE (programme countries) and highest in higher

education international. The costs per day of mobile member of staff are around 200€ (ranging between 100€ in youth and

close to or more than 300€ in adult education, schools and higher education international).

398

IFC, tab. 7.7

399

IFC, tab. 7.10

400

The Erasmus+ grant covered the majority of expenses for HE international students (e.g. Erasmus Mundus scholarship is

intended to cover all costs: travel, master participation, subsistence) and VET students (72% and 68% respectively), whereas

this was the case for only 19% of HE trainees (whom are paid/ compensated by their employers) and 8% HE students (as the

grant is not expected to cover all costs of living, but only the additional costs of mobility abroad).

401

9 to 15% of participating learners do not receive a grant from the programme (i.e. so called zero grant beneficiaries),

which suggest an additional added value of the programme and a spill-over effect; ICF, 7.3.6

402

The participants who did not receive grant (about 8,5%) are also included into these results.

kom (2018) 0050 - Ingen titel

significant impact on the solid assessment of cost-effectiveness

403

. These include

differentiated budget items, as well as variations between their types and sizes across and

within sectors (around 180,000€ per project on average)

404

. In spite of efforts made for this

evaluation, the conclusion on cost-effectiveness for cooperation projects cannot be

generalised as for mobility activities. Although quantitatively, the cost-effectiveness ratio can

seem positive at first sight (widened

participation, multiple layers of effects),

on a more

qualitative note, there are elements that can be improved in terms of efficiency of

implementation (application

process)

and effectiveness (quality

of project outputs)

405

. As a

general trend, cooperation projects seem to be very well designed and conceived at

application stage although they are not able to keep the same level of excellence at the

implementation stage. The majority of projects receive indeed lower quality scores on

completion than at application stage. In contrast, given the relatively

low costs of

collaborative platforms

for the EU, these types of actions appear as particularly cost-

effective.

5.4.2. Size of budget

Budget envelopes for most of the sectors were regarded as insufficient

406

and strongly

correlated to low success rates

407

. In some actions or countries, this had led to a

discouragement of applicants over the first years of implementation of Erasmus+. The

potential of the programme for broader organisational and system level effects, as discussed

under "effectiveness", could be enhanced by reaching a critical mass across all the fields of

education, training and youth.

The

demand largely exceeds the funding available.

In KA2 the competition and the scores

for successful projects are high

408

, which makes it more difficult for organisations with no or

little experience to access the programme. Given the increased demand for the cooperation

projects, the

budget allocated to KA2 and KA3 did not appear to be sufficient yet in

absolute terms.

Even in KA1, success rates can be low depending on countries and actions,

meaning that more learners could benefit from the programme if there were more funds

available

409

. Without prejudice to negotiations on the next Multi Financial Framework, the

evaluation, including the public consultation, points to the need for reconsidering the budget

envelope and the need for reviewing the budgetary distribution between programme sectors to

have a better alignment to their relative effectiveness. However, there is no evidence that

seriously questions the current overall balance between Key Actions. The vast majority of

403

404

ICF, 7.3.5 and 7.3.6

Depending on the year chosen the average project size varies from 160,000€ to 200,000€. The smallest projects (around

55,000€) are in the youth sector while the largest ones in higher education and VET (250,000€ to 270,000€).

405

These aspects refer mostly to the following: a) "one size fits all" regarding application forms (KA2 mainly) that is not

proportionate, meaning that small projects have to supply the same information as much larger ones; b) the ratio between

complexity of forms compared to the type of results expected that is not balanced (in KA2 mainly); and c) low success rates

vs complicated application forms. ICF, 7.1 and 7.4; NRS, 6.1.2 and OPC, 3.4

406

The European Parliament (CULT committee) study, 2016,

Erasmus+: decentralised implementation - first experience

found that the views of the national agencies on the adequacy of programme funding to be almost equally split. The concern

of insufficient budget size is also confirmed by the review of National Agencies’ reports. The insufficiency of budget was

mentioned in 67 of the 2007-2013 NA reports, the most problematic being Grundtvig. Over 2014-2015, the second most

frequent problem reported by the national level related to the insufficiency of budget, especially for KA2. Two NAs

suggested a greater flexibility in budget transfers. Only 51% of OPC respondents (strongly) agreed that the budget of the

Erasmus+ Programme is sufficient to achieve the objectives set for the Programme (n = 584). ICF, 7.2.3

407

For instance, from 3% (Not-for-profit European sport Events), 4% (Knowledge Alliances) or 15% (Strategic Partnerships

for Youth) to 43% (Jean Monnet strand) in 2014. However, the 2016 selection resulted in improved success rates; e.g. 31%

for Strategic Partnerships for Schools; ICF, tab. 4.10 for full data.

408

No KA2 selection was above 25% rate over 2014-15.

409

ICF, 7.1; NRS, 6.1.1 and OPC, 3.4

kom (2018) 0050 - Ingen titel

respondents to the consultation agreed with the current budget distribution between the three

Key Actions set in the Erasmus+ Regulation

410

5.4.3. Implementation modes and user-friendliness

The division of responsibilities, as inherited from predecessor programmes, between the

Commission, National Authorities, National Agencies and EACEA, is overall

clear and fit

for purpose

411

. It has not significantly changed over the two programming periods, if it was

not for the decentralisation of the management of the higher education international (non-EU)

programmes and some changes in the level of management of certain types of cooperation

projects. The only outstanding challenge stems from the rigidity inherent to the addition of the

development-related requirements of the EU external cooperation funds which are also used

to promote the international dimension of higher education

412

. This requires a complicated

management of multiple small budget envelopes with different rules at decentralised level and

a disproportionate effort compared to the numbers of beneficiaries

413

The guiding role of the Commission with regard to the programme management has been

acknowledged by 63% of agencies, due to efforts made in terms of coordination and

communication. However, many National Agencies expressed a need for EACEA to enhance

its cooperation with the national level.

Above all, there is a

general call for further simplification

of the management IT tools.

Lowering the burden of the application procedure through redefining the type as well as the

volume of information required depending on the actions of the programme and making better

use of the reporting system (see below) would contribute to continued improvement of the

programme's implementation

414

5.4.4. Efficiency gains and simplification through changes in the integrated

programme

Overall,

efficiency gains were acknowledged by a majority of countries

415

, sectors

416

and

most of National Agencies

417

. In general, countries recognised a certain simplification of

administrative procedures, while participants expressed concerns about application and

reporting.

Due to the

lack of time for adjustment between the adoption and the entry into force

the legal basis (i.e. less than 3 months) and for business continuity reason (e.g. the constraints

of the academic year), all players (the European Commission, the implementing bodies of the

programme and a large spectrum of programme stakeholders) had to go through a

challenging inception phase

to adapt to the many novelties of Erasmus+ over the first two

years of its implementation. This was mainly due to the late adoption of the Erasmus+

Regulation (end of 2013), coupled with the necessity to implement the majority of the actions

410

411

71% of OPC respondents (strongly) agreed that this distribution is appropriate (n = 480).

80% of OPC respondents (n = 497) believe that the distribution of Erasmus+ Programme activities between centralised

(through EACEA) and decentralised management (through National Agencies) is effective.

412

See above background section about the different external instruments

413

ICF, 7.4.4 and 7.5

414

ICF, 7.1 and 7.4

415

In 19 countries a positive opinion seems to prevail (BEnl, BG, EE, IE, EL, ES, FR, IT, CY, HU, PL, PT, SI, SE, UK, IS,

FY, CH,TR). In 6 other countries, stakeholders see insufficient efficiency gains or no gains overall (CZ, LT, MT, NL, RO,

SK, FI,). In a few countries, national authorities report different opinions depending on the NA, sector or key action; ICF, 7.5

and NRS, 6.3

416

It is evidenced through a number of National Reports that the youth sector received least well the changes in the

programme, claiming the new youth chapter to be less efficient than the Youth in Action Programme. NRS, 6.3

417

79% of agencies states that they have seen efficiency gains compared to predecessor programmes. Nearly two in five say

they have seen large efficiency gains; ICF, fig. 7.19

kom (2018) 0050 - Ingen titel

already by the first trimester of 2014

418

, which significantly reduced the time for the

Commission, the National agencies and the Executive Agency to put in place the necessary

normative framework and technical infrastructure to implement the programme smoothly. The

new architecture of the programme, although it is now positively valued, led to an overhaul of

previous administrative rules, definition of new criteria and new ways of operating. A new

generation of IT tools has replaced the previous one and new automated support has been

introduced for a number of management tasks.

National agencies and authorities recognised that initial challenges were gradually overcome

and that there has been a steep learning curve with substantial efficiency improvement and

simplification over the period 2014-2016. The challenge of digitalisation that was put in

practice through the online management of applications and reports is a case in point, and so

is the generalisation of simplified grants

419

. The programme management has now reached

cruising speed.

On the

positive side,

different stakeholders and countries have acknowledged the difference

made with the introduction of: a) simplified forms of grants, mainly lump sums and unit

costs

420

, b) uniform application forms, c) digitalised application procedure

421

, d) possibility to

apply as an institution instead of as a participant, and e) a single website to get easier access to

all programme information

On the

negative side

some other stakeholders and countries did not consider that the

integrated programme makes it easier for potential applicants to understand the funding

opportunities and even claimed that the standardised approaches (e.g. applications, reporting)

made work more time consuming than in the past

422

. Although acknowledging some

administrative simplification introduced with Erasmus+, according to National Reports

423

participants express concerns about too complex application and reporting procedures. For

instance, beneficiaries responded that forms tend to be repetitive and some questions are quite

long and difficult to interpret.

Overall there are a number of areas that

need further improvement and simplification,

such

as: a) better

field-customisation

of the unified procedures, b) further simplification of the

application and reporting

procedures, c) making the programme more

accessible to smaller

418

The first deadline for submitting grant proposals for mobility and cooperation actions under Erasmus+ was 1 February

2014

419

Study

Mid-term review of simplified grants used in the Erasmus+ programme,

DG EAC/PPMI, July 2017

420

Simplified grants

gave overall satisfaction to both National Agencies and projects’ coordinators. They have reduced their

workload, simplified budget planning, as well as project management and reporting. They give more flexibility to

participants, who can manage what they receive according to their needs. Most respondents advocated against any major

changes in the funding rules, due to efforts that were required to adjust to the current system. It has nevertheless been

suggested adjusting the level rates in some cases, especially for the longest distance bands, as well as introducing a number of

fine-tuning changes. See for specific evaluation findings: Study

Mid-term review of simplified grants used in the Erasmus+

programme,

DG EAC/PPMI, July 2017

421

92% of OPC respondents (strongly) agreed that the digitalisation of Erasmus+ is a progress (n = 338), and 89% that the

user-friendliness of IT tools in the Erasmus+ Programme has improved over time (n = 597), in particular as from 2016, when

IT tools were substantially streamlined.

422

31% of agency respondents (rather/strongly) disagree that the integrated programme makes it easier for potential

applicants to understand the funding opportunities. This negative sentiment is corroborated by several key interviews. A

majority of interviewees (national) claimed that the standardised approaches (e.g. applications, reporting) make work much

time consuming than in the past. Benefits were however pointed by several interviewees.

423

BEnl, BG, CZ, DK, DE, EE, IE, EL, ES, FR, HR, IT, CY, LU, HU, MT, LT, LV, NL, AT, PT, RO, SK, FI, SE, UK, IS,

FY, LI, NO, CH, TR.

kom (2018) 0050 - Ingen titel

organisations

424

, d) decreasing the time required for project management, especially in

KA2

425

, and e) making the terminology used

more user-friendly

426

In comparison with the previous programmes, the main changes which agencies associate

with

efficiency gains are online applications and reporting

as well as the fact that there is a

smaller diversity of types of actions to manage. However, the agencies are not very positive

about the evolution of their own administrative workload. There is also

room for

improvement

concerning the lowering of the administrative burden, as shown in the graph

below

427

To what extent do you agree with the following statements about efficiency gains

for your agency under Erasmus + compared to predecessor programmes?

Source: Survey of programme agencies

The rationale of the

economies of scale

expected from the consolidation of several

programmes into one was that an integrated programme with fewer but bigger actions/projects

would be simpler to administer, while the use of unit costs would further drive the

management costs down. If the model proposed in 2011 by the Commission to have a single

National Agency per programme country had been adopted in 2013, it would have allowed

according to estimates "to reach a cumulative effect of these simplifications to a productivity

424

Regarding the performance of success rates, country reports (BEnl, CZ, DE, EE, EL, ES, FR, HR, IT, CY, LT, HU, MT,

NL, AT, PL, PT, RO, SK, SE, UK, FY, CH, TR) alert that success rates are very low in some of the actions and fields.

Success rates appear to be particularly low in KA2, the youth field and adult education sectors.

425

Administrative procedures in KA2 are perceived particularly time-consuming (DK, LT, LV, HU, MT, NL, AT, LI, CH).

426

For instance, in the context of school education, references to "innovation", "intellectual outputs" and "multiplier events"

cannot be expected to be well understood by stakeholders

427

ICF, 7.5

kom (2018) 0050 - Ingen titel

increase of around 40%"

428

. This objective however could not be a target for Erasmus+, as

most countries continued to manage the programme through several National Agencies.

Management fees for NAs – LLP, Youth in Action and Erasmus+ (% of overall funding)

Source: ICF calculations based on a review of annual work programmes (% of overall funding)

The

management costs of National Agencies

compared to the programme value are

smaller

than under predecessor programmes,

when looking at LLP and Youth in Action. The

comparison looks inevitably less positive when taking into account the overall budgets of

predecessor international higher education programmes (Erasmus

Mundus, Tempus, etc.),

their complexity reduces the average efficiency of Erasmus+. This is because the

decentralised actions with partner countries have specific criteria and 12 different budget

envelopes within the EU external relations funds. In any case, the comparison is not a simple

one. There are important limitations in assessing the change of management fees: a) the base

is not comparable between 2007-2013 and 2014-2016, as some actions that used to be grants

managed by National Agencies have been moved to agencies' core budget and b) the

comparison with management costs of the Executive Agency is not available since EACEA

used to manage all international higher education mobility actions, of which an important

share has been decentralised to National Agencies.

Nonetheless

more efficiency gains are still expected to materialise,

as the Erasmus+ budget

will increase in the second half of the programming period whereas the management costs

will proportionally decrease. It is therefore too early to conclude at mid-term and the actual

cumulative effect on efficiency will have to be measured after 2020.

5.4.5. Monitoring mechanisms

The programme

monitoring has seen major improvements with Erasmus+

compared to

the predecessor programmes. Much more qualitative (e.g. beneficiary surveys showing results

beyond outputs) and comprehensive data is now available (e.g. international HE mobility

being now covered as well)

429

. Data is also more systematically used and disseminated

through new tools dedicated to Erasmus+, namely an on-line

Dashboard

for internal

management purpose and an

Annual Report

for public accountability purpose

430

Erasmus+ was expected to have a

much clearer performance-based management

than

predecessor programmes and indeed arrangements have been put in place to allow regular

monitoring:

key performance indicators

have been defined in the legal basis as well as in

428

429

Erasmus+ Impact Assessment,

Commission Staff Working Paper, SEC(2011) 1402 of 23.11.2011

See respective sections on method and effectiveness, as well as annex 3

430

http://ec.europa.eu/programmes/erasmus-plus/about/statistics_en

kom (2018) 0050 - Ingen titel

DG EAC's

Strategic Plan

431

, data is collected on ongoing basis in relation to most of these

indicators and targets have been set

432

However, as partly discussed in the section on effectiveness, to lower the

reporting burden

on beneficiaries

433

, there is room for improvement when it comes to: a) clarity and relevance

of some output indicators, as well as of the quality of the data

434

, b) robustness of the self-

reported results indicators, c) proportionality between the related burden on beneficiaries and

the actual use of the data, d) balance of monitoring efforts according to types of action, e)

user-friendliness and further inter-operability of IT tools

435

, f) and promotion of truly

performance-based management at the level of programme agencies. At Commission's level,

the reporting is used primarily for accountability; greater use of the information should be

made for programme monitoring and continuous improvement purposes

436

When looking at comparable national programmes, the monitoring arrangements of Erasmus+

are nevertheless

in line with existing practices

and perform well, given the size of the

programme

437

. Availability of objective data for results (e.g. based on actual pre-post

language proficiency assessment through the OLS), as well as availability of large volume of

self-reported feedback collected through systematic beneficiary surveys are two areas in

which Erasmus+ monitoring practice can even be considered better than that of other national

comparable programmes

438

Overall, the

monitoring mechanisms are effective

when it comes to providing an up-to-date

view of programme implementation to the Commission, the agencies and national authorities

but also to the general public, in particular via the Erasmus+ Project results platform

439

, as

discussed under "dissemination". However, there is

less evidence of the use of the data for

management and planning

purposes, which may be due to the fact that the monitoring

arrangements have only been enhanced since a couple of years

440

5.4.6. Anti-fraud measures

Overall there is a

limited scope for fraud

in the types of actions carried out under the

programme. The main concerns are project coordinators failing to honour obligations to other

partners, and possible multiple submissions of project applications.

According to the programme legal basis, a system of controls has been in place to prevent

fraud and irregularities at the EU and national levels

441

. The number of fraud cases reported is

very low compared to the scale of the programme,

with agencies proceeding to recover the

431

432

https://ec.europa.eu/info/publications/strategic-plan-2016-2020-education-and-culture_en

See annex 5e reporting on all output and result indicators from the legal basis

433

Disproportionately more information is collected for KA1 compared to other Key Actions; ICF, 7.6.1

434

E.g. various definitions of disadvantaged learners or why is country coverage is relevant indicator for Jean Monnet

435

As a legacy of the past, several IT tools co-exist supporting various monitoring functions, such as control and detection of

irregularities, accountability and transparency, as well as programme management. See annex 3 for description of tools

436

Only anecdotic examples have been found of agencies that take ownership of the data available and use it for their own

purposes of management and evaluation; ICF, 7.6

437

Looking at the comparator schemes, the level of monitoring is highly variable. For instance, four programmes collect

feedback from the participants (CEEPUS,

CTEP, Denmark-USA programme, German-French elementary school teacher

exchange);

others gather regular annual monitoring data (AKTION). Some perform regular evaluations, for example, the

IACOBUS Programme

started in 2014 and its activities in 2014-2015 have already been evaluated. The final evaluation of

the

International Inspiration Programme

(2007-2014) was conducted by an independent evaluator.

Prämienprogramm

has

annual reports for 2013 and 2014 and was audited by the German Federal Court of Auditors.

438

ICF, 7.6.5

439

http://ec.europa.eu/programmes/erasmus-plus/projects

440

ICF, 7.6.1

441

Guide for National Agencies: Implementing the Erasmus + programme

kom (2018) 0050 - Ingen titel

necessary amounts

442

. The vast majority of National Agencies (71%) consider that the

measures are fully sufficient

443

. Therefore the

anti-fraud strategy in vigour is considered

appropriate.

A few agency respondents made suggestions, such as using IT-tools to verify plagiarism

444

As the programme moves to its later phases, more checks could be performed on the

accumulated population of grants. In any case, the ratio of administrative burden compared to

the low number of cases reported and the money recovered is to be carefully considered.

442

No more than one or two cases per country in average and an overall low error rate indicated through primary checks of

less than 1%, in line with the predecessor programmes (DG EAC's Annual Activity Report 2016).

443

ICF, fig. 7.23

444

ICF, 7.7

kom (2018) 0050 - Ingen titel

5.5.

European added value

The evaluation highlights the

strong European added value

445

of EU programmes in the

fields of education, training and youth

446

The additional benefit of Erasmus+ was compared to what could be achieved at regional,

national or other international level (see 5.5.1), to what was achieved by EU predecessor

programmes (see 5.5.2) as well as to what would be the most likely consequences of

discontinuing the programme (see 5.5.3).

The European added value of Erasmus+ has been informed by 6 evaluation questions and

assessed against the following criteria that examined to what extent Erasmus+ supports:



a scale of actions between countries which would not otherwise be achieved (volume

effects);



target groups or sectors that would not be otherwise covered (scope effects);



processes which participating countries translate further into their own practice

(process effects)



cooperation between programme countries (integration effects)



innovation which would not be otherwise mainstreamed so widely (innovation

effects);

5.5.1. European added value as compared to what could be achieved at

regional, national or other international level

Firstly, two evidence sources show

clear added value of the programme due to its strong

volume effects

447

, except in the cases of the Student Loan Guarantee Facility

448

, sport

449

and

Jean Monnet activities

450

. The programme manages to reach out to substantially more

learners, staff and organisations than all comparable programmes reviewed taken together, as

well as groups that are much more rarely covered otherwise. The benchmarking analysis

shows in particular that the scale of Erasmus+ is much bigger than that of other comparable

schemes reviewed in the fields of education, training, youth and sport. There are only few

other actions that support credit mobility and they are of a much smaller scale

451

. In terms of

"market share", it is likely that Erasmus+ represents about half of credit mobility in EU higher

education

452

. It is also likely that the share of Erasmus+ in VET mobility is significant,

445

European added value

analysis looks for changes due to the EU programme over and above what could reasonably have

been expected from actions by other players let alone or from no action at all. It draws on the findings of the other evaluation

criteria, to conclude whether the programme is still justified.

446

ICF, 8; NRS 7; OPC 3.5

447

The assessment of volume effects is mostly based on the benchmarking of Erasmus+ against comparator programmes and

the agency survey.

448

For the Student

Loan

Guarantee Facility (SLGF), the European added value has not yet reached its full potential given the

very limited scale of the action at the time of this evaluation.

449

In the

sport

sector, the assessment of European added value is made difficult by the fact that the strand is too recent.

Another challenge is the fact that limited funding appears spread over too many different issues.

450

The European added value of

Jean Monnet

activities has declined over the years since the programme was initially

launched, as the volume of research and teaching about the EU grew and has become a common practice in higher education

within the EU. This growth cannot be related to Jean Monnet funding as this represents only a small share of research and

teaching activities. There is however potential for greater European added value targeting other audiences which have fewer

opportunities to learn about the EU.

451

ICF, fig. 8.3: The comparable programmes comprise from 10 to 3,522 participants each.

452

In 2015 the total number of EU higher education student who benefitted from

degree

mobility had reached 331,078

(Source: Eurostat (2017), Degree mobile graduates from abroad, figure lacks input for Spain, Greece and Poland. Available

at:

http://ec.europa.eu/eurostat/web/products-datasets/-/educ_uoe_mobg01).

Higher education learner

credit

mobility

supported by Erasmus+ that year had reached nearly the same number (301,267 individual students). As these two modes are

mutually exclusive it can be assumed that there were around 650,000 mobile students in total. However this calculation is

only rough as the Eurostat data covers full degree mobility and not credit mobility. There is no comparable data about

kom (2018) 0050 - Ingen titel

although available data is not of sufficient quality to calculate it

453

. Most respondents from

National Agencies consider that Erasmus+ funds more than 75% of actions of a given type, as

illustrated below

454

Perceived market share of Erasmus+ (number of responses)

Source: Survey of programme agencies

Secondly, three evidence sources point to the conclusion that

Erasmus+ scope effects are

significant

455

. Comparable national and international actions do exist (e.g. bilateral

programmes, philanthropic actions, etc.) but these are more focused on mobility than on

cooperation actions

456

. The Erasmus+ programme covers fields rarely covered by other

international interventions except in higher education. Only in the latter sector was it possible

for two-thirds of National Agencies to identify a number of comparable although much

smaller programmes

457

. For all other fields or types of actions less than half of respondents

could identify comparators

458

Erasmus+ has its own and unique boosting effect on mobility and cooperation in every

field it targets.

The findings from programme benchmarking

459

and agencies' survey (see fig.

below) reinforce each other to provide a solid case in that respect. None of the comparable

schemes combine as much as Erasmus+, in terms of country, sector as well as action

coverage. Moreover with a dedicated strand (KA3) since 2014, the EU programme

mobility for staff nor for students in other sectors. "Identifying mobile students, as well as their types of learning mobility, is

a key challenge for developing international education statistics since current international and national statistical systems

only report domestic educational activities undertaken within national boundaries" (OECD, Education at a Glance 2017).

453

A pilot study on IVET and general youth learning mobility (Eurostat, 2015) estimated that roughly 3% of VET learners

from 16 countries (estimated as about 320,000 learners by ICF) benefit from mobility actions, but the study acknowledges

that the data is not of sufficiently good quality to be extrapolated. ICF, 8.2.2.2: Erasmus+ supported around 150,000 VET

learners in these 16 countries over the first three years (common duration of VET studies).

454

Only when looking at system level actions a relatively high number of agency respondents thought the share of Erasmus+

was below 50%

455

Based on the agency survey, data about student mobility in general as well as data on comparable programmes

456

Out of the 58 comparable programmes shortlisted for benchmarking purpose, 2/3 supported mobility, while only 1/3

supported cooperation actions.

457

ICF, fig. 8.1

458

The agencies were asked about their awareness of other comparable interventions. The results as regards sectors shows

that around two thirds of respondents were aware of comparable actions in HE, slightly less than half in SE or in the Youth

sector, less than one third in AE, fewer in VET. As for types of action, 43% of respondents from National Agencies knew of

similar programmes that covered mobility of learners; this share was only of 42% and 38% for mobility of practitioners and

cooperation among organisations respectively.

459

Out of the total 58 benchmark programmes, around half cover SE, over 35% HE, below 20% VET or the Youth sector,

10% AE. Their target groups are students in nearly 80% of the 58 schemes, teachers at around 22%, youth at around 17% and

adults only around 10%.

kom (2018) 0050 - Ingen titel

encompasses system level cooperation actions that are very rarely supported by other

programmes.

In addition, the evaluated EU programmes show

specific added value to the benefit of

disadvantaged groups

whose opportunities for mobility largely result from the EU

intervention, more than from national initiatives. This is all the more true in the youth and

sport fields where there are fewer comparable programmes

460

. Over two thirds of the agency

respondents agreed with the notion that Erasmus+ offered more opportunities for

disadvantaged groups in comparison to other similar schemes. In all comparable fields,

Erasmus+ stands out as having the largest and most inclusive outreach

461

Comparison between Erasmus+ and other comparable initiatives

Source: Survey of agencies

Both

volume and scope effects magnify the impacts

of the programme as detailed above.

Thirdly, there are clear examples that the

evaluated EU programmes have influenced other

comparable programmes both in terms of process or types of actions

supported.

At national level,

Erasmus+ and its predecessors have contributed to shaping the

framework for student and staff mobility or youth exchanges across Europe

462

. For

instance, quality frameworks - in particular the concept of a European Quality Charter for

Mobility introduced first for higher education and extended to VET due to its success - are

also used for national schemes

463

. Best practices of EU programme management are often

reported to be transferred to national programmes

464

. Other comparable actions sometimes

follow the same rules

465

as Erasmus+. Generally speaking, among National Agencies which

manage programmes other than Erasmus+, 71% of respondents acknowledge a spill over

effect from the EU programmes under review to their own interventions

466

. These are good

examples of unintended positive effects of the evaluated programmes. Furthermore, 91% of

respondents to the public consultation strongly agreed or agreed that lessons learnt from the

460

Only 1 out of 6 benchmarking programmes had specific measures addressing the needs of disadvantaged groups. Only 3%

were directed at the sport field.

461

See sections on effectiveness and relevance about the share of participants with a disadvantaged background

462

E.g. At EU level, the "Bologna process" for HE or the "Copenhagen process" for VET. At national level, the European

concept of youth exchanges has inspired for instance the national programme "Youth meetings".

463

E.g. The quality assurance and administration of Nordplus mobilities (among DK, EE, LV, LT, FI, SE, IS, NO) are

inspired by Erasmus+.

464

E.g. accessibility grants for participants with special needs (NordPlus Higher Education)

465

E.g. Erasmus Belgica (among BE-de, BE-fr, BE-nl) follows the same rules as Erasmus+.

466

ICF, fig. 8.5. N.B. the number of agency responses (n=60) is particularly limited on this point.

kom (2018) 0050 - Ingen titel

Erasmus+ actions (which they were most aware of) are being applied elsewhere. In particular,

processes established to manage mobility have been mainstreamed into other national and

European interventions.

Several spill-over effects on process

have also been identified at EU level. Erasmus for

young entrepreneurs shares comparable approaches with Erasmus+ mobility. Schemes

launched recently, such as the European Solidarity Corps

467

or the EU AID scheme for

volunteers in humanitarian aid

468

, follow similar selection principles as the European

Voluntary Service (EVS).

Fourthly, as discussed in the effectiveness section, the network analysis showed that there are

overall

increasingly good levels of integration and cooperation between programme

countries

with some that were initially more peripheral moving to more central positions.

Trends in interconnections show that the programme is not necessarily dominated by large

countries. From a systemic perspective, the greatest added value comes from cooperation at

the level of staff and organisations and that of policy makers and stakeholders. As regards the

latter, many Open Method of Coordination activities organised in the context of ET2020, the

EU Youth strategy and the EU Work Plan for sport are funded through Erasmus+. The EU

programmes reviewed definitely enable cross-country cooperation and integration at a scale

that is incomparable to other actions.

While its volume, scope, process and integration effects are substantial,

the innovation

effects of Erasmus+ appear to be more modest.

This is the only type of added value

analysed where the results are less positive. The evaluation of effectiveness showed above

that while there are some examples of innovations that emanate from funded projects, these

are rather ad-hoc, soft and of modest scale compared to the volume of projects funded rather

than significant and mainstreamed. Although Erasmus+ has potential to enhance innovation

(collaborative approaches, specific KA3, brand attractiveness), its added value cannot be said

as emanating significantly so far from a role model in that respect.

5.5.2.

European added value of Erasmus+ as compared to what was

achieved by its predecessor programmes

Compared to its predecessors,

Erasmus+ manages to

reach out to more learners,

practitioners and organisations.

This scale potential results from the integration of several

programmes into one and from

a 40% increase in budget in comparison to 2007-2013.

This

increase has not yet fully materialised as most of it will be observed as of 2017. Meanwhile

the integration has generated efficiency gains

469

Through its well-known branding, the current programme benefits from

a better visibility

media/social media

470

, but also in policy making accompanied with a

strengthened

alignment with policy priorities

471

. The fragmented nature of its predecessors did not give

such a visibility. Increased attention has been continuously paid to synergies between the

programme and policy priorities in education, training, youth and sport. In particular, the

introduction of a Key Action devoted to system level support (KA3) is a novelty valued by

467

https://europa.eu/youth/solidarity_en

468

http://ec.europa.eu/echo/what/humanitarian-aid/eu-aid-volunteers_en

469

In particular thanks to the decentralised management mode through agencies but also increasingly on the side of

beneficiaries; besides certain processes (e.g. to manage mobility) have been streamlined from one sector to the others. See

section on efficiency

470

See section on relevance

471

See section on external coherence

kom (2018) 0050 - Ingen titel

most respondents concerned. It is nevertheless too early to evaluate whether this programme

will have achieved stronger system level changes.

Thanks to the integration of several programmes, the degree of

cross-sectoral cooperation

has increased

compared to the period 2007-2013. This is mainly due to the rising

participation of higher education organisations in other sectors and to a lesser extent to some

increase in participation of companies and public authorities.

The Erasmus+ programme has also been designed to pay

more attention than previously to

information about performance,

which helps to allocate the budget more appropriately. The

monitoring of Erasmus+ is much more developed than in the past. The dissemination and

exploitation of project results is underpinned by a clearer strategy and more systematic

publication on line. This represents actual progress, even though in both cases newly set-up

tools are not all as user-friendly as expected and though data remains insufficiently

analysed

472

The

international dimension,

where it exists (i.e. in higher education and youth), has been

made clearer with Erasmus+.

The integrated programme has addressed the fragmentation

issues raised in the interim evaluations of its predecessors (Erasmus Mundus, Tempus, Alfa).

There remains room for improvement notably in the management, visibility and

communication with regard to international actions.

From one programme generation to the next, the question of repetitive actions has to be

assessed

473

. Here again

added value is reflected in the differentiated level of repetitions.

Each year, EU programmes reach out to a new cohort of learners (i.e. though mobility is

repetitive it addresses different audiences each year), whereas the repeated participation of

practitioners and organisations in the funded activities over time enables the respective

actions to achieve deeper effect.

Overall, most of respondents to the Open Public Consultation value Erasmus+ for its

undisputed benefits

474

. More than 95% of respondents agreed or strongly agreed that it is

funding

activities which would not have been made possible otherwise,

and that it

contributes to improving what is already available at other geographical levels for the

education, training, youth and sports sectors

475

Most of the National Authority reports confirm the above findings by stating that Erasmus+

and its predecessor programmes produce

effects that could not have been attained through

actions at national level alone.

Country reports confirmed by the public consultation

responses highlight the impacts of Erasmus+ on mobility within the EU and beyond,

cooperation opportunities and internationalisation of organisations, quality improvements

based on learning from others, as well as the promotion of common European values and

intercultural awareness. Erasmus+ added value is more strongly praised in the youth sector,

while in the sports field, the European added value is rather acknowledged where activities go

beyond professional sport, such as cross-border cooperation between grassroots-level actors

476

472

See section on efficiency

473

See section on effectiveness

474

OPC, 3.5 (n=1264 to 1044). Opinions among individuals replying in their private capacity and representatives of an

organisation are consistent.

475

Meanwhile 86% (n=1111) agreed or strongly agreed that Erasmus+ is providing additional support to already existing

activities at the national, European or international level.

476

See for instance National Report of FI

kom (2018) 0050 - Ingen titel

5.5.2 European added value as compared to what would be the most likely

consequences of discontinuing the programme

The absence of Erasmus+ would result in a steep decrease in the volume, but also in the

quality, equity and efficiency

of both mobility and cooperation among the programme

countries and worldwide

477

. This would entail in turn a negative impact on education,

training, youth and sport sectors, as detailed under effectiveness above.

As shown in the efficiency section, Erasmus+ is managed particularly efficiently due to

economies of scale.

A decline in efficiency would result from the inevitable downscaling

of coordinating structures. According to National Reports, collaboration between institutions

and organisations at European level would be more complex to realise without Erasmus+. If

National Agencies

478

, European networks with central support function and European

collaborative platforms would not exist, the efficiency of comparable schemes, not to say their

mere existence, would probably be put at risk in many countries.

Both for mobility and cooperation actions, the absence of Erasmus+ might be compensated in

certain countries and sectors (mainly higher education) only in scaling up other comparable

schemes. However, when asked in a Delphi survey about plausible alternatives to Erasmus+,

the experts consider that these partial and

uncoordinated compensatory initiatives would

be highly unlikely to reach the scale of Erasmus+

479

. To achieve the full range of its results,

a greater variety of actions than the three main types of actions under Erasmus+ would be

needed. No single alternative would result in comparable results.

Unequal access to mobility and cooperation abroad would once again become a reality,

as was the case when these EU programmes did not exist. National Reports note that

participation in alternative activities would probably be linked more directly to organisational

or local resources. Therefore, differences in international opportunities between countries as

well as between sectors would increase as some would not be able to afford funding

international activities from national funds. As a consequence of the above, differences would

also increase between wealthy and less wealthy organisations or individuals, as confirmed by

the monitoring survey among higher education learners

480

and the Delphi survey among

experts

481

This in turn would

lead to a decline in positive adherence to the European project,

especially among young people, and a much weaker international outlook of staff. It would

also marginally negatively affect their transversal skills, especially foreign language

acquisition, though other channels to develop these skills would continue being used. Over

time the connections between organisations and staff would fade out, which would eventually

have a knock-on effect on learners and outcome recognition. Together this would lead to

much smaller international outlook of education, training, youth and sport sectors, including

beyond the EU, which would generate negative externalities given the globalised nature of

477

478

Source: Delphi survey and ICF, 8.4. extrapolating from evidence presented in part 5 of this document

NAs would certainly not survive a cut of EU programmes, even though they often manage other schemes since these are

of a much smaller scale, as shown above.

479

Confirmed for the international dimension of the programmes in

Evaluation of the EU Development Co-operation Support

to Higher Education in Partner Countries (2007-2014),

(2017), Particip, LeA, ECDPM, Ecorys, already mentioned:

"Probably

no single EU MS or even group of MS on their own would be well placed position to take the lead in organising

and managing a highly complex programme in support of global HE."

480

Two in five higher education learners would have not gone abroad without receiving the Erasmus+ grant (ICF, tab. 6.12).

However, one in five would have probably gone abroad also without receiving it with the remaining one third being unclear.

This suggests that there is some level of deadweight in the higher education mobility supported but this remains a minority of

beneficiaries. Comparable data is not available for VET in Erasmus+ monitoring survey.

481

According to the Delphi expert survey, given the budgetary constraints, alternative funding sources, including from

organisations and beneficiaries themselves, would not compensate in the same scale. This would in turn deepen the financial

barriers to access. Higher education would most likely see some compensation but inequality between sectors would increase.

kom (2018) 0050 - Ingen titel

modern economies. In other words,

bringing Erasmus+ to an end would leave negative

effects way beyond its direct beneficiaries.

In a nutshell, it might be worth quoting the High Level Group on Education and Training

concluding under the Estonian Presidency of the EU Council that "The

Erasmus+ programme

and its preceding programmes have 'created the first generation of young Europeans and

therefore have

the highest possible added value for Europe

as well as the potential to

promote common European values, empower young people, tackle socio-economic challenges

and foster competitiveness."

482

Meeting of the High Level Group on Education and Training, 14-16 June 2017, Tallinn

kom (2018) 0050 - Ingen titel

ONCLUSIONS

Overall, the programmes evaluated have proved to be

highly effective.

Erasmus+ appears

more coherent, relevant

and only

partly more efficient

than its predecessors.

The external evaluator made

11 recommendations

483

to the Commission to address

weaknesses of the current Erasmus+ programme. These recommendations will be followed-

up as stated in the Commission report on the mid-term evaluation of the Erasmus+

programme (2014-2020).



Five recommendations are

cross-cutting

(with a view to boosting respectively:

inclusiveness for more vulnerable groups; strategic investment in the sectors with highest

potential performance (schools, VET, youth); policy prioritisation; relevance of

innovation potential at sector level; systemic impact of projects through further

involvement of policy makers);

Four recommendations concern

specific areas

(aiming at addressing the relative

weaknesses of the Student Loan Guarantee Facility; Jean Monnet activities; adult

education; sport);

Two recommendations are more

process-oriented

(aiming at simplification and better

monitoring).



Effectiveness

The current programme and its predecessors deliver a

unique package of outcomes

in the

field of education, training, youth and sport. When concerns are expressed, these relate to

specific parts of the programme or to certain modalities of disbursement.

At mid-term, Erasmus+ is on track to

achieve or to exceed most of its performance

indicators,

as set in the legal basis. Beneficiaries report above 90% satisfaction rates for

learners and even higher for staff. In the period 2007–2016, all programmes reviewed have

supported together more than 4.3 million learners and 880,000 practitioners as well as

940,000 organisations. In terms of volume of outputs, the

main exception concerns the

Student Loan Guarantee Facility.

Even if the feedback received from its first users has been

positive, it has not yet lived up to volume expectations due to delays of its launch, low take-up

among financial institutions and lack of awareness among students.

The evaluation went beyond these indicators, which are not all equally relevant. A

counterfactual approach (quantifying the contribution of the programme in measuring the

difference in results between beneficiaries and non-beneficiaries) was used for the first time

across the programme, making this evaluation very reliable and more valid than previous

ones. However such an approach cannot exclude the possibility that the difference in

outcomes between beneficiaries and non-beneficiaries could be partially due to the selection

into the programme (in attracting the most motived individuals for instance) rather than purely

the effect of the programme. Based on this very sound methodology, the evaluation identified

a broad range of results and impacts at the level of learners (with particularly outstanding

effects of mobility), practitioners, organisations (in particular their internationalisation,

including beyond the EU) and also, to a certain extent, systems and policies (fostering the

recognition of learning outcomes across borders).

At individual level

483

ICF, 9

kom (2018) 0050 - Ingen titel

For most individuals, including when compared with the situation of non-beneficiaries, their

experience abroad has contributed not only to their

skills'

development and future career

prospects but also to deep changes to their

personal development.

The programmes

evaluated have stimulated networking among both learners and staff. Finally, they have

contributed to the development of openness to other cultures and

positive attitude towards

the European project.

Yet the evaluation points to the

need to do more to reach out to the

more vulnerable

in society and to facilitate the participation of smaller-size organisations.

At organisational level

The programmes reviewed have demonstrated a range of soft effects on organisations,

particularly a clear internationalisation of organisations, in and outside the EU. The

changes

at organisational level are however progressive and small scale

and continued

participation is needed for deeper transformations.

At system level

In the long run, the programmes have led to

a cultural shift in Europe in the way learning

mobility abroad is perceived and its learning outcomes are validated and recognised.

The programmes evaluated have also been

important for increasing the EU's global

outreach.

Though less visible, the evaluation also confirmed that the

programmes reviewed have had

a direct impact on education, training, youth and sport policies

where the related budget

was large enough or had an indirect effect by funding policy cooperation between authorities.

The system level changes through critical mass are much clearer in the higher education

sector than in other sectors which receive comparatively much less funding.

Systemic effectiveness is less in evidence for Jean Monnet, adult learning and sport,

where funded EU actions lack the critical mass and/or a sufficiently relevant focus. The

evaluation underlines in particular that the impact in the adult learning sector is diluted due to

the fragmented and diverse nature of the sector.

More generally, the resources for mainstreaming project results are spread across too many

topics at EU level and are insufficient at national level. Although local innovation (producing

something new for participating organisations) is achieved by the programme, this

innovation

does not enough follow the state of the art

developments in a given sector. Lastly, the

evidence of the

exploitation of project results by policy makers is not always clear,

especially when the latter are not engaged in the project from the beginning.

Coherence

The evaluation found a

high external coherence

between Erasmus+ and other relevant EU

policies and programmes (e.g. European Social Fund, Horizon 2020). Although the level of

synergies differs, it is notable that the evaluation detected

very few overlaps.

The

internal coherence of the programme resulting from the lifelong learning coverage

is strong as well. The further integration of fields often kept separate at domestic level into a

single EU programme is increasingly perceived as facilitating complementarities and

international outlook. This

consistent scope

sends the message that the learning opportunities

offered are equally important for the development of people, be it through formal or non-

formal and informal learning.

There is a

strong degree of cross-sectoral cooperation

within Erasmus+, which has

increased sharply compared to predecessor programmes. The majority of cooperation projects

in all sectors include at least one organisation which can be considered as coming from

kom (2018) 0050 - Ingen titel

another sector. There is an overall consensus that

an integrated programme boosted the

promotion of the actions

and the visibility of the different sectors.

Coherence

can still be improved in relation to sport

in reducing overlaps with non formal

education activities.

Relevance

The evaluation showed that Erasmus+ is

better aligned with EU policies

than its predecessor

programmes and entails sufficient flexibility to adapt to EU-level emerging needs. However,

to maximise the programme's impact, the evaluation recommends that

priorities at selection

stage be reduced and better focused.

The

contribution of Erasmus+ to a more cohesive Union

is also clearly evidenced. The

programme reaches out to disadvantaged people more than previous EU programmes or

comparable national schemes, although more still needs to be done to reach out to the more

vulnerable in society as outlined above. The geographical balance has also improved with

small countries and countries from Central and Eastern Europe being better integrated than in

the predecessor programmes.

The evaluation also noted that there is potential for introducing better targeted actions to

maximise the relevance of Jean Monnet activities.

Teaching and research on EU matters is

now relatively wide-spread in higher education. In contrast, there is a need to strengthen the

understanding of European integration and sense of belonging to Europe amongst the

youngest generations.

Efficiency

The economies of scale linked to the integration of several programmes did not deliver all

efficiency gains that had initially been anticipated in 2011 based on a "one agency-one

country" model which was not adopted by the legislator in 2013. Nevertheless, the

costs of

programme management appear reasonable

(6% of Erasmus+ administrative and

operational budget) and, in any case, lower than for similar programmes at national level

(14% in average). More efficiency gains are expected to materialise during the growing phase

foreseen by the budget profile of Erasmus+ until 2020.

The hybrid combination of different

programme management modes

and bodies is

fit for

purpose

with a good overall coordination and no major inefficiencies identified. Through

decentralised actions, the programmes get close to their target audience and offer the

possibility to align with national priorities. Very positive feedback was received on the

support provided by and to National Agencies. Centralised actions align more with EU level

priorities and help to respond to urgent political priorities. Some National Agencies report a

need for more communication with the Executive Agency (EACEA) to maximise synergies

between centralised and decentralised actions.

The first two years of Erasmus+' implementation were, however, very challenging for all

players. This was mainly due to the

late adoption of the Erasmus+ Regulation

(end of

2013), coupled with the necessity to implement the majority of the actions under the new

architecture and

with a new generation of IT tools already by the first trimester of 2014,

which significantly reduced the time for the Commission, the National agencies and the

Executive Agency to put in place the necessary normative framework and technical

infrastructure to implement the programme smoothly. National agencies and authorities have

recognised that

initial challenges have been gradually overcome

and that there has been a

steep learning curve with substantial efficiency improvement over the period 2014-2016.

Despite this difficult start, the programme has now reached its cruising speed.

kom (2018) 0050 - Ingen titel

Simplification

The

integration has led to greater simplicity of programme architecture,

notwithstanding

the separate chapters for Jean Monnet and sport, which is beneficial for both beneficiaries and

those in charge of management. It also makes programme monitoring more comprehensive

and clearer, although there is

room for a smarter collection and a better exploitation of

data,

enhancing transparency and accountability. The counterfactual approach (results

compared between beneficiaries and non-beneficiaries) should be pursued and developed for

evaluating the next programme.

The

procedural burden on beneficiaries

(application, reporting, etc.) is the main recurrent

issue. Many agree that Erasmus+ has considerably simplified a number of processes

(simplified grants, digitalisation, VET Charter, linguistic support, etc.). However, procedures

and IT management tools impose a burden on beneficiaries which is not always proportionate

to the grant they receive.

By contrast, the evaluation notes that the simplification process may have gone too far as

regards cooperation projects.

Applicants for small strategic partnerships

are too often

expected to meet the same requirements as applicants for large ones. Furthermore, it is hard to

clearly assess the cost-effectiveness of these actions considering their

relatively average level

of innovation.

There is not enough differentiation made in the way of handling smaller

(mainly collaborative) and larger (innovative) projects. The evaluation recommends

simplifying the application forms, reviewing the award criteria to better reflect key success

factors for effectiveness and strengthening the review at mid-term in particular for bigger

projects.

Another

challenge hampering efficiency is linked to the funding for international higher

education,

which remains managed separately for each global region in a too rigid manner.

European added value

30 years after its beginnings in the field of higher education,

Erasmus+ has become a

flagship programme of the EU.

The programme is highly valued by the general public as

well as by all stakeholders. It is consistently identified by citizens as

one of the three most

positive results of European integration.

Erasmus+ remains unique.

Whereas there are other schemes funding comparable actions at

national level, these are much smaller both in volume and scope. It is unlikely that alternative

measures would be able to compensate for Erasmus+ funding, demonstrating its

undisputable European added value.

For the future, without prejudice to negotiations on the next Multi Financial Framework, the

evaluation concludes that it would be relevant to reconsider the

overall budget.

It also

suggests

modifying the share-out between the programme sectors,

whereby potential

increases could be directed to sectors showing the highest performance, but which have, until

now, received relatively less funding.

School education and vocational education and

training

(VET), where the impact is not yet as widespread as in higher education, were

identified as

having the most promising potential

for an expanded participation in

Erasmus+ activities in the coming years. The unmet demand and the potential for maximising

effectiveness call for a stronger investment for the future development of the programme.

kom (2018) 0050 - Ingen titel

Annex 1: Procedural information



EAD

DG, De

CIDE

LANNING

/CWP

REFERENCES

DG Education, Youth, Sport and Culture

(DG EAC)

EAC/2015/014

RGANISATION AND TIMING

This evaluation has been steered by DG Education, Youth, Sport and Culture since July

2017

484

under the scrutiny of an

inter-service group (ISG)

comprising of representatives of

DG BUDG, DEVCO, EAC, EEAS, EMPL, FPI, HOME, JRC, NEAR, SG, as well as EESC.

The ISG was consulted at each stage of the evaluation process and reviewed each deliverable

produced by the contractor as well as this Staff Working Document.

ISG meetings took place on:



9 September 2015 (Kick-off),

18 July 2016 (Inception report);

21 November 2016 (Interim report 1);

24 March 2017 (Interim report 2);

10 July 2017 (Draft final report);

11 September 2017 (Final report)

The ISG was consulted on the draft Staff Working Document on 29 September and informed

about the Regulatory Scrutiny Board's opinion on 27 November.

XCEPTIONS TO THE BETTER REGULATION GUIDELINES

None

ONSULTATION OF THE

RSB (

IF APPLICABLE

)

The

Regulatory Scrutiny Board

was consulted on 16 October and met on 15 November

2017.

The Board acknowledged in its opinion with comments

485

the significant efforts of data and

evidence collection. It also noted the good methodology. The Board gave a

positive opinion,

with comments to improve the document. All have been addressed.

The following changes have been made to this Staff Working Document, in response to the

Board’s main considerations:

484

Ares(2015)3111961 - 24/07/2015 - Invitation to participate in an inter-service steering group for the

evaluation of Erasmus+ programme

485

Ares(2017)5629740 - 17/11/2017

kom (2018) 0050 - Ingen titel

Regulatory

Scrutiny Board's

recommendation

Changes made to the Staff Working Document

(1) The

conclusions are not

sufficiently

balanced and

precise. They do

not accurately

reflect some of the

programme

strengths and

weaknesses that the

evaluation and its

supporting study

identified.

They also do not

provide enough

guidance for future

decisions about the

programme.

(2) The report does

not clearly define

the scope of the

evaluation and the

baseline/benchmar

ks against which it

assesses the

programme.

Both the Executive summary and the conclusions have been revised to outline

more clearly the weaknesses pointed out by the external evaluator in its

conclusions and across the Staff Working Document.

Any evaluation Staff Working Document has to remain backward-looking

according to Better Regulation guidelines. For that reason,

the

recommendations are addressed in the Commission report to Council and

Parliament,

without anticipating the conclusions of a future Impact assessment.

The Commission can accept all 11 recommendations received from the external

evaluator (section 9 of ICF's final report) to a smaller or larger extent.

- 5 recommendations are cross-cutting (innovation; inclusiveness; strategic

investment in the best place sectors; systemic impact of projects; policy

prioritisation at selection stage)

- 4 recommendations concern a specific sector (SLGF; Jean Monnet actions;

Adult education; Sport)

- 2 recommendations deal with process (simplification; monitoring)

The baseline for this evaluation is the period 2007-2013 unless otherwise

specified.

This is now stated explicitly in the introduction and in the section

about the evaluation method (4.1 and 4.2). In addition, it has been specified

more systematically, especially across the section on effectiveness (5.1), where

the scope of a given finding concern all evaluated programmes (e.g. findings

from

evaluation

surveys of beneficiaries cover 10 years) or Erasmus+ only (e.g.

findings from

monitoring

surveys concern by definition only the current

programme).

The choice of an intervention logic (2.2) covering the two programming periods

has been further justified (4.1)

To better support the claim that Erasmus+ has achieved or exceeded most of its

targets, a new annex (5e) has been produced reporting on all indicators from the

legal basis against targets set in DG EAC's Strategic Plan

(3) The report does

not adequately

address

simplification and

efficiency

improvement.

The Staff Working Document (5.1, 5.4.1, 5.4.4, 5.4.5) has been completed with

more information about measures taken since 2014 to improve monitoring,

efficiency and to simplify the programme implementation. It is also explained

now why the Impact assessment regarding efficiency gains cannot be regarded

as a reference (5.4.4).

The Commission has not waited for the mid-term evaluation to start

working on simplification

(e.g. application forms). The level of complaints

about the administrative burden has decreased as of 2016 after a steep learning

curve on the new programme.

kom (2018) 0050 - Ingen titel

VIDENCE

SOURCES AND QUALITY

Sources of evidence

The following box provides an overview of the data sources from which the evidence was

drawn. A detailed description of the individual methods is provided in Annex 3.



Literature review

(150 sources)

Key informant

interviews

(190 interviews)

National reports

(34 countries)

Open Public Consultation

(4,786 responses of which 1,800 responses were fully

exploitable; 24 position papers submitted)

Social media

analysis (725,678 posts)

Benchmarking

(18 comparable national/transnational programmes)

Programme database

analysis (all beneficiaries) including network analysis

Monitoring surveys

(950,000 respondents - KA1)

Online Linguistic Support

(523,238 participants)

Beneficiary surveys

and

control groups

(over 47,000 respondents)

Survey of socio-economic organisations

(947 responses)

Agencies survey

(130 responses from National Agencies or the Executive Agency

EACEA)

Experts survey

(1,122 responses from project assessors)

Case studies

(38 organisations or policy cases; 233 respondents)

Review of selected

projects’ outputs

(386 outputs)

Expert panel assessment

of projects’ outputs (100 projects)

Jean Monnet actions (13,183 survey respondents; 5 interviews)

Student Loan Guarantee Facility (SLGF) (219 survey respondents; 33 interviews)

eTwinning Plus pilot action (2,967 respondents; 31 interviews)

kom (2018) 0050 - Ingen titel

Expert advice

National Reports

Programme countries have submitted their own evaluation reports on the implementation and

impact of decentralised actions of Erasmus+ in their respective territories, as required by

Erasmus+ legal basis. Their findings are summarised in the

National reports synthesis

(NRS)

drawn by the external evaluator

486

Source

National Reports

(ICF, stand lone

synthesis – "NRS")

Scope

Programme country reports on

the impact of decentralised

actions submitted by June 2017

as per Article 21(4) of

Volume

34 National reports

(MT and UK transmitted

their report to the

Commission with a four

month delay)

Regulation (EU) 1288/2013

and DG EAC’s guidelines

487

Expert Panel

Erasmus+ and predecessor programmes fund a high number of cooperation projects which

result in outputs (e.g. handbook, toolkits, methodologies, etc.) that are aimed to be used

within the organisations that took part in the project but also beyond. To get a better

understanding of their quality and dissemination potential, an

expert panel

reviewed

collectively the actual outputs of a selection of 100 cooperation projects.

Source

Expert panel

assessment of projects’

outputs

(ICF, Annexe 18)

Scope

Assess projects’ outputs

against a set of pre-

determined criteria

Volume

100 projects

(144 outputs)

External consultant

The external evaluator (ICF) contracted for this assignment

488

has carried out since May 2016

all tasks as required under the scrutiny of an inter-service group (ISG) and the daily steer of

DG Education, Youth, Sport and Culture.

Primary data were mainly collected from November 2016 to May 2017.

The only significant change compared to the initial work plan was limited to a two-month

delay of the (sub) contractor in launching the beneficiary surveys

489

486

487

488

489

National reports synthesis (NRS) annexed to ICF final report

Ares(2016)576506 of 2 February 2016

ICF Consulting Services Ltd under specific contract – EAC-2016-0219 implementing Framework contract EAC/22/2013

DG EAC wrote to ICF to mitigate that issue in March 2017. The delay had no consequence on the final

report.

kom (2018) 0050 - Ingen titel

Annex 2: Stakeholder consultation synopsis report

This evaluation has drawn on data from direct consultations with various stakeholder groups

involved in the implementation of the Erasmus+ and predecessor programmes.

A4.1 Overview of main stakeholders consulted

The table below provides an overview of the stakeholders consulted over the lifetime of the

evaluation through: semi-structured interviews (for the purpose of the key informant

interviews (KIIs) and the case studies), online surveys and an open public consultation. Other

consultation activities are described at the end of the section.

Table 1.

Overview of stakeholder consultation

Method

Beneficiary and control group

surveys

Nbr of

respondents/cases/records

Learners: 24,037 beneficiaries

and 2,695 from control group

Staff: 20,155 beneficiaries and 928

from control group

947 valid responses

Type of stakeholders consulted

Beneficiaries (learners and

practitioners) of mobility actions

and cooperation actions as well as

control groups

Organisations other than the

Socioeconomic actors survey

primary target group, i.e.

companies, public authorities, civil

society (other than youth

organisations)

Assessors supporting project

selection and those supporting

evaluation of final reports

Agencies in charge of programme

implementation

One respondent per sector

Key stakeholders in education and Key informant interviews

training, youth and sport

(EU/national level)

Staff, learners, leadership, project

leaders and other stakeholders if

relevant

Students participating in student

fairs for future (mobile) master’s

students and exhibitors at these

fairs

General public including key

stakeholders active in education

and training, youth and sport

Other: focused consultation

(Non)-participating financial

Student Loan Guarantee Facility

intermediaries and their

interviews

representatives; national student

loan schemes; HEIs and their

representatives; National

Authorities and agencies;

representatives of students and the

youth

Case studies

Experts survey

1,122 valid responses

Programme agencies survey

130 valid responses

59 at EU level

131 in 15 countries

233 respondents

38 case studies

Student Loan Guarantee Facility

student fair survey

119 students and 100 exhibitors

Open Public Consultation (OPC)

1,800 valid responses

33 interviews

kom (2018) 0050 - Ingen titel

Students studying about the EU –

beneficiary students and non-

beneficiary academic staff

Students studying about the EU –

beneficiary students and non-

beneficiary academic staff

Jean Monnet students survey

332 beneficiaries and 1,015 non-

beneficiaries

120 beneficiaries and 5,822 non-

beneficiaries

560 beneficiaries and 443 non-

beneficiaries

210 beneficiaries and 4,681 non-

beneficiaries

5 interviews

Jean Monnet section in the

beneficiary student survey

Staff teaching about the EU –

Jean Monnet staff survey

beneficiaries and non-beneficiaries

Staff teaching about the EU –

Jean Monnet section in the

beneficiaries and non-beneficiaries beneficiary staff survey

Interviews with professors who

have never applied for Jean

Monnet

Staff taking part in the pilot

project

Jean Monnet interviews

eTwinning Plus pilot survey

490

405 responses to partner countries

survey

2,562 responses to programme

countries survey

Teachers in programme and

partner countries, EU-level actors

and key stakeholders

Source: ICF

eTwinning Plus interviews

31 interviews

Based on the above, 68,675 stakeholders’ views were collated and analysed among which:



66,383 through the seven above-mentioned parallel online surveys

491

among which

47,815 through the sole beneficiary and control group surveys

492 through interviews

1,800 through the OPC

As the table also reflects, the range of stakeholders consulted was broad, encompassing both

direct and less direct beneficiaries of the programmes at the individual (learners and

practitioners), organisation and system level comprising (not in specific order):



general public including a variety of key stakeholders active in education and

training, youth and sport with different levels of knowledge and experience with

Erasmus+ and/or predecessor programmes: addressed through the OPC

492

all current programme target groups, both for KA1 (mobility) and KA2

(organisational cooperation): addressed though the surveys of beneficiary learners

and staff and related control groups



490

The Staff Working Document does not address this part of the contractor's assignment, not initially set out in

the Terms of reference, as eTwinningPlus pilot action is not funded by the Erasmus+ programme.

The Jean Monnet separate student and staff surveys outlined in the table are counted here as one survey

package.

491

492

Overall, those respondents who replied as ‘organisations’ were primarily active (in order) in the higher education, school

education, vocational education and training; adult education and youth fields. The remainder included respondents from

other sectors and lastly from the sport field. Those who replied as individuals were for the majority (59%) employed in

education and training, youth and sports. The remainder comprised learners (29%) in one of the fields above and 12%

reporting they had a different role in relation to these fields. For further details, see external evaluation final report volume 6.

kom (2018) 0050 - Ingen titel



socioeconomic actors (i.e. companies, public authorities, civic organisations, sectoral

bodies, etc.): addressed through the socioeconomic actors survey

project assessors (i.e. experts contracted by the EACEA and/or the NAs to assess

project applications and reports): addressed through the experts survey

NAs and EACEA staff members: addressed through the programme agencies survey

Selected EU and national level

493

key informants involved with programme(s)

implementation (EU level) or drawing on it and/or its predecessors: key stakeholders/

key stakeholder organisations representatives, EC officials (EC and agencies

(EACEA and ETF), ministries’ representatives (policy-making)

494

. Those were

addressed through the Key informant interviews (KIIs)

Selected national level

495

informants benefitting from the programme(s) in the three

fields above comprising: beneficiary organisations’ leadership, practitioners and

learners, targeted funded project leaders, other key stakeholders (where applicable).

Those were addressed through the case studies



Next to the consultation methods above that were designed and implemented to collate

stakeholders’ opinions on the current programme’s (and/or predecessors where appropriate)

performance against all or several of the main evaluation criteria, the evaluation team also

undertook ad hoc stakeholders consultation. This served to inform the evaluation of specific

part or activities of the programme namely: Jean Monnet programme

496

and the Student Loan

Guarantee Facility (SLGF)

497

. In each case, the consultation methods comprised tailored

surveys and interviews. For an overview of the types of stakeholders reached in both cases,

see table above.

Besides, to gain complementary insights and to ultimately assess the difference in results

between Erasmus+ beneficiaries and their counterparts who did not take part, control groups

of non-beneficiaries (targeted at practitioners

498

and learners

499

) were set-up.

Overall, the mix of consultation activities described above enabled the evaluation to

effectively address a relevant breadth of stakeholders. This in turn enabled the evaluation

team to gain insights from a range of relevant key players on different and meanwhile

493

Focusing on 15 Programme countries as per ICF proposal and agreed during the inception phase of the

evaluation study: BG, CY, CZ, DK, ES, FR, DE, HU, IE, IT, LV, PL, SE, TR, UK

National Authorities in charge of the Erasmus+ programme were not consulted because they contributed to

the evaluation directly through their national reports, as laid down by the programme legal basis.

Focusing on 15 Programme countries as per ICF proposal and agreed during the inception phase of the

evaluation study: BG, CY, CZ, DK, ES, FR, DE, HU, IE, IT, LV, PL, SE, TR, UK

Considering the specific features of the Jean Monnet programme as one of the two standalone Erasmus+

chapters (with Sport), the evaluation Steering Committee agreed with ICF, during the inception phase, that

the latter would be make the object of a standalone evaluation (for details, see external evaluation final

report, volume 3) .

As one the novelties introduced through Erasmus+, the SLGF made the object of a focused evaluation (for

details, see external evaluation final report, volume 2).

i.e. involved in all three fields and subfields (i.e. school education, VET, HE, adult education).

i.e. enrolled in school education, VET, HE and Erasmus+ Youth exchanges and EVS.

494

495

496

497

498

499

kom (2018) 0050 - Ingen titel

complementary dimensions of the programme(s)’s performance and/or on suggestions

towards the future programming period.

With regard to the representativeness of the data, sample sizes (more particularly applicable

to the main surveys) and targets had been agreed upon by the ISG during the inception stage

of the evaluation study. In practice, the number of responses received and processed have

been on or above targets in most cases. In the case of the beneficiary and control group

surveys the satisfactory response rate offered to the evaluation a solid and reliable basis for

data comparison amongst beneficiaries and sectors. A noticeable exception though was in the

sport and adult education fields where the control groups were too small. For details on the

actual response rates against minimum sample sizes defined for the above-mentioned main

surveys’ target and control groups see evaluation final report, volume 1 (section 3).

Concerning the other surveys, the minimum sample sizes

500

were attained. In the case of the

programme agencies, no ad hoc target had been fixed, but all programme countries were

covered.

In the case of the OPC, the number of total responses received and this of exploitable ones

(i.e. complete responses) was considered statistically reliable.

With regard to the key informant interviews (KIIs) undertaken to inform the main evaluation

report as well as accompanying standalone reports, the number of interviews foreseen at the

start of the evaluation process was attained in all cases. The main types of pre-identified

stakeholders were also consulted.

The same applied to the case studies (i.e. at organisational level (spread across all education

and training, youth and sport fields), at system level and focusing on Jean Monnet). With the

exception of one case study initially foreseen that could be not carried out, all the others were

effectively conducted either on-site (organisation level case studies) or over the phone

(system level and Jean Monnet ones).

In addition to the above, other consultation related activities are worth being mentioned.

These have not been listed in table 1:



Expert panel assessment of project outputs:

experts in education and training,

youth and sport fields were sub-contracted by ICF to assess a number of selected

project outputs and take part in a virtual expert panel assessment

Delphi survey of experts:

experts in the fields above were sub-contracted by ICF to

take part in the survey that whose main purpose was to inform the analysis of the EU

added value and cost-effectiveness of the current programme

Informal workshop

organised by two EU-level organisations (EUCIS-LLLP and the

European Youth Forum -EYF) in May 2017, prior to the closure of the OPC. It was

agreed with DG EAC that the event could be an opportunity for ICF to grasp

additional insights from key stakeholders ideally before OPC closure.



A4.2 Consistency of results across consultation activities

The table below presents key results per consultation activity, organised by evaluation

criteria, as well as the level of (1) consistency of results across consultation activities and (2)

complementarity of results across consultation activities.

500

i.e. at least 200 respondents to the socio-economic actors and to the experts surveys. At least 200 students

enrolled in JM courses or modules; at least 50 practitioners from Jean Monnet actions

kom (2018) 0050 - Ingen titel

Overall, there was a large degree of convergence in results from the different consultation

activities.

kom (2018) 0050 - Ingen titel

Mid-term evaluation of Erasmus+ and ex-post evaluation of predecessor programmes

Table 2.

Main results of the consultation processes and level of their consistency and complementarity

Complement-arity

of results across

consultation

activities

High

Headline results

Consistency of

results across

consultation

activities

High

Evaluation

criterion

Surveys

KIIs

Case studies

OPC

Relevance

Beneficiaries surveys:

Learners with different

backgrounds can benefit from

the programme.

The attractiveness of

disadvantaged groups is an area

for improvement though

Most interviewees see

strong alignment with key

EU priorities and high to

medium level of alignment

of Erasmus+ with key

national policies depending

on sectors

Concrete examples of

alignment of

Erasmus+ with key

national policies were

provided though not

by all respondents

The relevance of the E+

is positively rated by

most respondents.

A vast majority of

interviewed estimate that

Experts survey:

E+ best addresses learners’

Most respondents estimate that a needs. Those at national

high share of projects funded

level confirm that E+

align to high or average extent

actions are relevant in

with EU-and national level

terms of their positive

priorities

effects at practitioner level

too.

Improvement in quality of

applications and project reports Learners with different

is found between E+ and

backgrounds can benefit

predecessor programmes

from the programme. The

attractiveness of

Programme agencies survey:

disadvantaged groups is an

Over half of disagree that E+ is area for improvement

though

better aligned with national

priorities than predecessor

Mixed reports on the

programmes

programme’s ability to

meet the needs of

Mixed reports on the

programme’s ability to meet the education, training, youth

Some programmes’

objectives are perceived

to be notably more

Several similar

relevant than others in

findings as those

addressing the current

obtained through KIIs challenges and needs

within education,

training, youth and sport

October, 2017

kom (2018) 0050 - Ingen titel

Mid-term evaluation of Erasmus+ and ex-post evaluation of predecessor programmes

Surveys

KIIs

Case studies

OPC

needs of education, training,

youth and sport at organisation

and system level.

Most respondents believe that

the single programme is more

visible

A number of obstacles for

disadvantaged groups to

participate in E+ are reported.

Coherence

Programme agencies survey:

Most agencies perceive

synergies and lack of overlaps

across programme actions as

well as externally (other EU or

national programmes)

and sport at organisation

and system level.

A number of obstacles for

disadvantaged groups to

participate in E+ are

reported.

Most interviewees

(EU/national) consider that

Erasmus+ objectives and

actions are mutually

supportive in the different

fields.

Not all respondents were

The coherence of the programme able to give examples of

between centralised and

synergies

decentralised actions is valued

overlaps/duplications

by most respondents

No clear consensus on

Respondents’ views on the

increased E+

benefits of the integrated

decentralisation: not

programme are positive

positively perceived by all

The integrated programme

is valued by most

interviewees

Not all respondents

were able to give

examples of

synergies

overlaps/duplications

between different

types of actions and

in different sectors

Most respondents

believe that E+ does not

overlap with other

funding opportunities

Need to ensure synergies

with EU level

programmes is pointed

by some respondents.

Examples of

complementary national

schemes to E+ are

supplied in some OPC

position papers

High

October, 2017

Complement-arity

of results across

consultation

activities

High

Headline results

Consistency of

results across

consultation

activities

Evaluation

criterion

kom (2018) 0050 - Ingen titel

Mid-term evaluation of Erasmus+ and ex-post evaluation of predecessor programmes

Surveys

KIIs

Case studies

OPC

Effectiveness

Most interviewees have

positive views on the

Participation in E+ is associated effects of E+ in particular

with stronger attachment to the at individual and to varying

EU, networking, and greater use extent

organisation

of digital resources

levels.

The recognition of practitioners’ More mixed views on

achievement during mobility is examples of system-level

uneven

projects

and

Beneficiaries surveys

Disadvantaged learners show influence

positive results on a few

indicators

Socio economic actors survey

Most respondents agree that

their objectives have been met

Only a small number said their

projects led to implementation of

approaches

tackle

disadvantaged group

Programme agencies survey

The quality scores at project

selection are not a robust

predictor

quality

completion

Most respondents value the

D&E strategy. More mixed

Learners expressed a

broad

range

positive changes they

associate with E+.

All

practitioners

interviewed

cited

positive results on

themselves as well as

their learners and

organisations

In all case studies

concrete changes at

organisational level

were cited

A lot of the evidence

is about participation

disadvantaged

groups rather than

results for them.

71% of respondents

thought that as a whole,

the programme is

achieving its objectives to

a ‘very large’ or ‘large’

extent. There were some

notable differences

between programme

objectives, levels of

intervention and sectors

though

High

October, 2017

Complement-arity

of results across

consultation

activities

High

Headline results

Consistency of

results across

consultation

activities

Evaluation

criterion

kom (2018) 0050 - Ingen titel

Mid-term evaluation of Erasmus+ and ex-post evaluation of predecessor programmes

Surveys

KIIs

Case studies

OPC

views on the project results

platform

Efficiency

Several key informants

commented spontaneously

The size of budgets is considered on insufficient levels of

too small, in particular in the

funding for certain types of

case of projects applying for

actions or sectors.

KA2

Mixed views on efficiency

Programme agencies survey

gains are observed by the

programme beneficiaries

Most respondents value the

balance between centralised and The set of monitoring tools

decentralised actions

is commonly viewed as

Beneficiaries surveys

Mixed views on E+ management rather complex

tools

Most respondents see efficiency

gains compared to predecessor

programmes.

Mixed views on efficiency gains

are observed by the programme

beneficiaries

The set of monitoring tools is

considered rather complex

Socio economic actors survey

Vast majority of respondents are

very positive about the support

The majority of

Most respondents see

respondents welcome improvements in the

the use of unit costs

user-friendliness of E+ IT

tools and reporting

Several findings

procedures

concurrent with those

collated under KIIs.

A slightly smaller

majority of respondents

agree that the

management of the E+

has been simplified for

them

Mixed views on the

budget distribution

between the three Key

Actions. In particular

KA2 is considered as

underfunded

High

October, 2017

Complement-arity

of results across

consultation

activities

Headline results

Consistency of

results across

consultation

activities

Evaluation

criterion

kom (2018) 0050 - Ingen titel

Mid-term evaluation of Erasmus+ and ex-post evaluation of predecessor programmes

Surveys

KIIs

Case studies

OPC

received by EACEA or NAs

EU added value

Most respondents claimed

Programme agencies survey

Though less specifically, most

interviewees see a clear

that their organisation

The programme funds more

European added value in

could not have

mobility and cooperation actions E+

developed similar

than comparator programmes.

projects/actions

without Erasmus+

It covers system-level

funding

cooperation actions that are very

rarely covered by other

programmes

Most of the respondents

believe that E+ brings

certain benefits to the

actions implemented.

Respondents also

identified other areas

where they believed E+

has added value to the

actions implemented on

the EU, national and

international level

High

October, 2017

Complement-arity

of results across

consultation

activities

Headline results

Consistency of

results across

consultation

activities

Evaluation

criterion

kom (2018) 0050 - Ingen titel

Annex 3: Methods and analytical models

As the differences in the architecture, objectives and activities of the current and predecessor

programmes

501

made it difficult to make a straightforward comparison between their

respective actions, the current evaluation did not look at programmes along their own types of

actions but through a customised matrix. To ensure comparable approaches, the evaluation

framework developed by the external evaluator

502

classifies actions across the three

levels of

intervention

of the current programme (see fig. below) and uses that as the basis of

comparison, namely:

■

System

level

■

Organisation

level

■

Individual

level divided into actions for

learners

and

practitioners

across all

different

areas

(education and training, youth and sport) and

sectors

(school (SE),

vocational (VET), higher (HE) and adult education (AE)). For the purpose of the

evaluation, the overarching terms of

Learners

and

Practitioners

are used

503

501

ICF, 3 and Annexe 1; see section 2 above for background on Erasmus+; under the Erasmus+ the activities

were simplified, (as a simple example among many others, ‘ad personam’ Jean Monnet chair is now

integrated in the Jean Monnet Chair).

ICF, Annexe 14 (not published), where a matrix maps and compares the types of actions in the predecessor

programmes and the corresponding actions in the Erasmus+ programme

Learners

refer to all individuals involved in formal, non-formal and informal education as pupils, students,

apprentices, volunteers, young people, etc.

Practitioners

refer to those involved in the same respect as

teachers (including prospective teachers), trainers, youth workers, educators, coaches, etc.

502

503

kom (2018) 0050 - Ingen titel

Summary of analytical techniques used

Desk research and interviews

An extensive

literature review

was carried out across varied sources. In addition to the

contextualisation of other data, the main purpose was to review research notably on

programme outcomes, including success factors and obstacles to learning mobility and

transnational cooperation in the fields of education, training, youth and sport, mainly

encountered through predecessor programmes. This helped to inform whether and where the

programmes have had impacts at the individual, organisational and system levels and was

central to the analytical framework underpinning this evaluation. A full account of the

literature reviewed is annexed to the contractor's report. External statistics were also used to

put the programme in perspective

504

Key informant interviews

(KIIs) have been conducted at both national and European level

targeting EC/agencies officials, EU key stakeholder organisation representatives, national

policy makers and key stakeholders in 15 selected countries, as well as managers of

comparable programmes for benchmarking purpose. All were selected using category based

purposeful sampling

505

Source

Scope

Volume

Literature review

(ICF, Annexe 11 and 15 - not

Desk research: review of other

131 sources

reviewed

evaluations, NA yearly reports (since More than 400 yearly reports

504

For instance, although the market share of Erasmus+ was assessed in the HE field where more data is

available, this calculation was rough as Eurostat deals with full

degree

mobility and not

credit

mobility

mainly supported by the EU.

This means that interviewees were spread across each (sub-)field of the programme, at both EU and national

level, including policymakers and key stakeholders. The country selection (BG, CY, CZ, DE, DK, ES, FR,

HU, IE, IT, LV, PL, SE, TR and UK) took into account a variety of volume of participation in the

programme; the country size (covering more than 50% of the population); geographical distribution

(including one non-EU country) and the existence of national programmes for benchmarking in different

sectors.

505

kom (2018) 0050 - Ingen titel

Source

Scope

Volume

published)

2008), studies, academic papers at

EU, national, international, etc

506

from National Agencies

190 interviews:

59 at EU level

131 at national level

Interviews

of Key informants Semi-guided interviews of selected

(ICF, Annexe 16 - not

stakeholders in all sectors, at EU

published)

level and in 15 countries

National reports

Programme countries have submitted their own evaluation reports on the implementation and

impact of decentralised actions of Erasmus+ in their respective territories, as required by

Erasmus+ legal basis. Their findings are summarised in the

National reports synthesis

(NRS)

drawn by the external evaluator

507

. These reports involved the review of

documentation and statistics concerning the programme and related policies and ad hoc

consultations with relevant stakeholders. The most often used data collection methods were

interviews and surveys (both used by 25 out of 34 countries). In addition 11 countries used

focus groups.

Source

Scope

Volume

34 National reports

(MT and UK transmitted

their report to the

Commission with a four

month delay)

National Reports

(ICF, stand lone synthesis –

"NRS")

Programme country reports on the

impact of decentralised actions

submitted by June 2017 as per Article

21(4) of Regulation (EU) 1288/2013

and DG EAC’s guidelines

508

Open Public Consultation

The objective of the

Open Public Consultation

(OPC) was to gather the opinions, including

forward-looking perspectives, of the general public and various stakeholders, on all evaluation

criteria

509

. Different (closed and open-ended) questions were asked depending on the level

and scope of knowledge of the programmes declared by each respondent. Most respondents

were either organisations or practitioners. In any case, consultation findings can never be

representative due to the selection bias inherent to any open recruitment, but they can be

triangulated with other sources to inform the evaluation. Alongside their responses, the OPC

respondents submitted 24 position papers, which are included as well in the synopsis drawn

by the external evaluator

510

506

The Terms of reference set for this evaluation contain a list of literature references, which have been enriched

thereafter.

National reports synthesis (NRS) annexed to ICF final report

Ares(2016)576506 of 2 February 2016

https://ec.europa.eu/programmes/erasmus-plus/resources/consultations/erasmus-plus-mid-term-evaluation-

2017_en

Synopsis of the open Public Consultation (OPC) annexed to ICF final report

507

508

509

510

kom (2018) 0050 - Ingen titel

Source

Scope

Volume

4,786 complete (n =

1,219) and partial

responses of which

1,800

responses

were fully

exploitable (53%

answered in their personal

capacities; 47% on behalf

of an organisation)

511

24 position papers

submitted

Open Public Consultation

To gather the opinions and

(ICF, stand lone "OPC"

perspectives of the general public

synopsis)

and other stakeholders, the OPC

was available in all official EU

languages on the dedicated

European Commission website

from 28 February to 31 May 2017

Social media analysis

Social media analysis was used to collect evidence about the awareness on the programme

and its public image, globally and by sector. All relevant posts identified

512

were analysed for

trends in the

volume of conversation

about Erasmus+,

audience profile,

and content of

discussions both in terms of

topic and sentiment.

The volume of information shared was

compared with the volume for other EU programmes (e.g. Marie Curie, MEDIA, Citizens

programme).

Source

Social media

analysis

(ICF, Annex

19 - not

published

)

Scope

Twitter, Facebook and Instagram

were analysed between 5

November 2015 and 4 November

2016. Limited for qualitative

analysis to posts in English, French,

Spanish and German.

Volume

725,678 posts

on the topic of Erasmus+

(almost 2,000 posts every

day)

Benchmarking

Erasmus+ was benchmarked across the different evaluation criteria against 18 selected

comparator national and transnational mobility and cooperation programmes

513

. Three

different types of benchmarking were undertaken:

Strategic benchmarking

helped to

compare the relevance, coherence and added value of the programmes.

Process

benchmarking

compared efficiency and effectiveness of their management.

Financial

benchmarking

assessed the respective financial efficiency and effectiveness of the

511

A large number of partial responses only contained very basic background information about respondents and

were therefore removed from the analysis as unsuitable for analysis.

The Crimson Hexagon platform was used to identify the topics of conversations on social media according to

the frequency of use in posts of certain selected keywords (“Erasmus+”, “ Erasmus+”, “Erasmus plus” and

“Erasmusplus”). Results for Erasmus+ were fine-tuned using secondary keywords, such as youth, school or

schools, higher education or higher ed, VET or vocational, adult learning, sport, etc. Source:

https://www.crimsonhexagon.com/

18 national or international schemes were selected out of 58 for detailed benchmarking purpose and spread as

follows: HE (7), VET (5), SE (4), youth (2), sport (2), AE (1). Action-wise, the vast majority support

mobility. ICF, fig 5.5 about geographical balance

512

513

100

kom (2018) 0050 - Ingen titel

programmes compared. The mapping was not expected to be comprehensive but to inform

about the scale and profiles of comparable initiatives

514

. The large difference in scope and size

made comparison between programmes fairly complex. The contractor only had access to

information available in the public domain, which means that in a few cases, it was not

possible to collect data for some indicators.

Source

Benchmarking

with

comparator programmes

(ICF, Annex

)

Scope

Strategic/ process/ financial

benchmarking against national or

international schemes sharing some

similarities with Erasmus+

Volume

18 comparable

programmes

out of 58 shortlisted

schemes

15 interviews

Programme data

The evaluation used the wealth of the programme data available as regards

inputs

(funding)

and

outputs

(numbers of projects, numbers of beneficiaries, etc.). Moreover data on certain

results

(e.g. satisfaction-rate) is available since 2014 directly from the management or

monitoring tools set up for Erasmus+.

Overall

the quality of data was good for the predecessor programmes and even very

good for Erasmus+.

This shows improvement made in terms of monitoring since 2014 (see

table of monitoring tools below)

515

, albeit with certain limits

516

. The quality and completeness

of the data

517

was best for the Erasmus+ programme, followed by the LLP and Youth in

Action

518

. Fewer data was available for Erasmus Mundus and Tempus and even less for the

remaining higher education international programmes

519

and sport pilot actions

520

. The main

514

The initial mapping identified across Europe 58 schemes which were comparable to Erasmus+. Around half

covered SE, over 35% HE, below 20% VET or the Youth sector, 10% AE. Their target groups were students

in nearly 80% of the 58 schemes, teachers at around 22%, youth at around 17% and adults only around 10%.

Erasmus+ Dashboard, annual reports, Business Objects (BO) reports, Epluslink and Mobility Tool databases

Data is not always available for predecessor programmes. The data on Jean Monnet participants is in general

not very reliable. It reflects the intention as it is based on applications rather than the actual output, contains

double counting where the same persons take part in more than one activity and captures indistinctly those

involved in dissemination activities only.

ICF, Tab. 3.3 and 6.2 for a summary of the datasets coverage

Although data on YiA participants is only available for the European Volunteer Service, but not for Youth

exchanges.

For the remaining higher education international programmes, Alfa and Edulink in particular, only

consolidated findings were available from their distinct evaluation: European Commission,

Evaluation of the

EU Development Co-operation Support to Higher Education in Partner Countries (2007-2014),

Revised

draft final report (main report), March 2017.

For sport pilot actions only the funding and number of selected projects were available.

515

516

517

518

519

520

101

kom (2018) 0050 - Ingen titel

issues affected the analysis of participation patterns due to an incomplete coverage of data on

i) organisations

521

; ii) participants with few opportunities and disadvantaged backgrounds

522

Overview of Monitoring arrangements

Level and purpose IT systems and tools

Project level

management

– E+Link –main system for management of decentralised grants (NAs)

Pegasus –main system for managing centralised grants (EACEA)

Mobility Tool – system for managing individual mobility actions (NAs)

OEET – online expert evaluation tool

For project assessors

Business Object reporting



‘live’ Dashboard visualising key performance

indicators according to criteria defined by each user

For Commission services and agencies

Erasmus+ Project results platform – summary overview of all projects (KA2

and KA3) with success stories flagged - For all audiences

EU Survey – online survey administered for all KA1 beneficiaries through the

mobility tool

OLS – pre-post testing on language competence

PDM/URF – DGIT system for creation of unique organisational ID for all

beneficiaries of EU programmes

HERMES – DGIT system for management of application forms

Project level –

grant evaluation

Programme level

– reporting and

management

Programme level

– dissemination

Other tools

The

monitoring surveys of beneficiaries

carried out by DG EAC concern all learners or

practioners who take part in mobility under KA1 over 2014-2016

523

. The

reliability of the

data is strong.

Given the sample sizes (first two years of the programme) the data is

considered to be strongly reliable even though it was too early to include 2016 data. This

being said most questions are based on the self-perception of beneficiaries

524

in contrast to the

more objective OLS

525

assessment of progress in language proficiency for instance; even for

521

A unique ID for each participating organisation has been introduced only since 2014. Its absence beforehand

has affected the comparison with predecessor programmes and limited the network analysis to Erasmus+

(without KA1). Furthermore, although the number of types of organisations used was very large, some 40%

of organisations were classified as "other" in the programme dataset affecting the analysis of participation

per type of organisation.

See sections on effectiveness and relevance below

All KA1 individual beneficiaries are requested to fulfil an online survey on completion of their mobility. See

ICF, fig. 3.2 for breakdown by sector.

The most factual question (programme result indicator) is the rate of formal recognition of participation in

Erasmus+.

The

Online Linguistic Support

(OLS) was introduced with the Erasmus+ programme to help mobility

participants improve their knowledge of the relevant foreign language, so that they can make the most out of

their experience abroad. Before and at the end of the Erasmus+ mobility, Erasmus+ participants take the

language assessment to measure their level and progress in the language. OLS provides as well online

language courses.

522

523

524

525

102

kom (2018) 0050 - Ingen titel

very similar questions the responses offered vary across sectors affecting the comparability

across target groups.

In addition to descriptive and multivariate statistical analysis of programme data, a

social

network analysis

showed more specifically how organisations cooperate and participate in

Erasmus+ and assessed in particular cross-sector and trans-national cooperation. It is to be

noted that the analysis of repeated participation in the programme could not be pursued, as it

would have required substantial data cleaning across previous programmes' data sets (due to

multiple names used for the same organisations prior to 2014).

The analysis of programme data has been

enriched with other secondary data:

contextual

indicators from the Education and training monitor, the Youth indicators, relevant

Eurobarometer studies or the Sport indicators available on Eurostat. Furthermore, external

data sources on education and training institutions with high share of disadvantaged students

have been compared across a sample of countries with the programme data to identify the

schools that have (or have not) engaged with Erasmus+.

Most result indicators are based on self-perception and the link between the survey questions

and the indicator measured can sometimes be weaker. That is why monitoring data were

complemented through primary data collection such as surveys.

Source

Programme databases

(ICF, Annexe 2)

Monitoring surveys

Erasmus+ beneficiaries

(Commission data)

(ICF, Annexe 2)

Scope

Extracts from Commission IT

systems for programme

management over 2008-2016

526

Volume

Records concern

all

beneficiaries

and in some

case also applicants

All Erasmus+ beneficiaries of KA1 Over

955,000

have been surveyed on completion

respondents

since 2014

of which:

730,254 learners

227, 319 staff

total sample size:

523,238

participants

(95% in HE)

Based on programme data

from Erasmus + (KA2 and

KA3), LLP, Youth in

Action, Erasmus Mundus

and Tempus

Online Linguistic Support

Learners in KA1 (mostly higher

assessment (ICF, 3.4.2)

education students) since 2014

Network

analysis

(ICF, Annex 18 – not

published)

Based on programme data for KA2

and KA3 (or equivalent actions)

from Erasmus+, LLP, Youth in

Action, Erasmus Mundus and

Tempus

527

526

Some indicators were unavailable for 2007. 2016 data for centralised actions were not yet fully available at

the time of the data analysis. The sample for 2016 therefore represents a robust indication but not the final

state of the respective indicators.

For International Higher Education, data was available only for centralised projects. For Tempus the data

analysed only covered leading organisations as no data was available about partner organisations. Other

Higher Education international programmes were not covered.

527

103

kom (2018) 0050 - Ingen titel

Surveys of beneficiaries and control groups

To collect more comprehensive, objective and comparable quantitative data on programme

results, the external evaluator has carried out ten specific surveys of beneficiary

learners and

staff

and related control groups

covering all programme target groups,

both for KA1

(mobility) and KA2 (organisational cooperation)

528

. The Erasmus+ monitoring survey shows

a subjective valuation by beneficiaries themselves of their experience, whereas the external

evaluation surveys show

more objectively,

through factual questions whether the

beneficiaries demonstrate different attitudes, competences or beliefs than a control group that

was asked the same questions. As often as possible the questions used in the survey had been

previously tested in large scale surveys such as PISA, TALIS or Eurobarometer and in any

case were as often as possible factual, so as to enable a comparison with the control group of

non-beneficiaries. This means that to strength the validity of surveys statements about self-

perceived contribution of the programme were as often as possible avoided. The latter surveys

were disseminated to all contactable beneficiaries of Erasmus+ as well as predecessor

programmes

529

. Overall findings could be generalised to the whole programme with

sufficient

confidence

because the

sample sizes for all surveys were large enough

in all cases to make

judgements and the distribution of various background variables within the survey samples

was assessed as acceptable

530

In addition to the standard tests of significance and regressions, a

quasi-experimental

approach

was used to assess the contribution of the programme to the results measured

531

. As

a proxy of

counterfactual assessment

532

, control groups of non-beneficiaries were set up

533

528

Results at the level of organisations were measured through responses of staff

ICF/GfK, 3.3.3 on survey dissemination and recruitment methods as well as sample sizes (tab. 3.6)

However given the limited background variables about the beneficiaries it is not possible to assess how

comparable are the samples compared to the programme population. For several surveys (pupils, young

people, sport staff) there is no contact database of direct beneficiaries hence there is no background data on

the population to compare with. Respondents have been recruited through organisations. For all the staff

surveys though there is database for KA1 beneficiaries (Erasmus+ only) KA2 beneficiaries, for whom there

is no database that would provide data on the overall population, were also surveyed. Respondents have

been recruited among beneficiaries who took part not only in the current programme for which population

data is available for KA1 but also predecessor programmes. Except learners in higher education and VET

the information about background characteristics of the overall population for the other groups is weak or

inexistent. Where available the background information is besides not standardised across datasets. For

example, the database of HEI students who took part in Erasmus Mundus identifies their country of

residence, whereas the HEI student database of Erasmus students identifies their nationality or the country

they are being sent from. For all databases there are gaps in terms of the completeness of the information.

Even where population variables of interest are available, there are missing cases, meaning that it is not

possible to say with confidence what the profile of the population is.

The analysis includes tests of statistical significance, regressions, pre-post measurement and propensity score

matching; ICF, Annexe 3 for detailed statistical analysis of survey results; ICF/GfK, 3.4 concerning method

of statistical analysis and counterfactual estimation applied for the post surveys and the pre-post surveys

Counterfactual attempts to measure what would have been the outcome in the absence of the programme.

In addition to open recruitment through social media, control groups were recruited in asking either

participating organisations or unsuccessful applicants to disseminate the survey respectively to non-

participants or to their staff.

529

530

531

532

533

104

kom (2018) 0050 - Ingen titel

with similar profiles to compare their responses with those made by respondents who

participated in the Erasmus+ programme. This has been achieved in matching respondents in

the treatment group (i.e. Erasmus+ beneficiaries) with ‘similar’ individuals in the control

group, to come up with a ‘matched sample’ where subjects are alike in some background

characteristics, called covariates, such as gender, age, etc.

534

. Overall the comparability was

considered satisfactory even if there are some minor though statistically significant

differences on certain surveys and certain variables. For most variables there were no

significant differences between the control group and the treatment group and where they

were these concerned variables that were not likely to strongly influence the findings. In other

words, this might not be considered as genuine counterfactual assessment, as the design

cannot exclude that the differences in result variables measured between the beneficiaries and

the control group would be

related to the selection into the programme rather than the

programme.

Nevertheless it does control for various characteristics of both beneficiaries and

non-beneficiaries when judging the differences in results.

Most control groups reached a

significant size allowing comparison with beneficiaries,

except in sport and Adult

education sectors where the control groups were too small

535

In addition to the above ‘post’ surveys (i.e. beneficiaries were surveyed after they took part in

the programme), the external evaluator carried out pre-surveys in two sectors where short

term mobility exchanges apply (i.e. school pupils and Youth exchanges) for the purpose of

pre/post comparison.

This allowed - for the school sector only

536

- to draw conclusions on

selection into the programme and to observe changes during the time of the mobility

experience. The sample sizes suffered from a high attrition from the pre to the post survey,

especially for the youth sector where comparison of profiles prior to Erasmus+ mobility could

not been done. Pre-post results for the youth sector have therefore not been interpreted

considering the high likelihood of sample bias.

The surveys of learners and staff were complemented with a

survey of socio-economic

organisations

taking part in the programme: companies, public authorities, civic

organisations, sectoral bodies, etc.

Other surveys and interviews targeting beneficiaries of specific activities were carried out in

the case of

Jean Monnet

actions and the

Student Loan

Guarantee Facility (SLGF)

537

534

The match between co-variates was sought on: Age; Gender; Country; Academic level (i.e. whether they had

repeated a grade); Socio-economic background; Rural/urban area; Objective migrant status (i.e. at least one

parent born in a different country); Disadvantaged status (those who said ‘yes, and I feel disadvantaged by

this’ to any of the statements in the question about disabilities, problems and obstacles); Highest parents’

education level (i.e. by either the mother or father); Volunteering experience (Youth survey only).

As a consequence, the control group of VET staff was also used for adult education and the control group of

youth staff was also used for sport, as these were found to be sufficiently comparable against a range of

background characteristics.

ICF/GfK, tab. 3.7

ICF, 3.3.4 and 3.3.5

535

536

537

105

kom (2018) 0050 - Ingen titel

Source

Beneficiary surveys and

control groups

(ICF, Annexe 3 and 13 –

not published)

Scope

Beneficiaries of mobility actions

(learners and practitioners) and

cooperation actions (practitioners)

as well as control groups (non-

beneficiaries)

Volume

Over

47,000

respondents

538

Learners: 24,037

beneficiaries and 2,695

from control group

Staff: 20,155 beneficiaries

and 928 from control

group

Socio-economic actors

survey

(ICF, Annexe 6)

Organisations other than the

947

valid responses

primary target group, i.e.

companies, public authorities, civil

society (other than youth

organisations)

Topic-specific surveys

Students participating in student

fairs for future (mobile) Master

students and exhibitors of these

fairs

219 respondents

119 students and 100

exhibitors

Student Loan Guarantee

Facility

student fair survey

(ICF, Annexe 8 and

standalone SLGF section)

Student Loan Guarantee

Facility interviews

(ICF, 3.4.5)

(Non)- Participating Financial

33 interviews

Intermediaries and their

representatives; National student

loan schemes; HEIs and their

representatives; National

authorities and Agencies;

Representatives of students and the

youth

Students studying and Staff

2,350 respondents

teaching about the EU –

Supplemented with 330

beneficiaries and non-beneficiaries beneficiaries and 10,503

non-beneficiaries

from the general

beneficiary survey

5 interviews

4 case studies

Jean Monnet

surveys

(ICF, Annexe 7 and

standalone Jean Monnet

report)

Jean Monnet interviews and Interviews with professors who

case studies

have never applied for Jean

(ICF, 3.4.4)

Monnet (AU, CZ, UK, EL, US)

Surveys of staff involved in the programme implementation

538

Response rates are not available for these surveys, due to the fact that a large proportion of the responses were

collected through institutions or by a snowball method. If there is no way of knowing how many people

were actually contacted, ICF's technical report (ICF, annex 3a) maps how many invitations were sent to

which contact list. For instance, for the sport sector 197 out of a total of 329 participating organisations

replied to the survey.

106

kom (2018) 0050 - Ingen titel

National agencies and EACEA, as key managing bodies in the implementation of the

programme, were subject to a dedicated

programme agency survey,

especially useful to

inform efficiency of the programme management.

To complement data on projects, in particular about the quality of applications and final

reports but also the efficiency of the selection process, a

survey of project assessors

was also

carried out. They are experts assessing project applications and reports.

Lastly, a so-called

Delphi survey

was disseminated among these experts to collect project

assessors views about the added value of the programme and thus to provide inputs into the

analysis of cost effectiveness. They were asked for each sector to elaborate about hypothetical

impact of dismissing Erasmus+ and about alternatives

539

Source

Agencies

survey

(ICF, Annexe 4)

Scope

Agency staff in charge of

programme implementation (57

National Agencies and Executive

Agency EACEA)

One respondent per sector

designated by each agency

Volume

130

valid responses

Experts

(project assessors) Assessors supporting project

survey and Delphi survey

selection and those supporting

(ICF, Annexe 5)

evaluation of final reports

(centralised and decentralised

projects)

1122

valid responses

213

responses (Delphi

survey)

540

539

What would have happened in absence of Erasmus +? What are the most likely reactions when it comes to

alternative/ compensatory measures? What would have been the most likely effect of the absence of

Erasmus +? What alternative actions could lead to comparable results as Erasmus +?

That is to say between 25 and 40 expert responses per sector.

540

107

kom (2018) 0050 - Ingen titel

Case based and qualitative data collection

On a qualitative note, the

case studies

served to complement insights gained from key

informant interviews and surveys, in particular about how the programme generates

organisational and system level changes including spill over effects from individuals who are

direct beneficiaries to other individuals and to organisations. Case studies were selected using

a combination of criterion-based selection and random selection.

The

most significant change approach

was applied to case study interviews. It is based on

open ended questions which ask beneficiaries to give account

in their own words

of the

difference the programme makes to them. Indeed, the surveys measure always predefined

results from the perspective of the programme design. However the participants themselves

may judge how the programme has benefited to them differently.

Erasmus+ and predecessor programmes fund a high number of cooperation projects which

result in outputs (e.g. handbook, toolkits, methodologies, etc.) that are aimed to be used

within the organisations that took part in the project but also beyond. To get a better

understanding of their quality and dissemination potential, a

review of selected project

outputs,

mainly collected from national agencies, was carried out. Given that only relatively

few Erasmus+ funded projects were completed at the time of this evaluation, the sample is

somewhat biased towards predecessor programmes. All qualitative fieldwork was based on a

standard protocol: the outputs were captured in standardised grids, reviewed and where

appropriate enriched with feedback asked by interviewers. Lastly, an expert panel reviewed

collectively the actual outputs of cooperation projects.

Source

Case studies

(ICF, Annexe 9)

Scope

Cases mainly at organisation level

to assess spill over effects and

changes at organisation and system

levels

Staff, learners, leadership and other

stakeholders if relevant

Extract information from

cooperation projects and system

level project actions, mainly from

predecessor programmes

Volume

38 cases

of which

32 at organisation level,

4 at policy level,

4 for Jean Monnet actions

233 respondents

Review of selected

projects’ outputs

(ICF, Annexe 17 – not

published)

386 project reports

541

Expert panel

assessment of Assess projects’ outputs against a

projects’ outputs

set of pre-determined criteria

(ICF, Annexe 10)

100 projects

(144 outputs)

541

The sample of 386 projects selected for review has the following characteristics: 292 projects are funded by

LLP (of which 100 are transversal actions) and 70 by Erasmus+; 250 projects are centralised and 136

decentralised. The projects were spread across the sectors/strands as follows: school (65), VET (49), HE

(25), adult learning (33), Jean Monnet (50), Youth (42) and sport (22).

108

kom (2018) 0050 - Ingen titel

Annex 4: Evaluation criteria and evaluation questions

Source: Terms of Reference for Mid-term evaluation of Erasmus+ (January 2016)

Effectiveness



To what extent have the various programme fields (both within Erasmus+ and its

predecessor programmes) delivered the expected outputs, results and impacts? What

negative and positive factors seem to be influencing outputs, results and impacts?

What are the long-term impacts of the predecessor programmes? We are interested in

the impact of all actions/elements of the predecessor's programmes, in particular those

actions/elements that are continued in new Erasmus+ programme. We are also

interested in impact of actions/elements that have discontinued to the extent that it

might help to design the future programme.

Please identify, describe and quantify (if possible) the spill over effects between

various actions (clusters of actions) of the Programme.

What are the differences in impact of programme actions on specific disadvantaged

groups of the population who traditionally do not engage in transnational activities as

compared to other groups that benefit from the programme?

Are there positive /negative effects that existed in the previous programmes, but that

no longer exist with the new programme?

What conclusions can be drawn on the likely impact of Erasmus+ programme given

the fact that significant parts of their actions are continuation of predecessor's

programmes?

109



kom (2018) 0050 - Ingen titel



What have been the unintended effects and their magnitude of the Erasmus+ and the

predecessor programmes if any?

To what extent are the effects likely to last after the intervention ends?

How effective are the forms of cooperation and the types of actions under the

Programme for the purpose of supporting the political priorities? Which fields and

actions of the Programme are the most effective considering the needs?

How and to what extent is the existence of one integrated Programme a help or a

hindrance in promoting greater effectiveness and visibility for the EU's activities in the

field of education and training, youth and sport, compared with a sector-specific or

target group-specific approach to Programme design, using sector-specific action

names (e.g. Comenius, Leonardo, Grundtvig)?

To what extent are the Programme results adequately disseminated and exploited?

Relevance



To what extent are current and emerging key socio-economic needs and challenges

that Europe is facing reflected in the policy priorities, objectives and actions/activities

of the Programme?

To what extent are needs of different stakeholders and sectors addressed by

Programme objectives? How successful is the Programme in attracting and reaching

target audiences and groups within different fields of the programme's scope? How

well is the Erasmus+ programme known to the education and training, youth and sport

community? In case some target groups are not sufficiently reached, what factors are

limiting their access and what actions could be taken to remedy this?

To what extent is the design of the programme oriented and focused towards the hard-

to-reach groups or specific disadvantaged groups of the population who traditionally

do not engage in transnational activities? In case some target groups are not

sufficiently reached, what factors are limiting their access and what actions could be

taken to remedy this?

Based on the analysis of the impact of predecessor programmes, are there any

elements that have been discontinued and could have a possible value added in future

generation of programmes?

Coherence



To what extent are the objectives of different programme fields within Erasmus+

programme consistent and mutually supportive? What evidence exists of synergies

between the different programme fields and actions? How well do different actions

work together? To what extent there exist duplications, overlaps, or other

disadvantageous issues between the programme fields and how are they dealt with?

To what extent are the centralised and decentralised actions coherent? How do they

interact / complement each other?

110



kom (2018) 0050 - Ingen titel



To what extent is the Erasmus+ programme coherent with relevant EU policies and

programmes with similar objectives such as Creative Europe Programme, European

Structural and Investment Funds (ESIF) or other EU's funding programmes such as

Horizon 2020, financial instruments relating to justice and citizenship, health, external

cooperation or pre-accession assistance programmes? To what extent has the

Erasmus+ programme proved complementary to other EU interventions/initiatives in

the field of education and training, youth and sport?

To what extent does the Erasmus+ programme design (including needs, objectives,

activities, outputs and effects) provide appropriate links and support to the EU policy

agendas, in particular policy initiatives and political priorities within the EU 2020

Strategy, the Education and Training Strategic Framework ET 2020, the European

youth strategy or the EU policy priorities in the field of sport?

To what extent is the Erasmus+ programme coherent with various interventions

pursued at national and international level which have similar objectives? To what

extent has the Erasmus+ programme proved complementary to other Member States'

interventions/initiatives in the field of education, training, youth and sport?

Efficiency



What is the cost-effectiveness of various actions (clusters of actions) of Erasmus+

programme and its predecessor programmes? How the relative costs and outcomes

(effects) of various programme actions compare within and across the programme

fields? Provide quantifications.

To what extent is/was the size of budget appropriate and proportionate to what the

Programme and its predecessors set out to achieve? To what extent is/was the

distribution of funds across the programme fields and Key Actions appropriate in

relation to their level of effectiveness and utility?

What is the prospect of other policy instruments or mechanisms in providing better

cost-effectiveness ratio?

To what extent is the implementation and management structure of centralised and

decentralised actions appropriate, efficient, and well-functioning? How efficient is the

cooperation between the different management bodies (Commission –Executive

Agency –National Agencies –European Investment Fund – National Authorities –

Independent Audit Bodies – Erasmus+ Committee), and to what extent does the

Commission fulfil its guiding role in the process? How has this changed between the

two programming period? What are the areas for improvements?

Are there differences in efficiency of Programme management and implementation

between National Agencies, the Commission Executive Agency, the European

Investment Fund or between different programming periods? If so, what are the

differences and what are the underlying reasons for them? Compare the strategies,

approaches and outcomes of the different National Agencies.

To what extent are the monitoring mechanisms applied by the Commission, the

Executive Agency, the National Agencies and the European Investment Fund

111



kom (2018) 0050 - Ingen titel

efficient/cost effective? To what extent are the monitoring mechanisms of the

beneficiaries and participants by National Agencies and the Executive Agency

effective and proportionate? To what extent are internal monitoring mechanisms of

activities of the National Agencies and the implementation of the Programme at

national level effective and proportionate? What are the areas for improvement,

considering the need for a smooth and effective implementation of the Programme?



To what extent do the indicators identified for the Programme in the legal base

correspond to the monitoring purposes? How could the overall management and

monitoring system be improved?

To what extent are the management support tools (e.g. Epluslink, Mobility tool)

adequate and sufficient to support a sound management of the Programme?

What are the efficiency effects of the integration of previous programmes and

actions/activities into Erasmus+ programme? What are the advantages and

disadvantages of the integration of previous programmes and actions into Erasmus+

programme?

To what extent has the system of simplified grants resulted in a reduction of the

administrative burden for National Agencies, Executive Agency and programme

beneficiaries and participants? Are there differences across actions or fields? What

elements of the programme could be changed to further reduce the administrative

burden and simplify the programme's implementation, without unduly compromising

its results and impact?

To what extent have the anti-fraud measures allowed for the prevention and timely

detection of fraud?

European added value



What is the additional value and benefit resulting from EU activities, compared to

what could be achieved by MS at national and/or regional levels? What does the

Erasmus+ programme offer in addition to other education & training support schemes

available at both international and national levels?

What is the benefit and added value of the Erasmus+ programme compared to the

benefit of the predecessor programmes? What is the added value of repetitive

actions/activities of the Erasmus+ programme and the predecessor programmes?

What would be the most likely consequences of stopping the Erasmus+ programme?

To what extent does the Erasmus+ programme promote cooperation between

participating countries?



112

kom (2018) 0050 - Ingen titel

Annex 5a: The intervention logic of Erasmus+ and its predecessor programmes

113

kom (2018) 0050 - Ingen titel

Programmes’ aims

Drawing on the legacy of its predecessors (Lifelong Learning, Youth in Action, Erasmus Mundus, Tempus, Alfa, Edulink and the

Preparatory Actions in Sport), Erasmus+ aims to contribute to similar general objectives in the fields of education and training, youth

and sport. As for its predecessors, these objectives have been defined based on EU level priorities set out in key policy documents (e.g.

Europe 2020, ET 2020, Youth Strategy, EU Sport Action Plan and ‘Increasing the impact of EU development policy: an Agenda for

Change’

542

and since then confirmed through various European Commission Communications such as: ‘New Skills Agenda for

Europe’

543

; ‘Renewed EU agenda for higher education’

544

, etc.). These objectives jointly with the programme’s rationale are specified

in the legal basis

545

and the programme guide

546

Next to these overarching objectives, Erasmus+ is aimed to help achieve a number of specific objectives applying to the respective fields

it covers. These notably relate to:

■

improving the level of key competences;

improving quality, innovation, excellence and internationalisation of education, training and youth organisations;

promoting the emergence of European lifelong learning area;

enhancing the international dimension of education, training and youth;

promoting teaching and research about the EU;

promoting good governance in sport, volunteering in sport and tackle threats to integrity in sport.

542

October 2011:

https://eeas.europa.eu/sites/eeas/files/com3a20113a06373afin3aen3atxt.pdf

June 2016:

https://ec.europa.eu/education/news/20160610-education-skills-factsheet_en

May 2017:

https://ec.europa.eu/education/sites/education/files/he-com-2017-247_en.pdf

Available at :

http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32013R1288&from=EN

Erasmus+ Programme Guide, Part A:

http://ec.europa.eu/programmes/erasmus-plus/sites/erasmusplus/files/files/resources/erasmus-plus-programme-guide_en.pdf

543

544

545

546

114

kom (2018) 0050 - Ingen titel

The predecessor programmes and Erasmus+ share a number of specific objectives that remained common to both periods. Over both

programming periods specific emphasis has been put for instance on:

■

Competence development of participating learners;

■

Professional development of staff; and

■

Foreign language learning;

However, the predecessor programmes and the current programme were designed in quite different contexts. Hence, a number of

differences in objectives can be noted:

■

The Erasmus+ programme has stronger emphasis on high level policy objectives (and result-oriented approaches);

■

The emphasis on employability is also clearer in the current programme which was designed in a period when young people were

facing high unemployment, unlike the period of design of the predecessor programmes;

■

The predecessor programmes on the other hand had specific objectives about the quality and volume or mobility exchanges and

quality and volume of organisational cooperation. This ‘Europeanisation’ was perceived as an objective in its own right in the

predecessor programmes while in the current programme it seems to become a means to achieve other ends rather than a goal on its

own;

■

The predecessor programmes also emphasised quite strongly the use of ICT in education and training and the introduction of ICT-

based pedagogies was an objective of the programme while it does not figure in the Erasmus+ legal basis;

■

The youth in action programme put much more emphasis on the objectives of youth participation and the citizenship dimension of

the youth programme than the current programme.

There were also some issues that were topical at a given point in time for which specific actions were implemented temporarily. This is

for instance the case of the situation of Roma population across the EU which was rather high on the policy agenda in the period 2009-

2010. Regarding Erasmus+, the issue of social inclusion has become for instance prominent during the refugee crisis since 2015 and the

issue of violent radicalisation became urgency after the terrorist attacks that started in 2015.

Overall, despite progressive adjustments as above, the aims of Erasmus+ have not radically changed in comparison to those of its

predecessors. A noticeable difference between the current and the previous programmes is the integrated architecture of Erasmus+.

Bringing together the education and training (including Jean Monnet programme), youth and sport fields into a single integrated

programme is in particular expected to foster synergies, cross-fertilisation and to stimulate new forms of cooperation that did not or

failed to materialise in the past.

115

kom (2018) 0050 - Ingen titel

Inputs

To operate the main types of actions and achieve the expected outcomes and long-last impacts discussed below, three main types of

inputs underpin the Erasmus+ programme:

■

funding range of actions;

■

system and management structures, and;

■

support measures for dissemination and knowledge management.

Whilst these do not differ much in theory from those offered under the predecessor programmes, Erasmus+ inputs are in practice rather

different. A number of novelties have been indeed brought into Erasmus+. Besides the integrated structure mentioned above, the most

noticeable changes have taken the form of: an increased budget allocation; a renewed internal structure of the programme; the

introduction of new implementation and monitoring approaches and support measures for dissemination and knowledge management.

Types of actions

A major evolution compared to the predecessor programmes has been the change in programme structure. Rather than being structured

by sectors with each sector having embedded a variety of types of actions each specific to a given sector, the programme was

restructured according to main categories of types of actions (Key Actions - KA) which are common to the education and training and

youth fields. The sport field and the Jean Monnet programme have made the object of separate chapters. While there are still some

specificities in the fields and subfields, the main types of actions are shared. These are:

■

Mobility of individuals: through KA1 (in education and training and youth);

■

Cooperation partnerships: through KA2 (in education and training and youth) and other cooperation actions in sport and Jean

Monnet) ; and

■

System level projects: through KA3 and ad hoc actions in sport and Jean Monnet

This logic of these three main types of actions corresponds to those levels at which the programme aims to trigger change: individual,

organisation and system. This is an improvement in the logic of the programme compared to the myriad of actions with different names

under the predecessor programmes.

These broad categories of actions are further subdivided into a small number of types of actions which as often as possible share a

common name if they are common to several sectors. This enables to cater for a variety of needs within a broad category of activities

funded.

116

kom (2018) 0050 - Ingen titel

Expected outcomes and long-lasting impact

Erasmus+ aims to deliver outcomes and long-lasting impact at its three levels of intervention as the figure above illustrates. This can be

further summarised as follows:

■

At individual level: the programme is aimed to bring positive changes at both learners (students, trainees, apprentices, young people

and volunteers) and practitioners (teachers, trainers, youth workers) in the form of (not exhaustive): improved skills and

competences (including soft skills), self-empowerment and self-esteem, better awareness of the EU values, etc. For practitioners,

additional outcomes are expected such as: enhanced motivation, opportunities to test and implement new practices, ability to address

the needs of the disadvantaged, etc. The achievement of these outcomes is in turn expected to generate long-lasting impact at

individual level (e.g. enhanced employability, entrepreneurship, active participation in society, participation in formal/non-formal

education or training, etc.) but also at organisation and system levels (e.g. improved education attainment and completion rates,

employability, transition to further levels of education, solidarity and career progression of staff).

■

At organisation level: the transnational cooperation project opportunities offered by the programme are expected to generate the

following types of outcomes (not exhaustive): development and/or implementation of new pedagogies or curricula, implementation

of new organisational practices, enhanced networking with foreign partners (including outside Europe and from other fields),

improving the dialogue between the academic research arena and policy makers. etc. The achievement of these outcomes is in turn

expected to generate long-lasting impact at system level notably in the form of better quality of teaching, youth work and sport

activities, sustainable partnerships, increased levels of participation in sport, physical activity and voluntary activity, etc.

■

At system level: much greater systemic impact than in the past (e.g. KA3 clearly sets a framework for system level-oriented actions)

is expected overall. Anticipated outcomes at both EU and national levels relate to achieving: stronger awareness about key policy

challenges in education and training, youth and sport; enhanced mutual learning and good practice exchanges among policy makers

and key stakeholders; better understanding of key EU tools and policies; supporting research and training about the EU, etc. This is

in turn aimed to help achieve long-lasting impact in the form of:

–

Enhanced quality, efficiency and equity of education and training systems and youth policies through the OMC;

–

Effective implementation of reforms converging with the OMC at national level

–

Effective implementation of EU tools for assessment, transparency and recognition of skills and qualifications acquired through

formal, non-formal and informal learning at national level;

–

Increased visibility of the external dimension of the programme (both within and outside Europe) and credibility to support

structural reforms in partner countries, etc.

117

kom (2018) 0050 - Ingen titel

The different levels of intervention and related types of actions are not to be seen in isolation but on the contrary as aiming to contribute

to commonly shared objectives and to generate mutually reinforcing outcomes and impacts. More than in the past, spill-over effects are

expected to materialise across Erasmus+. Overall, the logic of the programme is that the simplification it offers should help reach greater

and long-lasting impact at the individual, organisation and system levels and contribute to the achievement of the key EU strategic

documents mentioned above.

118

kom (2018) 0050 - Ingen titel

Annex 5b: Erasmus+ Factsheet

119

kom (2018) 0050 - Ingen titel

120

kom (2018) 0050 - Ingen titel

Annex 5c: Specific objectives of Erasmus+ by level of intervention

Envisaged effect at:

Specific objective

Education and training

a To improve the level of key competences and

skills, with particular regard to their relevance

for the labour market and their contribution to a

cohesive society, in particular through

increased opportunities for learning mobility

and through strengthened cooperation

between the world of education and training

and the world of work

b To foster quality improvements, innovation

excellence and internationalisation at the level

of education and training institutions, in

particular through enhanced transnational

cooperation between education and training

providers and other stakeholders

c To promote the emergence and raise

awareness of a European lifelong learning area

designed to complement policy reforms at

national level and to support the modernisation

of education and training systems, in particular

through enhanced policy cooperation, better

use of Union transparency and recognition

tools and the dissemination of good practices

d To enhance the international dimension of

education and training, in particular through

cooperation between Union and partner-

country institutions in the field of VET and in

higher education, by increasing the

attractiveness of European higher education

institutions and supporting the Union's external

action, including its development objectives,

through the promotion of mobility and

cooperation between the Union and partner-

country higher education institutions and

targeted capacity-building in partner countries

e To improve the teaching and learning of

languages and to promote the Union's broad

linguistic diversity and intercultural awareness

To promote excellence in teaching and

research activities in European integration

through the Jean Monnet activities worldwide

Individual

level

Institutional

level

Systemic level

National

****

***

****

***

****

Envisaged effect in partner countries

****

***

****

121

kom (2018) 0050 - Ingen titel

Envisaged effect at:

Specific objective

Youth

a To improve the level of key competences and

skills of young people, including those with

fewer opportunities, as well as to promote

participation in democratic life in Europe and

the labour market, active citizenship,

intercultural dialogue, social inclusion and

solidarity, in particular through increased

learning mobility opportunities for young

people, those active in youth work or youth

organisations and youth leaders, and through

strengthened links between the youth field and

the labour market

b To foster quality improvements in youth work,

in particular through enhanced cooperation

between organisations in the youth field and/or

other stakeholders

c To complement policy reforms at local,

regional and national level and to support the

development of knowledge and evidence-

based youth policy as well as the recognition

of non-formal and informal learning, in

particular through enhanced policy

cooperation, better use of Union transparency

and recognition tools and the dissemination of

good practices

d To enhance the international dimension of

youth activities and the role of youth workers

and organisations as support structures for

young people in complementarity with the

Union's external action, in particular through

the promotion of mobility and cooperation

between the Union and partner-country

stakeholders and international organisations

and through targeted capacity-building in

partner countries

Sport

a To tackle cross-border threats to the integrity

of sport, such as doping, match-fixing and

violence, as well as all kinds of intolerance and

discrimination

b To promote and support good governance in

sport and dual careers of athletes

c To promote voluntary activities in sport,

together with social inclusion, equal

opportunities and awareness of the importance

of health-enhancing physical activity through

increased participation in, and equal access to,

sport for all

Individual

level

Institutional

level

Systemic level

National

****

***

****

***

****

Envisaged effect in partner countries

****

***

****

***

****

***

****

Source: Terms of reference of the mid-term evaluation

122

kom (2018) 0050 - Ingen titel

Annex 5d: Predecessor programmes factsheet

ERASMUS+ predecessor programmes

1. The Lifelong Learning Programme

The Lifelong Learning Programme (LLP)

547

was designed to enable people, at

any stage of their life, to take part in stimulating learning experiences, as

well as developing education and training across Europe. Nearly €7 billion,

which ran from 2007-2013, funded a range of transnational learning mobility

exchanges, study visits and networking activities. Most of the activities of the

LLP continue under the new Erasmus+ programme 2014-2020.

Over the course of its lifespan, the LLP provided support to school pupils, university

students, adult learners, and a variety of projects under the following main sub-

programmes:



Comenius for schools

Leonardo da Vinci for vocational education and training

Erasmus for higher education

Grundtvig for adult education

Jean Monnet actions, designed to stimulate teaching, reflection, and debate on

European integration

Comenius

The Comenius sub-programme focused on all levels of school education, as well as the

individuals involved, including pupils, teachers, local authorities, and education

institutions, among others. It aimed to:



547

Improve and increase the mobility of pupils and staff across the EU

Enhance and increase school partnerships across the EU

Encourage language learning, ICT for education, and better teaching techniques

http://ec.europa.eu/education/lifelong-learning-programme_en

123

kom (2018) 0050 - Ingen titel



Enhance the quality and European dimension of teacher training

Improve approaches to teaching and school management.

The total budget for Comenius in the 2007-2013 period was EUR 1,190 million and

benefited to nearly 980,000 participants.

The sub-programme funded the following main actions:

•

Mobility of pupils and school staff

Bilateral and multilateral partnerships between various schools in the EU

Multilateral projects and networks relating to improving language learning, ICT

for education, and better teaching techniques.

•

Leonardo da Vinci

The Leonardo da Vinci sub-programme funded practical projects in the field of vocational

education and training. The total budget for Leonardo da Vinci in the 2007-2013 period

was EUR 1,820 million and benefited to around 590,000 participants.

The sub-programme was aiming to:



Enhance the competitiveness of the European labour market by helping European

citizens to acquire new skills, knowledge and qualifications and have them

recognised across borders

Support innovations and improvements in vocational education and training

systems and practices.



Leonardo da Vinci funded the following main actions:

•

Mobility of vocational education and training students and staff

Bilateral and multilateral partnerships between various vocational education and

training providers in the EU for transfer of innovation, experience or good

practices.

Multilateral projects and networks relating to improving the quality of training

systems through the development of innovative contents, methods and

procedures for vocational education and training.

•

124

kom (2018) 0050 - Ingen titel

125

kom (2018) 0050 - Ingen titel

Erasmus

The total budget for Erasmus programme in the

2007-2013 period was EUR 3.1 million. In addition

to the 230,000 students supported each year (1.6

million over 2007-2013), Erasmus also provided

opportunities for over 300,000 academic and

administrative staff in higher education, with 4,000

institutions and 33 countries participating.

The sub-programme was aimed to:

•

Improve and increase the mobility of students and staff in higher education to

study, teach and train across the EU

Enhance and increase higher education institutions partnerships and networks,

thus promoting innovation, quality and relevance of higher education across the

EU.

The Erasmus sub-programme

supported student and staff

exchanges and transnational

cooperation in the field of higher

education.

The Erasmus programme supported the mobility through grants and provided co-funding

to transnational cooperation projects and networks.

Grundtvig

The Grundtvig sub-programme focused on the teaching and study needs of adult

learners, as well as developing the adult learning sector in general. The total budget for

Grundtvig programme in the 2007-2013 period was EUR 420 million and benefited to

nearly 170,000 participants.

Covering teachers, trainers, staff, and adult learners, among others, the sub-programme

aimed to:



Increase the number of people in adult education

Improve mobility conditions in adult learning

Improve the quality and cooperation between adult education organisations

Develop innovative educational and management practices

126

kom (2018) 0050 - Ingen titel



Ensure social inclusion through adult education

Support innovative ICT-based educational content, services, and practices.

Grundtvig supported:

•

Mobility of adult learners and adult learning staff

Bilateral and multilateral partnerships between various adult learning providers in

the EU

Multilateral projects and networks relating to improving the quality of adult

learning through the development of innovative contents, methods and

procedures for adult learning and making adult learning more accessible to the

potential users.

Jean Monnet

The Jean Monnet sub-programme was a component of LLP focusing on promoting

teaching and research on European integration matters. The total budget for the Jean

Monnet programme in the 2007-2013 period was EUR 140 million and benefited to over

1,100 practitioners.

It consisted of three key activities:

The Jean Monnet Action, designed to stimulate teaching, research, and reflection

on European integration, consisting of Jean Monnet Chairs, Centres of Excellence,

and Modules, among others;



Support for six specific academic institutions;



Support for Europe-wide associations active in the area of European integration

research.

These were complemented by conferences, thematic groups, and policy support within

the European Commission.



127

kom (2018) 0050 - Ingen titel

7. Youth in Action

Youth in Action

548

was the Programme of the European Union for young

people from 2007 to 2013. It also aimed to promote out of school mobility

within and beyond the EU’s borders, non–formal learning and intercultural

dialogue, and encouraged the inclusion of all young people regardless of their

educational, social and cultural background.

The total budget for the programme in the

2007-2013 period was EUR 885 million.

Benefiting to close to 1 million participants,

young people and youth workers, it had also

contributed to the effective recognition of

non-formal learning with 265,000 Youthpass

- the Youth in Action learning opportunities

certificate - delivered since 2007.

YiA aimed to inspire a sense of active

European citizenship, solidarity and

tolerance among young Europeans and to

involve them in shaping the Union's

future.

In order to achieve its objectives, the Youth in Action Programme implemented the

following actions:



Action 1 - Youth for Europe: groups of young people from different countries plan

together their Youth Exchange to learn about each other’s cultures; networking of

similar projects in order to strengthen their European aspect; support to young

people’s participation in the democratic life at all levels.



Action 2 - European Voluntary Service: young people take part individually or in

groups in non-profit, unpaid activities, within and outside the European Union



Action 3 - Youth in the World: cooperation with Partner Countries from other

parts of the world (exchange of good practice, etc)



Action 4 - Youth Support Systems: support for organisations and youth workers

(training, networking, partnerships, etc)

548

http://eacea.ec.europa.eu/youth/programme/about_youth_en.php

128

kom (2018) 0050 - Ingen titel



Action 5 - Support for European cooperation in the youth field:

between those

responsible for youth policy, those active in youth work and young people,

(seminars, Structured Dialogue, etc)

8. Erasmus Mundus 2009-2013

Erasmus Mundus II

549

was a cooperation and mobility programme in the field

of higher education over 2009-2013. The total budget for the programme in

the 2007-2013 period was EUR 950 million and benefited to around 13,000

participants in the 2007-2013 period.

The Erasmus Mundus programme provided support to:

Higher education institutions that wished to implement joint programmes at

postgraduate level (Action 1) or to set-up inter-institutional cooperation

partnerships between universities from Europe and targeted Third-Countries

(Action 2);



Individual students, researchers and university staff who wish to spend a study /

research / teaching period in the context of one of the above mentioned joint

programmes

cooperation

partnerships (Action 1 and Action 2);



Any organisation active in the

Erasmus Mundus aimed to enhance the quality of

field of higher education that wishes

European higher education and to promote

to develop projects aimed at

enhancing the attractiveness and

dialogue and understanding between people and

visibility

European

higher

cultures through cooperation with the third

education worldwide (Action 3).

countries



549

http://eacea.ec.europa.eu/erasmus_mundus/index_en.php

129

kom (2018) 0050 - Ingen titel

9. Tempus

Tempus

programme

550

supported

the

modernisation of higher education in the EU's

surrounding area. The total budget for Tempus IV

was EUR 258.7 million, with around 3,000

participants in the 2007-2013 period.

Tempus promoted a people-to-

people approach.

The programme aimed to promote voluntary convergence of the higher education

systems in the Partner Countries with EU developments in the field of higher education.

With regards to the Western Balkans, Tempus contributed to preparing the candidate

and potential candidate countries for a participation in the integrated Life Long Learning

Programme.

Tempus promoted through institutional cooperation the modernisation of higher

education systems in the Partner Countries of Eastern Europe, Central Asia, the Western

Balkans and the Mediterranean region.

In addition, Tempus provided support to consortia of institutions composed mainly of

universities or university associations. Non-academic partners could also be part of a

consortium.

The Tempus programme was implemented in close coordination with the Erasmus

Mundus programme which provided scholarships also to third country students allowing

them to participate in top-level Master courses and Doctorate programmes outside the

EU.

550

http://eacea.ec.europa.eu/tempus/programme/about_tempus_en.php

130

kom (2018) 0050 - Ingen titel

10. Alfa

The ALFA III Programme

551

aimed at the modernisation of Higher Education

in Latin America with a view to promoting sustainable and equitable

development in the region. The total budget for the programme in the 2007-

2013 period was EUR 950 million (EU contribution of EUR 75 million) and

benefited to 153 participating institutions from the EU and 341 from Latin

America.

In this regard, ALFA III aimed at

strengthening bilateral and multilateral

relations between the two regions, where

higher education institutions play a leading

role in the process of improving the quality

of national education systems which in turn

enables the socio-economic development.

This programme aimed to ensure a process

of ownership by the Latin American

countries through the creation of networks

and synergies between universities in Latin

America and Europe.

The third phase of the programme – ALFA III 2007-2013 – financed a diversity of

projects to improve the quality, relevance and accessibility of higher education in Latin

America and further regional integration through the creation of a higher education area.

The participating countries were the 28 Member States of the European Union and 18

countries of Latin America: Argentina, Bolivia, Brazil, Chile, Colombia, Costa Rica, Cuba,

Ecuador, El Salvador, Guatemala, Honduras, Mexico, Nicaragua, Panama, Paraguay,

Peru, Uruguay and Venezuela. Non-government organisations, chambers of commerce,

professional associations, private companies had an associate role.

551

http://ec.europa.eu/europeaid/where/latin-america/regional-cooperation/alfa/index_en.htm_en

131

kom (2018) 0050 - Ingen titel

11. Edulink

The total budget for Edulink

552

in the 2006-2013

period was EUR 58.3 million.

The programme aimed to increase access to quality

education that will enable ACP students to

undertake postgraduate studies, and to promote

student retention in the region, while increasing

the competitiveness of the institutions themselves,

through regional and multilateral networking

between higher education institutions, capacity

building and intra-ACP academic mobility of

students and staff.

The scheme provided support to:



Higher education institutions to set up inter-institutional cooperation partnerships

between universities from different countries within the ACP regions;

Individual students, researchers and university staff to spend a study / research /

teaching period in the context of one of the cooperation partnerships.

The programme was designed to

foster co-operation in higher

education between the countries of

the African, Caribbean and Pacific

Group of States (ACP States) and

the EU

552

http://www.acp-hestr.eu/

132

kom (2018) 0050 - Ingen titel

12.

Preparatory actions in sport 2009-2013

The main objective of Preparatory Actions

553

was to prepare future EU actions

in the field of sport. The funding in the 2007-2013 period for the preparatory

actions was EUR 14.5 million.

The actions funded transnational projects

put forward by public bodies or civil

society organisations in order to test

suitable networks and good practices in

the field of sport. The preparatory actions

also served to support knowledge-base in

the sport area through studies, surveys

and conferences.

Those were mostly aimed to support

cooperation among sport organisations,

through small scale partnerships and the

organisation of raising awareness events.

For instance, the areas covered in the preparatory actions in 2012-2013 included:



Strengthening of good governance and dual careers in sport through support for

the mobility of volunteers, coaches, managers and staff of non-profit sport

organisations,

Protecting athletes, especially the youngest, from health and safety hazards by

improving training and competition conditions,

Promoting traditional European sports and games,

Supporting the 'fight against match-fixing',

Promoting physical activity supporting active ageing,

Awareness-raising about effective ways of promoting sport at municipal level,

Trans-frontier joint grassroots sport competitions in neighbouring regions and

Member States.



553

http://ec.europa.eu/sport/policy/preparatory-actions/preparatory-actions_en.htm

133

kom (2018) 0050 - Ingen titel

Annex 5e: Indicators of the Erasmus+ programme legal basis

Overview of outputs (legal basis and strategic plan indicators)

Legend: D = DG EAC dashboard; AR = Annual report; ICF survey = ICF survey of beneficiaries; O = Other

Commission data (annual reports)

Target

Legal

basis 2014

Total (if

applicable)

State of play

Judgeme

2015

2016

2014–2016

Comment

Source

Definition

Number of

students and

trainees

participating in

the programme,

by country,

sector, action

and gender

Number of

students who

received the

guarantee

Learning mobility opportunities through Erasmus+

Substantia

lly above

target

Substantia

lly above

target

- Higher education

(without international)

236,0

239,0

248,0

723,000

869,615

- VET

78,00

80,00

236,000

279,982

Student Loan

Guarantee

200,000

(full period) 247

Substantia

lly below

target

Practitioners supported by the programme (Erasmus+, education and training)

Assuming

even

distribution

across

years:

150,000

No target was set

for years 2014 and

2015 so it is

assumed that it

was the same as in

2016

- HE practitioners

(excluding

international)

50,00

159,177

Above

target

- VET practitioners

11,00

As above:

33,000

50,556

Substantia

lly above

target

Idem

Substantia

lly above

target

Idem

Substantia

lly above

target

Idem

Number of

practitioners

supported by the

programme, by

country and in

the sector of

education and

training

-Schools practitioners

15,00

As above:

45,000

54,341

- Adult learning

3,000

As above:

9,000

12,971

134

kom (2018) 0050 - Ingen titel

Participants with special needs or fewer opportunities

Special needs

(4,034)

The categories are not

mutually exclusive.

Substantia Summing up the

lly above different categories

would lead to double

Disadvantaged target

according counting. However,

background

indicator of

(51,852)

programm disadvantaged young

Outmost

e data but people in higher

regions

issues of

education alone is

(9,148) - used data

substantially above

in HE

quality

the target

Fewer

opportunities

(9,580)

Education and training

15,00

Assuming

even

distribution

across

years:

45,000

Number of

participants with

special needs or

fewer

opportunities

supported by the

programme

Number of

participants with

special needs or

fewer

opportunities

supported by the

programme

(Erasmus+,

youth)

Participants with

special needs or fewer

opportunities (youth)

International higher education

Assuming

21,60 even

distribution

(2017) - 64,800

Special needs

– 6,551

Fewer

Substantia

opportunities - lly above

76,737

target

Involvement of non-EU

higher

education

institutions from partner

countries

n/a

1,000

1,100

2,200

Data concerns

numbers of

organisations in

general including EU

ones (capacity

building and

international student

and practitioner

mobility).

Considering that in

2016 there were more

grants awarded than

in 2015 the number is

likely to be

significantly over

target even given that

not all the

organisations

captured in this

indication are from

Substantial partner countries

2,523 in 2015 ly above

(some are from

alone

target

programme countries) R

The number of

partner country

higher education

institutions

(HEIs) involved

in credit and

degree mobility

and cooperation

actions, i.e.

capacity-building

projects under

the Erasmus+

programme and

having signed an

institutional

agreement with

an EU HEI

Number of

higher education

students and

practitioners

(134) receiving

support (a

scholarship) to

study in a partner

country, as well

as the number of

students and

practitioners

coming from a

partner country

to study in a

programme

country

students

and

practitioners going to

partner countries and

vice versa

38,800

39,448

On target

- Outgoing total

- Incoming total

n/a

3,800

4,000

7,800

31,000

The cap on number of

outgoing students

Substantial towards neighbouring

11,016 (2,670 ly above

countries was lifted in

learners)

target

2016

28,432

Below

Where the numbers

15,000 16,000

135

kom (2018) 0050 - Ingen titel

(16,413

learners)

target

for incoming mobility

are not used the

budgets are used to

fund outgoing

mobility

Youth mobility

Number of young

people engaged

in mobility

actions supported

by the

programme, by

country, sector,

action and gender

Number of

practitioners

supported by the

programme, by

country and for

the youth sector

Number of youth

organisations

from both

programme

countries and

partner countries

involved in

international

mobility and

cooperation

actions under the

Erasmus+

programme

Learning

mobility

opportunities through

Erasmus+ (youth)

69,00

70,00

70,000 0

209,000

277,913

Substantia

lly above

target

Practitioners supported

by the programme

(Erasmus+, youth)

21,00

22,00

21,000 0

64,000

101,437

Substantia

lly above

target

Involvement of EU and

non-EU

youth

organisations

(Erasmus+)

Services/information networks

5,500

5,600

Assuming

2015 is the

same as

2014:

16,600

87,773

(mobility) and

3150

(Strategic

Partnerships)

Substantia

lly above

target

indicator is

not

included in

the

Number of users of strategic

Euroguidance

plan)

n/a

Approx.

62,000 (2014–

2015)

The exact

indicators

reported differ

according to

reports. This

data reflects

participants in

activities

Number of users of

Euroguidance

The number of users

of Eurodesk (as a

support organisation

to Erasmus+,

Eurodesk, a network

of 1,200 youth

professionals, worked

with Eurodesk in 34

countries in 2016,

making information

on learning mobility

accessible to young

people and youth

professionals)

Users of Eurodesk

(enquiries answered)

140,0

140,00 140,0

420,000

The average

per year is

524,566

Substantia around

(2014–2015 - lly above 250,000 rather

n/a for 2016)

target

than 140,000

136

kom (2018) 0050 - Ingen titel

Jean Monnet

The indicator

on number of

participants

is not reliable

(double

counting)

and

Below target inconsistency

according to with other

programme

sources of

data and

data on

issues of data participants

quality

reached

Students trained through

Jean Monnet activities

215,00 235,00 260,00

710,239

667,199

Number of students

receiving training

through Jean Monnet

activities

Number of countries

where Jean Monnet

activities have been

performed

successfully,

increasing knowledge

in partner countries

Worldwide scope of

Jean Monnet activities

On target

Sport

Estimates

based on

survey:

- 75 % of

respondents

have 100 or

fewer

members

(including no

members)

- 58 % have

10 or fewer

employed

practitioners

Size of membership of

sport organisations

30 %

(2017)

Substantially

above target

This data is

not captured

in Erasmus+

monitoring

systems

Size of membership

of sport organisations

(% of small grassroots

organisations with

ICF

fewer than 1000

survey

members) applying

(practi

for, and taking part in,

tioners

the programme, by

)

country

137

kom (2018) 0050 - Ingen titel

Overview of results (legal basis and strategic plan indicators)

Legal

basis

Definition

strategic plan

2014–2016 performance

Judgeme

Source of

data

Result indicator

Education and training

Target

Comment

83 %

% of Erasmus+

participants

declaring that

they have

improved their

key competences

and/or their skills

relevant for

employability

87 % of beneficiaries state that they

Responses range from On target

have improved six or more competences 71 % for HE studies

from the proposed list

(long-term mobility)

to 88 % long-term

More than 80 % of learners agreed that mobility in VET

they improved the following

competences: cooperation in teams,

planning and organisation tasks,

planning and organising learning

activities, problem solving skills, seeing

the value of different cultures

83 % of learners state that they believe

they improved their professional skills

The responses range

from 72 % for HE

studies to 89 % for

VET

Above

target

Erasmus+

monitoring

survey of

beneficiari

(learners)

Better skills for

participants (Erasmus+,

education and training)

44 %

% of Erasmus+

participants

indicating that

participation in

the programme

contributed to

finding a job

88 % believe that their chances to find a From 82 % for HE

job improved

studies to 90 % for

international HE

92 % of respondents agree that their

Ranging from 90 %

opportunities for finding a job improved for pupils (blended

mobility) to 94 % for

higher education

Beneficiaries from VET and HE

experience shorter transition to

employment than control group

Taking part in an

Erasmus+

intervention appears

to improve the

transition from

education to

employment with a

13 % increase in the

indicator

The Erasmus+

monitoring survey as

well as ICF survey is

based on a survey of

practitioners taking

part in the programme

not organisations.

Hence there is only a

partial match between

the definition and this

data

Idem

ICF survey

beneficiary

learners

Employability of

participants (Erasmus+)

55 %

% of

organisations that

have

developed/adopte

d innovative

methods and/or

materials,

improved

capacity;

outreach

methodologies,

etc

When asked about expected changes to

their institutions, for 10 (out of 11)

changes more than 55 % (up to 85 %) of

practitioner respondents state that they

agree these changes have or will happen

On target

(possibly

above)

There is

not a

perfect

match with

the

definition

but the

data

collected

from

mobile

practitione

rs gives

very

positive

feedback

Erasmus+

monitoring

survey of

practitione

beneficiari

ICF survey

beneficiary

practitione

Innovation from

participating

organisations

(Erasmus+)

When asked about changes that have

happened after the mobility of project,

more than 55 % of respondents agree

with a high number of types of changes

asked about

Introduction of new teaching materials

(81 % schools, 72 % VET, 72 % HE,

77 % adult education)

New assessment or evaluation methods

(68 % schools, 63 % VET, 59 % HE,

68 % adult education)

Formal recognition of

participation (Erasmus+;

education and training)

HE –

100 %

% of Erasmus+

participants who

have received a

certificate,

diploma or other

kind of formal

recognition of

their participation

in the programme

Recognition of

On target

learning outcomes

The monitoring surveys of beneficiaries (academic

do not ask the question about whether

recognition) is at

learners received a certificate, etc.

83 %. It ranges from

However, considering that the issuing

75 % for international

of a transcript to mobile learners is one students to 89 % for

of the requirements of the HE

traineeships

University Charter which is a

precondition to receive Erasmus+

funding, it can be assumed the rate of

recognition of participation rate is

100 %

100 % (estimate)

Erasmus+

monitoring

survey of

beneficiari

es(learners

)

138

kom (2018) 0050 - Ingen titel

Result indicator

Legal

basis

Target

VET –

68 %

Definition

strategic plan

2014–2016 performance

88 % (2015)

Comment

The most common

tool for recognition is

Europass which is

received by 80 % of

participants

Judgeme

Above

target

Source of

data

Erasmus+

monitoring

survey of

beneficiari

es(learners

)

ICF survey

practitione

Practitio

ners

(not part

of the

targets)

85 % of practitioners got international

experience accepted as training (from

83 % in HE to 89 % in schools)

58 % state it was acknowledged by

hierarchy or peers (53 % in HE to 65 %

in AE)

45 % state that it helped them attain a

new function/ level of seniority (36 %

in schools to 48 % in adult education)

26 % received a financial reward (21 %

in schools – 28 % in HE)

HE –

95 %

% of Erasmus+

participants in

long-term

mobility

declaring that

they have

increased their

language skills

84 % HE studies

77 % international HE

76 % HE traineeships

Note: 14 % of students are fluent in the

language of the mobility before

departing on mobility

VET –

87 %

86 % VET learners

Given that HE

students start with a

higher level of foreign

language skills it

would be expected

that the improvement

is more modest than

in VET where the

entry level is lower.

The indicator has

been set assuming

that the duration of

mobility makes the

biggest difference but

the entry level makes

a very important

difference

Below

target if

taking as

share of all

students/

above

target if

excluding

those who

were

already

fluent

before

starting

mobility

On target

Erasmus+

monitoring

survey of

beneficiary

learners

Note the data for VET does not cover

only long-term mobility as suggested

by the indicator

According to OLS data:

HE – 34 % large improvement, 14 %

small improvement, 34 % no change

and 19 % regressed score

VET – very similar trends - 34 % large

improvement, 13 % small

improvement, 35 % no change, 18 %

regressed score

This data measures

actual change in a

score on a before and

after language test

It shows that there is a

big difference in self-

perception values and

actual competence

development

OLS data

on pre-post

tests

Language skills of

participants

Youth

77 %

% of Erasmus+

participants

declaring

that

they

have

improved

their

key competences

and/or their skills

relevant

for

employability

87 % of young people in youth

exchanges state having improved six or

more competences

More than 77 % of young people in

youth exchanges state having improved

problem solving (78 %), practical skills

(planning etc. – 78 %), learning to learn

(82 %), interpersonal and social

competence (92 %), cultural awareness

and expression (93 %) and intercultural

competences (84 %)

Above

target

Erasmus+

monitoring

survey of

beneficiari

es(learners

)

Better

skills

for

participants (Erasmus+,

youth)

Social and political

participation of young

80 %

% of Erasmus+ 74 % of EVS participants state that they The question asked in Below

young

intend to participate more actively in the

Erasmus+ target

Erasmus+

monitoring

139

kom (2018) 0050 - Ingen titel

people (Erasmus+)

participants

declaring being

better prepared to

participate

social

and

political life

their monitoring

survey

does

not

exactly

correspond to the

73 % state that they are more interested definition

in knowing what happens in the word

daily

It is not clear why the

target was set so high

71 % are more aware of social and

political concepts

There is a statistically significant

difference between young people who

took part in youth exchanges or EVS

when it comes to active citizenship

(voting,

etc),

participation

community activities and attitude to

protest strategies.

According to the ICF survey: 55 % of

youth beneficiaries think it is important

to engage in political discussions (50 %

control group), 89 % believe it is

important to participate in local

community activity (69 % control

group),

77 %

believe

active

membership in an association is

important (62 % in control group),

80 % believe that

organising a local-

level community activity is important

(69 % control group)

The data used for the

analysis in this report

is an index which

combines responses

on the perceived

importance of a range

of civic and political

activities

When looking at the

detailed responses the

differences with the

control group are

large in some cases

(see left cell)

social and

community

political

life

survey of

beneficiari

es(learners

)

ICF survey

of learners

90 %

Language

skills

participants (Erasmus+,

youth)

% of Erasmus+ 81 % of participants in EVS state they

participants

in improved their foreign language skills

voluntary actions

declaring

that

they

have

increased

their

language skills

According to OLS data 79 % of young

people taking part in youth actions

improved their foreign language skills

(69 % large improvement and 10 %

small improvement)

% of Erasmus+

participants who

have received a

certificate

(for

example

Youthpass),

diploma or other

kind of formal

recognition

their participation

in the programme

77 % of EVS volunteers and 88 % of

young people in youth exchanges state

having received some form of formal

recognition

Though the self-

perceived

improvement is below

the

target

the

improvement based

on OLS is large and

substantially greater

than for other groups

Below

target

according

Erasmus+

monitoring

data

but

strong

improveme

nt based on

OLS

Erasmus+

monitoring

survey of

beneficiary

learners

OLS pre-

post

language

tests

35 %

The vast majority Substantial Erasmus+

receive

Youthpass ly above monitoring

(87 % EVS and 88 % target

survey of

youth exchanges)

beneficiary

(but target learners

was

set

very low)

Formal recognition of

participation (Erasmus+,

youth)

Sport

Percentage of

participants

(expressed as %

of Erasmus+

sport

organisations)

who have used

the results of

cross-border

projects to

improve good

governance and

dual careers

Percentage of

participants

(expressed as

Erasmus+ sport

organisations)

67 % of respondents state that they used Data is not available

the results to improve governance or

through Erasmus+

manage quality

monitoring surveys

which are not carried

out among sport

beneficiaries

27 % of respondents state that their

project focused on the topic of good

governance

31 % state that their project focused on

dual careers

(multiple choice question)

62 % of respondents state they used the

results to combat threats to sport

The indicators are

developed based on

survey of sport

organisations

practitioners

Idem

Above

target

ICF survey

of sport

practitione

Above

target

ICF survey

of sport

practitione

Results increasingly

used to improve good

governance in sport and

dual careers

50 %

(2017)

Results increasingly

used to fight against

threats to sport

50 %

(2017)

10 % state that their project focused on

140

kom (2018) 0050 - Ingen titel

that use the

results of cross-

border projects to

combat threats to

sport

combating match fixing

12 % state the project focused on

combating doping

32 % state the project focused on

combating racism and violence

(multiple choice question)

68 % of respondents state that their

organisation introduced new ways of

working with disadvantaged young

people

73 % report that they implemented new

ways to reach out to people

Idem

Above

target

ICF survey

of sport

practitione

Results increasingly

used to enhance social

inclusion, equal

opportunities and sport

participation rates

50 %

(2017)

Percentage of

participants

(expressed as %

of Erasmus+

sport

organisations)

who have used

the results of

cross-border

projects to

enhance social

inclusion, equal

opportunities and

participation rates

53 % state the project focused on social

inclusion and equal opportunities

67 % state the project focused on

encouraging participation in physical

activities

(multiple choice question)

Source: ICF, tab. 6.3 and 6.13

141

kom (2018) 0050 - Ingen titel

Annex 6: Glossary

Term or acronym

Meaning or definition

Action

A strand or measure of the Erasmus+ Programme. Examples of

Actions are: Strategic Partnerships, Erasmus Mundus Joint Master

Degrees, Sectors Skills Alliances, etc.

A set of tasks carried out as part of a funded project. An activity

can be of different types (mobility activities, cooperation

activities, etc.). In the framework of Jean Monnet, an Activity is

equivalent to an Action (see definition above).

All forms of non-vocational adult education, whether of a formal,

non-formal or informal nature.

Any person who, having completed or is no longer involved in

initial education or training, returns to some forms of continuing

learning (formal, non-formal or informal), with the exception of

school and VET teachers/trainers.

participating organisation or informal group that submits grant

application. Applicants may apply either individually or on behalf

of a other organisations involved in the project. In the latter case,

the applicant is also defined as coordinator.

If the project is selected, the applicant becomes beneficiary of an

Erasmus+ grant. The beneficiary sings a grant agreement with –

or is notified of a grant decision by – the National or Executive

Agency that has selected the project. If the application was made

on behalf of other participating organisations, the partners may

become co-beneficiaries of the grant.

Europass is a portfolio of five different documents and an

electronic folder aiming to contain descriptions of the entire

holder's learning achievements, official qualifications, work

experience, skills and competences, acquired over time. Europass

also includes the European Skills Passport, a user-friendly

electronic folder that helps the holder to build up a personal,

modular inventory of his/her skills and qualifications. The aim of

Europass is to facilitate mobility and improve job and lifelong

learning prospects in Europe.

Education and training, youth, and sport.

The actions of Erasmus+ for education, training and youth include

Key Actions 1 (mobility for learners and practitioners), KA2

(cooperation among organisations) and KA3 (policy support).

Learning resulting from daily activities related to work, family or

leisure which is not organised or structured in terms of objectives,

time or learning support; it may be unintentional from the

142

Activity

Adult education

Adult learner

Applicant

Beneficiary

Europass

Fields

Key Action (KA)

Informal learning

kom (2018) 0050 - Ingen titel

learner's perspective.

Learners

Learners refer to all individuals involved in formal, non-formal

and informal education as pupils, students, apprentices,

volunteers, young people, etc.

Three levels are distinguished to evaluate the effects of the

programmes: individual level, divided into actions for learners

and practitioners, organisation level and system level.

All general education, vocational education and training, non-

formal learning and informal learning undertaken throughout life,

resulting in an improvement in knowledge, skills and

competences or participation in society within a personal, civic,

cultural, social and/or employment-related perspective, including

the provision of counselling and guidance services.

Level

intervention

Lifelong learning

Majority

/vast

"Majority of …" is used for [50 to 60%[ of the respondents

majority / most of

"Vast majority" for [56 to 75%[ ; "Most of…" for 75% and more

Non-formal learning

Learning which takes place through planned activities (in terms of

learning objectives and learning time) where some form of

learning support is present, but which is not part of the formal

education and training system.

In the context of Erasmus+ participants are considered those

individuals fully involved in a project and, in some cases,

receiving part of the European Union grant intended to cover their

costs of participation (notably travel and subsistence). Under

certain Actions of the Programme (i.e. Strategic Partnerships) a

distinction is hence to be made between this category of

participants (direct participants) and other individuals indirectly

involved in the project (e.g. target groups).

Any organisation or informal group of young people involved in

the implementation of an Erasmus+ project. Participating

organisations can be applicants or partners (also defined as co-

applicants, if they are identified at time of submission of the grant

application). If the project is granted, applicants become

beneficiaries and partners may become co-beneficiaries if the

project is financed through a multi-beneficiary grant.

Participating organisation involved in the project but not taking

the role of applicant.

countries which do not participate fully in the Erasmus+

Programme, but which may take part (as partners or applicants) in

certain Actions of the Programme.

Practitioners refer to those involved in formal and non-formal

education as teachers (including prospective teachers), trainers,

youth workers, educators, coaches, organisation leaders or staff.

143

Participants

Participating

organisation

Partner

(organisation)

Partner Countries

Practitioners

kom (2018) 0050 - Ingen titel

Programme

Countries

Sectors

EU and non EU countries (FY, IS, LI, NO, TR) that have

established a National Agency which participate fully in the

Erasmus+ Programme.

The education and training field is compounded of five sectors:

School education, vocational education and training, higher

education, international higher education and adult learning

Education and training which aims to equip people with

knowledge, know-how, skills and/or competences required in

particular occupations or more broadly on the labour market. For

the purpose of Erasmus+ projects focusing on initial or continuing

vocational education and training are eligible under VET actions.

The European tool to improve the recognition of the learning

outcomes of young people and youth workers from their

participation in projects supported by the Erasmus+ Programme.

Vocational

education and

training (VET)

Youthpass

144

kom (2018) 0050 - Ingen titel

Abbreviations



AE: Adult Education

DG EAC: Directorate General for Education and Culture

EACEA: Educational, Audiovisual & Culture Executive Agency

EaSI: Employment and Social Innovation

EC: European Commission

ECAS: European Commission Authentication System

ECHE: Erasmus Charter for Higher Education

ECTS: European Credit Transfer and Accumulation System

ECVET: European Credit System for Vocational Education and Training

EECS: European Economic and Social Committee

EIB: European Investment Bank

EIT: European Institute of Innovation and Technology

EMJMD: Erasmus Mundus Joint Master Degree

EP: European Parliament

EPALE: Electronic Platform for Adult Learning in Europe

EPRP: Erasmus+ Project Results Platform

EQAVET: European Quality Assurance in Vocational Education and Training

EQF: European Qualifications Framework

ERDF: European Regional Development Fund

ESF: European Social Fund

ESIF: European Structural & Investment Funds

E&T: Education and Training

ET 2020: Education and Training 2020 strategic framework for European cooperation

in education and training

ETF: European Training Foundation

EU: European Union

EUR: Euro

EVS: European Voluntary Service

EYE: Erasmus for Young Entrepreneurs

HE(I): Higher education (institution)

HERE: Higher Education Reform Experts

Horizon 2020: The EU Framework Programme for Research and Innovation

ICF: Consulting Service Ltd

145

kom (2018) 0050 - Ingen titel



ICT: Information and Communication Technology

INTERREG: European Territorial Cooperation (ETC), also known as Interreg

ISG: Interservice Group

JMO: Jean Monnet Operation

(subject of a stand-alone evaluation report)

KA1: Erasmus+ Key Action 1

KA2: Erasmus+ Key Action 2

KA3: Erasmus+ Key Action 3

KII: Key Informant Interview ("interview" in the text)

LLP: Lifelong Learning Programme

NA: National Agency

NGO: Non-governmental organisation

NRS: National Report Synthesis

(subject of a stand-alone evaluation report°

OECD: Organisation of Economic Cooperation and Development

OER: Open Educational Resources

OLS: Online Linguistic Support

OMC: Open Method of Coordination

OPC: Open Public Consultation

(subject of a stand-alone evaluation report)

SE: School Education

SLGF: Student [/Master] Loan Guarantee Facility

VET: Vocational Education and Training

WG: Working Group

YiA: Youth in Action

Add document

title 1

Add title 2

146