Snapshot

City

Paris

Country

France

Population

2,254,262

ERDF budget

EUR 4,995,793.16

Duration

01/11/2018 - 31/10/2021

Topic

Climate adaptation

About OASIS

pic
Credits to Ligue de l'enseignement

OASIS – Openness, Adaptation, Sensitisation, Innovation and Social Ties’ aimed to design and transform local urban areas in order to help the city of Paris to face the challenges of the 21st century. The project specifically set out to transform schoolyards in Paris into green oases accessible to both the school pupils and local communities. By bringing more greenery to the schoolyards, the project aimed to cool down the surrounding areas, making them more comfortable for children during their outdoor activities. It also strived to gather local communities (through citizen assemblies) around these newly accessible and pleasant areas, energising and empowering them to co-shape their environment, as well as raising awareness of climate change among the pupils. OASIS formed part of the Paris City Hall’s Resilience Strategy and fit well into the key challenges facing the metropolis: notably, anticipated warmer summers and more long, intense and frequent heatwaves, amplified by the phenomenon of Urban Heat Island (UHI). OASIS is also a part of the city’s broader aim to transform all the city’s public schoolyards by 2050.

The key components of the initiative consist of:

  1. The redesigning of ten selected pilot schoolyards and their physical transformation into spaces filled with greenery and water;
  2. Facilitation of community gatherings;
  3. Educational activities for children.

The physical transformation of the schoolyards takes advantage of innovative products, including materials (focusing on infiltration, thermal performance and a low carbon footprint), systems (for rainwater recovery, in particular) and nature-based solutions (for shading, evapotranspiration and drought resistance). OASIS also has a strong social cohesion and participatory element where end-users are invited to co-design the look of the new spaces.

The project implementation began in November 2018 and is anticipated to last until October 2021. This review took place in the second year of the project’s implementation when the ex-ante data collection was completed, while the ex-post stage was still being planned. Furthermore, both the project implementation and its evaluation were significantly impacted by the developing protests in Paris and the COVID-19 pandemic. Both reduced access to the schoolyards, the primary locations of the project.

This is a case study as part of an UIA report. You can access all of the project's resources on its project collection page.

Evaluation governance

Evaluation has been strongly embedded in the implementation of the OASIS project from the start and this has been considered an advantage. Being ‘in the middle of things’ allowed evaluators to feel at ease with those designing and implementing activities.

The evaluation didn’t only take place at the end, but throughout the process, which means it was an interactive evaluation that could lead to the adjustments.

Source: OASIS project hearing

 

pic
Credits to CAUE de Paris

Being part of the project team and the process provided evaluators with a good sense of what is feasible, in terms of suggesting changes and improvements, and helped with the design of activities with impact indicators in mind.

Evaluation has been managed by two entities; the Laboratory for Interdisciplinary Evaluation of Public Policies of the Sciences Po (LIEPP) for the behavioural aspects, and the Interdisciplinary Laboratory of Energies of Tomorrow (LIED), associated with ESIEE Paris, for the climate impact. The evaluation has also been strongly supported by Météo-France, the country’s key weather monitoring and forecasting entity. Météo-France was in charge of climate modelling and defining the climatic and micro-climatic evaluation protocol for quantifying the cooling effects created by the OASIS playgrounds at neighbourhood level.

The fact that the entities brought significant expertise in evaluating urban policies was considered important for the quality of the service. Importantly, LIED’s researchers have been involved in similar data collection activities at other sites in Paris, and this expertise facilitated the evaluation of OASIS. Evaluation governance has been characterised by clear definition of the responsibilities assigned to the involved entities and the scope of their engagement, as well as effective communication. This approach was helpful, for instance, when LIEPP worked on adjusting and fine-tuning its data collection instruments (questionaries for children). While LIEPP strived for simplicity of questions, its researchers consulted LIED and Météo-France to make sure that the instruments remained true to the scientific nature of the project.

Cooperation between the evaluation team and the project implementation stakeholders has been very active and enhanced learning loops for the project. The project implementing team was already able to efficiently use suggestions from the evaluators and quickly adjust activities as they evolved. For instance, thanks to monitoring inputs, the protocol initially developed for ensuring active participation of local community members was adjusted to better reflect the character of the groups.

Evaluation process

OASIS sets a blueprint for the evaluation of projects that tackle urban climate change by incorporating data collection on both environmental and human impact (social and climate impact assessment). It is precisely this relationship, looking at how the environmental and the social are interwoven in the urban area and how they depend on one other, that is worth exploring. From a broader policy perspective, the purpose of the evaluation is to develop a replicable OASIS framework, which could be further scaled up across the city.

The project represents a comprehensive evaluation approach built on two pillars: assessing the cooling effects of the newly designed schoolyards and evaluating impact the project has on children (their climate change awareness and social behaviour) and local communities (their active participatory approach to new spaces and levels of ownership). As such, this framework looks closely not only at how the physical environment changes due to the project, but also how these changes affect people. Furthermore, analysing how the educational activities increase children’s awareness about climate change can, in turn, give some indication for how they can engage with the physical environment in the future. The evaluation set out to explore how improved outdoor areas, offering a better temperature and quality of air as well as new sensory experiences (smells, textures, visuals), can impact children’s behaviour. Does it prompt cooperation instead of conflict? Does it encourage more collaborative and integrative patterns of engagement?

pic

The project opted for a mixed-methods approach, combining quantitative and qualitative data collection methods in the form of ex-ante and ex-post evaluations. Evaluation of the project’s impact on the people involved consists of analysing attitudes, interaction patterns and levels of ownership. Importantly, societal impact is assessed not only through methods based on declarations (in the case of this project, interviews and surveys), but also behaviour-based, ethnographic techniques (observation). If successful, such methodological triangulation should allow for accurate capturing of project results. Since one of the project’s key contributions was reclaiming schoolyards as communal spaces (contrary to the common practice of these premises being exclusively used by the school pupils and physically separated from the localities), the evaluation ecosystem included both school pupils and community members.

Mixed-methods approach

Text

Mixed-methods approach can be defined as “research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or a program of inquiry”.

Source: Tashakkori, A., Creswell, J.W., “Editorial: the new era of mixed methods”, J Mixed Methods Res 1: 3–7, 2007.

Triangulation

Text

Triangulation facilitates validation of data through cross verification from more than two sources. It tests the consistency of findings obtained through different instruments and increases the chance to control, or at least assess, some of the threats or multiple causes influencing our results. There are different types of triangulation: data (using different sources), investigator (involving different investigators), theory (using different theories for interpretation) and methodological (using different data collection methods).

Source: Better evaluation

Data collection followed the evaluation’s general two-pillar approach and, as such, LEID collected information about microclimatic and thermal changes through deployment of computerised weather sensors (Internet of Things). LIEEP was responsible for data collection on the potential impacts the project had on the people involved (children and community members) and key information was sourced through questionnaires and ethnographic observations, supported by qualitative interviews.

The first component, monitoring temperature and microclimate change, rested on a set of fixed and a set of mobile measurements. The climate diagnosis was designed to evaluate the reduction of the UHI effect and the increase in the levels of comfort experienced by citizens in the summer months by analysing and comparing measurements before/after the project transformation work on schoolyards took place. Instalment of mobile weather stations allowed for the development of a relative spatial layout of the microclimate. Complementarity of these two measurements allows for the better capturing of varying degrees of change in the environment.  Each of the ten project sites was equipped with two fixed weather measuring stations, one located in the playground and one in a nearby reference area (control stations) that would not experience any significant change during the project in terms of temperature, humidity and wind speed. Specifically, the improvement of summer comfort in the playgrounds is measured by the heat stress index, and in the buildings using temperature sensors integrated into the premises and modelling of the sites under heatwave conditions. Rainwater infiltration capacity is measured in the laboratory on different surfaces and will be characterised at playground level thanks to modelling of runoff and drainage phenomena. As a result, data collected through sensors and in the laboratories should allow for the creation of models and 3D simulations of the environmental impacts in the schoolyard premises, compared to areas not impacted by the project.

Co-design workshop credits to CAUE de Paris
Co-design workshop credits to CAUE de Paris

Human-centred evaluation progressed based on a protocol established during the project’s first year and is set out in the document as three main axes: the use of schoolyards, the project’s co-design approach, and pupils’ knowledge and attitudes on climate change.

Ten schools were selected for in-depth data collection on children’s behaviour and attitudes, although schools were covered by data collection to various degrees. Data collection on the human impact consisted of two core elements: surveys among the beneficiaries (pupils) in five schools and quantitative ethnographic observations of how pupils interacted in the schoolyards in six schools. Both sets of data collection instruments (questionnaires and ethnographic observation grids) were tested and fine-tuned before being rolled out in the selected schools.

Quantitative research

Qualitative research

Ex-ante questionnaire distributed in 5 schools 18 days of ex-ante observations in 6 schools

Internet of Things microclimate change measurements in five schools (through fixed and mobile measurement stations)

Ethnographic research through researchers’ participation in the design sessions and community meetings
  Qualitative interviews with teachers

The purpose of the questionnaire was to grasp changes in attitude among the children that might have occurred as a result of the educational and awareness raising activities conducted in the framework of the project. The multiple-choice ex-ante questionnaire was tailored to the contents of the workshops (e.g. awareness of environmental change, mastery of scientific notions, e.g. temperature). It was administered in a total of five classes, one per school.

While administration of the survey amongst older children was relatively straightforward, carrying out the same activity with the pre-school pupils (unable to read or write) required the development of adequate data collection techniques. The project evaluation team decided to employ two puppets and apply a simple puppet show scenario where the interviewer would make a claim about the environment and the two puppets would agree or disagree with what was being said. Children then had to indicate which of the two puppets they agreed with. Such adjustment allowed for interaction with the children, but also required careful preparation. It built on a lot of pre-testing (as well as previous studies), which included checking whether the colour of the puppets could influence the answers of children, and how to conduct the whole exercise without fostering any bias among the youngsters.

Our process of adapting surveys to all ages was successful. The pre-primary children looked at the interviewer as a ludicrativity and seemed to enjoy interaction a lot.

Source: OASIS project hearing

The second aspect of the evaluation, accounting for possible behavioural changes, was carried out through ethnographic observations in accordance with the observation grid. Data was collected on the use of yards and green spaces, cooperative/conflicting interactions between pupils in courtyards, gender segregation and patterns of socialising, and coded in accordance with the operational definition codebook which was developed for the evaluation. The data collection lasted one month and included a large sample of children in three age groups: three, five and six years old; between six and eleven years old; and teenagers between eleven and fifteen years old. Between June and September 2020, a total of 18 days of observation were completed in five schools, amounting to 90 hours of observation in total, and three days spent in each school (observations during morning breaks, lunch breaks and afternoon breaks). Data collectors then quantified pupils’ behaviour, categorising it as attendance, acts of violence or cooperation, and noting how space the was used and how the children interacted with the premises. The idea of the evaluation was to compare the quantitative data on the children’s patterns of behaviour from the ex-ante data collection (when pupils were interacting in the regular schoolyards) with data collected after the children had been exposed to the transformed, cooler and greener spaces. Naturally, final analysis will have to take changes in the children/young people’s behaviour arising simply from the fact that they are growing and maturing emotionally into account. Similarly, teens were characterised by significant non-participation levels in the survey data collection.

Since the project’s co-design aspect could not be evaluated using the ex-ante and ex-post approach, a more flexible observation grid was developed for data collection on the activities. 21 observations were conducted in three (primary and middle) schools during the co-design workshops, of which six were directed at adults and the rest directed at pupils.

Both project implementation and monitoring have been strongly impacted by widespread strikes occurring in Paris, and further by the COVID-19 pandemic. These events led to the temporary closure of schools and limited access to the schoolyards. As such, the community and cohesion building activities (citizen assemblies) had to be abandoned due to limitations on communal meetings. The evaluation team has been in the process of adjusting to these new conditions where new data collection methods are explored.

Ethnography

Text

“Ethnography is the study of social interactions, behaviours, and perceptions that occur within groups, teams, organisations, and communities. (…) The central aim of ethnography is to provide rich, holistic insights into people's views and actions, as well as the nature (that is, sights, sounds) of the location they inhabit, through the collection of detailed observations and interviews.”

Source: Reeves, S., Kuper, A., Hodges, B. (2008), “Qualitative research: Qualitative research methodologies: Ethnography,” BMJ (online), 337(7668):a1020, DOI:10.1136/bmj.a1020.

Horizontal issues

OASIS’s clear advantage comes from the fact that it is strongly embraced and incorporated into the Paris metropolitan policy, with significant chances for upscaling. The project is not the first instance where schoolyards have been transformed in the city, as over 30 schools were the sites for pilot schemes in 2018 and 2019, while fifteen more were set to be transformed in parallel to OASIS. The importance of OASIS in comparison to the other transformations comes precisely from its strong focus on learning and evaluation of impact. One of the main objectives for upscaling the UIA OASIS project is to develop a standardised cross-departmental process, from design to implementation, together with a set of guidelines and specifications for securing the high quality of the project.

pic

Given the project’s location within schools and the fact that it largely addresses minors as the key target group, a set of ethical issues was identified with regards to evaluation. Firstly, the question of informed consent to participation in data collection is problematic when it comes to children. The project evaluation team made all possible efforts to clearly convey the purpose and scope of the evaluation to school gatekeepers (principals), whose decision it then was to allow or prohibit in-depth evaluation of their school. Consequently, it was the schools’ principals who granted the evaluation team permission to question and observe the pupils. While the pupils were being observed during breaks and the time they spent in the school yards, they were never explicitly asked for consent. The pupils had the option to decline responding to the questionnaires (resulting in non-participation), but did not have a say in being observed by the data collectors during the ethnographic portion of the research. Whenever approached by children in the school yards, however, researchers gave an explanation of their presence on the school premises. Given the OASIS’s strong emphasis on participation, collective decision making and co-creation, it seems the evaluation framework could further benefit from addressing the issue of pupils as subjects rather than objects of data collection.

The evaluation team was also faced with ethical challenges in relation to involving pupils with learning disabilities in the data collection process. Here, decisions were made on individual case basis and in strong cooperation with the teachers supervising the pupils. This was a balancing act where priority was given to children’s wellbeing, whilst also attempting to ensure that they were allowed meaningful participation through the adjustment (simplification) of data collection instruments, when possible. Simultaneously, decisions were also made to exclude children from data collection if it was judged to be too cognitively challenging for them.

In pre-primary schools the issue was raised by some teachers disputing that a child could experience cognitive or language difficulties, in which case the interviewer negotiated with the teacher either the possibility not to invite a child to the interview or to have a simplified interview adapted to the child’s capacities, just to make sure the child wouldn’t feel excluded.

Source: OASIS project hearing

Lessons learnt

OASIS offers several good practices in terms of evaluation governance and approaches and data collection. Firstly, it is a case for the strong integration of evaluation experts into the project implementation team from the outset. Secondly, the project is a valuable example of how intervention with strong environmental and infrastructural components can effectively integrate and evaluate both physical (environmental) changes and the social aspects of a project. Lastly, careful preparation and implementation of data collection in the school environment, particularly among younger children, offers interesting approaches to data collection that can be replicated by other cities.

As such, the following lessons were identified by the project team:

  • Incorporation of the evaluators in the project design and implementation from the very start allows for shaping of the project cycle and activities with the end evaluation in mind. In the case of OASIS, such integration and early dialogue allowed for the building of rapport and trust between the evaluators and the partners directly implementing the activities. This further facilitated giving and receiving feedback. Additionally, being part of the project design phase enabled evaluators to enhance the shaping of activities with the data collection in mind.
  • Combining data collection on both the environmental and social impacts of the project ensures that the evaluation fully captures the possible effects of an intervention. In this regard, OASIS stands out as a project that has paid equal attention to monitoring of the microclimate changes produced by activities as it has to devoting resources for comprehensive quantitative and qualitative data collection on how the changed environment affects people. Only such a combination of data can reveal a full range of effects the project was able to achieve. This lesson is especially valuable for projects with strong environmental or material components, where the social impact can be easily overlooked.
  • Data collection in a school environment requires careful preparation, particularly if an intervention involves small children. OASIS offers a wealth of learning for other projects interested in data collection among young children, as it has developed and tested innovative survey techniques with the use of puppets. It proves that even children who cannot yet read or write can be meaningfully engaged in data collection, and their voices included and analysed. Young children are often overlooked in the evaluation of initiatives and OASIS proves that this does not have to be the case.
Transformation Work Maryse Hilsz School -credits CAUE de Paris
Transformation Work Maryse Hilsz School -credits CAUE de Paris

Review of the OASIS evaluation experience suggests that projects implemented within institutions (such as schools, in the case of OASIS) could benefit from approaching the establishments as systems, built out of various elements and groups. In this regard, OASIS evaluation focused first and foremost on how the intervention affected the pupils during the scope of their breaks between classes, specifically. A system approach to school could reveal interesting findings about how other school members (such as teachers and other staff) can also be impacted by transformed and friendlier school yards. Equally interesting could be the exploration of possible changes in pupils’ behaviour outside of the time spent directly in the school yards (during classes, for instance). Generally speaking, projects similar to OASIS in terms of their location within institutions should consider various elements of the systems in which they operate.

Text
school yard
Credits: CAUE 75

About this resource

Report
Location
Paris, France
About UIA
Urban Innovative Actions
Programme/Initiative
2014-2020

The Urban Innovative Actions (UIA) is a European Union initiative that provided funding to urban areas across Europe to test new and unproven solutions to urban challenges. The initiative had a total ERDF budget of €372 million for 2014-2020.

Go to profile
More content from UIA
1150 resources
See all

Similar content