Zoom-in #3. Evaluation and monitoring innovative environmental projects involving citizen participation: the DIAMS approach

Environmental projects involving citizen participation have the potential to provide a wide range of benefits that goes beyond the simple objectives of the project such as new environmental data collected; the potential in increasing public engagement; or in helping to detect rare environmental events, among others.

Environmental projects involving citizen participation have been popular in some areas such as biodiversity in non-marine environments, alien invasive species (Jordan et al., 2011), weather and climate (Sparrow et al., 2021); deposition of volcanic ashes (Stevenson et al., 2013); among others. Environmental projects with citizen participation on air quality monitoring increased in recent years across multiple cities worldwide (Mahajan et al., 2020; Oltra et al., 2017; Wesseling et al., 2019), in which the UIA DIAMS is indeed included. Most of these projects benefited from the availability of portable sensors, at a reduced cost compared to those used in scientific research projects, with the added benefit of their simplicity in their manipulation, without requiring a full scientific and technological expertise to operate.

Some of the most important opportunities and benefits of projects involving new environmental data as according to Blaney et al. (2016) include:

1. the increase of the spatial-temporal coverage of observations in areas without previous information

2. the increase of the cost effectiveness (or lower cost) compared to monitoring by professionals

3. the design of public engagement program around science and the environment in a practical and effective way.

However, citizen science projects are not free from problems and concerns, among which:

1. data collected using sensors might be of lower quality than that required by scientific research projects

2. lack of control in the monitoring, potentially leading to spatial bias, inadequate recording, and loss of authority in the results

3. lack of experience in citizen science projects by the scientific or technical leading group

4. need to council research scientist and social scientists for an effective public engagement project.

Another aspect to consider  in citizen science projects is the inclusiveness. Bonney et al. (2016) points that citizen science projects should try to reach a wider range of audiences and participants in order to truly contribute to the democratization of science. This includes gender, ethnicity, socio-economic and socio-cultural status, location, education level, alongside how these axes intersect to define hierarchies and power relations. Pateman et al. (2020) conducted a cross section survey of the UK population about the participation in environmental citizen science projects and concluded that men were more likely to participate than women; people identifying as from white ethnic groups, with particularly low participation by women from minority ethnic groups. Also, participation was highest amongst those in education (studying at school, college, or university) and lowest amongst the unemployed. The authors of this wide survey recommended that project leaders carefully consider the aims of projects and thus the  diversity of participants they wish to attract. 

Previous studies of air quality perception indicated there is not an association between the perceived air quality and the actual ambient concentrations (Brody et al., 2004; Oltra & Sala, 2018). General citizen’s knowledge about air quality is weak and the official online databases created by country’s Environmental Agency, which report the official air pollution levels, are not usually the main source of information for citizens (Canha et al., 2022). Usually, air pollution is perceived by sensory experience, awareness, knowledge, emotions (this latter usually linked to nuisance), communication and risk perception (Brody et al., 2004; Canha et al., 2022; Oltra & Sala, 2018). Generally, there is misconception about the sources which contribute the most to ambient concentrations (Maione et al., 2021). Education, age, place of residence or gender influence only to a small extent the perception of air pollution causes. This suggest that the lack of information and knowledge about the causes of air pollution is widespread across different socio-economic groups and countries (Maione et al., 2021). Some authors indicated that understanding people’s knowledge and response in front of poor air quality is key to design the best mitigation procedures to protect public health. Understanding the perception of citizens about air pollution is essential to induce a behavioural change in the society (Oltra & Sala, 2018). Some authors pointed that by increasing the knowledge about air quality, by making the problem into a tangible and personalized problem is the basis of the acceptance of future mitigation policies to improve air quality (Pantavou et al., 2017). Boosting education, training, public awareness and participation are some of the relevant actions for maximizing the opportunities to improve environmental pollution (UN, 2015).

Increasing the awareness of environmental issues through projects involving data collected by citizens has a clear benefit for local authorities. First, an educated public in environmental issues may be more likely to adopt pro-environmental behaviours and change their habits and activities that will limit the impact of their activities on the territory. Second, the acceptance and the adaption of new legislation and rules from local and regional authorities might be easier by citizens who have been informed about environmental threads. But benefits are not just one-sided. From the individual perspective, participating in environmental projects thought a community or interest group offers a number of benefits such as opportunities to socialize and enjoyment of a new activity. This latter was highlighted to be one of the main benefits from participants in environmental volunteer projects in England (Ockenden & Hutin, 2008). Also, the involvement in environmental projects might enforce the feeling of doing something worthwhile (i.e. ‘giving something back’) to society and nature; and increase the connection with nature and their local environment (Belluci et al., 2014). There are also a number of wider societal benefits, such as community-building and increased general environmental awareness amongst the population (Jordan et al. 2012).

Evaluation criteria are needed to inform both the funding body and the project management itself about the implementation process of the project and its impacts. Evaluation should assess the value of the innovative project for different outcomes and/or processes. This comprises a systematic assessment of both the effectiveness and efficiency of an activity or programme against a set of explicit or implicit standards and criteria. There are two aspects to evaluate according to Kieslinger et al. (2013):

1. Outcome-based evaluation, which assesses the overall goals of activities or programmes and the benefits to participants and recipients of the results and;

2. Process-based evaluation, which identifies the operational strengths and weaknesses of activities or programmes

Indicators quantify in a simpler manner the events, activities and help to communicate the complexities of the project. To be most meaningful, a monitoring and evaluation programme of a project should provide insights into cause-and-effect relationships between project activities and the anticipated impacts. Metrics can be quantitative and qualitative, and a combination of both is often best. Quantitative metrics put the fact and figures while qualitative metrics provides the anecdotal evidence.

Another aspect to consider in the evaluation program of innovative projects is the demographics of the project: who did the project reach? The demographics of the project quantifies the gender, the age, the socio-economic and socio-cultural class, the level of education, etc., that was reached (and responded) to the project. 

Environmental project with citizen participation are currently lacking of a comprehensive evaluation framework that allow comparability across projects and programmes (Kieslinger et al., 2013). Tredick et al. (2017) published a set of instructions and rules to evaluate citizen science projects based on the main elements found in the literature. These were grouped in six major areas: 1) Stakeholder collaboration and program resources; 2) Goals and objective; 3) Methods: Design and implementation of monitoring; 4) Data entry, storage, analysis, and synthesis; 5) Reporting and dissemination; and 6) Outcome assessment and program review. However, the methodology is still weak on the social implications of citizen science (Kieslinger et al., 2013). Moreover, there is a general trend that evaluation teams of citizen science projects continue to seek flexible evaluation strategies which adapt to specific project contexts (Schäfer & Kieslinger, 2016).

Some authors argue that evaluations are most successful when they are developed and implemented in partnership between the project lead in the evaluation project, the project implementers, and members of the public who are stakeholders in the program. Managing participation and improving facilitation by fostering good participation in planning, monitoring and evaluation it can be helpful to support empowerment, motivation and strengthen relationships among participants. This can support innovative projects and development approaches through social learning and adaptative management.  Others caution that evaluations should be conducted independent of the project participants to prevent subjectivity and bias in the process. For instance, Charnley & Engelbert (2005) argued that focus groups formed by community stakeholders and staff of the project led were extremely useful for helping the evaluation team to develop the evaluation criteria and to understand what constituted ‘success’ in relation to those criteria. However, sometime project staff can be over-extended with their existing responsibilities and despite being interested, sometimes they are unable to get involved in the evaluation process any more than necessary. Charnley & Engelbert (2005) suggested that dedicated staff should be involved full-time in the evaluation process.

Air pollution is the biggest environmental health threat in Europe and more and more people are acting claiming the right to breath clean air. In Europe, more than 95% of the urban population lives in areas that exceeds the limit in fine particles that the World Health Organization set to protect human health. Exposure to poor air is associated with cardiovascular and respiratory diseases, diabetes, dementia, among others, that lead to the loss of healthy years of life and, in the most serious cases, to premature deaths.

Initiatives involving the participation of citizens in new air quality data can produce useful information about local air quality. Such information can be used, for example, to improve official numerical models to better estimate air pollution levels and identify suitable actions to improve air quality.  Also, these initiatives often help to raise public awareness of air quality problems, in both the political and personal arena. On one hand, political awareness can lead to structural changes at the city level with wider and better cycle infrastructure or more affordable public transport fares. On the other hand, personal awareness can lead to behavioural change such as switching from driving to walking or cycling, which it can be in turn reinforced by public infrastructures. 

The DIAMS project led by the Aix Marseille Métropole aimed to increase the knowledge about air quality among all members of the society including young students and the education community, industrial partners and business, local authorities and public bodies and the general public on its broad sense. The tools used to empower society were broad and included workshops, talks, walks, etc., but all around the use of air quality data and numerical tools. The DIAMS online platform is the space where all air quality data generated in the territory, encompassing data from reference instruments - used to inform the European Commission - and that generated within the DIAMS framework are gathered in one single space and. New phone apps, numerical tools and models are generated around the DIAMS data to first represent the temporal-spatial distribution of air pollutants in the Métropole, to second foster the knowledge of what people is breathing to eventually identify measures and policies that might improve the quality of the air in the region. Projects like DIAMS are increasing and the availability of new data coupled with new data digitalisation approaches may represent a paradigm shift in the way that air quality is monitored.

The roadmap for the evaluation process of DIAMS is represented below. The starting point was the environmental challenge - elevated levels of pollutants due to emissions and meteorological conditions. The main goal of the project was empowering the society by providing better, more insightful and relevant information about air quality through the new approaches of portable sensors and new numerical tools. A set of engagement activities was planned. Monitoring these activities were undertaken on regular basis to better define (or re-define) them to improve its efficiency. Now, at the end of the project life-time, results of all activities and the overall project are evaluated and the achievement of the main project goal assessed to extract the lessons learnt for future projects.

The DIAMS project was led by the Aix-Marseille Métropole and the monitoring and the evaluation of the project was driven internally; members of the public and other stakeholder groups were not involved in the process. The monitoring and evaluation plan for DIAMS sits on the definition of quantitative and qualitative metrics which evaluate aspects of both the process (the how) and the outcome (what). Following the evaluation areas of Tredick et al. (2017), the metrics defined in each of the areas relevant to DIAMS are summarized in the table and an explanation of each of the areas is discussed below.

1. Stakeholder collaboration and program resources.      

DIAMS was composed of 8 partners since the beginning of the project and approached other 25 groups (including citizen associations, start-ups and service providers) to undertake and lead certain activities of the project. In terms of stakeholders, DIAMS worked with 91 classes reaching a total of 2,394 students which participate in the pedagogic program; 10,000 citizens worked with the data of portable sensors, though either individual initiative or through collective participation; 8 local authorities in the Métropole participated in DIAMS; and DIAMS also engaged with 11 industrial partners of which 8 belonged to the PIICTO association (an industrial and innovation platform in the Métropole).

All these figures confirm a large diversity of stakeholders, with good resources and trained partners that lead each of the engagement program or data exploitation of the project.

The main link between all stakeholders of the project was through the DIAMS online platform. Despite the innovative aspect of the online platform to link partners and stakeholders in one place, it might be seen as impersonal. And also, it risks to leave behind that part of the society without digital skills especially the older generation. Some citizen associations within the citizen engagement plan were linked through online messenger systems and that was seen as a way to foster the engagement and the level of participation compared to those citizens which worked individually. That confirms that the success of participation of citizens in this type of initiatives benefits when structured communication links exists.

The UIA funds for DIAMS secured the financial sources through the DIAMS life-time with high level of recruitment and stability throughout the project. However, as the project reaches its final point, funds to secure the functioning of the online platform and the data infrastructure should be still secured. This is one of the main weak points of environmental projects with a final end date.

2. Goals and objectives.

The goal of DIAMS is clearly defined in the proposal, and the link between the project activities and the overall goal was clearly articulated in the different workpackages of the project.

The objectives meet all SMART criteria and are prioritized on the basis of their contribution to the project mission.

3.  Methods: Design and implementation of monitoring.

Sampling protocols and data measured by sensors (either by citizens, schools or local authorities) were done following standard methods and state-of-the-art best practices. Portable sensors were compared to reference instruments in the fixed monitoring stations that official monitoring has in the territory. The consortium published in the DIAMS online platform tutorials to use the sensors, a video and a FAQs section to help the citizen the use of portable sensors. Fortnightly webinars were also schedule for hands-on sensor training.

The data analysis approach to explore the data collected in the project was undertaken by providers of data services. A plan for statistical analyses was not fully defined at the beginning of the project but innovative and state-of-the-art statistical methods were applied as the data started to be collected and available.

One of the most important factors in DIAMS is to evaluate how the data collected by the different stakeholders actually responded to the better and more insightful information about air quality was given to the society. A proper evaluation plan and data analysis of this was lacking at the beginning of the project and was lately designed towards the end of the project.

4. Data entry, storage, analysis, and synthesis.

Data collected within the project is highly organized and well managed in the data repository created for the project. Access to the data is by Application Programming Interface (API), csv data or through geojson data files, and access is granted beforehand. The data platform of DIAMS, including data and services developped around data is well structured and have a clear structure as shown in Figure 2.

Main metadata for each of the type sensors used in the project are available from the DIAMS online platform. It states the detection limit, the precision of the instrument and the range of concentrations (or values) measured. Any information about co-location and calibration procedures are however lacking.

With DIAMS, an increase of the data points with air pollution information collected in the territory. The postal service equipped their electric cars and vans with 300 portable sensors which logged around 40 million points. Fixed reference instruments log 8,760 hourly points in a year. The postal service mapping was equivalent to more than 4,500 instruments monitoring hour data for one year. Data was also used in 3D modelling exercises using fixed observations by either local authorities or industrial premises. Three-dimensional modelling enables the quantification of the possible impact on air pollution associated with the application of a policy before its real implementation. The number of fixed points in the territory with air pollution information increased with DIAMS with 50 monitoring sites deployed at 8 local authorities.

5. Reporting and dissemination

Much of the overall communication strategy in DIAMS was organized by the associations in charge to design and implement workshops and ateliers around the topic of air quality. A total of eight different associations developed the animations through service calls.

Some of the results associated with DIAMS activities were communicated to stakeholders in the form of webinars. Some of them caught people’s interests and were well attended. Other ways of communicating results of DIAMS activities were done through social media, restitution meetings and the final DIAMS event.

6. Outcome assessment and program review.

Assessment of the efficacy of the communication strategy with stakeholder uptake is not fully accounted for the ensemble of activities of the DIAMS framework. The main goal for DIAMS was to increase the knowledge about air population to the ensemble of the population. DIAMS reached a large number of different members of the society through its vast engagement plan and as well through the online platform. However, there is a lack of systematic evaluation of each activity to see if those were translated into proper changes in personal behaviour and/or new environmental policies and environmental measures to improve air quality.

One example developed within DIAMS that was effectively transpose into action came from the results from the three-dimensional numerical modelling developed for the Great Port of Marseille. A bespoken numerical model was developed to improve the manoeuvre within the port area to decrease the fuel consumption to eventually decrease atmospheric emission rates. Results from the modelling exercise was then adopted by the Great Port of Marseille.

One of the obstacles that make difficult the increase of citizen-driven environmental measures was the young age of the Aix Marseille Métropole authority. The UIA DIAMS project started when the Aix Marseille Métropole was created and the mechanisms to increase citizen participation in the design of local and regional policies about quality were not in place. For that, the local authority needs to have some maturity that Aix Marseille did not have at the moment.

DIAMS has certainly opened up the need for society engagement in the design, implementation and applicability of environmental measures in the Aix Marseille territory. However, for that, there is a need of opening-up from the political level. DIAMS has put the seeds for that and further acceptance from the political side is needed.

One interesting point to evaluate is if the new online platform with data and services developed for the interest of the territory has led to an increase of the interest of citizens to check the quality of the air they breathe. As highlighted by previous studies, general citizen’s knowledge about air quality is weak and the official online databases created by country’s Environmental Agency which report the official air pollution levels are not usually the main source of information for citizens (Canha et al., 2022). Usually, perception of air quality is perceived by sensory experience, awareness, emotions (usually linked to nuisance), communication and risk perception (Brody et al., 2004; Canha et al., 2022; Oltra & Sala, 2018). A way to quantify these will be asking the participants if there was a change in the way they were getting information about air quality.

DIAMS enabled the testing and demonstration of the applicability of an innovative practice based on the use of portable sensors and numerical models to reinforce the knowledge about air quality problems to all members of the society. The engagement project encompassed different stakeholders and approaches. Numerical modelling was used to developed and implement environmental measures to reduce atmospheric emissions from anthropogenic activities.

Evaluation the impact of environmental projects with large citizen participation requires of thoughtful design and planning involving the scientists, technologists, social scientists, policy markers and regulators and the stakeholders themselves.

1.  It might be a good idea to develop the evaluation program of the project in conjunction with the partners and the main stakeholders, as it is more likely about the success of the project. Specially in environmental project with citizen engagement, it might be good to involve citizens and main stakeholders in the definition of the evaluation metrics.
2. All activities developed within the framework of a project should be accompanied by the expected impacts and how they will be evaluated. When writing and deposing a project it is difficult to conceptualize which will be the useful metrics that should quantify the impact of all workpackages and activities. However, as the project shapes and the actions are concretized, those should be accompanied by a definition of the possible impacts and how they are going to be evaluated.
3. As recommended by the literature, project leaders are encourage to publish the demographics of the project (who participated, which gender, age range, socio-economic and socio-cultural level, ethnicity, etc.). Monitoring the demographics of the project ensures the inclusiveness of the project,  reaching all members of the society, whatever their social status, sociocultural origin, gender, literacy level, age, etc.

Blaney, R. J. P., Philippe, A. C. V., Pocock, M. J. O., & Jones, G. D. (2016). Citizen Science and Environmental Monitoring: Towards a Methodology for Evaluating Opportunities, Costs and Benefits. Citizen Science and Environmental Monitoring-Final Report. http://www.ukeof.org.uk/resources/citizen-science-resources/Costbenefitcitizenscience.pdf

Bonney, R., Phillips, T. B., Ballard, H. L., & Enck, J. W. (2016). Can citizen science enhance public understanding of science? Public Understanding of Science, 25(1), 2–16. https://doi.org/10.1177/0963662515607406.

Brody, S. D., Peck, B. M., & Highfield, W. E. (2004). Examining localized patterns of air quality perception in Texas: A spatial and statistical analysis. Risk Analysis, 24(6), 1561–1574. https://doi.org/10.1111/j.0272-4332.2004.00550.x

Canha, N., Justino, A. R., Gamelas, C. A., & Almeida, S. M. (2022). Citizens’ Perception on Air Quality in Portugal—How Concern Motivates Awareness. International Journal of Environmental Research and Public Health, 19(19). https://doi.org/10.3390/ijerph191912760

Charnley, S., & Engelbert, B. (2005). Evaluating public participation in environmental decision-making : EPA ’ s superfund community involvement program. 77, 165–182. https://doi.org/10.1016/j.jenvman.2005.04.002

Jordan, R. C., Gray, S. A., Howe, D. V., Brooks, W. R., & Ehrenfeld, J. G. (2011). Knowledge Gain and Behavioral Change in Citizen-Science Programs. Conservation Biology, 25(6), 1148–1154. https://doi.org/10.1111/j.1523-1739.2011.01745.x

Kieslinger, B., Schäfer, T., Heigl, F., Dörler, D., Richter, A., & Bonn, A. (2013). Evaluating citizen science : Towards an open framework. In H. Susanne, M. Haklay, A. Bowser, Z. Makuch, J. Vogel, & A. Bonn (Eds.), Citizen Science. Innovation in Open Science, Society and Policy. UCL Press. https://www.jstor.org/stable/j.ctv550cf2.13

Mahajan, S., Kumar, P., Pinto, J. A., Riccetti, A., Schaaf, K., Camprodon, G., Smári, V., Passani, A., & Forino, G. (2020). A citizen science approach for enhancing public understanding of air pollution. Sustainable Cities and Society, 52(June 2019), 101800. https://doi.org/10.1016/j.scs.2019.101800

Maione, M., Mocca, E., Eisfeld, K., Kazepov, Y., & Fuzzi, S. (2021). Public perception of air pollution sources across Europe. Ambio, 50(6), 1150–1158. https://doi.org/10.1007/s13280-020-01450-5

Ockenden, N., & Hutin, M. (2008). Volunteering to lead: a study of leadership in small, volunteer-led groups. In Institute for Volunteering Research (Issue March): https://www.bl.uk/britishlibrary/~/media/bl/global/social-welfare/pdfs/non-secure/v/o/l/volunteering-to-lead-study-of-leadership-volunteer-led-groups.pdf

Oltra, C., & Sala, R. (2018). Perception of risk from air pollution and reported behaviors: a cross-sectional survey study in four cities. Journal of Risk Research, 21(7), 869–884. https://doi.org/10.1080/13669877.2016.1264446

Oltra, C., Sala, R., Boso, À., & Asensio, S. L. (2017). Public engagement on urban air pollution: an exploratory study of two interventions. Environmental Monitoring and Assessment, 189(6). https://doi.org/10.1007/s10661-017-6011-6

Pantavou, K., Lykoudis, S., & Psiloglou, B. (2017). Air quality perception of pedestrians in an urban outdoor Mediterranean environment: A field survey approach. Science of the Total Environment, 574, 663–670. https://doi.org/10.1016/j.scitotenv.2016.09.090

Pateman, R., Dyke, A. & West, S. 2021. The Diversity of Participants in Environmental Citizen Science. Citizen Science: Theory and Practice, 6(1): 9, pp. 1–16. DOI: https://doi.org/10.5334/cstp.369

Schäfer, T., & Kieslinger, B. (2016). Supporting emerging forms of citizen science: A plea for diversity, creativity and social innovation. Journal of Science Communication, 15(2), 1–12. https://doi.org/10.22323/2.15020402

Sparrow, S., Bowery, A., Carver, G. D., Köhler, M. O., Ollinaho, P., Pappenberger, F., Wallom, D., & Weisheimer, A. (2021). OpenIFS@home version 1: A citizen science project for ensemble weather and climate forecasting. Geoscientific Model Development, 14(6), 3473–3486. https://doi.org/10.5194/gmd-14-3473-2021

Stevenson, J., Loughlin, S. C., Font, A., Fuller, G. W., MacLeod, A., Oliver, I. W., Jackson, B., Horwell, C. J., Thordarson, T., & Dawson, I. (2013). UK monitoring and deposition of tephra from the May 2011 eruption of Grímsvötn, Iceland. Journal of Applied Volcanology, 2(May 2011), 3. https://doi.org/10.1186/2191-5040-2-3

Tredick, C. A., Lewison, R. L., Deutschman, D. H., Hunt, T. A., Gordon, K. L., & Von Hendy, P. (2017). A Rubric to Evaluate Citizen-Science Programs for Long-Term Ecological Monitoring. BioScience, 67(9), 834–844. https://doi.org/10.1093/biosci/bix090

UN. (2015). The paris agreement. Towards a Climate-Neutral Europe: Curbing the Trend. https://doi.org/10.4324/9789276082569-2

Wesseling, J., de Ruiter, H., Blokhuis, C., Drukker, D., Weijers, E., Volten, H., Vonk, J., Gast, L., Voogt, M., Zandveld, P., van Ratingen, S., & Tielemans, E. (2019). Development and implementation of a platform for public information on air quality, sensor measurements, and citizen science. Atmosphere, 10(8). https://doi.org/10.3390/atmos10080445