Evaluating Innovative Projects
Monitoring and evaluating the impact of projects is essential to determine the success in implementing the action plan, achieving the objectives, and making an impact to the society. Monitoring the project along its implementation is encouraged to revise and reinforce the main objectives of the project; and reinforce the implementation plan in a more effective way. A final evaluation of the project set the basis to extract the lessons from it and build the basis for future successful projects.

Monitoring and evaluation process can be done in a quantitative manner for milestones that can be easily counted: number of participants engaged in a project; number of deployed activities, etc. However, when evaluating environmental projects with the focus on citizens (on its wide context, comprising all members of the society such as local authorities, industrial partners,  and general public, just to cite some of the actors) one of the most relevant impact to be evaluated falls on the person itself : “Did the project promote a better knowledge about the environmental problem?”; “Did the project promote a change of behaviour in citizens for more environmental conscious actions?”. Evaluating the knowledge acquired by citizens, the change of misconceptions of environmental issues and the change of behavioural change is not simple using quantitative evaluation indexes. Social science approaches are then more appropriate. 

Environmental projects and programmes which include the participation of citizens should be evaluated along three dimensions: (i) the knowledge created, (ii) the learning process and the empowerment of participants and (iii) the impact for wider society (Kieslinger et al., 2013).

This Zoom-In document revise the basis of monitoring and evaluation of innovative projects focused on environmental challenges and which comprise the engagement of citizens. The evaluation program of the UIA DIAMS project is discussed and some hints for future projects are given.

Key vocabulary

Monitoring : is the systematic collection and analysis of information as a project progresses and advances on its implementation. Monitoring a project aims to improve the efficiency and effectiveness of a project.
Evaluation : is the comparison of actual project impacts against the agreed strategic plans. It looks at what it was out to do, what it has been have accomplished, and how it has been accomplished.
Learning : is the reflection on lessons learnt from the project delivery to guide and enable continuous improvement for future project delivery.
Efficiency : measures if the input into the work is appropriate in terms of the output.
Effectiveness :  is the measure of the extent to which a project achieves the specific objectives it set.
Impact : measures how the project made a difference to the problem it was aiming to address.

Environmental projects involving citizen participation have the potential to provide a wide range of benefits that goes beyond the simple objectives of the project such as new environmental data collected; the potential in increasing public engagement; or in helping to detect rare environmental events, among others.

Environmental projects involving citizen participation have been popular in some areas such as biodiversity in non-marine environments, alien invasive species (Jordan et al., 2011), weather and climate (Sparrow et al., 2021); deposition of volcanic ashes (Stevenson et al., 2013); among others. Environmental projects with citizen participation on air quality monitoring increased in recent years across multiple cities worldwide (Mahajan et al., 2020; Oltra et al., 2017; Wesseling et al., 2019), in which the UIA DIAMS is indeed included. Most of these projects benefited from the availability of portable sensors, at a reduced cost compared to those used in scientific research projects, with the added benefit of their simplicity in their manipulation, without requiring a full scientific and technological expertise to operate.

Some of the most important opportunities and benefits of projects involving new environmental data as according to Blaney et al. (2016) include:

  1. the increase of the spatial-temporal coverage of observations in areas without previous information

  2. the increase of the cost effectiveness (or lower cost) compared to monitoring by professionals

  3. the design of public engagement program around science and the environment in a practical and effective way.

However, citizen science projects are not free from problems and concerns, among which:

  1. data collected using sensors might be of lower quality than that required by scientific research projects

  2. lack of control in the monitoring, potentially leading to spatial bias, inadequate recording, and loss of authority in the results

  3. lack of experience in citizen science projects by the scientific or technical leading group

  4. need to council research scientist and social scientists for an effective public engagement project.

Another aspect to consider  in citizen science projects is the inclusiveness. Bonney et al. (2016) points that citizen science projects should try to reach a wider range of audiences and participants in order to truly contribute to the democratization of science. This includes gender, ethnicity, socio-economic and socio-cultural status, location, education level, alongside how these axes intersect to define hierarchies and power relations. Pateman et al. (2020) conducted a cross section survey of the UK population about the participation in environmental citizen science projects and concluded that men were more likely to participate than women; people identifying as from white ethnic groups, with particularly low participation by women from minority ethnic groups. Also, participation was highest amongst those in education (studying at school, college, or university) and lowest amongst the unemployed. The authors of this wide survey recommended that project leaders carefully consider the aims of projects and thus the  diversity of participants they wish to attract. 

Previous studies of air quality perception indicated there is not an association between the perceived air quality and the actual ambient concentrations (Brody et al., 2004; Oltra & Sala, 2018). General citizen’s knowledge about air quality is weak and the official online databases created by country’s Environmental Agency, which report the official air pollution levels, are not usually the main source of information for citizens (Canha et al., 2022). Usually, air pollution is perceived by sensory experience, awareness, knowledge, emotions (this latter usually linked to nuisance), communication and risk perception (Brody et al., 2004; Canha et al., 2022; Oltra & Sala, 2018). Generally, there is misconception about the sources which contribute the most to ambient concentrations (Maione et al., 2021). Education, age, place of residence or gender influence only to a small extent the perception of air pollution causes. This suggest that the lack of information and knowledge about the causes of air pollution is widespread across different socio-economic groups and countries (Maione et al., 2021). Some authors indicated that understanding people’s knowledge and response in front of poor air quality is key to design the best mitigation procedures to protect public health. Understanding the perception of citizens about air pollution is essential to induce a behavioural change in the society (Oltra & Sala, 2018). Some authors pointed that by increasing the knowledge about air quality, by making the problem into a tangible and personalized problem is the basis of the acceptance of future mitigation policies to improve air quality (Pantavou et al., 2017). Boosting education, training, public awareness and participation are some of the relevant actions for maximizing the opportunities to improve environmental pollution (UN, 2015).

Increasing the awareness of environmental issues through projects involving data collected by citizens has a clear benefit for local authorities. First, an educated public in environmental issues may be more likely to adopt pro-environmental behaviours and change their habits and activities that will limit the impact of their activities on the territory. Second, the acceptance and the adaption of new legislation and rules from local and regional authorities might be easier by citizens who have been informed about environmental threads. But benefits are not just one-sided. From the individual perspective, participating in environmental projects thought a community or interest group offers a number of benefits such as opportunities to socialize and enjoyment of a new activity. This latter was highlighted to be one of the main benefits from participants in environmental volunteer projects in England (Ockenden & Hutin, 2008). Also, the involvement in environmental projects might enforce the feeling of doing something worthwhile (i.e. ‘giving something back’) to society and nature; and increase the connection with nature and their local environment (Belluci et al., 2014). There are also a number of wider societal benefits, such as community-building and increased general environmental awareness amongst the population (Jordan et al. 2012).

Evaluation criteria are needed to inform both the funding body and the project management itself about the implementation process of the project and its impacts. Evaluation should assess the value of the innovative project for different outcomes and/or processes. This comprises a systematic assessment of both the effectiveness and efficiency of an activity or programme against a set of explicit or implicit standards and criteria. There are two aspects to evaluate according to Kieslinger et al. (2013):

  1. Outcome-based evaluation, which assesses the overall goals of activities or programmes and the benefits to participants and recipients of the results and;

  2. Process-based evaluation, which identifies the operational strengths and weaknesses of activities or programmes

Indicators quantify in a simpler manner the events, activities and help to communicate the complexities of the project. To be most meaningful, a monitoring and evaluation programme of a project should provide insights into cause-and-effect relationships between project activities and the anticipated impacts. Metrics can be quantitative and qualitative, and a combination of both is often best. Quantitative metrics put the fact and figures while qualitative metrics provides the anecdotal evidence.

Another aspect to consider in the evaluation program of innovative projects is the demographics of the project: who did the project reach? The demographics of the project quantifies the gender, the age, the socio-economic and socio-cultural class, the level of education, etc., that was reached (and responded) to the project. 

Environmental project with citizen participation are currently lacking of a comprehensive evaluation framework that allow comparability across projects and programmes (Kieslinger et al., 2013). Tredick et al. (2017) published a set of instructions and rules to evaluate citizen science projects based on the main elements found in the literature. These were grouped in six major areas: 1) Stakeholder collaboration and program resources; 2) Goals and objective; 3) Methods: Design and implementation of monitoring; 4) Data entry, storage, analysis, and synthesis; 5) Reporting and dissemination; and 6) Outcome assessment and program review. However, the methodology is still weak on the social implications of citizen science (Kieslinger et al., 2013). Moreover, there is a general trend that evaluation teams of citizen science projects continue to seek flexible evaluation strategies which adapt to specific project contexts (Schäfer & Kieslinger, 2016).

Some authors argue that evaluations are most successful when they are developed and implemented in partnership between the project lead in the evaluation project, the project implementers, and members of the public who are stakeholders in the program. Managing participation and improving facilitation by fostering good participation in planning, monitoring and evaluation it can be helpful to support empowerment, motivation and strengthen relationships among participants. This can support innovative projects and development approaches through social learning and adaptative management.  Others caution that evaluations should be conducted independent of the project participants to prevent subjectivity and bias in the process. For instance, Charnley & Engelbert (2005) argued that focus groups formed by community stakeholders and staff of the project led were extremely useful for helping the evaluation team to develop the evaluation criteria and to understand what constituted ‘success’ in relation to those criteria. However, sometime project staff can be over-extended with their existing responsibilities and despite being interested, sometimes they are unable to get involved in the evaluation process any more than necessary. Charnley & Engelbert (2005) suggested that dedicated staff should be involved full-time in the evaluation process.

Air pollution is the biggest environmental health threat in Europe and more and more people are acting claiming the right to breath clean air. In Europe, more than 95% of the urban population lives in areas that exceeds the limit in fine particles that the World Health Organization set to protect human health. Exposure to poor air is associated with cardiovascular and respiratory diseases, diabetes, dementia, among others, that lead to the loss of healthy years of life and, in the most serious cases, to premature deaths.

Initiatives involving the participation of citizens in new air quality data can produce useful information about local air quality. Such information can be used, for example, to improve official numerical models to better estimate air pollution levels and identify suitable actions to improve air quality.  Also, these initiatives often help to raise public awareness of air quality problems, in both the political and personal arena. On one hand, political awareness can lead to structural changes at the city level with wider and better cycle infrastructure or more affordable public transport fares. On the other hand, personal awareness can lead to behavioural change such as switching from driving to walking or cycling, which it can be in turn reinforced by public infrastructures. 

The DIAMS project led by the Aix Marseille Métropole aimed to increase the knowledge about air quality among all members of the society including young students and the education community, industrial partners and business, local authorities and public bodies and the general public on its broad sense. The tools used to empower society were broad and included workshops, talks, walks, etc., but all around the use of air quality data and numerical tools. The DIAMS online platform is the space where all air quality data generated in the territory, encompassing data from reference instruments - used to inform the European Commission - and that generated within the DIAMS framework are gathered in one single space and. New phone apps, numerical tools and models are generated around the DIAMS data to first represent the temporal-spatial distribution of air pollutants in the Métropole, to second foster the knowledge of what people is breathing to eventually identify measures and policies that might improve the quality of the air in the region. Projects like DIAMS are increasing and the availability of new data coupled with new data digitalisation approaches may represent a paradigm shift in the way that air quality is monitored.

The roadmap for the evaluation process of DIAMS is represented below. The starting point was the environmental challenge - elevated levels of pollutants due to emissions and meteorological conditions. The main goal of the project was empowering the society by providing better, more insightful and relevant information about air quality through the new approaches of portable sensors and new numerical tools. A set of engagement activities was planned. Monitoring these activities were undertaken on regular basis to better define (or re-define) them to improve its efficiency. Now, at the end of the project life-time, results of all activities and the overall project are evaluated and the achievement of the main project goal assessed to extract the lessons learnt for future projects.

The roadmap of DIAMS: from the environmental challenge to the lessons learnt and legacy from the innovative project.
Figure 1. The roadmap of DIAMS: from the environmental challenge to the lessons learnt and legacy from the innovative project.

The DIAMS project was led by the Aix-Marseille Métropole and the monitoring and the evaluation of the project was driven internally; members of the public and other stakeholder groups were not involved in the process. The monitoring and evaluation plan for DIAMS sits on the definition of quantitative and qualitative metrics which evaluate aspects of both the process (the how) and the outcome (what). Following the evaluation areas of Tredick et al. (2017), the metrics defined in each of the areas relevant to DIAMS are summarized in the table and an explanation of each of the areas is discussed below.

1. Stakeholder collaboration and program resources.      

DIAMS was composed of 8 partners since the beginning of the project and approached other 25 groups (including citizen associations, start-ups and service providers) to undertake and lead certain activities of the project. In terms of stakeholders, DIAMS worked with 91 classes reaching a total of 2,394 students which participate in the pedagogic program; 10,000 citizens worked with the data of portable sensors, though either individual initiative or through collective participation; 8 local authorities in the Métropole participated in DIAMS; and DIAMS also engaged with 11 industrial partners of which 8 belonged to the PIICTO association (an industrial and innovation platform in the Métropole).

All these figures confirm a large diversity of stakeholders, with good resources and trained partners that lead each of the engagement program or data exploitation of the project.

The main link between all stakeholders of the project was through the DIAMS online platform. Despite the innovative aspect of the online platform to link partners and stakeholders in one place, it might be seen as impersonal. And also, it risks to leave behind that part of the society without digital skills especially the older generation. Some citizen associations within the citizen engagement plan were linked through online messenger systems and that was seen as a way to foster the engagement and the level of participation compared to those citizens which worked individually. That confirms that the success of participation of citizens in this type of initiatives benefits when structured communication links exists.

The UIA funds for DIAMS secured the financial sources through the DIAMS life-time with high level of recruitment and stability throughout the project. However, as the project reaches its final point, funds to secure the functioning of the online platform and the data infrastructure should be still secured. This is one of the main weak points of environmental projects with a final end date.

2. Goals and objectives.

The goal of DIAMS is clearly defined in the proposal, and the link between the project activities and the overall goal was clearly articulated in the different workpackages of the project.

The objectives meet all SMART criteria and are prioritized on the basis of their contribution to the project mission.

3.  Methods: Design and implementation of monitoring.

Sampling protocols and data measured by sensors (either by citizens, schools or local authorities) were done following standard methods and state-of-the-art best practices. Portable sensors were compared to reference instruments in the fixed monitoring stations that official monitoring has in the territory. The consortium published in the DIAMS online platform tutorials to use the sensors, a video and a FAQs section to help the citizen the use of portable sensors. Fortnightly webinars were also schedule for hands-on sensor training.

The data analysis approach to explore the data collected in the project was undertaken by providers of data services. A plan for statistical analyses was not fully defined at the beginning of the project but innovative and state-of-the-art statistical methods were applied as the data started to be collected and available.

One of the most important factors in DIAMS is to evaluate how the data collected by the different stakeholders actually responded to the better and more insightful information about air quality was given to the society. A proper evaluation plan and data analysis of this was lacking at the beginning of the project and was lately designed towards the end of the project.

4. Data entry, storage, analysis, and synthesis.

Data collected within the project is highly organized and well managed in the data repository created for the project. Access to the data is by Application Programming Interface (API), csv data or through geojson data files, and access is granted beforehand. The data platform of DIAMS, including data and services developped around data is well structured and have a clear structure as shown in Figure 2.

The data, service, usage and users scheme developed for DIAMS
Figure 2. The data, service, usage and users scheme developed for DIAMS. More details in https://www.airdiams.eu

Main metadata for each of the type sensors used in the project are available from the DIAMS online platform. It states the detection limit, the precision of the instrument and the range of concentrations (or values) measured. Any information about co-location and calibration procedures are however lacking.

With DIAMS, an increase of the data points with air pollution information collected in the territory. The postal service equipped their electric cars and vans with 300 portable sensors which logged around 40 million points. Fixed reference instruments log 8,760 hourly points in a year. The postal service mapping was equivalent to more than 4,500 instruments monitoring hour data for one year. Data was also used in 3D modelling exercises using fixed observations by either local authorities or industrial premises. Three-dimensional modelling enables the quantification of the possible impact on air pollution associated with the application of a policy before its real implementation. The number of fixed points in the territory with air pollution information increased with DIAMS with 50 monitoring sites deployed at 8 local authorities.

5. Reporting and dissemination

Much of the overall communication strategy in DIAMS was organized by the associations in charge to design and implement workshops and ateliers around the topic of air quality. A total of eight different associations developed the animations through service calls.

Some of the results associated with DIAMS activities were communicated to stakeholders in the form of webinars. Some of them caught people’s interests and were well attended. Other ways of communicating results of DIAMS activities were done through social media, restitution meetings and the final DIAMS event.

6. Outcome assessment and program review.

Assessment of the efficacy of the communication strategy with stakeholder uptake is not fully accounted for the ensemble of activities of the DIAMS framework. The main goal for DIAMS was to increase the knowledge about air population to the ensemble of the population. DIAMS reached a large number of different members of the society through its vast engagement plan and as well through the online platform. However, there is a lack of systematic evaluation of each activity to see if those were translated into proper changes in personal behaviour and/or new environmental policies and environmental measures to improve air quality.

One example developed within DIAMS that was effectively transpose into action came from the results from the three-dimensional numerical modelling developed for the Great Port of Marseille. A bespoken numerical model was developed to improve the manoeuvre within the port area to decrease the fuel consumption to eventually decrease atmospheric emission rates. Results from the modelling exercise was then adopted by the Great Port of Marseille.

One of the obstacles that make difficult the increase of citizen-driven environmental measures was the young age of the Aix Marseille Métropole authority. The UIA DIAMS project started when the Aix Marseille Métropole was created and the mechanisms to increase citizen participation in the design of local and regional policies about quality were not in place. For that, the local authority needs to have some maturity that Aix Marseille did not have at the moment.

DIAMS has certainly opened up the need for society engagement in the design, implementation and applicability of environmental measures in the Aix Marseille territory. However, for that, there is a need of opening-up from the political level. DIAMS has put the seeds for that and further acceptance from the political side is needed.

One interesting point to evaluate is if the new online platform with data and services developed for the interest of the territory has led to an increase of the interest of citizens to check the quality of the air they breathe. As highlighted by previous studies, general citizen’s knowledge about air quality is weak and the official online databases created by country’s Environmental Agency which report the official air pollution levels are not usually the main source of information for citizens (Canha et al., 2022). Usually, perception of air quality is perceived by sensory experience, awareness, emotions (usually linked to nuisance), communication and risk perception (Brody et al., 2004; Canha et al., 2022; Oltra & Sala, 2018). A way to quantify these will be asking the participants if there was a change in the way they were getting information about air quality.

 

Area

Subarea

Criteria

Metrics for DIAMS

1. Stakeholder collaboration and program resources

1a. Stakeholders

Does the community include scientist, technologists, local authorities, citizens, industries and business?

Is there a structured scheme for linking program participants?

Is there a mutual trust among project participants?

Number of actors mobilized in the project, including project partners and stakeholders

Percentage of stakeholders mobilized compared to stakeholders approached

Number of structured schemes including online discussion groups

 

1b. Resources

Is there a long-term commitment for funding?

Are the adequate staff with appropriate training?

Funds for keeping the DIAMS online platform, numerical tools and data infrastructure

Number of involved partners in running the program

2. Goals and objectives

2a. Goals

2a.1 Is there a clearly defined goal of the program?

 

 

2b. Objective

2b.1 Are the objective aligned with the overall goal?

2b.2 Are the objectives SMART (specific, measurable, attainable, relevant, and timely)?

2b.3 Are the objectives easy to explain and understand?

 

3. Methods: Design and implementation of monitoring

3a. Current understanding and conceptual model

Has all existing information on the system, background, and methods been compiled?

Has a conceptual model of the system being created?

Have the methods and protocols been tested (pilot data)?

Have appropriate analysis methods been outlined a priori?

Are planned statistical analyses being considered as part of monitoring program activities?

3.a.1 Number of protocols written & distributed; number of tutorials and videos published; number of pedagogical material created; number of numerical models compiled and run

3a.2 Online platform with the different elements of the project: protocols, data collected, models, results

 

3b Sample and protocol design

3b.1 Is the sampling protocol well design (scientifically sound, used established methods and easy to use by users)?

3b.2 Are the response metrics relevant and sensitive to change, and can they be measured against an appropriate reference state?

3b.3 Does the data collected by the program inform about the environmental challenge and walks towards the main goal?

3b.4 Have the methods and protocols been tested (pilot data)?

3b.5 Are planned statiscal analysis being considered as part of the monitoring program activities?

Evaluation reports about the collocation of sensor instruments against reference instrument

Statistical evaluating how far sensor data is from reference instruments.

 

 

3c Training and managing stakeholders

3c.1 Are protocols easy to understand and implement and appropriate for the level of expertise?

3c.2 Is there adequate training for stakeholders?

Number of workshops, webinars and tutorials for the correct usage of portable sensors

4. Data entry, storage, analysis and synthesis

4a. Organization and

management of data

 

Are data sets organized and well documented?

Are data housed in secure storage with long-term searchable archives?

Is there a coordinator to maintain the data and screen for errors?

Are data property and rules of access clear?

Are methods for uploading and downloading simple and clear?

Online platform and documentation library

Documents listed in the online library

Definition of keywords

 

4b. Quality assurance

and information integrity

Are there detailed specifications for methods, data entry, and QA/QC?

Are data-validation or -verification measures being used?

Are there regular review and quality checks for the data and database?

Number of calibration procedures and calibration protocols in place

Frequency of calibrations and co-locations

Frequency of data validation procedures of fixed and mobile sensors

 

4c) Data analysis and

interpretation

Does the analysis provide intended information about program goals and objectives?

Is the right amount of data being collected (not too much or too little)?

Are the data being collected at the correct scale(s)?

Are the current methods appropriate based on analysis of current monitoring data?

Percentage increment of the data points collected in the territory with air quality information

Model outputs and number of actions designed based on the model outputs

5. Reporting and dissemination

5b) Communication planning

Is there a clear commitment for public dissemination to provide educational value and facilitate community involvement?

Does the communication plan identify the audience and therefore the best medium for communication?

Is there a comprehensive communication strategy for disseminating results?

Is the communication plan designed to feedback to policy makers and/or regulators?

Number of associations designing activities for the public.

Number of activities designed for the public.

Webinars and meetings disseminating the results from engagement activities

 

 

5b) Outreach

implementation

and reporting

Is reporting regular, with results that are available quickly?

Are results being presented at the appropriate level of detail (scientific rigor)?

Are results being disseminated so that conclusions translate into action?

Does the communication include press releases, scientific publications, networking opportunities, and educational outputs?

Does communication reach and engage the public (e.g., an interactive website)?

Number of activities, including webinars, where results associated with results from activities

Number of different approaches where results are being communicated: press releases, technical reports, educational outputs.

Number of visitors to the online platform

6. Outcome assessment and program review

6a) Evaluating

outcomes: Science,

learning, and

engagement

Are scientific outcomes being assessed and measured?

Is the monitoring program informing conservation outcomes or outputs?

Are the data being used in decision-making?

Is the information contributing to the peer-reviewed scientific literature?

Is the program contributing to learning and engagement?

Number of policies and environmental measures designed and implemented on light with project results

Number of reports and scientific papers from the project

 

6b) Program review: Self

study and/or external

review

Do the participants reflect (periodically) on the program’s strengths and weaknesses?

Is there a mechanism for evaluation and feedback throughout the implementation of the program?

Are the participants and stakeholders open to external peer review?

Are reviews planned to be formative (less formal and part of program formation or development) or summative (periodic external reviews)?

Number of activities with surveys distributed to the public

Evaluation of citizen surveys

DIAMS enabled the testing and demonstration of the applicability of an innovative practice based on the use of portable sensors and numerical models to reinforce the knowledge about air quality problems to all members of the society. The engagement project encompassed different stakeholders and approaches. Numerical modelling was used to developed and implement environmental measures to reduce atmospheric emissions from anthropogenic activities.

Evaluation the impact of environmental projects with large citizen participation requires of thoughtful design and planning involving the scientists, technologists, social scientists, policy markers and regulators and the stakeholders themselves.

  1.  It might be a good idea to develop the evaluation program of the project in conjunction with the partners and the main stakeholders, as it is more likely about the success of the project. Specially in environmental project with citizen engagement, it might be good to involve citizens and main stakeholders in the definition of the evaluation metrics.
  2. All activities developed within the framework of a project should be accompanied by the expected impacts and how they will be evaluated. When writing and deposing a project it is difficult to conceptualize which will be the useful metrics that should quantify the impact of all workpackages and activities. However, as the project shapes and the actions are concretized, those should be accompanied by a definition of the possible impacts and how they are going to be evaluated.
  3. As recommended by the literature, project leaders are encourage to publish the demographics of the project (who participated, which gender, age range, socio-economic and socio-cultural level, ethnicity, etc.). Monitoring the demographics of the project ensures the inclusiveness of the project,  reaching all members of the society, whatever their social status, sociocultural origin, gender, literacy level, age, etc.

Blaney, R. J. P., Philippe, A. C. V., Pocock, M. J. O., & Jones, G. D. (2016). Citizen Science and Environmental Monitoring: Towards a Methodology for Evaluating Opportunities, Costs and Benefits. Citizen Science and Environmental Monitoring-Final Report. http://www.ukeof.org.uk/resources/citizen-science-resources/Costbenefitcitizenscience.pdf

Bonney, R., Phillips, T. B., Ballard, H. L., & Enck, J. W. (2016). Can citizen science enhance public understanding of science? Public Understanding of Science, 25(1), 2–16. https://doi.org/10.1177/0963662515607406.

Brody, S. D., Peck, B. M., & Highfield, W. E. (2004). Examining localized patterns of air quality perception in Texas: A spatial and statistical analysis. Risk Analysis, 24(6), 1561–1574. https://doi.org/10.1111/j.0272-4332.2004.00550.x

Canha, N., Justino, A. R., Gamelas, C. A., & Almeida, S. M. (2022). Citizens’ Perception on Air Quality in Portugal—How Concern Motivates Awareness. International Journal of Environmental Research and Public Health, 19(19). https://doi.org/10.3390/ijerph191912760

Charnley, S., & Engelbert, B. (2005). Evaluating public participation in environmental decision-making : EPA ’ s superfund community involvement program. 77, 165–182. https://doi.org/10.1016/j.jenvman.2005.04.002

Jordan, R. C., Gray, S. A., Howe, D. V., Brooks, W. R., & Ehrenfeld, J. G. (2011). Knowledge Gain and Behavioral Change in Citizen-Science Programs. Conservation Biology, 25(6), 1148–1154. https://doi.org/10.1111/j.1523-1739.2011.01745.x

Kieslinger, B., Schäfer, T., Heigl, F., Dörler, D., Richter, A., & Bonn, A. (2013). Evaluating citizen science : Towards an open framework. In H. Susanne, M. Haklay, A. Bowser, Z. Makuch, J. Vogel, & A. Bonn (Eds.), Citizen Science. Innovation in Open Science, Society and Policy. UCL Press. https://www.jstor.org/stable/j.ctv550cf2.13

Mahajan, S., Kumar, P., Pinto, J. A., Riccetti, A., Schaaf, K., Camprodon, G., Smári, V., Passani, A., & Forino, G. (2020). A citizen science approach for enhancing public understanding of air pollution. Sustainable Cities and Society, 52(June 2019), 101800. https://doi.org/10.1016/j.scs.2019.101800

Maione, M., Mocca, E., Eisfeld, K., Kazepov, Y., & Fuzzi, S. (2021). Public perception of air pollution sources across Europe. Ambio, 50(6), 1150–1158. https://doi.org/10.1007/s13280-020-01450-5

Ockenden, N., & Hutin, M. (2008). Volunteering to lead: a study of leadership in small, volunteer-led groups. In Institute for Volunteering Research (Issue March): https://www.bl.uk/britishlibrary/~/media/bl/global/social-welfare/pdfs/non-secure/v/o/l/volunteering-to-lead-study-of-leadership-volunteer-led-groups.pdf

Oltra, C., & Sala, R. (2018). Perception of risk from air pollution and reported behaviors: a cross-sectional survey study in four cities. Journal of Risk Research, 21(7), 869–884. https://doi.org/10.1080/13669877.2016.1264446

Oltra, C., Sala, R., Boso, À., & Asensio, S. L. (2017). Public engagement on urban air pollution: an exploratory study of two interventions. Environmental Monitoring and Assessment, 189(6). https://doi.org/10.1007/s10661-017-6011-6

Pantavou, K., Lykoudis, S., & Psiloglou, B. (2017). Air quality perception of pedestrians in an urban outdoor Mediterranean environment: A field survey approach. Science of the Total Environment, 574, 663–670. https://doi.org/10.1016/j.scitotenv.2016.09.090

Pateman, R., Dyke, A. & West, S. 2021. The Diversity of Participants in Environmental Citizen Science. Citizen Science: Theory and Practice, 6(1): 9, pp. 1–16. DOI: https://doi.org/10.5334/cstp.369

Schäfer, T., & Kieslinger, B. (2016). Supporting emerging forms of citizen science: A plea for diversity, creativity and social innovation. Journal of Science Communication, 15(2), 1–12. https://doi.org/10.22323/2.15020402

Sparrow, S., Bowery, A., Carver, G. D., Köhler, M. O., Ollinaho, P., Pappenberger, F., Wallom, D., & Weisheimer, A. (2021). OpenIFS@home version 1: A citizen science project for ensemble weather and climate forecasting. Geoscientific Model Development, 14(6), 3473–3486. https://doi.org/10.5194/gmd-14-3473-2021

Stevenson, J., Loughlin, S. C., Font, A., Fuller, G. W., MacLeod, A., Oliver, I. W., Jackson, B., Horwell, C. J., Thordarson, T., & Dawson, I. (2013). UK monitoring and deposition of tephra from the May 2011 eruption of Grímsvötn, Iceland. Journal of Applied Volcanology, 2(May 2011), 3. https://doi.org/10.1186/2191-5040-2-3

Tredick, C. A., Lewison, R. L., Deutschman, D. H., Hunt, T. A., Gordon, K. L., & Von Hendy, P. (2017). A Rubric to Evaluate Citizen-Science Programs for Long-Term Ecological Monitoring. BioScience, 67(9), 834–844. https://doi.org/10.1093/biosci/bix090

UN. (2015). The paris agreement. Towards a Climate-Neutral Europe: Curbing the Trend. https://doi.org/10.4324/9789276082569-2

Wesseling, J., de Ruiter, H., Blokhuis, C., Drukker, D., Weijers, E., Volten, H., Vonk, J., Gast, L., Voogt, M., Zandveld, P., van Ratingen, S., & Tielemans, E. (2019). Development and implementation of a platform for public information on air quality, sensor measurements, and citizen science. Atmosphere, 10(8). https://doi.org/10.3390/atmos10080445

About this resource

Author
Anna Font, UIA Expert
Project
Location
Aix-Marseille Provence metropole, France
About UIA
Urban Innovative Actions
Programme/Initiative
2014-2020

The Urban Innovative Actions (UIA) is a European Union initiative that provided funding to urban areas across Europe to test new and unproven solutions to urban challenges. The initiative had a total ERDF budget of €372 million for 2014-2020.

Go to profile
More content from UIA
1150 resources
See all

Similar content