Handbook of territorial & local development strategies Chapter 6 - Monitoring

Introduction

The monitoring of territorial and local development strategies is crucial for their success. Monitoring generates important data and knowledge to track progress with respect to previously identified targets and objectives and to inform necessary revisions. It is necessary to generate information that feeds into future evaluation activities. The absence (or unreliability) of monitoring data makes it difficult to measure the effectiveness of strategies in delivering territorial and local development outcomes, in absolute terms, or in relation to the effectiveness of other delivery methods (including mainstream EU cohesion policy programmes). Moreover, a robust monitoring system supports transparency, accountability and the visibility of EU support ‘on the ground’. By demonstrating what territorial and local strategies are achieving and what is working in terms of implementation, effective monitoring strengthens the ownership of initiatives by stakeholders. This is particularly important in the case of local strategies where an important objective is mobilising the participation of local communities in strategic development and ensuring their commitment over the long-term.

The role of monitoring is emphasised by the increasing efforts in the EU cohesion policy to design result-oriented strategies with a clear logic of intervention that is directly linked to a framework for measuring results. Put simply, the starting point for strategies is the analysis and prioritisation of the needs to be addressed, followed by the allocation of resources through operations to address these needs. Monitoring examines whether the anticipated results are being obtained or, conversely, whether changes are needed in the implementation and efforts should be re-targeted. Monitoring information is also used to design and inform evaluation activities to understand what has been achieved and how by programmes and strategies (see Figure 5).

Image

Thus, a monitoring system that is capable of tracking progress towards established results is an essential component of developing a territorial or local development strategy. This includes the selection of a coordinated set of indicators and also data gathering arrangements to support learning about policy actions in the territory. 

Evaluation also assesses whether the objectives have been met efficiently, as well as the reasons for its success or failure. It also addresses the issue of a causality between the effects and the policy intervention. Evaluation should also identify any unintended or unexpected effects, whether positive or negative. Evaluation complements monitoring in the sense that when a monitoring system signals that public action is diverging from is intended path, then the launch of an evaluation exercise can clarify the reason of such a diverging path (i.e. expected targets and results). The implication is that monitoring can be used to collect data for the evaluation (European Commission, 2021b).

Monitoring EU-funded, territorial and local development strategies is rather challenging. Developing indicators and generating data to capture integrated territorial effects and success is difficult and problems can arise in specific settings. In some localities, integrated territorial investments represent a new approach and delivery mechanism compared to traditional approaches based on more top-down, sectoral measures that target areas based on administrative units. This shift can create challenges in mobilising local participation and strengthening local capacity in monitoring these initiatives. Data-collection can be expensive, time-consuming and frustrating.

Beyond the local context, it is important to take into account that local strategies are part of multi-level architectures. Accordingly, monitoring systems are expected to demonstrate strategies’ contribution not only at a local level, but also at programme, national and EU levels. Different stakeholders will benefit from different types of monitoring information and each local strategy must consider the various arenas in which this data and the knowledge generated are disseminated. Monitoring will take place at several levels with regards to financial and substantive progress and the type of data required; its territorial and thematic scope will differ across these levels (see Table 5).

Image

This chapter addresses four main challenges of monitoring systems and presents potential solutions and sources of support to address these.

The first challenge relates to the fact that monitoring systems should be able to capture and follow the expected changes set out in the strategy’s logic of intervention. The key objectives and actions of strategies should be directly linked to a framework for measuring results. This emphasises the importance of selecting the most appropriate indicators and units of measurement of those indicators that relate to the objective to be achieved. However, local and territorial strategies face challenges in choosing indicators to address different dimensions of measurement: integrating indicators for sectoral and territorial priorities; ensuring accurate territorial coverage; balancing ‘hard’ indicators for tangible results with ‘soft’ indicators that capture less visible but important results; and combining the measurement of short-term/long-term results.

The second challenge acknowledges that for small local authorities, monitoring strategies represents a significant task. Data availability and collection capacity in territories with specific geographic features is often a challenge and data consistency can be problematic. In rural or remote areas with dispersed settlement patterns or in insular or cross-border settings, local-scale socio-economic processes are complex and collecting data at an appropriate scale to assess achievements of territorial or local development strategies against these patterns is challenging.

The third one outlines that monitoring by itself does not improve the performance of local and territorial strategies. In order to be effective, monitoring must play an integral role in the overall process of local strategy design and delivery. It is vital for local strategy owners to ensure links between monitoring, evaluation processes, communication arrangements and overall strategy governance so that the information collected is used to improve strategy implementation and policymaking.

Finally, the fourth challenge stresses that strategy owners must also consider how to involve stakeholders in monitoring. Engagement of local communities in monitoring increases ownership, autonomy and accountability and strengthens local commitment to implement corrective actions. However, there are challenges in engaging with citizens and communities, particularly more marginalised actors. Mobilisation demands careful planning as it involves the investment of time and human resources of strategy owners, communities and citizens.

Challenge 1 : How to design and implement a monitoring system with appropriate indicators and a strong intervention logic ? 

Monitoring systems should be able to capture and follow the expected changes set out in a strategy’s logic of intervention. The logic of intervention is a clear and well-thought-out representation of how planned actions are expected to lead to desired outcomes. Essentially, the logic of intervention clarifies how a change induced by policy action contributes to the achievement of strategic goals. The key objectives and actions of strategies should be directly linked to a framework for measuring results. This underlines the importance of developing a coordinated, tailored set of indicators for strategies. The main categories of indicators to consider are:

• Input – what is being committed? This relates to the resources committed to initiatives covered by interventions in strategies and can include financial inputs, technical expertise and commitment of human resources.
• Output – what does the action deliver? Values are used to measure the outputs of the operations supported or the outputs at operational programme level. Indicators here concern the specific deliverables of the intervention. They measure what is produced or bought about by the commitment of resources.
• Result – what does success look like? Values are used to measure the results generated by supported projects, or the results achieved at operational programme level. These indicators match the effects of the intervention with particular reference to the direct addressees i.e. the benefit and outcome of interventions related to (or derived from) the use of outputs. These results, for instance, refer to the performance of beneficiaries, investments triggered, increased access to services, etc. (European Commission, 2021a).

There are key questions that must be faced in developing indicators based on a strong intervention logic:

• What are the needs that the strategy will address and what is the expected contribution to its objectives?
  Example: the strategic objective is to increase the growth of those local small and medium sized enterprises (SMEs) with growth potential, in particular facilitating the creation of new economic activities linked to local products.
• What will be the relevant actions to support beneficiaries?
  Example: Support for the establishment of small enterprises and for the diversification of small enterprise activities. Output indicators: Number of enterprises receiving grants or financial support other than grants; total investment in enterprises.
• What is the change expected for beneficiaries?
  Example: No. of new/innovative methods adding value to local products, no. of new firms, products developed, employment increase within SMEs, Gross Added Value on supported enterprises; increase in the number and types of customers (result indicators) (INTERACT, 2020).

Strategy monitoring systems should take into account the multi-level architecture in which they are embedded, which translates into different information/data needs and interests.

Challenge 2 : How to address capacity challenges in ensuring the availability of datasets and arrangements for collecting and analysing monitoring data ?

Territorial and local strategies require a robust system for generating, collecting and analysing monitoring data. This can involve, for instance, direct data collection from participants/entities (e.g. questionnaires or surveys) or data extracted from administrative registers. Data should be accurate and collected and recorded in a timely way in order to inform strategy implementation and evaluation. A key challenge is ensuring the capacity for collecting information for measuring the results of integrated measures in specific territorial contexts and maintaining it over time. There is often substantial variability in the type of indicators and datasets typically used by these strategies, in keeping with their diverse territorial coverage and objectives.

Challenge 3 : How to embed monitoring in the policy cycle for better knowledge ? 

Monitoring by itself does not improve the performance of local and territorial strategies. To be effective, monitoring must play an integral role in the overall process of local strategy design and delivery.

It is vital for local strategy owners to ensure links between monitoring, evaluation processes, communication arrangements and overall strategy governance so that the information collected is used to improve strategy implementation and future policy making. Clarity and continuity between monitoring and these other elements requires effective working relationships between the implementing authorities and other stakeholders. Monitoring systems will only be effective if the gathered information is used to improve strategy design and delivery. Strategy owners should ensure that the knowledge generated through monitoring is relevant to different stakeholder audiences and that this knowledge is communicated in the most effective way.

Engaging relevant stakeholders and citizens strengthens the quality of monitoring systems of local strategies. It has the potential to create a substantial amount of localised data ‘on the ground’. It also promotes a dialogue between citizens, other stakeholders and policymakers, which may prove very beneficial for monitoring due to new ideas, suggestions, insights, etc. Finally, it raises awareness of the benefits of these territorial instruments in the local population, strengthens transparency, accountability and ownership.

However, mobilising and engaging citizens and communities is challenging. Monitoring is often seen as a technical task, run by public officials and experts. Even if stakeholders participate in the initial design phase, it can be difficult to keep them engaged in future monitoring activities. Once objectives and indicators have been established, there is a risk that actors lose interest and mutual learning processes decline (Marinelli, Guzzo and Gianelle, 2019).

There are capacity issues that can be exacerbated by monitoring systems that are either overly complex and unworkable in practice, or too simplistic, delivering the wrong information. Stakeholders may be willing to participate but are unable to do so because they don’t have the required skills and capabilities. Some may be able but unwilling to participate due to low trust and concern about how the authorities would use their contributions.

Participatory monitoring is based on the premise that there is consensus on strategic priorities between different stakeholders and that their goals are mutually compatible. This is not always the case. All of these challenges can be particularly apparent in encouraging participation from vulnerable or difficult to reach communities and individuals who may lack the resources or capacities to actively engage (e.g. women and young people) (European Union, 2022). In addressing these challenges strategy owners should take into account these key principles.

Recommendations

• Design and implement a monitoring system with appropriate indicators and a strong intervention logic.
  • Indicators should relate directly to the objectives of the local or territorial development strategy, territorial coverage and scope of the strategy.
  • Start by developing a complete and clear logic of intervention linked to indicators that make it more meaningful. Poor indicators that are not policy responsive are often the result of a vague or incomplete logic of intervention.
  • Hard output indicators should be complemented with softer qualitative indicators. Indicators should be adapted to address long-term processes as well as short-term results. This approach helps engage local communities to understand the real achievements on the ground, particularly those that are less tangible and not measurable in numerical terms. Input indicators that capture workload input, time and resources committed to strategy design and delivery are also important.
  • Local and territorial strategies should draw on existing monitoring resources at multiple levels. EU, national and sub-national administrations have monitoring systems and data sources that can be adapted to support monitoring of local and territorial strategies.
  • Develop monitoring systems that could remain stable over time for the implementation of territorial development instruments.
• Develop an efficient system for data collection and analysis.
  • There is a need for proportionality and flexibility. It is clear that ‘one size fits all’ solutions in setting up monitoring systems cannot be pursued. The huge variation in size in terms of population covered, thematic focus, budget, geographic scale and implementation approach, means that approaches to assessment must be tailored to specific circumstances.
  • Select a short list of indicators that reflect the impact of the integrated investments on an aggregate level. Use a limited number of key indicators with realistic possibilities of obtaining up-dated data.
  • Reflect on data-collection needs and capacity while defining the monitoring system. Identifying good indicators does not guarantee that they can be populated with adequate data. Those in charge of the process need to reflect on whether they and relevant stakeholders have the tools, resources and competences to collect and process suitable data at an appropriate time. Plan appropriate training and capacity building activities if needed.
• Embed monitoring in the policy cycle.
  • Monitoring should be planned from the outset as an iterative process, designed and managed so that monitoring results can feed into the decision-making process.
  • Integrate the needs and perspectives of multiple audiences when developing and implementing data collection and communication systems. Monitoring must produce simple information that is communicable and easily understood by both the provider and the user of the information.
  • Communicate the results of monitoring and evaluation at the right time so that learning and recommendation can also feed into new policy cycles.
• Mobilise relevant stakeholders (including citizens) in monitoring activities.
  • Co-create monitoring systems between strategy owners, local communities and citizens. Participatory monitoring should aim to share control over the content, the process and the results of monitoring activity and engage local communities in taking or identifying corrective actions.
  • Capacity-building actions and instruments (such as training, toolkits, peer to peer exchanges, focus groups) should be considered to facilitate engagement of target stakeholders and beneficiaries in monitoring. Innovative techniques and technologies to facilitate participatory monitoring should also be considered (e.g. use of social media, mobile phones).
  • Plan monitoring and evaluation milestones (e.g. annual reporting events, mid-term reviews) as opportunities to engage stakeholders in monitoring and modification of strategies.