Final (internal) evaluation and results: what can be replicated in other cities endowed with forest resources

The monitoring system of UFIL was developed against six objectives (two of which were structured into sub-objectives) and six results. A set of 12 indicators was used to measure progress against the six results. The figure below shows how the relation among these three building blocks of the system (objectives, results and indicators) is not linear. Results were measured by a minimum of one indicator (for example, R6 ‘Enhance collaboration between administrations for sustainable for-profit exploitation of the forest’) to a maximum of four indicators (for example, R5 ‘Favor urban integration with natural and rural environment’).

UFIL overall objective was to locally develop a forest-based bioeconomy. In practice, this was meant to trigger sustainable business development in the forest sector of the province, including through the improvement of capacities and skills and the attraction of talent. The core of the project was the innovation lab (for talent attraction and development of capacities/skills), but the overall objective clearly points to a broader impact on the local economy and to an innovative, environment-friendly way to conceive forest business development.

For any city endowed with sufficient forest resources and wishing to locally start a forest-based bioeconomy, it may be interesting to understand how UFIL approached the above ambitious overall objective. Below are described the individual specific objectives of the project and the results achieved over a period of four years and with €3.9 million co-financing from the ERDF. 

UFIL built the ‘innovative’ character of this Urban Innovative Action through the set-up of an ‘innovation lab’. The innovation lab was a residential training course located in the City of Cuenca where the teaching of innovative thinking and business development took place. Most of the teaching was delivered by the project partners. The residential course run three times over the project period. A total of 84 persons were selected and provided, over the three courses, with knowledge and tools to become innovative entrepreneurs in the forest sector. These future entrepreneurs launched, overall, 44 entrepreneurial ideas. However, only 14 of these ideas went a step forward and became registered businesses (see their listing in the picture below). The evaluation provides some more numbers: total applications to the residential courses were 329; out of the 84 selected candidates, 63% were men and 37% women; 46% were from Cuenca and 54% from outside Cuenca; finally, the average age of participants was 32 years.

Results versus the objective are mixed. The target of 40 created businesses (INDICATOR 1) was not achieved, so the project under-performed if the formal registration of businesses is considered. However, in terms of retention of these businesses in the territory, the project over-performed because 13 of the created businesses ended up in being registered in the Province of Cuenca (93% against a target of 60%) (INDICATOR 2).

The innovation lab is a replicable element in other contexts/cities. Teaching can be organised through the project partners (if the consortium is varied and qualified enough) or through the input of external providers. However, there are challenging aspects to be carefully considered:

• There must be a sufficient number of applications to the residential courses, so as to be in the position to select the best possible candidates. This requires appropriate advertising of the training opportunity offered by the project since the project kick-off (or even earlier, if feasible) and appropriate selection criteria for the identification of the right trainees.
• Trainees are likely to be insufficiently skilled to put forward truly innovative entrepreneurial ideas, thus the training has to provide concrete inspirations. Opportunities for innovation may be suggested by the industry or by academic entities specialised in the bioeconomy sector. The involvement of these specialised actors is essential to create an ‘ecosystem’ around the future entrepreneurs.
• Business creation takes time and if the formal registration of companies is to be achieved to meet the project’s objectives, targets need to be set carefully.

This rather broad objective was split into two more specific sub-objectives: improve the skills of Cuenca’s workforce for initiating innovative projects and raise awareness about the opportunities derived from the use of forests.

Several of the indicators used to measure the project’s achievements against this objective are qualitative and relate to the level of satisfaction about the project. In general, the evaluation shows that satisfaction is good among all those consulted (participants in the innovation lab and the businesses community). In particular, 75% of the participants in the innovation lab believe that their employability importantly improved after their participation in the project and even more participants (90% against an average target of 80%) are satisfied (INDICATOR 6). In addition to these qualitative results, the project surveyed the participants’ post-UFIL situation and concluded that 12 months after the completion of the training, 78% of UFIL participants were employed, either as employees or as entrepreneurs developing their new business (the target for this INDICATOR 3 was 60%, so in this case the project over-performed).

On the level of satisfaction about the facilities and materials made available by the project (UFIL spent over €0.7 million in equipment), the share of those who are ‘fully satisfied’ is rather low (28%), but, overall, 88% of the participants were at least ‘satisfied’ (see figure below).

On the side of the businesses, all (100%) companies that participated in the project as sponsors were satisfied about the ways they interacted/collaborated with the project (the target for this INDICATOR 4 was 75%, so the project over-performed). It is important to mention that participation of businesses in the project was limited, especially compared to the original plans and the originally envisaged ‘business challenge’ mechanism. Excluding the challenges proposed by the project partners in the first innovation lab course, only four companies made challenges in the second course. The project approach changed in the third course and the business challenge was replaced with face-to-face meetings between the participants in the innovation lab and the companies. UFIL internal evaluation reports that 67% of the sponsors having participated in the project will continue participating in next (potential) programs (the target for this INDICATOR 5 was 90%, so the project in this case under-performed).

The progress made against the second specific sub-objective (raising awareness about the opportunities derived from the use of forests) is measured comparing relevant enrollment data in the University of Castilla-La Mancha. In the academic year 2018/19, there were 10 new students enrolled for a Degree in Forestry and 4 new students enrolled for a Master in Forestry. In the academic year 2022/23, these numbers were 17 and 15, respectively. If the increase is evident, it is also true that students from Cuenca were (only) 2 in both academic years. Thus, the indicator ‘Rise in forest bioeconomy interest expressed as the share of students from Cuenca that decide to study bioeconomy studies’ (INDICATOR 7) was 0% (the target was 10%).  

The business challenge mechanism can be replicated anywhere there is the need to reveal talent. The way it was applied in UFIL is comprehensively discussed in my ZOOM-IN#1. Among the challenging aspects of this mechanism are:

• the business challenge mechanism does not create talent, it can only reveal it where (and if) it exists. This implies that the appropriateness of the mechanism depends very much on the level (high, low) of the delivered training and of the trainees. According to an investigation run by the project, the tutorials on the challenges were rated 2.93 by UFIL participants (over a 0-5 scale); similarly, the same investigation discloses that UFIL participants rated their applied knowledge of forest bioeconomy and their capacity to apply practical tools on real challenges below 3 (over a 0-5 scale);
• there must be clear incentives in place for the businesses to get involved in this type of mechanism.

As mentioned above, 13 of the created businesses were registered in the Province of Cuenca. However, 8 of these businesses are run by participants from outside Cuenca. The number of participants from outside Cuenca that decided to create their businesses in Cuenca represents INDICATOR 8. The project slightly under-performed against this indicator (the target was 10 companies).

The capacity of UFIL to keep these businesses in the province is the result of the identification and implementation of retention mechanisms by the project management and the project partners. I explained these (and other) mechanisms in my last web article on ‘Incentive mechanisms for visibility and engagement: acknowledgments, prizes and awards’. Retention mechanisms are crucial to ensure that investments made in human capital through the project give a ‘return’ to the territory. Among them, there are cash prizes from Globalcaja foundations, grants from the Provincial Council of Cuenca, use of the innovation lab equipment and tools to test and prototype products and services, and support services offered by the local Confederation of Employers CEOE CEPYME Cuenca (one of the partners in UFIL).

Retention mechanisms are replicable in many contexts. The provision of these mechanisms by a project implies to work out collaborations with local stakeholders and provide attractive benefits. Among the potential challenges related to their identification and implementation are:

• the time needed to identify, test and launch the most appropriate mechanisms which is substantial;
• there is the need to identify a range of mechanisms in order to address different needs and situations. For example, grants in UFIL were awarded on a competitive basis, thus they benefited only a limited number of companies among those established by the project. In order to widen the audience of beneficiaries, mechanisms with high accessibility are required. For example, UFIL made available the use of the innovation lab with its equipment and facilities. This is an easily accessible benefit for all those interested. Similarly, the support provided by CEOE CEPYME is not selective and available to all.

The focus of this objective was knowledge development. In the first year of implementation, the project dedicated significant time to the drafting of a ‘Study on forest bioeconomy and its opportunities for innovation in Cuenca’. The study provides a sort of baseline of the forest sector in Cuenca and identifies potential areas for business development. Completed in early 2020, the study was later (2022) updated. In addition, 11 annexes focusing on specific aspects (for example, wood engineering, carbon storage, production of resin, mycology, biomass and biofuels) were derived from the main document. Later on, four additional guides were developed by UFIL to focus on applied technologies in the forest sector and forest ownership issues. All these documents were also used as teaching material.  

In the end, the number of sectorial documents, reports and studies evaluating the opportunities of Cuenca’s forests for business innovation (INDICATOR 9) was higher (15) than the target (11).

In terms of replicability, all projects produce knowledge and the only peculiarity in the way UFIL did it is that the material had multiple uses: during the first year of the project, the development of the main study made all partners aware of the starting situation in the province of Cuenca (the baseline). So, it was a sort of internal awareness-raising exercise. Later, all the developed material and some derived (and shorter) documents were used in the classrooms. Finally, this material served to disseminate information on Cuenca’s forests and opportunities. Among the potential challenges are:

• the preparation of a sectorial baseline is important, but it cannot take over one year to be completed;
• the content and structure of knowledge developed during a project should be agile enough to adapt to multiple uses such as, in the case of UFIL, awareness-raising, teaching and dissemination purposes.

This objective sounds complex, but was treated in very simple terms from a monitoring (and implementation) point of view. There are two main indicators used by the project to measure the urban-rural linkages created, both of which relate to the place of origin of the participants in the project. Namely, UFIL attracted 46% of its participants in the innovation lab from the province of Cuenca (39 persons); 28 of these persons were from Cuenca and 11 from small municipalities in the rural areas. The number of participants from rural areas represents INDICATOR 10. The target for the indicator was 5, so the project over-performed.

Similarly, in terms of participating sponsors, two (2) were located in rural areas. The 'number of sponsors coming from the rural surrounding of the city and that decide to grow their businesses' represents INDICATOR 11. The target for this indicator was 2, so the project is reported (in the internal evaluation) to have fully met the target. The project undertook a few other small activities in rural areas. The pandemic is mentioned as one of the reasons preventing the implementation of more initiatives.

There are no elements for replication within this objective that was evidently not prioritised.

The scope of this objective was to enhance the collaboration between administrations towards the sustainable exploitation of forests. Besides the City of Cuenca, the project partnership also included the regional government of Castilla-La Mancha which is responsible for the management of forest resources, including those owned by the City of Cuenca. However, it was the city to play the main role under this objective. First, it established two collaboration agreements with local stakeholders: one with the Provincial Council of Cuenca and another one with the Local Action Group PRODESE. The former provided support to UFIL participants (economic and logistics), the latter was meant to facilitate the connection with the territory. In addition, the city strengthened, in close liaison with the regional government, the collaboration with FSC. This led to an extension of the certified forest area both in the province and in the whole region (additional significant increases in the certified forest area are expected in 2023). Notably, this aspect was facilitated by the participation of FSC in UFIL as a partner. This participation is discussed in my web article ‘Forest certification in and around UFIL: a multi-functional driver’.

Finally, the project set 13 actions in the Urban-Rural Agenda of Cuenca that require an integrated collaboration between the City of Cuenca, the Provincial Council of Cuenca and the Region of Castilla-La Mancha. The content of the Urban-Rural Agenda of Cuenca will be investigated in detail in a future web article. In brief, it can be anticipated here that the Agenda is what UFIL prepared for the future: a roadmap to 2030 for the promotion of its forest bioeconomy.

The vertical integration between administration was measured by INDICATOR 12 ‘Forest related initiatives proposed in the project that require collaboration between different administrations'. The set target was 20 initiatives and the project achieved 15 actions.

In terms of replicability, among the elements found under this objective, it is worth noting the formal collaboration agreements signed with institutional and non-institutional stakeholders.

The twelfth indicator is the last in the set of indicators used by UFIL to measure progress against its six expected results. Overall, it is evident that the focus of UFIL monitoring system remained essentially on results and did not capture the impact on the city or the territory. However, this approach seems to be commonly found across UIA projects (see the report on 'UIA lessons learnt on monitoring and evaluation practices').

In the previous sections, I gave an overview of the main results achieved by the project and made some personal remarks, especially in terms of elements that may be replicated in other cities/contexts. However, I believe it is necessary to conclude with a summary of the findings of the final and internal evaluation of the project. Although partially summarised, the text below is taken from the evaluation;  for this reason it is reported as quoted text.

In general,

UFIL has satisfactorily achieved the proposed indicators and is tackling many of the initially proposed objectives and results....... It also went a step forward by building the reputation of Cuenca, within the country, as a reference city for forest bioecomony ; and by pursuing awareness-raising and institutional commitment towards forest certification.

The UFIL evaluation also highlighted the importance of having the right participants in the innovation lab. This aspect requires time and resources as well as contacting and networking activities. The lab and its equipment...

....played a key role in the prototyping of products and services for entrepreneurship projects, supporting the success of the initiatives......The lab  is an important retention mechanism, it is a meeting point where experiences and knowledge are shared…... It is important that these facilities are kept open and available to entrepreneurs and to their projects.

In terms of the content of the training,...

...participants in the lab look for practicality, applicability and flexibility. This is a point to consider for future programs..... Structural support or aid for the future entrepreneurs could also be envisaged, at the beginning (when their idea is conceived) and at the end (when they are going to register their company)……Also, the mixing of standard classrooms with remote training should be considered....

Finally, on aspects to be improved... 

...UFIL needs to strengthen its presence in rural areas, relocating activities there…. In addition, it is necessary to have more links with companies (particularly local) and to seek greater commitment from participating companies as sponsors, despite the fact that the sponsorship model has been valued positively.