Utrecht U-RLP

In the overall project design, the evaluation constituted a separate workstream with dedicated partners. It was planned as an independent exercise to be conducted over three years by researchers representing two UK-based academic centres – University College London (UCL) and the University of Oxford.

The study was led by an expert from UCL trained in sociology and anthropology, supported by a small team. In addition to a trained research team, the evaluation benefited from the expertise of an academic Advisory Board, chaired by a representative from the University of Oxford. The board gathered distinguished scholars specialised in issues relevant to the project, such as the reception of asylum seekers, but also to evaluation itself. It met three times – to discuss the methodology, interim findings and draft final report.

Both the evaluators and the project management found the Advisory Board to offer important support. With its multidisciplinary in-depth academic expertise and independence, the Advisory Board was able to provide subject-matter expertise, but also challenge assumptions (underlying the theory of change, for example) or expose possible biases.

It was also in a position to provide guidance to the researchers in a dynamic evaluation context, which required flexibility in the methodology and added nuance to the findings.

While the project beneficiaries were involved in the evaluation through a large number of individual interviews, ideally, the lead evaluator would have liked to involve representatives of the target group in data collection as community researchers. This was not possible in the project’s short timeframe, however.

The project created cooperation mechanisms which enabled implementation of the evaluation, on the one hand, and feeding of the evaluation results into the partnership on the other. The evaluation included extensive cooperation with all project partners at various stages. They were included in the development of the approach and the project’s theory of change. They supported the data collection process and were informed on the evaluation’s progress. The evaluators, while not being involved in the main activities, closely followed the project’s work and participated in its various meetings. Specific meetings were dedicated to discussions related to the evaluation itself. One meeting was specifically devoted to extensive explanation and clarifications of the applied approach for project partners. Separate meetings were also organised to present and discuss evaluation results, both interim and final.

The experiences of U-RLP’s evaluation show how important it is to set aside appropriate resources for evaluation in order for it to produce expected learning. As the project evaluators noted, the aspiration to measure impact is an ambitious one, so it should be allocated an appropriate budget and timeframe.

If appropriate resources are not available, then expectations of what can realistically be delivered should be managed.

U-RLP’s experience underscores the need for sufficient time to be earmarked in two dimensions in particular – working time for researchers and time for the evaluation itself, including for its preparation. The project’s evaluation can be seen as an example of best practice for various reasons, and both its breadth and depth are impressive. This was possible thanks to the in-depth expertise, as well as strong personal engagement and commitment, of the otherwise rather under-resourced evaluation team, supported by project management.

One of the reasons why resources were insufficient related to the fact that not all obstacles and difficulties were anticipated from the start. As the lead evaluator observed, had there been more clarity in this respect, she would have budgeted for more research time. It is, thus, an argument for  devoting a sufficient amount of time for risk analysis at the evaluation design stage.

Apart from working time, U-RLP experience also shows that the project would have benefited from a preparatory phase. This would have allowed for a more comfortable development of a theory of change (see also below a discussion on how time-consuming this process was) and research tools prior to the implementation of activities themselves. The lead evaluator estimated such a phase to require roughly a couple of months, possibly as many as six. Importantly, such a preparatory phase within projects was introduced by UIA in subsequent funding editions.

U-RLP’s evaluation is an example of a comprehensive and coherent approach which responded to the project’s reality. The initial expectation was that the evaluation would follow an experimental or quasi-experimental approach. This would have helped determine causal relations between participation in the project and specific results or impacts. However, the project’s nature and complexity, as well as a number of resulting considerations, made implementation of such an approach impossible, even though it had seemed feasible at the time of designing the evaluation.

The final evaluation report lists a number of challenges which prevented application of the experimental approach, namely that:

• Participants experienced the project in different ways, to varying degrees and for different periods of time, which made comparisons difficult;
• Random assignment to experimental and control groups, i.e. provision of support to some asylum seekers and denial to others based on random selection, was highly problematic ethically in a project which targeted a vulnerable population;
• Asylum seekers themselves could choose to participate or not to participate in the project (or in parts of it);
• Due to varying conditions in asylum seeker centres in the country, it was hard to find any centre which could function as a comparable control group for the experiment.

Application of an experimental design was also hindered by the decision of the Dutch asylum authority (COA) to deny the evaluation team access to asylum seekers residing in the centre unless they participated in the U-RLP project. This meant that the evaluation team did not have access to those asylum seekers from the Overvecht centre who did not decide to be involved with the project. Consequently, the sample of accessed asylum seekers was biased towards those receiving support and the opportunities for comparison with those who did not receive it were limited. More information on the considerations which underly U-RLP’s decision not to pursue an experimental design can be found in the relevant part of the final evaluation report. In this respect, the U-RLP intervention shares similarities with the evaluation conducted within Rotterdam’s BRIDGE project.

Eventually, the evaluation followed a theory-based approach, involving the use of a theory of change for both designing and implementing the evaluation and determining the project’s contribution to expected outcomes. As the final evaluation notes, this approach “integrated rather than eliminated these important contexts”, which posed a challenge to the experimental approach “by seeking understanding of the interplay of project and effects”. In other words, instead of causal relations, this approach allowed evaluators to focus on the questions of how and why things had happened.

Rather than seek to prove causal attribution, the goal of the evaluation was to explain, with sensitivity to contexts, how the existence and operation of Plan Einstein likely contributed to observed outcomes, or made ‘a difference’.

 Source: U-RLP Final evaluation report

The U-RLP project did not have a theory of change from the start, so theory development was the first stage in its evaluation. This helped to define what the project was trying to deliver and to bring partners towards a common understanding of these aspirations. Theory of change proved to be a flexible enough methodology to enable collaborative work and allow space for different points of view. As this suggests, since it can help to define objectives and build agreement between diverse stakeholders, the approach can be particularly useful in projects which involve many partners. Similarly, it can fit those interventions which were designed by one group of people yet implemented by another. Both circumstances were present in the U-RLP project. Importantly, with academic partners leading the evaluation, the theory developed with U-RLP project partners was also rooted in the latest science related to asylum seeker reception, and their integration with local communities and the labour market. 

As U-RLP’s experience shows, theory development also posed some challenges. In particular, it proved very time-consuming as a process in a multiple partner setup. The innovative and pilot character of the project, which meant less clarity about the aims and possible outcomes at its outset, intensified this challenge. For this reason, the project evaluators underscored the importance of allowing an appropriate amount of time for this stage of the evaluation process, and for evaluation preparation more generally. This is particularly true if the theory of change is to be consistently used for monitoring the project, as well as designing and implementing the whole evaluation.

Another challenge in implementing the approach related to explaining the meaning of theory development to project partners who did not necessarily have a research or evaluation background.

This shows that in complex, multi-partner interventions, evaluators need to also be good communicators who are able to explain the meaning of various evaluation processes and, project-wise, bring all the partners to a common denominator. This is easier when evaluators have enough platform in the project to interact with other partners, as was the case in the U-RLP project.     

While theory-based evaluations sometimes face criticism for their actual limited use of the developed theory, Plan Einstein employed its theory extensively throughout the project and evaluation. Thus, the theory served as basis for project monitoring. It was also used to develop evaluation questions and determine data collection methods and sources, as well as to develop coding for data analysis. 

Importantly, however, the U-RLP lead evaluator felt that there was potential for pushing the approach even further. For interventions which consist of various workstreams, each devoted to different activities which beneficiaries can choose from, she felt it may be useful to develop nested theories of change. This means creating lower-level theories of change for specific project components. Such a solution could help evaluators to better distil the outcomes of particular components and show the workings of the project at a more granular level. 

The U-RLP evaluation was “inspired by the idea of learning through the process, providing not only accountability, but drawing out lessons that can be taken from the experience of innovation.” (Final evaluation report). Importantly, however, while the evaluators were inspired by the idea of learning, they felt constrained by the need to ensure project accountability.

To best capture evidence of early outcomes, the evaluation team proposed an elaborate mix-methods approach to the study. It used a combination of qualitative and quantitative methods for data collection and analysis, as well as a wide array of data sources summarised in the table below. 

Data collection methods
Source: Final evaluation report 

One of the challenges from the perspective of determining the project impact was securing an appropriate baseline, i.e. an overview of the situation prior to the intervention’s initiation. Even though the evaluation was started immediately, with no overall preparatory phase which would allow for development of research tools, the first evaluation measurements were delayed in relation to other activities.

The measurement of attitudes, aspirations and hopes for the project among young Dutch tenants was conducted after they had already moved into their apartments. It would have also been more valuable if the neighbourhood survey had been carried out at the time when the hostilities towards refugees were high. Consequently, the baseline measurements offered an already altered picture of the intervention, compared to its actual starting point.  

In light of the sensitive nature of the project, the implementation of the neighbourhood survey had to be approached with caution, specifically in relation to its language and distribution. The idea of sending the surveys out via post was eventually rejected, since it could have opened the results to hijacking and manipulation by groups hostile towards the asylum seekers’ centre in the neighbourhood. Instead, a face-to-face option was selected and implemented by trained researchers.

Mindful of the risk of overburdening refugees with research activities, during the research, the evaluation team decided to use the available NOA intake assessments, i.e. pre-existing quantitative baseline assessments completed by asylum seekers when joining the project. The initial assumption was that these would be later replicated for evaluation purposes to offer a comparison with the baseline assessments. However, repeating NOA assessment proved impossible during the evaluation. Asylum seekers perceived the surveys as a test and were likely to present a desired picture, which raised both ethical and validity questions for the evaluation. Therefore, a decision was made to pursue a qualitative approach to collecting data with asylum seekers through individual interviews.

The large component of individual interviews with foreigners activated considerations related to translation and interpretation, including the specificity of working with interpreters in sensitive, qualitative research. The evaluation team was aware of the power relations between interviewees and interpreters involved in such processes. For similar future studies, they would have therefore preferred involving community researchers. Not only would that reduce the power dynamic, but it would also increase the evaluation’s participatory character and produce additional opportunities from project beneficiaries employed as such researchers.    

It is clear that despite ethical, political, temporal, logistical or language-related limitations, the evaluation design allowed the team to gathered a wealth of data from different sources and give strong evidence backing for evaluation claims. 

In projects such as U-RLP, ethical and political considerations play a significant role in the evaluation. At the project outset, the vulnerability of the targeted population added a layer of complexity to the university ethical review. As recalled by the lead evaluator, the review was incredibly detailed and, consequently, time-consuming. In this respect, yet again, the project would have benefited from a development stage to account for that period.

Ethical considerations, among others about not burdening the asylum seekers with research activities, also influenced methodological changes, such as resignation from replicating the quantitative NOA assessments in favour of conducting qualitative individual interviews. They also played a part in the development and conduct of the neighbourhood survey. Considering the highly controversial subject, the survey questions had to be drafted with particular care so as not to stir controversy in the neighbourhood.

Political factors also had an influence on the project methodology. With respect to the survey, they influenced its distribution towards a face-to-face mode. The Dutch asylum authority’s refusal to provide the evaluation team with access to the asylum seekers’ centre prevented the researchers from talking to asylum seekers who did not benefit from the support offered as part of the U-RLP project. This meant that comparing the situation of U-RLP beneficiaries to Overvecht asylum seekers who did not receive support was not possible.

The U-RLP offers a rich source of insight for those embarking on an evaluation of a similar complex intervention related to serious social challenges. Many of those have been noted in previous parts, and below we additionally summarise our selection:

There should be time assigned for development of the project and its evaluation. The provision of such time would have allowed the evaluation team to recreate the theory of change, develop research tools and carry out appropriate baseline measurements prior to the initiation of actual project activities. Without such a phase, the baseline was compromised and some opportunities for capturing the project’s contribution to outcomes potentially lost, while the evaluation team were playing catch up from the word go. 

Theory-based evaluation and theory of change can be useful, yet possibly not ideal for innovative interventions. The selected approach worked in practice and offered a good opportunity for collaboration and inclusion of various points of view. The theory of change was used consistently and productively throughout the whole evaluation. At the same time, given the project’s complexity and with more time available, the evaluators would have liked to develop nested theories of change for particular project components. In view of the pilot character of the project, they would have generally preferred a more flexible approach, such as developmental evaluation, which does not impose a linear logic onto a dynamic innovative intervention. This would have helped to concentrate resources on generating insight for the project as to how and why things worked the way they did.

There is a need for significant resources. The ambitious goal of capturing early outcomes and impact should receive appropriate resources, both in terms of funding and time. In the space of the U-RLP project, budgetary shortcomings were compensated for with individual engagement of researchers. However, this cannot be a standard expectation for professionals who engage in evaluation. Time-wise, the funders interested in measuring impact should allow the evaluation to reach beyond the project implementation time.

Academic expertise is an asset. The U-RLP project could rely on the extensive expertise of its academic partner both at the level of the core evaluation team and the evaluation Advisory Board. U-RLP evaluation experience shows the importance of subject-matter expertise and research skills. These offer backing and add nuance to various elements of the research, be that the theory of change, methodological choices made throughout the study or its specific findings. This rich research experience allowed the team to develop a complex evaluation design, involving a dynamic and creative approach to data collection, to compensate for the lack of a counterfactual and to respond to various ethical and political challenges along the way.

The evaluator’s role can go beyond passing judgement on the project and their position may require balancing between different objectives and expectations. The U-RLP experience shows that in complex, multi-partner projects, evaluators need to be good communicators who are able to explain the meaning of various evaluation processes and, project-wise, bring all partners to a common denominator. Some tension was also revealed in the evaluator’s role between the need for accountability and the need for learning through the evaluation. This tension created a level of discomfort, and the need for accountability inevitably diverted resources from learning opportunities. With innovative interventions such as U-RLP, evaluations would have benefited from concentrated focus on generating insight and understanding about projects’ inner workings and achievements.