WORK PACKAGE 5:
Evaluation and pilot outcome assessment
OBJECTIVES:
The main objectives of this package are the following:
- To develop an evaluation strategy that includes a credible and feasible research design and analysis plan
- To coordinate the evaluation process at project level
- To ensure that evaluation data collection is carried out according to a common methodology across all pilot sites
- To analyze the results, mainly improvements of resilience capacity and mental wellbeing from quantitative and qualitative perspectives.
- To estimate the economic impact of the intervention at population level in mid-long term
Evaluation will be organized in three main phases:
- start-up scoping and preparatory work
- detailed evaluation specification, planning and methods development
- outcome assessment and reporting
Initial planning will start early to align with and, as appropriate, influence the resilience program development. Data collection across pilot sites will be organized according to established measurement points. Outcome and impact assessment of the intervention will be performed considering the point of view of all stakeholders, for which appropriate methods and indicators will be defined.
MAIN TASKS:
The evaluation will start with the agreement of the minimum dataset to be evaluated across all pilot sites. The evaluation approach will provide an overall methodological approach for the project containing the evaluation objectives (indicators), criteria (performance measures), evaluation design, data gathering process and analysis data process. Impact and effectiveness on the target groups (teachers, families, youths and school/community) will be assessed by using qualitative and quantitative analysis based on the indicators previously described in table 4.
The evaluation framework will also include a methodology to monitor the progress of the project ensuring the achievement of the milestones and outcomes defined. This protocol will require: 1) identification of performance indicators; 2) development of evaluation tools (e.g., questionnaires, activities, surveys); 3) data collection, analysis and interpretation; 4) generation, review and dissemination of appropriate reports; 5) development of action plans based on relevant data; 6) implementation of action plans.
All individual participants will need to sign the informed consent in order to be enrolled in the study. Parents/legal representatives of the youths will be responsible for the informed consent. Youths will have to assent to take part in the study as well.
Exclusion criteria: The legal representatives and youths are free to decline to participate in the project and this statement will be included as an exclusion criteria. At the same time, youths that consent to take part in the intervention (with the signed consent of their legal representatives) can withdraw at any time without providing any justification and will decide if their data is analyzed. In both cases (refusal or withdrawal), youths’ data will not be neither collected nor analyzed.
A data manager will be nominated in each pilot site responsible for: ensuring adequacy, accuracy and legitimacy of data collected; devising and implementing efficient and secure procedures for data management. In addition, a DPO will be nominated in UPRIGHT. Webinars and regular virtual meeting will be scheduled to monitor the data collection and management and support data managers in any issue that might arise.
In UPRIGHT, processed data will be result of quantitative and qualitative analysis. Depending on the evaluation purpose, raw data (surveys, questionnaires, records of focus groups and interviews) will be manipulated accordingly. This data processing will be carried out following the data protection regulation No 2016/679.
Prior to commencement of the resilience program implementation, the defined baseline data will be gathered by appropriate means in each pilot site. Appropriate means are those defined in the evaluation plan (defined in Task 5.1) and will consider standardized instruments, questionnaires, interviews, observations, data from logging software, etc. Quantitative analyses will be performed for each as subgroup analysis (target groups, intervention/control) as well as comparative analyses between the pilot sites for primary outcomes.
The baseline evaluation will be carried out before the intervention period commences. This evaluation will be based on the analysis of quantitative data collected by means of validated questionnaires completed by youth, families and teachers, and school records such as number of school dropouts, number of bullying cases etc. The questionnaires will be paper basis or online.
Data on early outcomes and intermediate effects will be gathered by the means, metrics and instruments defined in the evaluation methodology (defined in Task 5.1). Follow up evaluation process will be structured in two time points according to the two waves: first wave follow up at M21 and second wave follow-up at M30 (learning from the first wave implementation experience will improve the second waves´ performance). In this two evaluation phases, quantitative and qualitative data will be analyzed.
Quantitative analysis will be performed for each as subgroup analysis (target groups, intervention/control) as well as comparative analysis between the pilot sites for primary outcomes (mental wellbeing) and for some selected follow up outcomes (resilience capacity, quality of life, school drop-outs and absenteeism, bullying cases and substance use). Qualitative analysis through semi-structured interviews with schools (teachers and coordinators) and families, and focus groups with youths belonging to the intervention group will be performed to identify differences between sites and detect improvements in short term of mental wellbeing in youths (from baseline to follow up outcomes).
It will be built a Discrete Event Simulation (DES) model to estimate the impact of the intervention at the population level. Steps:
- Definition of the natural history of mental disorders among youths. This will be addressed in two phases: (i) early phase in which youths are at risk of imbalance in their mental wellbeing producing conflicts in the school environment and (ii) a more advanced stage in which these youths are seeking assistance in the health services, in particular in the psychology and psychiatry services.
- Obtaining simulation parameters for feeding the model from various information sources to define the current epidemiological scenario. This will include: literature review, analysis of incidents and other information from the school environment, analysis of contacts of youths with mental health problems with the health services, especially with the ambulatory psychology and psychiatry services and hospital psychiatric care.
- Construction of a risk score based on the scales analyzed. Then, the information from the scales will be integrated in a unique score measuring the probability of developing mental disorders.
- Construction and validation of the simulation model that represents the current epidemiological situation. DES technique will be used. -Estimation of the impact of the intervention at the population level by incorporating in the simulation model the results from the pilot site and the calculated risk score.
At the end of pilot operation, data will be collected and analyzed following the methodology defined in the evaluation framework.
Gathered data from self-reported questionnaires will be used for carry out a comprehensive statistical analysis in order to summarize and compare the results for the primary and final outcomes for the different pilot sites. Additionally, semi-structured interviews with schools and families, and focus groups with youths of the intervention group will be performed to evaluate satisfaction and other subjective matters.
A centralized analysis of the social, economic, health/mental well-being and satisfaction outcomes to achieve a complete understanding of each intervention and their impact. All results will be compiled, described, interpreted and discussed in a final report.
Articles for publication in peer reviewed scientific journals will be written based on the results of the project. In order to coordinate and manage drafting and submission of articles, and authorship, consortium partners will develop and agree on UPRIGHT publication policy.