|
| |
 |
|
| FIELD REPORT |
|
| Year : 2021 | Volume
: 19
| Issue : 1 | Page : 107-117 |
|
Development of a Tool to Assess Competencies of Problem Management Plus Facilitators Using Observed Standardised Role Plays: The EQUIP Competency Rating Scale for Problem Management Plus
Gloria A Pedersen1, Frezgi Gebrekristos2, Liyam Eloul3, Shannon Golden4, Mahmoud Hemmo5, Aemal Akhtar6, Alison Schafer7, Brandon A Kohrt8
1 Research Associate, Division of Global Mental Health, Department of Psychiatry and Behavioral Sciences, George Washington University, Washington, DC, USA
2 EQUIP Focal Person, The Center for Victims of Torture, Tigray, Ethiopia
3 Clinical Advisor, The Center for Victims of Torture, St. Paul, MN, USA
4 Research Associate, The Center for Victims of Torture, St. Paul, MN, USA
5 Research Associate, Department of Consultation-Liaison Psychiatry and Psychosomatic Medicine, University Hospital Zurich, University of Zurich, Zurich, Switzerland
6 Research Fellow, School of Psychology, University of New South Wales, Sydney, NSW, Australia
7 Technical Advisor, World Health Organization (WHO), Department of Mental Health and Substance Use, Geneva, Switzerland
8 Director, Division of Global Mental Health, Department of Psychiatry and Behavioral Sciences, George Washington University, Washington, DC, USA
| Date of Submission | 01-Oct-2020 |
| Date of Decision | 23-Dec-2020 |
| Date of Acceptance | 15-Feb-2021 |
| Date of Web Publication | 31-Mar-2021 |
Correspondence Address:
Brandon A Kohrt
Division of Global Mental Health, Department of Psychiatry and Behavioral Sciences, George Washington University, 2120 L St NW, Suite 600, Washington, DC 20037
USA
 Source of Support: None, Conflict of Interest: None
DOI: 10.4103/INTV.INTV_40_20
Problem Management Plus (PM+) is used to address mental health needs in humanitarian emergencies, including in response to COVID-19. Because PM+ is designed for non-specialist facilitators, one challenge is ensuring that trainees have the necessary competencies to effectively deliver the intervention and not cause unintended harm. Traditional approaches to evaluating knowledge of an intervention, such as written tests, may not capture the ability to demonstrate PM+ skills. As part of the World Health Organization Ensuring Quality in Psychological Support (EQUIP) initiative, we developed a structured competency rating tool to be used with observed standardised role plays. The role plays were designed to elicit demonstration of the key mechanisms of action for PM+. These role plays can be objectively rated by trainers, supervisors or other evaluators to determine facilitators’ competency levels in PM+. These competency assessments can highlight what skills require additional attention during training and supervision, thus supporting facilitators to fill gaps in competencies. The integration of role plays in training and supervision also allows organisations to establish quality control metrics for competency standards to deliver PM+. We describe lessons learned from piloting the PM+ competency role plays with the Center for Victims of Torture programme with Eritrean refugees in Ethiopia.
Keywords: competency, Ethiopia, emergencies, humanitarian, non-specialists, Problem Management Plus (PM+), psychological treatments, refugees, supervision, training
How to cite this article:
Pedersen GA, Gebrekristos F, Eloul L, Golden S, Hemmo M, Akhtar A, Schafer A, Kohrt BA. Development of a Tool to Assess Competencies of Problem Management Plus Facilitators Using Observed Standardised Role Plays: The EQUIP Competency Rating Scale for Problem Management Plus. Intervention 2021;19:107-17 |
How to cite this URL:
Pedersen GA, Gebrekristos F, Eloul L, Golden S, Hemmo M, Akhtar A, Schafer A, Kohrt BA. Development of a Tool to Assess Competencies of Problem Management Plus Facilitators Using Observed Standardised Role Plays: The EQUIP Competency Rating Scale for Problem Management Plus. Intervention [serial online] 2021 [cited 2023 Jun 7];19:107-17. Available from: http://www.interventionjournal.org//text.asp?2021/19/1/107/312725 |
| Introduction | |  |
With the proliferation of brief psychological interventions that can be delivered by non-specialists (Singla et al., 2017), there is a need to determine how to assure quality control in the provision of care (Jordans & Kohrt, 2020). To date, there has been considerable heterogeneity in how trainers, supervisors and organisations determine if a non-specialist is effectively trained and ready to deliver care safely (Kohrt & Bhardwaj, 2019). Without information on how non-specialists are trained and assessed, it is difficult to develop best practices for recruitment, training and supervision.
Competency refers to “the extent to which a therapist has the knowledge and skill required to deliver a treatment to the standard needed for it to achieve its expected effects” (Fairburn & Cooper, 2011, p. 373). To assess skills, Fairburn and Cooper (2011) suggest that “a role play-based method of assessment would be preferable. It would involve the trainee being the ‘therapist’ with a simulated patient who would enact a series of prepared clinical scenarios”. They also point out that, unfortunately, this technique is not consistently used to evaluate therapist competency. A recent review demonstrates gaps in the literature on implementing observed structured role plays for competency assessment and a lack of studies on the association of role play-assessed competency with client outcomes (Ottman et al., 2020). To address this gap, growing efforts in global mental health are now developing competency assessment tools and implementing this preferred role play assessment approach, with particular attention to non-specialists delivering psychological interventions (Kohrt et al., 2015a,b).
Ensuring Quality in Psychological Support (EQUIP, www.who.int/mental_health/emergencies/equip/en/) is a World Health Organization (WHO) initiative with numerous objectives, one of which is to develop competency assessment tools and guidance that can be used in training non-specialists to deliver psychological interventions (Kohrt et al., 2020a). By creating standardised competency assessment tools and procedures, there are opportunities for trainers, supervisors and organisations to compare outcomes of their trainings with similar trainings at other organisations or in other settings. This can ultimately help establish quality standards for delivering ethical and effective interventions (Jordans & Kohrt, 2020).
In this field report, we describe the process of developing and piloting a role play-based competency assessment tool for individual Problem Management Plus (PM+; World Health Organization, 2018). PM+ is a transdiagnostic intervention that incorporates multiple therapeutic techniques and is designed for delivery by non-specialists in humanitarian settings (Dawson et al., 2015). Although most psychological treatments delivered by non-specialists are adaptations of manuals developed for specialists, PM+ is designed from the ground-up with non-specialists in mind, which is reflected in the selection and description of therapeutic techniques (Dawson et al., 2015). Instead of using psychiatric diagnostic terminology, PM+ focuses on general concepts of stress and adversity, easily explained in non-stigmatising language to diverse communities. The focus on problem-solving addresses practical needs during a pandemic and complements the work done by humanitarian organisations, for example shelter, nutrition, livelihoods, education and healthcare.
PM+ is designed for rapid roll-out in humanitarian emergencies, with the intention that non-specialists can be trained in 10 days to deliver care with appropriate supervision. The intervention itself is brief: only five weekly 90-minute sessions. PM+ reduces depression with benefits maintained 3 months after treatment among women experiencing gender-based violence in Kenya and people living in a conflict-affected region of Pakistan (Bryant et al., 2017; Rahman et al., 2016, (2019)).
Below we outline the steps for developing the PM+ competency assessment tool. First, competencies for PM+ were identified through a manual review. Second, clinical staff with expertise in training on PM+ identified key competencies that could be feasibly evaluated in role plays. Next, we refined the competencies, rating scales and role play scripts for actors. Raters were trained to achieve acceptable inter-rater reliability. Finally, paraprofessionals working in refugee camps in Ethiopia were trained in PM+, and their competency in key PM+ skills was assessed with the structured role play tool. We conclude with reflections on how to improve use of the tool and its application in other settings.
Step 1: Identifying Competencies for PM+ by Reviewing the Intervention Manual
Development of the EQUIP competency tool for PM+ is modelled after the development of the Enhancing Assessment of Common Therapeutic factors (ENACT) tool and uses a similar structure for role play performance and scoring (Kohrt et al., 2015a,b). The ENACT includes structured role plays for actors and an assessment tool for raters. Each competency can be scored on different levels, and specific criteria (attributes) are provided for each level.
To identify key competencies, a review of manuals for PM+ and other non-specialist delivered psychological/psychosocial interventions was conducted (Pedersen et al., 2020). The review comprised creating a competency codebook that could be used across manuals and then identifying what competencies were included in particular interventions. For PM+, we focused on competencies related to the core mechanisms of action (Dawson et al., 2015). These included stress management, problem-solving, behavioural activation and interpersonal support. The competency items were reviewed and refined by an EQUIP network of experts, academics, researchers and field practitioners (including A.S. and reviewers listed in Acknowledgements). These competencies were originally named to align with psychotherapy treatment classes (e.g. behavioural activation, interpersonal support, problem-solving) rather than with a specific intervention. This was done to explore similar techniques across manualised interventions. For example, our review (Pedersen et al., 2020) found that problem-solving techniques in PM+ are also observed in the Friendship Bench (Chibanda et al., 2011), and behavioural activation techniques in PM+ have similarities with the Healthy Activity Programme (Anand et al., 2013). Initially, we identified 15 competencies in the individual PM+ manual ([Table 1]). Group facilitation competencies for Group PM+ have been addressed in a separate publication (Pedersen et al., under review).
Step 2: Selection of Competencies for PM+ Role Plays
Collaboration with Implementing Partners
Based on prior experiences with structured role plays, the team determined that to ensure timely and effective implementation of the competency role plays, including all 15 competencies related to PM+ in each trainee assessment would not be feasible. Therefore, the WHO EQUIP team collaborated with partners delivering PM+ in humanitarian settings to select and adapt key PM+ competencies likely to be most beneficial for trainees to practise. Competencies were selected based on those that could be assessed in brief (10–20 minutes) role plays during training and supervision.
The Center for Victims of Torture (CVT) partnered with WHO EQUIP to implement these activities from October 2019–March 2020 in Mai Aini and Adi Harush camps for Eritrean refugees in the Tigray region of Ethiopia. CVT has operated programming for Eritrean refugees in Ethiopia, including capacity development activities in mental health and psychosocial support (MHPSS) and direct services to clients since 2013. CVT’s primary intervention focuses on trauma rehabilitation through psychotherapy. However, PM+ fills a complementary role, helping refugees cope with psychosocial distress emerging from living in a setting of ongoing adversity, stressors and threat. All activities for this piloting received ethical approval from the Tigray Health Research Institute, National Research Ethics Committee of Ethiopia and George Washington University.
CVT has an experienced cadre of national psychotherapists and a large team of refugee paraprofessional psychosocial counsellors, with varying levels of tenure, training and experience in psychosocial interventions. CVT also has strong capacity development relationships with partner organisations, which allowed participation from a large group of lay practitioner trainees beyond CVT staff. This paraprofessional team has similar capacities to the staff of many other organisations working in humanitarian contexts. CVT staff were trained on PM+ and completed a translation of the PM+ intervention manual into the local language, Tigrinya. CVT’s lead trainer (F.G.) participated in a PM+ training-of-trainers and previously conducted a PM+ training in the context, which enabled him to identify PM+ competencies that needed more attention and structured feedback among previous PM+ trainees.
Another EQUIP partner team comprising the University of New South Wales (UNSW) and the University Hospital Zurich (USZ) are collaborating with local organisations in Jordan to implement PM+ for Syrian refugees. This team was involved in the early steps of adaptation. However, because of programme delays resulting from the COVID-19 pandemic in Jordan, we will focus on implementation learnings from the CVT programme in Eritrean refugee camps.
Prioritisation of Competencies
The CVT and UNSW/USZ teams started with the list of 15 PM+ competency items. With support from PM+ developers and the EQUIP team, CVT’s project clinical advisor (L.E.) and lead trainer (F.G.) selected eight competencies based on three criteria: first, they were considered most resonant for therapeutic change; second, based on prior experience, these competencies take longer for mastery; and, third, these competencies were valuable to guide trainers and supervisors when providing structured feedback to trainees. The eight competencies included six problem management competencies introduced in Session #2 of PM+, which are subsequently reviewed in Sessions #3–5; one stress management competency (deep breathing) introduced in Session #1 and then reviewed in Sessions #2–5; and one behavioural activation skill, introduced in Session #3 and reviewed in Sessions #4–5.
Development of Scoring Levels and Attributes
The scoring framework, adapted from ENACT, included four levels. Attributes for these four levels of each competency item were based on information extracted from the PM+ manual and feedback from trainers, supervisors and researchers with field experience implementing PM+. [Table 2] presents the eight selected competencies and attributes, their corresponding skill level and the CVT and UNSW/USZ team adaptations. At this point, the competency item names were revised to match terminology in the PM+ manual.
The CVT team assessed each item and its attributes for comprehensibility, acceptability, relevance, completeness and technical equivalence (based on Van Ommeren et al., 1999). All selected competencies required moderate adaptation to make them appropriate for the context of CVT’s programmes with Eritrean refugees in Ethiopia. The majority of adaptations related to terminology, such as removing potentially confusing or stigmatising language. This increased comprehensibility and facilitated ease of conceptual and literal translation into Tigrinya. Additionally, skills that may seem “basic” in some contexts (such as using a calendar) required extensive coaching and practising for the CVT Eritrean refugee programme, where information is not commonly organised in this way. All competency items were adapted to utilise local PM+ terminology reflected in the PM+ Tigrinya intervention manual.
Step 3: Development of PM+ Role Plays for Selected Competencies
Structure of Role Plays
The first author (G.P.) drafted two role play scripts for the actor-client and trainee based on the eight competencies selected. The scripts include instructions for the PM+ facilitator (the trainee) and the actor-client. The PM+ trainee instructions include a brief overview that specifies which PM+ session and what techniques they are expected to cover within the 10–20-minute role plays. To cover a range of techniques across several sessions, we designed the role play scripts into a set of acts or scenes, which cover key components of a session rather than an entire session. Instructions for the actor-client include a brief narrative of the client background (e.g. name, age, location, family and problems) and a list of specific prompts and verbal and physical responses for the actor to cue them to demonstrate their PM+ competencies. Trainees are informed that each role play only addresses a subset of skills, and they are not expected to demonstrate all PM+ skills from a particular session in a single role play.
The first role play covers PM+ Session #2 (Managing Problems). It is 15 minutes long and assesses four competency items related to Managing Problems steps 1–4. The second role play covers PM+ Session #3 and #4 (Managing Stress, Managing Problems, and Get Going, Keep Doing). It assesses four competencies involved in reviewing Managing Stress, Managing Problems, and behavioural scheduling of the Get Going, Keep Doing strategy. This second role play is intended to be performed in 15 minutes and uses two acts: one for Session #3 and one for Session #4. CVT translators produced Tigrinya versions of all materials, using back translation and group consultation to guide rounds of revisions (the role play scripts with instructions for actors in English and Tigrinya can be found in the Supplemental Online Materials.)
Contextualisation of Role Plays
The CVT team and the UNSW/USZ team contextualised the first and second PM+ role play actor scripts for their different settings (see [Table 3] for a comparison of adaptations). With the CVT programme for Eritrean refugees, appropriate allocation of observable skills to the four-level rating categories required significant debate. Adaptations included changing biographic details, symptoms reported by clients, coping mechanisms including religious practices, maladaptive coping including harmful substance use and self-isolation and relevant problems such as employment, family relationships and insecurity in camp settings. | Table 3 Comparison of PM+ Client-Actor Role Play Script Contextualisation
Click here to view |
Step 4: Training Raters and Actors for PM+ Role Play Competency Assessments
Six CVT staff were trained to be both actors and raters, so they could perform either function when completing competency assessments. The six actors-raters were two national staff counsellors with masters- and bachelor-level training and over 3 years of experience providing counselling, and four refugee paraprofessional psychosocial counsellors without formal education related to counselling, but with 2–6 years of experience and training with CVT. They had completed 7 days of training on the ENACT standardised role plays and rating tools to assess foundational helping skills. They then participated in 2 days of training focused on the PM+ competency assessment.
In the training of raters and actors, the trainers used role plays followed by group discussion to introduce the content of each PM+ competency area, including the behavioural attributes associated with each rating level. The group watched a training video of a mock session and independently scored the session. They rewatched the video, pausing to discuss each area and seek consensus on the appropriate score. This process was repeated to practise rating. Watching, rewatching and discussing the videos was a valuable method to increase overall understanding and improve the reliability of scoring across raters. To train on the acting component, the trainer introduced the role play content, and the actor-clients were given time to independently read the scripts and instructions. The team practised the scripts and used the PM+ competency rating scales, taking turns playing all roles (actor-client, rater and the trainee who is being rated). After practising, facilitated feedback sessions were conducted and focused on the individual performance of actors and areas to improve acting and rating consistency across individuals.
After training, the raters used the PM+ rating tool with a high level of inter-rater reliability (intraclass correlation coefficient, ICC = 0.725). At times, there were disagreements or ambiguity about ratings for levels three and four, but discussion effectively led to agreement about the conceptual distinctions. It was important that the actor-raters had already completed ENACT training, so they understood their roles, how to act out scripts, how to use the format of the tool and the conceptual distinctions between rating levels. This is also important because any rater assessing PM+ skills would also need to assess foundational helping skills with ENACT.
Step 5: Piloting the Role Play Competency Assessment in a PM+ Training
Although the EQUIP treatment-specific competency assessments were initially planned only for post-training and post-supervision, this training was an opportunity to also explore PM+ role play competency assessment pre-training. CVT, with a partner organisation, piloted the PM+ assessments before and after a 2-week PM+ training with a group of 35 refugee staff. CVT selected trainees based on their current roles providing services to the intended beneficiary populations. The prospective trainees were given information about the research study and were given the option to participate. If they chose not to participate in the research, they would still be able to receive the PM+ training and deliver the PM+ intervention. All participants were provided full information about the study, including discussing the risks of stress or other types of distress from participating in the competency assessments. All participants completed a full informed consent process.
The assessments were implemented by a team of nine CVT staff, including the six staff trained as actor-raters. CVT’s EQUIP focal person (F.G.) and two psychosocial supervisors coordinated the assessment process and monitored the actors and raters by observing a selection of assessments. Each trainee was assessed. They were invited into a room where the rater and an additional observer were waiting. The client-actor then entered, and the assessment began.
Pre-training Role Play Assessments
In pre-training assessments, most trainees quickly ran out of questions or other ways to move the session forward in the role play. Many trainees focused on providing support, offering advice and recommendations to the actor-client. All trainees, except one, spent fewer than 5 minutes in the role play. Most trainees had generally low competence levels, typically being scored at a level one or two in most areas. However, there was identifiable variation, with some skill areas having higher average scores. This was particularly true in areas that may be related to shared cultural expressions, such as norms for eye contact or postures, resulting in trainees performing relatively well in nonverbal communication.
On the other hand, areas that differ from common social behaviour or are more technical resulted in lower scores across trainees; for example most trainees provided a lot of advice, struggled to validate clients’ feelings, were unable to assess the risk of harm and did not adequately explain confidentiality. There were also individual variations, and a few trainees more consistently displayed basic or even advanced skills in these areas. This may be explained by their tenure with nongovernmental organisations, where they may have been previously exposed to similar concepts. This suggests that such qualities could potentially be identified during recruitment of new MHPSS providers.
Modifying PM+ Training Based on Competency Assessment Findings
A critical objective of EQUIP is to promote a competency-based training approach rather than traditional models of training that assume trainee groups have minimal or no competency of the training content. This pilot produced initially promising feedback. The pre-training competency assessment immediately provided information to PM+ trainers, allowing them to design and deliver more tailored training. CVT trainers used several strategies to modify the PM+ training based on the trainees’ demonstrated competencies. Trainers adjusted their daily plans to dedicate additional time for emphasis on particular skills the group was lacking. They added more in-depth explanations, hands-on approaches or relied more heavily on demonstration role plays to specifically target competencies that were weaker amongst the trainee cohort. This included some foundational helping competencies assessed through the ENACT role plays, a gap which may otherwise have been unidentified, potentially contributing to poorer training outcomes, and ultimately, lesser quality PM+ services. Among the PM+ competencies, Managing Problems was challenging for trainees.
To adjust to varying competence levels displayed among trainees, trainers formed practice groups with trainees of different skill levels. Each group focused on a particular competency so that they could support one another with feedback. With a smaller group of trainees, trainers could do individualised coaching tailored to trainees’ particular needs as assessed pre-training. Overall, CVT trainers felt that the knowledge they gained about the trainees through the pre-training assessments had increased their capacity to address trainees’ areas for growth and tailor the training delivery to target these competencies.
Post-Training Assessments
In the post-training assessments, the trainees demonstrated noticeable improvements. On average, trainees improved by at least one level in 6.5 of the eight PM+ areas assessed. All trainees improved in at least one competency area, and 31% improved in all eight areas. However, the standardised time allotted for the role plays created a barrier to completing a comprehensive assessment of the full range of trainees’ skills. Trainees were given instructions on which PM+ skills and topics to address with the client. Still, the 15 minutes for each PM+ role play was insufficient for the trainees to demonstrate all eight competencies cumulatively. For 35 trainees completing 45 minutes of role play assessments each (not including introduction and debriefing time), the pre-training assessments took 4 days, and the post-training assessment took just over 5 days. From the CVT team’s observations, this amount of time was not sufficient to devote to these assessments, suggesting modifications would be required.
All trainees participated in an individual informed consent process in which the purpose of the assessments and the research study was carefully explained; however, some trainees were visibly nervous during the role plays and providing a brief orientation about the assessment process improved their confidence. The team felt it could have been a more positive experience for the trainees if they were given time to prepare (15–20 minutes). For example, a break after the orientation could offer them space to think about what to say to the client and increase levels of calmness. Following the assessment, several trainees directly asked how they had performed and whether they had “passed the examination” or “covered all the items”. To debrief and provide support for the trainees, the project team offered additional explanations about the purpose of the role plays. Following this, trainees’ engagement in the process grew, and some expressed support for the evaluation method. Several trainees stated they appreciated the interactive experience and that it helped their learning. Also, limiting pre-training role plays to only common factors assessed with ENACT role plays and not including PM+ specific skills at pre-training could reduce the maximum allowable role play time from 45 minutes to 10 minutes per trainee. Having briefer assessments prior to training, when trainees are unfamiliar with the process, may help reduce distress and increase the feasibility of scaling up the process.
An additional concern was that, as the trainees are from the same population as the client-actors, aspects of the role play scripts may be triggering, which is also a concern for delivery of PM+. CVT took steps to help prevent this, including careful review and revision of the role play scenarios with local advisors and mentioning in their brief orientation that they could stop the role play at any time. There was also clinical support available during the evaluations in case any trainee experienced significant distress during a role play. Each trainee was given 5–10 minutes of individual debriefing following the evaluation, which gave the opportunity to check the trainee’s emotional state, normalise their reactions and give feedback on performance. While these problems did not occur during this pilot, it could be considered for future use of role play scripts.
| Discussion | | |
We have described the process of developing, adapting and piloting a competency assessment tool for PM+ based on a field feasibility study with CVT’s programmes for Eritrean refugees in Ethiopia. The process involved extracting a range of skills needed from the PM+ manual, reviewing these skills with experts in PM+, selecting priority competencies that could be feasibly assessed in brief role plays, adapting materials to the local context and training staff to perform as clients and use the rating tool. This was followed by piloting the tool in a PM+ training of 35 trainees.
The major benefit identified was the impact on trainers’ ability to tailor their trainings based on pre-training competency assessments. Additionally, the post-training assessments will help supervisors decide what to highlight to build competency during the supervised practice phase of PM+. The process also yielded several lessons for using the tool in other settings (see [Table 4] for recommendations). Moreover, the process highlighted the benefit of working with stakeholders directly implementing PM+ to ensure the recommended process is feasible and acceptable to organisations, trainers and supervisors. The role play scripts, assessment tool and guidance were draft versions of EQUIP materials, and the lessons learned are being used to revise the platform materials. | Table 4 Recommendations for Implementation of PM+ Competency Assessments
Click here to view |
One issue is the degree to which competency tools can be adapted and contextualised. Competency assessment tools need to be flexible based on who is delivering the intervention, where it is being delivered and to whom it is delivered. Demonstration of skills should fit culturally appropriate ways of expressing emotion and providing support. Similarly, flexibility is needed in treatment-specific tools to ensure the tool reflects face validity with local MHPSS practitioners. The highly consultative and consensus-based adaptation process requires ensuring relevant perspectives are engaged and empowered in decision-making. Although the professional clinical staff involved had significant experience and skills, they were reticent to challenge or make changes to an existing tool, which had been developed by “experts”.
Piloting the tools with the client-actors and raters from the implementing organisations provided vital information for adaptation to ensure clarity and ease of use. As these tools are meant to be usable by paraprofessionals, we advise having paraprofessionals on the adaptation team or piloting the tool with the actors and raters who will be using it. For treatment-specific competency role plays, we found that actors and raters need to be familiar with the specific intervention and ideally trained in it. Additionally, raters should have experience delivering MHPSS interventions, receive sufficient interactive training on the tools and have ongoing close monitoring by a supervisor. Although this pilot used trained paraprofessional actors and raters, another approach is to have programme staff with prior expertise (e.g. trainers and supervisors) take on the responsibilities of acting and rating. This use of trainer and supervisors in the acting/rating roles is how the ENACT has been used in other settings (Kohrt et al., 2018, Kohrt et al., 2020b).
Another concern was feasibility. As indicated in this pilot, the time and energy invested in implementing the competency assessments suggest this approach is well-suited to integration into the usual 10-day PM+ training. The ENACT has demonstrated feasible integration into pre- and post-training assessments, for example assessment of 20 trainees in 90–120 minutes with four to five actors who also serve as raters (Kohrt et al., 2018, Kohrt et al., 2020b). For the PM+ competency role plays, the feasibility requires refinement of the process. The CVT team concluded there was insufficient time to assess eight PM+ competencies with attention to quality and ensuring a positive trainee experience. One future option may be to reduce the number of treatment-specific competencies prompted in the role plays and split the corresponding assessment item into smaller pieces. For example, if a single competency item covers multiple treatment-specific steps, it can be broken down into two or three competencies with respective attributes. This process may allow the trainee to focus on fewer key competencies while still providing the trainer and supervisor with rich information from the assessment tool. Other key competencies not included in post-training assessments may be included in supervisor sessions.
To most effectively integrate the competence assessment results into a PM+ training and supervision with large groups of trainees, sufficient time should be allocated between the end of pre-training assessments and the start of training (e.g. 1 day). Similarly, time should also be allocated after reviewing post-training assessments to prepare for the supervision and practice phase. These periods are essential to review trainee scores and determine how to best adjust the training and supervision plans accordingly. With a large training, such as this pilot, a few hours are inadequate to change schedules, add new activities or modify presentations, handouts or other training materials.
Moreover, pre-training assessments must include ENACT to assess foundational helping competencies. For training novices in the MHPSS field, this information is invaluable in customising the training and allocating time to foundational helping skills and PM+ skills. A training of PM+ in Nepal began with ENACT competency assessment preceding a foundational helping skills training to achieve basic competencies, then a re-evaluation of foundational skills after the basic training and before the PM+ training (Sangraula et al., 2020). This allowed PM+ trainers to build core skills before proceeding to PM+ treatment-specific techniques.
Although PM+ assessments were conducted prior to the training in this pilot, it is advisable to conduct ENACT pre- and post-training and limit treatment-specific competency role plays (e.g. PM+ role plays) to only post-training assessment. A treatment-specific role play prior to training may be more burdensome than beneficial because non-specialist trainees may not have experience with treatment-specific skills. Also, given that PM+ role plays are considerably lengthier (20-30 minutes per trainee) than the ENACT foundational helping skills role plays (10 minutes), restricting treatment-specific competency role plays to post-training assessment will be most effective for time management. Some of the observed distress during pre-training assessment could also be mitigated by reducing pre-training assessment from 45 minutes to 10 minutes. Other approaches could be to integrate foundational and PM+ skill assessments throughout the training so that trainers can rate-as-they-go, which is being piloted in other studies now to determine feasibility. Allowing trainers to rate-as-they-go could lead to competency-based training approaches with flexible schedules for training content delivery (Kohrt et al., 2020a). It would require a trainer experienced in competency-based training to manage a continuously modifiable training programme.
Limitations
This study represents preliminary data on the implementation of PM+ competency assessments among Eritrean refugees in Ethiopia. Therefore, we cannot generalise the findings across humanitarian settings and different organisations. It will be important to assess this tool’s reliability and validity across various settings to support a more generalised evaluation of the tool. Given the ordinal make-up of the scale and use of objective raters, item response theory analysis may offer critical information on item discrimination and stability, which may also support the selection of key competencies for role play assessments at post-training and post-supervision (Cole et al., 2018; Raykov & Marcoulides, 2018). Key competencies included in the PM+ tool have been identified based on research, clinical and training experiences of the authors. Further research could assess whether strengthening of these key competencies among trained helpers are associated with positive client outcomes.
This study has not addressed other potential benefits of using the PM+ competency assessment tool, including potential cost–utility related to competency-based training and supervision for PM+. Costs incurred implementing PM+ competency-based training, including preparation time for adapting competency assessments and training raters and actors, should be measured alongside benefits received by the trained helpers and clients receiving the care compared to implementation as usual.
| Conclusion | | |
The CVT pilot was an important learning for the EQUIP platform, demonstrating the benefit and challenges of using a competency-based approach to training and assessment with structured role plays. Future work should consider potential modifications of the PM+ competencies prioritised for assessment during training and introduce additional role plays to assess other competencies during PM+ supervision sessions. It will be important to generate evidence for which competencies produce positive client outcomes and focus on these skills in role plays. High-quality training that can achieve competency among trainees is necessary but not sufficient to ensure effective care delivery. Therefore, competency-based training and supervision are part of a constellation of strategies needed for quality care.
Acknowledgements
Authors thank the CVT Ethiopia team who was instrumental to the implementation of these activities, with particular thanks to Samson Isaak, Afewerki Abrahaley, Eshete Teshager, Mulalem Fisseha, Dejen Yewhalaw, Fisseha Micheale, Esayas Kiflom, Maki Katoh and the CVT staff who are participating in the EQUIP programme. Authors also thank the reviewers of the PM+ competencies: Naser Morina, Richard Bryant, Katie Dawson, Sarah Watts, Stephanie Skavenski and Kenneth Carswell.
Financial support and sponsorship
This work is led by the World Health Organization through a grant from USAID. B.A.K. is supported by the U.S. National Institute of Mental Health (R01MH120649). The authors alone are responsible for the views expressed in this article and they do not necessarily represent the views, decisions or policies of the institutions with which they are affiliated.
Conflicts of interest
There are no conflicts of interest.
| References | | |
| 1. | |
| 2. | Bryant R. A., Schafer A., Dawson K. S., Anjuri D., Mulili C., Ndogoni L., Koyiet P., Sijbrandij M., Ulate J., Harper Shehadeh M., Hadzi-Pavlovic D., van Ommeren M. (2017). Effectiveness of a brief behavioural intervention on psychological distress among women with a history of gender-based violence in urban Kenya: A randomised clinical trial. PLoS Medicine, 14(8), e1002371. https://doi.org/10.1371/journal.pmed.1002371  |
| 3. | Chibanda D., Mesu P., Kajawu L., Cowan F., Araya R., Abas M. A. (2011). Problem-solving therapy for depression and common mental disorders in Zimbabwe: Piloting a task-shifting primary mental health care intervention in a population with a high prevalence of people living with HIV. BMC Public Health, 11, 828. https://doi.org/10.1186/1471-2458-11-828 |
| 4. | Cole M., Bandeen-Roche K., Hirsch A. G., Kuiper J. R., Sundaresan A. S., Tan B. K., Schleimer R. P., Kern R. C., Schwartz B. S. (2018). Longitudinal evaluation of clustering of chronic sinonasal and related symptoms using exploratory factor analysis. Allergy, 73 (8), 1715-1723. https://doi.org/10.1111/all.13470 |
| 5. | Dawson K. S., Bryant R. A., Harper M., Tay A. K., Rahman A., Schafer A., van Ommeren M. (2015). Problem Management Plus (PM+): A WHO transdiagnostic psychological intervention for common mental health problems. World Psychiatry, 14(3), 354-357. https://doi.org/10.1002/wps.20255 |
| 6. | Fairburn C. G., Cooper Z. (2011). Therapist competence, therapy quality, and therapist training. Behaviour Research and Therapy, 49(6), 373-378. |
| 7. | Jordans M. J. D., Kohrt B. A. (2020). Scaling up mental health care and psychosocial support in low-resource settings: A roadmap to impact. Epidemiology and Psychiatric Sciences, 29, e189. https://doi.org/10.1017/S2045796020001018 |
| 8. | Kohrt B. A., Jordans M. J. D., Rai S., Shrestha P., Luitel N. P., Ramaiya M., Singla D. R., Patel V. (2015a). Therapist competence in global mental health: Development of the Enhancing Assessment of Common Therapeutic factors (ENACT) rating scale. Behaviour Research and Therapy, 69, 11-21. http://dx.doi.org/10.1016/j.brat.2015.03.009 |
| 9. | Kohrt B. A., Ramaiya M. K., Rai S., Bhardwaj A., Jordans M. J. D. (2015b). Development of a scoring system for non-specialist ratings of clinical competence in global mental health: A qualitative process evaluation of the Enhancing Assessment of Common Therapeutic Factors (ENACT) scale. Global Mental Health, 2, e23. https://doi.org/10.1017/gmh.2015.21 |
| 10. | Kohrt B. A., Mutamba B. B., Luitel N. P., Gwaikolo W., Onyango Mangen P., Nakku J., Rose K., Cooper J., Jordans M.J., Baingana F. (2018). How competent are non-specialists trained to integrate mental health services in primary care? Global health perspectives from Uganda, Liberia, and Nepal. International Review of Psychiatry, 30(6), 182-198. https://doi.org/10.1080/09540261.2019.1566116 |
| 11. | Kohrt B. A., Bhardwaj A. (2019). Training and supervision. In Stein D. J., Bass J. K., Hofmann S. G. (Eds.), Global Mental Health and Psychotherapy (pp. 47-65). Academic Press. |
| 12. | Kohrt B. A., Schafer A., Willhoite A., van’t Hof E., Pedersen G. A., Watts S., Ottman K., Carswell K., van Ommeren M. (2020a). Ensuring Quality in Psychological Support (WHO EQUIP): Developing a competent global workforce. World Psychiatry, 19(1), 115-116. https://doi.org/10.1002/wps.20704 |
| 13. | Kohrt B. A., Turner E. L., Rai S., Bhardwaj A., Sikkema K. J., Adelekun A., Dhakal M., Luitel N.P., Lund C., Patel V., Jordans M. J. D. (2020b). Reducing mental illness stigma in healthcare settings: Proof of concept for a social contact intervention to address what matters most for primary care providers. Social Science & Medicine, 250, 112852. https://doi.org/10.1016/j.socscimed.2020.112852 |
| 14. | Ottman K. E., Kohrt B. A., Pedersen G. A., Schafer A. (2020). Use of role plays to assess therapist competency and its association with client outcomes in psychological interventions: A scoping review and competency research agenda. Behaviour Research and Therapy, 130, 103531. https://doi.org/10.1016/j.brat.2019.103531 |
| 15. | Pedersen G. A., Lakshmin P., Schafer A., Watts S., Carswell K., Willhoite A., Ottman K., van’t Hof E., Kohrt B. A. (2020). Common factors in psychological treatments delivered by non-specialists in low- and middle-income countries: Manual review of competencies. Journal of Behavioral and Cognitive Therapy, 30(3), 165-186. https://doi.org/10.1016/j.jbct.2020.06.001 |
| 16. | Pedersen G. A., Sangraula M., Shrestha P., Laksmin P., Schafer A., Ghimire R., Luitel N.P., Jordans M.J.D., Kohrt B. A. (under review, 2020). Development of the Group facilitation Assessment of Competencies Tool (GroupACT) for group-based mental health and psychosocial support interventions in humanitarian emergencies and low-resource settings. Journal on Education in Emergencies. |
| 17. | Rahman A., Hamdani S. U., Awan N. R., Bryant R. A., Dawson K. S., Khan M. F., Azeemi M.M.U.H., Akhtar P., Nazir H., Chiumento A., Sijbrandij M., Wang D., Farooq S., van Ommeren M. (2016). Effect of a multicomponent behavioral intervention in adults impaired by psychological distress in a conflict-affected area of Pakistan: A randomized clinical trial. JAMA, 316(24), 2609-2617. https://doi.org/10.1001/jama.2016.17165 |
| 18. | Rahman A., Khan M. N., Hamdani S. U., Chiumento A., Akhtar P., Nazir H., Nisar A., Masood A., Din I. U., Khan N.A., Bryant R.A., Dawson K. S., Sijbrandij M., Wang D., van Ommeren M. (2019). Effectiveness of a brief group psychological intervention for women in a post-conflict setting in Pakistan: a single-blind, cluster, randomised controlled trial. The Lancet, 393 (10182), 1733-1744. https://doi.org/10.1016/S0140-6736(18)32343-2 |
| 19. | Raykov T., Marcoulides G. A. (2018). A course in item response theory and modeling with Stata. Stata Press. |
| 20. | Sangraula M., Turner E. L., Luitel N.P., van’t Hof E., Shrestha P., Ghimire R., Bryant R., Marahatta K., Van Ommeren M., Kohrt B.A., Jordans M. J. D. (2020). Feasibility of Group Problem Management Plus (PM+) to improve mental health and functioning of adults in earthquake-affected communities in Nepal. Epidemiology and Psychiatric Sciences, 29, e130. https://doi.org/10.1017/S2045796020000414 |
| 21. | Singla D. R., Kohrt B. A., Murray L. K., Anand A., Chorpita B. F., Patel V. (2017). Psychological treatments for the world: Lessons from low- and middle-income countries. Annual Review of Clinical Psychology, 13(April), 149-181. https://doi.org/10.1146/annurev-clinpsy-032816-045217 |
| 22. | Van Ommeren M., Sharma B., Thapa S., Makaju R., Prasain D., Bhattaria R., de Jong J. T. V. M. (1999). Preparing instruments for transcultural research: Use of the translation monitoring form with Nepali-speaking Bhutanese. Transcultural Psychiatry, 36(3), 285-301. |
| 23. | |
[Table 1], [Table 2], [Table 3], [Table 4]
|
Search Pubmed for