Evaluations to objectively assess minimum competency are not routinely implemented for training and supervision in global mental health. Addressing this gap in competency assessment is crucial for safe and effective mental health service integration in primary care. To explore competency, this study describes a training and supervision program for 206 health workers in Uganda, Liberia, and Nepal in humanitarian settings impacted by political violence, Ebola, and natural disasters. Health workers were trained in the World Health Organization’s mental health Gap Action Programme (mhGAP). Health workers demonstrated changes in knowledge (mhGAP knowledge, effect size, d = 1.14), stigma (Mental Illness: Clinicians’ Attitudes, d = −0.64; Social Distance Scale, d = −0.31), and competence (ENhancing Assessment of Common Therapeutic factors, ENACT, d = 1.68). However, health workers were only competent in 65% of skills. Although the majority were competent in communication skills and empathy, they were not competent in assessing physical and mental health, addressing confidentiality, involving family members in care, and assessing suicide risk. Higher competency was associated with lower stigma (social distance), but competency was not associated with knowledge. To promote competency, this study recommends (1) structured role-plays as a standard evaluation practice; (2) standardized reporting of competency, knowledge, attitudes, and clinical outcomes; and (3) shifting the field toward competency-based approaches to training and supervision.