Background Global health diplomacy (GHD) is a burgeoning field bridging the priorities of global health and foreign affairs. Given the increasing need to mobilize disparate global health stakeholders coupled with the need to design complex public health partnerships to tackle issues of international concern, effective and timely cooperation among state actors is critical. Health Attachés represent this coordination focal point and are key diplomatic professionals at the forefront of GHD. Despite their unique mandate, little is published about this profession and the perspectives of those who work in the field. Methods Through purposive sampling, we performed in-depth qualitative interviews with seven Health Attachés: three foreign Health Attachés accredited to the United States and four U.S. Health Attachés accredited to foreign governments. Our interviews explored four key topics: the role and mission of Health Attachés, skills needed to perform GHD, examples of successes and challenges in accomplishing their respective missions, and suggestions for the future development of the diplomatic profession. Results We identified several lessons to apply to the growing field of GHD. First, GHD actors need to receive appropriate training to successfully negotiate the intersection of global health and foreign affairs. Participants suggested several areas of training that would benefit GHD actors: diplomacy and negotiation, applied science, and cross-cultural competency. Second, participants articulated the need for a career path for GHD practitioners, increased opportunities for on-the-job training and mentored experiences, and GHD competencies with defined levels of mastery that can be used in occupational evaluation and career development. Conclusions Our findings indicate that skills in diplomacy and negotiation, applied science, and cross cultural competency are essential for the statecraft of Health Attachés. Additionally, establishing a clear career pathway for Health Attachés is critical for future maturation of the profession and for fostering effective global health action that aligns public health and foreign diplomacy outcomes. Achieving these goals would ensure that this special cadre of diplomats could effectively practice GHD and would also better position Health Attachés to take the lead in advancing shared global health goals among nation states in a new era of twenty-first century diplomacy. Electronic supplementary material The online version of this article (10.1186/s12992-017-0316-7) contains supplementary material, which is available to authorized users.