Background Medical training occurs during peak childbearing years. However, the intense workload, long work hours, and limited financial compensation are potential barriers to having children during this time. Here, we aimed to identify gender-based differences in beliefs and experiences of having children during graduate medical education. We hypothesized that both genders face significant challenges, but women are more likely to experience stressors related to work-family conflicts. Methods We administered an anonymous web-based survey to all trainees at an academic medical center. Primary outcomes were gender differences in beliefs and experiences of having children during training. Multivariate logistic regression was performed using independent variables of gender, specialty type (surgical vs. medical), and parental status. Results In total, 56% of trainees responded (60% women, 40% men; n = 435). Women were more often concerned about the negative impact of having children and taking maternity leave on their professional reputation and career. The majority of women expressed concern about the potential negative impact of the physical demands of their jobs on pregnancy. Among parents, women were more likely than men to be the primary caregivers on weeknights and require weekday childcare from a non-parent. Conclusions Women face greater work-related conflicts in their beliefs and experiences of having a family during graduate medical education. Trainees should be aware of these potential challenges when making life and career decisions. We recommend that institutions employ solutions to accommodate the needs and wellbeing of trainees with families while optimizing training and workload equity for all trainees. Electronic supplementary material The online version of this article (10.1186/s12909-018-1373-1) contains supplementary material, which is available to authorized users.