Abstract Background Programs encouraging medical student research such as Scholarly Concentrations (SC) are increasing nationally. However, there are few validated measures of mentoring quality tailored to medical students. We sought to modify and validate a mentoring scale for use in medical student research experiences. Methods SC faculty created a scale evaluating how medical students assess mentors in the research setting. A validated graduate student scale of mentorship, the Ideal Mentor Scale, was modified by selecting 10 of the 34 original items most relevant for medical students and adding an item on project ownership. We administered this 11-item assessment to second year medical students in the Johns Hopkins University SC Program from 2011 to 2016, and performed exploratory factor analysis with oblique rotation to determine included items and subscales. We correlate overall mentoring quality scale and subscales with four student outcomes: ‘very satisfied’ with mentor, ‘more likely’ to do future research, project accepted at a national meeting, and highest SC faculty rating of student project. Results Five hundred ninety-eight students responded (87% response rate). After factor analysis, we eliminated three items producing a final scale of overall mentoring quality (8 items, Cronbach’s alpha = 0.92) with three subscales: advocacy, responsiveness, and assistance. The overall mentoring quality scale was significantly associated with all four student outcomes, including mentor satisfaction: OR [(95% CI), p-value] 1.66 [(1.53–1.79), p