Abstract Background Evidence suggests the possible impact of ambient high temperature on fetal growth and birth outcomes. However, little is known about the relative impact of exposure to heat and cold and the possible vulnerable window during pregnancy. Methods Data on a total of 237,585 pregnant women from January 1st, 2001 to December 31st, 2010 were acquired from the Queensland Health, Australia. Daily data on meteorological factors, including ambient temperature, relative humidity, barometric pressure, and air pollutants, such as PM10, SO2, NO2, and O3, were obtained from relevant government agencies. This study was to examine the associations of maternal exposure to ambient temperature (high and low temperatures, in early vs. late pregnancy) with the duration of gestation and birth weight. Results A J-shaped association between minimum temperature at conception and duration of gestation was observed after adjusting for seasonality and other confounders. Compared to women who were exposed to the minimum temperature of 15–20 °C in the first gestational week, exposure to the minimum temperature of > 20 °C significantly increased the duration of gestation by 0.029 weeks (95% CI: 0.008, 0.049). A cumulative effect was found when exposure across the first four weeks was examined. There was an inverted U-shaped relationship between minimum temperature at delivery and the duration of gestation. Compared to women exposed to 15–20 °C, exposure to minimum temperature of > 20 °C and ≤ 10 °C was associated with a shortened gestation by 0.030 weeks (95% CI: -0.052, − 0.008) and 0.018 weeks (95% CI: -0.057, − 0.004), respectively. By contrast, an inverse relationship between maximum temperature and birth weight was observed. Compared to exposure to the maximum temperature of > 30 °C in the last week of pregnancy, maternal exposure to 20–25 °C and