Catadioptric omnidirectional images exhibit serious nonlinear distortion due to the involved quadratic mirror. Conventional pinhole model-based methods perform poorly when directly applied to the deformed omnidirectional images. This study constructs a catadioptric geometry system to analyze the variation of the neighborhood of an object in terms of the elevation and azimuth directions in a spherical coordinate system. To accurately represent the distorted visual information, a parametric neighborhood mapping model is proposed based on the catadioptric geometry. Unlike the conventional catadioptric models, the prior information of the system is effectively integrated into the neighborhood formulation framework. Then the distortion-adaptive neighborhood can be directly calculated based on its measurable image radial distance. This method can significantly improve the computational efficiency of algorithm since statistical neighborhood sampling is not used. On the basis of the proposed neighborhood model, a distortion-invariant Haar wavelet transform is presented to perform the robust human detection and tracking in catadioptric omnidirectional vision. The experimental results verify the effectiveness of the proposed neighborhood mapping model and prove that the distorted neighborhood in the omnidirectional image follows a nonlinear pattern.