The relative vibration state of vehicle and rail has an important influence on the measurement of the track geometry. The defects of the traditional method of measuring the relative vibration state of vehicle and rail are analyzed. A method for measuring the relative vibration state of vehicle and rail based on vision is proposed. The world coordinate system was constructed with tracks, and the vehicle coordinate system was constructed with body. Taking into consideration lens distortion, the non-linear model of cameras was set up. Calibrating the camera and body based on robot hand-eye calibrations, and acquire the internal and external parameters of the cameras. According to the characteristic of the vehicle body vibration, a calculation method of vehicle vibration displacement was deduced based on binocular machine vision. Using experiment platform design verification experiment, the vibration displacement of the vehicle body is in agreement with the real value. As the speed increases, the vibration displacement error increases. The correctness and feasibility of the proposed method are verified. It provides a method for measuring relative vibration state of vehicle and rail.