The dual-optical path system is a commonly used detection device in precision mate parts assembly alignment tasks. Among them, the installation error of the beam splitting prism can cause the imaging point of the assembly parts to shift. Moreover, the surface processing quality of the mate parts and the differences in imaging optical paths on both sides of the parts can also lead to inconsistent image quality, which affects the accuracy of feature extraction. To address this issue, a calibration method based on homography matrix is adopted to eliminate the impact of prism installation errors. An automatic focus and light intensity adaptive adjustment method based on gray variance and information entropy is adopted to ensure consistency in image quality. Based on the improved arctangent method and combined with adaptive edge gradient threshold, subpixel edge extraction is performed. The above method simultaneously considers and overcomes the interference factors in image acquisition and processing, directly obtaining alignment control variables while avoiding prism pose adjustment, and improving the robustness of the algorithm. Through experimental testing, the image feature extraction accuracy of this method can reach 0.25 pixels under different working conditions, and the assembly coaxiality of the rotary mate parts is φ6.3 μm.