The near-infrared optical tracking system has rapidly developed into an important part of surgical navigation because of its high precision and convenience. A high-precision and low-cost optical tracking system was designed based on binocular vision. The system used the passive marker ball (NDI passive sphere) installed on the instrument as the marker, and added a near-infrared filter in front of the binocular camera lens to eliminate the interference of ambient light. Firstly, the contour filtering algorithm was used to extract the marker contour, and the least square ellipse fitting algorithm was used to obtain the pixel coordinates of the marker projection center; Secondly, an instrument recognition algorithm was designed so that each instrument could store the central pixel coordinates of the marker ball separately and orderly for the resolution of multiple instruments; Finally, by matching the mark centers corresponding to the left and right views, the spatial coordinates were reconstructed, and then the coordinates of the instrument tip in the world coordinate system were derived. The system was used to track the instruments in the experiment, and the irregular placement of instruments was used to verify the accuracy and robustness of the instrument recognition algorithm. The accuracy rate was 95%, and the average recognition time was only 4 ms. The stability, static positioning accuracy and dynamic tracking of the system were tested. The results showed that the stability error could reach 0.13 mm, the static positioning accuracy could reach 0.373 mm. The proposed near-infrared optical tracking system has high accuracy and stability, and can meet the requirements of surgical navigation.