Simultaneous localization and mapping (SLAM) based on visual-inertial information fusion is a hot topic in the field of robot navigation. Multimodal data synchronization is a premise of the data fusion algorithm, and the key to it is to accurately obtain the timestamps of the acquisition time of different sensors. While the timestamp of the vision sensor can be accurately obtained by the hardware method, that of the inertial sensor is usually roughly replaced by the output time, which lowers the accuracy of the visual-inertial fusion algorithm. In this paper, a method of time synchronization calibration of the visual-inertial sensor was proposed. A calibration device based on a planar simple pendulum was designed. The vision sensor and the inertial sensor captured data independently in the motion of the planar simple pendulum, and timestamps were added according to the same clock. Then, a method of estimating the angular displacement function and the angular velocity function of the pendulum’s center of gravity was proposed by the least squares method. Finally, the output delay and timestamp of the inertial sensor were obtained by comparing the phase differences of the two functions. Experimental results show that the standard deviation of repeated calibration by the proposed method is 0.018 ms, which proves the effectiveness of the method.