Computer visual travel aids (VTA) are effective means to assist the blind, while blind sidewalk recognition is an important function of VTA. The so-called blind sidewalk recognition is a method that segments blind sidewalk and detects boundary lines via image processing technology. After blind sidewalk recognition, VTA locate the bounda-ry based on stereo vision and then guide the blind to sidewalk by control signal. In order to solve the problem that existing blind sidewalk recognition algorithms have bad real-time performance, a highly real-time blind sidewalk recognition algorithm based on boundary tracking is proposed, mainly including accurate recognition and tracking recognition. First, the preprocessing of shadow removal is performed for each frame image before recognition, which calculates a residual model based on the Retinex theory to detect shadow and uses regional color compen-sation to remove shadow. Next, accurate recognition step mainly calculates gray level co-occurrence matrix of the initial frame, and uses clustering and Hough transform to find the boundary lines of blind sidewalk in image. Then tracking recognition step takes over next frame. The location of blind sidewalk’s boundary in previous frame is used to predict the small-scale region of interest (ROI) of the boundary in current frame, and boundary lines in that region are extracted based on gray gradient feature. After that, the algorithm checks up the validity of tracking by estimating the consistency of color distribution on both sides of the boundary in previous and current frames: tracking is considered to be valid if the consistency is high, and tracking recognition step continues, otherwise accurate recognition step restarts. We apply our algorithm on binocular VTA and the blind wear the VTA to walk along blind sidewalk for the algorithm performance test. In many experiments, the time of accurate recognition and tracking recognition in each image frame under normal illumination are about 0.8 s and 0.1 s, respectively, and the average time of recognition per frame decreases significantly while the recognition rate of blind sidewalk is more than 90%. Meanwhile, the adaptability is good in shadow environment and is acceptable in other envi-ronment, including strong and weak light, damage of blink sidewalk, and blurring. Experimental results indicate that the algorithm can significantly enhance the real-time performance of blind sidewalk recognition in the prem-ise of ensuring the recognition rate. Therefore, our algorithm is more suitable for real-time visual navigation than traditional ones.