The traffic sign recognition (TSR) system is an important research direction in the field of intelligent transport system. Due to traffic complexity, large scale of traffic signs database and other reasons, the feasibility of TSR design must take computational complexity and recognition rate into consideration. An efficient and fast traffic sign algorithm is proposed based on the improved principal component analysis (PCA) and extreme learning machine (ELM), as known as PCA-ELM. Firstly, the histogram of gradient direction (HOG) features for each TSR are extracted from traffic sign database. HOG dimensional features are reduced by the improved PCA algorithm. ELM model training is presented based on the HOG after dimension reduction. Image recognition is tested based on the trained ELM model. Experimental results show that the recognition algorithm based on PCA-HOG and ELM model can get a high recognition rate of 97.69% and perform low in computational complexity.