A handheld swept source optical coherence tomography (SSOCT) system is described for imaging human skin in vivo. The performance of the system is analyzed in detail, and compared to the spectral domain optical coherence tomography (SDOCT) system in our laboratory. The results show that the sensitivity of the handheld SSOCT system and the traditional SDOCT system is 101 dB and 94 dB, respectively. A phantom experiment and two in vivo experiments on different human skin areas are carried out to demonstrate the better performance of the handheld SSOCT system in terms of signal to noise ratio (SNR) and the maximum imaging depth. The inner side of a human arm is imaged successfully with the handheld SSOCT system, which suggests that the handheld SSOCT system can image any position of human skin. All the results indicate that the handheld SSOCT system can not only be used to image skin anywhere conveniently, but also offer the structure images with high quality.