A novel overlapping code system is proposed to enhance the communication performance of double-layer overlay coding visible light communication system. With full consideration of the characteristics of far and near distance communication, the proposed system makes a creative combination of trace orthogonal coding and overlapping systems. This combination ensures the trace orthogonal property of two layers signals and therefore eliminates the interlayer interference. Moreover, an enhanced Euclidean distance and coding gain can be obtained to extract better performance. For this system, the same decoding performance can be obtained with the sum detection algorithm of linear complexity as well as with maximum-likelihood (ML) detection. The effect of different power allocations on system performance is analyzed by deriving the closed expression of bit error rate based on ML, and three constellation designs are then presented. Simulation results show that compared with existing double-layer overlapping systems, the new system based on layered trace-orthogonal coding can effectively improve communication performance.