We introduce a simple one-dimensional (1D) structure in the design of 1D color splitters (1D-CSs) with RGB unit cells for color imaging and propose a single-to-double-layer design in 1D-CSs. Based on inverse design metasurfaces, we demonstrate numerically a single-layer 1D-CS with a full-color efficiency of 46.2% and a double-layer 1D-CS with a full-color efficiency of 48.2%; both of them are significantly higher than that of traditional color filters. Moreover, we demonstrate a 1D-CS that has application value by evaluating the double-layer 1D-CS’s performances in terms of incident angle sensitivity, polarization angle sensitivity, and assembly tolerance.