A compact directional backlight module of time-multiplexed auto-stereoscopic display based on side-glowing polymer optical fiber (SGPOF) is proposed. The optical system is mainly composed of SGPOF array and cylindrical lens array. Spatial crosstalk is reduced by inserting a grating film as multi-slit diaphragm between the SGPOF array and the cylindrical lens array. A theoretical model is constructed based on the imaging optics principle of the off-axis ray. In the experiments, the cylindrical lens array concentrates a small number of views on three different view zones, the display can provide high luminance. The measurement results show that the luminance uniformity of the backlight module is up to 89.6%, and in the viewing zone the crosstalk is lower than 10%. The backlight module is compacted that the thickness being only 7 mm. The full-resolution and low-crosstalk 3D images are realized by using SGPOF backlight.