The off-resonant nonlinear optical properties of the composite film of CdSeS quantum dots (QDs) stabilized in polystyrene (PS) nanosphere are investigated by Z-scan technique with femtosecond laser (800 nm, 50 fs, 1 kHz). It is observed that the signs of the nonlinear absorption coefficients and refractive indices changes from negative to positive with laser intensity increasing, which is due to the transition from saturable absorption to two-photon absorption (TPA). With laser intensity of 4.2 GW/cm2 and 16.8 GW/cm2, the two-photon absorption coefficient (β) are 1.6×10-8 cm/W and 2.8×10-8 cm/W, respectively, and the third-order nonlinear refractive indices (n2) are -6.2×10-11 cm2/W and 2.7×10-12 cm2/W, respectively, which shows an increase of about one magnitude for β and two magnitudes for n2 compared with previous reports. The results indicate that CdSe QDs doped PS has high potential application value for all-optical switching devices.