In normal conditions, image quality of space camera optical system changes very little in a permissible range of depth of focus，which will appear different degrees of falling when the image plane is out of depth of focus however. The wavefront coding technology (WFC) can extend the depth of focus very well，and enable the optical system to be insensitive to defocus. The user-defined merit function using modulation transfer function as image quality guideline is proposed based on extended polynomial to import cube phase distribution in the pupil, the optical system is designed of WFC system, and the test device is builded. The test results show that the image of WFC system is not only correspond to image of original system, but also keeps stably in a big range of depth of focus, which proves the big value of WFC on enhancing the stability of space camera.