On the base of traditional genetic algorithm,a new fitness function is constructed by introducing a standard deviation function. Meanwhile,a proposed method of automatically lowering temperature is used to control the temperature of controlling anneal-selecting strategy. A new method used to optimize and design film system is developed by combining the improved float-coded genetic algorithm with simplex algorithm,and an optimized program is coded. Examples show that this program is excellent in both global and local searching capacity. Some notch filters with center wavelength 534 nm,bandwidth 35 nm and neutral beam splitter mirrors with high performance in visible band are designed.