An approach for designing multilayer optical coatings is presented by use of genetic algorithm. With a predetermined maximum layer number, the algorithm automatically specifies the appropriate layer number and material and thickness of each layer. Evaluation function used here is different from the others before. The 1064 nm, 532 nm frequency-doubled anti-reflection coating is designed on LBO (LiB3O5) substrate with two evaluations functions. Two coating structures are found with high transmissivities, one with transmissivity 99.98% at 1064 nm and 99.99% at 532 nm, the other with transmissivity of 100.00% at both wavelengths. The broad-band anti-reflection coating on K9 glass substrate is designed. The transmissivities of two broad-band antireflection coatings optimized over a range of wavelength (450～650 nm) and (390～780 nm) are above 99.91% and 99.30% respectively. If the evaluation function is sensitive to the deviation of coating structure, the approach by genetic algorithm is effective.