The memory simulated annealing approach is presented for designing an optical FIR filter based on crystal birefringence, which is able to provide arbitrary spectrum output with double channels. As an effective way to make global optimization through random search, simulated annealing algorithm simulates the slowly physical process of solid annealing to get lowest inner energy. The procedure of employing the algorithm to get satisfied results in relatively fewer crystal slices is described, and an example of broadband G/M filter, including the experimental results is given to demonstrate its feasibility and effectiveness.