A general calculation method to obtain the transmission spectrum of Faraday anomalous dispersion optical filter (FADOF) is presented. The theoretical scheme can be applied to any electrical dipole transmission of various atomic vapors and is valid for arbitrary magnetic fields. Moreover, the method can be used to predict the performance of both the passive FADOF and excited state FADOF (ESFADOF). It can also be extended to calculate the transmission spectrum of other new type atomic optical filters (such as Stark type atomic optical filters). The theoretical transmission curve of Rb 775.9 nm ESFADOF obtained by the general method is given, which agrees well with the experimental result.