Spectroscopic ellipsometric measurements in the infrared region is used to characterize the structure of CVD-diamond films. According to the status of surface and interface, different models are developed for diamond films on smooth silicon substrate and rough alumina substrate respectively with theoretical simulations followed. The Bruggeman effective media approximation, which includes the factors such as roughness, void and phase mixture, is applied to simulate the effects of rough surface and complicated interface on the ellipsometric data. It is found that the establishment of appropriate model has the strongest influence on the fit of ellipsometric spectra. The films on silicon substrates are well described by a two-layer film model. The bulk film(first layer) is pure diamond. The second layer(surface) represents a rough surface sublayer about(879 nm) thick, which contains 28.3% voids. But satisfactory agreements between experimental and calculated data cannot be obtained when the two-layer film model is aplied to diamond films on alumina substrates. The best fit is achieved for the samples on alumina substrates with a three-component interface layer(64.13% alumina+23.34% diamond+12.53% void) included by Bruggeman effective media approximation.