The phase modulation and demodulation of double beam interferometer often use ellipse fitting to calculate relevant parameters, and a method is developed to assess calculation precisions of these parameters. The maximum deviations of parameters calculated by ellipse fitting should be computed in different levels of noise according to adaptive Monte Carlo method; then the precisions of fitted parameters can be estimated according to the noise floor of interference signal which is measured in experiments. The algorithm of fiber-optic hydrophone based on 3×3 coupler using two interference signals is analyzed as an example, and the relationship between the parameter value and device properties needs to be studied in order to improve the demodulation effect of hydrophone. As a result, the precisions of parameters calculated by ellipse fitting in experiments should be assessed firstly. The assessment method of precisions is applied to hydrophone experiments, and the noise floor of measured interference signal is close to -70 dB·V²/Hz from 20 Hz to 4 kHz. The deviations of parameters are less than 0.06% and 0.08° according to adaptive Monte Carlo method.