A novel method of directional adaptive filter for fringe image is proposed based on the investigation of pretreatment method of fringe image on three-dimensional pattern profilometry technique. For a fringe image filter, the direction and the size of filter window are quite sensitive characteristics. The gray values of the stripe image change periodically in the normal direction, and keep almost the same values in the stripe direction. The fringe direction is calculated by gravity method. The size of data window is automatically adjusted according to the linear dependency to get the correct direction. The length and the width of the filter window are optimized based on the filtering result checking. By this algorithm, it can get the correct stripe direction and the right size parameters of the filter windows, as a result to get the qualified filtered image. This method is verified by the experiments.