Precision sensing and measuring based on optical trap technology is innovation and deepening for application of optical force effect from micro-precise manipulation to precise measurement of physical quantities. Precision measurement of position information of the particle in optical trap is key technology of precise sensing and measuring. The method was put forward, which used digital image processing and curve fitting algorithm to detect particle positions in optical trap. The normalized self-correlation function of particle positions was obtained by digital image correlation, and the quadratic curve fitting was used to the normalized self-correlation function curve by least-square method to realize particle position detection of sub-pixel accuracy. Experiments show that the method described in this paper can effectively suppress the quantization effect of hardware and realize fast and high-precision detection of particle position in optical traps. Compared with the direct correlation method, the detection accuracy can be improved by at least one order of magnitude to 0.03 pixel.