Owing to the necessity and importance of the technique of three-dimensional (3D) displacement measurement in the filed of engineering, a 3D displacement measurement system based on single CCD camera and digital image correlation is developed. With regard to the relationship between the center and the slope of the displacement vector and displacements in- and out-of the object plane, two dimensional Savitzky-Golay (SG) digital differentiator (an operator based on local subset coordinates) is employed to calculate constant terms (corresponding to the in-plane components) and the first order terms (corresponding to the out-of-plane components).On the theoretical basis of the pin-hole camera imaging model, a numerical simulation and an experiment on a non-uniformly deformed cantilever beam are conducted. In addition, the simulated speckle images associated with the 3D deformation are generated, which is further employed to validate the effectiveness and precision of the Newton-Raphson iterative method with a shape function of the second order. Simulated and experimental results demonstrate that both in-plane and out-of-plane displacements can be accurately retrieved using the proposed method.