Sub-aperture stitching interferometry plays an important role in large-aperture surface testing. Instead of employing large aperture interferometers, a displacement mechanism is often utilized for testing each sub-aperture from the tested surface. In sub-aperture stitching interferometry, positioning error and alignment error between two adjacent sub-apertures are the major error source for surface map testing. The positioning error, mainly introduced by the mechanical scanning of each sub-aperture, will cause a mismatch of the overlapping regions of two adjacent sub-apertures, and will severely decrease the accuracy of stitching testing. The alignment error is mainly caused by piston, tip, tilt, and defocus between two adjacent sub-apertures. Some iterative algorithms utilized alternate optimization to make the positioning error and alignment error converge in sequence. However, due to the coupling between positioning error and adjustment error, the alternative optimization method may not accurately solve the relationship between these two errors. Different from alternate optimization, global searching algorithms can realize synchronous optimization of these two kinds of errors.Particle Swarm Optimization (PSO) is a random searching algorithm derived by simulating the foraging behavior of birds. PSO is a global optimization method, and the model parameters setting is simple, making it widely used in many areas. In this paper, PSO was selected as the optimization method for sub-apertures stitching. However, if PSO is conducted in the original surface maps directly, it will take a long time to get an accurate estimation of positioning and alignment errors. In order to accelerate global searching in PSO, an accurate stitching method based on down-sampled PSO was proposed to realize the synchronous elimination of positioning error and alignment error. To improve the efficiency of PSO, down-sampling was applied to reduce the searching range of PSO algorithm. Then the pixel-level positioning error was obtained by the gradient method. Finally, the coefficients of all error terms were solved, and the positioning error and alignment error were eliminated.To validate the proposed algorithm, two adjacent sub-apertures were selected from the surface data of a spherical mirror tested by a 4-in Zygo interferometer for simulation analysis. To simulate the actual testing process, random noise and positioning error, and alignment error were introduced into the two sub-apertures artificially. The coefficients of these errors were selected randomly. The sub-aperture stitching results of the conventional PSO, and the proposed algorithm based on down-sampled PSO were obtained. The positioning accuracy of the two methods can reach the pixel-level accuracy, but the operation speed of the algorithm is increased by about 12 times by employing 4^th order down-sampling. In order to verify the feasibility of the proposed algorithm, a 4-in plane mirror and a 3-in off-axis parabolic mirror were selected as test samples. The surface maps obtained by the proposed stitching method were consistent with the full-aperture direct test, in which the Peaks and Valleys (PV) and Root Mean Square (RMS) values of the proposed algorithm were closer to full-aperture test results. Compared with traditional Least Square (LS) method, the PV value and RMS of residual surface errors of the proposed algorithm are both smaller than those of LS method, indicating that by the proposed method, the positioning error and alignment error are eliminated precisely, and the obtained surface map is more coincident with the full-aperture direct testing result.