In order to detect the resolution of the concave gratings, it needs to build a precise detection system. Based on the theoretical analysis of the system of the bandwidth of light source, the slit width and the pixel size of the detector, combining with the aberration analysis of concave gratings, a method for the resolution detection system is proposed. According to the standard method in JB/T 8239.2-95 which is named The technical conditions for diffraction grating, the system analyzes the above three factors quantitatively and puts forward a reasonable calculation method. Then it can obtain the practical value of concave grating resolution in the detection system. By experimental verification, the method can distinguish the difference between the test results and the theoretical analysis results from 0.001 0 nm to 0.003 7 nm.