To study the theoretical detection limit of the magnetic gradient tensor system （MGTS）， the differential tensor measurement range formula was derived with the magnetic dipole forward equation， the tensor matrix characteristic equation and the tensor invariant， and the MGTS theoretical detection limit estimation method was proposed. First， the target magnetic moment was estimated with the magnetic gradient tensor derivative invariant relations， and the difference calculation principle was used to estimate the measurement accuracy of the tensor component of the MGTS in the static sampling of the background field； then， the theoretical detection limit of on-site MGTS was estimated with the proposed method； and finally， the magnetic object was slided away from the MGTS and continuous sampling was performed， recording the actual tensor signal observable range， and verifying the estimation result of detection limit. The results show that the theoretical detection limit of MGTS is related to the baseline distance d， the measurement accuracy q， the target magnetic moment intensity M， the angle θ between the target magnetic moment and the observation point， etc.； The longer d， the higher q， and the larger M， the farther detect distance； The longer d and the closer observation， the greater the theoretical differential measurement error； The detection range of MGTS reaches the maximum in the direction parallel to the magnetic moment， but decreases sharply in the direction perpendicular. The experiments show that the estimation accuracy of detection limit of the planar-cross MGTS for four typical magnets is ± 0.5 m.