Nanostructuring, structure distortion, and/or disorder are the main manipulation techniques to reduce the lattice thermal conductivity and improve the figure of merit of thermoelectric materials. A single-phase α-MgAgSb sample, MgAg0.97Sb0.99, with high thermoelectric performance in near room temperature region was synthesized through a high-energy ball milling with a hot-pressing method. Here, we report the average grain size of 24–28 nm and the accurate structure distortion, which are characterized by high-resolution neutron diffraction and synchrotron x-ray diffraction with Rietveld refinement data analysis. Both the small grain size and the structure distortion have a contribution to the low lattice thermal conductivity in MgAg0.97Sb0.99.