A quantitative phase imaging system capable of revealing three-dimensional changes of microstructure and light scattering properties of cancer tissues is developed based on the transport-of-intensity equation using the differential interference contrast (DIC) microscope. The DIC images of unstained paraffin embedded tissue sections of 257 patients with invasive ductal carcinoma are acquired with the quantitative phase imaging system. The corresponding two-dimensional phase maps are obtained with the calculation of the transport-of-intensity equation. The spatially resolved optical properties of the tissues are computed from the two-dimensional phase maps using the scattering-phase theorem afterwards. The results show that the grade of invasive ductal carcinoma has strong correlation with reduced scattering coefficients and anisotropy of tissues. When the grade of invasive ductal carcinoma increases, the reduced scattering coefficient of invasive ductal carcinoma tissues increases significantly, while the anisotropy decreases significantly. The retrieved optical parameters are shown to distinguish invasive ductal carcinoma with different grades with a high accuracy of 88%. Owing to the label-free and high-contrast characteristics of DIC microscope, the proposed approach is also applicable for fresh or frozen tissues, and may have promising applications in real-time and rapid cancer diagnosis.