While effectively enhancing the transmission capacity of current commercial single-mode optical fiber communication systems, Polarization Division Multiplexing (PDM) technology also faces serious challenges from Polarization Mode Dispersion (PMD) and Rotation of State of Polarization (RSOP). PMD causes the broadening of optical pulses, resulting in crosstalk and distortion of the signals, which may significantly increase the Bit Error Rate (BER) of the system. Also, RSOP might cause rapid change in the polarization state of optical signals up to several hundred krad/s, preventing the two polarization signals from being correctly separated at the receiver. In practical optical fiber links, PMD and RSOP usually coexist and impose significant impairments on the system performance. The main work of this paper is to build an impairment model of PMD and RSOP and to conduct a joint compensation of the polarization-relevant impairments.Traditionally, the Stokes formalism is employed to analyze the PMD and RSOP in commercial single-mode fibers that actually support two orthogonal polarization modes. The corresponding Stokes vector is a 3-dimensional real vector with clear physical meaning, and can be intuitively represented in the geometrical Poincare sphere. However, the Stokes formalism requires 3 auxiliary 2×2 Pauli matrices, and when extended to the treatment of Modal Dispersion (MD) and Mode Coupling (MC) in an N-mode optical fiber (i.e., a modal space of dimension N), a number of N2-1 auxiliary N×N Gell-Mann matrices are required, which can drastically complicate the analysis of MD and MC effects as N increases. Recently, borrowing methodology from quantum mechanics, we proposed and developed the Density Matrix (DM) formalism for the MD and MC in a modal space of arbitrary dimension N≥2. Without requiring any auxiliary matrices, the DM formalism is simple and straightforward in formulation and application, and is thus particularly suitable for the study of modal properties and signal compensations in the optical communication system.In this paper, we apply the DM formalism in the PDM system to construct a joint polarization-impairment compensation scheme. We establish a polarization impairment model for coexisting PMD and RSOP based on traceless Hermitian impairment matrices. By tracking the corresponding independent parameters in the matrices, we achieve the joint impairment compensation of PMD and RSOP. To verify the proposed scheme, we build a 28 GBaud PDM quadrature phase-shift keying coherent optical communication transmission simulation system, and implement a joint impairment-compensation for Differential Group Delay (DGD) over a wide range of 30~170 ps and fast RSOP over 300 krad/s~2 Mrad/s. In simulations, with about 100 iterations, the tracking and compensation have already converged for the impairment of 100 ps Differential Group Delay (DGD), and with only 0.5 dB and 0.7 dB optical Signal-to-noise Ratio (OSNR) cost, joint compensation of RSOP can be achieved in typical (600 krad/s) and extreme conditions (2 Mrad/s), respectively, both in the presence of 100 ps DGD. When 170 ps DGD and 2 Mrad/s fast RSOP coexist, the BER is 3.22×10^-3, which still meets the criterion for error-free transmission of signals. The fast convergence, the small OSNR cost, and the stable BER performance verify the validity and efficacy of our polarization impairment model and the corresponding joint compensation scheme in optical communication systems.Furthermore, taking advantage of the DM formalism, our joint compensation scheme can be readily generalized to modal spaces of arbitrary dimension N for the impairment analysis and signal compensation of MD and MC, simply by the extending the relevant matrices from 2×2 to N×N. Therefore, our work potentially provide a simple and effective theoretical approach for the impairment analysis and compensation of optical-signals in more general mode-division multiplexing communication systems.