Loss is inevitable for the optical system due to the absorption of materials, scattering caused by the defects, and surface roughness. In quantum optical circuits, the loss can not only reduce the intensity of the signal, but also affect the performance of quantum operations. In this work, we divide losses into unbalanced linear losses and shared common losses, and provide a detailed analysis on how loss affects the integrated linear optical quantum gates. It is found that the orthogonality of eigenmodes and the unitary phase relation of the coupled waveguide modes are destroyed by the loss. As a result, the fidelity of single- and two-qubit operations decreases significantly as the shared loss becomes comparable to the coupling strength. Our results are important for the investigation of large-scale photonic integrated quantum information processes.