In this paper, a novel modeling and simulation method for general linear, time-invariant, passive photonic devices and circuits is proposed. This technique, starting from the scattering parameters of the photonic system under study, builds a baseband equivalent state-space model that splits the optical carrier frequency and operates at baseband, thereby significantly reducing the modeling and simulation complexity without losing accuracy. Indeed, it is possible to analytically reconstruct the port signals of the photonic system under study starting from the time-domain simulation of the corresponding baseband equivalent model. However, such equivalent models are complex-valued systems and, in this scenario, the conventional passivity constraints are not applicable anymore. Hence, the passivity constraints for scattering parameters and state-space models of baseband equivalent systems are presented, which are essential for time-domain simulations. Three suitable examples demonstrate the feasibility, accuracy, and efficiency of the proposed method.