Aiming at the sensitivity of dual-UAV passive radar to the initial value of Direct Position Determination(DPD) of broadband radiator targets, a direct passive location algorithm is proposed based on Niche Topology Improved Vector Particle Swarm Optimization(NTVIPSO). The proposed algorithm combines the direct method and the indirect method, relying on the indirect method to construct the constraint conditions, and using the niche vector particle swarm to solve the exact position. The simulation results show that the proposed particle swarm algorithm is superior to the vector particle swarm algorithm and the neighborhood topology particle swarm algorithm. The accuracy of the proposed direct passive determination algorithm under the constraint of scale difference in the presence of disturbance is lower than the direct method but better than the indirect method, and the computational complexity is lower than the direct method but higher than the indirect method.