An algorithm for small target detection with high speed was proposed. For optimization of algorithm, Particle Swarm Optimization (PSO) was utilized. In order to overcome some defects with PSO, a method about adaptive adjustment of swarm topology was developed, which improved the particles swarm diversity and strengthened their searching capabilities. For small target detection, the local variance increment can depict small target which acts as a sudden change with grayscale in local pixel space. PSO was combined with detection for decreasing the computation cost. Simulation shows that the improved PSO gets excellent search ability which accelerates detection, and the detection result is reliable and effective.