Classical multi-channel algorithms usually employ minority discrete channels. It is considered that the channels are independent and their multi-mountain like distribution is neglected. Double-layer particle filter for target tracking with multi-channel energy extreme points set method is proposed. Visual target is firstly decomposed to multiple frequency and angle channels. And then energy extreme points set is calculated as target feature. Double-layer particle filter algorithm framework is established. Target motion state is globally constrained by first-order particle. Second-order particles represent the accessory multi-channel features and cluster to feature peak by local optimization method. Multi-channel features are examined and then second-order particles are weighted. The result is feedback to first-order particles as weighting parameters. Thus global constraint and multi-feature reliability distribution are balanced. Experimental results show that the algorithm runs steadily with moderate computational complexity. Compared to correlation methods, the average tracking error is decreased by about 0.5 pixels.