Because panoramic video has a 360 degree viewing angle, it is quite suitable for the multi-object detection in intelligent video surveillance systems. First, this paper introduces the DSP-based system hardware architecture. Second, the panoramic image unrolling algorithm and the MHOEI multi-object detection algorithm are given. Then the look-up method of block table on DSP platform for panoramic image unrolling and the resolution reduction method for MHOEI are designed respectively. Finally, the optimized scheme is implemented on DSP processor. Experimental results show that it achieves 120 f/s processing speed for unrolling the panoramic video resolution from 720×576 into 896×180 on the selected DM6437 processor. And after combined with the multi-object detection, the average speed is up to 35 f/s, which meets real-time requirements. Meanwhile, it can detect moving and motionless objects in panoramic video effectively.