To achieve real-time 3-D medical image despeckling, a novel fast algorithm and high-performance Very Large Scale Integrated (VLSI) architecture for 3-D median filter were presented. According to the data’s distributing character of the 2-D array ordered by a 2-D partial sorting technique, the proposed method could efficiently reduce the implementation complexity of the 3-D median filter by using a 3-D partial sorting technique. Theoretical analysis and experimental results demonstrate that the presented method could reduce the computation complexity and/or hardware resources by about 33%, which could significantly decrease the computation delay of the 3-D median filter implemented by software, and/or the hardware cost and power consumption.