MATLAB/Simulink is a good system modeling and algorithm simulation tool and Modelsim is the most commonly used HDL simulation tool in China. Modelsim can cooperate with Simulink using Link for modelsim. And the algorithm design and validation are finished in the same software so that compatibility issues under different development environment are avoided, design complexity is simplified greatly and development process is advanced. The design method based on models is introduced. The algorithm with hardware can bring great convenience to algorithm development and subsequent hardware implementation. The interface technology and model building of Modelsim and MATLAB/Simulink united simulation are introduced. And the validity of the system is verified by the application example of 16 QAM modulation and demodulation.