In practical application, the signals measured by distributed optical fiber vibration sensing system are mostly non-stationary random signals, and the key to realize pattern recognition is to obtain the amplitude-time-frequency instantaneous characteristics of the signals accurately. Existing related research shows that Hilbert transform combined with empirical mode decomposition can obtain the instantaneous energy and instantaneous frequency of the intrinsic modal component of measuring signal. The subsequent improved ensemble empirical mode decompostion method, has pseudo component and large reconstruction error, while complementary ensemble empirical mode decompostion method reduces the reconstruction error, but increases the amount of computation, which cannot guarantee the efficiency and accuracy of feature extraction and classification. In this paper, the feature extraction of distributed optical fiber vibration sensing system was realized based on modified ensemble empirical mode decompostion with Hilbert transform, the evaluation mechanism of permutation entropy was introduced to optimize the iteration times of random noise in the decomposition process. Through simulation analysis and experimental comparison, it was verified that the method could effectively solve the problems existing in the above methods and improve the system's performance in processing time and feature accuracy. Experimental results show that the average extraction accuracy of the proposed method for single-frequency vibration signals is 99.2%. Compared with EMD and CEEMD, the average feature extraction accuracy of mixed vibration signal is 98.1%, which is 15.6% and 7% higher than EMD and CEEMD respectively. The average time of the algorithm is the shortest, which is 3.8259 s. It provides a reliable and efficient method for signal feature extraction of distributed fiber vibration sensing system.