Compared with other electric energy storage devices, dielectric capacitors made of dielectric composites have great advantages in fast charging and discharging capacity with high power density. A dilemma of improving the energy density of dielectric composites and synchronous optimizing their breakdown performance is becoming an intriguing research direction. To further adjust the contradiction between dielectric constant and dielectric breakdown performance, here a finite element numerical simulation based on dielectric breakdown model (DBM) was proposed to study the effect of the distribution of inorganic fillers on the electric field and breakdown damage morphology in flexible polydimethylsiloxane(PDMS) based dielectric composite system. The results show that a large dielectric difference is observed between filler and matrix, which indicates that polymer matrix with a large dielectric constant or inorganic filler with a small dielectric constant can realize reducing the size of the high electric field area at the interface and improving the breakdown resistance of the material. This study further reveals that the more dispersed structure of inorganic fillers, the more likely its dendritic damage channels tend to branch, indicating that this situation is conducive to the increase of damage sites of dielectric breakdown dendritic damage channels, the decrease of damage rate, and the improvement of breakdown resistance of materials. All above data demonstrate that this study provides certain guidance for the development of organic-inorganic dielectric composites with both high energy storage and excellent breakdown performance.