In this paper, the localized surface plasmon resonance (LSPR) peak of Ag-Fe and Au-Fe alloy nanoparticles for the triangular prism is calculated by using the discrete dipole approximation (DDA) method. We investigate the variation of the resonance wavelength, refractive index sensitivity, and figure of merit with the particles size and alloy compositions. We perform a comparative study on the refractive index sensitivity and figure of merit of alloys in order to find the considered (Ag-Fe and Au-Fe) alloys with high sensitivity. The refractive index sensitivity of the Au-Fe alloy is found higher than that of the Ag-Fe alloy. Therefore, to optimize the size of alloy nanoparticles (NPs) for the triangular prism, the figure of merit is calculated and observed that the optimized size is 50 nm and 20 nm for Ag-Fe and Au-Fe alloys, respectively. A comparison of Ag-Fe shows that the Au-Fe alloy NPs have greater figure of merit (FOM) and thus may be more suitable for applications in biosensing.