The application of nanofluids in spectral beam splitting (SBS) hybrid PV/T system can improve its efficiency, in which nanoparticles (NPs) with appropriate size can effectively filter solar radiation outside the spectral response of photovoltaic cells. The optical properties of Au, Ag, Cu, Fe₃O₄, ZnO and TiO₂ NPs were simulated by finite difference time domain (FDTD) method. Moreover, taking spectral response of monocrystalline silicon solar cells as example, the optical absorption properties of six NPs with diameters ranging from 20 nm to 200 nm were studied, and degree of appropriateness (DOA) was used as the evaluation index to optimize the particle size. The results show that the optical properties of NPs are very sensitive to their particle size. By changing the particle size of NPs, the positions of scattering, absorption and extinction peaks can be adjusted in a wide range, and the peaks increase with the increase of particle size. The absorption capacity of metal NPs to solar radiation is better than that of non-metal NPs. The maximum absorption power per unit volume of six NPs are 21.88 GW/m³, 17.95 GW/m³, 20.16 GW/m³, 2.54 GW/m³, 1.02 GW/m³, 0.27 GW/m³, respectively. The DOA index analysis shows that the optimal particle sizes of six NPs suitable for SBS hybrid PV/T system are 20 nm, 50 nm, 20 nm, 170 nm, 110 nm, 20 nm, respectively.