Silica nanoparticles (NPs) are more and more useful in many engineering areas, but the dynamic behaviors of adsorption of NPs at surface are not clear, especially when there exist surfactants on the surface. The modified NPs with the nonionic dimethyl silane are partially hydrophobic, and in this paper, the surface behavior is investigated which is determined by interfacial tension and surface compression modulus. It is concluded that the dimethyl silane coverage, the brine salinity and the surfactant would affect the NPs’ adsorption. Higher salinity in brine or higher dimethyl silane coverage causes lower steady state surface tension, which is related to the hydrophobicity and adsorption amount of NPs at the surface. When the cationic surfactant concentration is lower than critical micelle concentration (CMC), the surface tension of mixture system is a little bit higher than cationic surfactant’s. Cationic surfactant can be adsorbed at NPs’ surface to change the hydrophobicity based on the electrostatic attraction, and then some surfactants are dissolved in liquid phase together with NPs, while the anionic surfactant will not do so. In the shrinking droplet process, the surface tension of the NPs with 1.5 μmol/m² dimethyl silane decreases from ~59 mN/m at the steady state to ~50 mN/m, which proves that the NPs’ adsorption density can be higher even after infinite long time equilibrium due to the repulsive force between the NPs. Besides, the curve of interfacial tension (IFT) versus surface area shows three parts with different declining slopes. In the first part, the relatively low adsorption of NPs at the surface means weak interaction between NPs. Then in the second part, due to the irreversible adsorption, the spacing between NPs decreases with adsorption amount increasing and surface area lowering, so the increasing of NPs’ interaction leads to high surface compression modulus. After that, the IFT curve keeps flat since the NPs assembly reaches to the closest peck. With the increase of NPs’ hydrophobicity, the compression modulus increases up to ~70 mN/m, which satisfies the Gibbs criterion to resist coarsening of the foam or emulsion. However, for the mixing system, increasing surfactant concentration leads to a lower surface tension at steady state, then the surface tension difference or compression modulus decreases too. Finally, we find that the compression modulus order from high to low is as follows: NPs, cationic surfactant-NPs, anionic surfactant-NPs, surfactants. This investigation is meaningful for accounting for the enhancement of foam or emulsion stability with NPs affected by salinity and surfactant.