Strong nonlinearity of plasmonic metamaterials can be designed near their effective plasma frequency in the epsilon-near-zero (ENZ) regime. We explore the realization of an all-optical modulator based on the Au nonlinearity using an ENZ cavity formed by a few Au nanorods inside a Si photonic waveguide. The resulting modulator has robust performance with a modulation depth of about 30 dB/μm and loss less than 0.8 dB for switching energies below 600 fJ. The modulator provides a double advantage of high mode transmission and strong nonlinearity enhancement in the few-nanorod-based design.