Nonreciprocal elements such as circulators and isolators are essential for the realization of integrated optical circuits. The design of nonreciprocal components requires breaking the time reversal symmetry[2,3]. This can be achieved through the use of nonlinear materials[4,5], materials with time-dependent properties, and magneto-optical (MO) materials. However, since the MO response of natural materials is weak at optical wavelengths, designing nonreciprocal devices that are based on MO materials results in bulky structures that are much larger than the wavelength. The advent of silicon photonics and photonic crystals has reduced the size of nonreciprocal optical components down to wavelength scale[9–12]. To further decrease the size down to the subwavelength scale, one needs to beat the diffraction limit. Nanoscale metallic structures that support surface plasmon polaritons can be used to achieve subwavelength scale optical components, because they can beat the diffraction limit[13,14]. Combining metallic and MO materials can therefore pave the way for highly compact nonreciprocal plasmonic elements[15–22]. One of the promising ways to engineer integrated plasmonic circuits is to employ metal-dielectric-metal (MDM) waveguides. Several different nanoscale plasmonic components based on MDM waveguides have been proposed, including filters[24,25], couplers[26,27], sensors[28–30], switches[31–35], and rectifiers.