Optical filter based on microring resonator usually has drawbacks such as narrow bandwidth and larger radius of curvature. An ultra-compact optical filter consisting of rectangular ring resonator incorporated with two Mach-Zehnder interferometers is proposed and optimized, and the device is formed by trench-based nanophotonic frustrated total internal reflection couplers and total internal reflection mirror-based 90o bends waveguide. The performances such as transfer function, quality factor, and transmission characteristic are analyzed by using statespace representation and finite difference time domain (FDTD) methods. The dependence of filtering features on both different coupling coefficients and loss coefficients is calculated with state-space representation method. The FDTD simulation results show the device exhibits an asymmetric wavelength interleaver filtering effect with the different ratios of pass-band and stop-band bandwidths when the coupler reflection coefficient is close to the coupler transmission coefficient. The quality factor of about 3120 and the free spectral range up to 50.6 nm are observed in such small 25 μm ×8 μm chip area when the coupler reflection coefficient is far larger than the coupler transmission coefficient. The compact configuration of device layout is beneficial to the extension at two-dimensional directions for highly dense photonic integrated circuits.