An adaptive microwave photonic angle-of-arrival (AOA) estimation approach based on a convolutional neural network with a bidirectional gated recurrent unit (BiGRU-CNN) is proposed and demonstrated. Compared with the previously reported AOA estimation methods based on phase-to-power mapping, the proposed method is unnecessary to know the frequency of the signal under test (SUT) in advance. The envelope voltage correlation matrix is obtained from dual-drive Mach–Zehnder modulator (N-DDMZM, N > 2) optical interferometer arrays first, and then AOA estimations are performed on different frequency signals with the aid of BiGRU-CNN. A three-DDMZM-based experiment is carried out to assess the estimation performance of microwave signals at three different frequencies, and the mean absolute error is only 0.1545°.