Holographic microscopy has emerged as a vital tool in biomedicine, enabling visualization of microscopic morphological features of tissues and cells in a label-free manner. Recently, deep learning (DL)-based image reconstruction models have demonstrated state-of-the-art performance in holographic image reconstruction. However, their utility in practice is still severely limited, as conventional training schemes could not properly handle out-of-distribution data. Here, we leverage backpropagation operation and reparameterization of the forward propagator to enable an adaptable image reconstruction model for histopathologic inspection. Only given with a training dataset of rectum tissue images captured from a single imaging configuration, our scheme consistently shows high reconstruction performance even with the input hologram of diverse tissue types at different pathological states captured under various imaging configurations. Using the proposed adaptation technique, we show that the diagnostic features of cancerous colorectal tissues, such as dirty necrosis, captured with 5× magnification and a numerical aperture (NA) of 0.1, can be reconstructed with high accuracy, whereas a given training dataset is strictly confined to normal rectum tissues acquired under the imaging configuration of 20× magnification and an NA of 0.4. Our results suggest that the DL-based image reconstruction approaches, with sophisticated adaptation techniques, could offer an extensively generalizable solution for inverse mapping problems in imaging.