Digital staining in optical microscopy using deep learning - a review

Lucas Kreiss; Shaowei Jiang; Xiang Li; Shiqi Xu; Kevin C. Zhou; Kyung Chul Lee; Alexander Mühlberg; Kanghyun Kim; Amey Chaware; Michael Ando; Laura Barisoni; Seung Ah Lee; Guoan Zheng; Kyle J. Lafata; Oliver Friedrich; Roarke Horstmeyer

doi:10.1186/s43074-023-00113-4

[1] Aresta G, Araújo T, Kwok S, Chennamsetty SS, Safwan M, Alex V, et al. Bach: Grand challenge on breast cancer histology images. Med Image Anal. 2019;56:122–39.

[2] Haft-Javaherian M, Fang L, Muse V, Schaffer CB, Nishimura N, Sabuncu MR. Deep convolutional neural networks for segmenting 3D in vivo multiphoton images of vasculature in Alzheimer disease mouse models. PLoS ONE. 2019;14(3):e0213539.

[3] Muthumbi A, Chaware A, Kim K, Zhou KC, Konda PC, Chen R, et al. Learned sensing: jointly optimized microscope hardware for accurate image classification. Biomed Opt Express. 2019;10(12):6351–69.

[4] Chen T, Chefd’Hotel C. Deep learning based automatic immune cell detection for immunohistochemistry images. In: International workshop on machine learning in medical imaging. Springer; 2014. p. 17–24.

[5] Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. Springer; 2015. p. 234–241.

[6] Liu MY, Breuel T, Kautz J. Unsupervised image-to-image translation networks. Adv Neural Inf Process Syst. 2017;30. https://proceedings.neurips.cc/paper_files/paper/2017/hash/dc6a6489640ca02b0d42dabeb8e46bb7-Abstract.html.

[7] Zhu JY, Park T, Isola P, Efros AA. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. Computer Vision Foundation; 2017. p. 2223–32.

[8] Zhang Y, Tang F, Dong W, Huang H, Ma C, Lee TY, et al. Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning. arXiv preprint arXiv:2205.09542. 2022.

[9] Kingma DP, Dhariwal P. Glow: Generative flow with invertible 1x1 convolutions. Adv Neural Inf Process Syst. 2018;31.

[10] Kobyzev I, Prince SJ, Brubaker MA. Normalizing flows: An introduction and review of current methods. IEEE Trans Pattern Anal Mach Intell. 2020;43(11):3964–79.

[11] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. Adv Neural Inf Process Syst. 2020;33:6840–51.

[12] Saharia C, Chan W, Chang H, Lee C, Ho J, Salimans T, et al. Palette: Image-to-image diffusion models. In: ACM SIGGRAPH 2022 Conference Proceedings. Association for Computing Machinery; 2022. p. 1–10.

[13] You A, Kim JK, Ryu IH, Yoo TK. Application of generative adversarial networks (GAN) for ophthalmology image domains: a survey. Eye Vis. 2022;9(1):1–19.

[14] Jiang H, Zhou Y, Lin Y, Chan RC, Liu J, Chen H. Deep Learning for Computational Cytology: A Survey. arXiv preprint arXiv:2202.05126. 2022.

[15] Wu Y, Cheng M, Huang S, Pei Z, Zuo Y, Liu J, et al. Recent Advances of Deep Learning for Computational Histopathology: Principles and Applications. Cancers. 2022;14(5):1199.

[16] Jo Y, Cho H, Lee SY, Choi G, Kim G, Min HS, et al. Quantitative phase imaging and artificial intelligence: a review. IEEE J Sel Top Quantum Electron. 2018;25(1):1–14.

[17] Rivenson Y, de Haan K, Wallace WD, Ozcan A. Emerging advances to transform histopathology using virtual staining. BME Front. 2020:9647163.

[18] Pillar N, Ozcan A. Virtual tissue staining in pathology using machine learning. Expert Rev Mol Diagn. 2022;22(11):987–9. .

[19] Bai B, Yang X, Li Y, Zhang Y, Pillar N, Ozcan A. Deep learning-enabled virtual histological staining of biological samples. Light Sci Appl. 2023;12(1):57.

[20] Rivenson Y, Wang HD, Wei ZS, de Haan K, Zhang YB, Wu YC, et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat Biomed Eng. 2019;3(6):466–77. .

[21] Christiansen EM, Yang SJ, Ando DM, Javaherian A, Skibinski G, Lipnick S, et al. In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images. Cell. 2018;173(3):792. .

[22] Ounkomol C, Seshamani S, Maleckar MM, Collman F, Johnson GR. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nat Methods. 2018;15(11):917. .

[23] Gupta L, Klinkhammer BM, Boor P, Merhof D, Gadermayr M. GAN-Based Image Enrichment in Digital Pathology Boosts Segmentation Accuracy. Med Image Comput Comput Assist Interv Miccai 2019, Pt I. 2019;11764:631–639. .

[24] He YR, He S, Kandel ME, Lee YJ, Hu C, Sobh N, et al. Cell cycle stage classification using phase imaging with computational specificity. ACS Photon. 2022;9(4):1264–73.

[25] Somani A, Sekh AA, Opstad IS, Birgisdottir ÅB, Myrmel T, Ahluwalia BS, et al. Virtual labeling of mitochondria in living cells using correlative imaging and physics-guided deep learning. Biomed Opt Express. 2022;13(10):5495–516.

[26] Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative adversarial nets. Adv Neural Inf Process Syst. 2014;27. https://proceedings.neurips.cc/paper_files/paper/2014/hash/5ca3e9b122f61f8f06494c97b1afccf3-Abstract.html.

[27] Rana A, Yaunery G, Lowe A, Shah P. Computational Histological Staining and Destaining of Prostate Core Biopsy RGB Images with Generative Adversarial Neural Networks. In: 2018 17th Ieee International Conference on Machine Learning and Applications (Icmla). 2018. p. 828–834. .

[28] Shaban MT, Baur C, Navab N, Albarqouni S. Staingan: Stain style transfer for digital histological images. In: 2019 Ieee 16th international symposium on biomedical imaging (Isbi 2019). IEEE; 2019. p. 953–956.

[29] Wang RH, Song PM, Jiang SW, Yan CG, Zhu JK, Guo CF, et al. Virtual brightfield and fluorescence staining for Fourier ptychography via unsupervised deep learning. Opt Lett. 2020;45(19):5405–8. .

[30] Ye S, Zou J, Huang C, Xiang F, Wen Z, Wang N, et al. Rapid and label-free histological imaging of unprocessed surgical tissues via Dark-field Reflectance Ultraviolet Microscopy. iScience. 2022;105849.

[31] Petersen D, Mavarani L, Niedieker D, Freier E, Tannapfel A, Kotting C, et al. Virtual staining of colon cancer tissue by label-free Raman micro-spectroscopy. Analyst. 2017;142(8):1207–15. .

[32] Pradhan P, Meyer T, Vieth M, Stallmach A, Waldner M, Schmitt M, et al. Computational tissue staining of non-linear multimodal imaging using supervised and unsupervised deep learning. Biomed Opt Express. 2021;12(4):2280–98. .

[33] Bocklitz TW, Salah FS, Vogler N, Heuke S, Chernavskaia O, Schmidt C, et al. Pseudo-HE images derived from CARS/TPEF/SHG multimodal imaging in combination with Raman-spectroscopy as a pathological screening tool. BMC Cancer. 2016;16. .

[34] de Haan K, Zhang Y, Zuckerman JE, Liu T, Sisk AE, Diaz MF, et al. Deep learning-based transformation of H &E stained tissues into special stains. Nat Commun. 2021;12(1):1–13.

[35] Opstad I. Data set: Fluorescence microscopy videos of mitochondria in H9c2 cardiomyoblasts. DataverseNO. 2023. .

[36] Hong Y, Heo YJ, Kim B, Lee D, Ahn S, Ha SY, et al. Deep learning-based virtual cytokeratin staining of gastric carcinomas to measure tumor-stroma ratio. Sci Rep. 2021;11(1). .

[37] Ghahremani P, Li Y, Kaufman A, Vanguri R, Greenwald N, Angelo M, et al. Deep learning-inferred multiplex immunofluorescence for immunohistochemical image quantification. Nat Mach Intell. 2022;4(4):401–12.

[38] Park Y, Depeursinge C, Popescu G. Quantitative phase imaging in biomedicine. Nat Photonics. 2018;12(10):578–89.

[39] Tomczak A, Ilic S, Marquardt G, Engel T, Forster F, Navab N, et al. Multi-Task Multi-Domain Learning for Digital Staining and Classification of Leukocytes. IEEE Trans Med Imaging. 2021;40(10):2897–910. .

[40] Zhou KC, Qian R, Dhalla AH, Farsiu S, Izatt JA. Unified k-space theory of optical coherence tomography. Adv Opt Photonics. 2021;13(2):462–514.

[41] Drexler W, Fujimoto JG, et al. Optical coherence tomography: technology and applications. vol. 2. Springer; 2015.

[42] Lin SE, Jheng DY, Hsu KY, Liu YR, Huang WH, Lee HC, et al. Rapid pseudo-H&E imaging using a fluorescence-inbuilt optical coherence microscopic imaging system. Biomed Opt Express. 2021;12(8):5139–58. .

[43] Mari JM, Aung T, Cheng CY, Strouthidis NG, Girard MJA. A Digital Staining Algorithm for Optical Coherence Tomography Images of the Optic Nerve Head. Transl Vis Sci Technol. 2017;6(1). .

[44] Wang Z, Millet L, Mir M, Ding H, Unarunotai S, Rogers J, et al. Spatial light interference microscopy (SLIM). Opt Express. 2011;19(2):1016–26.

[45] Nguyen TH, Kandel ME, Rubessa M, Wheeler MB, Popescu G. Gradient light interference microscopy for 3D imaging of unlabeled specimens. Nat Commun. 2017;8(1):1–9.

[46] Kandel ME, He YR, Lee YJ, Chen THY, Sullivan KM, Aydin O, et al. Phase Imaging with Computational Specificity (PICS) for measuring dry mass changes in sub-cellular compartments. Nat Commun. 2020;11(1):1–10.

[47] Konda PC, Loetgering L, Zhou KC, Xu S, Harvey AR, Horstmeyer R. Fourier ptychography: current applications and future promises. Opt Express. 2020;28(7):9603–30.

[48] Cooke CL, Kong F, Chaware A, Zhou KC, Kim K, Xu R, et al. Physics-enhanced machine learning for virtual fluorescence microscopy. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. Computer Vision Foundation; 2021. p. 3803–3813.

[49] Croce AC, Bottiroli G. Autofluorescence spectroscopy and imaging: a tool for biomedical research and diagnosis. Eur J Histochem EJH. 2014;58(4):2461.

[50] Li XY, Zhang GX, Qiao H, Bao F, Deng Y, Wu JM, et al. Unsupervised content-preserving transformation for optical microscopy. Light-Sci Appl. 2021;10(1). .

[51] Zipfel WR, Williams RM, Webb WW. Nonlinear Magic: Multiphoton Microscopy in the Biosciences. Nat Biotechnol. 2003;21(11):1368–76. .

[52] Lemire S, Thoma OM, Kreiss L, Völkl S, Friedrich O, Neurath MF, et al. Natural NADH and FAD Autofluorescence as Label-Free Biomarkers for Discriminating Subtypes and Functional States of Immune Cells. Int J Mol Sci. 2022;23(4):2338.

[53] Gehlsen U, Szaszak M, Gebert A, Koop N, Huettmann G, Steven P. Non-Invasive Multi-Dimensional Two-Photon Microscopy enables optical fingerprinting (TPOF) of immune cells. J Biophotonics. 2015;8(6):466–79.

[54] Schürmann S, Weber C, Fink RH, Vogel M. Myosin Rods are a Source of Second Harmonic Generation Signals in Skeletal Muscle. In: Proceedings Volume 6442, Multiphoton Microscopy in the Biomedical Sciences VII; 2007. p. 64421U. .

[55] Heuke S, Vogler N, Meyer T, Akimov D, Kluschke F, Röwert-Huber HJ, et al. Multimodal Mapping of Human Skin. Br J Dermatol. 2013;169(4):794–803.

[56] Rosencwaig A. Photoacoustics and Photoacoustic Spectroscopy. vol. 57. Wiley; 1980. ISBN 10: 0894644505.

[57] Xu M, Wang LV. Photoacoustic Imaging in Biomedicine. Rev Sci Instrum. 2006;77(4):041101.

[58] Yao J, Wang LV. Photoacoustic microscopy. Laser Photonics Rev. 2013;7(5):758–78.

[59] Kang L, Li XF, Zhang Y, Wong TTW. Deep learning enables ultraviolet photoacoustic microscopy based histological imaging with near real-time virtual staining. Photoacoustics. 2022;25. .

[60] Li X, Kang L, Lo CT, Tsang VT, Wong TT. High-Speed Ultraviolet Photoacoustic Microscopy for Histological Imaging with Virtual-Staining assisted by Deep Learning. J Visualized Exp Jove. 2022;(182).

[61] Boktor M, Ecclestone B, Pekar V, Dinakaran D, Mackey JR, Fieguth P, et al. Deep-Learning-Based Virtual H&E Staining Using Total-Absorption Photoacoustic Remote Sensing (TA-PARS). In: Sci Rep. 2022;12:10296. .

[62] Boktor M, Ecclestone BR, Pekar V, Dinakaran D, Mackey JR, Fieguth P, et al. Virtual histological staining of label-free total absorption photoacoustic remote sensing (TA-PARS). Sci Rep. 2022;12(1):1–12.

[63] Karita M, Tada M, Okita K, Kodama T. Endoscopic Therapy for Early Colon Cancer: the Strip Biopsy Resection Technique. Gastrointest Endosc. 1991;37(2):128–32.

[64] Stefanchik D. Endoscopic Tissue Resection Device. Google Patents; 2010. US Patent 7,780,691.

[65] Akiyama M, Ota M, Nakajima H, Yamagata K, Munakata A. Endoscopic Mucosal Resection of Gastric Neoplasms Using a Ligating Device. Gastrointest Endosc. 1997;45(2):182–6.

[66] Ramos-Vara J, Miller M. When tissue antigens and antibodies get along: revisiting the technical aspects of immunohistochemistry–the red, brown, and blue technique. Vet Pathol. 2014;51(1):42–87.

[67] Wright DK, Manos MM. Sample Preparation from Paraffin-Embedded Tissues. PCR Protocol Guide Methods Appl. 1990;19:153–9.

[68] Mager S, Oomen MH, Morente MM, Ratcliffe C, Knox K, Kerr DJ, et al. Standard Operating Procedure for the Collection of Fresh Frozen Tissue Samples. Eur J Cancer. 2007;43(5):828–34.

[69] Cardiff RD, Miller CH, Munn RJ. Manual Hematoxylin and Eosin Staining of Mouse Tissue Sections. Cold Spring Harb Protoc. 2014;2014(6):073411.

[70] Whittaker P, Kloner R, Boughner D, Pickering J. Quantitative Assessment of Myocardial Collagen with Picrosirius Red Staining and Circularly Polarized Light. Basic Res Cardiol. 1994;89(5):397–410.

[71] Rivenson Y, Liu TR, Wei ZS, Zhang Y, de Haan K, Ozcan A. PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning. Light-Sci Appl. 2019;8. .

[72] Zhang YJ, de Haan K, Rivenson Y, Li JX, Delis A, Ozcan A. Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue. Light-Sci Appl. 2020;9(1). .

[73] Zhang Y, de Haan K, Li J, Rivenson Y, Ozcan A. Neural network-based multiplexed and micro-structured virtual staining of unlabeled tissue. In: Conference on Lasers and Electro-Optics, Technical Digest Series (Optica Publishing Group, 2022), paper ATh2I.2.

[74] Bautista PA, Yagi Y. Digital simulation of staining in histopathology multispectral images: enhancement and linear transformation of spectral transmittance. J Biomed Opt. 2012;17(5). .

[75] Mayerich D, Walsh MJ, Kadjacsy-Balla A, Ray PS, Hewitt SM, Bhargava R. Stain-less staining for computed histopathology. Technology. 2015;3(01):27–31.

[76] Gadermayr M, Appel V, Klinkhammer BM, Boor P, Merhof D. Which way round? A study on the performance of stain-translation for segmenting arbitrarily dyed histological images. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. Lecture Notes in Computer Science. vol. 11071. Cham: Springer. 2018. p. 165–173. .

[77] Fujitani M, Mochizuki Y, Iizuka S, Simo-Serra E, Kobayashi H, Iwamoto C, et al. Re-staining pathology images by FCNN. In: 2019 16th International Conference on Machine Vision Applications (MVA). IEEE; 2019. p. 1–6.

[78] Li D, Hui H, Zhang YQ, Tong W, Tian F, Yang X, et al. Deep Learning for Virtual Histological Staining of Bright-Field Microscopic Images of Unlabeled Carotid Artery Tissue. Mol Imaging Biol. 2020;22(5):1301–9. .

[79] Zhang GH, Ning B, Hui H, Yu TF, Yang X, Zhang HX, et al. Image-to-Images Translation for Multiple Virtual Histological Staining of Unlabeled Human Carotid Atherosclerotic Tissue. Mol Imaging Biol. 2022;24(1):31–41. .

[80] Yang X, Bai B, Zhang Y, Li Y, de Haan K, Liu T, et al. Virtual stain transfer in histology via cascaded deep neural networks. ACS Photonics. 2022;9(9):p3134-3143. .

[81] Otto F. DAPI Staining of Fixed Cells for High-Resolution Flow Cytometry of Nuclear DNA. In: Methods in Cell Biology. vol. 33. Elsevier; 1990. p. 105–10. .

[82] Jiang Z, Li B, Tran TN, Jiang J, Liu X, Ta D. Fluo-Fluo translation based on deep learning. Chinese Opt Lett. 2022;20(3):031701.

[83] Kandel ME, Kim E, Lee YJ, Tracy G, Chung HJ, Popescu G. Multiscale Assay of Unlabeled Neurite Dynamics Using Phase Imaging with Computational Specificity. Acs Sensors. 2021;6(5):1864–74. .

[84] Cheng SY, Fu SP, Kim YM, Song WY, Li YZ, Xue YJ, et al. Single-cell cytometry via multiplexed fluorescence prediction by label-free reflectance microscopy. Sci Adv. 2021;7(3). .

[85] Yuan E, Matusiak M, Sirinukunwattana K, Varma S, Kidzinski L, West R. Self-Organizing Maps for Cellular In Silico Staining and Cell Substate Classification. Front Immunol. 2021;12. .

[86] Guo SM, Yeh LH, Folkesson J, Ivanov IE, Krishnan AP, Keefe MG, et al. Revealing architectural order with quantitative label-free imaging and deep learning. Elife. 2020;9. .

[87] Liu Y, Yuan H, Wang ZY, Ji SW. Global Pixel Transformers for Virtual Staining of Microscopy Images. IEEE Trans Med Imaging. 2020;39(6):2256–66. .

[88] Burlingame EA, Margolin AA, Gray JW, Chang YH. SHIFT: speedy histopathological-to-immunofluorescent translation of whole slide images using conditional generative adversarial networks. Med Imaging 2018 Digit Pathol. 2018;10581. .

[89] Gu S, Lee RM, Benson Z, Ling C, Vitolo MI, Martin SS, et al. Label-free cell tracking enables collective motion phenotyping in epithelial monolayers. iScience. 2022;25(7):104678. .

[90] Ling C, Majurski M, Halter M, Stinson J, Plant A, Chalfoun J. Analyzing u-net robustness for single cell nucleus segmentation from phase contrast images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Computer Vision Foundation; 2020. p. 966–67.

[91] Goswami N, He YCR, Deng YH, Oh C, Sobh N, Valera E, et al. Label-free SARS-CoV-2 detection and classification using phase imaging with computational specificity. Light-Sci Appl. 2021;10(1). .

[92] Hu CF, He SH, Lee YJ, He YC, Kong EM, Li H, et al. Live-dead assay on unlabeled cells using phase imaging with computational specificity. Nat Commun. 2022;13(1). .

[93] Kolln LS, Salem O, Valli J, Hansen CG, McConnell G. Label2label: training a neural network to selectively restore cellular structures in fluorescence microscopy. J Cell Sci. 2022;135(3). .

[94] Jo Y, Cho H, Park WS, Kim G, Ryu D, Kim YS, et al. Label-free multiplexed microtomography of endogenous subcellular dynamics using generalizable deep learning. Nat Cell Biol. 2021;23(12):1329. .

[95] Hermsen M, Volk V, Bräsen JH, Geijs DJ, Gwinner W, Kers J, et al. Quantitative assessment of inflammatory infiltrates in kidney transplant biopsies using multiplex tyramide signal amplification and deep learning. Lab Investig. 2021;101(8):970–82.

[96] Tondeleir D, Lambrechts A, Müller M, Jonckheere V, Doll T, Vandamme D, et al. Cells lacking β-actin are genetically reprogrammed and maintain conditional migratory capacity. Mol Cell Proteomics. 2012;11(8):255–71.

[97] Anti-MAP2 antibody Data sheet [EPR19691] ab183830. Accessed Oct 2023. https://www.abcam.com/products/primary-antibodies/map2-antibody-epr19691-ab183830.html.

[98] Chen X, Kandel ME, Shenghua H, et al. Artificial confocal microscopy for deep label-free imaging. Nat Photonics. 2022. .

[99] Li D, Cho YK. High specificity of widely used phospho-tau antibodies validated using a quantitative whole-cell based assay. J Neurochem. 2020;152(1):122–35.

[100] Anti-Ki67 Antibody [Ki-67] data sheet (PE) (A86642). Antibodies.com 2023. Accessed Oct 2023. https://www.antibodies.com/de/ki67-antibody-ki-67-pe-a86642.

[101] Xu ZD, Li X, Zhu XH, Chen LY, He YH, Chen YP. Effective Immunohistochemistry Pathology Microscopy Image Generation Using CycleGAN. Front Mol Biosci. 2020;7. .

[102] Zhang R, Cao Y, Li Y, Liu Z, Wang J, He J, et al. MVFStain: Multiple virtual functional stain histopathology images generation based on specific domain mapping. Med Image Anal. 2022;80:102520.

[103] Liu Y, Li X, Zheng A, Zhu X, Liu S, Hu M, et al. Predict Ki-67 positive cells in H &E-stained images using deep learning independently from IHC-stained images. Front Mol Biosci. 2020;7:183.

[104] Bai B, Wang H, Li Y, de Haan K, Colonnese F, Wan Y, et al. Label-free virtual HER2 immunohistochemical staining of breast tissue using deep learning. BME Front. 2022;2022:9786242. .

[105] Gareau DS. Feasibility of digitally stained multimodal confocal mosaics to simulate histopathology. J Biomed Opt. 2009;14(3). .

[106] Bini J, Spain J, Nehal K, Hazelwood V, DiMarzio C, Rajadhyaksha M. Confocal mosaicing microscopy of basal cell carcinomas ex vivo: progress in digital staining to simulate histology-like appearance. Adv Biomed Clin Diagn Syst Ix. 2011;7890. .

[107] Bini J, Spain J, Nehal K, Hazelwood V, DiMarzio C, Rajadhyaksha M. Confocal mosaicing microscopy of human skin ex vivo: spectral analysis for digital staining to simulate histology-like appearance. J Biomed Opt. 2011;16(7). .

[108] Amrania H, Antonacci G, Chan CH, Drummond L, Otto WR, Wright NA, et al. Digistain: a digital staining instrument for histopathology. Opt Express. 2012;20(7):7290–9. .

[109] Giacomelli MG, Husvogt L, Vardeh H, Faulkner-Jones BE, Hornegger J, Connolly JL, et al. Virtual Hematoxylin and Eosin Transillumination Microscopy Using Epi-Fluorescence Imaging. PLoS ONE. 2016;11(8). .

[110] Elfer KN, Sholl AB, Wang M, Tulman DB, Mandava H, Lee BR, et al. DRAQ5 and Eosin (‘D &E’) as an Analog to Hematoxylin and Eosin for Rapid Fluorescence Histology of Fresh Tissues. PLoS ONE. 2016;11(10). .

[111] Fan X, Tang ZY, Healy JJ, O’Dwyer K, Hennelly BM. Label-free Rheinberg staining of cells using digital holographic microscopy and spatial light interference microscopy. Adv Opt Imaging Technol Ii. 2019;11186. .

[112] Fereidouni F, Todd A, Li YH, Chang CW, Luong K, Rosenberg A, et al. Dual-mode emission and transmission microscopy for virtual histochemistry using hematoxylin- and eosin-stained tissue sections. Biomed Opt Express. 2019;10(12):6516–30. .

[113] Fan X, Healy JJ, O’Dwyer K, Hennelly BM. Label-free color staining of quantitative phase images. Opt Lasers Eng. 2020;129. .

[114] Bautista PA, Abe T, Yamaguchi M, Yagi Y, Ohyama N. Digital staining of unstained pathological tissue samples through spectral transmittance classification. Opt Rev. 2005;12(1):7–14. .

[115] Bautista PA, Abe T, Yamaguchi M, Yagi Y, Ohyama N. Digital Staining of Pathological Images: Dye amount correction for improved classification performance. Med Imaging 2007 Comput-Aided Diagn Pts 1 2. 2007;6514. .

[116] Hanselmann M, Kothe U, Kirchner M, Renard BY, Amstalden ER, Glunde K, et al. Toward Digital Staining using Imaging Mass Spectrometry and Random Forests. J Proteome Res. 2009;8(7):3558–67. .

[117] Hinton G, LeCun Y, Bengio Y. Deep learning. Nature. 2015;521(7553):436–44.

[118] Park Y, Park W, Jo Y, Min H, Cho H. Method and Apparatus for Generating 3D Fluorescent Label Image of Label-Free using 3D Refractive Index Tomography and Deep Learning. European Patent Application. 2021. Patent number: 11450062, Filed: March 19, 2020 Date of Patent: September 20, 2022.

[119] Stenman S, Bychkov D, Kücükel H, Linder N, Haglund C, Arola J, et al. Antibody supervised training of a deep learning based algorithm for leukocyte segmentation in papillary thyroid carcinoma. IEEE J Biomed Health Inf. 2020;25(2):422–8.

[120] Zhuge H, Summa B, Hamm J, Brown JQ. Deep learning 2D and 3D optical sectioning microscopy using cross-modality Pix2Pix cGAN image translation. Biomed Opt Express. 2021;12(12):7526–43. .

[121] Segerer FJ, Nekolla K, Rognoni L, Kapil A, Schick M, Angell H, et al. Novel Deep Learning Approach to Derive Cytokeratin Expression and Epithelium Segmentation from DAPI. In: Medical Imaging with Deep Learning. CoRR; 2022. .

[122] Li J, Garfinkel J, Zhang X, Wu D, Zhang Y, De Haan K, et al. Biopsy-free in vivo virtual histology of skin using deep learning. Light Sci Appl. 2021;10(1):1–22.

[123] Shi L, Wong IH, Lo CT, Wong TT. One-side Virtual Histological Staining Model for Complex Human Samples. In: 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), Ioannina, Greece; 2022. p. 1–4..

[124] Fanous MJ, He S, Sengupta S, Tangella K, Sobh N, Anastasio MA, et al. White blood cell detection, classification and analysis using phase imaging with computational specificity (PICS). Sci Rep. 2022;12(1):1–10.

[125] Cao R, Nelson SD, Davis S, Liang Y, Luo Y, Zhang Y, et al. Label-free intraoperative histology of bone tissue via deep-learning-assisted ultraviolet photoacoustic microscopy. Nat Biomed Eng. 2023;7(2):124–34. .

[126] de Bel T, Hermse, M, Kers J, van der Laak J, Litjens G. Stain-Transforming Cycle-Consistent Generative Adversarial Networks for Improved Segmentation of Renal Histopathology. In: Proceedings of The 2nd International Conference on Medical Imaging with Deep Learning, in Proceedings of Machine Learning Research. 2019;102:151–163 Available from: https://proceedings.mlr.press/v102/de-bel19a.html.

[127] Teramoto A, Yamada A, Tsukamoto T, Kiriyama Y, Sakurai E, Shiogama K, et al. Mutual stain conversion between Giemsa and Papanicolaou in cytological images using cycle generative adversarial network. Heliyon. 2021;7(2):e06331.

[128] Cohen JP, Luck M, Honari S. Distribution matching losses can hallucinate features in medical image translation. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. Lecture Notes in Computer Science. vol. 11070. Cham: Springer. 2018. p. 529–36. .

[129] Salimans T, Goodfellow I, Zaremba W, Cheung V, Radford A, Chen X. Improved techniques for training gans. Adv Neural Inf Process Syst. 2016;29. https://proceedings.neurips.cc/paper_files/paper/2016/hash/8a3363abe792db2d8761d6403605aeb7-Abstract.html.

[130] Durugkar I, Gemp I, Mahadevan S. Generative multi-adversarial networks. arXiv preprint arXiv:1611.01673. 2016.

[131] Frogner C, Zhang C, Mobahi H, Araya M, Poggio TA. Learning with a Wasserstein loss. Adv Neural Inf Process Syst. 2015;28. https://proceedings.neurips.cc/paper/2015/hash/a9eb812238f753132652ae09963a05e9-Abstract.html.

[132] Brunet D, Vrscay ER, Wang Z. On the mathematical properties of the structural similarity index. IEEE Trans Image Process. 2011;21(4):1488–99.

[133] Wang Z, Simoncelli EP, Bovik AC. Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003. Vol. 2. Pacific Grove; 2003. p. 1398-1402. .

[134] Borhani N, Bower AJ, Boppart SA, Psaltis D. Digital staining through the application of deep neural networks to multi-modal multi-photon microscopy. Biomed Opt Express. 2019;10(3):1339–50.

[135] Bayramoglu N, Kaakinen M, Eklund L, Heikkila J. Towards Virtual H &E Staining of Hyperspectral Lung Histology Images Using Conditional Generative Adversarial Networks. In: 2017 IEEE International Conference on Computer Vision Workshops (Iccvw 2017). 2017. p. 64–71. .

[136] Nygate YN, Levi M, Mirsky SK, Turko NA, Rubin M, Barnea I, et al. Holographic virtual staining of individual biological cells. Proc Natl Acad Sci USA. 2020;117(17):9223–31. .

[137] Trullo R, Bui QA, Tang Q, Olfati-Saber R. Image Translation Based Nuclei Segmentation for Immunohistochemistry Images. In: Mukhopadhyay A, Oksuz I, Engelhardt S, Zhu D, Yuan Y. (eds) Deep Generative Models. DGM4MICCAI 2022. Lecture Notes in Computer Science, vol 13609. Cham: Springer. .

[138] Bautista PA, Abe T, Yamaguchi M, Yagi Y, Ohyama N. Digital staining of pathological tissue specimens using spectral transmittance. Med Imaging 2005 Image Process Pt 1-3. 2005;5747:1892–1903. .

[139] Bautista PA, Abe T, Yamaguchi M, Yagi Y, Ohyama N. Digital staining for multispectral images of pathological tissue specimens based on combined classification of spectral transmittance. Comput Med Imaging Graph. 2005;29(8):649–57. .

[140] Bautista PA, Yagi Y. Digital Staining for Histopathology Multispectral Images by the Combined Application of Spectral Enhancement and Spectral Transformation. In: 2011 Annual International Conference of the Ieee Engineering in Medicine and Biology Society (Embc); 2011. p. 8013–8016. .

[141] Lotfollahi M, Daeinejad D, Berisha S, Mayerich D. Digital Staining of High-Resolution Ftir Spectroscopic Images. In: Appl Spectrosc. 2019;73(5):556–64. .

[142] Bulten W, Bándi P, Hoven J, Loo Rvd, Lotz J, Weiss N, et al. Epithelium segmentation using deep learning in H &E-stained prostate specimens with immunohistochemistry as reference standard. Sci Rep. 2019;9(1):1–10.

[143] Lotfollahi M, Berisha S, Daeinejad D, Mayerich D. Digital Staining of High-Definition Fourier Transform Infrared (FT-IR) Images Using Deep Learning. Appl Spectrosc. 2019;73(5):556–64. .

[144] Perez-Anker J, Malvehy J, Moreno-Ramirez D. Ex Vivo Confocal Microscopy Using Fusion Mode and Digital Staining: Changing Paradigms in Histological Diagnosis. Actas Dermo-Sifiliograficas. 2020;111(3):236–42. .

[145] Schuurmann M, Stecher MM, Paasch U, Simon JC, Grunewald S. Evaluation of digital staining forex vivoconfocal laser scanning microscopy. J Eur Acad Dermatol Venereol. 2020;34(7):1496–9. .

[146] Mercan C, Mooij GCAM, Tellez D, Lotz J, Weiss N, van Gerven M, Staining Virtual, for Mitosis Detection in Breast Histopathology. In, et al. IEEE 17th International Symposium on Biomedical Imaging (ISBI). Iowa City. 2020;2020:1770–4. .

[147] Jackson CR, Sriharan A, Vaickus LJ. A machine learning algorithm for simulating immunohistochemistry: development of SOX10 virtual IHC and evaluation on primarily melanocytic neoplasms. Mod Pathol. 2020;33(9):1638–48.

[148] Oszutowska-Mazurek D, Parafiniuk M, Mazurek P. Virtual UV Fluorescence Microscopy from Hematoxylin and Eosin Staining of Liver Images Using Deep Learning Convolutional Neural Network. Appl Sci-Basel. 2020;10(21). .

[149] Picon A, Medela A, Sánchez-Peralta LF, Cicchi R, Bilbao R, Alfieri D, et al. Autofluorescence image reconstruction and virtual staining for in-vivo optical biopsying. IEEE Access. 2021;9:32081–93.

[150] Fredman G, Christensen RL, Ortner VK, Haedersdal M. Visualization of energy-based device-induced thermal tissue alterations using bimodal ex-vivo confocal microscopy with digital staining. A proof-of-concept study. In Skin Res Technol. 2022;28:564–70. .

[151] Meng XY, Li X, Wang X. A Computationally Virtual Histological Staining Method to Ovarian Cancer Tissue by Deep Generative Adversarial Networks. Comput Math Methods Med. 2021;2021. .

[152] Vladimirova G, Ruini C, Kapp F, Kendziora B, Ergün EZ, Bağcı IS, et al. Ex vivo confocal laser scanning microscopy: A diagnostic technique for easy real-time evaluation of benign and malignant skin tumours. J Biophoton. 2022;15(6):e202100372.

[153] Soltani S, Ojaghi A, Qiao H, Kaza N, Li X, Dai Q, et al. Prostate cancer histopathology using label-free multispectral deep-UV microscopy quantifies phenotypes of tumor aggressiveness and enables multiple diagnostic virtual stains. Sci Rep. 2022;12(1):1–17.

[154] Liu K, Li B, Wu W, May C, Chang O, Knezevich S, et al. VSGD-Net: Virtual Staining Guided Melanocyte Detection on Histopathological Images. In: IEEE Winter Conf Appl Comput Vis. 2023;2023:1918–1927. .

[155] Ruini C, Vladimirova G, Kendziora B, Salzer S, Ergun E, Sattler E, et al. Ex-vivo fluorescence confocal microscopy with digital staining for characterizing basal cell carcinoma on frozen sections: A comparison with histology. J Biophotonics. 2021;14(8). .

[156] Kaza N, Ojaghi A, Costa PC, Robles FE. Deep learning based virtual staining of label-free ultraviolet (UV) microscopy images for hematological analysis. In: Label-free Biomedical Imaging and Sensing (LBIS) 2021. vol. 11655. Proceedings of the SPIE; 2021. p. 116550C. .

[157] Kaza N, Ojaghi A, Robles FE. Automated virtual staining, segmentation and classification of deep ultraviolet (UV) microscopy images for hematological analysis. In: Biophotonics Congress: Biomedical Optics 2022 (Translational, Microscopy, OCT, OTS, BRAIN), Technical Digest Series (Optica Publishing Group, 2022), paper MW4A.5. .

[158] Ortner VK, Sahu A, Cordova M, Kose K, Aleissa S, Alessi-Fox C, et al. Exploring the utility of Deep Red Anthraquinone 5 for digital staining of ex vivo confocal micrographs of optically sectioned skin. J Biophotonics. 2021;14(4). .

[159] Tavakkoli A, Kamran SA, Hossain KF, Zuckerbrod SL. A novel deep learning conditional generative adversarial network for producing angiography images from retinal fundus photographs. Sci Rep. 2020;10(1):1–15.

[160] Sasajima K, Kudo SE, Inoue H, Takeuchi T, Kashida H, Hidaka E, et al. Real-time in vivo virtual histology of colorectal lesions when using the endocytoscopy system. Gastrointest Endosc. 2006;63(7):1010–7. .

[161] Cetin O, Chen M, Ziegler P, Wild P, Koeppl H. Deep learning-based restaining of histopathological images. In: 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE Computer Society; 2022. p. 1467–1474.

[162] Wagner SJ, Matek C, Shetab Boushehri S, Boxberg M, Lamm L, Sadafi A, et al. Make deep learning algorithms in computational pathology more reproducible and reusable. Nat Med. 2022;28(9):1744–6.

[163] Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell. 2019;1(5):206–15. .

[164] Lu S, Stein JE, Rimm DL, Wang DW, Bell JM, Johnson DB, et al. Comparison of biomarker modalities for predicting response to PD-1/PD-L1 checkpoint blockade: a systematic review and meta-analysis. JAMA Oncol. 2019;5(8):1195–204.

[165] Özbey M, Dar SU, Bedel HA, Dalmaz O, Özturk Ş, Güngör A, et al. Unsupervised Medical Image Translation with Adversarial Diffusion Models. arXiv preprint arXiv:2207.08208. 2022.

[166] Guan H, Li D, Park Hc, Li A, Yue Y, Gau YA, et al. Deep-learning two-photon fiberscopy for video-rate brain imaging in freely-behaving mice. Nat Commun. 2022;13(1):1–9.