Deep learning-based color transfer biomedical imaging technology

Yinxu Bian; Tao Xing; Weijie Deng; Qin Xian; Honglei Qiao; Qian Yu; Jilong Peng; Xiaofei Yang; Yannan Jiang; Jiaxiong Wang; Shenmin Yang; Renbin Shen; Hua Shen; Cuifang Kuang

doi:10.3788/IRLA20210891

[1] N Chantziantoniou, A D Donnelly, M Mukherjee, et al. Inception and development of the papanicolaou stain method. Acta Cytologica, 61, 266-280(2017).

[2] A H Fischer, K A Jacobson, J Rose, et al. Hematoxylin and eosin staining of tissue and cell sections. Cold Spring Harbor Protocols, 2008, pdb.prot4986(2008).

[3] E H Beutner. Immunofluorescent staining: The fluorescent antibody method. Bacteriological Reviews, 25, 49-76(1961).

[4] H Irshad, A Veillard, L Roux, et al. Methods for nuclei detection, segmentation, and classification in digital histopathology: A review—current status and future potential. IEEE Reviews in Biomedical Engineering, 7, 97-114(2013).

[5] S T Chari, S Echelmeyer. Can histopathology be the “Gold Standard” for diagnosing autoimmune pancreatitis?. Gastro-enterology, 129, 2118-2120(2005).

[6] D Onder, S Zengin, S Sarioglu. A review on color normalization and color deconvolution methods in histopathology. Applied Immunohistochemistry & Molecular Morphology, 22, 713-719(2014).

[7] Matos J De, Jr A S Britto, L E S Oliveira, et al. Histopathologic image processing: A review. arXiv preprint, 1904.07900(2019).

[8] R Ursache, T G Andersen, P Marhavý, et al. A protocol for combining fluorescent proteins with histological stains for diverse cell wall components. The Plant Journal, 93, 399-412(2018).

[9] S A Taqi, S A Sami, L B Sami, et al. A review of artifacts in histopathology. Journal of Oral and Maxillofacial Pathology: JOMFP, 22, 279(2018).

[10] Celis R, Romero E. Unsupervised col nmalisation f H E stained histopathology image analysis[C]11th International Symposium on Medical Infmation Processing Analysis. International Society f Optics Photonics, 2015, 9681: 968104.

[11] T Abraham, A Shaw, D O'Connor, et al. Slide-free MUSE microscopy to H&E histology modality conversion via unpaired image-to-image translation GAN models. arXiv preprint, 2008.08579(2020).

[12] Haan K de, Y Rivenson, Y Wu, et al. Deep-learning-based image reconstruction and enhancement in optical microscopy. Proceedings of the IEEE, 108, 30-50(2019).

[13] Liang H, Plataniotis K N, Li X. Stain style transfer of histopathology images via structurepreserved generative learning[C]International Wkshop on Machine Learning f Medical Image Reconstruction. Cham: Springer, 2020: 153162.

[14] Mahapatra D, Bozgtabar B, Thiran J P, et al. Structure preserving stain nmalization of histopathology images using self supervised semantic guidance[C]International Conference on Medical Image Computing ComputerAssisted Intervention. Cham: Springer, 2020: 309319.

[15] M Shi, K L McMillan, J Wu, et al. Cisplatin nephrotoxicity as a model of chronic kidney disease. Laboratory Investigation, 98, 1105-1121(2018).

[16] Y Rivenson, Z Göröcs, H Günaydin, et al. Deep learning microscopy. Optica, 4, 1437-1443(2017).

[17] C C Bilgin, J Rittscher, R Filkins, et al. Digitally adjusting chromogenic dye proportions in brightfield microscopy images. Journal of Microscopy, 245, 319-330(2012).

[18] M Veta, J P W Pluim, Diest P J Van, et al. Breast cancer histopathology image analysis: A review. IEEE Transactions on Biomedical Engineering, 61, 1400-1411(2014).

[19] S Hooja, N Pal, B Malhotra, et al. Comparison of Ziehl Neelsen & Auramine O staining methods on direct and concentrated smears in clinical specimens. The Indian Journal of Tuberculosis, 58, 72-76(2011).

[20] K K Mak, M R Pichika. Artificial intelligence in drug development: present status and future prospects. Drug Discovery Today, 24, 773-780(2019).

[21] G Gunčar, M Kukar, M Notar, et al. An application of machine learning to haematological diagnosis. Scientific Reports, 8, 1-12(2018).

[22] H Chen, O Engkvist, Y Wang, et al. The rise of deep learning in drug discovery. Drug Discovery Today, 23, 1241-1250(2018).

[23] C Krittanawong. The rise of artificial intelligence and the uncertain future for physicians. European Journal of Internal Medicine, 48, e13-e14(2018).

[24] B T Grys, D S Lo, N Sahin, et al. Machine learning and computer vision approaches for phenotypic profiling. Journal of Cell Biology, 216, 65-71(2017).

[25] Y Rivenson, T Liu, Z Wei, et al. PhaseStain: The digital staining of label-free quantitative phase microscopy images using deep learning. Light: Science & Applications, 8, 1-11(2019).

[26] Shaban M T, Baur C, Navab N, et al. Staingan: Stain style transfer f digital histological images[C]2019 IEEE 16 th International Symposium on Biomedical Imaging (ISBI 2019). IEEE, 2019: 953956.

[27] F L Lizzi, M Astor, T Liu, et al. Ultrasonic spectrum analysis for tissue assays and therapy evaluation. International Journal of Imaging Systems and Technology, 8, 3-10(1997).

[28] M Linzer, S J Norton. Ultrasonic tissue characterization. Annual Review of Biophysics and Bioengineering, 11, 303-329(1982).

[29] M Xu, L V Wang. Photoacoustic imaging in biomedicine. Review of Scientific Instruments, 77, 041101(2006).

[30] M Diem, L Chiriboga, H Yee. Infrared spectroscopy of human cells and tissue. VIII. Strategies for analysis of infrared tissue mapping data and applications to liver tissue. Biopolymers: Original Research on Biomolecules, 57, 282-290(2000).

[31] Xiao X, Ma L. Col transfer in crelated col space[C]Proceedings of the 2006 ACM International Conference on Virtual Reality Continuum Its Applications, 2006: 305309.

[32] E Reinhard, M Adhikhmin, B Gooch, et al. Color transfer between images. IEEE Computer Graphics and Applications, 21, 34-41(2001).

[33] Welsh T, Ashikhmin M, Mueller K. Transferring col to greyscale images[C]Proceedings of the 29 th Annual Conference on Computer Graphics Interactive Techniques, 2002: 277280.

[34] M Byra, M Galperin, H Ojeda-Fournier, et al. Breast mass classification in sonography with transfer learning using a deep convolutional neural network and color conversion. Medical Physics, 46, 746-755(2019).

[35] Hwang Y, Lee J Y, So Kweon I, et al. Col transfer using probabilistic moving least squares[C]Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, 2014: 33423349.

[36] Y Jing, Y Yang, Z Feng, et al. Neural style transfer: A review. IEEE Transactions on Visualization and Computer Graphics, 26, 3365-3385(2019).

[37] S Wan, Y Xia, L Qi, et al. Automated colorization of a grayscale image with seed points propagation. IEEE Transactions on Multimedia, 22, 1756-1768(2020).

[38] L A Gatys, A S Ecker, M Bethge. A neural algorithm of artistic style. arXiv preprint, 1508.06576(2015).

[39] Albawi S, Mohammed T A, AlZawi S. Understing of a convolutional neural wk[C]2017 International Conference on Engineering Technology (ICET). IEEE, 2017: 16.

[40] Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions[C]Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, 2015: 19.

[41] Long J, Shelhamer E, Darrell T. Fully convolutional wks f semantic segmentation[C]Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, 2015: 34313440.

[42] Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional wks[C]Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, 2017: 47004708.

[43] K Simonyan, A Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint, 1409.1556(2014).

[44] A Krizhevsky, I Sutskever, G E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 1097-1105(2012).

[45] Y Li, Z B Hao, H Lei. Survey of convolutional neural network. Journal of Computer Applications, 36, 2508-2515(2016).

[46] F J Pineda. Generalization of back-propagation to recurrent neural networks. Physical Review Letters, 59, 2229(1987).

[47] Johnson J, Alahi A, Li Feifei. Perceptual losses f realtime style transfer superresolution[C]European Conference on Computer Vision. Cham: Springer, 2016: 694711.

[48] Liu X C, Cheng M M, Lai Y K, et al. Depthaware neural style transfer[C]Proceedings of the Symposium on NonPhotealistic Animation Rendering, 2017: 110.

[49] Goodfellow I, PougetAbadie J, Mirza M, et al. Generative adversarial s[C]Advances in Neural Infmation Processing Systems, 2014 : 26722680.

[50] H Liu, Z Fu, J Han, et al. Single satellite imagery simultaneous super-resolution and colorization using multi-task deep neural networks. Journal of Visual Communication and Image Representation, 53, 20-30(2018).

[51] M Izadyyazdanabadi, E Belykh, X Zhao, et al. Fluorescence image histology pattern transformation using image style transfer. Frontiers in Oncology, 9, 519(2019).

[52] Y Rivenson, H Wang, Z Wei, et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nature Biomedical Engineering, 3, 466-477(2019).

[53] Y Rivenson, Haan K de, W D Wallace, et al. Emerging advances to transform histopathology using virtual staining. BME Frontiers, 2020, 9647163(2020).

[54] Nishar H, Chavanke N, Singhal N. Histopathological stain transfer using style transfer wk with adversarial loss[C]International Conference on Medical Image Computing ComputerAssisted Intervention. Cham: Springer, 2020: 330340.

[55] H Touvron, M Douze, M Cord, et al. Powers of layers for image-to-image translation. arXiv preprint, 2008.05763(2020).

[56] Z Zuo, Q Xu, H Zhang, et al. Multimodal image-to-image translation via mutual information estimation and maximization. arXiv preprint, 2008.03529(2020).

[57] Isola P, Zhu J Y, Zhou T, et al. Imagetoimage translation with conditional adversarial wks[C]Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, 2017: 11251134.

[58] Dai B, Fidler S, Urtasun R, et al. Towards diverse natural image deions via a conditional gan[C]Proceedings of the IEEE International Conference on Computer Vision, 2017: 29702979.

[59] Zhu J Y, Park T, Isola P, et al. Unpaired imagetoimage translation using cycleconsistent adversarial wks[C]Proceedings of the IEEE International Conference on Computer Vision, 2017: 22232232.

[60] Almahairi A, Rajeshwar S, Sdoni A, et al. Augmented cyclegan: Learning manytomany mappings from unpaired data[C]International Conference on Machine Learning. PMLR, 2018: 195204.

[61] M E Tschuchnig, G J Oostingh, M Gadermayr. Generative adversarial networks in digital pathology: A survey on trends and future potential. Patterns, 1, 100089(2020).

[62] A H Petriceks, J C Olivas, S Srivastava. Trends in geriatrics graduate medical education programs and positions, 2001 to 2018. Gerontology and Geriatric Medicine, 4, 2333721418777659(2018).

[63] D Shen, G Wu, H I Suk. Deep learning in medical image analysis. Annual Review of Biomedical Engineering, 19, 221-248(2017).

[64] J J Barcia. The Giemsa stain: Its history and applications. International Journal of Surgical Pathology, 15, 292-296(2007).

[65] M E Daykin, R S Hussey. Staining and histopathological techniques. An Advanced Treatise on Meloidogyne, 2, 39-48(1985).

[66] P Pradhan, T Meyer, M Vieth, et al. Computational tissue staining of non-linear multimodal imaging using supervised and unsupervised deep learning. Biomedical Optics Express, 12, 2280-2298(2021).

[67] H B Barlow. Unsupervised learning. Neural Computation, 1, 295-311(1989).

[68] A Teramoto, A Yamada, T Tsukamoto, et al. Mutual stain conversion between Giemsa and Papanicolaou in cytological images using cycle generative adversarial network. Heliyon, 7, e06331(2021).

[69] S Padma, R PV, R Kante, et al. A comparative study of Staining characteristics of Leishman-Geimsa cocktail and Papanicolaou stain in Cervical Cytology. Asian Pacific Journal of Health Sciences, 5, 233-236(2018).

[70] B Gollapudi, O P Kamra. Applications of a simple Giemsa-staining method in the micronucleus test. Mutat Res, 64, 45-46(1979).

[71] M Woolman, A Tata, E Bluemke, et al. An assessment of the utility of tissue smears in rapid cancer profiling with desorption electrospray ionization mass spectrometry (DESI-MS). Journal of The American Society for Mass Spectrometry, 28, 145-153(2016).

[72] M K K Niazi, A V Parwani, M N Gurcan. Digital pathology and artificial intelligence. The Lancet Oncology, 20, e253-e261(2019).

[73] Y C Lo, I F Chung, S N Guo, et al. Cycle-consistent GAN-based stain translation of renal pathology images with glomerulus detection application. Applied Soft Computing, 98, 106822(2021).

[74] Khojasteh M, Ward R, MacAulay C. Quantification of membrane IHC stains through multispectral imaging[C]2012 9th IEEE International Symposium on Biomedical Imaging (ISBI). IEEE, 2012: 752755.

[75] Z Xu, C F Moro, B Bozóky, et al. GAN-based virtual re-staining: A promising solution for whole slide image analysis. arXiv preprint, 1901.04059(2019).

[76] Bel T de, J M Bokhorst, der Laak J van, et al. Residual cyclegan for robust domain transformation of histopathological tissue slides. Medical Image Analysis, 70, 102004(2021).

[77] M B Bornstein. Reconstituted rat-tail collagen used as substrate for tissue cultures on coverslips in Maximow slides and roller tubes. Laboratory Investigation, 7, 134-137(1958).

[78] H Boonstra, J W Oosterhuis, A M Oosterhuis, et al. Cervical tissue shrinkage by formaldehyde fixation, paraffin wax embedding, section cutting and mounting. Virchows Archiv A, 402, 195-201(1983).

[79] E C Jensen. Quantitative analysis of histological staining and fluorescence using ImageJ. The Anatomical Record, 296, 378-381(2013).

[80] L Kang, X Li, Y Zhang, et al. Deep learning enables ultraviolet photoacoustic microscopy based histological imaging with near real-time virtual staining. Photoacoustics, 25, 100308(2021).

[81] J W Baik, H Kim, M Son, et al. Intraoperative label-free photoacoustic histopathology of clinical specimens. Laser & Photonics Reviews, 15, 2100124(2021).

[82] D Morrison, D Harris-Birtill, P D Caie. Generative deep learning in digital pathology workflows. The American Journal of Pathology, 191, 1717-1723(2021).

[83] Zhou N, Cai D, Han X, et al. Enhanced cycleconsistent generative adversarial wk f col nmalization of H&E stained images[C]International Conference on Medical Image Computing ComputerAssisted Intervention. Cham: Springer, 2019: 694702.

[84] H Cho, S Lim, G Choi, et al. Neural stain-style transfer learning using gan for histopathological images. arXiv preprint, 1710.08543(2017).

[85] B Li, A Keikhosravi, A G Loeffler, et al. Single image super-resolution for whole slide image using convolutional neural networks and self-supervised color normalization. Medical Image Analysis, 68, 101938(2021).

[86] X Chen, J Yu, S Cheng, et al. An unsupervised style normalization method for cytopathology images. Computational and Structural Biotechnology Journal, 19, 3852-3863(2021).

[87] A Vahadane, T Peng, A Sethi, et al. Structure-preserving color normalization and sparse stain separation for histological images. IEEE Transactions on Medical Imaging, 35, 1962-1971(2016).

[88] Macenko M, Niethammer M, Marron J S, et al. A method f nmalizing histology slides f quantitative analysis[C]2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. IEEE, 2009: 11071110.

[89] A M Khan, N Rajpoot, D Treanor, et al. A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution. IEEE Transactions on Biomedical Engineering, 61, 1729-1738(2014).

[90] A Gupta, R Duggal, S Gehlot, et al. GCTI-SN: Geometry-inspired chemical and tissue invariant stain normalization of microscopic medical images. Medical Image Analysis, 65, 101788(2020).

[91] Y Zheng, Z Jiang, H Zhang, et al. Adaptive color deconvolution for histological WSI normalization. Computer Methods and Programs in Biomedicine, 170, 107-120(2019).

[92] Shaban M T, Baur C, Navab N, et al. Staingan: Stain style transfer f digital histological images[C]2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). IEEE, 2019: 953956.

[93] D Tellez, G Litjens, P Bándi, et al. Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. Medical Image Analysis, 58, 101544(2019).

[94] Y Bian, Y Jiang, J Wang, et al. Deep learning colorful ptychographic iterative engine lens-less diffraction microscopy. Optics and Lasers in Engineering, 150, 106843(2022).

[95] H Shen, J Gao. Deep learning virtual colorful lens-free on-chip microscopy. Chinese Optics Letters, 18, 121705(2020).

[96] Y Bian, Y Jiang, Y Huang, et al. Deep learning virtual colorization overcoming chromatic aberrations in singlet lens microscopy. APL Photonics, 6, 031301(2021).