volume | PIER Journals

Abstract

We propose a 3-band iteration method to transfer knowledge learned from RGB (red, green and blue) data pretrained models to the hyperspectral domain. We demonstrate classification of a Multi-spectral Choledoch database for cholangiocarcinoma diagnosis. The results show quicker and more stable training progress: 92%+ top-1 accuracy in the initial 3 epochs. Some advanced training techniques in the RGB computer vision field can be easily utilized and transferred to the hyperspectral domain without adding more parameters to the original architecture. The computational cost and hardware requirements remain the same. After voting, the highest top-1 accuracy on the validation set reached 95.4%, and the highest top-1 accuracy on the test set reached 94.3%. We can directly use our models trained on high-dimensional spectral images to test and infer on RGB color images. We visualized some results by Grad-CAM (Gradient-weighted Class Activation Mapping) on RGB test data, and it shows the transferability of knowledge. We trained the models solely on classification task on spectral data, and these models showed their ability to predict on RGB images with different fields of views. The results indicate good segmentation even when the model has never been trained on any segmentation task.

1. Deng, Jia, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li, "Imagenet: A large-scale hierarchical image database," 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248-255, 2009.

2. Schuhmann, Christoph, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, "LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs," ArXiv Preprint ArXiv:2111.02114, 2021.

3. Schuhmann, Christoph, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev, "LAION-5B: An open large-scale dataset for training next generation image-text models," Advances in Neural Information Processing Systems, Vol. 35, 25278-25294, 2022.

4. Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, "Learning transferable visual models from natural language supervision," Proceedings of the 38th International Conference on Machine Learning, PMLR, 8748-8763, 2021.

5. West, Jeremy, Dan Ventura, and Sean Warnick, "Spring research presentation: A theoretical foundation for inductive transfer," Brigham Young University, College of Physical and Mathematical Sciences, Vol. 1, No. 08, 2007.

6. George Karimpanal, Thommen and Roland Bouffanais, "Self-organizing maps for storage and transfer of knowledge in reinforcement learning," Adaptive Behavior, Vol. 27, No. 2, 111-126, 2019.

7. Maitra, Durjoy Sen, Ujjwal Bhattacharya, and Swapan K. Parui, "CNN based common approach to handwritten character recognition of multiple scripts," 2015 13th International Conference on Document Analysis and Recognition (ICDAR), 1021-1025, 2015.

8. Bird, Jordan J., Jhonatan Kobylarz, Diego R. Faria, Anikó Ekárt, and Eduardo P. Ribeiro, "Cross-domain MLP and CNN transfer learning for biological signal processing: EEG and EMG," IEEE Access, Vol. 8, 54789-54801, 2020.
doi:10.1109/ACCESS.2020.2979074

9. Kabir, H. M. Dipu, Moloud Abdar, Abbas Khosravi, Seyed Mohammad Jafar Jalali, Amir F. Atiya, Saeid Nahavandi, and Dipti Srinivasan, "Spinalnet: Deep neural network with gradual input," IEEE Transactions on Artificial Intelligence, Vol. 4, No. 5, 1165-1177, 2023.

10. Pham, Hieu, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, Mingxing Tan, and Quoc V. Le, "Combined scaling for zero-shot transfer learning," Neurocomputing, Vol. 555, 126658, 2023.

11. Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean, "Distilling the knowledge in a neural network," ArXiv Preprint ArXiv:1503.02531, 2015.

12. Bao, Hangbo, Li Dong, Songhao Piao, and Furu Wei, "Beit: Bert pre-training of image transformers," ArXiv Preprint ArXiv:2106.08254, 2021.

13. Ilharco, Gabriel, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, et al. "OpenCLIP,", 2021.

14. Fang, Yuxin, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao, "Eva: Exploring the limits of masked visual representation learning at scale," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 19358-19369, 2023.

15. Lannelongue, Loïc, Jason Grealey, and Michael Inouye, "Green algorithms: Quantifying the carbon footprint of computation," Advanced science, Vol. 8, No. 12, 2100707, 2021.
doi:10.1002/advs.202100707

16. Tamburrini, Guglielmo, "The AI carbon footprint and responsibilities of AI scientists," Philosophies, Vol. 7, No. 1, 4, 2022.

17. Zhu, Shaolong, Jinyu Zhang, Maoni Chao, Xinjuan Xu, Puwen Song, Jinlong Zhang, and Zhongwen Huang, "A rapid and highly efficient method for the identification of soybean seed varieties: Hyperspectral images combined with transfer learning," Molecules, Vol. 25, No. 1, 152, 2020.

18. Zhong, Chongxiao, Junping Zhang, Sifan Wu, and Ye Zhang, "Cross-scene deep transfer learning with spectral feature adaptation for hyperspectral image classification," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 13, 2861-2873, 2020.

19. Yang, Bin, Shunshi Hu, Qiandong Guo, and Danfeng Hong, "Multisource domain transfer learning based on spectral projections for hyperspectral image classification," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 15, 3730-3739, 2022.

20. Khaki, Saeed, Hieu Pham, and Lizhi Wang, "Simultaneous corn and soybean yield prediction from remote sensing data using deep transfer learning," Scientific Reports, Vol. 11, No. 1, 11132, 2021.

21. Zhang, Haokui, Ying Li, Yenan Jiang, Peng Wang, Qiang Shen, and Chunhua Shen, "Hyperspectral classification based on lightweight 3-D-CNN with transfer learning," IEEE Transactions on Geoscience and Remote Sensing, Vol. 57, No. 8, 5813-5828, 2019.

22. Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, "Imagenet classification with deep convolutional neural networks," Advances in Neural Information Processing Systems, Vol. 25, 2012.

23. Simonyan, Karen and Andrew Zisserman, "Very deep convolutional networks for large-scale image recognition," ArXiv Preprint ArXiv:1409.1556, 2014.

24. He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, "Deep residual learning for image recognition," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770-778, 2016.

25. Tan, Mingxing and Quoc Le, "Efficientnet: Rethinking model scaling for convolutional neural networks," Proceedings of the 36th International Conference on Machine Learning, PMLR, 6105-6114, 2019.

26. Liu, Z., H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, "A convnet for the 2020s," Proc. IEEECVF Conf. Comput. Vis. Pattern Recognit. CVPR, 2022.

27. Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, "An image is worth 16x16 words: Transformers for image recognition at scale," ArXiv Preprint ArXiv:2010.11929, 2020.

28. Liu, Ze, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 10012-10022, 2021.

29. Tu, Z., H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Maxvit Li, "Multi-axis vision transformer," Proceedings of the Computer Vision --- ECCV 2022: 17th European Conference, 459-479, 2022.

30. Fang, Yuxin, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao, "Eva-02: A visual representation for neon genesis," ArXiv Preprint ArXiv:2303.11331, 2023.

31. Chang, Chein-I., Hyperspectral Imaging: Techniques for Spectral Detection and Classification, Vol. 1, Springer Science & Business Media, 2003.

32. Grahn, Hans and Paul Geladi, Techniques and Applications of Hyperspectral Image Analysis, John Wiley & Sons, 2007.
doi:10.1002/9780470010884

33. Hagen, Nathan A. and Michael W. Kudenov, "Review of snapshot spectral imaging technologies," Optical Engineering, Vol. 52, No. 9, 090901, 2013.

34. Sumbul, Gencer, Marcela Charfuelan, Begüm Demir, and Volker Markl, "Bigearthnet: A large-scale benchmark archive for remote sensing image understanding," IGARSS 2019 --- 2019 IEEE International Geoscience and Remote Sensing Symposium, 5901-5904, 2019.

35. Lu, Guolan and Baowei Fei, "Medical hyperspectral imaging: A review," Journal of Biomedical Optics, Vol. 19, No. 1, 010901, 2014.

36. Zhang, Qing, Qingli Li, Guanzhen Yu, Li Sun, Mei Zhou, and Junhao Chu, "A multidimensional choledoch database and benchmarks for cholangiocarcinoma diagnosis," IEEE Access, Vol. 7, 149414-149421, 2019.

37. Wang, Qiang, Susan Fernandes, Gareth O. S. Williams, Neil Finlayson, Ahsan R. Akram, Kevin Dhaliwal, James R. Hopgood, and Marta Vallejo, "Deep learning-assisted co-registration of full-spectral autofluorescence lifetime microscopic images with H&E-stained histology images," Communications Biology, Vol. 5, No. 1, 1119, 2022.

38. Jiao, Changwei, Zijian Lin, Yinghe Xu, and Sailing He, "Noninvasive Raman imaging for monitoring mitochondrial redox state in septic rats," Progress In Electromagnetics Research, Vol. 175, 149-157, 2022.
doi:10.2528/PIER22101504

39. McLean, Ian S., Beating the Atmosphere, 39-75, Electronic Imaging in Astronomy Springer Praxis Books (Berlin, Heidelberg: Springer), 2008.

40. Majaess, D., "Discovering protostars and their host clusters via WISE," Astrophysics and Space Science, Vol. 344, 175-186, 2013.

41. Lacar, F. M., M. M. Lewis, and I. T. Grierson, "Use of hyperspectral imagery for mapping grape varieties in the Barossa Valley, South Australia," IGARSS 2001. Scanning the Present and Resolving the Future. Proceedings. IEEE 2001 International Geoscience and Remote Sensing Symposium (Cat. No.01CH37217), 2875-2877, 2001.

42. Vermeulen, Ph., P. Flémal, O. Pigeon, P. Dardenne, J. A. Fernández Pierna, and V. Baeten, "Assessment of pesticide coating on cereal seeds by near infrared hyperspectral imaging," Journal of Spectral Imaging, Vol. 6, 2017.

43. Zea, Maria, Augusto Souza, Yang Yang, Linda Lee, Krishna Nemali, and Lori Hoagland, "Leveraging high-throughput hyperspectral imaging technology to detect cadmium stress in two leafy green crops and accelerate soil remediation efforts," Environmental Pollution, Vol. 292, 118405, 2022.

44. Zhou, Zehao, Song Qiu, Yan Wang, Mei Zhou, Xinyuan Chen, Menghan Hu, Qingli Li, and Yue Lu, "Swin-spectral transformer for cholangiocarcinoma hyperspectral image segmentation," 2021 14th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), 1-6, 2021.

45. Zhu, He, Jing Luo, Jiaqi Liao, and Sailing He, "High-accuracy rapid identification and classification of mixed bacteria using hyperspectral transmission microscopic imaging and machine learning," Progress In Electromagnetics Research, Vol. 178, 49-62, 2023.

46. Xu, Zhanpeng, Yiming Jiang, Jiali Ji, Erik Forsberg, Yuanpeng Li, and Sailing He, "Classification, identification, and growth stage estimation of microalgae based on transmission hyperspectral microscopic imaging and machine learning," Optics Express, Vol. 28, No. 21, 30686-30700, 2020.

47. Wang, Meiling, Yongchang Xu, Zhisheng Wang, and Changda Xing, "Deep margin cosine autoencoder based medical hyperspectral image classification for tumor diagnosis," IEEE Transactions on Instrumentation and Measurement, Vol. 72, 2023.

48. Shan, Xinxin, Tai Ma, Anqi Gu, Haibin Cai, and Ying Wen, "TCRNet: Make transformer, CNN and RNN complement each other," ICASSP 2022 --- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1441-1445, Singapore, Singapore, 2022.

49. Dai, Kaijie, Zehao Zhou, Song Qiu, Yan Wang, Mei Zhou, Mingshuai Li, and Qingli Li, "A generative data augmentation trained by low-quality annotations for cholangiocarcinoma hyperspectral image segmentation," 2023 International Joint Conference on Neural Networks (IJCNN), 01-09, 2023.

50. Zhan, Gan, Yutaro Uwamoto, and Yen-Wei Chen, "Hyperunet for medical hyperspectral image segmentation on a choledochal database," 2022 IEEE International Conference on Consumer Electronics (ICCE), 1-5, IEEE, Las Vegas, NV, USA, 2022.

51. Yun, Sangdoo, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo, "CutMix: Regularization strategy to train strong classifiers with localizable features," Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 6023-6032, 2019.

52. Zhang, Hongyi, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz, "Mixup: Beyond empirical risk minimization," ArXiv Preprint ArXiv:1710.09412, 2017.

53. Cubuk, Ekin D., Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le, "Autoaugment: Learning augmentation strategies from data," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 113-123, 2019.

54. Cubuk, Ekin D., Barret Zoph, Jonathon Shlens, and Quoc V. Le, "Randaugment: Practical automated data augmentation with a reduced search space," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 702-703, 2020.

55. Zhong, Zhun, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang, "Random erasing data augmentation," Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, No. 07, 13001-13008, 2020.

56. Falkner, Stefan, Aaron Klein, and Frank Hutter, "BOHB: Robust and efficient hyperparameter optimization at scale," Proceedings of the 35th International Conference on Machine Learning, PMLR, 1437-1446, 2018.

57. "Anon algorithms for hyper-parameter optimization," Proceedings of the 24th International Conference on Neural Information Processing Systems.

58. Selvaraju, Ramprasaath R., Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra, "Grad-CAM: Visual explanations from deep networks via gradient-based localization," Proceedings of the IEEE International Conference on Computer Vision (ICCV), 618-626, 2017.

Dr. Lei Wang

Prof. Sailing He