volume | PIER Journals

Abstract

Due to the optical diffraction limit, the resolution of a wide-field (WF) microscope cannot easily go below a few hundred nanometers. Super-resolution microscopy has the disadvantages of high cost, complex optical equipment, and high experimental environment requirements. Deep-learning-based super-resolution (DLSR) has the advantages of simple operation and low cost, and has attracted much attention recently. Here we propose a novel DLSR model named Modified Hybrid Task Cascade U-Net (MHTCUN) for image super-resolution in optical microscopy using the public biological image dataset BioSR. The MHTCUN has three stages, and we introduce a novel module named Feature Refinement Module (FRM) to extract deeper features in each stage. In each FRM, a U-Net is introduced to refine the features, and the Fourier Channel Attention Block (FCAB) is introduced in the U-Net to learn the high-level representation of the high-frequency information of different feature maps. Compared with six state-of-the-art DLSR models used for single-image super-resolution (SISR), our MHTCUN achieves the highest signal-to-noise ratio (PSNR) of 26.87 and structural similarity (SSIM) of 0.746, demonstrating that our MHTCUN has achieved the state-of-the-art in DLSR. Compared with the DLSR model DFCAN used for image super-resolution in optical microscopy specially, MHTCUN has a significant improvement in PSNR and a slight improvement in SSIM on BioSR. Finally, we fine-tune the trained MHTCUN on the other biological images. MHTCUN also shows good performance on denoising, contrast enhancement, and resolution enhancement.

1. Zhao, Z.-Q., P. Zheng, S.-T. Xu, and X. Wu, "Object detection with deep learning: A review," IEEE Transactions on Neural Networks and Learning Systems, Vol. 30, No. 11, 3212-3232, 2019.
doi:10.1109/TNNLS.2018.2876865

2. Trajanovski, S., C. Shan, P. J. C. Weijtmans, S. G. B. de Koning, and T. J. M. Ruers, "Tongue tumor detection in hyperspectral images using deep learning semantic segmentation," IEEE Transactions on Biomedical Engineering, Vol. 68, No. 4, 1330-1340, 2020.
doi:10.1109/TBME.2020.3026683

3. Zhao, S., D. M. Zhang, and H. W. Huang, "Deep learning-based image instance segmentation for moisture marks of shield tunnel lining," Tunnelling and Underground Space Technology, Vol. 95, 103156, 2020.
doi:10.1016/j.tust.2019.103156

4. Yang, W., X. Zhang, Y. Tian, W. Wang, J.-H. Xue, and Q. Liao, "Deep learning for single image super-resolution: A brief review," IEEE Transactions on Multimedia, Vol. 21, No. 12, 3106-3121, 2019.
doi:10.1109/TMM.2019.2919431

5. Ledig, C., L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z.Wang, et al. "Photo-realistic single image super-resolution using a generative adversarial network," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4681-4690, 2017.

6. Kim, K. I. and Y. Kwon, "Single-image super-resolution using sparse regression and natural image prior," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No. 6, 1127-1133, 2010.
doi:10.1109/TPAMI.2010.25

7. Kirkland, E. J., "Bilinear interpolation," Advanced Computing in Electron Microscopy, 261-263, Springer, 2010.
doi:10.1007/978-1-4419-6533-2_12

8. Liu, T., K. De Haan, Y. Rivenson, Z. Wei, X. Zeng, Y. Zhang, and A. Ozcan, "Deep learning-based super-resolution in coherent imaging systems," Scientic Reports, Vol. 9, No. 1, 1-13, 2019.

9. Dong, C., C. C. Loy, K. He, and X. Tang, "Image super-resolution using deep convolutional networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38, No. 2, 295-307, 2015.
doi:10.1109/TPAMI.2015.2439281

10. He, K., X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770-778, 2016.

11. Lim, B., S. Son, H. Kim, S. Nah, and K. M. Lee, "Enhanced deep residual networks for single image super-resolution," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 136-144, 2017.

12. Zhang, Y., K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, "Image superresolution using very deep residual channel attention networks," Proceedings of the European Conference on Computer Vision (ECCV), 286-301, 2018.

13. Huang, G., Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4700-4708, 2017.

14. Zhang, Y., Y. Tian, Y. Kong, B. Zhong, and Y. Fu, "Residual dense network for image super-resolution," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2472-2481, 2018.

15. Li, Z., J. Yang, Z. Liu, X. Yang, G. Jeon, and W. Wu, "Feedback network for image super-resolution," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3867-3876, 2019.

16. Ronneberger, O., P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," International Conference on Medical Image Computing and Computer-assisted Intervention, 234-241, Springer, 2015.

17. Chen, K., J. Pang, J. Wang, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Shi, W. Ouyang, et al. "Hybrid task cascade for instance segmentation," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4974-4983, 2019.

18. Cai, Z. and N. Vasconcelos, "Cascade r-cnn: Delving into high quality object detection," Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 6154-6162, 2018.

19. Hell, S. W. and J. Wichmann, "Breaking the diffraction resolution limit by stimulated emission: Stimulated-emission-depletion uorescence microscopy," Optics Letters, Vol. 19, No. 11, 780-782, 1994.

20. Hess, S. T., T. P. K. Girirajan, and M. D. Mason, "Ultra-high resolution imaging by fluorescence photoactivation localization microscopy," Biophysical Journal, Vol. 91, No. 11, 4258-4272, 2006.

21. Rust, M. J., M. Bates, and X. Zhuang, "Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (storm)," Nature Methods, Vol. 3, No. 10, 793-796, 2006.

22. Gustafsson, M. G. L., "Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy," Journal of Microscopy, Vol. 198, No. 2, 82-87, 2000.

23. Weigert, M., U. Schmidt, T. Boothe, A. Muller, A. Dibrov, A. Jain, B. Wilhelm, D. Schmidt, C. Broaddus, S. Culley, et al. "Content-aware image restoration: Pushing the limits of fluorescence microscopy," Nature Methods, Vol. 15, No. 12, 1090-1097, 2018.

24. Wang, H., Y. Rivenson, Y. Jin, Z. Wei, R. Gao, H. Gunaydin, L. A. Bentolila, C. Kural, and A. Ozcan, "Deep learning enables cross-modality superresolution in fluorescence microscopy," Nature Methods, Vol. 16, No. 1, 103-110, 2019.

25. Qiao, C., D. Li, Y. Guo, C. Liu, T. Jiang, Q. Dai, and D. Li, "Evaluation and development of deep neural networks for image super-resolution in optical microscopy," Nature Methods, Vol. 18, No. 2, 194-202, 2021.

26. Shi, W., J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1874-1883, 2016.

27. Howard, A. G., M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, "Mobilenets: Efficient convolutional neural networks for mobile vision applications,", arXiv preprint arXiv:1704.04861, 2017.

28. Ramachandran, P., B. Zoph, and Q. V. Le, "Searching for activation functions,", arXiv preprint arXiv:1710.05941, 2017.

29. Lin, T.-Y., P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2117-2125, 2017.

30. Allen, D. M., "Mean square error of prediction as a criterion for selecting variables," Technometrics, Vol. 13, No. 3, 469-475, 1971.

31. Wang, Z., A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity," IEEE Transactions on Image Processing, Vol. 13, No. 4, 600-612, 2004.

32. Szegedy, C., W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1-9, 2015.

33. Descloux, A., K. S. Gruβmayer, and A. Radenovic, "Parameter-free image resolution estimation based on decorrelation analysis," Nature Methods, Vol. 16, No. 9, 918-924, 2019.

34. Abramoff, M. D., P. J. Magalhaes, and S. J. Ram, "Image processing with imagej," Biophotonics International, Vol. 11, No. 7, 36-42, 2004.

Dr. Dawei Gong

Dr. Tengfei Ma

Dr. Julian Evans

Prof. Sailing He