Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Multidimensional vision sensors for information processing

Nature Nanotechnology (2024)Cite this article

Subjects

Abstract

The visual scene in the physical world integrates multidimensional information (spatial, temporal, polarization, spectrum and so on) and typically shows unstructured characteristics. Conventional image sensors cannot process this multidimensional vision data, creating a need for vision sensors that can efficiently extract features from substantial multidimensional vision data. Vision sensors are able to transform the unstructured visual scene into featured information without relying on sophisticated algorithms and complex hardware. The response characteristics of sensors can be abstracted into operators with specific functionalities, allowing for the efficient processing of perceptual information. In this Review, we delve into the hardware implementation of multidimensional vision sensors, exploring their working mechanisms and design principles. We exemplify multidimensional vision sensors built on emerging devices and silicon-based system integration. We further provide benchmarking metrics for multidimensional vision sensors and conclude with the principle of device–system co-design and co-optimization.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Evolution from image sensors to vision sensors.
Fig. 2: Multidimensional vision sensors using emerging device technology.
Fig. 3: Multidimensional vision sensors using system integration.
Fig. 4: Integration technology of multidimensional vision sensors.
Fig. 5: Device and system performance metrics for vision sensors.
Fig. 6: Device–system co-design and co-optimization.

Similar content being viewed by others

References

  1. He, C., Shen, Y. & Forbes, A. Towards higher-dimensional structured light. Light. Sci. Appl. 11, 205 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Oike, Y. Expanding human potential through imaging and sensing technologies. In 2022 International Electron Devices Meeting (IEDM) 1.2.1–1.2.5 (IEEE, 2022). This is a paper on the imaging and sensing technologies in different dimensions.

  3. Zhou, F. & Chai, Y. Near-sensor and in-sensor computing. Nat. Electron. 3, 664–671 (2020).

    Article  Google Scholar 

  4. Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).

    Article  CAS  PubMed  Google Scholar 

  5. Wan, T. et al. In-sensor computing: materials, devices, and integration technologies. Adv. Mater. 35, 2203830 (2023).

    Article  CAS  Google Scholar 

  6. Chai, Y. In-sensor computing for machine vision. Nature 579, 32–33 (2020). This is the first paper that defines ‘in-sensor computing’.

    Article  CAS  PubMed  Google Scholar 

  7. Gollisch, T. & Meister, M. Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron 65, 150–164 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Dunn, F. A., Lankheet, M. J. & Rieke, F. Light adaptation in cone vision involves switching between receptor and post-receptor sites. Nature 449, 603–606 (2007).

    Article  CAS  PubMed  Google Scholar 

  9. Stockman, A., MacLeod, D. I. A. & Johnson, N. E. Spectral sensitivities of the human cones. J. Opt. Soc. Am. A 10, 2491–2521 (1993).

    Article  CAS  Google Scholar 

  10. Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020). This is a paper on ultrafast machine vision.

    Article  CAS  PubMed  Google Scholar 

  11. Liao, F. et al. Bioinspired in-sensor visual adaptation for accurate perception. Nat. Electron. 5, 84–91 (2022). This paper reports a vision sensor with adaptation functions.

    Article  Google Scholar 

  12. Zhang, Z. et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol. 17, 27–32 (2022).

    Article  PubMed  Google Scholar 

  13. Tan, H. & van Dijken, S. Dynamic machine vision with retinomorphic photomemristor-reservoir computing. Nat. Commun. 14, 2169 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Vijjapu, M. T. et al. A flexible capacitive photoreceptor for the biomimetic retina. Light. Sci. Appl. 11, 3 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Zhu, Q. B. et al. A flexible ultrasensitive optoelectronic sensor array for neuromorphic vision systems. Nat. Commun. 12, 1798 (2021). This paper reports a vision sensor with high sensitivity.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. Jayachandran, D. et al. Insect-inspired, spike-based, in-sensor, and night-time collision detector based on atomically thin and light-sensitive memtransistors. ACS Nano 17, 1068–1080 (2022).

    Article  Google Scholar 

  17. Jayachandran, D. et al. A low-power biomimetic collision detector based on an in-memory molybdenum disulfide photodetector. Nat. Electron. 3, 646–655 (2020).

    Article  Google Scholar 

  18. Long, Z. et al. A neuromorphic bionic eye with filter-free color vision using hemispherical perovskite nanowire array retina. Nat. Commun. 14, 1972 (2023). This is a paper on vision sensors that preprocess information in the spectral dimension.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Liu, Y. et al. Perovskite-based color camera inspired by human visual cells. Light. Sci. Appl. 12, 43 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Du, J. et al. A robust neuromorphic vision sensor with optical control of ferroelectric switching. Nano Energy 89, 106439 (2021).

    Article  CAS  Google Scholar 

  21. Jiang, T. et al. Tetrachromatic vision-inspired neuromorphic sensors with ultraweak ultraviolet detection. Nat. Commun. 14, 2281 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. KIM, M. et al. Cuttlefish eye–inspired artificial vision for high-quality imaging under uneven illumination conditions. Sci. Robot. 8, eade4698 (2023).

    Article  PubMed  Google Scholar 

  23. Chen, J. et al. Optoelectronic graded neurons for bioinspired in-sensor motion perception. Nat. Nanotechnol. 18, 882–888 (2023). This is a paper on vision sensors that preprocess information in the spatiotemporal dimension.

    Article  CAS  PubMed  Google Scholar 

  24. Chang, J., Sitzmann, V., Dun, X., Heidrich, W. & Wetzstein, G. Hybrid optical–electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8, 12324 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  25. Chen, Y. et al. All-analog photoelectronic chip for high-speed vision tasks. Nature 623, 48–57 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. Wang, Y. et al. A three-dimensional neuromorphic photosensor array for nonvolatile in-sensor computing. Nano Lett. 23, 4524–4532 (2023).

    Article  CAS  PubMed  Google Scholar 

  27. Wang, C.-Y. et al. Gate-tunable van der Waals heterostructure for reconfigurable neural network vision sensor. Sci. Adv. 6, eaba6173 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  28. He, Z. et al. An organic transistor with light intensity-dependent active photoadaptation. Nat. Electron. 4, 522–529 (2021).

    Article  CAS  Google Scholar 

  29. Seung, H. et al. Integration of synaptic phototransistors and quantum dot light-emitting diodes for visualization and recognition of UV patterns. Sci. Adv. 8, eabq3101 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Qin, S. et al. A light-stimulated synaptic device based on graphene hybrid phototransistor. 2D Mater. 4, 035022 (2017).

    Article  Google Scholar 

  31. Sun, L. et al. In-sensor reservoir computing for language learning via two-dimensional memristors. Sci. Adv. 7, 20 (2021).

    Article  Google Scholar 

  32. Arbabi, A. & Faraon, A. Advances in optical metalenses. Nat. Photon. 17, 16–25 (2022). This is a review paper on the applications of metalenses for imaging.

    Article  Google Scholar 

  33. Arbabi, E., Kamali, S. M., Arbabi, A. & Faraon, A. Full-Stokes imaging polarimetry using dielectric metasurfaces. ACS Photon. 5, 3132–3140 (2018).

    Article  CAS  Google Scholar 

  34. Yang, Z. et al. Generalized Hartmann–Shack array of dielectric metalens sub-arrays for polarimetric beam profiling. Nat. Commun. 9, 4607 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  35. Yang, J. et al. Ultraspectral imaging based on metasurfaces with freeform shaped meta-atoms. Laser Photon. Rev. 16, 2100663 (2022).

    Article  Google Scholar 

  36. Wang, R. et al. Compact multi-foci metalens spectrometer. Light. Sci. Appl. 12, 103 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Kwon, H., Arbabi, E., Kamali, S. M., Faraji-Dana, M. & Faraon, A. Computational complex optical field imaging using a designed metasurface diffuser. Optica 5, 924–931 (2018).

    Article  Google Scholar 

  38. Saidaminov, M. I. et al. Perovskite photodetectors operating in both narrowband and broadband regimes. Adv. Mater. 28, 8144–8149 (2016).

    Article  CAS  PubMed  Google Scholar 

  39. Wang, J. et al. Self-driven perovskite narrowband photodetectors with tunable spectral responses. Adv. Mater. 33, 2005557 (2021).

    Article  CAS  Google Scholar 

  40. Cao, F. et al. Bionic detectors based on low-bandgap inorganic perovskite for selective NIR-I photon detection and imaging. Adv. Mater. 32, e1905362 (2020).

    Article  PubMed  Google Scholar 

  41. Rao, H. S., Li, W. G., Chen, B. X., Kuang, D. B. & Su, C. Y. In situ growth of 120 cm2 CH3 NH3PbBr3 perovskite crystal film on FTO glass for narrowband-photodetectors. Adv. Mater. 29, 1602639 (2017).

    Article  Google Scholar 

  42. Liu, Q. et al. Electron-donating amine-interlayer induced n-type doping of polymer:nonfullerene blends for efficient narrowband near-infrared photo-detection. Nat. Commun. 13, 5194 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Kublitski, J. et al. Enhancing sub-bandgap external quantum efficiency by photomultiplication for narrowband organic near-infrared photodetectors. Nat. Commun. 12, 4259 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  44. Armin, A., Jansen-van Vuuren, R. D., Kopidakis, N., Burn, P. L. & Meredith, P. Narrowband light detection via internal quantum efficiency manipulation of organic photodiodes. Nat. Commun. 6, 6343 (2015).

    Article  CAS  PubMed  Google Scholar 

  45. Siegmund, B. et al. Organic narrowband near-infrared photodetectors based on intermolecular charge-transfer absorption. Nat. Commun. 8, 15421 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. Schraml, S., Belbachir, A. N., Milosevic, N. & Schön, P. Dynamic stereo vision system for real-time tracking. In Proc. 2010 IEEE International Symposium on Circuits and Systems (IEEE, 2010).

  47. Finateu, T. et al. A 1,280 × 720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86 µm pixels, 1.066 GEPS readout, programmable event-rate controller and compressive data-formatting pipeline. In 2020 IEEE International Solid-State Circuits Conference - (ISSCC) (IEEE, 2020).

  48. Maruyama, Y. et al. 3.2-MP back-illuminated polarization image sensor with four-directional air-gap wire grid and 2.5-um pixels. IEEE Trans. Electron Devices 65, 2544–2551 (2018).

    Article  CAS  Google Scholar 

  49. Lefebvre, M., Moreau, L., Dekimpe, R. & Bol, D. A 0.2-to-3.6TOPS/W programmable convolutional imager SoC with in-sensor current-domain ternary-weighted MAC operations for feature extraction and region-of-interest detection. In 2021 IEEE International Solid- State Circuits Conference (ISSCC) 118–120 (IEEE, 2021).

  50. Tsai, T.-H. et al. A cascaded PLL (LC-PLL + RO-PLL) with a programmable double realignment achieving 204 fs integrated jitter (100 kHz to 100 MHz) and –72 dB reference spur. In 2022 IEEE International Solid- State Circuits Conference (ISSCC) 1–3 (IEEE, 2022).

  51. Oike, Y. Evolution of image sensor architectures with stacked device technologies. IEEE Trans. Electron Devices 69, 2757–2765 (2022). This is a paper on 3D integration technology for image sensors.

    Article  CAS  Google Scholar 

  52. Haruta, T. et al. A 1/2.3 inch 20 Mpixel 3-layer stacked CMOS Image Sensor with DRAM. In 2017 IEEE International Solid-State Circuits Conference (ISSCC) (IEEE, 2017).

  53. Guo, M. et al. A three-wafer-stacked hybrid 15-MPixel CIS + 1-MPixel EVS With 4.6-GEvent/s readout, in-pixel TDC, and on-chip ISP and ESP Function. IEEE J. Solid State Circuits 58, 2955–2964 (2023).

    Article  Google Scholar 

  54. Mahajan, R. & Sankman, B. 3D Microelectronic Packaging: From Fundamentals to Applications (Springer, 2017).

  55. Sakakibara, M. et al. A back-illuminated global-shutter CMOS image sensor with pixel-parallel 14b subthreshold ADC. In 2018 IEEE International Solid-State Circuits Conference - (ISSCC) (IEEE, 2018).

  56. Tan, Y. et al. A bioinspired retinomorphic device for spontaneous chromatic adaptation. Adv. Mater. 34, 2206816 (2022).

    Article  CAS  Google Scholar 

  57. Bhatnagar, P., Hong, J., Patel, M. & Kim, J. Transparent photovoltaic skin for artificial thermoreceptor and nociceptor memory. Nano Energy 91, 106676 (2022).

    Article  CAS  Google Scholar 

  58. Eki, R. et al. A 1/2.3 inch 12.3 Mpixel with on-chip 4.97TOPS/W CNN processor back-illuminated stacked CMOS image sensor. In 2021 IEEE International Solid- State Circuits Conference (ISSCC) 154–156 (IEEE, 2021).

  59. Yuan, S., Naveh, D., Watanabe, K., Taniguchi, T. & Xia, F. A wavelength-scale black phosphorus spectrometer. Nat. Photon 15, 601–607 (2021).

    Article  CAS  Google Scholar 

  60. Ma, C. et al. Intelligent infrared sensing enabled by tunable moire quantum geometry. Nature 604, 266–272 (2022).

    Article  CAS  PubMed  Google Scholar 

  61. Song, J. K. et al. Stretchable colour-sensitive quantum dot nanocomposites for shape-tunable multiplexed phototransistor arrays. Nat. Nanotechnol. 17, 849–856 (2022).

    Article  CAS  PubMed  Google Scholar 

  62. Qin, T. et al. Mercury telluride colloidal quantum-dot focal plane array with planar p–n junctions enabled by in situ electric field-activated doping. Sci. Adv. 9, eadg7827 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  63. Meng, Y. et al. Ultrafast multilevel optical tuning with CSb2Te3 thin films. Adv. Opt. Mater. 6, 1800360 (2018).

    Article  Google Scholar 

  64. Bai, W. et al. Near-infrared tunable metalens based on phase change material Ge2Se2Te5. Sci. Rep. 9, 5368 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  65. Luo, Z. D. et al. Artificial optoelectronic synapses based on ferroelectric field-effect enabled 2D transition metal dichalcogenide memristive transistors. ACS Nano 14, 746–754 (2020).

    Article  CAS  PubMed  Google Scholar 

  66. Chen, J.-Y., He, L., Wang, J.-P. & Li, M. All-optical switching of magnetic tunnel junctions with single subpicosecond laser pulses. Phys. Rev. Appl. 7, 021001 (2017).

    Article  Google Scholar 

  67. Wang, L. et al. Femtosecond laser-assisted switching in perpendicular magnetic tunnel junctions with double-interface free layer. Sci. China Inf. Sci. 65, 142403 (2021).

    Article  Google Scholar 

  68. Ciuciulkaite, A. et al. Magnetic and all-optical switching properties of amorphous TbxCo100−x alloys. Phys. Rev. Mater. 4, 104418 (2020).

    Article  CAS  Google Scholar 

  69. Stupakiewicz, A., Szerenos, K., Afanasiev, D., Kirilyuk, A. & Kimel, A. V. Ultrafast nonthermal photo-magnetic recording in a transparent medium. Nature 542, 71–74 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  70. Zheng, H. et al. Meta-optic accelerators for object classifiers. Sci. Adv. 8, 30 (2022).

    Article  Google Scholar 

  71. Shao, B. et al. Crypto primitive of MOCVD MoS2 transistors for highly secured physical unclonable functions. Nano Res. 14, 1784–1788 (2020).

    Article  Google Scholar 

  72. Brunet, L. et al. First demonstration of a CMOS over CMOS 3D VLSI CoolCube™ integration on 300mm wafers. In 2016 IEEE Symposium on VLSI Technology (IEEE, 2016).

  73. Seo, S. et al. Artificial optic-neural synapse for colored and color-mixed pattern recognition. Nat. Commun. 9, 5106 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  74. Huo, Q. et al. A computing-in-memory macro based on three-dimensional resistive random-access memory. Nat. Electron. 5, 469–477 (2022).

    Article  Google Scholar 

  75. Zhu, C. et al. Optical synaptic devices with ultra-low power consumption for neuromorphic computing. Light. Sci. Appl. 11, 337 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  76. Rao, M. et al. Thousands of conductance levels in memristors integrated on CMOS. Nature 615, 823–829 (2023). This is a paper on electronic computing by memristors with 2,084 conductance states.

    Article  CAS  PubMed  Google Scholar 

  77. Posch, C., Matolin, D. & Wohlgenannt, R. A QVGA 143dB dynamic range asynchronous address-event PWM dynamic image sensor with lossless pixel-level video compression. In 2010 IEEE International Solid-State Circuits Conference - (ISSCC) (IEEE, 2010).

  78. Guo, Q. et al. Compact single-shot metalens depth sensors inspired by eyes of jumping spiders. Proc. Natl Acad. Sci. USA 116, 22959–22965 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This work is supported by MOST National Key Technologies R&D Programme (SQ2022YFA1200118-04), Research Grant Council of Hong Kong (CRS_PolyU502/22), Shenzhen Science and Technology Innovation Commission (SGDX2020110309540000), Innovation Technology Fund (ITS/047/20), The Hong Kong Polytechnic University (1-ZE1T and WZ4X), and The Hong Kong Polytechnic University Shenzhen Research Institute (I2022A013).

Author information

Author notes

  1. These authors contributed equally: Zhaoqing Wang, Tianqing Wan.

Authors and Affiliations

  1. Department of Applied Physics, The Hong Kong Polytechnic University, Kowloon, Hong Kong, China

    Zhaoqing Wang, Tianqing Wan, Sijie Ma & Yang Chai

  2. Joint Research Centre of Microelectronics, The Hong Kong Polytechnic University, Kowloon, Hong Kong, China

    Zhaoqing Wang, Tianqing Wan, Sijie Ma & Yang Chai

Authors
  1. Zhaoqing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tianqing Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sijie Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yang Chai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang Chai.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Nanotechnology thanks Saptarshi Das, Sung Kyu Park and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Table 1.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Z., Wan, T., Ma, S. et al. Multidimensional vision sensors for information processing. Nat. Nanotechnol. (2024). https://doi.org/10.1038/s41565-024-01665-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41565-024-01665-7

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing