Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Mar 4;22(5):2014.
doi: 10.3390/s22052014.

A Vision-Based System for In-Sleep Upper-Body and Head Pose Classification

Yan-Ying Li  1 , Shoue-Jen Wang  2 , Yi-Ping Hung  1
Affiliations

A Vision-Based System for In-Sleep Upper-Body and Head Pose Classification

Yan-Ying Li et al. Sensors (Basel). .

Abstract

Sleep quality is known to have a considerable impact on human health. Recent research shows that head and body pose play a vital role in affecting sleep quality. This paper presents a deep multi-task learning network to perform head and upper-body detection and pose classification during sleep. The proposed system has two major advantages: first, it detects and predicts upper-body pose and head pose simultaneously during sleep, and second, it is a contact-free home security camera-based monitoring system that can work on remote subjects, as it uses images captured by a home security camera. In addition, a synopsis of sleep postures is provided for analysis and diagnosis of sleep patterns. Experimental results show that our multi-task model achieves an average of 92.5% accuracy on challenging datasets, yields the best performance compared to the other methods, and obtains 91.7% accuracy on the real-life overnight sleep data. The proposed system can be applied reliably to extensive public sleep data with various covering conditions and is robust to real-life overnight sleep data.

Keywords: deep multi-task learning; head and upper-body detection; head and upper-body pose classification; sleep monitoring; sleep posture.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
Architecture of the proposed framework.
Figure 2
Figure 2
SleePose-FRCNN-Net architecture.
Figure 3
Figure 3
Sample images for each pose class from the SLP database [42].
Figure 4
Figure 4
Illustration of the system in the experiment [43].
Figure 5
Figure 5
Examples of images from the iSP pilot dataset [43].
Figure 6
Figure 6
The 10 types of sleep postures [43].
Figure 7
Figure 7
Examples of RGB/IR images from the iSP real-life dataset.
Figure 8
Figure 8
Confusion matrix for head pose classification on the SLP dataset.
Figure 9
Figure 9
Confusion matrix for upper-body pose classification on the SLP dataset.
Figure 10
Figure 10
Confusion matrix for head pose classification on the iSP pilot dataset.
Figure 11
Figure 11
Confusion matrix for upper-body pose classification on the iSP pilot dataset.
Figure 12
Figure 12
Repurposed SimpleBaseline [49] (first row) and OpenPose [50] (second row) performance on the iSP dataset.
Figure 13
Figure 13
Sleep posture detection on real-life data. The left column shows images in IR mode and the right column shows in RGB mode.
Figure 14
Figure 14
Pictorial representation of sleep poses and motion over time of subjects.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Huang T., Redline S. Cross-sectional and Prospective Associations of Actigraphy-Assessed Sleep Regularity with Metabolic Abnormalities: The Multi-Ethnic Study of Atherosclerosis. Diabetes Care. 2019;42:1422–1429. doi: 10.2337/dc19-0596. - DOI - PMC - PubMed
    1. Huang T., Mariani S., Redline S. Sleep Irregularity and Risk of Cardiovascular Events: The Multi-Ethnic Study of Atherosclerosis. J. Am. Coll. Cardiol. 2020;75:991–999. doi: 10.1016/j.jacc.2019.12.054. - DOI - PMC - PubMed
    1. Yu C.K.C. The effect of sleep position on dream experiences. Dreaming. 2012;22:212–221. doi: 10.1037/a0029255. - DOI
    1. Neilson J., Avital L., Willock J., Broad N. Using a national guideline to prevent and manage pressure ulcers. Nurs. Manag. 2014;21:18–21. doi: 10.7748/nm2014.04.21.2.18.s22. - DOI - PubMed
    1. Hoyer E.H., Friedman M., Lavezza A., Wagner-Kosmakos K., Lewis-Cherry R., Skolnik J.L., Byers S.P., Atanelov L., Colantuoni E., Brotman D.J., et al. Promoting mobility and reducing length of stay in hospitalized general medicine patients: A quality-improvement project. J. Hosp. Med. 2016;11:341–347. doi: 10.1002/jhm.2546. - DOI - PubMed