Skip to content
CaMLSysUniversity of Cambridge
Home People Publications Projects Contact

← All publications

IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition

  • Hyeokhyen KwonGeorgia Institute of Technology
  • Catherine TongUniversity of Oxford
  • Harish HaresamudramGeorgia Institute of Technology
  • Yan GaoUniversity of Oxford
  • Gregory D. AbowdGeorgia Institute of Technology
  • Nicholas D. LaneUniversity of Oxford
  • Thomas PlötzGeorgia Institute of Technology

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 2020

Abstract

The lack of large-scale, labeled data sets impedes progress in developing robust and generalized predictive models for on-body sensor-based human activity recognition (HAR). Labeled data in human activity recognition is scarce and hard to come by, as sensor data collection is expensive, and the annotation is time-consuming and error-prone. To address this problem, we introduce IMUTube, an automated processing pipeline that integrates existing computer vision and signal processing techniques to convert videos of human activity into virtual streams of IMU data. These virtual IMU streams represent accelerometry at a wide variety of locations on the human body. We show how the virtually-generated IMU data improves the performance of a variety of models on known HAR datasets. Our initial results are very promising, but the greater promise of this work lies in a collective approach by the computer vision, signal processing, and activity recognition communities to extend this work in ways that we outline. This should lead to on-body, sensor-based HAR becoming yet another success story in large-dataset breakthroughs in recognition.

Materials

  • PDF
BibTeX
@article{kwon2020imutube,
  title = {{IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition}},
  author = {Kwon, Hyeokhyen and Tong, Catherine and Haresamudram, Harish and Gao, Yan and Abowd, Gregory D. and Lane, Nicholas D. and Plötz, Thomas},
  year = {2020},
  journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)},
  volume = {4},
  number = {3},
  pages = {1--29},
  publisher = {Association for Computing Machinery (ACM)},
  doi = {10.1145/3411841},
  url = {https://doi.org/10.1145/3411841}
}
Cambridge Machine Learning Systems Group
Department of Computer Science and Technology
University of Cambridge
ContactGitHub