Skip to main content
Springer Nature Link
Log in

Sparse Deep LSTMs with Convolutional Attention for Human Action Recognition

  • Original Research
  • Published:
Save article
View saved research
SN Computer Science Aims and scope Submit manuscript

Abstract

Deep learning has recently gained remarkable results in action recognition. In this paper, an architecture is proposed for action recognition, including ResNet feature extractor, Conv-Attention-LSTM, BiLSTM, and fully connected layers. Furthermore, a sparse layer after each LSTM layer is added to overcome overfitting. In addition to RGB images, optical flow is also used to incorporate motion information into our architecture. Due to similarities of consecutive frames, video sequences are divided into equal parts. Frames of successive parts are used as consecutive frames to obtain the flow. Furthermore, to find the significant regions, the convolutional attention network is applied. The proposed method is evaluated using two popular datasets, UCF-101, and HMDB-51 and its accuracy for these datasets is 95.24 and 71.62, respectively. Overfitting is reduced due to using a sparse layer instead of a dropout based on the results achieved. Moreover, a deep LSTM network leads to a higher recognition rate than one-layer LSTM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

References

  1. Subetha T, Chitrakala S (2016). A survey on human activity recognition from videos. In: 2016 International Conference on Information Communication and Embedded Systems (ICICES) (pp 1–7). IEEE.

  2. Herath S, Harandi M, Porikli F. Going deeper into action recognition: a survey. Image Vis Comput. 2017;60:4–21.

    Article  Google Scholar 

  3. Saini R, Kumar P, Roy PP, Dogra DP. A novel framework of continuous human-activity recognition using Kinect. Neurocomputing. 2018;311:99–111.

    Article  Google Scholar 

  4. Saini R, Kumar P, Kaur B, Roy PP, Dogra DP, Santosh KC. Kinect sensor-based interaction monitoring system using the BLSTM neural network in healthcare. Int J Mach Learn Cybernet. 2019;10(9):2529–40.

    Article  Google Scholar 

  5. Kumar P, Mukherjee S, Saini R, Kaushik P, Roy PP, Dogra DP. Multimodal gait recognition with inertial sensor data and video using evolutionary algorithm. IEEE Trans Fuzzy Syst. 2018;27(5):956–65.

    Article  Google Scholar 

  6. Zhang HB, Zhang YX, Zhong B, Lei Q, Yang L, Du JX, Chen DS. A comprehensive survey of vision-based human action recognition methods. Sensors. 2019;19(5):1005.

    Article  Google Scholar 

  7. Deng L, Yu D (2014) Deep learning: methods and applications. Found Trends® Signal Process 7(3–4):197–387.

  8. Wu D, Sharma N, Blumenstein M (2017) Recent advances in video-based human action recognition using deep learning: a review. In: 2017 International Joint Conference on Neural Networks (IJCNN) (pp 2865–2872). IEEE.

  9. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE. 1998;86(11):2278–324.

    Article  Google Scholar 

  10. Yang H, Zhang J, Li S, Lei J, Chen S. Attend it again: Recurrent attention convolutional neural network for action recognition. Appl Sci. 2018;8(3):383.

    Article  Google Scholar 

  11. Bengio Y, Simard P, Frasconi P. Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw. 1994;5(2):157–66.

    Article  Google Scholar 

  12. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735–80.

    Article  Google Scholar 

  13. Graves A, Schmidhuber J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 2005;18(5–6):602–10.

    Article  Google Scholar 

  14. Piergiovanni AJ, Ryoo MS (2019) Representation flow for action recognition. In: Proceedings of the ieee conference on computer vision and pattern recognition (pp 9945–9953).

  15. Carreira J, Zisserman A (2017) Quo vadis, action recognition? a new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp 6299–6308).

  16. Sevilla-Lara L, Liao Y, Güney F, Jampani V, Geiger A, Black MJ (2018) On the integration of optical flow and action recognition. In: German conference on pattern recognition (pp 281–297). Springer, Cham.

  17. Li Z, Gavrilyuk K, Gavves E, Jain M, Snoek CG. Videolstm convolves, attends and flows for action recognition. Comput Vis Image Underst. 2018;166:41–50.

    Article  Google Scholar 

  18. Ji S, Xu W, Yang M, Yu K. 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell. 2012;35(1):221–31.

    Article  Google Scholar 

  19. Laptev I. On space-time interest points. Int J Comput Vision. 2005;64(2–3):107–23.

    Article  Google Scholar 

  20. Sicre R, Gevers T (2014) Dense sampling of features for image retrieval. In: 2014 IEEE International Conference on Image Processing (ICIP) (pp 3057–3061). IEEE.

  21. Dollár P, Rabaud V, Cottrell, G., & Belongie, S. (2005, October). Behavior recognition via sparse spatio-temporal features. In: 2005 IEEE international workshop on visual surveillance and performance evaluation of tracking and surveillance (pp 65–72). IEEE.

  22. Laptev I, Marszalek M, Schmid C, Rozenfeld, B. (2008) Learning realistic human actions from movies. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition (pp 1–8). IEEE.

  23. Klaser A, Marszałek M, Schmid C (2008) A spatio-temporal descriptor based on 3d-gradients.

  24. Zhu J, Qi J, Kong X (2012) An improved method of action recognition based on sparse spatio-temporal features. In: International conference on artificial intelligence: methodology, systems, and applications (pp 240–245). Springer, Berlin, Heidelberg.

  25. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, ..., Rabinovich A (2015). Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 1–9).

  26. Tieleman T, Hinton G (2017) Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning. Technical Report.

  27. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems (pp 91–99).

  28. Ohn-Bar E, Trivedi MM. Multi-scale volumes for deep object detection and localization. Pattern Recogn. 2017;61:557–72.

    Article  Google Scholar 

  29. Sharma S, Kiros R, Salakhutdinov R (2015) Action recognition using visual attention. arXiv preprint arXiv: 1511.04119.

  30. Wang L, Xiong Y, Wang Z, Qiao Y, Lin D, Tang X, & Van Gool L (2016) Temporal segment networks: Towards good practices for deep action recognition. In: European conference on computer vision (pp 20–36). Springer, Cham.

  31. Ijjina EP, Chalavadi KM. Human action recognition using genetic algorithms and convolutional neural networks. Pattern Recogn. 2016;59:199–212.

    Article  Google Scholar 

  32. Cheng C, Lv P, Su B (2018) Spatiotemporal pyramid pooling in 3d convolutional neural networks for action recognition. In: 2018 25th IEEE international conference on image processing (ICIP) (pp 3468–3472). IEEE.

  33. Tran D, Bourdev L, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE international conference on computer vision (pp 4489–4497).

  34. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. In: Advances in neural information processing systems (pp 568–576).

  35. Wang X, Gao L, Wang P, Sun X, Liu X. Two-stream 3-d convnet fusion for action recognition in videos with arbitrary size and length. IEEE Trans Multimedia. 2017;20(3):634–44.

    Article  Google Scholar 

  36. Yang H, Zhang J, Li S, Luo T. Bi-direction hierarchical LSTM with spatial-temporal attention for action recognition. J Intell Fuzzy Syst. 2019;36(1):775–86.

    Article  Google Scholar 

  37. Yang S, Yu X, Zhou Y (2020) LSTM and GRU Neural Network performance comparison study: taking yelp review dataset as an example. In: 2020 International workshop on electronic communication and artificial intelligence (IWECAI) (pp 98–101). IEEE.

  38. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.3555.

  39. Weiss G, Goldberg Y, Yahav E (2018) On the practical computational power of finite precision RNNs for language recognition. arXiv preprint arXiv: 1805.04908.

  40. Srivastava N, Mansimov E, Salakhudinov R (2015) Unsupervised learning of video representations using lstms. In: International conference on machine learning (pp 843–852).

  41. Ma CY, Chen MH, Kira Z, AlRegib G. TS-LSTM and temporal-inception: exploiting spatiotemporal dynamics for activity recognition. Signal Process. 2019;71:76–87.

    Google Scholar 

  42. Majd M, Safabakhsh R (2019) Correlational Convolutional LSTM for human action recognition. Neurocomputing.

  43. Ullah A, Ahmad J, Muhammad K, Sajjad M, Baik SW. Action recognition in video sequences using deep bi-directional LSTM with CNN features. IEEE Access. 2017;6:1155–66.

    Article  Google Scholar 

  44. Zang J, Wang L, Liu Z, Zhang Q, Hua G, Zheng N (2018) Attention-based temporal weighted convolutional neural network for action recognition. In: IFIP international conference on artificial intelligence applications and innovations (pp 97–108). Springer, Cham.

  45. Gammulle H, Denman S, Sridharan S, Fookes C (2017). Two stream lstm: A deep fusion framework for human action recognition. In: 2017 IEEE winter conference on applications of computer vision (WACV) (pp 177–186). IEEE.

  46. Hassibi B, Stork DG (1993) Second order derivatives for network pruning: Optimal brain surgeon. In: Advances in neural information processing systems (pp 164–171).

  47. LeCun Y, Denker JS, Solla SA (1990) Optimal brain damage. In: Advances in neural information processing systems (pp 598–605).

  48. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR (2012) Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 1207.0580.

  49. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 2014;15(1):1929–58.

    MathSciNet  MATH  Google Scholar 

  50. Wan L, Zeiler M, Zhang S, Le Cun Y, Fergus R (2013) Regularization of neural networks using dropconnect. In: International conference on machine learning (pp 1058–1066).

  51. Changpinyo S, Sandler M, Zhmoginov A (2017) The power of sparsity in convolutional neural networks. arXiv preprint arXiv: 1702.06257.

  52. Yuan M, Lin Y. Model selection and estimation in regression with grouped variables. J R Stat Soc. 2006;68(1):49–67.

    Article  MathSciNet  Google Scholar 

  53. Wen W, Wu C, Wang Y, Chen Y, Li H (2016) Learning structured sparsity in deep neural networks. In: Advances in neural information processing systems (pp 2074–2082).

  54. Murray K, Chiang D (2015) Auto-sizing neural networks: With applications to n-gram language models. arXiv preprint arXiv: 1508.05051.

  55. Srinivas S, Babu RV (2015) Data-free parameter pruning for deep neural networks. arXiv preprint arXiv: 1507.06149.

  56. Narang S, Elsen E, Diamos G, Sengupta S (2017) Exploring sparsity in recurrent neural networks. arXiv preprint arXiv: 1704.05119.

  57. Wen W, He Y, Rajbhandari S, Zhang M, Wang W, Liu F, ... Li H (2017). Learning intrinsic sparse structures within long short-term memory. arXiv preprint arXiv: 1709.05027.

  58. Zach C, Pock T, Bischof H (2007) A duality based approach for realtime TV-L 1 optical flow. In: Joint pattern recognition symposium (pp 214–223). Springer, Berlin, Heidelberg.

  59. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.1556.

  60. Kulkarni K, Evangelidis G, Cech J, Horaud R. Continuous action recognition based on sequence alignment. Int J Comput Vision. 2015;112(1):90–114.

    Article  Google Scholar 

  61. Soomro K, Zamir AR, Shah M (2012) UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.0402.

  62. Kuehne H, Jhuang H, Garrote E, Poggio T, Serre T (2011) HMDB: a large video database for human motion recognition. In: 2011 International conference on computer vision (pp 2556–2563). IEEE.

  63. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C., ... Ghemawat S (2016) Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv: 1603.04467.

  64. Wang H, Schmid C (2013) Action recognition with improved trajectories. In: Proceedings of the IEEE international conference on computer vision (pp 3551–3558).

  65. Dhar P, Alvarez JM, Roy PP (2017) Efficient framework for action recognition using reduced fisher vector encoding. In: Proceedings of international conference on computer vision and image processing (pp 343–354). Springer, Singapore.

  66. Schuldt C, Laptev I, Caputo B (2004) Recognizing human actions: a local SVM approach. In: Proceedings of the 17th international conference on pattern recognition, 2004. ICPR 2004. (Vol. 3, pp 32–36). IEEE.

  67. Blank M, Gorelick L, Shechtman E, Irani M, Basri R (2005) Actions as space-time shapes. In: Tenth IEEE international conference on computer vision (ICCV'05) Volume 1 (Vol. 2, pp. 1395–1402). IEEE.

  68. Yang H, Yuan C, Li B, Du Y, Xing J, Hu W, Maybank SJ. Asymmetric 3d convolutional neural networks for action recognition. Pattern Recogn. 2019;85:1–12.

    Article  Google Scholar 

  69. Sun L, Jia K, Chen K, Yeung DY, Shi BE, Savarese S (2017) Lattice long short-term memory for human action recognition. In: Proceedings of the IEEE International Conference on Computer Vision (pp 2147–2156).

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Engineering, Shahid Beheshti University, Tehran, Iran

    Atefe Aghaei, Ali Nazari & Mohsen Ebrahimi Moghaddam

Authors
  1. Atefe Aghaei
    View author publications

    Search author on:PubMed Google Scholar

  2. Ali Nazari
    View author publications

    Search author on:PubMed Google Scholar

  3. Mohsen Ebrahimi Moghaddam
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Mohsen Ebrahimi Moghaddam.

Ethics declarations

Conflict of interests

Atefe Aghaei declares that she has no conflict of interest. Ali Nazari declares that he has no conflict of interest. Mohsen Ebrahimi Moghaddam declares that he has no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aghaei, A., Nazari, A. & Moghaddam, M.E. Sparse Deep LSTMs with Convolutional Attention for Human Action Recognition. SN COMPUT. SCI. 2, 151 (2021). https://doi.org/10.1007/s42979-021-00576-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • DOI: https://doi.org/10.1007/s42979-021-00576-x

Keywords