Development of a Stroke Tele-Rehabilitation System Using Markerless Augmented Reality and a Hybrid AI-Based Clinical Decision Support Dashboard
Keywords:
Markerless Augmented Reality, Hybrid Artificial Intelligence, Stroke Rehabilitation, Clinical Decision Support Dashboard, Healthcare Serious GamesAbstract
This study aimed to develop a stroke tele-rehabilitation prototype integrating markerless augmented reality with a hybrid AI-based clinical decision support dashboard, in order to reduce the limitations of wearable-device-dependent rehabilitation and to support continuous home-based monitoring. The system employed MediaPipe for real-time tracking of 21 hand landmarks from a standard webcam and transmitted the extracted features to a Hybrid AI architecture that combines a rule-based engine with a large language model to generate feedback and clinical summaries under the Serious Game Design Assessment framework and the concept of neuroplasticity. The methodology consisted of engineering evaluation, simulated-data testing using 100 clinical cases, and expert validation by nine specialists. Evaluation in a simulated environment showed a frame drop rate of 5.36%, a spatial error of 10.51 pixels, a frame rate of 49.6–58.4 FPS, and a latency of 45.2–68.7 ms. For AI performance, the dashboard achieved 92.00% accuracy and 89.36% recall on 100 simulated clinical cases, while expert evaluation rated the system architecture and functional appropriateness at the highest level (mean = 4.59–4.75). The key contribution of this work lies in the integration of Markerless AR, Hybrid AI, and a clinical dashboard into a single prototype. However, the findings are limited to prototype evaluation in a simulated environment and expert-based validation; therefore, further clinical studies are required before real-world deployment.
References
[1] A. Moldoveanu et al., “Gamified virtual reality in post-stroke neurorehabilitation: A systematic review,” Frontiers in Virtual Reality, vol. 7, Art. no. 1737482, 2026, doi: 10.3389/frvir.2026.1737482.
[2] C. Lugaresi et al., “MediaPipe: A framework for building perception pipelines,” arXiv preprint arXiv:1906.08172, 2019, doi: 10.48550/arXiv.1906.08172.
[3] S. Rasetto et al., “ReHAb Playground: A DL-based framework for game-based hand rehabilitation,” Future Internet, vol. 17, no. 11, Art. no. 522, 2025, doi: 10.3390/fi17110522.
[4] F. Jiang et al., “Artificial intelligence in healthcare: Past, present and future,” Stroke and Vascular Neurology, vol. 2, no. 4, pp. 230-243, 2017, doi: 10.1136/svn-2017-000101.
[5] World Health Organization, International Classification of Functioning, Disability and Health (ICF). Geneva, Switzerland: World Health Organization, 2001.
[6] R. Zou et al., “MIND: Empowering mental health clinicians with multimodal data insights through a narrative dashboard,” in Proc. CHI Conf. Human Factors in Computing Systems (CHI), 2026, doi: 10.1145/3772318.3790529.
[7] J. A. Kleim and T. A. Jones, “Principles of experience-dependent neural plasticity: Implications for rehabilitation after brain damage,” Journal of Speech, Language, and Hearing Research, vol. 51, no. 1, pp. 225-239, 2008, doi: 10.1044/1092-4388(2008/018).
[8] C. M. Meira Jr. and M. Massa, “Genetic and environmental influences on perceptual-motor abilities,” Psychology, vol. 8, pp. 1669-1678, 2017, doi: 10.4236/psych.2017.811110.
[9] A. Diamond, “Executive functions,” Annual Review of Psychology, vol. 64, pp. 135-168, 2013, doi: 10.1146/annurev-psych-113011-143750.
[10] E. Weissblueth and Y. Nissim, “The contribution of virtual reality to social and emotional learning in pre-service teachers,” Creative Education, vol. 9, pp. 1551–1564, 2018. doi:10.4236/ce.2018.910114.
[11] F. Zhang et al., “MediaPipe Hands: On-device real-time hand tracking,” arXiv preprint arXiv:2006.10214, 2020, doi: 10.48550/arXiv.2006.10214.
[12] S. Karthik, S. Bhaggiaraj, E. S. Soji, L. S. Deve, S. S. Priscila, and S. S. Rajest, “Real-time hand gesture recognition for stroke rehabilitation using deep learning approach,” AVE Trends in Intelligent Computing Systems, vol. 1, no. 3, pp. 157–172, 2024.
[13] A. Faux-Nightingale et al., “Exploring stroke survivors’ and physiotherapists’ perspectives of the potential for markerless motion capture technology in community rehabilitation,” Journal of NeuroEngineering and Rehabilitation, vol. 21, Art. no. 168, 2024, doi: 10.1186 /s12984- 024-01467-x.
[14] P. Y. Hsu et al., “The evolution of augmented reality to augment physical therapy: A scoping review,” Journal of Rehabilitation and Assistive Technologies Engineering, vol. 11, Art. no. 20556683241252092, 2024, doi: 10.1177/20556683241252092.
[15] K. Mitgutsch and N. Alvarado, “Purposeful by design?: A serious game design assessment framework,” in Proc. International Conference on the Foundations of Digital Games (FDG), 2012, pp. 121–128, doi: 10.1145/2282338.2282364.
[16] E. L. Deci and R. M. Ryan, “The ‘what’ and ‘why’ of goal pursuits: Human needs and the self-determination of behavior,” Psychological Inquiry, vol. 11, no. 4, pp. 227–268, 2000, doi: 10.1207 /S15327965PLI1104_01.
[17] K. Hayase, “Concerns involving the self: What is the real target of anxiety, regret, or worry, when things do not go right for you? More evidence from Sweden and the United States,” Psychology, vol. 9, pp. 1475–1515, 2018, doi: 10.4236 /psych.2018.96090.
[18] B. F. Skinner, The Behavior of Organisms: An Experimental Analysis. New York, NY, USA: Appleton-Century, 1938.
[19] E. H. Shortliffe, Computer-Based Medical Consultations: MYCIN. New York, NY, USA: Elsevier, 1976.
[20] A. J. Thirunavukarasu et al., “Large language models in medicine,” Nature Medicine, vol. 29, no. 8, pp. 1930–1940, 2023, doi: 10.1038/s41591-023-02448-8.
[21] X. Meng et al., “The application of large language models in medicine: A scoping review,” iScience, vol. 27, Art. no. 109713, 2024, doi: 10.1016/j.isci.2024. 109713.
[22] J. Long, “Large language model guided tree-of-thought,” arXiv preprint arXiv:2305.08291, 2023, doi: 10.48550/arXiv.2305.08291.
[23] A. Y. Bischof et al., “A collection of components to design clinical dashboards incorporating patient-reported outcome measures: Qualitative study,” Journal of Medical Internet Research, vol. 26, Art. no. e55267, 2024, doi: 10.2196/55267.
[24] D. J. Hekman et al., “Dashboarding to monitor machine-learning-based clinical decision support interventions,” Applied Clinical Informatics, vol. 15, pp. 164–169, 2024, doi: 10.1055/a-2219-5175.
[25] P. Dullabh et al., “Advancing patient-centered clinical decision support in today’s health care ecosystem: Key themes from the Clinical Decision Support Innovation Collaborative’s 2023 Annual Meeting,” JAMIA Open, vol. 7, no. 4, Art. no. ooae109, 2024, doi: 10.1093/jamiaopen/ooae109.
Downloads
Published
How to Cite
Issue
Section
Categories
License
Copyright (c) 2026 Journal of Engineering and Industrial Technology, Kalasin University
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
ลิขสิทธิ์ของวารสาร
เนื้อหาและข้อมูลในบทความที่ลงตีพิมพ์ในวารสารศูนย์ดัชนีการอ้างอิงวารสารไทย ถือเป็นข้อคิดเห็นและความรับผิดชอบของผู้เขียนบทความโดยตรงซึ่งกองบรรณาธิการวารสาร ไม่จำเป็นต้องเห็นด้วย หรือร่วมรับผิดชอบใด ๆ
บทความ ข้อมูล เนื้อหา รูปภาพ ฯลฯ ที่ได้รับการตีพิมพ์ในวารสารศูนย์ดัชนีการอ้างอิงวารสารไทย ถือเป็นลิขสิทธิ์ของวารสารศูนย์ดัชนีการอ้างอิงวารสารไทย หากบุคคลหรือหน่วยงานใดต้องการนำทั้งหมดหรือส่วนหนึ่งส่วนใดไปเผยแพร่ต่อหรือเพื่อกระทำการใด จะต้องได้รับอนุญาตเป็นลายลักอักษรจากวารสารศูนย์ดัชนีการอ้างอิงวารสารไทยก่อนเท่านั้น
