(Universitas Negeri Malang, Indonesia)(2) Sholikhatul Amaliya
(Universitas Negeri Malang, Indonesia)(3) Muhammad Iqbal Akbar
(Universitas Negeri Malang, Indonesia)(4) Muhammad Zaki Wiryawan
(Universitas Negeri Malang, Indonesia)(5) Yeoh Wen Liang
(Saga University, Japan)(6) Wendy Cahya Kurniawan
(Saga University, Japan)*corresponding author
AbstractSIBI is less widely adopted, and the lack of an efficient recognition system limits its accessibility. SIBI gestures often involve subtle hand movements and complex finger configurations, requiring precise feature extraction and classification techniques. This study addresses these issues using a Hand Keypoint-based Convolutional Neural Network (HK-CNN) for SIBI classification. The research utilizes Kinect 2.0 for precise data collection, enabling accurate hand keypoint detection and preprocessing. The optimal data acquisition distance between 50 and 60 cm from the camera is considered to obtain clear and detailed images. The methodology includes four key stages: data collection, preprocessing (keypoint extraction and image filtering), classification using HK-CNN with ResNet-50, EfficientNet, and InceptionV3, and performance evaluation. Experimental results demonstrate that EfficientNet achieves the highest accuracy of 99.1% in the 60:40 data split scenario, with superior precision and recall, making it ideal for real-time applications. ResNet-50 also performs well with 99.3% accuracy in the 20:80 split but requires longer computation time, while InceptionV3 is less efficient for real-time applications. Compared to traditional CNN methods, HK-CNN significantly enhances accuracy and efficiency. In conclusion, this study provides a robust and adaptable solution for SIBI recognition, facilitating inclusivity in education, public services, and workplace communication. Future research should expand dataset diversity and explore dynamic gesture recognition for further improvements.
KeywordsConvolutional Neural Network (CNN); Hand Keypoint; Image Classification; Sistem Isyarat Bahasa Indonesia (SIBI)
|
DOIhttps://doi.org/10.31763/ijrcs.v5i2.1745 |
Article metrics10.31763/ijrcs.v5i2.1745 Abstract views : 205 | PDF views : 65 |
Cite |
Full Text Download
|
References
[1] K. Emmorey, “Ten Things You Should Know About Sign Languages,” Current Directions in Psychological Science, vol. 32, no. 5, pp. 387–394, 2023, https://doi.org/10.1177/09637214231173071.
[2] R. Rastgoo, K. Kiani, S. Escalera, V. Athitsos, and M. Sabokrou, “A survey on recent advances in Sign Language Production,” Expert Systems with Applications, vol. 243, p. 122846, 2024, https://doi.org/10.1016/j.eswa.2023.122846.
[3] K. K. Sukhadan, V. D. Bakhade, G. S. Thakare, K. G. Dhanbhar, and A. C. Deshmukh, “Sign Language Recognition System,” International Journal for Research in Applied Science & Engineering Technology, vol. 12, no. 3, pp. 140–143, 2024, https://doi.org/10.22214/ijraset.2024.58758.
[4] D. Lillo-Martin and J. A. Hochgesang, “Signed languages – Unique and ordinary: A commentary on Kidd and Garcia (2022),” First Language, vol. 42, no. 6, pp. 789–793, 2022, https://doi.org/10.1177/01427237221098858.
[5] D. Novaliendry, M. F. P. Pratama, K. Budayawan, Y. Huda, and W. M. Y. Rahiman, “Design and Development of Sign Language Learning Application for Special Needs Students Based on Android Using Flutter,” International Journal of Online and Biomedical Engineering (iJOE), vol. 19, no. 16, pp. 76–92, 2023, https://doi.org/10.3991/ijoe.v19i16.44669.
[6] A. Taupiq, M. Wildan Fajri, and Dannylee, “Identification of Indonesian Sign Language System Using Deep Learning in Yolo-based,” Media Journal of General Computer Science, vol. 1, no. 2, pp. 40–47, 2024, https://doi.org/10.62205/mjgcs.v1i2.22.
[7] M. Marlina, A. Mahdi, and Y. Karneli, “The effectiveness of the Bisindo-based rational emotive behavior therapy model in reducing social anxiety in deaf women victims of sexual harassment,” The Journal of Adult Protection, vol. 25, no. 4, pp. 199–214, 2023, https://doi.org/10.1108/JAP-10-2022-0024.
[8] M. Marlina, Y. T. Ningsih, Z. Fikry, and D. R. Fransiska, “Bisindo-based rational emotive behaviour therapy model: study preliminary prevention of sexual harassment in women with deafness,” The Journal of Adult Protection, vol. 24, no. 2, pp. 102–114, 2022, https://doi.org/10.1108/JAP-09-2021-0032.
[9] N. K. I. Wahyuni, I. M. Suarjana, and D. A. P. Handayani, “Development Of Video Learning Based On The Indonesian Language Signing System (SIBI) Method For Class II Deaf Chill At SDN 2 Bengkala Academic Year 2022/2023,” Journal of Psychology and Instruction, vol. 5, no. 3, 2023, https://doi.org/10.23887/jpai.v5i3.64832.
[10] R. Rastgoo, K. Kiani, and S. Escalera, “Sign Language Recognition: A Deep Survey,” Expert Systems with Applications, vol. 164, p. 113794, 2021, https://doi.org/10.1016/j.eswa.2020.113794.
[11] D. Straupeniece, D. Bethere, E. Ozola, “Sign Language of the Deaf People: A Study on Public Understanding,” Education. Innovation. Diversity, vol. 2, no. 7, pp. 109–114, 2024, https://doi.org/10.17770/eid2023.2.7356.
[12] S. Arooj, S. Altaf, S. Ahmad, H. Mahmoud, and A. S. N. Mohamed, “Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign language,” Journal of King Saud University - Computer and Information Sciences, vol. 36, no. 2, p. 101934, 2024, https://doi.org/10.1016/j.jksuci.2024.101934.
[13] N. K. Jyothi, V. Harshitha, M. Puneira, P. S. Nikitha, “Image Processing Model for Sign Language Recognition System,” International Journal for Research in Applied Science & Engineering Technology (IJRASET), vol. 12, no. 5, pp. 2235–2237, 2024, https://doi.org/10.22214/ijraset.2024.61671.
[14] S. Renjith and R. Manazhy, “Sign language: a systematic review on classification and recognition,” Multimedia Tools and Applications, vol. 83, pp. 77077–77127, 2024, https://doi.org/10.1007/s11042-024-18583-4.
[15] M. Alaftekin, I. Pacal, and K. Cicek, “Real-time sign language recognition based on YOLO algorithm,” Neural Computing and Applications, vol. 36, pp. 7609–7624, 2024, https://doi.org/10.1007/s00521-024-09503-6.
[16] E. Aldhahri et al., “Arabic Sign Language Recognition Using Convolutional Neural Network and MobileNet,” Arabian Journal for Science and Engineering, vol. 48, no. 2, pp. 2147–2154, 2023, https://doi.org/10.1007/s13369-022-07144-2.
[17] S. Dwijayanti, S. I. Taqiyyah, H. Hikmarika, and B. Y. Suprapto, “Indonesia Sign Language Recognition using Convolutional Neural Network,” International Journal of Advanced Computer Science and Applications, vol. 12, no. 10, 2021, https://dx.doi.org/10.14569/IJACSA.2021.0121046.
[18] A. Osman Hashi, S. Zaiton Mohd Hashim and A. Bte Asamah, "A Systematic Review of Hand Gesture Recognition: An Update From 2018 to 2024," IEEE Access, vol. 12, pp. 143599-143626, 2024, https://doi.org/10.1109/ACCESS.2024.3421992.
[19] D. Pribadi, M. Wahyudi, D. Puspitasari, A. Wibowo, R. Saputra, and R. Saefurrohman, “Real Time Indonesian Sign Language Hand Gesture Phonology Translation Using Deep Learning Model,” Scitepress, vol. 1, pp. 172–176, 2024, https://doi.org/10.5220/0012446000003848.
[20] G. Chaganava and D. Kakulia, “Keypoint Detector Retraining Techniques for the Communication System of Sign Language Speakers,” Eski?ehir Technical University Journal of Science and Technology A - Applied Sciences and Engineering, vol. 21, pp. 74–86, 2020, https://doi.org/10.18038/estubtda.822295.
[21] G. Kim, J. Cho, G. Kim, B. Kim, and K. Jeon, “A Keypoint-based Sign Language Start and End Point Detection Scheme,” KIISE Transactions on Computing Practices, vol. 29, no. 4, pp. 184–189, 2023, https://doi.org/10.5626/KTCP.2023.29.4.184.
[22] R. Rastgoo, K. Kiani, and S. Escalera, “Hand sign language recognition using multi-view hand skeleton,” Expert Systems with Applications, vol. 150, p. 113336, 2020, https://doi.org/10.1016/j.eswa.2020.113336.
[23] T. G. K and M. N. Nachappa, “Sign Language Recognition by Image Processing,” International Journal of Advanced Research in Science, Communication and Technology, vol. 4, no. 4, pp. 306–310, 2024, https://doi.org/10.48175/IJARSCT-15954.
[24] S. Mittal, S. Srivastava and J. P. Jayanth, "A Survey of Deep Learning Techniques for Underwater Image Classification," IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 10, pp. 6968-6982, 2023, https://doi.org/10.1109/TNNLS.2022.3143887.
[25] A. Sharma, N. Sharma, Y. Saxena, A. Singh, and D. Sadhya, “Benchmarking deep neural network approaches for Indian Sign Language recognition,” Neural Computing and Applications, vol. 33, no. 12, pp. 6685–6696, 2021, https://doi.org/10.1007/s00521-020-05448-8.
[26] M. Momeny et al., “Learning-to-augment strategy using noisy and denoised data: Improving generalizability of deep CNN for the detection of COVID-19 in X-ray images,” Computers in Biology and Medicine, vol. 136, p. 104704, 2021, https://doi.org/10.1016/j.compbiomed.2021.104704.
[27] J. Hai et al., “R2RNet: Low-light image enhancement via Real-low to Real-normal Network,” Journal of Visual Communication and Image Representation, vol. 90, p. 103712, 2023, https://doi.org/10.1016/j.jvcir.2022.103712.
[28] A. Wadhawan, P. Kumar, “Sign language recognition systems: A decade systematic literature review,” Archives of computational methods in engineering, vol. 28, pp. 785-813, 2021, https://doi.org/10.1007/s11831-019-09384-2.
[29] R. Sutjiadi, “Android-Based Application for Real-Time Indonesian Sign Language Recognition Using Convolutional Neural Network,” TEM Journal, vol. 12, no. 3, pp. 1541–1549, 2023, https://doi.org/10.18421/TEM123-35.
[30] Y. Obi, K. S. Claudio, V. M. Budiman, S. Achmad, and A. Kurniawan, “Sign language recognition system for communicating to people with disabilities,” Procedia Computer Science, vol. 216, pp. 13–20, 2022, https://doi.org/10.1016/j.procs.2022.12.106.
[31] U. Özsoy, Y. Y?ld?r?m, S. Kara?in, R. ?ekerci, and L. B. Süzen, “Reliability and agreement of Azure Kinect and Kinect v2 depth sensors in the shoulder joint range of motion estimation,” Journal of Shoulder and Elbow Surgery, vol. 31, no. 10, pp. 2049–2056, 2022, https://doi.org/10.1016/j.jse.2022.04.007.
[32] L.-F. Yeung, Z. Yang, K. C.-C. Cheng, D. Du, and R. K.-Y. Tong, “Effects of camera viewing angles on tracking kinematic gait patterns using Azure Kinect, Kinect v2 and Orbbec Astra Pro v2,” Gait & Posture, vol. 87, pp. 19–26, 2021, https://doi.org/10.1016/j.gaitpost.2021.04.005.
[33] C. Posner, A. Sánchez-Mompó, I. Mavromatis, and M. Al-Ani, “A dataset of human body tracking of walking actions captured using two Azure Kinect sensors,” Data in Brief, vol. 49, p. 109334, 2023, https://doi.org/10.1016/j.dib.2023.109334.
[34] C. Neupane, A. Koirala, Z. Wang, and K. B. Walsh, “Evaluation of Depth Cameras for Use in Fruit Localization and Sizing: Finding a Successor to Kinect v2,” Agronomy, vol. 11, no. 9, p. 1780, 2021, https://doi.org/10.3390/agronomy11091780.
[35] I. D. M. B. A. Darmawan, Linawati, G. Sukadarmika, N. M. A. E. D. Wirastuti, and R. Pulungan, “Temporal Action Segmentation in Sign Language System for Bahasa Indonesia (SIBI) Videos Using Optical Flow-Based Approach,” Jurnal Ilmu Komputer dan Informasi, vol. 17, no. 2, pp. 195–202, 2024, https://doi.org/10.21609/jiki.v17i2.1284.
[36] A. Albar, H. Hendrick, and R. Hidayat, “Segmentation Method for Face Modelling in Thermal Images,” Knowledge Engineering and Data Science, vol. 3, no. 2, pp. 99-105, 2020, http://dx.doi.org/10.17977/um018v3i22020p99-105.
[37] M. A. Ridwan and H. Mubarok, “The Recognition of American Sign Language Using CNN with Hand Keypoint,” International Journal on Information and Communication Technology, vol. 9, no. 2, pp. 86–95, 2023, https://socjs.telkomuniversity.ac.id/ojs/index.php/ijoict/article/view/845.
[38] E. Rakun and N. F. Setyono, “Improving Recognition of SIBI Gesture by Combining Skeleton and Hand Shape Features,” Jurnal Ilmu Komputer dan Informasi, vol. 15, no. 2, pp. 69–79, 2022, https://doi.org/10.21609/jiki.v15i2.1014.
[39] I. A. Adeyanju, O. O. Bello, and M. A. Adegboye, “Machine learning methods for sign language recognition: A critical review and analysis,” Intelligent Systems with Applications, vol. 12, p. 200056, 2021, https://doi.org/10.1016/j.iswa.2021.200056.
[40] T. Rabie, M. Baziyad, R. Sani, T. Bonny, and R. Fareh, “Color Histogram Contouring: A New Training-Less Approach to Object Detection,” Electronics, vol. 13, no. 13, p. 2522, 2024, https://doi.org/10.3390/electronics13132522.
[41] H. Ohno, “One-shot reflectance direction color mapping for identifying surface roughness,” Precision Engineering, vol. 85, pp. 65–71, 2024, https://doi.org/10.1016/j.precisioneng.2023.09.004.
[42] A. Shah, N. Azam, E. Alanazi, and J. Yao, “Image blurring and sharpening inspired three-way clustering approach,” Applied Intelligence, vol. 52, no. 15, pp. 18131–18155, 2022, https://doi.org/10.1007/s10489-021-03072-0.
[43] T. Wu, J. Shao, X. Gu, M. K. Ng, and T. Zeng, “Two-stage image segmentation based on nonconvex approximation and thresholding,” Applied Mathematics and Computation, vol. 403, p. 126168, 2021, https://doi.org/10.1016/j.amc.2021.126168.
[44] M. M. Tall, I. Ngom, O. Sadio, A. Coulibaly, I. Diagne, and M. Ndiaye, “Automatic detection and counting of fisheries using fish images,” Bulletin of Social Informatics Theory and Application, vol. 7, no. 2, pp. 150–162, 2023, https://doi.org/10.31763/businta.v7i2.655.
[45] J. Zhang, X. Bu, Y. Wang, H. Dong, Y. Zhang, and H. Wu, “Sign language recognition based on dual-path background erasure convolutional neural network,” Scientific Reports, vol. 14, no. 1, p. 11360, 2024, https://doi.org/10.1038/s41598-024-62008-z.
[46] R. Gupta and A. Kumar, “Indian sign language recognition using wearable sensors and multi-label classification,” Computers & Electrical Engineering, vol. 90, p. 106898, 2021, https://doi.org/10.1016/j.compeleceng.2020.106898.
[47] A. P. Wibawa et al., “Frontier Energy System and Power Engineering Forecasting Hourly Energy Fluctuations Using Recurrent Neural Network (RNN),” Frontier Energy System and Power Engineering, vol. 5, no. 2, pp. 50–57, 2023, http://dx.doi.org/10.17977/um049v5i2p50-57.
[48] L. Latumakulita et al., “Web-Based System for Medicinal Plants Identification Using Convolutional Neural Network,” Bulletin of Social Informatics Theory and Application, vol. 6, no. 2, pp. 158–167, 2022, https://doi.org/10.31763/businta.v6i2.601.
[49] M. C. Bagaskoro, F. Prasojo, A. N. Handayani, E. Hitipeuw, A. P. Wibawa, and Y. W. Liang, “Hand image reading approach method to Indonesian Language Signing System (SIBI) using neural network and multi layer perseptron,” Science in Information Technology Letters, vol. 4, no. 2, pp. 97–108, 2023, https://doi.org/10.31763/sitech.v4i2.1362.
[50] L. Zhou, X. Ma, X. Wang, S. Hao, Y. Ye, and K. Zhao, “Shallow-to-Deep Spatial–Spectral Feature Enhancement for Hyperspectral Image Classification,” Remote Sensing, vol. 15, no. 1, p. 261, 2023, https://doi.org/10.3390/rs15010261.
[51] I. H. Sarker, “Deep Learning: A Comprehensive Overview on Techniques, Taxonomy, Applications and Research Directions,” SN Computer Science, vol. 2, no. 6, p. 420, 2021, https://doi.org/10.1007/s42979-021-00815-1.
[52] K. Narang, M. Gupta, R. Kumar and A. J. Obaid, "Channel Attention Based on ResNet-50 Model for Image Classification of DFUs Using CNN," 2024 5th International Conference for Emerging Technology (INCET), pp. 1-6, 2024, https://doi.org/10.1109/INCET61516.2024.10593169.
[53] J. Liu, “Face recognition technology based on ResNet-50,” Applied and Computational Engineering, vol. 39, no. 1, pp. 160–165, 2024, https://doi.org/10.54254/2755-2721/39/20230593.
[54] B. Deng et al., “Application of RESNET50 Convolution Neural Network for the Extraction of Optical Parameters in Scattering Media,” arXiv, 2024, https://doi.org/10.48550/arXiv.2404.16647.
[55] M. Yin, X. Li, Y. Zhang, S. Wang, “On the Mathematical Understanding of ResNet with Feynman Path Integral,” arXiv, 2019, https://doi.org/10.48550/arXiv.1904.07568.
[56] A. Kumar, L. Nelson and S. Singh, "ResNet-50 Transfer Learning Model for Diabetic Foot Ulcer Detection Using Thermal Images," 2023 2nd International Conference on Futuristic Technologies (INCOFT), pp. 1-5, 2023, https://doi.org/10.1109/INCOFT60753.2023.10425447.
[57] L. M. K. Sheikh, A. Shaikh, A. Sandupatla, R. Pudale, A. Bakare, M. Chavan, “Classification of Simple CNN Model and ResNet50,” International Journal for Research in Applied Science & Engineering Technology, vol. 12, no. 4, pp. 4606-4610, 2024, https://doi.org/10.22214/ijraset.2024.60677.
[58] D. F. Laistulloh, A. N. Handayani, R. A. Asmara, and P. Taw, “Convolutional Neural Network in Motion Detection for Physiotherapy Exercise Movement,” Knowledge Engineering and Data Science, vol. 7, no. 1, pp. 27-39, 2024, http://dx.doi.org/10.17977/um018v7i12024p27-39.
[59] M. Dolla Meitantya, C. Atika Sari, E. Hari Rachmawanto, and R. Raad Ali, “VGG-16 Architecture on CNN for American Sign Language Classification,” Jurnal Teknik Informatika, vol. 5, no. 4, pp. 1165–1171, 2024, https://doi.org/10.52436/1.jutif.2024.5.4.2160.
[60] S. Sony, K. Dunphy, A. Sadhu, and M. Capretz, “A systematic review of convolutional neural network-based structural condition assessment techniques,” Engineering Structures, vol. 226, p. 111347, 2021, https://doi.org/10.1016/j.engstruct.2020.111347.
[61] S. Das, Md. S. Imtiaz, N. H. Neom, N. Siddique, and H. Wang, “A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifier,” Expert Systems with Applications, vol. 213, p. 118914, 2023, https://doi.org/10.1016/j.eswa.2022.118914.
[62] A. W. Salehi et al., “A Study of CNN and Transfer Learning in Medical Imaging: Advantages, Challenges, Future Scope,” Sustainability, vol. 15, no. 7, p. 5930, 2023, https://doi.org/10.3390/su15075930.
[63] M. Iman, H. R. Arabnia, and K. Rasheed, “A Review of Deep Transfer Learning and Recent Advancements,” Technologies, vol. 11, no. 2, p. 40, 2023, https://doi.org/10.3390/technologies11020040.
Refbacks
- There are currently no refbacks.
Copyright (c) 2025 Anik Nur Handayani, Sholikhatul Amaliya, Muhammad Iqbal Akbar, Muhammad Zaki Wiryawan, Yeoh Wen Liang, Wendy Cahya Kurniawan
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
| About the Journal | Journal Policies | Author | Information |
International Journal of Robotics and Control Systems
e-ISSN: 2775-2658
Website: https://pubs2.ascee.org/index.php/IJRCS
Email: ijrcs@ascee.org
Organized by: Association for Scientific Computing Electronics and Engineering (ASCEE), Peneliti Teknologi Teknik Indonesia, Department of Electrical Engineering, Universitas Ahmad Dahlan and Kuliah Teknik Elektro
Published by: Association for Scientific Computing Electronics and Engineering (ASCEE)
Office: Jalan Janti, Karangjambe 130B, Banguntapan, Bantul, Daerah Istimewa Yogyakarta, Indonesia