Psychometric Evaluation of a START Triage Psychomotor Observation Instrument: Content Validity and Interrater Reliability
DOI:
https://doi.org/10.53713/htechj.v4i4.739Keywords:
clinical competence, disaster nursing, mass casualty incidents, psychometrics, psychomotor performanceAbstract
Accurate psychomotor assessment in Simple Triage and Rapid Treatment (START) is critical during mass-casualty incidents. Observation-based evaluations often yield inconsistent judgments when utilizing unvalidated instruments, compromising competency assessments. This study aimed to establish the content validity and interrater reliability of a newly developed psychomotor observation sheet for START triage among healthcare professionals. This methodological study evaluated a 12-item dichotomous observation sheet derived from standard START protocols. Content validity was determined by three emergency and disaster nursing experts using a four-point relevance scale, analyzed with Aiken's V. Interrater reliability was assessed with 36 medical staff members, independently evaluated by two trained raters. Agreement metrics included percent agreement and Cohen's kappa. Expert evaluation yielded a perfect Aiken's V of 1.00 across all 12 items, indicating exceptional content validity. Interrater reliability testing demonstrated percent agreement ranging from 94.44% to 100%, alongside Cohen's kappa coefficients between 0.85 and 1.00. These metrics significantly exceeded minimum acceptability thresholds, reflecting outstanding consistency and minimal scoring ambiguity among raters. The START triage psychomotor observation sheet exhibits robust content validity and excellent interrater reliability. This validated instrument provides educators with a standardized, objective tool for evaluating clinical competencies in disaster preparedness training, though future validation in high-fidelity simulation environments is recommended.
References
Abdul-Nabi, S. S., & Hitti, E. (2025). A review of mass casualty incident triage tools for hospital-based triage. Turkish Journal of Emergency Medicine, 25(4), 251–255. https://doi.org/10.4103/tjem.tjem_77_25
Aiken, L. R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45(1), 131–142. https://doi.org/10.1177/0013164485451012
Armstrong, J. B. P. (2024). Preparing your emergency department for disaster: Optimizing surge capacity during mass casualty events. Healthcare Management Forum. https://doi.org/10.1177/08404704231199403
Bourassa, M., Kolb, W. H., Cunningham, S., Neurauter, V., Bourassa, A., Grieshaber, E., Loyd, K., Walden, R. R., & Wassinger, C. (2024). Psychomotor skills assessment in healthcare education: A scoping review. Physical Therapy Reviews, 29(1–3), 64–86. https://doi.org/10.1080/10833196.2024.2348927
Carenzo, L., Ingrassia, P. L., Foti, F., Albergoni, E., Colombo, D., Sechi, G. M., Zoli, A., & Sironi, S. (2022). A region-wide all-hazard training program for prehospital mass casualty incident management: A real-world case study. Disaster Medicine and Public Health Preparedness, 17, e184. https://doi.org/10.1017/dmp.2022.84
Elshami, S., Yakti, O., Mohamed Ibrahim, M. I., Awaisu, A., Sherbash, M., & Mukhalalati, B. (2025). Instruments for the assessment of disaster management among healthcare professionals: A scoping review. Frontiers in Public Health, 13, 1540743. https://doi.org/10.3389/fpubh.2025.1540743
Faizatiwahida, N., Alfrisa, B., Nurjannah, I., & Alim, S. (2025). The effect of Simple Triage and Rapid Treatment (START) training on cognitive, affective, and psychomotor skills in performing START triage among medical staff and healthcare professionals. Jurnal Berita Ilmu Keperawatan, 18(2), 173–184. https://doi.org/10.23917/bik.v18i2.9861
Faizatiwahida, N., Irnu, J. A., Ayuningtyas, P. A., Hatta, H., Nurjannah, I., & Alim, S. (2024). Validity and reliability of the START triage knowledge and attitude instrument in medical personnel and health workers. Jurnal Berita Ilmu Keperawatan, 17(2), 202–210. https://doi.org/10.23917/bik.v17i2.5147
Franc, J. M., Kirkland, S. W., Wisnesky, U. D., Campbell, S., & Rowe, B. H. (2022). METASTART: A systematic review and meta-analysis of the diagnostic accuracy of the Simple Triage and Rapid Treatment (START) algorithm for disaster triage. Prehospital and Disaster Medicine, 37(1), 106–116. https://doi.org/10.1017/S1049023X2100131X
Gabbe, B. J., Veitch, W., Mather, A., Curtis, K., Holland, A. J., Gomez, D., Civil, I., Nathens, A., Fitzgerald, M., Martin, K., Teague, W. J., & Joseph, A. (2022). Review of the requirements for effective mass casualty preparedness for trauma systems: A disaster waiting to happen? British Journal of Anaesthesia, 128(2), e158–e167. https://doi.org/10.1016/j.bja.2021.10.038
Hosseinzadeh, M., Nematollahi, A. V., Afra, Y., Amini, P., & Navaei, A. R. (2024). Effectiveness of distance training based on the Simple Triage and Rapid Treatment (START) triage system in pre-hospital emergency. Journal of Medical Education and Curricular Development, 11. https://doi.org/10.1177/23821205241278658
Huabbangyang, T., Thepmanee, D., Buaprasert, P., Chansomboon, P., Sri-on, J., & Rojsaengroeng, R. (2024). Content validity assessment of a newly developed emergency medical dispatch and triage protocol in Thailand. Journal of Clinical Medicine, 14(19), 7125. https://doi.org/10.3390/jcm14197125
Javid, S. A., Jamil, M., Noor, B., Umer, M. R., Asghar, A., Hassan, M. R., Akbar, A., Hussain, S., & Khan, W. (2025). Emergency preparedness and response: Strategies for mass casualty incidents. Cureus, 17(7), e89158. https://doi.org/10.7759/cureus.89158
Lin, Y., Chen, K., Wang, J., & Lai, P. (2022). Simple Triage and Rapid Treatment protocol for emergency department mass casualty incident victim triage. The American Journal of Emergency Medicine, 53, 99–103. https://doi.org/10.1016/j.ajem.2021.12.037
Mokkink, L. B., Herbelet, S., Tuinman, P. R., & Terwee, C. B. (2025). Content validity: Judging the relevance, comprehensiveness, and comprehensibility of an outcome measurement instrument—A COSMIN perspective. Journal of Clinical Epidemiology, 185, 111879. https://doi.org/10.1016/j.jclinepi.2025.111879
Moura, B. R. S., Oliveira, G. N., Medeiros, G., Vieira, S., & Nogueira, S. (2022). Rapid triage performed by nurses: Signs and symptoms associated with identifying critically ill patients in the emergency department. International Journal of Nursing Practice, 28(1), e13001. https://doi.org/10.1111/ijn.13001
Pandawa, R. M., Saleh, F. M., & Muhammad, R. (2025). Development of disaster nursing competency assessment instruments. International Journal of Public Health Science, 14(1), 94–106. https://doi.org/10.11591/ijphs.v14i1.24697
Setyawati, A.-D., Lu, Y.-Y., Wu, S.-F. V., & Liang, S.-Y. (2025). Effects of disaster triage training program on knowledge, skills, and resilience among nurses: A randomized controlled trial. Disaster Medicine and Public Health Preparedness, 19, e298. https://doi.org/10.1017/dmp.2025.10233
Shabani, M., Heidarimoghadam, R., Faradmal, J., & Babamiri, M. (2025). Identifying the cognitive, psychomotor, and sensory-perceptual abilities required for the role of pre-hospital emergency medical personnel. WORK. https://doi.org/10.1177/10519815241295907
Wisnesky, U. D., Kirkland, S. W., Rowe, B. H., Campbell, S., & Franc, J. M. (2022). A qualitative assessment of studies evaluating the classification accuracy of personnel using START in disaster triage: A scoping review. Frontiers in Public Health, 10, 676704. https://doi.org/10.3389/fpubh.2022.676704
Zhao, X., Feng, G. C., Ao, S. H., & Liu, P. L. (2022). Interrater reliability estimators tested against true interrater reliabilities. BMC Medical Research Methodology, 22, Article 232. https://doi.org/10.1186/s12874-022-01707-5
