Mass-casualty incidents demand rapid, structured triage decisions to optimize the allocation of limited healthcare resources during sudden surges in patient volume (Armstrong, 2024). Emergency responders must quickly prioritize patients requiring urgent intervention to prevent the inappropriate distribution of personnel, equipment, and referral capacity (Gabbe et al., 2022). Recent literature on mass-casualty triage emphasizes that effective tools must remain simple, fast, and highly applicable in resource-limited contexts where healthcare professionals must manage overwhelming numbers of victims (Abdul-Nabi & Hitti, 2025). This global necessity underscores the critical importance of standardized triage protocols in modern disaster response frameworks (Javid et al., 2025).

The Simple Triage and Rapid Treatment (START) protocol requires healthcare professionals to translate complex clinical assessments into immediate physical actions within a strict 30- to 60-second timeframe per victim (Franc et al., 2022). This approach classifies victims by rapidly evaluating walking ability, respiration, perfusion, and mental status (Lin et al., 2022). The instantaneous translation of assessment rules into decisive physical actions makes psychomotor competence a central component of safe triage performance. Cognitive understanding alone is insufficient to ensure accurate patient categorization in the chaotic environment of a mass casualty incident (Abdul-Nabi & Hitti, 2025).

Recent educational interventions have successfully demonstrated that targeted START triage training significantly enhances the cognitive, affective, and psychomotor domains of medical staff and healthcare professionals (Hosseinzadeh et al., 2024). Training studies indicate substantial improvements in the execution of practical skills following structured disaster preparedness programs (Faizatiwahida et al., 2025). Previous instrument development efforts have also established that knowledge and attitude instruments related to START triage can be effectively validated for medical and health personnel (Faizatiwahida et al., 2024). These advancements highlight a strong institutional focus on improving disaster response capabilities through comprehensive educational strategies.

Observation-based psychomotor assessments remain highly vulnerable to rater variation and subjective bias when the underlying measurement items lack sufficient clarity and standardized interpretation (Shabani et al., 2025). Direct observation of physical skills requires absolute precision in item formulation to prevent divergent scoring among different evaluators (Wisnesky et al., 2022). This methodological issue is critically important because modern healthcare education increasingly demands highly valid instruments for accurately assessing observable clinical skills (Bourassa et al., 2024). The absence of rigorously validated psychomotor tools creates a significant barrier to objectively measuring actual clinical competence in disaster scenarios.

Establishing robust content validity ensures that assessment items accurately represent the intended clinical construct, while evaluating interrater reliability confirms that different evaluators assign consistent scores to identical performances (Huabbangyang et al., 2024). Content validity is a fundamental step in instrument development, evaluating whether items adequately represent the construct being measured and remain understandable to users (Mokkink et al., 2025). Researchers commonly utilize Aiken's V to quantify expert agreement on item relevance and clarity (Aiken, 1985). Interrater reliability metrics, including Cohen's kappa and percent agreement, are frequently used in health research to assess observational consistency (Zhao et al., 2022).

The primary novelty of the current investigation lies in the development and rigorous psychometric validation of a dichotomous observation sheet specifically designed to evaluate START triage psychomotor execution. Existing literature predominantly focuses on cognitive and affective assessments, leaving a distinct gap in tools designed for direct physical observation. This study introduces a specialized 12-item instrument that captures the essential, observable steps of victim assessment, including initial sorting, respiration, perfusion, and mental status evaluation (Moura et al., 2022). The dichotomous scoring format specifically mitigates ambiguity during direct observation.

The present study aims to examine the content validity and interrater reliability of this newly developed psychomotor observation sheet, providing educators with a standardized, objective tool for competency assessment in disaster preparedness training. Addressing this methodological gap is urgently needed to ensure that healthcare professionals possess the verified physical skills required in real-world mass-casualty incidents (Carenzo et al., 2022). The study contributes vital psychometric evidence for an observation-based instrument that can be seamlessly integrated into START triage training, competency assessment, and broader disaster preparedness education initiatives.

The findings are presented in two parts: content validity based on expert judgment and interrater reliability based on agreement between two independent raters.

The present study established exceptional content validity for the newly developed START triage psychomotor observation sheet, with all twelve items achieving a perfect Aiken's V score of 1.00. This perfect score reflects unanimous consensus among the expert panel regarding the relevance, clarity, and representativeness of the assessment items. Content validity extends beyond mere surface relevance to ensure that the instrument comprehensively captures the underlying construct being measured while remaining comprehensible to the intended users (Mokkink et al., 2025). The START triage construct encompasses rapid, observable clinical actions, including victim sorting, respiratory evaluation, perfusion assessment, and mental status assessment. Expert agreement confirms that the selected items accurately map onto these critical psychomotor domains without introducing unnecessary complexity. The operational nature of START triage demands that assessment tools mirror the rapid translation of clinical signs into definitive triage categories. Quantitative validation through Aiken's V provides robust empirical evidence that expert judgment consistently supports this structural alignment (Aiken, 1985). This rigorous validation process confirms that the observation sheet functions effectively as a comprehensive psychomotor assessment tool rather than a superficial checklist of theoretical knowledge.

Interrater reliability testing further confirmed the robust consistency of the observation sheet, yielding percent agreement values above 94% and Cohen's kappa coefficients ranging from 0.85 to 1.00 across all indicators. These statistical outcomes indicate very good agreement beyond chance, demonstrating that independent raters interpret and score the psychomotor performances in a highly uniform manner. Observation-based assessment in clinical education depends heavily on rater interpretation, making the mitigation of subjective bias a critical priority. Inconsistency between raters can severely undermine the credibility of competency decisions and compromise the fairness of high-stakes evaluations (Zhao et al., 2022). The exceptionally high agreement observed in this study suggests that the operational definitions within the observation sheet are sufficiently clear to prevent divergent interpretations. The use of a dichotomous scoring format, which required raters to classify actions strictly as performed or not performed, played a pivotal role in reducing scoring ambiguity. This binary approach forces raters to focus on the presence or absence of specific, observable psychomotor steps, thereby minimizing the influence of subjective halo effects or personal grading leniency. The resulting statistical consistency proves that the instrument can be reliably deployed across different evaluators without sacrificing measurement precision.

The psychometric strengths of this observation instrument align seamlessly with recent empirical evidence emphasizing the critical need for competency-based disaster triage training. Faizatiwahida et al. (2025) demonstrated that targeted START triage training significantly improves the psychomotor execution skills of medical staff and healthcare professionals during simulated mass casualty incidents. The current study builds upon these educational outcomes by providing a specific psychometric tool to objectively measure these psychomotor improvements. Previous literature has also highlighted the broader impacts of disaster preparedness education on healthcare workers. Setyawati et al. (2025) found that structured disaster triage training programs substantially enhance nurses' knowledge, practical skills, and psychological resilience in emergency scenarios. The validated observation sheet from the present research serves as a crucial missing link in this educational ecosystem, providing educators with a reliable means to quantify the practical skill acquisition reported in those training studies. By supplying a rigorously tested assessment instrument, this research enables future educational interventions to move beyond self-reported confidence and instead measure actual, observable clinical competence.

This investigation also successfully bridges a significant methodological gap in the existing literature by shifting the focus from cognitive and attitudinal assessments to the direct evaluation of observable clinical skills. Prior instrument development efforts in this domain have predominantly concentrated on measuring the theoretical understanding and disaster-related attitudes of healthcare personnel. Faizatiwahida et al. (2024) successfully validated knowledge and attitude instruments for START triage among medical personnel, establishing a strong foundation for cognitive assessment. Cognitive knowledge and positive attitudes, although necessary, do not automatically translate into flawless psychomotor execution during the chaotic environment of a mass casualty incident. The current study addresses this translational gap by introducing a validated tool specifically designed to capture the physical execution of triage protocols. Evaluating psychomotor performance requires direct observation under standardized conditions, an approach that captures the physical dexterity, speed, and sequential accuracy demanded by real-world triage. The provision of this psychomotor observation sheet completes the triad of assessment tools, allowing researchers and educators to comprehensively evaluate the cognitive, affective, and psychomotor domains of disaster nursing competence (Pandawa et al., 2025).

The establishment of strong content validity and interrater reliability carries profound implications for disaster preparedness education and clinical competency evaluation within healthcare institutions. Validated psychomotor instruments are absolute prerequisites for ensuring that training outcomes are measured fairly, consistently, and objectively across diverse educational settings. Nursing educators and hospital training coordinators can now utilize this observation sheet to conduct standardized evaluations during simulation-based learning exercises. The tool enables the identification of specific psychomotor deficits, allowing instructors to provide targeted corrective feedback to individual learners before they encounter real mass-casualty incidents (Elshami et al., 2025). Healthcare institutions can integrate this instrument into their routine disaster drill protocols to objectively assess the readiness of their medical staff. Regulatory bodies and credentialing organizations may also adopt this validated sheet to standardize the certification processes for emergency and disaster nursing competencies. The availability of a reliable, dichotomous observation format significantly reduces the administrative burden on evaluators while maintaining high scientific rigor, facilitating the widespread implementation of competency-based assessments in resource-constrained training environments.

The robust psychometric properties demonstrated in this study are accompanied by methodological limitations that warrant careful interpretation of the findings. The interrater reliability testing phase was conducted with a relatively modest sample size of 36 respondents evaluated by only two trained raters. A larger and more diverse sample of evaluatees might introduce greater variability in performance execution, potentially challenging the universal applicability of the current reliability coefficients. The assessment took place in a controlled training environment, which inherently lacks the extreme physiological stress, environmental distractions, and chaotic dynamics characteristic of actual mass casualty incidents. Rater performance in a calm, simulated setting may not perfectly predict rater consistency under the high-pressure conditions of a real emergency. The exclusive reliance on a dichotomous scoring format, while excellent for reducing ambiguity, fails to capture the quality, speed, or fluidity of the psychomotor execution. Future investigations must address these constraints to ensure the instrument's utility extends beyond controlled academic settings into the unpredictable reality of disaster response.

Subsequent research endeavors should prioritize expanding this validation framework to encompass broader demographic samples and more complex simulation paradigms. Future studies must involve larger cohorts of healthcare professionals across diverse clinical settings to establish the observation sheet's generalizability. Incorporating high-fidelity, real-time mass-casualty simulations with standardized patients or advanced manikins will test the instrument's resilience under realistic stress conditions. Researchers should also explore integrating video-based remote assessment protocols to evaluate interrater reliability when evaluators are not physically present in the simulation room. Investigating the correlation between scores obtained from this psychomotor observation sheet and actual patient outcomes in simulated scenarios would provide additional criterion-related validity evidence. Longitudinal studies tracking the psychomotor retention of healthcare professionals over time using this specific instrument could yield valuable insights into the decay rates of triage skills. These progressive research steps will ultimately refine the tool and solidify its position as a gold standard for disaster triage competency assessment.

The newly developed START triage psychomotor observation sheet exhibits exceptional psychometric robustness, characterized by comprehensive expert consensus and outstanding interrater consistency. All evaluated items achieved maximum content validity and superior agreement across all behavioral indicators. This rigorous validation confirms the instrument's suitability for objectively measuring clinical execution and procedural adherence among healthcare providers during mass-casualty preparedness exercises. Educational institutions and emergency training coordinators can confidently deploy this standardized tool for competency assessment and simulation-based learning. Future investigations must expand the validation framework to encompass larger, more diverse clinical cohorts and high-fidelity disaster scenarios to establish broader generalizability.