Reliability and Validity of the Red Flag App: Encouraging Inclusive Education for Children at Risk in the School System

Early identification of ‘risk for disability and mental illness’ has many advantages including better prognosis, and an improved quality of life. Teachers and school management spend long contact hours with children, and are typically among the first to notice developmental delays or behavioural changes. Although they are uniquely poised to identify risks, this often does not occur due to lack of teacher training. In India, there is a dearth of trained therapists for children with special needs. Additionally, many teachers have not been adequately trained in identifying and helping children with mental health issues, neurodevelopmental disabilities, or learning disabilities. The current study seeks to validate an online app which can aid teachers in identifying children at risk across seven areas namely Attention Deficit Hyperactivity Disorder (ADHD), Autism Spectrum Disorder issues (ASD), abuse issues, intellectual deficits, mental health issues, specific learning disorders (SLD), and visual deficits. These buckets have been identified through expert focus group discussions and appropriate literature reviews. The current paper describes the necessity and relevance of such an instrument, and outlines the validation process being conducted.


Introduction
Children are among the world's most vulnerable populations. They are completely dependent on caregivers for nurturance and protection, and are also susceptible to various health issues. In India, the pediatric healthcare burden is extremely high. Out of the approximately 1 billion children globally at risk for disabilities, more than 100 million are in India (Gavi, 2016). Disability is a wide term that encompasses physical ailments, neurodevelopmental disabilities, and mental health disorders. Statistically speaking, nearly 2.4 million children in India are at risk for neurodevelopmental disorders such as Autism Spectrum Disorder (ASD) and Attention Deficit Hyperactivity Disorder (ADHD) (Gavi, 2016). Similarly, community level studies suggest that the prevalence of child and adolescent mental health disorders is around 12.5% to 16.5% (Hussain and Purohit, 2019). Children in India are also at an increased risk for abuse, this is due in part to the patriarchal society, and cultural beliefs regarding what are acceptable child rearing practises. Although prevalence rates vary greatly, it is suggested that around 8-31% of children experience sexual abuse, and even more experience physical abuse or negelct (Choudhary et al., 2018). The incidence of intellectual difficulties and specific learning disorders is also very high.
Many childhood disorders are chronic in nature and cause significant dysfunction (American Psychiatric Association, 2013). Children with disability are at an increased risk for various negative life events such as high school dropout rates, severe learning crisis, high unemployment rates, criminal activities and poverty (Rohwerder, 2015). In fact, UNICEF estimates that in developing countries. Nearly 90% of children with disabilities do not stay in school (UNICEF, 2014). Such risks jeopardise the future of children with disabilities and hinder their progress in academics, interpersonal relationships, and daily quality of lie, eventually result in their exclusion from society and workplace (Rohwerder, 2015).
Early identification of disability can help mitigate many of the risk factors mentioned above. Early identification allows the child to receive the support they need, which in turn facilitates better development. However, as of now India is severely lacking in the ability to identify children at risk. Research suggests that only 10% of Indian children are currently being accurately diagnosed-and even less are getting the required intervention (Gururaj et al., 2016). This can be explained in part by the dearth of pediatric health care professionals-India has a psychiatrist to client ratio of 1 for every million (Mishra & Galhotra., 2018), and only 5000 practicing Occupational Therapist as of 2016 (World Federation of Occupational Therapists, 2016)-but it is also contingent on the lack of mental health awareness in primary caregivers and educators.
Arguably, educators spend a significant amount of time with children, coming second only to the primary caregivers. This puts them in the unique position of observing and interacting with children in a structured environment, with other similar aged children nearby. Such an advantage means that educators are often the first to notice developmental delays, or social difficulties in children (Soneson, 2020). However, a lack of awareness and teacher training regarding children with special needs (CWSN) results in these observations being brushed off as 'circumstantial'. A study conducted in India found that only 9% of teachers had received International Journal of Education ISSN 1948-5476 2021 prior training on ADHD management (Shetty and Rai, 2014). Other research has highlighted that 85.5% of teachers have never undergone any special training programmes on learning disabilities. 69.5% of them had never come across any children with learning disabilities in their classrooms, and only 30.5% of the them were able to identify learning disabilities in their students (Moothedath & Vranda, 2015). Mental health literacy is also quite low-more than 80% of teachers are NOT aware of mental health challenges. These challenges result in only 10% of children at risk being identified, and even less receiving the help they require. This could result in 50 million children not receiving appropriate schooling in India by 2025 (Singhal, 2009). To make matters worse, the ratio of educators to students is around only 1 teacher for every 182 children. While this is a huge imbalance even for neurotypical children, the desired requirement for teachers for Children with Special Needs (CWSN) is "1 teacher for every 8 children" (Singhal, 2009).
Despite the lack of awareness, studies have suggested that teachers are already attune to the needs of children. Mathews et al. (2020) found that teacher concern was an important predictor of poor mental health in children. The study found that a teacher's concern for a child's wellbeing was moderately predictive of the presence of a mental health disorder. Further, lack of teacher concern was highly predictive of the absence of mental health issues. These results indicate the extent of the ability teacher's possess, and reiterate the importance of providing them with appropriate screening tools that could aid empirical identification and referral to specialist mental health services (Mathews et al., 2020).
Adequate teacher training can greatly improve the health outcomes for children. Short term teacher training programmes implemented in developed countries such as the UK have shown a lot of promise for the mental health of children, especially for those children who are already struggling (Ford et al., 2018). The emergence and growth of teletherapy can aid modify such programmes to better fit the requirements of developing countries. Teletherapy can be defined as providing health care at a distance. It makes use of technology such as smartphones, videocalling services, and high speed internet to provide expert medical care to anyone who might need it. Telehealth has made it possible for all sections of the population to access healthcare services at affordable costs. A systematic meta-analysis indicated that for OT especially, telehealth has been found to be as advantageous, if not more, than regular face-to-face therapy. The major advantages include ease of implementation, reduced costs for all stakeholders, and a wide reach. The benefits of telehealth have the potential to make school based riskidentification a substantially smoother process, and increased research in this realm is the need of the hour (Hung and Fong, 2018).
The benefits of early identification extend to all spheres of a child's life, but are especially apparent in the classroom. Early identification facilitates appropriate intervention which in turn can aid improve concentration, motivation, and achievement in the class. It also helps with socialization. Having adequate social skills and relationships in the classroom can help enhance the child's self confidence, sense of belonging, and foster a strong support system, all of which are strong protective factors against future risk. Behavioural issues such as acting out, or aggression can also be mitigated before they become causes of concern (Koegel et al., 2013; Doncheva, 2018). ISSN 1948-5476 2021 Children who face risks are often likely to face not one but multiple risks at a time and therefore need to be screened & identified holistically. For instance, a child with a disability might have faced abuse or might have mental health issues but the current single parameterbased screening and diagnosis ignores the interconnectedness of such risks. 91% of children affected by multiple risks experience school failure (Rohwerder, 2015). The need of the hour is to have a screening tool to identify the multiple risks children face and equip all stakeholders to address them immediately.

International Journal of Education
The current study seeks to validate a risk assessment tool as the beginning of a holistic endeavour undertaken in collaboration with the government to improve the educational inclusion of students at risk. The process can be divided into 5 stages namely Screen, Learn, Act, Track, and Enable. The first stage 'screen' is outlined in the current paper, and aims to validate a web-based risk assessment survey to screen for children who are at risk of disability, abuse, or mental health issues. The second stage 'Learn' seeks to provide comprehensive teacher training in order to equip educators in identifying and managing children at risk in the classroom. The third stage 'Act' aims to connect such children and their families with various beneficial government schemes. The fourth stage 'Track' plans to map the implementation of schemes and policies by various stakeholders. Lastly, the fifth stage 'Enable' seeks to create a better future for children at risk by empowering all stakeholders. The primary objective of this study is to design and validate a questionnaire that will aid teachers in screening children who might be at risk for various forms of disability.

Method
This study will employ a mixed-methods design to construct and validate a questionnaire on the status of teacher identification of risk in the classrooms. Our work will have four sequential phases: (1) item generation through review of the evidence and experts' opinions, (2) evaluating the face and content validity of the questionnaire to gain consensus regarding the relevancy and clarity of items, (3) determination of a range of possible scenarios for each item of the questionnaire and (4) evaluation of reliability.

Survey Development
Item generation for the development of the questionnaire was done through interviews with experts and a thorough literature review.
Phase I -Qualitative Phase/Development of the instrument This phase included two stages: (1) Development of an item pool; (2) Item reduction and development of the app based tool.

Procedure and Participants
A qualitative approach using focus groups was adopted in this study and aimed to explore experts' perspectives on the factors that influence the ability of teachers, school management, International Journal of Education ISSN 1948-5476 2021 and rehabilitation professionals to identify children at risk in the classroom. Purposive sampling was conducted between October 2020 and January 2021. A total of 20 participants were recruited for each focus group which included experts, teachers, resource teachers, directors of organisations working with people with disabilities, and individuals with lived experience of disability. Participants completed a semi-structured interview. The following topics guided the interviews: (1) what is the current knowledge and skills teachers in the school system have in relation to different disabilities that might impact learning. (2) Considering your training experiences as experts, teachers, resource teachers, directors of the organisations working with individuals with disabilities and people with lived experience what factors facilitate and hinder the application of what has been learned during the training process.
The data was analysed using qualitative content analysis. The model of the transfer process presented by Baldwin and Ford was used to guide the analysis which could inform the initial direction of qualitative data analysis without limiting the identification of new themes. The above qualitative findings provided directions for developing a practical screening tool to identify knowledge and skill teachers in the school system have in relation to identification of different disabilities that might impact learning. The themes that emerged from the qualitative data were used to build on the related constructs, which have provided the basis for the development of scale items. Statements included in the survey were written in clear, understandable and unambiguous language. In order to minimize the possibility of set responses, items were written in both positive and negative directions. The qualitative data analysis resulted in the emergence of 7 themes, namely ADHD, ASD issues, abuse issues, intellectual deficits, mental health issues, specific learning disorders, and visual deficits.

Procedure and Participants
The items were divided into 7 buckets-one section for each area that might prevent induction based on the earlier focus group data. Each section consisted of 10-12 questions, and had between 10-12 experts review the items.
The criteria used by the experts for retention or deletion of items included relevance, ambiguity, and difficulty of understanding. Each expert offered comments and suggestions regarding items they thought should be added, removed or modified. They also evaluated the level of relevance of each item to its corresponding construct on a 5-point scale. The suggestions provided by the experts were reviewed and a modified list was sent back to them. Once all the changes were incorporated, 2 experts reviewed the questions one final time. Based on these themes and the literature review a 90 item pool was generated. Through expert review and discussion this was streamlined and reduced to 74 items across all seven buckets.

Data Analysis
The given dataset consists of five point Likert scale data. The dependent variable is measured International Journal of Education ISSN 1948-5476 2021 on ordinal scale i.e., Strongly disagree (1), disagree (2), neutral (3), agree (4) and strongly agree (5) and the independent variable consists of two categorical independent or distinct groups namely male and female. There is no relationship between each group of the independent variable. These points satisfy the assumptions of the Mann Whitney U test or T-test and make the dataset appropriate to apply the test along with descriptive statistics. SPSS software was used to perform analysis.
The reliability (R); content validity index for items (I-CVI); scale-level content validity index, universal agreement calculation (S-CVI/UA); scale-level content validity index, averaging calculation method (S-CVI/Ave) were computed to indicate the content validity.
Prior to the calculation of the content validity index (CVI), the relevance rating was re-coded as 1 (likert scale of 4 or 5) or 0 (likert scale of 1 or 2 or 3), Universal agreement (UA) score was given as 1 when the item achieved 100% agreement between experts. If this was not achieved then the UA score was given as 0.
Following formulas were used to compute S-CVI/Ave and S-CVI/UA S-CVI/Ave = (sum of I-CVI scores)/(number of item)

S-CVI/UA = (sum of UA scores)/(number of item)
A reliability test was also conducted to measure the consistency in the rating by the different ratters. If Cronbach's Alpha test results lie between 0.5 to 1 it indicates a high positive correlation.
The table below (table 1) gives the bucket wise content validity. From these results, we can conclude that the buckets having S-CVI/UA ≥ 0.8 (ADHD, Autism Issues, Visual Deficits) and a S-CVI/Ave ≥ 0.9 (ADHD, Autism Issues, Intellectual Deficits, Specific Learning Disorders, and Visual Deficits) have high content validity/agreeability. The remaining buckets i.e. the Abuse Issues and Mental Health Issues buckets need to be revised. ISSN 1948-5476 2021 The Reliability (Cronbach's alpha) test indicates that agreeability among the raters was highly correlated. The psychometric properties of the final app based instrument will be assessed in a sample of 1000 teachers.

Discussion
The primary goal of this study is to validate items that can be used to develop a questionnaire that has utility for identifying risk in classrooms. Based on the initial data collected through Focus Group Discussions, for this study risk has been divided into seven areas of risk, namely ADHD, ASD issues, abuse issues, intellectual deficits, mental health issues, specific learning disorders, and visual deficits.
The necessity for such a screening tool is well established. Community level studies of psychological health in India have reported that between 12.5% to 16.5% children and adolescents suffer from varying mental health disorders. The prevalence of such disorders is nearly double in urban children as compared to children living in rural areas (Hussain and Purohit, 2019). Although there is greater availability of healthcare services in urban areas, their accessibility is severely mitigated by socioeconomic status (SES). SES is an important factor that influences health, nutrition, and mortality of a population. It is also directly linked to acceptability, affordability, and appropriate utilization of health facilities. Individuals belonging to a lower socioeconomic class find it much harder to access the same quality of basic healthcare as more affluent families (Wani et al., 2019). However, the recent growth of telehealth options has the potential to bridge this gap and improve health outcomes for all populations.
The current feasibility study found that the proposed items were significantly able to identify 'at risk' children across most of the risk areas the implications of this are multifold. Individuals from low income households tend to have low health literacy, and are less likely to seek preventive health care. This includes routine health checks which are often instrumental in diagnosing chronic disorders (Fleary et al., 2013). The validation of a low cost, teacher-report survey creates the opportunity for children who would not typically have access to advanced health care, to receive preliminary screening, and to be connected to appropriate government programmes as needed. Such early assessment can have many benefits as has been supported by research that reports that the early diagnosis of Neurodevelopmental Disorders such as ASD can greatly improve the prognosis as well as the quality of life of the individual (Fernell et al., 2013). In a similar vein, early identification of mental health disorders such as anxiety and depression, can greatly improve the health, behavioural and educational outcomes of individuals (Mental Health America, 2016).
The growth of smartphones and telehealth has helped increase the availability of screening tools for low-resource areas. For example, Peek Acuity is an android app developed by the London School of Hygiene and Tropical Medicine and the International Centre for Eye Health, that can be used to ascertain one's visual acuity. Similarly, voice tests are being encouraged by the WHO as cost-effective means of measuring auditory ability. However, these tools are International Journal of Education ISSN 1948-5476 2021 limited to 'overt' types of disability. Open access tools that can measure cognitive or motor function are hard to come by (Hatch et al., 2018). Additionally, there is also a dire need for screening tools that assess the more psychosocial aspects of 'risk'-such as abuse. A metaanalysis of 2280 articles relating to child physical, sexual or emotional abuse and neglect, concluded that as of 2012 the quality and accuracy of tools being used to identify abused children was incredibly low, and that the development of a valid screening instrument was an urgent prerequisite (Bailhache et al., 2013).
The instrument studied in this paper seeks to fill this important gap in risk assessment. The current feasibility study was able to validate items useful for identifying risk across 7 areas of risk (ADHD, ASD issues, abuse issues, intellectual deficits, mental health issues, specific learning disorders, and visual deficits) which include both observable and more 'hidden' risk factors. The creation of one tool which can accurately assess a variety of factors has multiple positive implications for low resource communities where both time and manpower are in short supply.
Risk assessment tools form an important aspect of preventive medicine. Such tools identify individuals at risk early enough for appropriate interventions to be provided, and help mitigate long term consequences. Additionally, interventions targeting disability in its early stages can be significantly less expensive, making them an important aspect of healthcare systems in less developed countries (Iragorri and Spackman, 2018). The care of children with serious disabilities both mental or physical often requires large public expenditure. However, Research has found that the successful identification of just one high risk child can save society between 1.7 -2.3 million dollars. Arguably, this figure is on the lower end of things after taking into account the benefits to the immediate family, and factors such as inflation (Jones et al., 2002). Early intervention is also profitable in terms of increased productivity and human potential. It also helps break the vicious cycle of poverty and disability. Coupling the benefits of early identification for both the individual and society, with the relative inexpensiveness and wide scope of telehealth, it would seem almost irresponsible to not focus on its widespread adoption (Rohwerder, 2015).
The present study marks the first phase of a multi-stage endeavour. It seeks to validate a webbased instrument that can be used by educators to identify children at risk across areas such as ADHD, ASD issues, abuse issues, intellectual deficits, mental health issues, specific learning disorders, and visual deficits. The tool has good validity, and is cost effective, making it a feasible instrument to be used in low income areas. It has utility as a screening device for children who would not otherwise be able to access advanced medicine, and can be used as the basis for referrals to specialists.