Abstract

Objective

A unique and comprehensive self-report scale to help assess sleep in Turkish is needed. This study aimed to develop the Troy Sleep Scale (TSS) and perform validity and reliability analyses.

Materials and Methods

The study was conducted with 494 medical students and residents. The psychometric properties of the devised scale were evaluated through exploratory and confirmatory factor analyses, Cronbach’s alpha and McDonald’s omega reliability evaluations, and item analyses based on both classical test theory and item response theory (IRT). The reliability assessment of the TSS was carried out through test-retest, and criterion-referenced validity was employed to provide additional validation.

Results

The TSS consists of 11 items with a 3-factor structure with eigenvalues greater than 2. The factors provided information about (I) dysfunction due to sleep problems, (II) affective symptoms due to sleep problems, and (III) sleep quality. The reliability levels of all three sub-factors and overall scale scores were 0.82 and above. The test-retest reliability level had a correlation value of >0.80. The item analyses conducted in line with the IRT showed that the scale items provided a high level of information and functioned together with the answer set. The correlation between the TSS and the Pittsburgh Sleep Quality Index was analyzed for criterion-referenced validity, yielding a correlation of 0.73.

Conclusion

The psychometric properties of the TSS indicated that it was valid and reliable, making it suitable for use as a screening test for individuals with sleep-related complaints.

Introduction

Sleep is a fundamental requirement for human life. Regular and consistent sleep is crucial for maintaining overall health and well-being. Sleep is a dynamic process involving intricate neurochemical activity characterized by periodic transitions between different sleep stages. The structure and duration of sleep are influenced by various factors such as age and genetic predisposition.¹ Insomnia, a common symptom, often disappears when the underlying cause is addressed. Chronic insomnia is believed to affect approximately 10% of the population.^{2, 3} Insomnia is generally defined as the inability to initiate and maintain sleep which impairs sleep quality and integrity despite favorable conditions.⁴ The negative impact of poor sleep quality is evident during the day, causing disability, impairment of cognitive and emotional functions, and behavioral problems. Sleep deprivation and fatigue often lead to daytime sleepiness, mood disturbances, attention and memory impairment, and an increased likelihood of accidents and errors.⁵ Sleep problems are a common health problem in university students, insomnia is seen at a higher rate than in the general population in systemic reviews (18.5% vs. 7.5%), and sleep problems cause medical problems and stress in students and negatively affect their academic performance.^6-9According to subjective experiences, sleep quality can be defined as satisfaction with the previous night’s sleep and satisfactory wakefulness the next day. However, when subjective sleep experiences and evaluations cannot be measured or expressed numerically, they only have descriptive features. Scientific research can be conducted using numerical data obtained with valid and reliable measurement tools, and measurements made with sensitive measurement tools make the research valuable. For this purpose, scales facilitate the measurement of variable characteristics and determine the quality of the results.¹⁰ Objective measurements such as polysomnography and actigraphy are costly and challenging for patients with sleep-related complaints.¹¹ Self-report methods, such as sleep diaries, sleep recording, and sleep scales, are commonly used to assess sleep quality, although they do not offer information about sleep structure. These methods attempt to gauge both the quantitative and qualitative aspects of sleep. The components and importance of sleep quality differ among individuals. Therefore, a self-report-based assessment is necessary to measure sleep quality.¹² In addition, self-report scales have many advantages, such as being inexpensive, practical, and quick to administer. Many scales have been developed to assess sleep problems and are used in Turkey (see review).¹³ Most scales for sleep problems originate from abroad, and Turkish validity and reliability studies have been conducted.^14-19 In Turkey, there is a need for a local and comprehensive self-report scale to help distinguish individuals with sleep problems from healthy individuals and to determine sleep problems qualitatively and quantitatively. Therefore, this study aimed to develop a comprehensive and reliable self-report scale, the Troy Sleep Scale (TSS), to assess adult sleep quality.

Materials and Methods

Participants

The study was conducted between January 2022 and August 2022 with students studying at the Çanakkale Onsekiz Mart University Faculty of Medicine during the 2021-2022 academic year. The data were obtained from 494 medical students. The mean age of the students was 21.4±3.0 years. Of the participating students, 181 (36.6%) were male and 313 (63.4%) were female. Eighty-seven (17.6%) were class 1.144 (29.1%) were class 2.63 (12.8%) were class 3.40 (8.1%) were class 4.94 (19%) were class 5.35 (7.1%) were class 6 (intern doctors), and 31 (6.3%) were residents. Data from 494 medical students were randomly divided into two groups. Hair et al.²⁰stated that exploratory factor analysis (EFA) results should be validated in a split sample from the original dataset or a separate sample obtained with a new application. The larger group comprised approximately 59% of the data (n=289) and was utilized for EFA, Cronbach’s alpha, and McDonald’s reliability coefficient calculations. The remaining 197 medical students’ data were employed for confirmatory factor analysis (CFA). All 494 students’ data were utilized for item response theory (IRT).

Process

Drafting of the Troy Sleep Scale

The TSS draft form was prepared as follows: (I) The scale aimed to evaluate adult sleep. (II) It was decided to use 5-degree Likert-type items (never, rarely, sometimes, most of the time, and always) as the answer set for the scale. Studies show that the options for the 5-point Likert structure work well.²¹ (III) A literature review was conducted on scales assessing sleep, and candidate items were created by reviewing the relevant scales. (IV) The candidate item pool was presented to four expert psychiatrists working on sleep and one expert academician for measurement and evaluation. The experts kept items deemed appropriate on the trial form and removed those deemed inappropriate. While there were 29 items in the item pool, 23 remained after expert evaluation. (V) For Turkish language comprehensibility and plain expression, the opinion of an expert in Turkish was considered. The Turkish language expert evaluated the remaining 23 items and suggested ensuring more straightforward expressions. The items were revised in line with the recommendations of the linguist. Following the abovementioned process, a draft trial form consisting of 23 Likert-type items with a 5-point response set was obtained. Ten students participated in a pilot study using the draft form. This pilot study evaluated whether the scale items and expressions were understandable for the target group. The pilot study showed that the items and expressions were understandable.

Several scales have been developed to evaluate sleep quality, with the Pittsburgh Sleep Quality Index (PSQI) being the most commonly used.²² The PSQI, which comprises 19 questions and seven dimensions, assesses subjective sleep quality, sleep latency, sleep duration, sleep efficiency, sleep disturbance, sleep medication use, and daytime dysfunction. A validity and reliability study of the PSQI was conducted by Ağargün et al.¹⁴ in Turkey, highlighting potential limitations of the scale, such as subjective interpretations affecting scores for dimensions like sleep duration, latency, and efficiency. In this study, the PSQI was administered to all participants.

After the Development of the Draft Scale

The research was completed as follows. The approval of Çanakkale Onsekiz Mart University Clinical Research Ethics Committee (approval number: 2021-07, date: 20.10.2021) was obtained. Before the study, a consent form was obtained from all participants, indicating their agreement to participate. A draft form of the TSS was also prepared, and a trial application was carried out to obtain validity and reliability evidence of the scale. The trial applications were conducted face-to-face. To assess the validity of the TSS, item-total correlations were examined based on the classical test theory (CTT), which constructs validity within the scope of factor analysis and CFA. The scale’s validity was further assessed through item discrimination and difficulty levels, item characteristic curves, and item and test information functions based on the IRT. The scale’s reliability was examined using Cronbach’s alpha internal consistency coefficient, McDonald’s composite reliability coefficient, and marginal reliability coefficients.

Statistical Analysis

Data File Preparation Phase

The study involved analyzing forms completed by participants. It was observed that all data were present on the participant’s forms. The data were subsequently transferred to JAMOVI and R statistical software. The “mvn” package of R was utilized to examine the multivariate normal distribution of the 23 items using the “henze-zirkler,” “mardia,” and “doornik-hansen” methods.²³ The Kaiser-Meyer-Olkin (KMO) test and Bartlett’s test of sphericity were employed to analyze the data. The KMO value was interpreted based on the reference values provided. Specifically, values above 0.89 (0.90 to 1.00) were considered “excellent,” values between 0.80 and 0.89 were deemed “good,” values between 0.70 and 0.79 were labeled “acceptable,” values between 0.60 and 0.69 were categorized as “moderate,” values between 0.50 and 0.59 were classified as “low level,” and values below 0.50 were deemed “unacceptable”.²⁴ In Bartlett’s test of sphericity, the hypothesis H₀ was evaluated to determine whether the correlation matrix was a unit matrix, indicating the absence of relationships between the items.

Factor Identification with Exploratory Factor Analysis (EFA)

Factor analyses were performed utilizing the principal axis factoring (PAF) methodology. PAF analyzes the shared variance among the measures instead of the error sources unique to each one. PAF, commonly used in social and behavioral science research, models the shared variance in a series of X-measures.²⁵ In determining the number of EFA factors, the reference value of “eigenvalue” was accepted as “1”. A meaningful discussion in EFA involves assessing the extent to which the total variance (variability) of the characteristic being measured through the scale is explained. The structure obtained from factor analysis is based on Hair et al.²⁰ who states that in social sciences, where information is generally less precise, a solution that explains 60% of the total variance is acceptable. Warner²⁵ suggests that the acceptable limits are between 40% and 70%. In interpreting the factor structure, the minimum values for factor loading should be between ± 0.30 and ± 0.40. A factor-loading value of ± 0.50 is important, while ± 0.70 and above is indicative of a well-defined structure.²⁰

Determination of Reliability

Cronbach’s alpha, McDonald’s omega, and marginal reliability coefficients were used to determine the scale’s reliability. A test-retest application was performed to determine the reliability level. Sixty-six participants were administered the scale again one month later. The level of consistency was examined using a correlation analysis between the scores of the first and last applications. The test-retest administration assessed the consistency of the measurement tool, whether it gave stable results over time, and provided evidence of reliability. Adequate reliability should be 0.70 and above, according to Nunnally and Bernstein²⁶.

Confirmatory Factor Analysis (CFA)

In CFA analysis, the degree of model fit is important. By looking at the fit values, the suitability of the model can be determined. The literature suggests that reference values for fit indexes determined for CFA are acceptable for 0.05< root mean square error of approximation (RMSEA) ≤0.08 and excellent for 0>RMSEA≤0.05; a good fit for a Tucker-Lewis index (TLI) is 0.95 and above; and an acceptable fit for a comparative fit index (CFI) is 2< X²/standard deviation (sd)≤5 and a good fit is 0>X²/sd≤2.^27-34

Data Analyses with Item Response Theory (IRT)

IRT-based measurement tools offer several advantages, including the independence of item parameters and group characteristics. Additionally, IRT allows for unique standard error estimates for each participant. In IRT analyses, item parameters are independent of the participant group, and group characteristics are independent of the item sample.¹⁶ Furthermore, standard errors can be estimated separately by analyzing the test results for each respondent. In this context, a standard framework for evaluating their ability according to IRT can be revealed, even if the respondents are tested with different questions.³⁵The validity and reliability analyses using the IRT should include an examination of the unidimensionality and local independence assumptions.³⁶ Unidimensionality requires that only one characteristic is assessed (the relevant items of the measurement tool are only for one characteristic), which affects the performance of individuals on the measurement tool.³⁷ An item correlation matrix or EFA can be used to assess unidimensionality. This study used EFA to analyze unidimensionality, as explained in detail in the results section. According to the EFA results, the TSS had three subdimensions. In this case, each factor was accepted as unidimensional separately, and IRT analyses were performed accordingly. The assumption of local independence was tested using the Q3 statistic³⁸, and IRT calibrations were performed using the “Mirt v.1.30” package within the R v.4.1.2 program.³⁹ In the IRT, the discrimination value of an ideal scale item (i.e., “a” parameter) should be between 0.5 and 2. The parameter is generally within the acceptable range of 0.75 to 2.50 in the literature.⁴⁰ The ideal (medium difficulty level) limits for item difficulty levels (i.e., “b” parameter) are accepted between -1.00 and 1.00.⁴¹ In aptitude or achievement tests, items with difficulty levels lower than -1.00 are considered easy, while items with difficulty levels higher than 1.00 are considered difficult.⁴² The item information function is a graphical representation that illustrates the range of the trait (the trait being measured on the scale) in which the item best distinguishes individuals taking the measurement tool.⁴³

Results

CTT Validity Evidence of the Troy Sleep Scale

Construct Validity (EFA)

The normal distribution of the 23 items on the scale was analyzed. Multivariate normal distribution examination revealed a multivariate normal distribution (p>0.05). The KMO value was calculated as 0.89. Bartlett’s sphericity test value was calculated as 2040, and the result was significant (df=55, p<0.05). As explained in the data analysis section, these results demonstrate that the data are suitable for factor analysis. PAF was adopted for the EFA. According to the data of the 23 items in the EFA data file, item-total correlation values and factor analysis input loadings were analyzed. Twelve items (3, 4, 6, 7, 8, 10, 11, 12, 17, 18, 19, and 20) with initial, extraction, and item-total correlation values below 0.30 were identified. These items were removed from the scale. The results are presented in Table 1. The initial, extraction, and corrected item-total correlation values of the remaining 11 items on the TSS were between 0.45 and 0.79. As explained in the data analysis section, these values are within the ranges recommended in the literature for EFA. Eigenvalue and scree plot analyses were performed to determine the number of factors the TSS showed with the remaining 11 items. According to the eigenvalue data, these three factors had values greater than 1. The eigenvalue of the first factor was 2.66, and the explained variance was 24.2%. The eigenvalue of the second factor was 2.38, and the variance explained by it was 21.7%. The eigenvalue of the third factor was 2.24, and the explained variance was 20.4%. Together, the three factors explain 66.2% of sleep features. As described in the Data Analysis section, this value has been accepted in the literature. The scree plot obtained from the TSS confirmed a three-factor structure (Figure 1). The scree plot shows that the TSS had three factors, all of which had eigenvalues greater than 1. Based on these results, the scale was determined to have a three-factor structure. Axis rotation was performed to determine the factors in which the 11 items of the TSS were located. As three factors were determined to be related, oblimin rotation was performed. The items for these factors are listed in Table 2. Four items were grouped under factor 1, three under factor 2, and four under factor 3. After the rotation process, the factor load values ranged from 0.44 to 0.97. The items under these factors were analyzed, and the factors were named. Factor 1 consisted of the 9^th, 14^th, 15^th, and 16^th items and was named Dysfunction Due to Sleep Disorder (DSD) (minimum 4, maximum 20 points). Factor 2 consisted of items 21, 22, and 23 and was labeled Affective Symptoms Due to Sleep Disorder (ASD) (minimum 3, maximum 15 points). Factor 3 consisted of items 1, 2, 5, and 13 and was called sleep quality (SQ) (minimum 4, maximum 20 points). No reverse-scored items were used for these factors.

Construct Validity (CFA)

The analyses revealed a 3-factor structure for the TSS, which consisted of 11 Likert-type items. The accuracy of this structure was analyzed using CFA. The diagram obtained from the analysis of the CFA data file is shown in Figure 2. When the diagram obtained from the CFA was examined, the 5^th item under the 3^rd factor had the lowest correlation (0.52). The highest correlation value was the 22nd item under the 2nd factor. As a result of the CFA, the fit indices were calculated as X²/sd=1.98, CFI=0.98, TLI=0.97, SRMR=0.04, and RMSEA=0.06. In the DFA analysis, a covariance link was established by modifying the error terms of items 1-2 and 15-16. In this case, a correlation between the error terms of items 1 and 2 as well as 15 and 16 was accepted. Creating a covariance modification between the unknown parts of these items also showed that there were some semantic links between the items.

Reliability of Troy Sleep Scale

The reliability of the TSS was analyzed using Cronbach’s alpha, McDonald’s omega, and the marginal reliability coefficients. The 11 items were subjected to reliability analyses as a single-factor scale with three separate factors. Cronbach’s alpha reliability value of the first factor was 0.89, McDonald’s omega reliability value was 0.89, and the marginal reliability value was 0.88. The Cronbach’s alpha reliability value of the second factor was 0.88, the McDonald’s omega reliability value was 0.89, and the marginal reliability value was 0.86. The Cronbach’s alpha reliability value of the third factor was 0.82, McDonald’s omega reliability value was 0.83, and the marginal reliability value was 0.89. The Cronbach’s alpha reliability value of the TSS was 0.91 and the McDonald’s omega reliability value was 0.92. The values obtained from the analyses of the reliability levels of the scale are 0.70 and above and had sufficient reliability.¹⁵ A test-retest procedure was performed to determine the reliability level. Sixty-six participants were administered the scale again one month later. The level of consistency was examined by comparing the scores of the test and retest administrations. Table 3 presents the correlation results. The first-factor correlation of the scale, which was administered to 66 participants with an interval of one month and whose structure was discovered according to EFA, was 0.81, the second-factor correlation was 0.80, and the third-factor correlation was 0.86. For the 11-item total score, the correlation between the data from the first and second administration was analyzed, and a correlation value of 0.88 was obtained. The correlation values obtained correspond to high reliability.

Validity Evidence for the Troy Sleep Scale

Factor 1 IRT Validity Evidence

IRT analyses were conducted using data from 494 students. The generalized partial credit model (GPCM) calibrated the IRT factor item. The GPCM calculated S\_c² (degrees of freedom), RMSEA, level of significance, “a” (item discrimination) and “b” (item difficulty) parameters, and standard errors for each item separately (Table 4). According to the GPCM, the RMSEA threshold value for acceptable item fit is 0.08. With a value below this threshold, an item is considered to fit the model. The item fit statistics revealed that all items had RMSEA values below 0.05, indicating that the provided model fit according to the GPCM. The discrimination value, or “a” parameter, of an ideal scale item in IRT should be between 0.5 and 2. In the literature, the range for this parameter is between 0.75 and 2.50, which is considered acceptable. Based on the GPCM estimations (log-likelihood, p<0.05), the compatibility of the items was confirmed. The item characteristic curves depicted that Items 9, 14, 15, and 16 worked well with their options in the response set. The item information function is a graphical representation that illustrates the range of the feature measured in the scale. The higher the peak of the curve in the item information function, the more information an item provides. Items 15 and 16 in Factor 1 were the most informative, while Item 9 provided relatively less information than the other items.

Factor 2 IRT Validity Evidence

According to the item fit statistics, all the items’ RMSEA values were less than 0.05. Based on this result, it was determined that the three items of Factor 2 provided an adequate fit according to the generalizability theory (GPCM). The discrimination levels of Items 21 and 23 were optimal. The estimations based on the GPCM (log-likelihood ratio, p<0.05) confirmed the fit of the items. The item characteristic curves revealed that all options for items 21, 22, and 23 were functional. Upon analyzing the item information functions of Factor 2 items, it was found that Item 22 was the most informative. Item 23 provided relatively less information than the other items (Table 4).

Factor 3 IRT Validity Evidence

Furthermore, the RMSEA values for all the items in Factor 3 were also less than 0.05. Consequently, it was concluded that the four items of Factor 3 provided an adequate fit according to the GPCM. Items 2, 5, and 13 discrimination levels for Factor 3 were ideal. The GPCM (log-likelihood ratio, p<0.05) demonstrated the fit of the measurement tool items. The item characteristic curve of Item 13 revealed that the “most of the time” option was less effective than the other options. The remaining items functioned effectively with their respective options. Upon analyzing the item information functions of Factor 3 items, it was observed that Items 1, 2, and 5 were the most informative. Item 13 provided relatively less information than the other items (Table 4).

Concurrent Validity

The correlation between the scores obtained by the participants from the sleep assessment scale developed in this study and the scores obtained by the same participants from the PSQI was 0.73. In line with these results, the obtained correlation value corresponded to a robust and high-level correlation.^44-46 In this case, the concurrent validity level of the sleep assessment scale developed in this study was high. The Turkish form of the TSS is presented in Appendix 1.

Discussion

In this study, the TSS scale was developed to assess subjective sleep problems and their consequences and its psychometric properties were analyzed. The results showed that this scale is valid and reliable for medical students in the Turkish population. The 5-point Likert scale consists of 11 items, and the construct validity analyses in the study showed that the scale had 3 factors. When the factors were analyzed, it was determined that the items of the first factor were related to “DSD” the items of the second factor were related to “ASD” and the items of the third factor was related to “ (SQ; problems with initiating and maintaining sleep)”. The CFA confirmed this structure. In addition, to evaluate its criterion-based validity, its correlation with the PSQI, accepted as the gold standard in the assessment of sleep quality, was examined, and a strong correlation was found. The reliability value was calculated for all scale items, and the test-retest reliability value was also high. To evaluate the structural validity, EFA indicated that the scale had three factors, and this three-factor structure explained 66% of the total variance. The KMO and Bartlett’s sphericity test values were significant in EFA. The initial, subtracted, and corrected item-total values of the final 11 items in the TSS were between 0.45 and 0.79. These values are within acceptable limits when compared to other scales such as the PSQI, Athens Insomnia Scale, and Jenkins Sleep Scale.^47-49 Subsequent CFA supported this three-factor structure. The literature shows these fit indices confirm this structure.^{27, 28, 30, 32, 34} Internal reliability results were similar to or better than the PSQI, Athens Insomnia Scale, and Jenkins Sleep Scale scores. When the test-retest correlation for the TSS was analyzed as another reliability indicator, a high correlation level was observed. Criterion-based validity analysis of the TSS was performed by examining its correlation with the PSQI scores, which are accepted as the gold standard for sleep. The results showed a strong correlation between the TSS and the PSQI. The present findings demonstrated the potential utility of a three-dimensional assessment of sleep disturbance by scoring three factors of the TSS rather than a single sleep quality index. Notably, the sleep efficiency, perceived sleep quality, and daily disturbance subscales of the PSQI’s three-factor validation study were similar to those of the TSS.⁵⁰ The three-factor scale has the advantage of obtaining various assessments of sleep problems on a single scale. More information on the type and nature of sleep problems may be necessary to guide treatment choices. Despite these advantages, caution is advised when generalizing these findings, as they were obtained using only a young non-clinical population. It is also vital to recognize that these three factors may help to differentiate between those with and without sleep disorders. Sleep-related dysfunction, one of the factors on the TSS, is present in individuals with sleep disorders.⁵¹ Sleep quality is closely related to an individual’s physical, cognitive, or emotional functionality.^{52, 53} Sleep quality also affects cognitive areas, such as attention and memory (related to functionality).⁵⁴ In addition, quality of life, which is related to functionality, has been reported to be related to sleep quality.⁵⁵ There is a bidirectional relationship between mood and sleep disorders; sleep and symptoms are included in the diagnostic criteria for mood disorders, and sleep disorders are risk factors for the emergence and recurrence of depression.⁵⁶ Taken together, sleep disturbances are closely related to mood symptoms and functioning beyond sleep initiation, maintenance, and adequate sleep duration, and this has made this sleep scale helpful. The findings of this study should be evaluated considering its limitations. First, since it was conducted with university students and residents, it is not representative of the general population because it was administered to a limited age group. Second, this study was conducted on individuals with self-efficacy. This may limit the ability to determine the validity of these results in a broader population. Third, the scale was not administered to a clinical sample; it must be investigated for sleep disturbances due to mental disorders or other medical illnesses. Fourth, using PSQI as the only scale for concurrent validity is another weakness of the study. Despite these limitations, the study also has strengths. The fact that the scale was prepared considering Turkish culture and language features increased the comprehensibility of the scale. To our knowledge, a scale related to sleep in Turkish has not been developed before. In addition, the fact that IRT analysis was performed while developing the scale is another strength of this study.

Conclusion

In conclusion, our research demonstrated that the TSS is a valid and dependable instrument for assessing sleep. This newly devised scale exhibited desirable properties, including self-administration, internal consistency, reliability, and construct validity, in a sample of university students. Consequently, it is a sound psychometric measurement tool for medical student populations. However, future studies are necessary to evaluate the scale’s psychometric properties across various age groups, such as adolescents and the elderly, as well as in clinical samples with sleep disorders.