Psychometric Reevaluation of the Columbia-Suicide Severity Rating Scale: Findings From a Prospective, Inpatient Cohort of Severely Mentally Ill Adults

Psychometric Reevaluation of the Columbia-Suicide Severity Rating Scale:

Findings From a Prospective, Inpatient Cohort of Severely Mentally Ill Adults

Alok Madan, PhD, MPHa,b,*; B. Christopher Frueh, PhDa,c; Jon G. Allen, PhDa,b; Thomas E. Ellis, PsyDa,b; Katrina A. Rufino, PhDa,d; John M. Oldham, MDa,b; and J. Christopher Fowler, PhDa,b

Accurate prediction of suicide-related behaviors continues to remain elusive due to lack of knowledge of the sensitive and specific long-term risk factors, few short-term warning signs, and variability of suicide risk over time.1,2 The Institute of Medicine1 and the American Psychiatric Association’s Practice Guideline for the Assessment and Treatment of Suicidal Behaviors3 concluded that predicting suicide (especially on the individual level) appears nearly impossible. Longitudinal prediction using relatively distal variables (eg, psychiatric diagnoses) yields high false-positive rates, limiting practical predictive value.1,4,5 Within this context, the Columbia-Suicide Severity Rating Scale (C-SSRS)6 has become the preferred suicide risk instrument, and the US Food and Drug Administration now strongly encourages the C-SSRS for use in all antidepressant clinical trials.7 Given its emerging “gold-standard” status, further psychometric evaluation, particularly of its factor structure and predictive validity, is necessary to supplement 4 existing studies6,8-10 to date. Interrater reliability is reported as moderately good,8,10 convergent validity is reasonably established,6,9,10 and predictive validity of lifetime most severe suicidal ideation was significant in 2 studies.6,9 Unfortunately, no studies have examined the factor structure of the measure. This gap in knowledge is particularly important given that clinical and research decisions often are based on theoretically proposed (as yet to be empirically validated) summary scores from the C-SSRS. Further, inpatients at high risk for suicide-related behaviors are underrepresented in previous studies. The current study addresses these gaps by assessing the factor structure, concurrent validity (compared to other established measures of suicidal ideation and hopelessness), and predictive validity (6-month suicide-related behaviors and service utilization) of the C-SSRS in a large, adult inpatient population with high prevalence rates of lifetime suicide-related ideation and behavior.

METHODS

Participants

Between July 1, 2012, and June 30, 2014, 1,075 adults were admitted to a specialized psychiatric hospital; 1,055 (98.1%) participated in a larger treatment outcomes study11 from which these cross-sectional analyses are based. There were no exclusion criteria for study entry; 1.9% of patients declined participation or were discharged before the C-SSRS was administered. Gender distribution was relatively even with 515 women (48.8%) and 540 men (51.2%). The sample was young (35.2 ± 14.7 years), relatively well-educated (88.1% with at least some college), but mostly unemployed (61.3%). The majority of participants were single or never married (57.7%); the remainder were married/partnered (26.8%), divorced (9.6%), separated (4.6%), or widowed (1.3%). Six respondents declined to provide data pertaining to their marital status. In terms of racial background, the majority identified as Caucasian (90.2%); the remainder identified as multiracial (5.7%), African-American (1.8%), Asian (1.3%), American Indian (0.5%), or Native Hawaiian/Other Pacific Islander (0.5%). One participant declined to indicate racial background. Seventy-two patients identified as being of Hispanic or Latino ethnicity (6.8%).

The study sample was severely mentally ill. Patients reported an extensive psychiatric history and having received treatment from multiple previous therapists (4.1 ± 3.8), psychopharmacologists (3.1 ± 3.0), acute psychiatric hospitalizations (1.4 ± 2.4), and extended (> 5 days) psychiatric hospitalizations (1.1 ± 2.1). Admission diagnoses based on Structured Clinical Interview for the DSM-IV (SCID-I)12 indicated that patients met criteria for multiple Axis I disorders (2.9 ± 1.6); 37.1% also met criteria for an Axis II personality disorder diagnosis based on SCID-II13 assessments. Axis I spectrums included depressive disorders (62.7%), bipolar spectrum disorders (18.9%), anxiety disorders (57.6%), substance use disorders (57.4%), and psychotic disorders (8.3%). The most common personality disorders were borderline personality disorder (17.1%) and avoidant personality disorder (14.2%).

• Accurate prediction of suicide remains elusive due to lack of predictive measures.
• This first independent psychometric evaluation of the Columbia-Suicide Severity Rating Scale’s (C-SSRS) found the measure to be an internally consistent and valid instrument to quantify suicide severity, intensity, and behaviors.
• An empirically derived factor score composed of severity of ideation and suicidal behavioral items from the C-SSRS was modestly correlated with suicide-related behaviors within the 6 months following a psychiatric hospitalization. The C-SSRS provides a clinically useful assessment tool that balances sensitivity and specificity relative to other measures of suicidality.

Not all patients provided complete data. Relative to patients who provided only baseline data (n = 737), patients who provided baseline and follow-up data (n = 318, discussed next) had received treatment from more previous therapists (4.5 ± 4.3 vs 3.9 ± 3.6, P = .013, d = 0.162), psychopharmacologists (3.5 ± 3.9 vs 2.9 ± 2.5, P = .004, d = 0.175), acute psychiatric hospitalizations (1.7 ± 2.5 vs 1.3 ± 2.3, P = .025, d = 0.148), and extended (> 5 days) psychiatric hospitalizations (1.4 ± 2.7 vs 1.0 ± 1.8, P = .003, d = 0.182). Additionally, patients who provided baseline and follow-up data (47.2%) were more likely to receive a formal Axis II diagnosis than patients who provided only baseline data (32.4%), P < .0001, φ = 0.143. There were no other differences across sociodemographic characteristics, Axis I diagnoses, or treatment histories. Of note, the few observed differences between groups were below conventional standards of small effect sizes (ie, < 0.20).

Procedures

Data were collected as part of the hospital’s clinical outcomes project to assess treatment response over time11 with a smaller subsample also participating in a suicide-specific treatment trial.14 Baseline measures were collected within 72 hours of admission and at 2-week intervals during hospitalization as part of a larger battery of assessments. Data were collected via laptop computers. All assessments were designed and implemented as an element of routine clinical care and integrated into treatment planning and monitoring of progress. As such, obtaining formal informed consent was not necessary. Patients and their treatment teams were provided with profile scores and feedback regarding diagnostic findings. In addition, they were informed that findings were used to evaluate the effectiveness of treatment and for research purposes. Follow-up phone interviews were conducted at scheduled intervals postdischarge at 2, 12, and 24 weeks. Use of all data—including analysis and reporting of findings—was approved by Baylor College of Medicine’s institutional review board.

Measures

Demographic variables as well as history of psychiatric hospitalization and psychiatric service usage were assessed using a standardized patient information survey.11

The C-SSRS is a semistructured interview that assesses severity of suicidal ideation, intensity of ideation, suicidal behavior, and lethality of suicide attempts.6 The Severity of Ideation subscale uses a 5-point scale comprising (1) a passive wish to be dead, (2) nonspecific active thoughts of suicide, (3) active suicidal ideation including methods, (4) suicidal intent without a specific plan, and (5) active suicidal ideation with intent and a plan to act. The Intensity of Ideation subscale is composed of 5 items (frequency, duration, controllability, deterrents, and reasons for ideation), each rated on a 5-point scale with increasing values indicating increasing intensity. The Suicidal Behavior subscale assesses actual attempts, nonsuicidal self-injurious behavior, interrupted attempts, and aborted attempts using a nominal scale. The lethality subscale uses an ordinal scale to assess the lethality of actual attempts or the potential for lethality if the lethality was none or minimal.6

The Patient Health Questionnaire (PHQ-9)15 is a 9-item self-report measure assessing core depression symptomatology over the prior 2 weeks using a 4-point Likert scale. Only item 9, which pertains to suicidal ideation (ie, “Thoughts that you would be better off dead or thoughts of hurting yourself in some way”), was retained for inclusion in these analyses.

The Beck Scale for Suicidal Ideation (BSS)16 is a self-report instrument consisting of 21 sets of statements containing content such as wish to live, wish to die, frequency of ideation, perceived capability to carry out an attempt, and extent of actual preparation.15 Scores range from 0 to 38. The BSS is widely used in suicide research.17

The Beck Hopelessness Scale (BHS)18 is a 20-item self-report instrument intended to measure negative future thinking. Items are rated as true or false. Hopelessness has been shown to be a key mediator between depression and suicidal ideation, and the BHS has demonstrated sensitivity (0.91) for deaths by suicide.17

The Suicide Cognitions Scale (SCS)19 is a self-report instrument consisting of 18 items rated on a 5-point scale. Items were constructed to be consistent with schemas of unbearability and unlovability with scores ranging from 18 to 90. The SCS has excellent psychometric properties.19,20

Data Analysis

Given the large number of analyses and potential for α inflation, statistical significance was set at P < .01. All analyses were conducted using SAS/STAT software, version 9.3 of the SAS System for Microsoft Windows, Cary, North Carolina.

Reliability

The scale of measurement for the 20 items across 3 subscales of the C-SSRS includes items that have potential responses that are (1) dichotomous (yes vs no; eg, “actual attempt”), (2) ordinal (rank-ordered; eg, frequency of thoughts from low to high), (3) categorical (multiple, non-ordered classes; eg, “reasons for ideation”), and (4) continuous (at-least interval scale; eg, “total number of attempts”). Given this variability in scale of measurement, standard approaches to evaluating internal consistency are not possible. In fact, Posner and colleagues commented accordingly in 2011: “The severity and behavior subscales use an ordinal scale and are therefore not subject to internal consistency analysis.”6(p1269) However, subsequent advances (ie, in 2012)21 in psychometric theory, research, and practice have revealed ordinal α to be an appropriate measure of internal consistency for instruments that have items with variable scales of measurement. Interpretation of results obtained from ordinal α analyses are the same as with traditional approaches, such that higher α values are better.22 Consequently, in the current analyses, ordinal α was calculated separately for the 3 subscales of the C-SSRS (severity of ideation, intensity of ideation, and behaviors) as well as for the full measure.

Construct Validity

Although the C-SSRS is divided into 3 theoretical subscales, there are no published data on the factor structure of the C-SSRS. Consequently, an exploratory factor analysis was undertaken to assess the latent structure of the scale. Again, because of the variable scales of measurement of the 20 items of the C-SSRS, traditional factor analytic approaches are inappropriate given that they assume an underlying correlation matrix (eg, Pearson’s ) among continuously scaled items.22 When there are deviations from the aforementioned assumption, a polychoric correlation matrix among items must first be calculated and then factor analytic approaches applied.22 Consequently, principal components analysis (PCA) of the underlying polychoric correlation matrix of nonredundant items from the C-SSRS was used to establish the construct validity of the measure. Component extraction was based on Horn’s parallel analysis23 as it is among the least subjective, most accurate, and sensitive methods of factor extraction.24 The more commonly used Kaiser criterion (ie, eigenvalues ≥ 1), on the other hand, is among the least accurate methods and tends to overestimate number of factors to retain.24 The factor structure was rotated using a varimax rotation for ease of interpretation. Note that the PCA was run on all items of the C-SSRS except for “most severe ideation”; this item is redundant in that it represents a rank ordering of the previous 5 items that constitute the suicidal ideation subscale. Summing scores from the items that loaded on each of the factors derived from the PCA allowed for the calculation of clinically useful summary scores.

Concurrent Validity

The total summed score of nonredundant items from the C-SSRS, the summary scores from the PCA, and the single item most severe ideation were correlated with total scores of commonly used measures of suicidal ideation: item 9 from the PHQ-9, the BSS, BHS, and SCS. In addition to the C-SSRS, all patients completed the PHQ-9. However, only a subsample (n = 318) completed the BSS, BHS, and SCS as part of their participation in a concurrent suicide-specific treatment trial.

Predictive Validity

Suicide-related behaviors. The total summed score of nonredundant items from the C-SSRS, the summary scores from the PCA, and the single item most severe ideation were correlated separately with each of the following self-reported suicide-related behaviors: actual attempts, aborted attempts, interrupted attempts, and preparatory acts obtained from the Suicide Behaviors subscale from the C-SSRS during any 1 of 3 scheduled follow-up contacts (2, 12, and 24 weeks) after discharge from the hospital. A dichotomous (yes vs no) variable that captured any of the aforementioned suicide-related behaviors was calculated and also correlated with the total score, the factor scores from the PCA, and the single item most severe ideation.

Of note, receiver operating characteristic (ROC) analyses were calculated to provide clinically meaningful cut-scores for the 4 scores from the C-SSRS (total score, factor 1 from the PCA, factor 2 from the PCA, and single item most severe ideation) as well as other commonly used measures of suicidality in correctly classifying any suicide-related behaviors within 6 months of discharge from the hospital. Results are presented in terms of area under the curve, sensitivity, and specificity (see Table 3).

Service utilization. The total summed score of nonredundant items from the C-SSRS, the summary scores from the PCA, and the single item most severe ideation were correlated with the emergency department visit(s) and psychiatric hospitalization(s) during any 1 of 3 scheduled follow-up contacts (2, 12, and 24 weeks) after discharge from the hospital.

RESULTS

Reliability

When analyzed in aggregate, the 20 items comprising the C-SSRS evidenced excellent internal consistency (ordinal α = .95). Subscale analyses revealed variable findings. The 5-item Severity of Ideation subscale (ordinal α = .98) and the Suicidal Behavior subscale (ordinal α = .95) evidenced excellent internal consistency as well. The 5 items comprising the Intensity of Ideation subscale revealed poor/unacceptable levels of internal consistency (ordinal α = .59).

Validity

Horn’s parallel analyses indicated forcing a 2-factor solution to the PCA. All 19 items loaded onto the orthogonally rotated solution with factor loadings above 0.51 and in a solution that accounts for 65.3% of the variance across items. The severity of ideation and behavioral items rotated onto the first factor, and the intensity of ideation items loaded onto the second factor. This PCA provides support to the construct validity of the C-SSRS. See Table 1 for details.

Table 2 provides details of the multiple additional validity analyses. The total summed score of nonredundant items from the C-SSRS, the summary scores from the PCA, and the single item most severe ideation were found to be significantly correlated (r = 0.27 to 0.58) with frequently used measures of suicidality, demonstrating adequate concurrent validity. The total score from the C-SSRS and the summary score from factor 1 were found to be moderately correlated with individual suicide-related behaviors and more strongly correlated with any suicide-related behavior within the 6-month period following hospitalization compared to the intensity score and the single item most severe ideation from the C-SSRS. Of the 4 scores from the C-SSRS, the total summed score and intensity summary score were found to be correlated with emergency department visits and psychiatric hospitalizations within 6 months following discharge from the hospital; the single item was correlated with emergency department visits only. Notably, the severity of ideation/behavior factor was not correlated with service utilization.

Table 3 provides details of the ROC analyses. All 4 scores from the C-SSRS as well as the 4 other commonly used measures of suicidality performed better than chance in correctly classifying any suicide-related behavior within 6 months after discharge from the hospital (P < .001). The C-SSRS total score and factor 1 from the PCA provided the best balance between sensitivity and specificity, performing almost identically. Although the single item most severe ideation and the single suicide-related item from the PHQ-9 were found to be highly sensitive to any suicide-related behaviors (0.861), both evidenced poor specificity (< 0.41).

DISCUSSION

An emerging consensus in the suicide research literature contends that suicide risk is best conceptualized as a complex diathesis-stress phenomenon involving genetic vulnerability triggered by early adverse events, resulting in impaired development and function of neurobiological systems regulating behavior, affect, and cognitive function.25-29 Stress-response systems may become overwhelmed in response to episodic negative life events, increasing the likelihood of triggering a suicidal crisis. Studies generally support diathesis-stress models for predicting suicide risk, finding that interactions between early adverse events and current impulsivity,29 recent stressful life events,30,31 and level of psychopathology and recent stressful life events in alcoholics32 confer increased risk of suicide-related behaviors. In the context of diathesis-stress models, any single measure, regardless of the robustness of its psychometric properties, will exhibit limited validity in predicting outcome. Therefore, despite the relatively small magnitude of the correlations and marginally acceptable sensitivity and specificity in correctly classifying suicide-related behaviors, the results of the current study provide support for the predictive validity and clinical utility of the C-SSRS.

The results of reliability analysis revealed that, overall, the C-SSRS functions well as an internally coherent construct; however, the Intensity of Ideation subscale (frequency, duration, controllability, deterrents, and reasons for suicidal ideation) had poor internal consistency. One potential reason for this result is the confounding use of a zero point that can indicate either greater intensity of ideation or no problem with ideation. Future research should assess the internal consistency of the Intensity of Ideation subscale with the zero-point anchor eliminated in order to clarify its psychometric properties. In the interim, clinicians, researchers, and policy-makers should judiciously use this subscale as an independent measure of the intensity of suicidal ideation.

The results from the factor analysis revealed new information regarding the C-SSRS. Contrary to the developers’ assumptions of multiple domains, the results support a 2-factor solution, with severity of suicidal ideation and all suicide-related behavior items loading onto the first factor, and the intensity of ideation items loading on a second factor. The 2 sets of items comprising the first factor are conceptually related insofar as the severity of ideation encompasses behavioral methods/plans (ie, rehearsal) and line up with actual behaviors. These items differ from items that load on the second factor, where the “intensity” items pertain more to the extent of preoccupation than the specific contents. While it may be expedient for clinicians to draw conclusions based on single items such as total number of suicide attempts, aborted attempts, and interrupted attempts, the PCA results point to the potential benefit of computing factor scores that can then be regressed with other outcomes.

Overall, the C-SSRS demonstrated reasonable concurrent validity with self-report measures of suicidal ideation; however, the wide range of small-to-medium effect sizes indicates that the interview measure assesses similar yet distinct constructs compared to measures of suicidal ideation, hopelessness, or suicide cognitions. This finding should not come as a surprise in part because the C-SSRS assesses suicide-related behaviors and ideation. Furthermore, the C-SSRS is an interview-based measure whereas the BSS and BHS are self-report measures, and this difference thereby introduces hetero-method measurement error variance.

The C-SSRS total score, summary scores from the PCA, and the single item most severe ideation demonstrated differential predictive validity. The total score and factor 1 demonstrated significant association with postdischarge preparatory, interrupted, aborted, and actual suicide attempts, as well as all suicide-related behaviors (small-to-medium correlations). Factor 2 was predictive of postdischarge interrupted attempts, all suicide-related behaviors, and service utilization, whereas most severe ideation was predictive of preparatory and interrupted attempts. Compared to other studies indicating most severe ideation6,9 and past suicide attempt as most predictive of future suicide attempts, the current study indicated that total score (r = 0.289) and factor 1 (r = 0.282) were more robust predictors of suicide-related behaviors than most severe ideation (r = 0.165). Even though all 4 scores from the C-SSRS performed better than chance in correctly classifying any suicide-related behaviors within 6 months of discharge from the hospital, the total score and summary score from factor 1 provided the best (but still only marginally acceptable) balance between sensitivity and specificity.

This study has a number of limitations. First, the sample is largely Caucasian, which limits generalizability to individuals from other racial/ethnic backgrounds. The inpatient sample may not generalize to all clinical populations; however, the importance of studying patients with high burden of illness and high risk for suicide is a necessary step in improving diagnostic and treatment approaches to suicide risk. Second, as is evident from the significant difference between the number of patients completing the C-SSRS at admission (N = 1,055) and the relatively fewer individuals providing data during the 6-month period following the hospitalization (n = 318), patients lost to follow-up may bias the findings. This potential for bias, however, is somewhat tempered. There were few differences between patients who provided follow-up data and those who did not. Although differences between treatment histories and likelihood of meeting criteria for formal Axis II disorder were statistically significant, all observed differences between groups were below conventional standards of even small effect sizes (ie, < 0.20). Nonetheless, future research should strategize efforts to improve longitudinal participation as well as efforts to obtain external measures of suicide lethality.

This study has a number of important strengths, including a large sample of adult inpatients with high base rates of suicide-related behaviors, as well as advanced analytic approaches appropriate to the measurement challenges of the C-SSRS. Despite losing a large percentage of the original study sample during the follow-up period, 6-month, cross-sectional data from a large cohort of high-risk patients sets this study apart from much of the existing suicide literature. Finally, this is the first report on the latent factor structure of the measure and the first independent study to examine the psychometric properties of the C-SSRS outside of the measure’s original developers. These notable strengths and essentially confirmatory findings from this study should help to allay concerns (eg, Health Policy Advisory Committee on Technology, State of Queensland, Australia33) regarding the widespread diffusion of this measure into clinical and research practice despite few psychometric evaluations. The C-SSRS’s solid psychometric properties merit its use as a suicidal risk assessment instrument.

Submitted: April 20, 2015; accepted October 26, 2015.

Potential conflicts of interest: Aside from partial support noted below, the authors have no other potential conflicts of interest to disclose.

Funding/support: This research was partially supported by the Menninger Clinic Foundation, McNair Medical Institute, and the Brown Foundation, Inc, of Houston, Texas.

Role of the sponsor: The study sponsors were not involved in any aspect of the research activities. Thus, the authors were independent from study sponsors in the context of the research.

Additional information: The study follows the guidelines on good publication practices. Drs Frueh and Madan are McNair Scholars.

REFERENCES

1. Goldsmith S, Pellmar T, Kleinman A, et al. Reducing Suicide: A National Imperative. Washington, DC: National Academies Press; 2002.

2. Fowler JC. Suicide risk assessment in clinical practice: pragmatic guidelines for imperfect assessments. Psychotherapy (Chic). 2012;49(1):81-90. PubMed doi:10.1037/a0026148

3. Jacobs DG, Baldessarini RJ, Conwell Y, et al; Work Group on Suicidal Behaviors. Practice Guideline for the Assessment and Treatment of Suicidal Behaviors. Arlington, VA: American Psychiatric Publishing; 2003.

4. Rudd MD. Fluid Vulnerability Theory: A Cognitive Approach to Understanding the Process of Acute and Chronic Suicide Risk. In: Ellis TE, ed. Cognition and Suicide: Theory, Research, and Therapy. Washington, DC: American Psychological Association; 2006. doi:10.1037/11377-016

5. Oquendo M, Halberstam B, Mann J, et al. Risk factors for suicidal behavior: Utility and limitations of research instruments. In: First MB, ed. Standardized Evaluation in Clinical Practice. Washington, DC: American Psychiatric Publishing Inc; 2003.

6. Posner K, Brown GK, Stanley B, et al. The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry. 2011;168(12):1266-1277. PubMed doi:10.1176/appi.ajp.2011.10111704

7. US Department of Health and Human Services. Guidance for Industry: Suicidal Ideation and Behavior: Prospective Assessment of Occurrence in Clinical Trials. US Food and Drug Administration Web site. http://www.fda.gov/drugs/guidancecomplianceregulatoryinformation/guidances/ucm315156.htm. Accessed September 9, 2014.

8. Mundt JC, Greist JH, Gelenberg AJ, et al. Feasibility and validation of a computer-automated Columbia-Suicide Severity Rating Scale using interactive voice response technology. J Psychiatr Res. 2010;44(16):1224-1228. PubMed doi:10.1016/j.jpsychires.2010.04.025

9. Mundt JC, Greist JH, Jefferson JW, et al. Prediction of suicidal behavior in clinical research by lifetime suicidal ideation and behavior ascertained by the electronic Columbia-Suicide Severity Rating Scale. J Clin Psychiatry. 2013;74(9):887-893. PubMed doi:10.4088/JCP.13m08398

10. Kerr DCR, Gibson B, Leve LD, et al. Young adult follow-up of adolescent girls in juvenile justice using the Columbia suicide severity rating scale. Suicide Life Threat Behav. 2014;44(2):113-129. PubMed doi:10.1111/sltb.12072

11. Fowler JC, Allen JG, Oldham JM, et al. Exposure to interpersonal trauma, attachment insecurity, and depression severity. J Affect Disord. 2013;149(1-3):313-318. PubMed doi:10.1016/j.jad.2013.01.045

12. First MB, Spitzer RL, Gibbon M, et al. Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Research Version, Patient Edition With Psychotic Screen (SCID-I/P W/ PSY SCREEN). New York: Biometrics Research, New York State Psychiatric Institute; 2002.

13. First MB, Gibbon M, Spitzer RL, et al. Structured Clinical Interview for DSM-IV Axis II Personality Disorders, (SCID-II). Washington, D.C.: American Psychiatric Press, Inc.; 1997.

14. Ellis TE, Rufino KA, Allen JG, et al. Impact of a suicide-specific intervention within inpatient psychiatric care: The Collaborative Assessment and Management of Suicidality (CAMS) [published online ahead of print January 12, 2015]. Suicide Life Threat Behav. PubMed doi:10.1111/sltb.12151

15. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606-613. PubMed doi:10.1046/j.1525-1497.2001.016009606.x

16. Beck A, Steer R. Manual for the Beck Scale for Suicidal Ideation. San Antonio, TX: The Psychological Corporation; 1991.

17. Brown GK, Jeglic E, Henriques GR, et al. Cognitive therapy, cognition, and suicidal behavior. In: Ellis TE, ed. Cognition and Suicide: Theory, Research, and Therapy. Washington, DC: American Psychological Association; 2006. doi:10.1037/11377-003

18. Beck A, Steer R. Manual for the Beck Hopelessness Scale. San Antonio, TX: The Psychological Corporation; 1988.

19. Bryan CJ, David Rudd M, Wertenberger E, et al. Improving the detection and prediction of suicidal behavior among military personnel by measuring suicidal beliefs: an evaluation of the Suicide Cognitions Scale. J Affect Disord. 2014;159:15-22. PubMed doi:10.1016/j.jad.2014.02.021

20. Ellis TE, Rufino KA. A psychometric study of the Suicide Cognitions Scale with psychiatric inpatients. Psychol Assess. 2015;27(1):82-89. PubMed doi:10.1037/pas0000028

21. Gadermann A, Guhn M, Zumbo B. Estimating ordinal reliability for Likert-type and ordinal item response data: a conceptual, empirical, and practical guide. PARE. 2012;17(3):1-13.

22. Tabachnick B, Fidell L. Using Multivariate Statistics. 4th ed. Needham Heights, MA: McGraw-Hill; 2001.

23. Horn JL. A rationale and test for the number of factors in factor analysis. Psychometrika. 1965;30(2):179-185. PubMed doi:10.1007/BF02289447

24. Costello A, Osborne J. Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. PARE. 2005;10(7):1-9.

25. Currier D, Mann JJ. Stress, genes and the biology of suicidal behavior. Psychiatr Clin North Am. 2008;31(2):247-269. PubMed doi:10.1016/j.psc.2008.01.005

26. Mann JJ, Waternaux C, Haas GL, et al. Toward a clinical model of suicidal behavior in psychiatric patients. Am J Psychiatry. 1999;156(2):181-189. PubMed

27. Turecki G. Dissecting the suicide phenotype: the role of impulsive-aggressive behaviours. J Psychiatry Neurosci. 2005;30(6):398-408. PubMed

28. Rudd MD, Berman AL, Joiner TE Jr, et al. Warning signs for suicide: theory, research, and clinical applications. Suicide Life Threat Behav. 2006;36(3):255-262. PubMed doi:10.1521/suli.2006.36.3.255

29. Brodsky BS, Oquendo M, Ellis SP, et al. The relationship of childhood abuse to impulsivity and suicidal behavior in adults with major depression. Am J Psychiatry. 2001;158(11):1871-1877. PubMed doi:10.1176/appi.ajp.158.11.1871

30. Caspi A, Sugden K, Moffitt TE, et al. Influence of life stress on depression: moderation by a polymorphism in the 5-HTT gene. Science. 2003;301(5631):386-389. PubMed doi:10.1126/science.1083968

31. Chang EC, Sanna LJ, Hirsch JK, et al. Loneliness and negative life events as predictors of hopelessness and suicidal behaviors in Hispanics: evidence for a diathesis-stress model. J Clin Psychol. 2010;66(12):1242-1253. PubMed doi:10.1002/jclp.20721

32. Conner KR, Beautrais AL, Conwell Y. Risk factors for suicide and medically serious suicide attempts among alcoholics: analyses of Canterbury Suicide Project data. J Stud Alcohol. 2003;64(4):551-554. PubMed doi:10.15288/jsa.2003.64.551

33. Health Policy Advisory Committee on Technology (HealthPact). Technology Brief—Columbia Suicide Severity Rating Scale. State of Queensland, Australia: Australian Safety and Efficacy Register of New Interventional Procedures—Surgical, Royal Australasian College of Surgeons; 2012.