Negative Response Bias in a Disability and Civil Liability Claim setting: A Dutch Study on Prevalence Rates and Associations with Neuropsychological Test Outcomes

In the present study, we examined NRB prevalence in a disability or liability claim setting. To our knowledge, few Dutch studies have been powered to estimate the prevalence of NRB, used multiple PVT/SVTs, and examined associations with diagnostic data, neuropsychological outcomes, and self-reported depression outside a clinical setting. Given substantial differences in the context of the complaints of patients and those referred for assessment of disability or liability claims, additional focus on these factors in nonclinical cases is needed. We included different cases with a suspected or established general medical, neurological, traumatic brain injury, or psychiatric diagnosis, and examined effects on cognitive and self-reported outcomes.

In this cross-sectional study of archival data, we included persons who were referred by their company or insurance doctor, or the Courts in the context of a liability or disability claim. Participants were consecutively assessed at Diagnostic Centre Clinics Amsterdam (DCCA) between 2006 and 2017. Referrals were made either directly to DCCA-affiliated registered clinical neuropsychologist (JDJ) or neuropsychologist (TS), or via DCCA affiliated psychiatrists and neurologists. Many participants had received or were still on active psychiatric or psychological treatment or psychical rehabilitation. Only one was a psychiatric inpatient at the time of the assessment. Relevant medical information was presented upon referral. Other inclusion criteria included fluent in Dutch or another language supported by the WMT software.

Probable NRB was defined as a positive score on any combination of two or more PVT/SVT’s. Two positive PVT/SVT in a liability or disability claim context corresponds with Slick criteria for malingered cognitive dysfunction [20]. All participants had taken the WMT. In 3.8% of all cases only WMT data were available and were used to classify NRB. Reasons for missing WMT data were visual problems, language barrier, refusal to further cooperate or a dramatic symptom presentation, such as reporting extreme fatigue or, e.g., a patient slowly sliding from a chair onto the floor after finishing WMT immediate and delayed recall. In these few instances of a single positive PVT, the participant was classified as NRB positive.

We collected sociodemographic information, including age, gender, education and primary diagnosis. Information on suspected or established diagnoses was presented by the referring physician. The authors (JDJ and TS) reviewed all available medical and other diagnostic information prior to cognitive testing and classified cases according to the WMT demographic and clinical characteristics section that includes different neurological, psychiatric, or other medical conditions. In case of multiple co-existing disorders the most clinically relevant one describing cognitive or affective status was chosen in order to classify participants. E.g., if the medical information stated both a suspected or differential diagnosis of depression and burn-out, we chose depression as the principle diagnosis.

We assessed cognitive function and self-rated depression using a standard protocol of neuropsychological tests, including Digit Span forward and backward [21], Groningen Intelligence Test short version [22], Stroop [23], 15-item Auditory Verbal Learning Test [24], the Beck Depression Inventory (BDI-II) [25], and Assessment of Depression Inventory Dutch version [26]. Other cognitive tests were also included in the protocol, but no data were available for analysis purposes. As participants did not take all protocol tests for reasons described above, pairwise comparisons were made.

We collected data on NRB using six PVT/SVT’s: WMT, ASTM, SIMS, Reliable Digit Span (RDS) [7], the Visual Association Test-extended (VAT-E) [27], and the ADI-Dutch version over-reporting subscale. Some PVT/SVTs were added to the test protocol during the inclusion period for reasons of test development and diagnostic purposes.

Participants provided written informed consent to store data digitally to be used for scientific purposes. This retrospective patient record study design used archival data and was not reviewed by a medical research ethics committee.

PVT and SVT scores were categorized based on established cut-offs. We used the average WMT percentage score on three of its performance validity indicators, i.e., (Immediate Recall + Delayed Recall + Consistency)/3. NRB was analyzed as a dichotomous variable, and cognitive measures and questionnaires were continuous variables. We used the GIT-2 intelligence quotient (average = 100, SD = 15), the Digit Span test, the Stroop I-III, the AVLT total score and delayed recall t-scores (average = 50, SD = 10), and the BDI-II raw score. DS was not analyzed as a cognitive outcome because of collinearity with RDS. Level of education was condensed into low, intermediate, and high (low = lower secondary education or less, intermediate = upper and post-secondary education, and high = first and second stage of tertiary education). We used the kappa statistic to evaluate level of agreement between individual PVT/SVTs and the NRB criterion. A kappa value below 0.40 can be considered poor agreement, between 0.40 - 0.75 as fair to good and over. 0.75 is excellent. We compared demographic data across groups with or without NRB using the t-test and Chi-square test. We used t-test or AN(C)OVA to examine NRB and demographic data as predictors of cognitive impairment and self-rated depression. Predictors that were significant in univariate analysis were subsequently used in multivariate analyses. Effect size was examined using Cohen’s Delta or Partial Eta2, where d = 0.2 stands for a small effect, d = 0.5 a medium, and d = 0.8 a large effect, and partial eta squared can be considered a small (ηp2 = 0.01), a medium (ηp2 = 0.06), or a large (ηp2 = 0.14) effect.

Prevalence rates

We included a total of 393 cases who had taken only the WMT, or the WMT and one or more other PVT/SVT’s, i.e., AKTG (n = 270, 68.7%), SIMS (n = 318, 80.9%), RDS (n = 274, 69.7%), VAT-E (n = 164, 41.7%), or ADI (n = 172, 43.7%). Prevalence of NRB was 48.3%. Below chance level responding on WMT Delayed recall was noted in 4.8% of cases and random responding in 15.7% (score 50 +/- 16%). Overreporting of symptoms on the SIMS (score > 15) was present in 65.8% of cases. Agreement between individual PVT/SVT outcomes and NRB classification was kappa = 0.78 for the WMT, ASTM kappa = 0.70, SIMS kappa = 0.62, RDS kappa = 0.54, ADI kappa = 0.50 and VAT-E Kappa = 0.30.

While the male/female ratio (n = 229/164) nor age (average 46.9, SD 10.5, range 18-71 years) was associated with NRB, level of education was (Chi2 = 10.4, p = 0.006). Post hoc analyses of those cases with Primary (n = 28, NRB + 54%), Lower secondary (n = 198, NRB + 56%), or Upper secondary education (n = 167, NRB + 39%) showed that the prevalence of NRB was higher in the Lower secondary education sub-sample than in the Higher secondary education sub-sample (Chi2 = 10.0, p = 0.002).

NRB and diagnosis

Referral questions concerned four broad diagnostic classes: Neurological (n = 73, 18.6%), Psychiatric (n = 249, 63.4%), General medical problem (n = 21, 5.3%), Traumatic Brain Injury (TBI) (n = 45, 11.5%). Diagnostic information was missing in 5 cases (‘no diagnosis’, 1.7%). Level of education was not associated with diagnostic group membership (Chi2 = 3.3, p = 0.51). Comparison of three relatively large subgroups of neurological, psychiatric, and TBI cases showed that NRB was associated with diagnosis (Chi2 = 13.1, p = 0.001). Post hoc analysis revealed that NRB was more prevalent in psychiatric cases as compared to neurological cases (Chi2 = 11.5, p = 0.001) (Table 1). Stroke patients made up the largest neurological sub-sample, and NRB was present in 9/28 (32.1%) of cases. Focusing on relatively larger psychiatric patient sub samples, we found that NRB was present in 39/60 (65%) outpatients with major depressive disorder, in 28/48 (58.3%) with pain disorder, in 22/36 (61.1%) with PTSD, in 12/27 (44.4%) with anxiety disorder other than PTSD/OCD, in 3/14 (21.4%) with burn-out, in 10/13 (76.9%) with somatoform/conversion, and in 7/10 (70%) with chronic fatigue.

NRB, cognitive test outcome, and self-rated depression

NRB was associated with demographically corrected test outcomes for attention/executive control, intelligence, and verbal memory (STROOP cards I, II, and III, GIT-2, 15WT). Subjects who underperformed generally yielded low scores on all neuropsychological tests. Large effect sizes of NRB were found on the STROOP and GIT-2 (Cohen’s d > 0.8), and medium or large on 15WT total score and 15WT delayed recall (Cohen’s d = 0.749 and d = 0.558, respectively) (Table 2).

Age was associated with BDI-II scores (r = -0.24, p = <0.001), yet gender and level of education were not. NRB was independently associated with self-rated depression, and the effect size was large (Partial Eta2 = 0.165).

We examined over-reporting of symptoms and underperformance on cognitive tests during neuropsychological assessment in a disability and liability claim setting. Overall prevalence of NRB was particularly high in patients with a suspected or established psychiatric diagnosis. NRB was associated with lower cognitive test scores and higher self-rated depression scores. Strong points of this study are the large number of included cases with a panoply of general medical, neurological, psychiatric or traumatic brain injury diagnoses, the use of up to six performance or symptom validity tests, and the homogeneous non-clinical setting.

Implications for practice

The present study outcomes may well have implications for neuropsychological evaluation in a disability and liability claim setting. Almost 50% of cases overreported symptoms or underperformed, and their scores on cognitive tests and a depression questionnaire were often abnormal, though probably unreliable. Agreement between individual PVT/SVTs and the NRB criterion varied between very modest and almost excellent, a finding that may urge psychologists to use several PVT/SVTs, as it may increase the NRB detection rate. The importance of our findings cannot easily be overestimated, given that NRB was particularly high in psychiatric cases, and a large number of people in the Netherlands receive disability pensions and also report mental problems.

Our results are in line with research findings outside the Netherlands. Studies in a disability or liability claim setting have reported negative response bias in 30-50% of cases [3,7,28]. Different European study groups examined NRB, many of which were Dutch [1]. Excluding those Dutch studies that focused on forensic cases, many included patients in a clinical context. NRB, as measured with the ASTM, was prevalent in Whiplash patients referred to a university hospital [8]. Whiplash patients with NRB scored dramatically worse on cognitive tests compared to those without. Another study showed that 30% of Chronic Fatigue (CF) patients scored positive on the ASTM as compared to 13% with Multiple Sclerosis (MS) [17]. Possible or probable NRB as measured with the ASTM, the nonverbal Test of Memory Malingering (TOMM) and the Warrington’s Warrington’s Recognition Memory Test for Faces (WMT), was present in 57.2% of cases with a provisional diagnosis of suspected Chronic Toxic Encephalopathy [18]. In a study of psychiatric outpatients, with a preponderance of patients with Attention Deficit Hyperactivity Disorder or Autism Spectrum Disorder, 33.9% failed the ASTM and/or SIMS [5]. Symptom Checklist-90 scores or composite cognitive tests scores differed 1-2 SD between groups with or without NRB. A study examined NRB in a large memory clinic sample using the WMT and TOMM [19]. NRB was practically non-existent in patients over the age of 65, but it was present in 13% of relatively younger cases. NRB may influence therapy outcome, as was demonstrated in a study of Cognitive Behavior Therapy: drop-out rate was 23% for CF patients with NRB, almost a threefold of what was found for patients without NRB [29]. Most Dutch studies on NRB examined specific outpatient samples, did not measure overreporting of depressive complaints, and used one or two measures that showed varying prevalence rates of NRB. By including cases with a suspected or established diagnosis of neurological, psychiatric, general medical or TBI problems, and by using several PVT/SVT’s, we were able to reliably estimate NRB prevalence in a disability and liability setting, and examine NRB associations with both neuropsychological tests and self-rated depression.

Underperformance or overreporting of symptoms in clinical settings are conceptually akin to poor working alliance or medication nonadherence. A physician or psychologist might think that the therapist - patient working alliance is congruent, to soon find out that it is not when patients do not show expected progress. But relevance of NRB is not confined to individual clinical assessments. The Dutch Employee Insurance Agency (UWV) reported that a total of 93.900 persons received sick leave payments and 808.300 persons received disability payments in January 2019 [30]. The associated overall payment costs in 2017 amounted to 9.35 billion euros [31]. A study by the Dutch Central Bureau of Statistics (CBS) showed that 40% of all persons reporting disability in 2011 also reported mental problems as measured with a 5-item questionnaire that consists of depression items [10]. NRB was not measured in that study. Meta-analyses demonstrating relatively modest effect sizes in high quality studies of psychotherapy and antidepressant medication for affective disorder [10-13, 32], report that self-rating questionnaires are used as study outcome measure. However, the use of PVT/SVT is not mentioned in those studies. Even low NRB prevalence rates could potentially confound treatment study outcomes, particularly when effects sizes are small, and it may invalidate psychiatric diagnoses, thereby leading to excess research and societal costs.

Differences between neurological and psychiatric cases with regard to objectified underperformance highlights validity of the PVT concept. Response bias measures are often based on the assumption that patients with mild TBI or mild dementia do well on simple cognitive tests and that lay persons are unaware of what constitutes a genuine memory problem. In this study almost one third of the stroke patients cases were as classified probable NRB. By contrast, more than half the cases with depression, pain disorder or PTSD were NRB positive. Notably, assessment setting was the same for all participants. Differences found between stroke and psychiatric patients highlight importance of effort during test taking. It is not because of cognitive decline that psychiatric outpatients score positive on PVT, but it is primarily the internally and externally motivated behavior that is related to failure instead. NRB may not be obvious to the clinician and previously mentioned findings on prevalent NRB in cases referred to a psychiatrist, neurologist but not a neuropsychologist support that notion (de Jonghe et al., 2019). Therefore, a plausible suggestion would be that incorporating effort measures in any clinical evaluation enhances validity of diagnostic conclusions.

Potential limitations of this study involve the use of archival data with the implication of less control over variables affecting internal validity as compared to, e.g., a known groups design. Secondly, not every participant completed all PVT/SVT and other tests used. In general, sensitivity and specificity differs between PVT/SVT’s and this may lead to inconsistencies in study outcomes. Thirdly, potential selection bias exists as participants were referred for a neuropsychological examination. Not all cases referred to a DCCA psychiatrist or neurologist were subsequently seen by a neuropsychologist, and language barrier was an exclusion criterion in this study. That may have led to overestimation of NRB prevalence. We performed a naturalistic study in a specific setting where dramatic symptom presentations can be seen. It seems impossible to always complete a lengthy assessment procedure as some participants may simply stop taking a particular test. However, more than one PVT/SVT was used in the vast majority of cases, thereby underlining reliability of results. Secondly, a pilot study of all referrals to either a DCCA based neurologist (n = 26) or psychiatrist (n = 17) showed that 47-81% of cases overreported depression on the ADI, and 26-36% overreported symptoms on a SIMS short version (de Jonghe et al., 2019). None of these cases were referred to a neuropsychologist. Thirdly, a pilot study at DCCA included neuropsychological referrals (n = 23) with language barrier or illiteracy (de Winter, 2017) [33]. NRB was present in 73.9% of all cases. So, although selection bias cannot be ruled out, preliminary findings on prevalent NRB in cases not included in the study seem to indicate similar outcomes, thereby supporting internal validity of this study.

An important diagnostic question is whether complaints reflects real symptoms or not. This is one of few studies examining response bias in a Dutch disability and liability claim setting. The use of several performance and symptoms validity tests proved to be helpful in determining NRB prevalence rates in people referred with an established or suspected clinical diagnosis. Considering the potential confounding effects and costs associated with NRB, it is evident that no psychological assessment, and clinical evaluation or relevant research endeavor for that matter, may be complete without adequate control measures of cognitive test performance and self-reported mental problems.