-
Quality assessment of expert answers to lay questions about cystic fibrosis from various language zones in Europe: the ECORN-CF project
Background:
The European Centres of Reference Network for Cystic Fibrosis (ECORN-CF) established an Internet forum which provides the opportunity for CF patients and other interested people to ask experts questions about CF in their mother language. The objectives of this study were to: 1. develop a detailed quality assessment tool to analyze quality of expert answers, 2. evaluate the intra- and inter-rater agreement of this tool, and 3. explore changes in the quality of expert answers over the time frame of the project.
Methods:
The quality assessment tool was developed by an expert panel. Five experts within the ECORN-CF project used the quality assessment tool to analyze the quality of 108 expert answers published on ECORN-CF from six language zones. 25 expert answers were scored at two time points, one year apart. Quality of answers was also assessed at an early and later period of the project. Individual rater scores and group mean scores were analyzed for each expert answer.
Results:
A scoring system and training manual were developed analyzing two quality categories of answers: content and formal quality. For content quality, the grades based on group mean scores for all raters showed substantial agreement between two time points, however this was not the case for the grades based on individual rater scores. For formal quality the grades based on group mean scores showed only slight agreement between two time points and there was also poor agreement between time points for the individual grades. The inter-rater agreement for content quality was fair (mean kappa value 0.232+/-0.036, p<0.001) while only slight agreement was observed for the grades of the formal quality (mean kappa value 0.105+/-0.024, p<0.001). The quality of expert answers was rated high (four language zones) or satisfactory (two language zones) and did not change over time.
Conclusions:
The quality assessment tool described in this study was feasible and reliable when content quality was assessed by a group of raters. Within ECORN-CF, the tool will help ensure that CF patients all over Europe have equal possibility of access to high quality expert advice on their illness.
-
Crisis-repair sequences - considerations on the classification and assessment of breaches in the therapeutic relationship
Background:
Recent research indicates that temporary deteriorations of variables monitored continuously in the course of the therapeutic relationship are important characteristics of psychotherapeutic change. These so-called rupture-repair episodes were assessed by different authors using different mathematical methods.
Methods:
The study deals with the criteria for identifying rupture-repair episodes that have been established in previous studies. It proposes modifications of these criteria which prospectively could make it possible to identify rupture-repair episodes more precisely and consistently. The authors developed an alternative criterion. This criterion is able to include crisis patterns which had not been considered before, as well as to characterize the length of the crises. As a sample application, the different criteria were applied to continuously measured assessments of the therapeutic interaction in psychodynamic therapy courses (ten shorter processes and one long-term therapy).
Results:
The analysis revealed that the number of the identified rupture-repair episodes differed depending on the criterion that was used. Considerably more crises were identified with the newly developed criterion. The authors developed a classification of crisis patterns. They distinguished five patterns of crises and their resolution in therapy processes and ascertained the frequency of distribution. The most frequent pattern was the simple V-shape. The second most common pattern was a decline over more than one session with a sudden repair. The longest downward trend comprised a period of six sessions.
Conclusions:
The findings of the study give insight into basic mechanisms of change within the therapeutic relationship. A phenomenological discussion of how a crisis is defined is useful to create a methodological approach to the operationalization of crises, to differentiate specific characteristics and to specifically link these characteristics to the outcome in future studies. The methodological deliberations might be applyable to different research areas where the analysis of fluctuations in a variable of interest over time is relevant.
-
Enhanced secondary analysis of survival data: reconstructing the data from published Kaplan-Meier survival curves
Background:
The results of Randomized Controlled Trials (RCTs) on time-to-event outcomes that are usually reported are median time to events and Cox Hazard Ratio. These do not constitute the sufficient statistics required for meta-analysis or cost-effectiveness analysis, and their use in secondary analyses requires strong assumptions that may not have been adequately tested. In order to enhance the quality of secondary data analyses, we propose a method which derives from the published Kaplan Meier survival curves a close approximation to the original individual patient time-to-event data from which they were generated.
Methods:
We develop an algorithm that maps from digitised curves back to KM data by finding numerical solutions to the inverted KM equations, using where available information on number of events and numbers at risk. The reproducibility and accuracy of survival probabilities, median survival times and hazard ratios based on reconstructed KM data was assessed by comparing published statistics (survival probabilities, medians and hazard ratios) with statistics based on repeated reconstructions by multiple observers.
Results:
The validation exercise established there was no material systematic error and that there was a high degree of reproducibility for all statistics. Accuracy was excellent for survival probabilities and medians, for hazard ratios reasonable accuracy can only be obtained if at least numbers at risk or total number of events are reported.
Conclusion:
The algorithm is a reliable tool for meta-analysis and cost-effectiveness analyses of RCTs reporting time-to-event data. It is recommended that all RCTs should report information on numbers at risk and total number of events alongside KM curves.
-
Improvement of maternal Aboriginality in NSW birth data
Background:
The Indigenous population of Australia was estimated as 2.5% and under-reported. The aim of this study is to improve statistical ascertainment of Aboriginal women giving birth in New South Wales.
Methods:
This study was based on linked birth data from the Midwives Data Collection (MDC) and the Registry of Births Deaths and Marriages (RBDM) of New South Wales (NSW). Data linkage was performed by the Centre for Health Record Linkage (CHeReL) for births in NSW for the period January 2001 to December 2005. The accuracy of maternal Aboriginal status in the MDC and RBDM was assessed by consistency, sensitivity and specificity. A new statistical variable, ASV, or Aboriginal Statistical Variable, was constructed based on Indigenous identification in both datasets. The ASV was assessed by comparing numbers and percentages of births to Aboriginal mothers with the estimates by capture-recapture analysis.
Results:
Maternal Aboriginal status was under-ascertained in both the MDC and RBDM. The ASV significantly increased ascertainment of Aboriginal women giving birth and decreased the number of missing cases. The proportion of births to Aboriginal mothers in the non-registered birth group was significantly higher than in the registered group.
Conclusions:
Linking birth data collections is a feasible method to improve the statistical ascertainment of Aboriginal women giving birth in NSW. This has ramifications for the ascertainment of babies of Aboriginal mothers and the targeting of appropriate services in pregnancy and early childhood.Key words:Birth, Aboriginality, data, Australia
-
'You give us Rangoli, we give you talk' - Using an art-based activity to elicit data from a seldom heard group
Background:
The exclusion from health research of groups most affected by poor health is an issue not only of poor science, but also of ethics and social justice. Even if exclusion is inadvertent and unplanned, policy makers will be uninformed by the data and experiences of these groups. The effect on the allocation of resources is likely to be an exacerbation of health inequalities.
Methods:
We subject to critical analysis the notion that certain groups, by virtue of sharing a particular identity, are inaccessible to researchers - a phenomenon often problematically referred to as 'hard to reach'. We use the term 'seldom heard' to move the emphasis from a perceived innate characteristic of these groups to a consideration of the methods we choose as researchers. Drawing on a study exploring the intersections of faith, culture, health and food, we describe a process of recruitment, data collection and analysis in which we sought to overcome barriers to participation. As we were interested in the voices of South Asian women, many of whom are largely invisible in public life, we selected a method of data collection which was culturally in tune with the women's lives and values. A collaborative activity mirroring food preparation provided a focus for talk and created an environment conducive to data collection. We discuss the importance of what we term 'shoe leather research' which involves visiting the local area, meeting potential gatekeepers, and attending public events in order to develop our profile as researchers in the community. We examine issues of ethics, data quality, management and analysis which were raised by our choice of method.SummaryIn order to work towards a more theoretical understanding of how material, social and cultural factors are connected and influence each other in ways that have effects on health, researchers must attend to the quality of the data they collect to generate finely grained and contextually relevant findings. This in turn will inform the design of culturally sensitive health care services. To achieve this, researchers need to consider methods of recruitment; the makeup of the research team; issues of gender, faith and culture; and data quality, management and analysis.
-
Prediction intervals for future BMI values of individual children - a non-parametric approach by quantile boosting
Background:
The construction of prediction intervals (PIs) for future body mass index (BMI) values of individual children based on a recent German birth cohort study with n =2007 children is problematic for standard parametric approaches, as the BMI distribution in childhood is typically skewed depending on age.
Methods:
We avoid distributional assumptions by directly modelling the borders of PIs by additive quantile regression, estimated by boosting. We point out the concept of conditional coverage to prove the accuracy of PIs. As conditional coverage can hardly be evaluated in practical applications, we conduct a simulation study before fitting child- and covariate-specific PIs for future BMI values and BMI patterns for the present data.
Results:
The results of our simulation study suggest that PIs fitted by quantile boosting cover future observations with the predefined coverage probability and outperform the benchmark approach. For the prediction of future BMI values, quantile boosting automatically selects informative covariates and adapts to the age-specific skewness of the BMI distribution. The lengths of the estimated PIs are child-specific and increase, as expected, with the age of the child.
Conclusions:
Quantile boosting is a promising approach to construct PIs with correct conditional coverage in a non-parametric way. It is in particular suitable for the prediction of BMI patterns depending on covariates, since it provides an interpretable predictor structure, inherent variable selection properties and can even account for longitudinal data structures.
-
Non-linear mixed models in the analysis of mediated longitudinal data with binary outcomes
Background:
Structural equation models (SEMs) provide a general framework for analyzing mediated longitudinal data. However when interest is in the total effect (i.e. direct plus indirect) of a predictor on the binary outcome, alternative statistical techniques such as non-linear mixed models (NLMM) may be preferable, particularly if specific causal pathways are not hypothesized or specialized SEM software is not readily available. The purpose of this paper is to evaluate the performance of the NLMM in a setting where the SEM is presumed optimal.
Methods:
We performed a simulation study to assess the performance of NLMMs relative to SEMs with respect to bias, coverage probability, and power in the analysis of mediated binary longitudinal outcomes. Both logistic and probit models were evaluated. Models were also applied to data from a longitudinal study assessing the impact of alcohol consumption on HIV disease progression.
Results:
For the logistic model, the NLMM adequately estimated the total effect of a repeated predictor on the repeated binary outcome and were similar to the SEM across a variety of scenarios evaluating sample size, effect size, and distributions of direct vs. indirect effects. For the probit model, the NLMM adequately estimated the total effect of the repeated predictor, however, the probit SEM overestimated effects.
Conclusions:
Both logistic and probit NLMMs performed well relative to corresponding SEMs with respect to bias, coverage probability and power. In addition, in the probit setting, the NLMM may produce better estimates of the total effect than the probit SEM, which appeared to overestimate effects.
-
A proof of principle for using adaptive testing in Routine Outcome Monitoring: the efficiency of the Mood and Anxiety Symptoms Questionnaire -Anhedonic Depression CAT
Background:
In Routine Outcome Monitoring (ROM) there is a high demand for short assessments. Computerized Adaptive Testing (CAT) is a promising method for efficient assessment. In this article, the efficiency of a CAT version of the Mood and Anxiety Symptom Questionnaire, - Anhedonic Depression scale (MASQ-AD) for use in ROM was scrutinized in a simulation study.
Methods:
The responses of a large sample of patients (N = 3597) obtained through ROM were used. The psychometric evaluation showed that the items met the requirements for CAT. In the simulations, CATs with several measurement precision requirements were run on the item responses as if they had been collected adaptively.
Results:
CATs employing only a small number of items gave results which, both in terms of depression measurement and criterion validity, were only marginally different from the results of a full MASQ-AD assessment.
Conclusions:
It was concluded that CAT improved the efficiency of the MASQ-AD questionnaire very much. The strengths and limitations of the application of CAT in ROM are discussed.
-
Comparing multiple competing interventions in the absence of randomized trials using clinical risk-benefit analysis
Background:
To demonstrate the use of risk-benefit analysis for comparing multiple competing interventions in the absence of randomized trials, we applied this approach to the evaluation of five anticoagulants to prevent thrombosis in patients undergoing orthopedic surgery.
Methods:
Using a cost-effectiveness approach from a clinical perspective (i.e. risk benefit analysis) we compared thromboprophylaxis with warfarin, low molecular weight heparin, unfractionated heparin, fondaparinux or ximelagatran in patients undergoing major orthopedic surgery, with sub-analyses according to surgery type. Proportions and variances of events defining risk (major bleeding) and benefit (thrombosis averted) were obtained through a meta-analysis and used to define beta distributions. Monte Carlo simulations were conducted and used to calculate incremental risks, benefits, and risk-benefit ratios. Finally, net clinical benefit was calculated for all replications across a range of risk-benefit acceptability thresholds, with a reference range obtained by estimating the case fatality rate - ratio of thrombosis to bleeding.
Results:
The analysis showed that compared to placebo ximelagatran was superior to other options but final results were influenced by type of surgery, since ximelagatran was superior in total knee replacement but not in total hip replacement.
Conclusions:
Using simulation and economic techniques we demonstrate a method that allows comparing multiple competing interventions in the absence of randomized trials with multiple arms by determining the option with the best risk-benefit profile. It can be helpful in clinical decision making since it incorporates risk, benefit, and personal risk acceptance.
-
Analysis of human immune responses in
quasi-experimental settings: tutorial in biostatistics
Background:
Human immunology is a growing field of research in which experimental, clinical, and analytical methods of many life science disciplines are utilized. Classic epidemiological study designs, including observational longitudinal birth cohort studies, offer strong potential for gaining new knowledge and insights into immune response to pathogens in humans. However, rigorous discussion of methodological issues related to designs and statistical analysis that are appropriate for longitudinal studies is lacking.
Methods:
In this communication we address key questions of quality and validity of traditional and recently developed statistical tools applied to measures of immune responses. For this purpose we use data on humoral immune response (IR) associated with the first cryptosporidial diarrhea in a birth cohort of children residing in an urban slum in south India. The main objective is to detect the difference and derive inferences for a change in IR measured at two time points, before (pre) and after (post) an event of interest. We illustrate the use and interpretation of analytical and data visualization techniques including generalized linear and additive models, data-driven smoothing, and combinations of box-, scatter-, and needle-plots.
Results:
We provide step-by-step instructions for conducting a thorough and relatively simple analytical investigation, describe the challenges and pitfalls, and offer practical solutions for comprehensive examination of data. We illustrate how the assumption of time irrelevance can be handled in a study with a pre- post- design. We demonstrate how one can study the dynamics of IR in humans by considering the timing of response following an event of interest and seasonal fluctuation of exposure by proper alignment of time of measurements. This alignment of calendar time of measurements and a child's age at the event of interest allows us to explore interactions between IR, seasonal exposures and age at first infection.
Conclusions:
The use of traditional statistical techniques to analyze immunological data derived from observational human studies can result in a loss of important information. Detailed analysis using well-tailored techniques allows the depiction of new features of immune response to a pathogen in longitudinal studies in humans. The proposed staged approach has prominent implications for future study designs and analyses.
|
