Individual research projects

Host Institution: Université Paris Cité

Supervisors

Main supervisor: Raphaël Porcher (Université Paris Cité)

Co-supervisor: Catrin Tudur Smith (University of Liverpool)

Context

The term “research gap” is not well defined and its meaning can differ according to the research context. In health research, a research gap generally refers to a clinical question for which missing or insufficient information limits the ability to reach a conclusion. It is closely linked to research needs, priorities and evidence-based decision-making. Identification of research gaps has the potential to inform the design and conduct of research, evidence-based decision-making, health policies, and practice. Audiences including consumers, patients, researchers, clinicians, advocacy groups, and funders can also benefit from understanding the current status of research gaps.

Initiatives such as the James Lind Alliance, Cochrane Priority Setting Methods Group, and the Evidence-Based Research Network are three prominent examples of existing efforts to identify gaps and uncertainties in research to improve research, research prioritization and evidence-based decision-making. This project aims to describe the different methods used to identify and display gaps in clinical research and develop methodological guidance.

There are two main frameworks that have explicitly focused on the identification of gaps from literature and systematic reviews. This study aims to advance these efforts by exploring other methods used and / or reported for identifying and displaying research gaps.
guidelines.

Objectives

This Ph.D. project aims to:

Identify the definitions and concepts related to research gaps as reported in scientific articles
Describe the methods for identifying and displaying research gaps
Explore key stakeholders’ perceived needs for, strengths of, and barriers to methods in identifying and displaying research gaps
Conduct a proof of concept study and develop methodological guidance, for identifying and displaying research gaps

Methods

The first stage of the project will be a scoping review to describe methods used and/ or reported for identifying and displaying research gaps in clinical research. In the second stage, a mixed method study, including key informant semi-structured interviews and a quantitative survey will be conducted to determine perceived needs for identifying and displaying research gaps. Finally, the last stage will involve a proof of concept study and the development of methodological guidance for identifying and displaying research gaps.

Expected results

This project will provide an overview of methods used and/ or reported for identifying and displaying research gaps in health research and propose methodological guidance. The project will also produce a validated questionnaire on key stakeholders’ perceived needs for, strengths of, and barriers for methods to identify and display research gaps.

Start date: October 2016

Duration: 36 months

Planned secondments

Host 1: University of Liverpool, 4 months

Host 2: Cochrane, 3 months

Host 3: ECRIN, 2 months

Host Institution: University of Amsterdam

Supervisors

Supervisor: Patrick Bossuyt (University of Amsterdam)

Co-supervisor: Els Goetghebeur (University of Ghent)

Context

In health care, the development and evaluation of innovative medical tests lag behind the evaluation of pharmaceuticals and other interventions. Some of the reasons can be found in the lower evidentiary requirements that are needed for access to the marker, marketing, recommendations and reimbursement. In many countries these requirements are fairly minimal. If any exist, they are typically defined in terms of analytical performance requirements. The need for clinical demonstrating acceptable clinical performance is far less outspoken, and clinical effectiveness – the ability of tests to improve patient outcomes – is rarely demonstrated. As a result, the field can be characterized by sometimes highly optimistic claims paralleled by disappointing clinical results.

One of the reasons for the current suboptimal state of test development and evaluation is the multiphase approach that is typically used. This strategy is inspired by the multiphase strategy for staging clinical trials used in the development of pharmaceuticals. There Phase I studies, in which safety and initial effectiveness are evaluated in small groups of patients, precede Phase II studies, in larger groups of participants, and Phase III trials, typically performed in even larger groups, to document effectiveness and to monitor side-effects.

Several dozen comparable multiphase strategies for test development and evaluation have been proposed. Many of these seem to be inspired by the hierarchy of evidence defined by Fryback in the 1970s for the evaluation of CT imaging. There, Level 1 referred to analytical performance, Level 2 to diagnostic accuracy (sensitivity and specificity), while higher levels included evaluations of the effect on patient outcomes in the target use population.
In the biomarker development pipeline, this multiphase approach has resulted in a clear separation between biomarker discovery, biomarker verification, and biomarker validation, with separate – and typically disjoint – evaluations of the clinical effectiveness of these markers.

This silo approach has been pointed to as one of the main reasons for the failure of the current biomarker pipeline to generate meaningful, clinically useful biomarkers. It has also resulted in the premature introduction of medical test of dubious effectiveness, and discussions about reimbursement.
This situation testifies of the need for a thorough critique of the current multiphase approach to test evaluation, and the exploration of an alternative approach, which will break down the silos between the separate phases, allowing potentially a more efficient and more valid approach to test development.

Objectives

This PhD project aims to:

Evaluate success and failures in the current multiphase approach at test development.
Develop an alternative strategy, one that more closely integrates analytical performance requirements, clinical performance specifications, and clinical effectiveness (effect on patient outcomes from testing).
Apply that alternative strategy to specific areas of testing.

Methods

The first partof the project will be based on a critical appraisal of a set of markers and tests for a well-defined area. The area of testing will be selected based on the knowledge and interests of the student who runs the project, and the possibilities for collaboration with clinical colleagues. We will evaluate a number of clinical trials in which markers where evaluated, and we will use the typology proposed by dr Diamandis to explain why novel protein biomarkers for patients with ovarian cancer failed to reach the clinic (BMC Medicine 2012; 10:87). Diamandis distinguished between reports that could be classified as cases of fraud (rare), true discoveries of markers but with disappointing performance, and false discoveries, resulting from inappropriate methodology.

In the second part of this project we will develop the circular approach to test development and evaluation, proposed by dr Horvath and colleagues in a working group for the European Federation of Laboratory Medicine (ClinicaChimicaActa 2014; 427:49–57). These laboratory professionals proposed an approach where the intended use of the novel test, in a well-defined care pathway, is central in evaluations of analytical performance, clinical performance, clinical effectiveness and cost-effectiveness. By positioning the intended use in a focus of all evaluations. Required levels of performance could be defined and studies to evaluate whether tests meet these requirements can be designed appropriately.

This approach resembles what has been labeled as a linked evidence approach, where evidence of the effectiveness of downstream actions, guided by the test results, is combined with other pieces of information, such as the distribution of test results in the target use population, for defining acceptability criteria. In this approach, analytical and clinical performance criteria become necessary conditions for the test to be useful. This is in essence, a multiphase approach where the phases are not isolated but connected with an overarching “fit-for-purpose” question.

In the final part of this project we will apply this approach, guided by targeted use, to a number of areas in health care where novel tests are highly needed and/or being developed. Here also decisions about the actual clinical area will be guided by the specific interests of the student and the possibilities for fruitful collaboration with the partners in MiRoR. The applications are not restricted to clinical medicine, but could extend to population-based screening programs and risk stratification of asymptomatic individuals.

Expected results

One or more manuscripts resulting from this project will be systematic analyses of biomarker failures and successes in specific health care areas. The central manuscript will describe a worked out strategy for the implementation of a circular approach to test development guided by the intended use, which is different from the dominant linear approach, with clearly separated phases. Additional manuscripts will apply that strategy to a number of areas in health care where novel tests are eagerly awaited or in development, such as in applications of personalized and precision medicine.

Start date: October 2016

Duration: 36 months

Planned secondments

Host 1: University Paris Descartes, 4 months, studying their current practice of literature review and writing a protocol

Host 2: NICE (M25), 3 months, to map existing practices into the projects evaluated in the NICE Diagnostics Assessment Programme

Host 3: University of Ghent, 3 months, to include causal modelling in the analysis of design effects on biomarker discovery studies

Host Institution: University of Liverpool

Supervisors

Supervisors: Paula Williamson and Bridget Young (University of Liverpool)

Co-supervisor: Philippe Ravaud (University Paris Descartes)

Context

The COMET (Core Outcome Measures in Effectiveness Trials) Initiative brings together people interested in the development and application of agreed standardised sets of outcomes, known as ‘core outcome sets’ (COS). These sets represent the minimum that should be measured and reported in all clinical trials of a specific condition, and are also suitable for use in clinical audit or research other than randomised trials. The existence or use of a COS does not imply that outcomes in a particular trial should be restricted to those in the relevant core outcome set. Rather, there is an expectation that the core outcomes will be collected and reported, making it easier for the results of trials to be compared, contrasted and combined as appropriate; while researchers continue to explore other outcomes as well. COMET aims to collate and stimulate relevant resources, both applied and methodological, to facilitate exchange of ideas and information, and to foster methodological research in this area.

COS need to include those outcomes that are most relevant to patients and carers, so it is vital that patients and carers are involved in their development. There are examples of where involving patients in the process identified an outcome that was important to them as a group but which might have been overlooked if the outcome set was developed by practitioners on their own.

Researchers are increasingly including patients and the public alongside other stakeholders in identifying what outcomes (such as pain, quality of life, etc.) to measure in clinical trials, but no one knows how best to do this. Whilst only 22% of published COS reported that there was input from patients in their development, in ongoing studies nearly 90% include patients as participants. The question now is not whether patients should participate, but rather the nature of that participation.

There are numerous challenges in facilitating patient participation in a COS study and these will depend on the patient group and the methods chosen. This studentship will explore some key challenges for patient participation in consensus processes, including finding the right language to explain COS studies, strategies to maintain the input of patients over time, enabling the inclusion of patients in face to face meetings with health professionals, and developing end of study information for patients.

This project will compare the ways that patients have been included in outcome selection across several clinical areas and whether the current methods are fit for purpose. The findings will inform guidance and interventions to enhance the ways that COS developers include patients in outcome selection in future studies.

Objectives

Core outcome set (COS) developers face particular challenges in achieving consensus among diverse stakeholder groups about what outcomes to include in a COS. The project will investigate the ways in which patients have been involved as participants in COS studies including consensus development and whether the current methods of involvement are fit for purpose (i.e. adequately capture the perspectives of patients). The project will also investigate whether current methods used when including patients are considered acceptable to them.

Methods

Systematic review of methods of including patients as participants in (i) trial and clinical guideline outcome selection and (ii) core outcome set development: Previous reviews of patient participation in COS studies only involved extraction of information from COS publications, and there were gaps in reporting. There was also quite a low percentage of COS studies (22%) where patients were included. This has been increasing over time however thus increasing the number of relevant studies. The new review would involve contact with the COS developers to find out more about the methods they used to include patients as participants.

Interviews with patients and health professionals involved as participants in core outcome set development: It will be important to find out from patients their views on the methods used in the studies they were involved in, to compare various aspects but also to try to pull out what may be seen to be good practice. This will be achieved using patient interviews.

Comparison of methods of patient participation: The work above will identify methods of interest for further comparative study, including those particularly aimed at involving a larger number of patients from different countries. The methods will be compared in at least one condition to see whether they produce different results. In addition, how well the methods perform in terms of facilitating diverse samples in reporting on their perspectives about outcomes will be assessed.

Expected results

Guidance will be developed on ways to address key challenges and enhance the involvement of patients in COS development. A manuscript will be written reviewing current methods of including patients as participants. A study will be undertaken comparing at least two methods of interest, including those particularly aimed at involving a larger number of patients from different countries, and submitted for publication.

Start date: October 2016

Duration: 36 months

Planned secondments

Host 1: University Paris Descartes, 5 months, to study their approach of COS

Host 2: ECRIN, 2 months, to explore the impact for trialists

Host 3: NICE, 3 months, to impact for guidelines developers

Host Institution: University of Liverpool

Supervisors

Supervisors: Carrol Gamble and Paula Williamson (University of Liverpool)

Co-supervisor: Roser Rius (Universitat Politecnica de Catalunya)

Context

Successfully recruiting the pre-specified number of patients to time and target within clinical trials remains a difficult challenge that negatively impacts all stakeholders in a clinical trial. Recruitment problems have practical and financial impacts, delaying completion of research or reducing its timely impact on patient health and well-being. Despite significant investment in infrastructure to support clinical trials the challenge of recruitment remains. A Delphi survey of clinical trial units in the UK identified methods for boosting recruitment as the number one priority area for methodological research.

Little is known about how applicants requesting funding for clinical trials estimate the recruitment period for the proposed trial. If there is an absence of information, or the information used is unreliable then the ensuing predictions will often be inaccurate. Extensions to length of recruitment time and funding are therefore frequently required.

In a systematic review of models to predict recruitment to multicentre trials, five major classes of models were identified from eight studies. The models identified were: unconditional; conditional; Poisson; Bayesian; and Monte Carlo Simulation Markov Model. Barnard reported that the majority of trials adopted the unconditional approach to recruitment prediction suggesting that all their planned centres will start recruiting to their maximum capacity on day one. They concluded that across the models the focus was restricted to patient accrual with insufficient account taken of recruitment of centres with a new model required as a matter of importance.

When the trial is underway, monitoring recruitment is of key importance to ensure that the target sample size can be achieved. Current methods to monitor recruitment in practice appear limited to the visual comparison of the predicted and actual recruitment curves and the size of the discrepancy. However, additional metrics could be beneficial to monitor recruitment progress against milestones from the perspectives of funders, CTUs, and trial sites, and while statistical methods have been proposed their application is limited. There is also little known about methods used to adapt recruitment curves when revising recruitment periods or including additional sites. Factors to consider are: the stage at which the decision to revise the curves is considered necessary; the frequency of revisions; and how the information on recruitment to date is incorporated and extrapolated.

Objectives

This Ph.D. project aims to:

Identify, compare and develop statistical methodology used to predict and monitor recruitment in clinical trials,
Develop guidance supported by software with a web-based interface.

Methods

The project will involve conducting systematic reviews on methods to predict, monitor and revise recruitment projections in clinical trials. This will include an update to an existing systematic review (literature search date: July 2008), additional systematic reviews will cover methods for monitoring recruitment and adaptive adjustments to recruitment projections.

Surveys will be used to establish current methods used within clinical trials units to predict, monitor, and adjust patient accrual. These surveys will include identifying the awareness, practical implementation, and barriers to the methods identified within the systematic reviews.

Factors considered to have impacted translation of the predicted recruitment curve to that observed will be identified from ongoing clinical trials. This will include the source and translation of the data underpinning the projection, for example from pilot and feasibility studies or expert opinion, together with establishing additional core factors that should be routinely included in models to predict participant accrual. Recruitment data from across a large portfolio of clinical trials will be used to estimate these core factors to inform model development which may include disease areas, impact of seasonal variation and holiday periods, rate of centre openings, and trial stage on recruitment rates(e.g. evidence of research fatigue).Finally models will be developed and validated against retrospective and prospective clinical trials data.

Expected results

Methods used to predict and monitor recruitment in clinical trials have received little attention. The results of this project will improve the reliability of predictions of recruitment in to clinical trials and methods used to monitor adherence reducing waste in research. Guidelines will be developed on when and how to adjust recruitment graphs and the development of continuously adaptive curves against which to monitor progress along with web-based programmes to allow wide-spread application of the models.

Start date: October 2016

Duration: 36 months

Planned secondments

Host 1: Universitat Politecnica de Catalunya, 5 months, to explore their methods in terms of recruitment

Host 2: ECRIN, 3 months, to evaluate the possible implementation for trialists

Host Institution: Université Paris Cité

Supervisors

Supervisor: Isabelle Boutron (Université Paris Cité)

Co-supervisor: Bridget Young (University of Liverpool)

Context

How clinical research is actually performed has recently been questioned, and a series of articles published in The Lancet demonstrates the enormous problem of waste in producing and reporting research evidence. According to Chalmers, up to 85% of all research investments may be wasted. A large part of waste is related to inadequate planning. Flaws in design can bias results of randomised controlled trials and the systematic reviews that include them, thus leading to potentially erroneous conclusions with serious consequences for patients. Empirical evidence found exaggerated estimates of intervention effect in trials with inadequate sequence generation or allocation concealment, lack of blinding, or exclusion of patients from analyses. Previous studies showed that this waste could be avoided in part by simple and inexpensive methodological adjustments during the planning stage. Similarly inadequate design raises questions related to the applicability of the trial results.

Collective intelligence is defined as a shared or group intelligence that emerges from the collaboration, collective efforts, and competition of many individuals. Collective intelligence is already used in various disciplines. However, it has never been proposed to improve the planning of clinical research.

Objectives

This Ph.D. project aims to:

Determine whether and how mobilising collective intelligence could be used in the planning of a randomized controlled trial.

Methods

We will proceed in 3 steps:

– Identification of the methods used to involve collective intelligence and development of a typology of these methods. The ESR 5 will perform a systematic review. A variety of standard databases (e.g., Cochrane Central Register of Controlled Trials (CENTRAL), PubMed, EMBASE) but also full-text databases (e.g. Google Scholar) will be searched, and researchers involved in this field will be contacted.

– Identification of the barriers and facilitators for involving collective intelligence. The ESR 5 will perform qualitative studies to identify the barriers and facilitators for involving collective intelligence in the context of the planning of a clinical trial. It will benefit from the expertise in qualitative studies of the teams from Liverpool University and Split University.

– Assessment of the impact of mobilising collective intelligence on the planning of a randomized controlled trial. The ESR 5 will use case vignette of protocols of randomised controlled trials to evaluate and compare the impact of different methods mobilising collective intelligence on the planning of the trial. The outcome will be the changes of the protocol.

Expected results

This project will propose recommendations for the use of innovative methods to improve the planning of clinical trials. The ESR will benefit from the expertise of the consortium in the planning and conduct of clinical trials. The ESR will also have the opportunity, during a secondment at ECRIN – European Clinical Research Infrastructure Network – to explore the impact of these methods on research projects.

Start date: October 2016

Duration: 36 months

Planned secondments

Host 1: University of Split, 3 months, to develop the protocol and conduct the qualitative study

Host 2: University of Liverpool, 5 months, to study their practice in planning clinical trials

Host 3: ECRIN, 2 months, to explore the future implementation by trialists

Host Institution: University of Ghent

Supervisors

Supervisor: Els Goetghebeur (University of Ghent)

Co-supervisor: Aeilko Zwinderman (University of Amsterdam)

Context

The randomized controlled trial is the classic gold standard design when it comes to drawing conclusions on causal effects of exposures. Whilst it has the uncontested advantage of avoiding bias in the studied population, it also has serious limitations. The design is infeasible or unethical for some exposures, while feasible randomized trials are typically conducted in restricted, fairly homogeneous populations. Historically, far more male than female subjects were for instance studied. To learn about effects in the relevant broader population, observational studies are increasingly important. The US Institute of Medicine, showed that 85% of the comparative effectiveness research comes from observational studies. The key problem then lies in overcoming confounding between treatment (often prescribed by indication) and baseline covariates.

Over the past decades, important new methods for causal inference have entered the epidemiologist’s toolkit. The use of suboptimal methodology in a specific context may lead to added complexity, unclear interpretability and reduced accuracy. For instance, when absence of unmeasured confounders is assumed, available techniques include outcome regression models, matching based on propensity scores, inverse probability weighting, principal stratification, etc.

Corresponding estimators may differ in their direct target of estimation, in their definition of exposure, in the (sub)population to which the effect pertains, in the further assumptions they rely on and in their properties at the level of bias and precision. The best method depends therefore on context, on the analysis goal and on the structure and size of the available database. While the critical assumption of `no unmeasured confounders’ cannot be checked on the data, components of the methods involve models which should fit the observational data. All else being equal one may wish to impose as few additional assumptions as possible, or otherwise balance robustness and precision against the ease and transparency of the technique.

Differences in interpretation of the causal effect measure may appear subtle, but are real and can be practically significant. They become more pronounced when true treatment effects depend on patient characteristics, a feature which in its own right may be exploited in precision medicine. All this enters in nontrivial ways in the evaluation of point exposure or time constant treatments, it becomes more complex when observed treatment regimes vary over time in function of observed covariates.

Finally, it is important to acknowledge that selective treatment choices may in practice occur in tandem with missing data elements which themselves can be subject to selection bias. There is a parallel in the principles and methods for incorporating selective exposure and selective missingness.
In this project, we will examine the current state of reporting on causal effects and the causal inference methods used with observational studies in the medical literature. Representative examples will serve as test cases to help clarify the meaning of the estimated causal effect in context and the relative value of the different methods of analysis available. A guidance document will be derived to help support the method of choice for a given research set-up.

Objectives

This Ph.D. project aims to:

Examine causal claims based on observational data in the medical literature, in terms of the method used, acknowledgment and justification of the assumptions invoked, interpretation of the effect and the reporting of method and results.
Develop case studies demonstrating the value of an effective choice of method – and lack thereof – under typical scenarios for two specific disease areas representing long term as well as short term treatments.
Provide guidance in the choice of causal inference method and expression of conclusions in a range of typical set-ups

Methods

The project starts with a review of the main types of methods currently available for causal effect estimation,both under the assumption of no unmeasured confounders and in the presence of instrumental variables. These methods will be compared in terms of the specific assumptions they rely on, their direct target of estimation and statistical properties with drawback and advantages in different settings. This will first be done for point exposures, and in a second instance for time-varying treatments

We will identify 2 distinct disease areas for which an important treatment has been evaluated over the past decade, with serious reliance on observational studies. We will consider one treatment in a chronic disease setting with long term follow-up (e.g. statins for high cholesterol or increased risk of a heart attack) and one where a shorter curative intervention is envisaged. The specific choice will be guided by current scientific interest, background and interest of the student and the expertise of the supervising team.

The set of methods used and conclusions drawn will be reviewed and summarized. Results will be compared among observational studies and with those of clinical trials.Intrinsic explanations will be sought for any systematic differences discovered and we will look for meaningful comparisons of obtained effects through derived standardized measures with shared interpretation. When some study results are available only in marginalized form over the study-specific population, i.e. in more coarse resolution than necessary for the envisaged standardization, we will reconstruct the desired standardized result by first estimating expected results in finer strata incorporating auxiliary covariate information from the study at hand, as well as established covariate effect measures.

Similarly we will review what happened in recent years in terms of causal effect claims in the BMJ. Where possible the role of referees and editors in steering the methods to date will be examined.

Expected results

This project will yield insight in the current practice of causal effect analysis in specific areas of clinical research. A guidance document defining adequate analysis and reporting methods for causal inference will be derived. The latter can serve as a tool to perform such monitoring in other disease areas and as a support to authors and referees in medical journals.Finally, new methods will be derived to allow for adequate direct standardization allowing to compare results from different studies that are originally given in different resolution (conditional on more or less finely defined strata).

One or more manuscripts will result from this project. One will focus on identifying the different causal effect methods used in two disease areas and examine whether they are justified and lead to evidence based conclusions that are most relevant for guiding treatment choice in practice. A review paper with critical appraisal of methods frequently used and indications for improvement will follow. A guidance document for further use in similar disease areas is envisaged. Special attention will be given to communicating results to the target audience including both practical statisticians and clinical researchers.

Start date: October 2016

Duration: 36 months

Planned secondments

Host 1: Cochrane, 3 months, to study their current practice of literature review and write a protocol for the proposed review on causal effect methods and presentation of results

Host 2: University of Amsterdam, 4 months, to work out the protocol for 2 well-chosen disease areas

Host 3: BMJ, 3 months, to study their practice of reviewing on causal effects and causal effect methods

Host Institution: Université Paris Cité

Supervisors

Main supervisor: Agnes Dechartres ( Université Paris Cité)

Co-supervisor: Patrick Bossuyt (University of Amsterdam)

Context

Systematic Reviews (SRs) synthesize all existing evidence on an important clinical question, using a precise and reproducible methodology. SRs may be affected by bias at the level of the single studies. Consequently, assessing the Risk of Bias (RoB) of individual studies is a crucial step while conducting a SR.

The Cochrane Collaboration has recently revised the tool for assessing Risk of Bias (RoB) in randomized controlled trials (RCTs), developing the RoB 2.0 Tool. The new version includes important theoretical and procedural changes in the definition and evaluation of bias. In this update, the use of signalling questions and algorithms could lead to a simpler and faster evaluation, while also increase the agreement between reviewers; at the same time the suppression of the “unclear risk” judgment may affect the RoB evaluation of studies and modify conclusions regarding bias in systematic reviews (SRs) and meta-analyses.

Given its recent development, the RoB 2.0 tool has still not been subject of studies evaluating the effect of the variations proposed; the aim of our project is to examine the possible impact of using the new tool in RoB evaluation. We will also explore ways to tutoring/help the assessment, creating a learning tool which could facilitate the use and adoption of the RoB 2.0 tool.

Objectives

This Ph.D. project aims to:

Evaluate how the RoB tool has been used in Cochrane Systematic Reviews;
Assess how will the new RoB tool (RoB 2.0) work in terms of consequences on risk of bias and treatment effect estimate;
Develop and validate a learning tool for the RoB 2.0 tool to explore interventions to improve reproducibility.

Methods

We will proceed in 2 steps:

1) Reproducibility of Risk of Bias assessment in Cochrane Systematic Reviews: We will use a random sample of reviews from a collection of 2796 SRs published between March 2011 and September 2014 from the Cochrane database of systematic reviews. Two researchers will independently re-grade single studies according to the RoB 2.0 tool, using the original manuscripts. Agreement between reviewers will be examined and RoB judgment will be compared to the previous assessment, differences will be highlighted. The in-depth analysis will focus on discordant cases, which will be examined thoughtfully in search of association with certain domains/topics/outcomes, with single-case discussion regarding episodes in which changes result from the revision of the tool.

2) New way to assess risk of bias. The second version of Cochrane tool and its implication: The RoB tool has recently been updated and the RoB 2.0 tool of Cochrane was recently launched. In this project, we will compare assessment with RoB toll 1.0 and RoB tool 2.0. examine how bias has been evaluated in previous works and with a meta-epidemiological model we will evaluate the impact on treatment effect of the new domains from the RoB 2.0 tool.

Expected results

This project aims to explore the issue of reproducibility of RoB evaluation and develop methods to improve reproducibility of this assessment.

Start date: October 2016

Duration: 36 months

Planned secondments

Host 1: University of Amsterdam, 5 months, to study their practice in meta-epidemiology

Host 2: Cochrane, 2 months, to explore the impact of this work on the conduct of meta-analysis

Host 3: EQUATOR Network, 3 months, to explore these approaches

Host Institution: University of Ghent

Supervisors

Supervisor: Stijn Vansteelandt (University of Ghent)

Co-supervisor: Raphaël Porcher (University Paris Descartes)

Context

Systematic reviews and meta-analyses are essential tools to synthesize all available evidence in the field of health, especially therapeutic evaluation. A meta-analysis in particular, is a statistical analysis to combine the results of several studies with the aim to obtain a more accurate estimate as well to gain insight into potential sources of heterogeneity between studies. Summary data meta analyses combine summary (aggregate) data (e.g. treatment effect measures, such as odds ratios, hazard ratios, …) extracted from study publications or obtained from investigators. Individual patient data meta-analyses instead seek the original research data directly from the researchers responsible for each study. These data are then re-analysed centrally and combined, if appropriate.

Three much overlooked statistical problems in meta-analysis are: (a) the fact that different studies may focus on different versions of the same treatment; (b) non-collapsibility and non-transportability of effect measures across studies; and (c) insufficient adjustment for confounding in some observational studies that are part of a larger meta-analysis. Problem (a) arises in the presense of so-called `compound treatments’. For instance, a meta-analysis on the effect of weight loss on mortality in obese people may be complicated by the fact that the different studies focus on different weight loss strategies (e.g. diet, physical exercise, …), each having a different impact on weight loss and thus a different impact on mortality. Problem (b) may arise due to the fact that certain treatment effect measures, such as odds ratios and hazard ratios, tend to suggest stronger treatment effects when they are calculated on more homogenous patient populations; this phenomenon is known as `non-collapsibility’ of effect measures. This may make the effect estimates assessed for one patient population non-transportable to other patient populations. It may thereby invalidate simple strategies to combine data from different research studies. It may moreover suggest or hide heterogeneity between studies, that is then erroneously interpreted as signaling the presence or absence of heterogeity on treatment efficacy. Problem (c) leads to bias in the summary results for some studies. As was the case for problem (b), this may invalidate simple strategies to combine data from different research studies, and may bias assessment of the degree of heterogeneity between studies.

Objectives

This Ph.D. project aims to:

Review how frequently problems of compound treatments, non-collapsibility, non-transportability and differential adjustment for confounding occur in meta-analysis of randomised experiments and observational studies.
Develop meta-analysis approaches for (combinations of) randomised experiments and observational studies that infer the same effect measure across all contributing studies (i.e. they infer the effect of the same `version’ of the treatment for the same population), and thereby delivers valid measures of between-study heterogeneity.
Develop meta-analysis approaches for (combinations of) randomised experiments and observational studies that accommodate the lack of data on some confounder variables in some studies.

Methods

This project will use data from Cochrane reviews published between 2011 and 2014 that we obtained from the Cochrane Collaboration, focusing on 1 or 2 disease areas as specific case studies. We will select all reviews with at least one meta-analysis of at least 3 studies and record whether the aforementioned problems of compound treatments, non-collapsibility, non-transportability and differential adjustment for confounding occur.

We will next develop direct standardisation methods for meta-analyses of individual patient data from randomised experiments in order to ensure that the same effect measure is inferred for the same population across all contributing studies. In particular, we will standardise the results for a given study to either the combined population across all studies, or the study population of specific trials. We will develop a new class of models for the vector of intention-to-treat effects, standardised to the populations from each of the considered trials, along with accompanying estimation methods. The proposed models will include 2 random effects: one to capture heterogeneity between the results from a given study, as they are standardised to the populations from the different trials; and one to capture heterogeneity between the results of different studies standardised to the same population. The first source of heterogeneity reflects heterogeneity in the effect of the same treatment when applied to different patient populations. The second source of heterogeneity reflects heterogeneity in the effect of different versions of the treatment (e.g. due to noncompliance, …) applied to the same population. Through the analysis of various case studies, we will develop insight into the relative magnitudes of the corresponding variance components. The results will be generalised to meta-analyses of combinations of randomised experiments and observational studies.

We will finally adapt methods from the missing data literature to handle lack of data on some confounder variables in some studies. We will propose novel extrapolation methods. These will use the studies with rich covariate data to evaluate the impact of using fewer variables in the analysis on the resulting treatment effect estimates; on the basis of the findings, the results obtained for studies with data on few variables will then be extrapolated to a hypothetical setting where more covariates were recorded.

Based on the methodological findings, guidance will be developed on the level of information that is ideally provided in published papers.

Expected results

This project will lead to a better understanding of the extent to which published meta-analyses suffer from problems due to compound treatments, non-collapsibility, non-transportability and differential adjustment for confounding occur. In addition, it will lead to novel statistical methods for meta-analysis.

These methods will lead to less biased meta-analysis results and more accurate assessments of between-study heterogeneity, by ensuring that the different considered studies infer the same effect measure across all contributing studies, and adjust for the same variables. The results of this project could therefore have a major impact on routine strategies to conduct meta-analyses. This impact will be evaluated via the re-analysis of one or more published meta-analyses.

We foresee that the research will lead to one publication that reviews the use of meta-analysis, and several methodological publications on novel statistical methods for meta-analysis.

Start date: October 2016

Duration: 36 months

Planned secondments

Host 1: University Paris Descartes, 6 months, to build on the approach to combine treatment effects measures in clinical trials

Host 2: Cochrane, 2 months, to study the practice of supplying and requesting supplementary information

Host 3: University of Amsterdam, 2 months, to learn from the practice of presentingausal inference results in papers

Host Institution: University of Split

Supervisors

Supervisor: Darko Hren (University of Split)

Co-supervisor: Guadalupe Gomez (Universitat Politecnica de Catalunya)

Context

Recommendations on the reporting of research are developed to improve reporting quality, thus facilitating readers’ understanding, critical assessment, and decision whether and how results can be included in systematic reviews. The STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) Statement was developed to provide guidance on adequate reporting of observational research. It is a checklist of items that should be addressed in articles reporting on the 3 main study designs of analytical epidemiology: cohort, case-control, and cross-sectional studies.

Much of medical research is observational and the practical value of STROBE can be extended beyond simple reporting guidelines to provide a framework for teaching the principles of scientific research and reporting in medicine. This is particularly important for students and novice researchers who lack experience and may have trouble understanding underlying connections between those aspects. Using these guidelines as a basis for an educational intervention can facilitate learning enhance the user’s experience with STROBE.

Objectives

The PhD will be structured with 4 main objectives

Project 1
1. To classify changes made in the extensions to identify strengths and weaknesses of the original STROBE checklist.
2. To determine the prevalence and typology of endorsement by journals in fields related to extensions.

Project 2
1. To assess current researcher’s use, self-perceived ability to properly use the checklist (self-efficacy), knowledge and understanding of the STROBE checklist.

Project 3
1. To develop and evaluate the impact of a structured educational intervention based on the STROBE statement for teaching observational research methods and reporting.

Methods

Project 1 comprises two different study designs. First, a qualitative analysis focused on the additions made within the extensions will be conducted. Second, a cross-sectional bibliometric study will be conducted to determine the prevalence and typology of endorsement by journals.
Project 2 is a cross-sectional study that will utilize convenience sampling online to attempt to reach current authors in the field of biomedical research.
Project 3 will develop an online course for observational studies.

Expected results

The qualitative assessment of the established extensions to STROBE will inform the content and level of specificity needed for the educational intervention. While investigating endorsement aims to provide insight into the uptake of STROBE and its extensions and will create a pool of journals to survey for the second project.
The second project, a survey for authors of observational studies, will describe how authors currently interact with and feel towards STROBE. These attitudes and beliefs, combined with the content information gathered in study one will then help inform the educational intervention based on STROBE.

Start date: October 2016

Duration: 36 months

Planned secondments

Host 1: EQUATOR Network, 3 months, to refine the survey for authors of observational studies

Host 2: University Paris Descartes, 2 months, to develop the intervention

Host 3: Universitat Politecnica de Catalunya, 5 months, to analyze and write up results

Host Institution: University of Amsterdam

Supervisors

Supervisor: Patrick Bossuyt (University of Amsterdam)

Co-supervisor: Isabelle Boutron (University Paris Descartes)

Context

Accurate presentation of the results of clinical research is the cornerstone of the dissemination of the results and their implementation in clinical practice. The Declaration of Helsinki states that “Authors have a duty to make publicly available the results of their research on human subjects and are accountable for the completeness and accuracy of their reports.”

In practice, investigators usually have broad latitude in writing their articles; they can choose which data to report, how to report them, and how to interpret the results. Consequently, scientific articles are not simply reports of facts, and authors have many opportunities to consciously or subconsciously shape the impression of their results for readers, that is, to add “spin” to their scientific report. The use of spin in scientific writing can result from ignorance of the scientific issue, unconscious bias, or willful intent to deceive.

“Spin” can be defined as specific reporting that could distort the interpretation of results and mislead readers. The most disturbing form would be to suggest that health care interventions are beneficial, despite the absence of convincing evidence. Such “spin” has been documented in the reporting of RCTs, diagnostic accuracy studies, systematic reviews, and other forms of evaluation.

Objectives

The aim of this project is to identify, develop and evaluate strategies for preventing, eliminating, or limiting “spin” in clinical research.

Methods

A structured search of the literature to identify active strategies for reducing “spin” in scientific reporting,
A series of systematic reviews of active strategies to prevent, eliminate or limit “spin” in clinical research,
A field trial comparing editorial strategies to limit “spin” in reports from clinical trials of health care interventions.

Expected results

• A structured report describing proposals for active measures to limit “spin” in reports of clinical research, or to curb its effects ;
• A series of reports of systematic reviews on the effectiveness of active strategies to limit “spin” ;
• An overview of these reviews, with documentation of the comparative effectiveness of these strategies, and analyses of likely determinants of the variability in the effectiveness of these strategies ;
• A guidance document, with recommendations for different stakeholders to limit “spin” and the effects of “spin” in evaluations of the effectiveness of health care interventions.

Start date: October 2016

Duration: 36 months

Planned secondments

Host 1: University Paris Descartes, 4 months, to extend and strengthen the systematic review of strategies to limit “spin”

Host 2: BMJ, 4 months, to prepare for a field trial of strategies to limit “spin” in reports from clinical trials of health care interventions

Host 3: Cochrane, 2 months, to prepare for a field trial of strategies to limit “spin” in systematic reviews of health care interventions

Host Institution: CNRS

Supervisors

Main supervisor: Patrick Paroubek (CNRS)

Co-supervisor: Patrick Bossuyt (University of Amsterdam)

Context

Reporting is a crucial activity of scientific research. An important aspect of scientific reports is the claims that they contain. These can be drawn from references to other works through citation, as well as from the analysis of the theoretical and experimental results of the reported study. The complexity of the task and a variety of other factors are sometimes responsible for the presence of inadequate reporting in published studies that may escape the scrutiny of authors, peer-reviewers, editors and readers. Scientific misinformation, which can be aggravated over successive citations, must be avoided, in particular when scientific reports can be at the origin of important decisions, for instance in clinical practice.

Growing concern about the effects of inadequate reporting has recently motivated studies in the field of “research on research”. Distorted presentation of results following a variety of strategies, or “spins”, has for instance been shown to be present in as many as 60% of a set of abstract conclusions of randomized controlled trials. There is thus a need to assist scientific authors in identifying possible instances of inadequate claims in their reports. The hypothesis explored in this research project is that Natural Language Processing techniques can be leveraged to implement efficient interactive strategies to avoid inadequate reporting.

Objective

This PhD project aims to:

Provide scientific authors with tools that can support the self-assessment of various causes for inadequate claims. For this purpose,
Develop Natural Language Processing techniques for (a) identifying important claims in scientific reports and (b) extracting candidate supporting information,
Implement an enriched authoring mode using the above automatic predictions associated with confidence levels,
Develop an annotation scheme to enrich scientific reports with explicit information that may subsequently be used by peer reviewers and readers,
Update automatic predictions over time using the supervised data obtained through interactive annotation.

Methods

This project will use research articles on randomized controlled trials from the biomedical literature to build corpora annotated for important claims and their supporting evidence. Claims will be limited to those present in (a) abstracts, (b) titles, and (c) citations to other works. A single annotation tool will be developed for both annotation by independent experts, intended to bootstrap the corpus construction, and by authors, intended to assist them in their authoring activity. Once a representative corpus has been obtained, a contrastive study of the characteristics and textual expressions used per type of claims will be conducted.

Annotated data will be used to learn machine learning modules to automatically detect important claims and their supporting evidence. This problem lies in the domain of textual entailment and may be approached by partial textual entailment in an interactive setting, where a claim to be demonstrated from text can be decomposed into facets. The corresponding machine learning modules will be used to make automatic predictions associated with confidence levels and will be updated with new supervised data resulting from the use of the authoring tool. Issues regarding the efficient integration of the tool within classical authoring software will be studied, and an evaluation of the usability and the effects of the tool on authored articles will be conducted.

Expected results

The resulting tool will be made available to enrich existing articles through claim supporting annotations and to assist authors in improving their articles before submission to peer review. We expect that the developed annotation scheme will contribute to the dissemination of improved scientific reports that will be self-explaining and thus be accessible to larger communities of readers. The implemented approach may subsequently be tested on other scientific domains such as Natural Language Processing, for which large electronic corpora of articles are available.

Start date: November 2016

Duration: 36 months

Planned secondments

Host 1: Cochrane, 3 months, to prepare the corpus

Host 2: EQUATOR Network, 2 months, to prepare the corpus

Host 3: University Paris Descartes, 4 months, to evaluate the tool

Host Institution: CNRS

Supervisors

Supervisor: Aurélie Névéol (CNRS),

Co-supervisor: Mariska Leeflang (University of Amsterdam)

Context

In medicine, Diagnostic Tests are any kind of tests performed to assist clinicians with the diagnosis of a disease or detection of a specific health condition. Diagnostic tests can be invasive (e.g. amniocenteses), minimally invasive (e.g. blood test) or non-invasive (e.g. urine analysis). Therefore, it is crucial to weigh possible benefits against the financial and psychological burden associated with tests and resulting follow-up. The Cochrane collaboration has developed a methodology to systematically review diagnostic test accuracy based on the published literature.

Diagnostic Tests are a major influence on clinical decisions. Yet, information on their utility and accuracy is not always readily available in the published literature. In addition to the delay in publishing reports on diagnostic test studies, there is a strong publication bias: diagnostic test studies are first presented in the major medical conferences, and only about 50% of studies go on to be published in medical journals. The exact nature of the publication bias is not well understood.

Another challenge when looking for diagnostic test accuracy studies, is that these studies are often not indexed in an easy retrievable way. Hence, a typical search strategy for diagnostic test accuracy will retrieve around 5000 initial hits, of which a couple of hundred will have to be read as full text and only around 10 to 20 will be included in the review.

There is a need to understand and monitor the information trail on diagnostic tests in order to inform clinicians and patients. The hypothesis explored in this research project is that text mining methods can offer an efficient way to gather information on diagnostic tests based on medical conference abstracts and articles published in medical journals.

Objectives

This Ph.D. project aims to:

Provide the scientific community with tools that track information about diagnostic test studies. For this purpose,
Develop Natural Language Processing techniques for (a) identifying conference abstracts and journal articles reporting on diagnostic tests (b) extracting specific information about diagnostic tests (study characteristics) including test description, accuracy and use
Implement a recommendation system that will retrieve diagnostic test studies for inclusion in systematic reviews, as well as identify specific study characteristics
Populate a knowledge base with comprehensive information about diagnostic tests which will inform researchers and clinicians.
Update automatic predictions over time using the supervised data obtained through interactive annotation

Methods

This project will use a selection of Cochrane diagnostic test accuracy reviews to build a large reference corpus of conference abstracts and journal articles relevant to test diagnostic studies. Furthermore, the dataset will also be annotated with gold standard characterization for test description, accuracy and use.
A classification tool will be developed both for experts to independently identify relevant test diagnostic studies and for systematic review writers, intended to assist them with the selection of studies to be included in the reviews. Once a representative corpus has been obtained, a contrastive study of diagnostic test study reporting in conferences vs. journals will be conducted. Similarly, an information extraction module will be developed to extract specific information related to test diagnosis studies, in collaboration with systematic review writers who will act as experts and end-users.

Annotated data will be used to learn machine learning modules to automatically detect diagnostic test studies and their characteristics. The problems lie in the domain of information extraction and may be approached as named entity recognition informing classification models. The corresponding machine learning modules will be used to make automatic predictions for global studies and in the context of systematic review writing. Issues regarding the efficient integration of the tool within the systematic review creation workflow will be studied and the effects of the tool on systematic review literature coverage will be conducted.

Expected results

The resulting algorithms can be applied to the available data sources in order to create a comprehensive repository of information on diagnostic test studies. We expect that this effort will provide useful information for database curators by providing support to database curation, and to clinicians and systematic review writers by providing a comprehensive characterization of studies addressing specific diagnostic tests.

Start date: October 2016

Duration: 36 months

Planned secondment(s)

Host 1: University of Amsterdam, 5 months, to build the gold standard reference corpus

Host 2: Cochrane, 5 months, to identify relevant medical conferences, and work on the creation of diagnostic test repository

Host Institution: University of Split

Supervisors

Supervisor: Darko Hren (University of Split)

Co-supervisor: Erik Cobo (Universitat Politecnica de Catalunya)

Context

Peer review is used by a vast majority of biomedical scientific journals to assess and improve the quality submitted manuscripts. Despite dealing with scientific output and potentially having an impact on the quality of research published, the efficiency of the manuscript peer review process has been questioned. Within the biomedical field, the apparent roles and tasks of peer reviewers differ among the journals. Research has also shown that there is a dissonance between the most important tasks in peer review, as perceived by peer reviewers and as perceived by journal editors. These differences may influence quality of peer review reporting, and thus quality of the peer review process across journals. The primary functions of peer reviewers are poorly defined. Thus far no body of literature has systematically identified the roles and tasks of peer reviewers of biomedical journals. A clear establishment of these can lead to improvements in the peer review process. This research aims to determine what is known on the roles and tasks of peer reviewers.

Research indicates that there are social and subjective dimensions of the peer review process that contribute to this perception, including how key stakeholders – namely authors, editors and peer reviewers – communicate. In particular, it has been suggested that the expected roles and tasks of stakeholders need to be more clearly defined and communicated if the manuscript review process is to be improved. Disentangling current communication practices, and outlining the specific roles and tasks of the main actors, might be a first step towards establishing the design of interventions that counterbalance social influences on the peer review process. This research aims to identify and characterize the roles and tasks of the different actors in the process of peer review from the perspective of key stakeholder, journal editors.

Besides authors and editors, peer reviewers are key stakeholders within the manuscript peer review

process, hence is it important to explore their understanding of their roles and tasks in manuscript

review. Thus far most studies on peer reviewers in the biomedical field report on surveys (administered to peer reviewer or editor). However, there is an increasing drive towards promoting scientific integrity via meta-data analysis using journal peer review data. Peer reviewer reports are rich sources of data that can be used to examine the performance/behaviour of peer reviewers through their output. This is particularly pertinent given that, generally, peer reviewer comments of accepted and rejected manuscripts in biomedicine have not been examined on a large scale due to issues of confidentiality. There are few existing studies on peer review in the biomedical field that have systematically analysed the content of peer reviewer reports.

This research aims to generate a typology of peer review comments on manuscripts that report on randomized controlled trials (RCTs) submitted to biomedical journals. The analysis of peer reviewer reports, particularly those of rejected manuscripts that are typically not readily available, will offer a unique opportunity to reveal areas that peer reviewers address and elicit their perceived roles and tasks.

Objectives

To determine the role and tasks of peer reviewers in the biomedical journal editorial process through a scoping review of the literature;
To explore editors’ understanding of the roles and tasks of peer reviewers in biomedical journals through qualitative interviews;
To explore peer reviewers’ understanding of their roles and tasks within the manuscript review process through a content analysis of peer reviewer reports.

Methods

Scoping review of the literature

We will use the methodological framework first proposed by Arksey and O’Malley and subsequently adapted by Levac and the Joanna Briggs Institute.

The scoping review will consist of:

1) Database search of the literature (all study designs, editorials, letter to the editor) and

2) Review of grey literature (review of journal guidelines to peer reviewers, review of grey literature such as blogs, network websites, etc.)

Qualitative study design using semi-structured interviews

Semi-structured interviews will be carried out with editors of biomedical journals who are involved in decision making about the fate of manuscripts; determine journal content and policy; interact with peer reviewers; and are accountable for all material published in their journals. A heterogeneous sample of participants representing a wide range of journals will be sought through purposive maximum variation sampling. Interviews will be thematically analysed following the method outlined by Braun and Clarke. The qualitative data analysis software NVivo v11 will be used to aid data management and analysis

Retrospective cross-sectional study using peer reviewer reports

A retrospective content analysis of a sample of peer reviewer reports to participating medical journals will be performed. Reports will be categorized according to: final editorial decision (i.e. publication status of the manuscript) and reviewer recommendation (accept/reject).

Reviewer comments will be analysed using:

1) CONSORT reporting guidelines.

2) Review Quality Instrument

Expected results

The overall aim of this PhD is to identify ways of improving the quality of reviewer reports submitted to biomedical journals through an in-depth view analysis of content and communication in the peer review process. Through identifying most common themes and issues addressed by reviewers it will provide important information for future development of educational interventions for authors, editors and reviewers.

Start date: October 2016

Duration: 36 months

Planned secondments

Host 1: BMJ, 4 months, studying their current practice of peer-review and editorial process

Host 2: University Paris Descartes, 2 months, to develop analysis framework

Host 3: Universitat Politecnica de Catalunya, 4 months, analysis of the methodological and reporting issues required

Host Institution: Universitat Politecnica de Catalunya

Supervisors

Supervisor: Erik Cobo (Universitat Politecnica de Catalunya)

Co-supervisor: Jamie Kirkham (University of Liverpool)

Context

The peer review process is a cornerstone of biomedical research publication. The editorial peer review process is described as journal editors relying on the views of independent experts in making decisions on, for example, the publication of submitted manuscripts or presentation of reports at meetings. The peer review system is considered the best method for evaluating publications of health research.

In 2010, a report ordered by the UK House of Commons showed that the cost to the UK Higher Education Institutions in terms of staff time was £110 to £165 million per year for peer review and up to £30 million per year for the work done by editors and editorial boards. Worldwide, peer review costs an estimated £1.9 billion annually and accounts for about one-quarter of the overall costs of scholarly publishing and distribution. The human cost was evaluated at 15 million hours by 2010.

One of the main goals of the peer review process is to ensure that the report submitted is completely transparent and adheres to the relevant reporting guidelines to allow an appropriate appraisal of the methods and results.

Nevertheless, the effectiveness of the peer review system is questioned. The peer review system fail to identify important flaws in the quality of manuscripts and published articles. A recent study by Hopewell et al. showed that peer reviewers could not detect important deficiencies in the reporting of methods and results of randomized trials.

Objectives

This Ph.D. project aims to:

Identify interventions to improve the adherence to reporting guidelines during the peer review process of submitted manuscripts,
Evaluate the barriers and facilitators for using these interventions,
Evaluate the impact of the intervention deemed as the most promising.

Methods

We will proceed in 3 steps:

1) Identification of interventions used by editors to improve the adherence to reporting guidelines reporting during the peer review process.
The student will first perform a comprehensive literature review. He/she will develop the search strategy, perform the search and select eligible articles. He/she will also develop a data extraction form and record the data. The student will also review the tools for quality improvement during the regulatory process. In a second step, the student will survey editors and reviewers. He/she will contact all editors of the 10 highest impact factor journals of each specialty and ask them which interventions they are implementing to improve the adherence to reporting guidelines during the peer review process. The student will propose a typology of the interventions used.

2) Evaluation of barriers and facilitators for using these intervention and identify other possible interventions.
The student will perform an online qualitative study. The stakeholders invited to participate will be editors and peer reviewers. The student will develop the protocol, perform the study and perform a content analysis following the « grounded theory approach », and an automatic textual analysis. This study will allow identifying the barriers and facilitators for these interventions, identifying new interventions and identifying the most promising intervention(s).

3) Evaluation of the impact of the intervention deemed as the most promising in the qualitative study.
We will perform a pilot study to evaluate the feasibility of evaluating the most promising intervention in a randomized controlled trial. We will use a fabricated manuscript with some items incompletely reported. Reviewers will be randomized to the usual process or to the intervention selected. The primary outcome will be the percentage of items incompletely reported accurately identified. The student will write the protocol. The study will be performed in collaboration with the MiRoR partners BMJ and BMC.

Expected results

This project will allow a clear understanding of the possible interventions that could be implemented during the peer review process to improve the completeness of reporting of published reports.

Start date: October 2016

Duration: 36 months

Planned secondments

Host 1: BMJ, 5 months, studying their current practice of peer-review and editorial process

Host 2: University of Liverpool, 5 months, assessment of the intervention

Host Institution: Universitat Politecnica de Catalunya

Supervisors

Supervisor: José Antonio Gonzalez (Universitat Politecnica de Catalunya)

Co-supervisor: Darko Hren (University of Split)

Context

Editorial peer review is a longstanding and established process usually aimed at providing a fair decision-making mechanism and improving the quality of a submitted manuscript. Despite the long history and employment of the peer review system, its efficacy is still a topic of controversy. In recent years, certain tools have been developed that are aimed at measuring the quality of a peer review report. However, the variation in the quality of peer review both within and between journals is a concern; particularly troubling is how biomedical editors assess the quality of review reports and which quality components are used to evaluate reviewers’ work.

Objectives

This Ph.D. project aims to:

Identify tools used to assess the quality of peer review reports;
Describe the practices and understand the perspectives of biomedical editors and authors toward the quality of peer review reports;
Develop a checklist of items to measure the quality of peer review reports.

Methods

We will proceed in 3 steps:

1) The student will perform a methodological systematic review of tools used to assess the quality of peer review reports. She will develop a search strategy, perform the search, select eligible articles, develop a standardized extraction form and record the data.
2) The student will perform a qualitative study using an online survey to describe the practices and understand the perspectives of biomedical editors and authors toward the quality of peer review reports. She also aim to solicit editors and authors’ opinions on how they define the quality of peer review reports and in particular which quality components they consider when they assess a peer review report.
3) The survey will be followed by a Delphi survey with international experts whereby they will rate the importance of the quality components previously obtained from the aforementioned methodological systematic review and survey. In this step, the experts will decide on the final set of quality components that will be include in the checklist.

Expected results

The resulting checklist will be made available to biomedical editors to assess the quality of a reviewer’s work. It might be further used to assess the quality of peer review reports in developing programs to train new reviewers.

Start date: March 2017

Duration: 36 months

Planned secondments

Host 1: EQUATOR Network, 3 months, to finalize the methodological systematic review

Host 2: University of Split, 4 months, to conduct the qualitative study

Host 3: Cochrane Editorial Unit, 3 months, to explore future implementation by editors