Evgeny Bobrov on strategies and approaches to increase the value of biomedical research.

In an influential series of opinion pieces published in early 2014 (see introductory comment by Macleod et al., 2014), the medical journal The Lancet tackled the issue of “increasing value, reducing waste” in biomedical research. This series laid out in detail the issues that had led Chalmers & Glasziou to conclude in 2009 that 85% of investment in biomedical research was wasted. The Lancet series did not give a clear definition of “value”, but the picture emerged that biomedical research is valuable if it uses appropriate, unbiased methods (leading to robustness), and if its results are useful for both other researchers (accessibility) and potential users (relevance). Thus, this concept of “value” goes beyond the framework of Open Science, and includes relevance to society at large. Hardly anyone would contradict these pillars of value in principle, with the possible exception of relevance for basic research. In this case, because the assessment of relevance for users must be based on far-reaching assumptions about possible future applications and their probability, it is disputed whether relevance is suited to assess value, even though the case for basic research at large most often draws on it.

Evgeny Bobrov, Open Data and Research Data Management Officer at QUEST Center, Berlin Institute of Health (BIH) (© BIH/Thomas Rafalzyk)

But how is the reality of the academic biomedical research system? What are its workflows, its standards, its incentive structures, and ultimately its principles of conduct? Analyses from multiple points of view offer a picture which indicates often low research value (Freedman et al., 2015), including inappropriate study designs (Bailoo et al., 2014), tendentious analyses (Simmons et al., 2011), reporting bias (Avey et al., 2016, Landis et al., 2012), incomplete methodological information (Leung et al., 2018), lack of compliance etc. (Jilka, 2016). Choices of research focus, model system and the specific questions addressed often do not sufficiently consider relevance and existing evidence. These shortcomings are entrenched in the biomedical research system, and thus can only be addressed by all stakeholders in unison – researchers, institutions, funders, journals, and legislators. Of these, the researcher level is ultimately the one where research practice would have to change, while research-performing organizations have the most immediate and arguably most powerful influence on researchers to implement such a change (not least, because researchers themselves in large part constitute the institutions).

The QUEST Center

The quest for improving value in biomedical research, I would thus argue, has to both address and be driven by researchers and institutions. For the research area of the Berlin Institute of Health (BIH), which includes the Charité – Universitätsmedizin Berlin (on which this article focuses) and the Max Delbrück Center for Molecular Medicine, a dedicated “BIH Center for Transforming Biomedical Research” was created in 2017 to support this transformation. It goes by the name of QUEST (Quality, Ethics, Open Science, Translation), and is a unique structure with a holistic view of the aspects in need of improvement in biomedical research.

Approaches

The QUEST Center has eight “Approaches”, which aim to make biomedical research in the BIH research area and beyond more robust and useful:

1. Quality assurance aims to increase robustness, including reproducibility
2. Open Science supports transparency and thus usefulness to researchers, while
3. Patient and stakeholder engagement increases the relevance for users of research
4. Incentives & rewards acknowledges the crucial role of the reward system, and is thus an overarching approach, similarly to
5. Education & training, which aims to qualify researchers in the aforementioned areas
6. Translational bioethics develops evidence-based recommendations for the implementation of ethical requirements for biomedical translational research
7. Meta-research, i.e. research on research, is conducted by QUEST members and guest scientists, and serves as evidence basis for further activities and self-evaluation, as well as contributing to the body of knowledge at large
8. Thinktank services of QUEST draw from the different approaches and offer advice to different stakeholders in biomedicine like funders and policy-makers

Thus, QUEST approaches cover activities both directly within the biomedical research community and on a meta-level. Also, in case of the former, these activities are oriented at both the conduct of research (pursuing robustness and transparency) and its translation into applications (relevance). These different activities inform each other, and this allows the knowledge about biomedical research practices and the implementation of value-increasing changes to advance together. On a practical level, this also means that QUEST staff is often involved in both support and research activities, which helps to integrate the sometimes differing mindsets.

QUEST’s activities are conducted by an office team of seven staff members with research background, as well as the research groups “Transforming biomedical research”, “Translational bioethics” and “Meta-research”, associated researchers and international visiting fellows.

QUEST’s conception rests on the assumption – in my opinion, justified – that the meta-research community had already reached at least an approximate consensus on the main problems biomedical research is facing. QUEST was thus given the mission to act upon it, and with this mission of driving change in its sphere of activity, the questions to ask are: Where can the QUEST Center have an impact? Which resources are already available at the institution? And starting from there, what is the resulting strategy?

Strategies

There are different strategies which the QUEST Center pursues to achieve its mission of increasing value in biomedical research, in particular at the Charité and the Max Delbrück Center. These are shaped by its broad mission and multidisciplinary team, as well as the – for the most part, pre-existing – institutional structures and concurrent efforts of other stakeholders.

One strategy is spearheading: with its innovation-orientation and limited resources, QUEST is well suited to develop initiatives and services, and implement them in a pilot phase, but less suited to provide continuous services for a whole institution, be it Charité or Max Delbrück Center. With growing acceptance and demand, services are thus ideally taken over by institutional facilities. For example, QUEST had an officer for Open Access (OA), who served as institutional OA Representative, secured DFG funding for OA publications, and disseminated information on OA through presentations, consulting, and online resources. However, with OA becoming increasingly common at the institution, it became unfeasible for one person to cover all related activities. The medical library took over the responsibility for OA services, establishing a dedicated OA team of four.

Another strategy could be called linking: in a topic as complex as research data management (RDM), no single person can cover all expertise necessary. As QUEST officer for Open Data and Research Data Management, it is one of my key tasks to link expertise available in IT, library, data protection and other departments, with a double goal: on the one hand to offer researchers a single entry point to data management services, on the other hand to develop joint strategies for the further development of services and infrastructures. The role of the QUEST Center in the long run remains to be established here, but whatever the organizational structure of RDM services, it is to be expected that with growing demand, the member institutions will need to provide additional, long-term staffing for consulting, education and infrastructures.

To inform about its activities, establish direct contact and get informed about demands and expectations of researchers, the QUEST Center uses the pitching format, where individual research groups or institutes are visited. A short introduction to QUEST is followed by a topic-specific presentation, with the topic being chosen by the hosts. Meta-research also informs and facilitates QUEST activities. For example, a data scientist at QUEST developed a text mining algorithm for the detection of those publications, for which data have been openly shared. This algorithm was then used to find the institutional publications with Open Data, and with this knowledge it was possible to use Open Data as an additional criterion in the distribution of performance-oriented funding. At the same time, this algorithm is now available for the wider community and can thus be used for the monitoring of data sharing and for meta-research.

Educating is also an important strategy to drive change, and QUEST offers seminars e.g. on computational reproducibility, quality assurance, and peer review. As before, such educational activities are often to be considered pilots. As the demand for certain qualification grows and formats are incorporated by PhD programs or even in MSc level education, other providers have to take over from QUEST and its education officer.

Highlighting is an important strategy particularly when it comes to raising awareness for existing problems, informing about solutions in a general way, and contributing a certain point of view to larger discussions. Shortcomings of experimental planning and analysis are addressed in lectures, talks, and opinion pieces especially by QUEST director Ulrich Dirnagl and Professor for Translational Bioethics Daniel Strech. To shift hiring and evaluation criteria away from counting publications and impact factors, and towards a broader and more responsible assessment, the QUEST officer for incentives and indicators emphasizes criteria of responsible research in hiring committees and evaluates applications for intramural funding along these criteria.

Way forward

The QUEST Center thus addresses all aspects of research practice which are primarily the responsibility of individual researchers and institutions. This includes all steps of the research process: How are priorities of research set, how is it conducted, how analyzed, and how shared and disseminated? Importantly, QUEST’s activities aim not only at “research” as such, but at individual researchers in the research system by working to foster both capabilities and motivations to conduct science which is robust and useful. With this ambitious mission and little example to follow, the QUEST Center hopes to be both an inward facilitator of institutional change and an outward example of best practice. Possibly, it can serve as inspiration not only to institutions of similar focus, but also in other areas of research, with different research cultures and other obstacles to increasing the value of research. It would be interesting to see, which generic or discipline-specific strategies such efforts would pursue, and whether similar centers can cover a wide range of research fields, as would be the case for a whole university. And ultimately, time (and research) will have to tell, how large the impact of QUEST and similar centers was in either driving or facilitating change – a change which is concurrently driven and facilitated by a multitude of other stakeholders as well, from governments to individual researchers.

Acknowledgements: I thank Stephanie Müller-Ohlraun (QUEST) and Benedikt Fecher (Elephant in the Lab, HIIG) for their helpful comments on earlier versions of this article.