Scientists are developing vast repositories of evidence to develop effective policies

Scientists are developing vast repositories of evidence to develop effective policies

Investors are pouring tens of thousands of millions of dollars into an ambitious plan to solve the biggest problem in scientific advice: providing evidence to governments. Their goal is to create a system that enables policymakers worldwide to generate rapid syntheses of scientific evidence that will help them develop evidence-based policies critical problems such as climate change to solve.

“We could ultimately benefit enormously from a world in which comprehensive evidence syntheses on every major social problem are available in one place, continually updated,” says Will Moy, who leads the Campbell Collaboration, an international nonprofit that supports social science assessments.

Although researchers in policy-relevant areas produce a variety of studies, syntheses that represent the weight of evidence on a topic are rare in many areas and are not routinely used to guide policymaking. “There is huge demand” from policymakers for such syntheses, says Jen Gold, director of research at the Economic and Social Research Council (ESRC), a British funding agency. “But the offer doesn’t match.”

Evidence synthesis is “everything the world knows about how to solve an important problem in one place,” explains Moy. In medicine, doctors routinely use thousands of systematic reviews - thorough syntheses of studies such as randomized drug trials – that show whether treatment helps or harms. In most other areas, however, such a comprehensive basis is missing (see 'Missing syntheses'). It can take months or years to extract meaning from a vast corpus of research—and funding agencies have historically spent comparatively little on synthesizing knowledge compared to the billions they spend on new research.

To address this, the ESRC and Wellcome, the biomedical research funder in London, announced on September 21 that they were investing £9.2 million ($12.2 million) and about £45 million, respectively, over five years in databases and tools that can help synthesize research. British Science Minister Patrick Vallance and Wellcome boss John-Arne Røttingen announced the funding at an event in New York tied to the United Nations Summit on the Future, a meeting aimed at shaping a better world, including through science.

Researchers have welcomed the news - believed to be one of the largest single investments in evidence synthesis - and say it comes at the right time as advances in artificial intelligence (AI) speed up the process of finding and combining studies. “It's so exciting,” says Isabelle Mercier, a researcher with the United Nations Development Program Supported evidence syntheses at the UN. “Four years ago this was too big to think about, but now we’re starting to see that we can actually make it happen,” she says.

But AI also makes the task more difficult, as AI chatbots like ChatGPT can generate credible-sounding but potentially misleading summaries of research results. “The challenge is how to separate what is truly reliable from what is not,” says Moy.

The production of syntheses is usually slow, difficult and expensive. Researchers conducting a systematic review must search worldwide databases of published and unpublished papers to find potentially relevant studies. They then reduce a long list of thousands of studies to the most relevant ones, assess their reliability, extract the data, and combine the results, sometimes using a statistical method called meta-analysis. Even when completed, evidence syntheses often do not reach policymakers and quickly become outdated as new research arrives. “When a policymaker comes with a question, it shouldn’t take three months to find the research,” says James Thomas, a research synthesis specialist at University College London. “This is ridiculous.”

The problem worsened during the COVID-19 pandemic, when authorities everywhere needed rapid synthesis to make decisions about medicines, masks and lockdowns. At first, scientists couldn't provide them fast enough - but then they produced them too many duplicate syntheses and poor evaluations.

Scientists' dream is that anyone, anywhere, could put together a synthesis tailored to their question and region with the push of a button. To do this, researchers want to create 'evidence banks': shared databases of pre-selected studies, tagged with information such as method and location, and containing data in a common format so they can be combined. Trained AI tools should do most of the tedious work of sorting studies and synthesizing data, while humans check quality – for example, by assessing possible biases in the underlying studies.

Some databases are already on the right track. The Education Endowment Foundation (EEF), a charity in London, has a database of more than 3,500 education studies. Using this database, the organization has created a series of systematic reviews that Influence of tutoring, homework and class size on learning reveal. It shares the database and overviews with multiple countries to avoid others repeating the work. Ideally, “instead of doing six separate systematic reviews, you create one great review that we can use together,” says Jonathan Kay, who leads the evidence synthesis work at the EEF.

The latest investments could eventually lead to a series of databases similar to the EDF, ready to be synthesized for key policy areas such as environmental protection. From these, advocates want to build 'living' - or constantly updated - evidence syntheses that show, for example, which helps reduce climate change, improving mental health and reducing youth unemployment.

Wellcome intends to fund consortia that develop data platforms and tools that help achieve this goal. This is “an unusual move by Wellcome,” says Tariq Khokhar, the charity’s head of data for science and health. The organization is known for funding health research, but this money could help make evidence across disciplines more understandable. “It really is a foundation that anyone can build on,” he says.

The ESRC plans to fund a consortium to accelerate evidence synthesis and develop first versions of living syntheses in areas such as healthy aging. The two efforts would be different but could overlap, Khokhar says.

The ESRC also wants its consortium to develop ways to make it easier for policymakers to use evidence synthesis. Some British officials are starting to use an AI tool called Redbox Copilot to analyze and summarize government documents and speeches. Researchers could develop tools that “incorporate systematic review evidence into this process,” Gold says.

The £55 million will not be enough to achieve the kind of seamless evidence synthesis that advocates dream of. But funders hope to aggregate more funds — and that this initial investment will encourage other funders to get involved. “The idea is that each investment can build on a lot of work that has already been done,” says Khokhar.