Do research findings need to be timely to influence policymaking?

Many research funders are increasingly focused on demonstrating clear positive impacts on society. Researchers can help to realise these positive impacts, in part, by seeking to ensure that their work gets taken up in the design of policies and programs. However, bridging the research-to-policy gap requires finding balance points between a number of factors that shape how researchers and policymakers do their work. In particular, the pace and timing of many policy environments is much more rapid than those found in traditional academic research settings. Based on this, it seems that researchers would need to provide insights at the pace of policymaking if they want to have impact. This post tests this idea by reviewing some recent studies about whether or not research evidence needs to be timely in order to be taken up by policymakers.

Is timeliness part of the “recipe” for research impact?

Let’s start with a great example of the type of paper that one finds when searching academic databases on this subject. Geddes et al. (2017) did a study that sought to find a “recipe” for research impact in UK parliament. They held interviews and a workshop with parliamentary officers (e.g. committee clerks, librarians, or workers from a specialist science and technology unit). Overall, the authors found that there is no one single “recipe” to fit all circumstances, but a clear message emerged from the study about timeliness. Research findings need to be provided in a timely fashion and researchers need to be ready to engage with policy issues on very short notice (i.e. days or weeks) if they would like their research to influence an inquiry or debate. This finding has some logical implications about the types of research evidence that can be provided on such short time horizons. Such a short window of time means that researchers’ methodological options might be limited to meet policymakers’ needs. Things like a rapid desktop review of previous research (if there is sufficient research on the topic), a quick series of interviews with experts, or some statistics using an existing database might be feasible. One might be able to hope that a larger research project on the topic was recently completed, but in most cases, several more time-consuming, but robust research methods wouldn’t be feasible to meet a tight deadline.

The time pressure that policymakers feel can be seen in other findings from the study as well. The study found that parliamentary officers had a clear preference for data in the form of statistics and/or narratives — outputs that can quickly encapsulate an issue on a population or individual levels. Further, they said that research outputs should be clear (e.g. clearly written, transparently described research findings) and accessible (e.g. in plain language and published in open access sources). Participants called out the “…effectiveness of academic blog posts in translating academic research” (pg 269) as one example of potentially clear and accessible communication (I pat myself on the back for writing this post…). Based on this, it seems likely that timely delivery of research evidence is a necessary element of research uptake, but timeliness alone may not be sufficient for uptake. Ensuring that research results are also efficiently communicated in ways that are understandable or preferred by policymakers may ultimately be required for the work to have impact.

Balancing rigour with timeliness

There is a rich body of scholarship about providing research findings within key windows or opportunity that can also add to our discussion of timeliness. One recent paper in this area of study is Meisel et al. (2019), which aimed to understand how to capitalise on an urgent window of opportunity engendered by the opioid epidemic and rapid shifts happening in healthcare systems. The authors interviewed 18 policymakers about how to bridge the research-to-practice gap in implementing evidence-based prevention and treatment programs for substance use disorder.  

Overall, the participants in this study felt that researchers need to use study designs that prioritise timeliness. One of the key themes that emerged from the interviews also echoes the logical conclusion that we made from Geddes et al.’s (2017) findings. Policymakers said that for research to influence policy, researchers need balance rigor with timeliness. A representative quote from the paper is quite telling:

“For the decisions I need to make, it actually needs to be a bit less rigorous, because more rigor equals more time. I don’t need the exact answer out there to three decimal points. I need to be directionally correct.” (pg 4)

One participant in this study even went so far as to say that rigorous research might be problematic if the findings from the project arrive too late to be used, however, most participants merely indicated that research rigor should be considered as one of several factors when designing research that can be used to address an ongoing public health crisis. Broadly, this adds support for the notion that timeliness is necessary, but may not be sufficient. It also adds another consideration to the mix — research rigor or quality. The policymakers in this study seem to strongly prioritise timeliness over any rigor beyond an amount that is minimally sufficient for their decision-making needs.

According to a simulation study, greater certainty can’t compensate for late-arriving findings

The studies that we’ve looked at so far used qualitative methods to understand the role of timeliness in policymaking, but there are also other new and emerging methods being applied in this space. Lammers et al. (2024) used computer simulations to explore how the timing of research findings delivery and the level of uncertainty of research evidence may interact. They considered questions like does communicating about research findings earlier compensate for a higher level of uncertainty (and vice versa)? The study’s methods are a little complex, so I’ll spend a little more time than usual breaking down the methods (for myself as well as you!).

It should be mentioned that using computer simulations are often particularly useful for gaining insight into complex systems that are difficult to study in real life. In this case, it would be a proper challenge to observe how groups of policymakers might change their opinions in several different scenarios related to the timing and uncertainty of research findings presented to them. As the two previous articles demonstrated, there are researchers exploring these sorts of questions in real life, but their qualitative methods mean that the findings are frequently limited to a providing a nuanced understanding of a specific context or scenario, like UK parliamentary officers’ opinions on evidence use[1].

With that out of the way, here’s the detail on the Lammers et al.’s (2024) methods. The authors used an established computer simulation technique called a Hegselmann–Krause model. This is an agent-based model, which means that it is a technique that can be used to study how people (or other things, referred to as “agents”) interact over time by building on a set of initial factors and assumptions about a scenario. The model in this case was designed to consider how a range of factors shape if and when simulated policymakers change their minds in reference to the introduction of research evidence and the views of their peers over time. The factors in the study included things like how many policymakers are part of a group, their prior opinions on an issue, their level of open-mindedness to new ideas, their urgency for making a decision, and the level of uncertainty in the scenario[2].

To explore their research question, the authors developed models for two distinct decision-making scenarios based on previous research — intelligence analysts looking for terrorist activity and inspectors assuring the safety of chicken meat. These two scenarios were chosen because previous research indicated that they likely had very different combinations of factors shaping decisions.

After undertaking several simulations for both scenarios, the authors found evidence to support the idea that timely delivery of evidence and a sufficient degree of certainty were necessary for policymakers to use the evidence in their decision-making.  However, relating to how timing and certainty interact they found that “…evidence should come early, and certainty cannot compensate for late timing. If one waits too long with communicating evidence, it will not be relevant anymore…” (pg 188). The authors began the study with a suspicion that timing and certainty could, to some degree, compensate for each other, but they were surprised to find that no amount of additional certainty (likely built on higher-quality research) would overcome late-arriving research findings. This confirms the real-world findings from the Meisel et al. (2019) article. If researchers are seeking to influence policy, they may want to design questions that can achieve a sufficient level of certainty or rigour, but beyond that, ensuring that findings can be delivered at the right moment is paramount.

As a final comment on the findings from both Meisel et al. (2019) and Lammers et al. (2024), some experts back up this “balance rigor with timeliness” sentiment. In an article by Christopher Whitty cited in a previous post on choosing policy-relevant research questions, he says:

“Academics underestimate the speed of the policy process, and publish excellent papers after a policy decision rather than good ones before it….Papers which are as simple as possible (but no simpler) are most likely to be taken up in policy.” (pg 1)

Sense checking what we’ve learned with a systematic review

A recent systematic review can provide a useful sense check of our findings so far in this post[3]. Ouimet et al. (2024) reviewed studies on research evidence use in legislatures, including factors that facilitate and hinder use. Their findings were largely consistent with what we’ve found so far. Several studies indicated that research being available and accessible when a window of opportunity arises (6 of 21 studies) or researchers having the capabilities to rapidly respond to policymaker research needs when they arise (1 of 21 studies) were key facilitators of research use. Moreover, lack of timeliness was highlighted as a hindrance of research use by 2 studies.

Additionally, 8 of the 21 studies affirmed that research findings should be “…concise, clear, and understandable…for nonspecialists…”, as well as the fact that presenting findings with narrative elements like stories or anecdotes (5 of 21 studies) or statistics (5 of 21 studies) facilitate research use.

Finally, the review also highlights another key factor at that was only briefly mentioned earlier — policymakers are frequently time poor. A key hindrance to research use was the lack of time and need to make quick decisions (8 of 21 studies). This is, indeed, a key factor to consider when thinking about timeliness and research use. A study from 2015 (Cherney et al.) tells this story even more clearly.  The authors surveyed more than 2,000 public officials in Australia about their use of social research and more than half of their participants said they didn’t have enough time to read the academic literature on the key topics related to their work. This finding further underscores the notion that research delivered in a timely fashion must also be delivered in a useable format for the work to have policy influence.

Conclusions

This post set out to understand if providing research in a timely fashion was important for having policy influence. Unsurprisingly, timeliness is important. However, this closer examination of the topic reframed and added important nuances to the original question. We found that timeliness is necessary but not sufficient for policy influence, and that there are a number of other conditions that may need to be met in conjunction with timeliness. The conditions include: the methods have a minimum level or rigour or provide a minimum level of certainty, the findings are communicated in language that is right for the audience, and/or the research is designed to produce types of data that policymakers prefer or understand. These findings have clear implications for how researchers write research questions and design projects — researchers need to find ways to strike a balance between all of these factors if policy influence is the goal.

It should also not go unsaid that, in some cases, it may not be possible to produce research that is sufficiently timely while maintaining a minimum required level of rigor for a researcher to stand by their findings or to do their work in an ethical way. In these cases, the way forward is more difficult.

The next post will zoom in on the relative uncertainty that there is about how research gets used in the policy making process.

Note: this essay is continuously updated as relevant articles are added to Good Questions Review.

[1] It should also be noted that simulation studies have several limitations. To name a few for this study, the model does not account for unequal power dynamics within the group, the fact that in many real life situations the level of uncertainty about a topic is unknown, or a wide range of known barriers and facilitators to the uptake of research  in policymaking.

[2] To my specialist colleagues, please let me know if I have oversimplified this for the sake of clear communication.

[3] I work on a team that specialises in review methodologies, so this won’t be the last time that a post will include a systematic review or two…

Articles cited

Cherney, A., Head, B., Povey, J., Ferguson, M., & Boreham, P. (2015). Use of academic social research by public officials: Exploring preferences and constraints that impact on research use. Evidence & Policy, 11(2), 169–188. https://doi.org/10.1332/174426514X14138926450067

Geddes, M., Dommett, K., & Prosser, B. (2017). A recipe for impact? Exploring knowledge requirements in the UK Parliament and beyond. Evidence and Policy, 14(02), 259–276. https://doi.org/10.1332/174426417X14945838375115

Lammers, W., Pattyn, V., Ferrari, S., Wenmackers, S., & Walle, S. V. D. (2024). Evidence for policy-makers: A matter of timing and certainty? Policy Sciences, 57(1), 171–191. https://doi.org/10.1007/s11077-024-09526-9

Meisel, Z. F., Mitchell, J., Polsky, D., Boualam, N., McGeoch, E., Weiner, J., Miclette, M., Purtle, J., Schackman, B., & Cannuscio, C. C. (2019). Strengthening partnerships between substance use researchers and policy makers to take advantage of a window of opportunity. Substance Abuse Treatment, Prevention, and Policy, 14(1), 12. https://doi.org/10.1186/s13011-019-0199-0

Ouimet, M., Beaumier, M., Cloutier, A., Côté, A., Montigny, É., Gélineau, F., Jacob, S., & Ratté, S. (2024). Use of research evidence in legislatures: A systematic review. Evidence & Policy, 20(2), 226–243. https://doi.org/10.1332/174426421X16656568731041

Whitty, C. J. M. (2015). What makes an academic paper useful for health policy? BMC Medicine, 13(1), 301, s12916-015-0544–0548. https://doi.org/10.1186/s12916-015-0544-8