Taking a systematic approach to evaluating new tools and tech

In elite sport, we strive to operate in environments that are evidence-informed and development-focused. However, when it comes to integrating new practices—whether that’s adopting new technology, applying a fresh training methodology, or trialling a recovery strategy—organisations often face a barrage of options with little guidance on how to separate meaningful innovation from fleeting trends.

This article will outline a structured approach to help sports organisations make smarter, evidence-based decisions when evaluating anything new within the performance landscape.

The rapid pace of innovation over the last decade has been both a blessing and a burden. While the range of tools and ideas available to practitioners has grown exponentially, the sheer volume has led to confusion, inefficiency, and at times poor outcomes and wasted resources. Without a consistent system to assess new approaches, decisions can come down to influence or the latest trend. That leaves coaches vulnerable to wasting time—their own, their peers’, and their athletes’—and money, on top of making ill informed decisions.

Over the course of my career, I’ve learned the value of applying a graded recommendation model to guide these choices. Dr. Alan McCall introduced me to this approach while at Arsenal Academy. 

Whether it’s a training load model, a wellness questionnaire, a nutritional strategy, or a GPS platform, this approach provides a reliable filter for separating high performance solutions from unproven or unsuitable ideas.

Dealing with the innovation overload

Sport is full of innovative intention and promise. From breathwork protocols and neuromuscular diagnostics to player tracking and cognitive training, innovation is everywhere. Each promises improved performance, injury reduction, or optimised recovery.

Few innovations are backed by quality evidence. Fewer still are adaptable to every environment or population.

I have encountered this scenario many times in environments that I have worked in or visited. Managers feel harried by the queue of contacts from companies pitching the latest new technology and methods. Those sales reps are quite savvy about letting you know which of your competitors have already embraced this technology, and the results they are seeing. This exploits the intrinsic risk that innovation presents, particularly in competitive environments: a fear of missing out, or of falling behind. If other clubs have the latest shiny toy from the sports technology industry and we don’t, it’s easy to feel like we’re always playing catch-up.

The end result is too often technology sitting in a storeroom due to limited people power to bring it alive. Or, the technology or method didn’t do what it promised.

This leads to frustration from your team and frustration from senior people in the organisation who invested in the approach.

Managers, high performance directors, and coaches have to be attuned to the danger of adopting new methods reactively. What can seem like logical, good ideas, can fail in the wrong context or culture.

Graded recommendation model

To navigate the increasingly complex sports science landscape, a systematic and layered evaluation process is essential. The model is built on a three-step approach adapted from sports medicine to systematically and fairly evaluate new ideas and tools across four key criteria.

First is evidence quality. Is the method supported by peer reviewed, independent research? Has it been tested outside of promotional case studies?

Next is effect size. Does it offer a meaningful improvement? In high performance settings, even marginal gains matter, but only when implemented in the relevant performance areas.

The risk vs. reward profile forces you to weigh the downsides. Every new approach brings potential drawbacks: the time investment, athlete compliance, unintended side effects. Do the benefits outweigh these risks?

Finally, assess the fit between the tool and your specific context. Will this proposed new solution work within your culture, with your people, your resources, and your existing systems?

Step 1: Search for the evidence in the research

Begin with a comprehensive search for existing literature related to the method, technology, or practice being considered. Use academic databases and consult field-specific experts to gather peer reviewed studies, systematic reviews, and any emerging research.

Step 2: Assess the quality of what you find

Critically appraise the methods or study design used in the research. Look out for bias, conflicts of interest, and small sample sizes that may limit reliability. Ask yourself if the methods are appropriate for assessing the main research question, or if the methods appear aimed at obtaining a desired result.

Similarly, after reviewing the results, interpret the data and draw your own conclusions before reading the article’s “Discussion” or “Conclusion” sections. Compare your interpretation against the authors’. If they are different, try to determine why. Evaluate how the results may transfer or generalise to your sport or athletes: the research may be rigorous and the results promising, but that does not guarantee that it is appropriate for your context.

Be cautious with conclusions drawn solely from company-provided or loosely controlled studies. There’s much more value in independent research that supports or advises against a method.

Step 3: Integrate with practice-based evidence

Combine research findings with practical, context-specific insights. Consider the feasibility, cost effectiveness, and the cultural or logistical fit within your team or organisation. Reflect on practitioner expertise, athlete feedback, and real world trial outcomes.

This systematic evaluation is then distilled into the graded recommendation framework, which allows teams to clearly communicate whether an intervention is strongly advisable, acceptable, questionable, or best avoided.

Template for a graded evaluation

The framework below is based on the Arsenal-adapted graded recommendation system, which in turn is based on “Unravelling confusion in sports medicine and sports science practice: A systematic approach to using evidence to guide decision making” ^[1]. It blends internal assessments, expert opinion, and published evidence.

Each new idea—whether it’s a method, strategy, or technology—should be assessed using this four-tier system.

During my time at Arsenal, Dr. Alan McCall would complete the assessment and bring it for discussion at the club’s regular research and development meetings with all stakeholders. I continue to adopt this approach in my current role as Director of Sport and Physical Wellbeing at the University of Galway. 

Graded recommendation system

• A: Strong evidence to recommend
  • Peer-reviewed studies support use of the approach
  • Endorsed by recognised experts
  • Positive internal testing and athlete feedback
• B : Acceptable evidence to recommend
  • Expert endorsement present (without any conflict of interest)
  • Acceptable outcomes from internal assessments
  • Reasonable quality company-led or pilot study data
• C : Weak evidence to recommend
  • Inconclusive internal evaluation
  • Limited or poor quality supporting evidence
• D: Insufficient or strong negative evidence
  • Expert consensus against use
  • Negative or failed internal testing
  • Lack of, or misleading, evidence

The evaluation process should include pilot testing or a short term trial (2–4 weeks); staff and coach input surveys; and athlete usability or feedback reviews. This ensures decisions are not only data-informed but context-relevant and supported by the team.

This method isn’t only about assessing equipment. The graded recommendation model works just as well for evaluating new screening or testing protocols; athlete load monitoring or readiness metrics; recovery techniques (e.g. cryotherapy, massage tools, supplements); psychological strategies (e.g. mindfulness training); communication platforms or feedback systems.

Anything that requires staff time, player buy-in, or financial investment should be run through this filter. The model helps teams decide with clarity.

Avoiding the pitfalls of poor adoption

When new methods are adopted without robust review, the costs go beyond the financial investment.

Onboarding and retraining staff takes time. The staff can get fatigued from shifting priorities from the current strategy to the “next big thing,” and always being in those onboarding and retraining sessions when they could otherwise be doing their core jobs. And it’s not just the staff. Athletes get sceptical (in addition to fatigued and frustrated) when our approaches change too often.

One environment I worked with trialled a new training load algorithm. It required constant subjective input, yet the results didn’t correlate with performance or injury outcomes. Coaches quickly lost trust in the approach. The problem wasn’t just the method: it was the lack of structured appraisal prior to implementation.

The graded recommendation model for evaluation provides a buffer against such mistakes.

Bringing the graded approach to sports science

I am often asked for advice and opinion by sporting organisations. Recently, a Gaelic Games county asked for my advice about expanding the use of GPS systems from senior down to the U17 teams, for both Gaelic Football and Hurling. This is a considerable potential investment by the sporting organisation.

I had previously completed a graded recommendation approach on the Statsport technology. That put me in a position to advise the decision makers that the technology ranked A and that there was strong evidence to recommend use. I could explain that there are peer reviewed studies supporting its use, and that the system is also endorsed by recognised experts with positive internal testing and athlete feedback from the adult teams already using the system.

The county and I discussed the people power they would need to implement the system, and created a monthly meeting between the teams to ensure a standardised approach to implementation across all teams.

This resulted in the ability to create player profiles across the squads from U17 to adults. The graded recommendation approach enhanced the quality of the advice from merely my personal opinion to a systematic approach.

Similarly, from time to time, I am asked to advise on new technology. A recent example is KineMo, a technology that tracks and quantifies whole body movement kinematics during exercise using only the camera from a mobile device.

The graded recommendation approach helped me decide my level of involvement as an advisor during early discussions.

I was encouraged when the team had peer reviewed papers examining the technology. The technology team had done their due diligence, which was very encouraging. There was also positive feedback from another user, a professional working in a top professional rugby team. I also had a positive experience trialling the technology in my own environment.

These factors, understood together, gave me the confidence to become an advisor for the organisation.

While this article will not get into technologies and systems that scored C or D, there are surprisingly many technologies that have been widely adopted in the industry despite significant limitations. When dealing with these companies, the conversation often stops abruptly when I request the published or internal papers validating the technology.

Disciplined curiosity and culture in sport

Innovation will continue to drive sport forward, especially when managed with due diligence, discipline and clarity. The graded recommendation model does not discourage new ideas, it simply asks them to prove their worth.

In the best high performance environments, new ideas are welcomed, but not accepted blindly. A shared understanding of what good evaluation looks like helps prevent unnecessary turnover in methods; increases staff confidence in the decision-making process; and ensures better integration and adherence.

It is essential to bring coaches, analysts, medics, and athletes into the conversation. The more voices involved, the more robust the decision-making process.

By evaluating methods, tools, and technologies through a structured, collaborative lens, we protect the integrity of our environments; empower staff to make confident decisions; and ensure that athletes experience consistency, clarity, and care.

Using every available solution is neither possible nor the right approach. It’s about using the right things that add value to your environment.