Original article written by Dean Miller, Charli Sargent and Gregory Roach and colleagues
Background
Just a few years ago, capturing athlete preparedness and recovery metrics could be a laborious and long-winded task, taking up a big chunk of a sport scientist’s day. The recent advent of wearable smart watches, rings, patches and the algorithms that go with them has meant that hundreds of metrics are now available with a quick swipe or a click of a button.
As helpful as that can be, these improvements in technology bring new issues for sport scientists to contend with. To name just a few:
- Which devices should we recommend to athletes?
- What is the accuracy of the measurements they provide?
- Which product claims can we trust, and which should we view with caution?
With the number of products on the market it’s nigh-on impossible to be able to test all of them yourself. Thankfully, since wearable devices are also peaking the interest of many academic researchers, more and more studies are being conducted independently comparing and documenting their accuracy.
So, how do some of the most popular devices stack up in terms of sleep, heart rate and heart rate variability?
What the authors did
The sleep of 53 recreationally active adults (26 female, 27 males, aged 25.4 ± 5.9 y) was studied for one night in a sleep laboratory (9 h from 23:00 – 08:00).
Their metrics were tracked by gold-standard polysomnography (PSG; sleep) and 2-lead electrocardiography (ECG; heart rate, HRV) along with 6 common wearable devices:
- Apple Watch Series 6
- Garmin Forerunner 245 Music
- Polar Vantage V
- Oura Ring Generation 2
- Whoop 3.0
- Somfit
Data analyses were then performed to quantify the validity of the wearable devices against the gold-standard measures.
What the authors found
For sleep, all wearable devices were valid for 2-stage classification (whether the person was asleep or awake). Similar values of agreement with gold-standard PSG were found across the board – 86-89%.
However, agreement was much lower for multi-stage sleep classification (whether someone was awake, in light sleep, deep sleep or REM sleep), with more pronounced differences between devices. The Garmin device had 50% agreement, with Somfit (an adhesive forehead patch) showing 65% agreement.
For heart rate and HRV, the devices ranged from almost perfect agreement with ECG to poor relative agreement, with Whoop 3.0 performing best and Garmin being worst performing.
Limitations
The researchers reached out to the manufacturers of each device requesting direct data access for their validity assessments. Only Whoop and Somfit provided access, meaning that they were sometimes working with summary data over longer sampling periods for other devices. This may account somewhat for the greater agreement with gold-standard measures seen for Whoop with HR and HRV, and multistage sleep classification for Somfit, and therefore may not completely reflect how these devices perform “in the wild”.
Since the gold-standard measure of sleep (PSG) is hard to conduct outside a lab, studies like these can struggle to capture how well wearable devices perform under many of the conditions that often affect an athlete’s sleep in the real-world. For example, if they’re woken up multiple times a night by the pain of an injury, or by their children, does that change how well these devices perform at detecting these metrics of preparedness and recovery?
What this means for coaches
The most popular wearable devices seem to provide a decent estimate of how long an athlete is sleeping for, allowing us to pick up any general red flags related to sleep behaviours.
Although some wearable companies may place a lot of marketing emphasis on the value of detecting REM and deep sleep amounts, these data suggest that the accuracy just isn’t there right now to read too much into any changes you see in these metrics. If these are of particular interest to you, devices that monitor things like brain activity and eye movement (so head-worn, like the Somfit, or Muse S Headband) may be better albeit still far from perfect.
While most of these devices showed excellent accuracy for heart rate measurement, they each differ in how they calculate HRV. For example, they sample over different periods (the whole night, part of the night, morning measurements), and apply different signal processing algorithms. Depending on how you want to use these metrics, it’s worth doing a deep dive into some of these details to figure out the best device to use with your athletes.
Reviewer’s comments
The pace at which the product industry moves tends to outstrip the pace of academia, meaning that most of the wearable products tested here are now onto next generation versions. Hopefully, that means the numbers we see here are only like to have improved as manufacturers tweak their hardware and update their algorithms.
As we can’t know for sure, it’s always worth running in-house testing with any new products and/or asking manufacturers for these kinds of numbers from their own validity testing before dishing them out to athletes. If companies don’t have these numbers available or aren’t willing to share, that should be a bit of a red flag in itself.
Recommended resources
Read – Wearable technology: Action of distraction for athlete performance – Sian Allen and Pete Tierney
Read – Assessing the accuracy of popular commercial technologies that measure resting heart rate and heart rate variability – Jason Stone and colleagues