Reading between the research lines – factors in studies that can affect how much we can trust the impact of the study.
In this podcast, Dr Gillian Tabor discusses the intricacies of evaluating research in animal physiotherapy, particularly focusing on horses. She emphasises the importance of understanding research limitations, the necessity of robust methodologies, and the critical evaluation of statistical significance versus clinical relevance. The conversation provides insights into how to effectively critique research papers and apply findings to clinical practice.
Dr Gillian Tabor is MSc ResM MCSP HCPCreg, a Chartered Physiotherapist, Cat A member ‘Association of Chartered Physiotherapists in Animal Therapy and has lots of great help and advice for horse owners – check out all of her work here – Dr Gillian Tabor
CLICK here to head over to the main Podcast page to see the complete list of podcasts available.
Transcription
Here is a transcription of the podcast. (Please note this is an automated transcription, so we apologise for any errors from the original podcast.)
Hello everybody, this is Gillian Tabor. I’m a physiotherapist, and you may have listened to some of my previous podcasts where I’ve done some reviews of research papers. As well as being a practising clinician, I specialise in the treatment and rehab of horses. I also work at Hartpury University, teaching students how to progress their human physio skills to being able to treat animals. Additionally, I supervise and lead some research there. I have read a number of papers, conducted my own doctoral study, and I have what they call an early career researcher view on the main topics related to interventions in the treatment of horses, especially from a musculoskeletal perspective, as well as a fundamental understanding of biomechanics research.
I thought it would be beneficial to put together a podcast following a list of ideas to help others critique and understand research. The reality of carrying out a research study is that there are always limitations. It is incredibly difficult to conduct a completely robust study that is free from any criticism. As researchers, we become accustomed to this, but there are some very obvious flaws in research that can make us question its conclusions. It might seem that critiquing research is overly harsh, as people work extremely hard on their studies. For students conducting research, it can be particularly challenging to achieve a high level of quality given the resources available. However, there are clear reasons why we should not always trust the conclusions drawn from certain studies.
When listening to critiques, it might seem that we are being overly critical, but it is essential to be able to read between the lines of research and understand why certain studies may not be as reliable as they appear, even if they claim to prove something groundbreaking. On the other hand, if a study is conducted without these limitations and has ticked all the necessary boxes, we can have confidence that its conclusions are valid and have a meaningful impact.
Research papers typically begin with background and rationale, and it is crucial for the authors to review relevant research. For example, if I were investigating the use of laser therapy, I would not expect to find references to the benefits of ultrasound or shockwave therapy unless they were directly relevant to the research. If there is limited research in the area, the authors should clearly state this and demonstrate that they have thoroughly explored existing studies without missing anything significant. The introduction should end with clear aims and objectives, and ideally, a hypothesis. A hypothesis should state whether a difference is expected between two variables or whether an improvement is anticipated.
Research aims to falsify claims through methodology and results. If a claim is unfalsifiable, no amount of money, time, or equipment can disprove it. This is where pseudoscience can be problematic. An example I use is the idea that there is a chocolate teapot orbiting the Earth—this cannot be disproved, but that does not mean it is true. It is essential to scrutinise the aims, objectives, and hypotheses of research studies carefully.
Poor methodology is another significant issue in research. It is frustrating when a research team has spent an inordinate amount of time designing and conducting a study but fails in fundamental aspects of methodology. One of the most critical elements of a robust study is the presence of a control group or a comparable group with a change in one variable. A control group consists of subjects that do not receive the treatment or intervention being tested. This could be a separate group of horses or the same horses at a different point in time.
If a study lacks a proper control group, it becomes difficult to attribute changes solely to the intervention. In crossover trials, half the subjects receive the intervention first while the other half serve as a control, before switching roles. Randomisation of subjects into groups is also crucial. Additionally, in some cases, a placebo or sham treatment is necessary to ensure the person assessing the outcomes does not know which group the subjects belong to, thereby minimising bias.
One example of a flawed study design involves a trial assessing bone mineral density changes when horses stand on a whole-body vibration plate. In this study, the treatment group stood on the plate, while the control group remained in their stables. However, this design fails to account for the possibility that simply walking to the plate, rather than the vibration itself, may have influenced bone mineral density. To improve this study, the control group should have also walked to the plate and stood on it, albeit without the vibration.
Bias in research is another concern, particularly when the person conducting measurements is aware of which subjects are in which group. Studies that control for this, such as the Stubbs et al. paper on multifidus changes with dynamic mobilisation exercises, ensure that the person analysing results is blinded to the groups. In contrast, a study on PEMF rugs failed to blind the person measuring range of motion, potentially leading to bias in the results.
When assessing research, it is also important to consider the number of outcome measures being tested. In human studies, strict criteria often limit the number of primary and secondary outcome measures. However, in equine research, studies frequently measure numerous variables, such as pain, range of motion, muscle tone, behaviour, and stride length. Given that statistical significance is often set at p < 0.05, meaning a 5% probability of results occurring by chance, multiple outcome measures increase the likelihood of false positives. If a study measures 20 variables, it is likely that at least one will show significance purely by chance.
Beyond significance, researchers must also consider the reliability and validity of their outcome measures. A reliable measure produces consistent results, while a valid measure accurately assesses what it claims to measure. For example, if we attempted to measure lameness by the colour of a horse’s head collar, this would be highly reliable (as colour can be consistently identified) but entirely invalid. Therefore, new measures should be tested against gold standards, such as using 3D motion capture systems to validate AI-based assessments of upper body asymmetry.
Standardisation of measures is also essential, particularly in multi-centre studies where data is collected in different locations. Training and calibration of assessors ensure consistency in measurements. The Ridden Horse Pain Ethogram, for instance, involves training to standardise assessments across different users.
Confounding variables, such as differences in exercise levels, breed, or age, must also be accounted for, as they can influence results. In terms of statistical analysis, it is crucial to apply appropriate methods and clearly justify their use. Many studies rely on a p-value of 0.05, but statistical significance does not necessarily equate to clinical significance. For instance, a 0.3-degree change in joint angle after a manipulation may be statistically significant but clinically irrelevant if the total range of motion is 90 degrees. Effect size tests, such as Cohen’s d, provide additional insight into whether changes are meaningful in practice.
The discussion section of a research paper should not only interpret results but also acknowledge limitations. If authors openly discuss the constraints of their study and suggest how future research can address them, their conclusions are more credible. However, over-extrapolation of results is a common issue, where researchers make claims beyond what their data supports. The conclusions should align with the study’s aims and objectives, and if in doubt, readers should refer back to the introduction to verify this alignment.
I hope this has been helpful. Some of you may already be familiar with research evaluation, while others may find this new, but it is important to understand how to critically assess research to determine its impact on clinical practice. As always, if you have any questions or would like to continue the discussion, please do. Enjoy reading research papers!
