Experience from the reindeer review. Interview with Jon Moen
‘There needs to be a proper discussion with stakeholders about what sort of evidence base they want and what answers researchers are able to deliver,’ says Jon Moen, who chaired the reindeer project. He found the review both instructive and frustrating.
Long, hard journey to destination
Jon Moen, a professor of ecology at Umeå University, has been researching land use issues for many years, not least conflicts between different interests all wanting to use mountain areas for their own ends. In the late 1990s, he blew a hole in the idea that grazing by reindeer is generally harmful to vegetation, but also realised that the question called for collaboration with political scientists and historians. When he agreed to chair EviEM’s reindeer grazing project in 2012, it was because he was keen to get an overall picture of research on the subject. And to learn how a systematic review was carried out. It was the start of a long journey.
‘None of us on the review team had any idea how much work would be involved, and consequently we hadn’t planned enough time for it in our schedules. It took much longer than we expected. There was almost a year of “bookkeeping” discussions – about how to delimit the question, what studies to exclude and why – before we got to the fun part: the scientific analysis. It was quite frustrating.’
The initial screening of the literature was relatively plain sailing. But once 100 studies had been selected for close reading, things became harder. A systematic review is far more detailed than an ordinary research review.
‘Every step has to be documented. And at each one there are countless decisions to be made. If you haven’t worked that way before, you don’t appreciate what the problems are.’
In practical terms, it would be a great help – and save a lot of time – if review teams did not have to build up a database themselves to handle all the data, Moen believes.
‘All projects will come up against the same problems. It’s a bit impractical having to reinvent the wheel every time.’ He feels in two minds about the systematic review approach.
‘It’s always good to work in a structured way – it forces you to think about your basic assumptions. But as a systematic review is time-consuming and quite costly, you also have to be able to show that it offers added value. I’m not sure that you can. Not because there’s anything wrong with the process, but because the scientific data from conservation research are so disparate, or at any rate may require a lot of interpretation.’
His conclusion is that the basic premises of a review need to be made clearer.
‘There has to be a clear link between the question asked and the decisions stakeholders need to take,’ says Moen. ‘What sort of evidence base do they want? And what are researchers able to deliver? As I see it, there’s a great need for an informed discussion on this.’