Five dysfunctions of ‘democratised’ research. Part 2 – Researching in our silos leads to false positives

@leisa’s second post about the common dysfunctions about ‘democratised’ research, this focusing on researching in silos, the query effect and false positives.

How do we fall victim to the query effect?

By focussing our research around the specific thing our team is responsible for, we increase our vulnerability to the query effect. That little feature is everything to our product team and we want to understand everything our users might think or feel about it, but are we perhaps less inclined to question our team’s own existence in our research?

What is a false positive?

Research that is focussed too tightly on a product or a feature increases the risk of a false positive result. A false positive is a research result which wrongly indicates that a particular condition or attribute is present.

Why are false positives a problem?

False positives are problematic for at least two reasons. Firstly they can lead teams to believe that there is a greater success rate or demand for the product or feature they are researching than is actually the case when experienced in a more realistic context. And secondly, they can lead to a lack of trust in research – teams are frustrated because they have done all this research and it didn’t help them to succeed. This is not a good outcome for anyone.

How do we avoid positives and gain more relevant insight?

The role of the trained and experienced researcher is to not only have expertise in methodology but also to help guide teams to set focus at the right level, to avoid misleading ourselves with data. To ensure we not only gather data, but we are confident we are gathering data on the things that really matter. Even if that requires us to do research on things our team doesn’t own and cannot fix or to collaborate with others in our organisation. In many cases, the additional scope and effort can be essential to achieving a valid outcome from research that teams can trust to use to move forward.