Sunday, November 1, 2020
Shoot RAW, Think Fine, Research, Design

The quantitative research traps

By Admin , in UX Research , at September 15, 2020

In 2016 everybody was convinced that Hilary Clinton was going to be the next and the first woman president of the USA. Even Trump thought so, and he looked very surprised when he won.

After all, every poll and survey (with some small exceptions) had shown Clinton winning.

It was a major screw up of quantitative research. Coupled with the wishful thinking of the media and mass deception from surveyed voters a small percentage error resulted in the greatest surprise of 21st century America.

This and other great examples from history and science should make every researcher cautious about anything with “quantitative” attached to the title. If we base our findings only in quantitative research, we will come to the definitive conclusion that god exists; 90% of the world says so. Furthermore, the scientific method of precise science cannot be fully applied to social phenomena research, where usability research is part of, even though mixed with tech jargon and deceptively often hidden inside a machine.

Quantitative research in usability is always cursed by limited geography, improper targeting of users, and nearly always the exclusion of a good portion of users, that for reasons of introvercy or physical limitations are not part of the data collected.

Another big problem is the inability of environmental control of the data collection. While the quantitative data is piled up with the contribution of many nodes in the system, the analysis failures are owned by the researcher. The conclusions arrived using flawed info can only lead to flawed decisions.

Qualitative research is like history: it cannot be undone! One cannot experiment on it, and it cannot be falsifiable. Any hypothesis that the usability researcher can formulate with not be testable based on the data present. Only a designed experiment can prove or disprove a theory, and apart from Microsoft, Facebook, and other giants, it is impossible to do for anybody else.

A redo is time-consuming and always expensive.

The last and most underestimated element is the researcher. Scientific analysis is done and controlled by groups of researchers. A single usability researcher, even a genius one, will have it impossible to reach the right conclusions, based on initially flawed and truncated quantitative data.

For all these reasons, the numbers of quantitative data must be taken with a pinch of salt, because more often than not they are deceptive. Relying only in cold numbers is a recipe for disaster in later stages of the development of the product.

Keep in mind that even all the scientific data shows that the world is round, for all of us walking on it, for all intents and purposes, is flat.