How to create an insights robot
The goal of most market research today is “insight” of one form or another. The word “insight” itself has as many definitions as there are market researchers, marketing directors and exasperated advertising executives; but if there is one thing all are likely to agree on, it is that “insight” involves discovering something new. But how can we discover better insights? Often novelty may be the only requirement, and plenty of research, marketing and advertising has focused on acting on “insight” that merely offers the new without further qualification. I’ve been working on figuring out where new ideas and insights come from; in particular I want to understand how new insights come about and how we use this understanding to come up with better, not just new, insights. I would therefore like to be more demanding in my definition, and for the purposes of this essayette “insight” in market research is defined as:
“A tentatively accepted hypothesis that is sufficiently radical as to allow a competitive advantage by acting upon it.”
Logical reasoning
To fully understand how new ideas and insights come about we need to start with the basics; with how people reason about the world and the philosophical foundations that underpin that reasoning. The two most commonly recognised forms of reasoning, familiar to all who have either through interest or basic schooling dabbled in philosophy, are deductive and inductive reasoning. To quickly recap the two: deduction means determining the conclusion (‘People buy umbrellas’) by using a rule (‘When it rains, people buy umbrellas’) and its precondition (‘It is raining’); induction means determining the rule (‘When it rains, people buy umbrellas’) after numerous examples of the conclusion (‘People buy umbrellas’) following the precondition (‘It is raining’).
What basic philosophy lessons tend to not cover is a third sort of reasoning, related to induction, called abduction. Casting aside any suggestions of aliens or illegal activities in the phrase, abduction was first popularised as a form of logical reasoning by Charles Sanders Peirce almost 150 years ago. His insight was that by rearranging the Aristotelian syllogism (rule, conclusion, precondition) you could develop abductive inferences. To use our earlier example, abduction uses the conclusion (‘People are buying umbrellas’) and the rule (‘When it rains, people buy umbrellas’) to support the underlying reasoning that the precondition (‘Therefore, it may be raining’) could explain the conclusion. You may have noticed that rather than a certainty, the abductive method provides a plausible explanation, or hypothesis, for the conclusion. It is this openness to uncertainty that is key to how I would argue abduction should inform insight discovery in market research – but first let us have a closer look at Peirce’s reasoning as he developed it towards practical application.
Pragmatism
After setting the theory, Peirce focused less on the logical form of abduction and more on its practical function within the creation of scientific knowledge. He identified that abduction was key in the creation of hypotheses, and that along with deduction and induction, abduction formed a part of a process of developing scientific knowledge. This process would go something along these lines:
- An anomaly, or surprising observation, is made which cannot be explained by our existing knowledge.
- A plausible hypothesis, or “flash of insight” is developed to account for the anomaly. (abduction)
- Predictions must be made which will be found to be true if the hypothesis is correct. (deduction)
- Experimental observations are then made and compared against the predictions. (induction)
- These observations when compared against the predictions allow us to accept our hypothesis, or discover new anomalies which invalidate the hypothesis and starts the process again.
All this may seem familiar, or obvious, but I would like to draw attention to the way the second stage, abduction, has been much neglected by market research. While most research busies itself with the application of predictions (e.g. product or marketing concepts) to observations (research data on the reception of said product or marketing concepts), there is less importance being placed on the development of the underlying hypothesis. Hypotheses are the beginning of a good insight, but the creation of hypotheses is rarely given the emphasis it deserves in the research process. After all, we already know what we need to know, right? We just define our objectives or research questions in the brief and wait for the insights to roll out at the other end. Which simply isn’t the case. The quality of the initial hypothesis (precondition, or ‘guess’) at the start of the research process has a huge bearing on the quality of insight you end up with at the end of a project. As an industry, market research is already very efficient at invalidating/validating predictions – the challenge is to figure out how to develop radical yet plausible hypotheses that underlie the predictions. In less scientific terms, how can we use this idea of abduction to make more creative, radical guesses which, if validated, would deliver better insights or ultimately provide a greater advantage for a business?
Using robots and abduction to discover insights
I’m interested in how we can use machines to help us discover insights. In particular how we can get a machine to roughly follows Peirce’s step-by-step explanation of scientific research, above. As with any market research project, the process starts with the knowledge of an anomaly that needs to be explained. This could be that a certain group of people seem different from another group (segmentation) or that another group of people don’t seem to be buying as much of a product as expected (communications), etc.
Once you have an anomaly you want to investigate, you need to develop hypotheses which explain the anomaly. As well as the standard criteria for the creation of hypotheses (falsifiability, directionality, establishment of variables, etc.) for a hypothesis to lead to a good insight it needs to meet one additional condition: it needs to be radical. If proved valid, the hypothesis should be different enough from our existing knowledge as to allow a business to gain some advantage by acting upon it.
This “radicalness” comes about by joining disparate ideas together. Ideas are never completely original. Ideas beget other ideas and new ideas are an amalgamation of those that have come before them - Stuart Kauffman’s notion of the Adjacent Possible does a great job of explaining this process. The Adjacent Possible refers to everything that is possible by combining what we already have; for instance all the elements which made up the Gutenberg press – the movable type, the ink, the paper – had all been invented before the press itself, and only by putting them together could the Gutenberg press have been conceived. The interesting implication being that each step you take leads to a whole new set of possibilities. Whilst the Adjacent Possible helps us consider what can be created, it also reminds us of the limits of creation. At any given point in time there are only a certain number of possible ideas adjacent to where we are now. The same is true for creating hypotheses by joining together disparate ideas in new ways you can develop better hypotheses, that are radical yet still within the realms of possibility.
It follows that the more ideas and information you have access to, and the better you are at connecting those ideas in a meaningful way, the better your hypotheses and ultimately the better the insights. As such the first job of any insights machine is to gain as much knowledge, information and as many ideas as possible. At this point in time the logical place to start is the Internet - the world’s largest and most accessible database.
The next step is to link these ideas together in unexpected ways to develop hypotheses which are sufficiently plausible and radical. This can be done using what’s called “object-relation-object” triplets:
1. Object (or conclusion): Take the knowledge that we already know about the anomaly we are trying to explain and use this as a starting point. For example, if we are looking to understand why a particular group of people is different from other groups and our previous research says they like cats and tend to live in the US, the machine is programmed with this information (location US and lover of cats) and then pulls all the information we have about cat lovers in the US, from the internet and any other sources we have access to.
2. Relation (or rule): The next step is to link this existing knowledge object to another object, to perform some sort of “logical leap” to connect one idea to another. There are different types of relationships or rules that can be used to link ideas together. We currently focus on a few different sorts of relationships, one of the most basic being word proximity. For example, if we look at all the information related to cat lovers in the US and we find that the noun “knitting” frequently occurs in close proximity to “cats” we can infer that there is some link between ”cats” and “knitting”. There are many other types of logical leaps you can make using different sorts of textual analysis include spotting linking verbs, using dates or other meta-data like tags, introducing elements of randomness, or indeed using a combination of different relation types.
3. Object (or precondition): Using this newly connected knowledge to create a hypothesis. The final part of the puzzle is to analyse the information in away which balances looking at objects/knowledge which have lots of connections and those which are obscure enough to suggest that some form of competitor advantage could be gleaned if the connection and preceding hypotheses were proved to be valid and reliable. You couldn’t rely simply on the strength or frequency of connections between objects to discover potential insights because the objects with the strongest connections tend to point to obvious or common-sense hypotheses, which would prove to be perfectly valid, but wouldn’t provide much a advantage for a business because they are so well known.
The full realisation of this approach isn’t quite complete yet. But it’s not so far off.
