Using experiments in consumer research

Published 01|03|10

Foreword

1.1 Why did we decide to commission a report on experiments?

Ofcom has recently been doing some work on behavioural economics and its implications for regulation. Behavioural economics explores insights from psychology on the behaviour of individuals, and uses these to develop a more realistic understanding of how individuals make decisions. Traditional economics assumes that individuals are good at making decisions: behavioural economics relaxes this assumption and highlights that individuals are subject to cognitive limitations, impulses and emotions, which can lead to apparent "errors" or "biases" in decision-making.

A key insight from the work we have been doing on behavioural economics is that evidence on actual consumer behaviour can be helpful in understanding how consumers react to different market features, and what type of remedies are likely to be successful. Behavioural economics has recently garnered a lot of interest in policy-making, and some organisations have started to use experiments to understand how behavioural biases can influence outcomes in particular markets. Experimental-style techniques are well-established in areas such as consumer product research, but have only recently been used more often in regulation and policy-making contexts.

For example, the Federal Trade Commission in the US commissioned an experiment to understand how consumers would react if they were given information on the commission received by mortgage brokers. The experiment yielded the interesting result that with additional information, consumers were less likely to choose the lowest cost product. Additional information in this case appeared to confuse consumers rather than improve consumers' decisions . This is one example of the potential of experiments to uncover effects that traditional theory may not predict, and provide more evidence on the effectiveness of remedies before they are implemented.

Other organisations which have commissioned experiments in the UK include the FSA and the OFT . The OFT and Competition Commission (CC) have recently published a report on Road-testing of consumer remedies which describes how testing remedies can help to identify the most effective remedy and to fine-tune its design. The OFT and CC report also discusses different ways of testing remedies, including qualitative methods, simulation, quantitative surveys and economic experiments. These different methods all have strengths and weaknesses, and in some cases, some types will be suitable than others. For example, the report highlights that experiments provide data on actual consumer behaviour and allow the performance of different remedies to be compared. However, the report also notes that care needs to be taken in extrapolating the results of an experiment to the real world, and drawing inferences about how much real world consumer behaviour may actually change. A general conclusion of the report is that "which road testing method is employed, how it is employed and whether more than one method is used depends on the particular policy question and what type of information is required to answer that question."

We consider that experiments may have a role in developing policy for markets that Ofcom regulates. We therefore commissioned a report from London Economics and University College London (UCL) to investigate the potential uses of experiments in understanding consumer behaviour, and to develop a better understanding of the potential benefits of experiments if used in Ofcom's work.

1.2 The potential uses of experiments

Experiments test the actual behaviour of individuals under different conditions. In an experiment, individuals may be faced with choices to make under different circumstances, and the experimenter is able to observe how consumers react. The main advantages include:

• Experiments allow you to observe what consumers actually do, not what they say they will do.
• They allow testing that may be impossible or very expensive to carry out in the actual market.
• It may be easier to establish causality as the experimenter can vary one feature at a time.

However, set against this:

• Experiments may not be able to provide information on the likely magnitude of effects in the real world.
• Where information on consumers' beliefs and perceptions are required, other types of testing are likely to be more useful.

Where the effectiveness of a remedy depends on how consumers react, testing (potentially using methods such as experiments) may play a useful role. However, as highlighted in the OFT and CC report on Road-testing of consumer remedies, it will not be useful or appropriate to use experiments across all areas of our work. In some cases, other types of research will be more informative. We already undertake substantial research on consumer behaviour, and conduct a number of different types of research. We therefore believe that the use of experiments is an additional tool that complements the work that we already do.

To assess the potential benefits and limitations of experiments in Ofcom's work, we asked London Economics and UCL to carry out an experimental study in the specific context of devising effective ways of providing information on the price of telephone calls.

1.3 Experiment on providing price information

This specific experiment was designed to consider a number of options for improving the understanding of consumers when choosing services funded through call charges.

Consumers are able to find out the cost of a call in many ways. All phone companies are bound, to a greater or lesser extent depending broadly on the services that they offer and to whom, by rules called General Conditions which require them to be transparent about the cost of calls. Where applicable, these General Conditions require them to publish details of call prices and ensure that their customer service staff point consumers to these charges.

There may also be other sources of information where consumers can find out about call tariffs. For example, for certain types of calls such as Premium Rate Services, some information about the cost of calls is published by the provider of that service.

However, consumers in general do not always know the exact cost of a call before they make a call, and we have identified that this uncertainty can lead to consumer detriment.

Ofcom has in the past looked at developing policies to address this potential uncertainty. For example, the National Telephone Numbering Plan (the Numbering Plan) sets out the allocation of telephone numbers to communications providers and the permitted use of those ranges (for example, restrictions on the type of service or BT's retail price).

However, many of the price-related requirements n the Numbering Plan only apply to BT, and other communications providers (CPs) are able to charge their own tariffs for calls to individual number ranges. Consumers therefore often do not know the exact tariff of a call before making that call, unless they have actively looked up their CP's prices.

We are currently engaged in considering options to address identified consumer concerns around price transparency in this area. This review is one of Ofcom's proposed projects in its draft Annual Plan for 2010/11. It will be undertaking an examination of the impact on consumers of relevant aspects of the current regime and will analyse the full costs and benefits of different remedies to address any identified consumer problems.

In the past, Ofcom has considered a number of informational remedies to tackle these problems, such as pre-call announcements at the point-of-call and improved provision of price lists. There are clear costs to any informational remedy. For example, we have found that the cost of pre-call announcements in particular is likely to be very high because of the high degree of variation in charges within a number range. There are also other relevant considerations that need to be made to ensure such interventions are viable, for example, we need to consider the risk of pre-call announcements disrupting certain machine-to-machine calls, including personal, fire and burglar alarms, leading to potential life-threatening consequences .

Nonetheless, in assessing the benefits to consumers, we have previously found it very difficult to predict how effective these measures would be if introduced. For example, stakeholders have previously suggested that consumers may find pre-call announcements annoying and, if offered the choice, would switch off the feature shortly after its introduction, thereby limiting the effectiveness of such a facility.

Our experiment therefore sought to understand how different interventions may influence callers to make better decisions when making a call. In this specific experiment, participants performed best when provided with pre-call announcements that stated the exact call price, however, consumers appeared to benefit to some degree from all types of interventions. The experiment also provided some interesting results which might not immediately be predicted by traditional theory. These included:

• the ability of participants to learn to make better decisions over time, regardless of the intervention; and
  
• total bill costs can impact future behaviour. However, the impact on future behaviour is driven by the size of the total bill, rather than the information provided on the bill on the prices of individual calls and the associated per minute charges.

The experiment has provided some helpful indications of the type of interventions that are most likely to be effective in this situation, although it did not test the full range of possible interventions. The outcome does need to be reviewed with some caution as it only provides us with evidence on one aspect of the issue. As noted above, experiments can be very helpful in indicating the relative effectiveness of different interventions, but care needs to be taken in extrapolating the results to the real world.

For example, the report notes that interventions that do not work well in the laboratory experiment are unlikely to work in the real world. However, interventions that appear to work well in the laboratory experiment may not perform as well in the real world. It is also difficult to predict and quantify the size of real-world welfare gain that different interventions may deliver. We would therefore need to look to other forms of analysis to assess the likely size of real-world welfare gain, and weigh any potential benefits against the full set of costs, which as described earlier, may be substantial.

We will further consider these findings when we review our approach to regulating non geographic call services to ensure that the regulatory framework delivers optimal consumer outcomes in terms of range of services and clarity and appropriateness in costs and charges.

1.4 Conclusions on the use of experiments

Our work on behavioural economics has further highlighted the usefulness of consumer research in developing policy. We consider that experimental-style techniques are likely to act as a useful complement to the substantial consumer research that we already do. However, as with other research methodologies, there are advantages and disadvantages of experiments, and their appropriateness will vary depending on the context. It is unlikely that carrying out experiments will be helpful in all cases.

Therefore, nothing in the report means we will necessarily use experiments in all cases and any action we take, in any particular case but, for example, in enforcing general consumer protection law, is subject to the position under existing law, including rules about evidence, which Ofcom applies.

The full document is available below