Solving Market Failure

What makes markets break, and how can we prevent it from happening?

Traditional economic theory tells us that ‘information failure’ is a form of ‘market failure’ which occurs when anyone anywhere lacks the ‘perfect information’ necessary to enable proper decision making.

“With perfect information in a market, all consumers and producers have perfect and instantaneous knowledge of all market prices, their own utility, and own cost functions.” Wikipedia

This article makes three central claims:

1. Firstly, all market failures can ultimately be conceptualised as either problems surrounding the provision of public goods, or problems resulting from information failure.
2. Secondly, true public goods such as national defence are few and far between. Many oft-cited public goods are in fact excludable, or rivalrous, and may be club or private goods.
3. Thirdly, all information failures are addressable through innovation and improved technology.

These claims lend themselves to a profound conclusion. If we accept they are true, we must also accept that the vast majority of market failure is remediable through technological progress. And if we believe that, we may then conclude that in time technology stands to obsolete all market regulation.

Public goods are those goods and services which are both non-rivalrous and non-excludable by nature. This is to say, the marginal cost of providing them is zero, and it is impossible to prevent people who have not paid for them from benefiting.

So what is market failure?

Typically we think of markets, through price signals, as efficiently allocating goods and services across economies. But in practice, markets are often not efficient, and there are many types of market failure which lead to losses in net social welfare, and pareto suboptimal outcomes.

Leaving public goods aside, which are under-provided and under-consumed when left to free-markets, examples of market failure include allocative and productive inefficiencies, non-competitive markets with high degrees of monopoly or monopsony power, and missing or incomplete markets.

What is information failure?

Information failure comes in many forms, but can be separated into two distinct categories: failings which arise from a lack of access to information, and failings arising from an inability to process it.

Human inability to process information stems from our inherently ‘boundedly rational’ nature as creatures. All of us (even the best) have brains only so big, and therefore possess necessarily limited cognitive processing power. Too much information, too many variables, and an incomplete or incorrect understanding of how systems work stymie our ability to accurately compute outcomes, and make rational choices. We may also exhibit cognitive biases such as ‘time-inconsistent preferences’, and fall prey to traps such as the ‘sunk cost fallacy’, leading us to make uneconomic or irrational decisions with predictable regularity.

Lack of access to information meanwhile results from issues which can broadly be divided into ‘known unknowns’, and ‘unknown unknowns’.

In the case of known unknowns, informational availability may be restricted due to cost. The act of collecting or gathering information may be costly to the point that it proves prohibitive to exchange at Pareto optimal levels. This may be because information is hard to gather, or exists but requires further work to make useful (e.g. translation from one language to another). Time-lag in reporting may also result in information being outdated or of questionable relevance by the time it reaches an individual.

In the case of unknown unknowns, ignorance may exist for many reasons: the inaccuracy of available information and conclusions subsequently drawn, incomplete or unreliable information sets, or (following the logic herein) in any number of other unconsidered ways. With respect to recent events and emerging trends, adversaries may also deliberately seek to undermine the integrity of information stores, or otherwise misdirect consumers of information, for their own benefit.

It is also the case that sometimes one or both parties in a transaction may not know exactly what information is relevant in a given scenario. Where nobody knows a piece of information’s relevance, there is unrealised ‘Knightian uncertainty’. Ongoing scientific research and discovery reduces this kind of uncertainty. In other circumstances, where one party possesses relevant information but the other does not, we say there is ‘information asymmetry’. Such asymmetries need not result from malicious action on the part of the enlightened party (e.g. by deliberately obscuring relevant facts from view), but frequently emerge through accidental omission and natural variation in individual preferences (for instance where one person’s prime concern isn’t even considered by another).

These ‘root’ information failures combine to give way to many secondary failures: chiefly amongst them imperfect communications between parties, and an inability to coordinate activities or arrive at a consensus even where mutually beneficial. Collective action problems represent a prime example of this.

Expressing types of market failure as information failure

Expressing market failures in terms of the informational failures which contribute to them enables us to see how informational and communicational solutions (technology) may mitigate or eliminate market failure.

Allocative and productive inefficiencies

These result from:

1. price signals not being communicated accurately, completely, or reliably; and
2. human failure in interpreting price signals (i.e. falling prey to cognitive biases, such as distrusting accurate but unexpected market-wide data which may not align with one’s anecdotal or localised knowledge).

More sensors (IoT), and better tools for information fusion, visualisation, and inference (big data) solve these problems.

Monopoly and monopsony power

Monopolies have long been held by policymakers to be bad (because competition inversely is seen as positive). But monopolies are neither inherently nor necessarily bad, can adequately reward investors for the risks they take, and in practice are often the only entities with the requisite scale to conduct truly ambitious technological research and development. This thinking is reflected in the existence of institutions such as the patent system which grants inventors state-assured temporary monopolies for precisely this purpose.

We know that over time, even badly behaving monopolists who gorge consumers by generating super-supernormal profits, will eventually come under attack from other rent-seeking would-be monopolists (see: Taxi cartels’ being supplanted by Uber). And these challenger monopolists will in turn face their own challengers (see: Didi Chuxing, Lyft, MyTaxi, and a dozen others).

Whilst challenging dominant monopolies requires greater resources than most individual entrepreneurs can muster, compelling innovators may raise the requisite capital from alternative sources. State investment vehicles, and supersized venture capitalists such as Softbank will enable this going forward. Such wars between monopolists and their challengers will generally prove good for consumers, resulting in the large-scale redistribution of surplus value to them (e.g. through cheaper services, safer/better service, and more anytime availability). Provided consumers are given freedom of choice and there remain enough diversified active investors to support challenges, at the end of the day Schumpeterian creative destruction will prevail. Just because a service has a monopoly does not mean it is immune from competition. Behaviour branded anti-competitive by regulators, such as predatory pricing by incumbents, can in fact be good for consumers. And that which is not (for instance collusion) tends to be temporary in nature and especially susceptible to external disruption. Most “anti-competitive behaviour” can be effectively addressed (and deterred) through greater deployment of capital. Whilst regulators worry about breaking up firms like Google and Facebook, the truth is that they would make the best natural competitors to the eCommerce monopoly that is Amazon.

Missing markets

Something is said to be Pareto optimal when “it is impossible to reallocate [resources] so as to make any one individual or preference criterion better off without making at least one individual or preference criterion worse off.” (Wikipedia) But sometimes markets don’t exist where Pareto-optimality might lead one to expect they should. Even after realisation, this may remain the case for some time, for a number reasons: high transaction costs, a lack of trust between parties, time-lag in exchange, and failures in coordination. These are all informational problems.

1. High transaction costs occur when the means of setting up exchange for transactions of that type are prohibitively expensive. The internet is a large global marketplace that reduces search and information costs, helping buyers and sellers determine what goods are already available on the market and at what price. As a function of the distance between the bid and the ask, bargaining costs can be reduced through the internet as a real-time, globally accessible, costless communication platform.
2. A lack of trust can be remedied through increasingly globalising and standardising reputational enforcement systems such as review mechanisms and public transaction histories on exchange platforms like Airbnb, eBay, and Amazon. Policing and enforcement costs are consequently reduced, and the need for any two individual parties to have bilaterally established trust before engaging in exchange is eliminated. The threat of a bad review permanently damaging one’s status incentivises consumers and suppliers to behave well.
3. Technology’s role in increasing the pace of exchange should be obvious to anybody who has used Amazon Prime, ordered a pizza from Deliveroo or Uber Eats, FedEx’d a letter across the world in a single day, or even just flown on a plane themselves. Whilst historically a slow pace of exchange may have rendered exchanges obsolete by the time both parties became ready to transact, the time it takes to discover solutions, ship products, and process payments has been cut significantly (and in the case of digital products reduced to virtually nothing).
4. Coordination failures and collective action problems are likewise being rendered non-issues by the advent of technology. Group voting and crowdfunding mechanisms such as Kickstarter, and even their business equivalents such as Crowdcube, enable interested members to transact when they otherwise would not be able to. Platforms like Airbnb have unlocked economic surplus owned by ordinary middle-class homeowners, never before realised. In a redistribution of wealth away from a homogenised hotel industry, the service has provided consumers with a greater variety of choice, allowed more people than ever to enjoy lower-cost holidays, and turned many hundreds of thousands of people into buyers and sellers now they are able to quickly and cheaply locate and communicate with one another. This has never before been possible in human history, but thanks to technology it is, and in any market imaginable. Whilst this increase in demand may put upwards pressure on prices (e.g. in the housing market), in a free market suppliers would step in to provide (and take advantage) of the profit opportunities presented. It is only when regulation artificially constrains supply (i.e. preventing the building of new homes), that problems arise. The threat comes not from Airbnb, or any death march of technology, but market regulation which however well intended exists rarely without unintended consequences.

Negative externalities

Negative externalities (such as pollution) that result from consumers’ behaviour can be reduced through increasingly connected social networks, and an improved technological ability to reliably attribute and publicise bad behaviour. Attribution problems (informational availability issues) currently prevent the naming and shaming of many polluters, and only the most obvious offenders are subjected to any sort of social sanction. Social punishment (boycotts, opprobrium, or other action that might deter those propagating negative externalities) often does not occur because free-riders are hard to detect. Similarly, a lack of social rewards for good behaviour often result in its under-provision (think contributions to open-source, or nice Wikipedia editors). But improvements in sensors, the growth of reputation platforms, decentralisation of the means of communication, and improvements in global identification all contribute towards making more effective non-governmental reduction of negative externalities a likely reality. And even without this, technological innovation within a category of product may further result in diminishing its negative externalities (e.g. Juul versus traditional tobacco, or natural gas versus coal).

Limitations to human cognitive processing power

Through growth in open data, and advancements in artificial intelligence and modelling/simulation tools, decision-making can be enhanced through software agents capable of making decisions on an individual’a behalf. For instance, by automatically switching energy supplies to the most competitive provider, as proposed in a new FSB/Fingleton report entitled Open Energy.

In Conclusion

Technology will not replace governments and the law, but it does have the potential to make the vast bulk of existing market-facing regulations redundant in the near-term future.

We will never have perfect information. But to think that today’s technology (nevermind that imaginable tomorrow) might do a better job than current regulations — avoiding the deadweight losses, market distortions, and perverse incentives they produce — is no stretch of the imagination. Vested interests seeking to protect existing rules will make change slow, but we must not let them block progress or advance techno-dystopian narratives.

At the same time, path dependency matters.

Designing an ideal state from scratch, few would imagine the world we live in today. But it is imperative to recognise that the world we inhabit is the one of the present, and it is the starting point for any change. It is important to think about how to get from where we are to where we know we need to be. Blue-sky thinking, although well and good, still requires pragmatic political economists to realise it. As such, we might still recognise the role which regulators may play in enabling the emergence of sustainable, purely social norms.

If you liked this article, please check out HASH, a real-time modeling and simulation platform which aims to enable anybody to capture the complex adaptive systems we see around us. If eliminating information failure sounds up your street, we’re also hiring.

If you’re interested in working with us, please get in touch at hq@hash.ai

Special thanks to John Myers and Evan Davis for their feedback on earlier drafts of this article.