There is no shortage of predictions about the electricity industry. Every week, it seems, experts excitedly announce the coming of another grid transformation. These are electric times.
Yet every energy prognosticator should maintain a sense of humility—forecasts about the future of energy are notoriously inaccurate . In the past decade alone, we’ve heard over-inflated promises of a nuclear renaissance and persistent underestimations of how much the costs of solar, battery storage, oil, and natural gas would decline. Indeed, energy prognosticators have a talent for just one thing: being wrong.
These prognosticators often predict a glorious future for a flashy technology. What they overlook, however, is that the success of a new technology depends on a number of underlying economic factors. Only once we understand these conditions can we seek to smooth the path of innovations to market. In the case of the electricity market, it is the regulatory environment (state and national) that largely determines its openness to innovation. But we should not imagine a zero-sum battle, with regulation on one side and free markets on the other: Rather, the key question is whether regulation should facilitate or substitute for competition.
A century ago, government policy said the answer was “substitute.” State and local governments treated electricity as a “natural monopoly,” granting a single company exclusive rights to a service territory. In exchange, monopoly utilities let governments fix prices based on their estimation of prudent costs and a reasonable rate of return on capital.
This “regulatory compact” incentivized utilities to accumulate assets like facilities and equipment, and to develop business models that relied on favorable regulatory treatment. Travis Kavulla, vice chairman of the Montana Public Service Commission, notes  that this “so-called cost-of-service regulation suggest[ed] to the utility that it should spend as much as possible, even when less might do.” He continues, “The barometer for whether an investment [was] wise for a utility [was] not capital productivity, but whether expenditures [would] be disallowed by the regulator. This seldom occur[red].”
Like most monopolies, the utility companies had an allergy to change. Harvard economist William Hogan sums up the situation nicely : “The more innovative the idea, the less likely it would [have been] adopted.” Furthermore, since monopolies “socialize” risks by saddling captive ratepayers—not investors—with any costs associated with their investments, the utilities did little to reduce those risks. The monopolies had no reason to develop economic discipline, so they did not, and thus they also did not pursue the technologies that would have cut costs or improved service.
By the late 1980s, monopoly utilities around the world were imposing high costs on customers and had accumulated unnecessary assets, in large part due to errors in technology choice . Thanks to a political backlash, governments began regulatory reforms in the 1990s that overhauled incentives, changed the locus of investment decisions, and shifted the allocation of risk from captive customers to private investors. These reforms, in effect, split up the grid into regulated monopoly and competitive parts. They created competitive wholesale markets for power generation and retail markets for services supplied directly to customers. However, transmission and distribution services (which refer to the transmissionof power from plants to electric substations, and the distribution of power from substations to customers, respectively) remained under the regulated cost-of-service model.
These restructured markets—sometimes referred to as “deregulated” markets—are hardly free from regulation. Indeed, thanks to the unique physical characteristics of electricity, it is impossible to free the market entirely from regulation. Though this may one day change, producers cannot store electricity in bulk in a cost-effective manner, and so have no backup supply in case of shortages. Meanwhile, most end-use customers cannot respond to real-time conditions on the grid, such as by reducing consumption when a shortage occurs. Regulators have thus treated the reliability of transmission as a “public good,” creating regional grid operators to direct the operation of power plants to match changes in consumer demand.
Such regulations result in highly designed markets  that are less organic than most commercial environments. In these markets, the “visible hand” of regulation facilitates competition by enabling the “invisible hand” to go to work. Admittedly, natural gas prices rose after the market reforms of the 1990s and 2000s, causing electricity prices to increase. Critics pointed to the increase as proof that restructuring was not cost-effective. But over the course of the past decade, the cost of gas and of various technologies has declined, unleashing a wave of new investment and renewed faith in markets.
Although a far cry from laissez faire, restructured markets have been an immense improvement over the monopolies of the past. The markets of the future, however, will be better still.
Not all states pursued restructured markets, leaving the United States with a patchwork quilt of local regulatory environments—which today vary widely in effectiveness.
In the states that retain the monopoly model, utilities have built expensive new plants premised on inaccurate economic projections, passing the costs on to captive ratepayers. A particularly egregious example is South Carolina’s abandoned nuclear project : The state poured $9 billion in total into the facility, expecting customers to foot the bill. But South Carolina was not alone in doubling down on wastefulness; monopoly utilities in many states have kept old, expensive plants open when it increased their regulated returns. (The value of a regulated utility’s assets determines the utility’s profitability.) Predictably, this has stifled innovation.
Meanwhile, energy producers in competitive states have built innovative plants to take advantage of the fracking revolution, siting these plants where they have the lowest fuel costs and provide the most value. These low-cost additions  allow competitive states to retire old, dirty, and inefficient power plants. In a quest to make plants more profitable still, merchants have worked to boost output and labor productivity while lowering fuel and other operating costs . Markets in these states also reduced the costs of integrating renewables and complying with environmental rules. (In the latter case, for instance, merchants have used cleaner fuels instead of installing the expensive air pollution controls that monopoly utilities prefer—because those controls boost regulated returns.) Meanwhile, market incentives made consumers more responsive  to grid conditions, rewarding them financially for adjusting their consumption based on the fluctuating cost of supply.
However, even in “deregulated” states, most market rules are based on conventional technologies. Regulation has yet to catch up with the recent wave of innovation. At times, the “visible hand” of market design inhibits competition, such as by creating barriers to entry. Excessive eligibility requirements, for instance, have slowed the adoption of energy storage technologies. In addition, the architects of the old “visible hand” assumed a one-way flow of power, from centralized power plants to relatively passive consumers. They didn’t expect the advent of distributed resources (production technologies, like solar panels or wind turbines, sited close to customers or even on their property), which provide two-way flows of power—meaning some energy flows forward to the consumer and some flows back into the grid. We need to adjust the “visible hand” to accommodate new technologies, or else its vise grip will suffocate the market.
Meanwhile, other factors have interfered with the success of markets: legislative interventions and regulatory decisions that increasingly track the political preferences of governors and other key officials. Although the Federal government regulates wholesale electricity in 47 states, the Federal Energy Regulatory Commission (FERC) typically defers  to regional grid-operators and their stakeholders. (These stakeholders include representatives of states, public power companies, private companies, industries, environmental organizations, and more). The stakeholders make decisions in quasi-legislative processes that often favor the incumbent owners of power plants and transmission facilities, and disfavor new entrants. Grid operators also tend to prioritize the wishes of key stakeholders or state political interests, sometimes at the expense of market-design quality. These influences occasionally hamper reforms that would improve efficiency.  The more market design reflects political preferences and the influence of entrenched interests, the less well the “visible hand” can do its job.
The intrusion of politics into market design can change the public’s perception of electricity markets, to damaging effect. The influence of interventionist green advocacy  and rent-seeking incumbents has particularly distorted public understanding of how the market works. The Right often succumbs to alarmism about the retirement of “baseload”  power plants, while the Left underestimates the costs of mandating high levels of renewables. Predictably, the result is a maze of government interventions that work at cross-purposes, with customers left to foot the bill. As Brookings scholar David Victor has noted , “When the politics of serious energy policy become impossible to manage . . . a torrent of symbolic actions fills the space.”
What electricity markets need is a commitment by politicians to stay out of it, letting the markets operate on their own and restoring investor confidence. Yet governments often bail out legacy power plants and subsidize new resources. This will be a key question for the future of markets: Can government regulators make sound technocratic decisions, even as more voices call for politicized intervention? Texas is the lone state that seems to have avoided the problem; its politicians and regulators are deeply committed to market-driven outcomes. And the Texas model is outperforming the field .
The Grid in Flux
By the early 2020s, we will see a landscape of different energy markets across the United States, with far more variation than we see today. Monopoly states will continue to lag behind in adopting lower-cost technologies. Customers in these states will have to fight hard for unusual supply arrangements and compensation for customer-sited distributed resources, like rooftop solar.
Yet customers in monopoly states with intervention-prone legislators will have it even worse. Goaded by the green agenda, legislators will interfere more drastically in investment decisions, circumventing the regulatory process that weighs the public interest. Monopolies have proven to be willing enablers of green symbolism, provided that it increases capital expenditures. The cost: As political connections increasingly determine investments, those decisions get worse and customers get hammered. Meanwhile, the regulatory process  itself becomes less transparent and accountable, and people no longer trust it.
For example, green industrial policy—including subsidies for offshore wind and nuclear bailouts—has plagued electricity markets in the Northeast, driving up costs for customers with little or no emissions benefit . Beyond market distortions, such pervasive intervention will undermine grid governance. Most notably, FERC recently made a decision that compromised the quality of market design , apparently to accommodate state interventions in New England. In an effort to prevent state-subsidized resources from depressing market prices, the decision introduces a “substitution” auction wherein existing power plants may sell their obligations to new, subsidized clean energy facilities. In other words, more economic plants can be bought out by less economic ones, per the discretion of technocrats. This decision will further distort market incentives, harm customers, and keep market prices high. It will also not prove politically durable, which could lead voters to call either for further erosion of competitive markets, or the reverse—taking a leaf from Texas’s book.
Texas, which boasts the least-intrusive “visible hand” and most competition-friendly retail policies , will become the envy of customers, free-marketeers, and pragmatic greens alike. Ironically, though it lacks a climate policy, the state will demonstrate the green benefits of competitive markets, thanks to its policy of not interfering in the markets for political reasons. It will proceed with de-carbonization more effectively and affordably than any other state. In recent months, Texas has already retired a large number of fossil-fuel plants, and brought online efficient natural gas, wind, solar, and energy storage  resources.
However, despite these variations in design, what will really transform marketplaces across the country will be the emergence of new technologies. These will do more than just make electricity cheaper; they will change the entire economic calculus of the energy market. One such game-changer will be the proliferation of economical bulk storage: The cost of batteries is slated to fall 50 percent by the early 2020s . Indeed, Bain & Company estimate s that by 2025, battery storage “could be cost-competitive with peaking power plants” (plants that operate only during periods of highest demand). As renewables take over a greater part of the market, these batteries will have even more value—battery storage can solve a persistent problem with solar and wind, which is that you can’t count on these sources to produce consistent power supply.
Renewables, especially solar, will see a precipitous fall in installation costs. Customers will gradually learn to respond to grid conditions, such as by switching electricity consumption from low-supply to high-supply periods. This will further reduce the costs of integrating renewables into the market, as they are the most variable sources of power. We will also see improvements in the market thanks to advances in “smart grid” technologies, which enable the exchange of electricity and information between customers and suppliers Companies will be able to manage their customers’ demand in real time (with their consent), saving consumers money without hassle, in response to price signals at the wholesale level.
The deployment of smart meters and advanced monitoring technologies will allow grid operators to curtail service to individual customers who cause supply shortfalls, such as by consuming more than what they contracted for. While an adequate supply of electricity is now considered a “common good,” with this technology, individual customers will be able to pay more for reliable service if they value it highly. But first, the “visible hand” must permit retail suppliers to pay for different grades of wholesale grid services (with different degrees of reliability), rather than one-size-fits-all. Texas is the closest to this vision; markets with a more paternalistic “visible hand” will fail to unleash these benefits without reform.
Other, perhaps more startling technologies are also on the horizon. As blockchain reduces the costs of transactions , we will have the opportunity to make decentralized transactions common across the electricity ecosystem, creating greater flexibility as well as savings. But to do so, further reforms are needed. In the 2020s, regulation may catch up to the “internet of things ,” but will they catch up to the blockchain of things?
The market-based systems that allow permission-less innovation  will unleash rapid investment in emerging technologies. These will drive down emissions and costs while increasing customers’ options. Customers and environmental interests will form a powerful alliance and call for even more regulatory reform. Indeed, by the late 2020s, political forces will start to converge around a market-based agenda.
The Grid of the Future
Customer empowerment will be the theme of the grid of the future. Thanks to open markets, customers will be able to control the nature and quality of their own service. They will have far more options, and they will be much better equipped to evaluate them. If a customer wishes, she will be able to pay more for enhanced service reliability, green power, or various contractual terms—or all three.
Large customers will become particularly creative in exploring their options. They will make better investments in infrastructure—energy efficiency upgrades to buildings and on-site generation to meet some of their own needs—as well as customized financial arrangements with retail suppliers. As they generate some of their own power, these consumer-producers will interact with the transmission system in new ways. For example, commercial electric-vehicle fleet owners and data centers with on-site backup power may consume more energy from the centralized transmission system during times of excess supply, then provide services back to the central grid during shortages.
Meanwhile, automated customer-facing platforms will revolutionize the experience of small customers. Robo-energy advisers , not unlike those that are currently transforming the financial industry , will simplify sophisticated energy decisions for customers of all stripes. For example, advanced algorithms can instantly evaluate retail supply plans and energy service companies. With such apps at their fingertips, even once-passive customers will likely find it worth their while to shop around.
As mentioned earlier, blockchain  and other advanced digital technologies will reduce coordination costs and unlock the economic potential of energy exchanges outside the traditional market. We will see the rise of private and community networks that can sequester themselves on microgrids when power from the central system is unavailable. Such decentralization, however, will not replace the central grid anytime soon.
Thanks to economies of scale , the centralized system will still take care of most of our bulk power production for at least the next couple of decades . In the long term, however, there’s no way of telling when, where, or how centralized systems will triumph over decentralized systems (or vice versa). Breakthroughs in modular nuclear reactors could secure the dominance of the centralized system for the rest of the century, or massive advances in photovoltaics and storage could usher in the age of distributed enlightenment. Uncertainty about future technology choices is truly at an all-time high.
At the very least, we can say that distributed resources will play a more dynamic role in conjunction with the centralized system. Customers will increasingly be able to self-supply with on-site generation, or contract with local resources to hedge against service disruptions in the central system. This, in turn, will lower demand for reliable service from the centralized system. Instead of the expensive, excess supply we have now on the central system, we could see the rise of inexpensive plants with lower reserves that periodically oversupply and undersupply local networks. These distributed resources will absorb excess energy when the central system is oversupplied, and self-generate when it is undersupplied. Customers will be able to find a reliable supply from a mix of sources—and pay for the degree of reliability they value, of course. Some manufacturers will pay extra for reliable service to avoid equipment damage due to power outages, while families will save money by using automated systems that shift the times at which their dishwashers operate.
Beyond self-supply, new technologies and evolving consumer preferences will do more to promote green energy than government intervention ever could. Customers will be able to use home energy-management services (like the aforementioned dishwasher system), as well as unconventional mechanisms like crowdsourcing, which could provide additional revenue for popular projects like community solar farms. The latter will allow customers to finance projects that institutional investors deem uneconomic. Together, such mechanisms ensure that only those individuals and companies willing to pay a premium for green energy do so. Over time, competition will make green energy more affordable, and we will see a marked rise in decarbonization.
Ultimately, we should expect open markets to triumph, providing incentives for private investment and customer participation. As markets provide superior incentives for risk management, they will have a particular advantage in an era when technology choice is so uncertain. Energy monopolies may eventually become a thing of the past (though regulation will always be with us).
Preparing for the Future
The prognosis for the grid of the future is as uncertain as ever . Rather than pretending we “know” what’s best and relying on central planning, we should embrace policies that respond best to uncertainty. Indeed, the less we know, the more we should realize that the best system relies on the judgement of the many, not the few.
Just as merchant generators drove innovation and managed risk better than monopolies over the past two decades, a freer electricity system will empower all private actors to make better investment decisions for themselves and the environment. To deal with the uncertainty we now face, politicians and institutions should create a durable and adaptive framework  to drive innovation. Nothing is more adaptive than free-flowing private capital. If politicians and regulators commit to an open, competitive grid, it will bring about a revolution in the electricity industry, making it dynamic, innovative, and customer-centric . If we limit government and unleash market forces, the grid of the future, whatever it looks like, will bring with it an era  of prosperity and environmental health.
 For more examples of political influence, see this piece , which I co-wrote with Josiah Neeley of R Street. See also the writings of Scott Hempling  on the regulatory capture of public utility commissions. Most commissioners I know acknowledge that politics influences big decisions. For an analysis of the nuanced regional politics of stakeholders, see this paper  commissioned by R Street last year.
Image credit: Lumppini 
- “notoriously inaccurate”: https://www.foreignaffairs.com/articles/2016-10-18/back-energy-future
- “notes”: https://americanaffairsjournal.org/2017/05/no-free-market-electricity-can-ever/
- “sums up the situation nicely”: https://sites.hks.harvard.edu/fs/whogan/Hogan_Market_Design_012310.pdf
- “ errors in technology choice”: https://sites.hks.harvard.edu/fs/whogan/Hogan_Market_Design_012310.pdf
- “ highly designed markets”: http://www.pjm.com/~/media/library/reports-notices/special-reports/20160505-resource-investment-in-competitive-markets-paper.ashx
- “abandoned nuclear project”: https://www.postandcourier.com/business/top-scana-accountant-accused-executives-of-mismanaging-s-c-nuclear/article_743584d4-3295-11e8-8465-47a2cc905671.html
- “low-cost additions”: https://kleinmanenergy.upenn.edu/blog/2017/06/01/once-lifetime-investment-opportunity-trigger-more-baseload-retirements-pjm
- “operating costs”: http://faculty.haas.berkeley.edu/wolfram/papers/FRW_AER_0607.pdf
- “complying ”: http://bushnell.ucdavis.edu/uploads/7/6/9/5/76951361/economics-literature.pdf
- “responsive”: http://2o9ub0417chl2lg6m43em6psi2i.wpengine.netdna-cdn.com/wp-content/uploads/2016/07/RSTREETSHORT30.pdf
- “typically defers”: http://2o9ub0417chl2lg6m43em6psi2i.wpengine.netdna-cdn.com/wp-content/uploads/2017/10/112.pdf
- “”: https://www.the-american-interest.com/2018/06/18/the-grid-of-the-future/#_ftn1
- “green advocacy”: https://www.economist.com/news/leaders/21679193-global-warming-cannot-be-dealt-using-todays-tools-and-mindsets-so-create-some-new
- “retirement of “baseload””: https://www.rstreet.org/wp-content/uploads/2017/05/97.pdf
- “noted”: https://www.brookings.edu/research/energy-and-climate-moving-beyond-symbolism/
- “ outperforming the field”: https://www.resausa.org/sites/default/files/RESA_Restructuring_Recharged_White%20Paper_0.pdf
- “regulatory process”: http://wtvr.com/2018/03/07/senator-warns-that-bill-will-tilt-scales-in-favor-of-dominion-energy/
- “no emissions benefit”: https://www.utilitydive.com/news/new-yorks-nuclear-subsidies-contradict-economic-principles/436978/
- “compromised the quality of market design”: https://www.utilitydive.com/news/split-ferc-approves-iso-ne-2-part-capacity-market-plan/518904/
- “competition-friendly retail policies”: https://object.cato.org/sites/cato.org/files/serials/files/regulation/2017/9/regulation-v40n3-4.pdf
- “efficient natural gas, wind, solar, and energy storage”: http://www.rstreet.org/2018/01/05/testing-texas-power/
- “ fall 50 percent by the early 2020s”: https://auto.economictimes.indiatimes.com/news/industry/moodys-warns-of-low-return-on-ev-manufacturing-us-ev-makers-may-lose-7000-per-unit/62633930
- “ estimate”: https://www.forbes.com/sites/baininsights/2018/02/01/the-energy-revolution-of-2018-electricity-storage/#8ab48096208e
- “blockchain reduces the costs of transactions”: https://docs.wixstatic.com/ugd/e0ea17_ac7d85a46554443c8348f8a234920625.pdf
- “internet of things”: http://www.independent.org/pdf/tir/tir_20_03_13_kiesling.pdf
- “permission-less innovation”: http://permissionlessinnovation.org/
- “Robo-energy advisers”: https://www.zentility.com/
- “transforming the financial industry”: https://www.forbes.com/sites/stephaniemacconnell/2017/09/12/robo-advisors-and-the-great-money-movement/#591810e011d7
- “blockchain”: https://www.greentechmedia.com/articles/tag/blockchain#gs.OiMrczs
- “economies of scale”: http://files.brattle.com/files/7626_comparative_generation_costs_of_utility-scale_and_residential-scale_pv_in_xcel_energy_colorado
- “couple of decades”: https://www.nap.edu/catalog/24836/enhancing-the-resilience-of-the-nations-electricity-system
- “ uncertain as ever”: https://www.eenews.net/stories/1060075897
- “ durable and adaptive framework”: https://www.sciencedirect.com/science/article/pii/S2214629614000243
- “dynamic, innovative, and customer-centric”: https://object.cato.org/sites/cato.org/files/serials/files/regulation/2017/9/regulation-v40n3-4.pdf
- “era”: http://www.leadingwithconservation.org/wp-content/uploads/2015/07/Kiesling_Paper.pdf
- “”: https://www.the-american-interest.com/2018/06/18/the-grid-of-the-future/#_ftnref1
- “this piece”: https://www.rstreet.org/2016/08/04/unnatural-monopolies-do-power-companies-have-to-be-crony-capitalists/
- “writings of Scott Hempling”: http://www.scotthemplinglaw.com/essays/regulatory-captureI
- “paper”: http://2o9ub0417chl2lg6m43em6psi2i.wpengine.netdna-cdn.com/wp-content/uploads/2017/10/112.pdf
- “Lumppini”: https://www.shutterstock.com/g/gnidets%20iurii