The rational purpose of insurance is to allow people to switch a bet from the side they cannot afford to lose (or if society cannot afford them losing) to the side they can afford to lose, even if the expected value of switching sides is technically lower.
Here's a bet: you can take the A or B side of it. A pays B \$1. B flips a fair coin 20 times; if it comes up all heads, B pays A \$1,000,000. If you go by pure expected value, you want to take B's side of the bet, since the probability that B has to pay \$1,000,000 is 1/220, or 1/1,048,576. In the long run, B wins \$1 1,048,575 times, and loses \$1,000,000 once; thus his expected value for each bet is \$48,575/1,048,576 or about \$0.046.
Here's the rub: if you take B's side and lose, and you don't have \$1,000,000 in your pocket right now, you die.
This is why insurance companies exist: An insurance company does have \$1,000,000 in its pocket, so it can take the B side of the bet. You do not have \$1,000,000 in your pocket, and you (presumably) don't want to take a 1/1,048,576 chance of dying, certainly not for less than a nickel, so you take the A side of the bet and pay \$1 to the insurance company.
Even worse, it may be that if B doesn't have \$1,000,000, someone else will die. The state, then, requires that you take the A side of the bet, or convince the state that you really do have \$1,000,000.
Hence, the state requires, and I happily comply, that I buy auto insurance. I'm a very good driver, but even the best driver makes mistakes, and it's a matter of chance whether those mistakes might lead to a large liability. I also have a high deductible, because I can afford to lose \$1,000; it would hurt, but it wouldn't be a catastrophe, and I'm a good enough driver that the probability that I'll cause an accident, for which I would pay at most \$1,000, is low enough that my expected value of betting that I will not have an accident is higher than betting that I will.
The general rule for buying insurance is: if you can afford to "lose the bet," don't buy insurance. If you cannot afford to lose the bet, buy the insurance.
Thus I never pay for extended warranties, which are just insurance bets. I can replace my phone, my computer, my printer, etc., and the money I save by not buying insurance more than pays for the replacement costs.
On the other hand, I cannot afford the medical bills of someone I might seriously injure in a car accident, so I have auto insurance. When I owned a house, I could not afford for it to burn down and leave me homeless, bankrupt, and destitute, so I paid for fire insurance. I also paid extra for earthquake and flood insurance.
It makes a certain degree of sense, even to this communist*, for insurance to be private. Private, profit-maximizing businesses have an incentive to set prices based on an accurate calculation of risk. If one company sets premiums too high, other insurance companies, who calculate risk more accurately, will take your customers with a lower price. If one company sets premiums too low, it will go bankrupt. There's an element of moral hazard, too; an insurance company has a strong incentive to renege on contracts, and it has an incentive to charge too-low premiums to insure against events that are rare not only for the individual, but in the aggregate (e.g. earthquakes). Hence insurance companies are (usually) heavily regulated by the state.
*I don't believe it's necessary for insurance to be private.
To sum up: Private insurance is the appropriate model when:
- The risk to any individual is low
- The cost of "losing" is too high for a typical individual to pay
- The risk can be accurately calculated
- There is little, if any, ability to differentiate between individuals' risk
It used to be the case that medical care was very much an insurance-like proposition. Most injuries, illness, or disease were either fatal (and hence not worth insuring against), or immediately and expensively treatable. More importantly, it was impossible to predict whether any individual would get injured or sick. Medical insurance really was, like all other forms of insurance, a way of socializing risk.
However, modern medicine transformed many illnesses, especially geriatric illnesses, from fatal to chronically treatable at substantial expense. Furthermore, it made these expenses predictable. Now, almost everyone will eventually have an expensive chronic illness, if for no other reason than old age. Because old age is known to be expensive and virtually certain, we have Medicare, which socializes not risk (since there's no risk in a near-certainty) but cost.
The situation is a little different for younger people.
With modern medicine, everyone, including each individual him- or herself, knows which individuals are at low risk, and which are at high risk, of developing a chronic, expensive illness. Even if there were perfect competition in the insurance industry, there would be price discrimination between low- and high-risk individuals. Low risk individuals do not want to pay more to subsidize high-risk individuals, and high-risk individuals, because everyone knows they're going to lose the expensive bet, have to pay nearly the full (and very expensive) cost of their treatment to "switch sides" of the bet.
Furthermore, the market for medical care developed very weirdly; most routine medical care was fee for service, and rare, expensive medical care was covered by private insurance. There was no such thing as an expensive chronic condition. (Even diabetes was relatively inexpensive to treat chronically, and the complications, when they inevitably arose even with treatment, untreatably fatal.) Today, however, expensive chronic medical conditions, especially cancer (if you have cancer once, you have a much higher risk of developing cancer again) don't fall into either category. Furthermore, because, as a society, we mandate that hospitals have to provide emergency treatment to everyone without regard to ability to pay, and we don't give them enough tax money to cover expenses, they have to charge someone more. The insurance companies have enough power to pay, so they shift the costs to people without insurance who need even routine medical care. Demand for medical care is relatively price inelastic: an increase in the cost of medical care does not cause a proportionate decline in the demand for care. This means that private, competitive producers of medical care have a market incentive to increase costs high enough to price many people out of the market, especially in a society with a lot of wealth and income inequality.
The problem we need to solve in medical care is no longer how to social risk, but whether and how to socialize costs. Like any other attempt to socialize costs, some people will be net payers, and some will be net beneficiaries. This is true even of costs and risks socialized through the private sector: Because I am a good driver, I have paid far more in liability insurance than liabilities I have caused others: in 30 years of driving, for which I've paid conservatively about \$20,000 in liability insurance payments, I've had one accident where I was at fault, with about \$2,000 of liability, and I've received about \$2,100 for the one accident where the other driver was at fault.
There are basically three choices to socializing anything. First, don't socialize costs. If a poor person has an expensive chronic condition, we don't treat it, and that person dies or lives in pain. Second, we socialize costs through taxation, as in Medicare. Third, we socialize costs through the private sector, kind of as we socialize risk with auto insurance.
The PPACA (a.k.a. Obamacare), like its template in Massachusetts (a.k.a. Romneycare), is an attempt to socialize medical care costs through the private sector, by compelling people to buy insurance and compelling insurance companies to insure everyone equally without regard to individual risk.
The alternative is to socialize costs through taxation. Everyone pays some sort of tax, and the government pays for all medical care, either directly by also socializing the production of medical care, or indirectly by paying private producers.
I'll talk about the relative merits of these strategies, and why the PPACA is deeply problematic, in another post.