Capitalism prior to the Great Depression was driven primarily by production to make the working classes more productive*. There was tremendous pressure not only to use machines to make workers more immediately productive, but also to improve their standard of living to make them better workers overall and to support more workers. (Note that Marx was not incorrect in noting the poor quality of life of industrial workers in the mid-19th century; there was no "altruism" whatsoever in the capitalists drive to improve workers' standard of living. Utopian fuckwits notwithstanding, it still remains arguable, however, that the quality of life of pre-industrial agrarianism was even lower than under industrial capitalism. Secondarily, capitalist production created the professional-managerial middle class, which absorbed much of the luxury production afforded by improved productive efficiency.
*Per-capita consumption as a percentage of total production by the ruling class has fallen substantially from feudalism, and continues to fall. Contrast Versailles with the Hearst Castle and Bill Gates' home.)
The problem, becoming apparent even by Marx's time and underlying the recessions of the 19th century, was that the personal demand of the upper and middle classes was beginning to fail: they no longer wanted stuff. Contributing to this failure was the social attitude that created the capitalist ruling class in the first place (and which permeated to the middle class): a distaste for consumption in favor of saving and investment. The feudal ruling class consumed not only ostentatiously but proudly: they were, in their own eyes (and to some degree justifiably) turning what little surplus labor there was — surplus labor that couldn't be invested — into works of great art and beauty, which they (again to some degree justifiably) conceived themselves to be holding in trust for future generations. One reason why the capitalist ruling class arose as a distinct class — rather than the feudal ruling class simply embracing industrial production — was precisely because the capitalist attitude favoring investment over consumption afforded a much higher rate of industrial development.
This crisis of demand came to a head in the 1929 stock market crash and subsequent depression. The professional-managerial middle class (relatively large compared to early capitalism), whose individual desire to consume was saturated, tried to gain entry in the capitalist ruling class by exploiting loopholes in the stock market.
It is entirely incorrect to view the loopholes and weaknesses in the stock market in 1929 or the shadow banking system today as the problem that needs to be corrected. It's certainly a problem, but it's impossible to build a system with no loopholes or weaknesses at all, especially when there's any negative selection pressure that can be escaped by finding or creating loopholes. (Most simple systems work only because no one is actively trying to subvert them; the most well-designed complex systems fail even though no one is trying to subvert them.) At best, advocates of "regulation" are fighting the last war.
FDR's New Deal and Keynesian economics was the last, best hope to "rescue" capitalism from its own contradictions. It succeeded — at least for a while — because not only did FDR et al. improve regulations on the stock market and financial system, but also because it pushed a considerable amount of social demand — the ability to buy stuff — on the working class. Of course, the United States supplied this demand in no small part by creating the most powerful international economic empire in the history of the world, hyper-exploiting tens or hundreds of millions of foreign workers as well as tens of millions of American minority and illegal immigrant workers. Even so, without the demand of a huge chunk of the American working class, American imperialism would have faltered.
However, the New Deal and Keynesian economics did not address the primary selection pressure operating on members of the capitalist class: losing their money and being ejected from the capitalist class. Indeed the entry of hundreds of thousands of new members to the lower rungs of the capitalist class increased this selection pressure. It doesn't matter, for example, that there is a vastly expanded demand for shoes if you yourself cannot efficiently produce shoes: you'll still go out of business. The selection pressure remains, and to evolution it doesn't matter how you escape selection pressure; all that matters is that you do escape it.
"Efficiency" in capitalist economics has two* definitions: producing goods with less actual labor time ("good" efficiency), and producing goods with less actual labor cost ("bad" efficiency). One company that can produce 100,000 shoes a year with 100 workers paid $10/hour, is "more efficient" than another company that can produce 100,000 shoes with 75 workers paid $20/hour. There are always loopholes and weaknesses in regulations that mandate that workers be paid $20/hour; the companies that find and exploit those loopholes will escape selection pressures that will eliminate companies that don't find them or have some "moral" compunction against exploiting them. [see comments]
As long as there is labor cost selection pressure, the labor market will find ways to "fall downhill" to a market for labor power rather than a market for labor time, i.e. the price for labor will fall to its cost according to the law of supply and demand. All you can do with regulation is select against some particular methods of making the market for labor a market for labor power, and whatever you don't select against will end up being selected "for".
*Three, actually: the third is producing more (or better) goods for the same time/cost.
From the late 1940's to the 1970's, we had tremendous economic growth in the United States (and Western Europe). The vastly increased demand (ability to buy) afforded mostly by labor unions and government labor regulations eased selection pressure overall: with demand so high, even relatively inefficient producers were not selected against. The chief problem of the era (and we were lucky to have that problem) was convincing people to want to consume what was being produced. There also was a certain symbiosis between foreign and American workers. The hyper-exploited foreign were producing mostly raw materials and some technologically unchallenging manufactured products, such as textiles; American workers were producing more complex and technologically challenging products, such as automobiles and airplanes. Technological issues, especially shipping, worker education, and management, prevented "outsourcing" and made American workers more cost-efficient overall, despite a much higher unit cost.
But selection pressure was not absent: companies were going out of business and a lot of capitalists in the 50s and 60s failed and were expropriated. Even if only a small fraction of companies just looked for ways to reduce labor costs rather than labor time, that small fraction still escaped selection pressure more often than companies that didn't look for (or didn't find) ways to reduce labor costs. As evolution shows us, even a small reduction in selection pressure against a minority always* leads to that minority eventually dominating the population and often completely eliminating alternative variations.
*Almost always: there are some exceptions due to higher-level selection pressures.
Because of the way selection pressures work within the capitalist ruling class, no amount of regulation or "artificial" constraints on the capitalist class will ever be effective in the long (i.e. 30-40 years) term. We would literally have to regulate with the power, knowledge and benevolence of an deity* to successfully overcome the selection pressures inherent in the capitalist system.
*It should be noted that no such omnipotent, omniscient and omnibenevolent deity actually exists.
In the New Deal years, while the upper middle professional-manager class was struggling — often successfully — to enter the capitalist ruling class, the working class and lower-middle class were storing some of their new ability-to-buy demand into housing and pensions. Naturally, when labor costs started to decline — i.e. actually working was creating less ability-to-buy demand — ability-to-buy demand devolved to first pension funds and then home prices, producing the bubbles of the 21st century. When these bubbles were exhausted (in the space of just a few years each), ability-to-buy demand completely collapsed.
To a certain extent, this stored demand was simply stolen through chicanery and outright fraud. It makes a certain amount of sense to look for ways this chicanery and fraud can be prevented and eliminated, and it seems that a lot of "progressive" economists have focused their entire attention on this effort. But the fundamental problem is that there was an opportunity for these areas of stored demand to be stolen in the first place. Two conditions are necessary for anything to be stolen: the stolen property must of course exist, but more importantly it must be placed at risk by its owners. No matter how much gold there is in Fort Knox, it's not going to be stolen from there. It's very difficult to take someone's house when it's paid for (or they have enough ability-to-buy demand to maintain their mortgage payments): if you want to steal someone's house, you have to convince them to put it at risk by taking out a mortgage they can't pay for. Because ability-to-pay demand on the basis of wages was declining for the working and lower-middle classes, that's exactly what they did: take out second mortgages and home-equity loans on their more-or-less paid-for (or being paid-for) houses. And they more-or-less "rationally" had to do so: because their mortgage payments were tied to their incomes (and their pensions tied to the health of the companies), had they not propped up overall ability-to-pay demand by tapping their houses, they risked losing their jobs (predicated on supplying that demand) and their houses anyway. Because of the fundamental nature of the capitalist system, the working class and lower-middle class was in a "damned (tomorrow) if you do, damned (today) if you don't" double bind.
In a purely engineering sense, any competent, honest and caring economist could design a system that would maintain aggregate demand, keep full employment, and guarantee a dignified standard of living to everyone on the planet. Throw in the scientific, engineering and technical knowledge we already have, and we could probably support ten times as many people as we have, all with dignity and a reasonable level of comfort. Even I myself could probably make a decent attempt, with my amateur and half-assed understanding of economics and physical science.
Figuring out how to design an economic system is not the problem. If we had a reasonable level of good will and common desire, I can't see that it's really even as technically or intellectually challenging as designing a computer operating system. The problem is that we don't have good will. Even if 90% of the capitalist ruling class had even a little Keynesian good will and common cause with the working and middle-classes, they would be eliminated by the other 10% of Randians long before they could actually implement any economic system that was good for everyone. They partially succeeded in the 1940s only because a "perfect storm" of circumstances diminished the power of the proto-Randians (i.e. there was a "group selection" event) leaving them in a position of temporary advantage. But the proto-Randians weren't completely eliminated; more importantly, the fundamental selection pressures didn't change: the selection pressures that afforded a Randian attitude (improved cost-efficiency, not time-efficiency) a relative advantage didn't change.
To be honest, drilling down to the evolutionary pressures on politics and economics isn't all that comforting. One imagines, per Marx, that somehow the working class can directly exert selection pressure: they certainly appear to have the raw power to do so. But they do not appear to have the political and psychological will to do so, and a century of communist and socialist agitation and propaganda failed to make this will pervasive in the working class: FDR and the Keynesians threw the working class a bone in the 1940's, and they couldn't run away from communism fast enough. The massive anti-communist propaganda in the 50's and 60's had a lot to do with the outcome, but even the best propaganda cannot succeed where there isn't fertile ground in people's minds for its ideas.
With all due respect to many of my friends, whom I like and admire greatly (and even many people I consider to be well-intentioned jackasses), the radical anti-capitalist (communist, socialist and anarchist) movement is entirely ineffectual and meaningless. Even the capitalist liberal and "progressive" left appears almost completely unable or unwilling to exploit the catastrophic — and frankly egregiously stupid — blunders of the Randian capitalist right.
We're running out of options, and running out of time. All I can see is a literal mass extinction of most human societies, perhaps of all humanity. Even if some are left standing to rebuild the world, the "victors" will mostly be selected by pure chance, not by any intrinsic moral virtue.