What I find really incomprehensible is that in Europe there are quite nice family cars with clean, small diesel engines that are getting around 55 mpg. They are not available here.
A lot of it is just a series of chicken and egg issues--will consumers switch if diesel is not readily available at their local filling station? Will manufacturers absorb the extra cost to switch to diesel engines if consumers won't be willing to pay those extra costs? And so on and so forth.
Another big issue are emissions regulations. Manufacturers aren't going to create different versions for different state's emission regulations, and California's stringent regs make diesel passenger cars mostly impractical to sell in the US. The extra tech required to make diesels clean enough for California will make the cars seem too pricey. Unfortunately, diesel is not sexy or seen as environmentally responsible so although consumers will pay a premium for hybrid tech, most won't for a diesel.
Many will argue that it is the high cost of fuel in Europe due to taxes that lead to the very high proportion of diesel passenger cars, although Japan also has very expensive fuel and they do not drive diesel passenger cars, so every country/region has its own unique reasons for what kinds of fuels are used. Look at us just a few short years ago when gasoline prices were high and seemed like that they would stay that way forever--instead of diesel, seemed like ethanol stormed onto the scene and was going to be our savior (with lot's of political lobbying involved behind-the-scenes, I'm sure).
Who knows, 20 years from now, maybe we'll all be filling up with algae-based fuels? The US military may be using it widely by then based on preliminary results right now, like already approving algae-based aviation fuel for military aircraft. Commercial aviation is clamoring for algae-based fuels based on the military's early success with it.
Another old technology that I always wondered about was the good ol' two-stroke engine. If the emissions problem could be solved economically, that could also lead to small, powerful, fuel efficient engines in our cars. But that's been looked at extensively and I guess it just doesn't work for the bigger cars that Americans want.