You haven't seen any of the new diesels then. (Or maybe you have and you just didn't notice they were diesel.) Mercedes, VW, Jeep, you can't even tell they're diesels. No more noise than a regular gas engine and very little diesel smell. Even the new diesel engines in the large pickups are leaps and bounds better than the last generation as far as refinement.

As far as the fuel costing more, a diesel gets 30% or better mileage than a similar gas engine. With gas at $3.00, diesel would have to be $3.90+ just to break even. Generally, the only time the difference is almost that large is in winter [when home heating oil purchases drive up the cost of diesel fuel]. Most of the year diesel sits at slightly above regular. Either way, cost per mile is often much lower with diesel, even with fuel itself generally costing more. Over time it ends up saving you money.

If you really want to save money you could run biodiesel (legal) or untaxed dyed diesel (illegal), both cost well under what diesel costs and even under what regular gas costs.

The other thing I like about diesel is the safety factor. It isn't as volatile as gas, so you can store it with little explosion risk. There is also little risk of fire or explosion in a crash compared to a gas vehicle. For our construction business we buy diesel when the price is low and store it for the rest of the year in 10,000 gallon tanks. It ends up saving us even more money than when we would buy at the pump, and we have a large stock of fuel on hand in case of an emergency (like a fuel shortage).