There is lots of supposition on why diesel is higher, and I have read lots of posts and reports on it. It does cost more to produce it today because of the added requirements, but I believe the single factor that drives the price the most is that the government wants to regulate how much diesel is used due to the more harmful combustion byproducts of it. They also regulate the amount of diesels on the road- look at how hard the truck manufacturers (light trucks) have to work to meet standards, and what has been compromised in dependability because of it. Now tractors. Look at how few manufacturers are putting diesel cars and trucks on the road, compared to how many make them for use in other countries? Almost every one builds a diesel, but can't get it to pass emissions regulations here in the US. Because of emissions, they regulate the number of consumer diesels on the road through stringent requirements and higher fuel costs. Between the reduced economy on pickups (due to hard to meet regulations), the added cost of that diesel engine in a new truck, and the higher cost of diesel, it no longer pays to buy a diesel pickup unless you tow real heavy or drive a lot of miles a year. A friend that works for Ford told me that they did a study, and concluded that you needed to drive 32,000 miles a year to cost justify a powerstroke- and that was more than 10 years ago.
My daughter drives an 03 Jetta TDI- great power, extremely durable engine, and on her worse day she gets 48 mpg. I'd take that any day over a hybrid, and the added expense of diesel fuel is offset by the staggering fuel economy.
You'd think that the government would be more in favor of diesel, if it is in a clean burning engine/exhaust system. You don't have the adverse effects with biodiesel like you do ethanol-enriched gas, and you can completely manufacture diesel fuel out of most any organic material. Why not invest in a fuel that you can manufacture, versus one that you have to drill for and deplete supplies?
I think the price for diesel is mostly political in nature. I could be wrong- it happens more than I want it to. But, that's what I believe.