I went from a one tone Chevy crew cab diesel (SRW) - that had lots of good mods. Like you I no longer had a heavy trailer to tow, just a boat and/or smaller open trailer for quads, SBS etc. I was afraid I would miss the diesel power and the real truck goodness of the HD.
I found a buyer for my truck, so I decided to make the change. I bought a 2014 Crew Cab Raptor and have never looked back (owned it 18 months and 43k miles later) - love the Raptor ride and how it drives and feels. Super nice truck. Given what I tow the Raptor with 6.2L is all I need.
Only downside is I get 12 mpg (I drive fast) in the Raptor and about 9 mpg towing. I did better than that by about 3 mpg in the diesel - on both fronts.
I am very happy I made the change. My mods are just SVC rear Bump Kit, deaver + 2 rear springs, some lights, blacked out the bumper and wheel arch's, Method wheels etc...
Unless I get a heavy trailer I will be in a Raptor going forward! When the new one comes out I will probably make the change after it has been out for a year or so and give mine to my son.
Good luck with your decision!