I’d love to do the loop. Any sketchy parts you’d avoid with a 4 year old and wife that’s only semi-adventurous?
I would say the actual Alpine Loop itself is not sketchy at all. The hardest trails out there are not actually part of of the loop (Black Bear and the notorious wall at Poughkeepsie). The two main trails of the loop are Engineer Pass and Cinnamon Pass, both of which are not sketch at all but they will have some shelf roads. The shelf roads look more scarier than they actually are. Don't be scared of width, I never felt like the raptor was ever too wide while on the loop. Just take it slow and in 4LO. If you're starting in Ouray, it might recommend Mineral Creek, I would skip it and go south just a bit to Corkscrew Gulch/Hurricane Pass to connect you to Engineer/Cinnamon.
Doing the whole loop in a day is definitely do-able but not enjoyable in my opinion (esp with a family). I would definitely split it up into a few days so you can pop out occasionally to take some pics, enjoy the views, and scope out the old mining remnants. Splitting it up also allows you to end your day a little early so you can make it into town to grab a bite to eat as most places in the mountain towns do not stay open that late.
If you come off the loop pretty confident (in terms of driving, handling the heights, etc), I highly recommend doing Imogene. I always run Imogene from Ouray to Telluride. I start in the morning and I'm in Telluride by lunch time. It's just a hair more technical than what you'll be driving on the loop but that's what makes it fun imo.
Feel free to PM me if you ever end up planning a trip and have any questions!