I had VenomRex 20's on my Raptor from the beginning, and had no issues at all off-road with Nitto Ridge Grapplers. I even took my truck to Moab and did alot of off-roading without issue; BUT, I never aired down at all. Too risky with 35" slim sidewalls on 20's.
HOWEVER, I just recently changed to 17's, and am much more relaxed when going off-roading these days. Not so paranoid about my wheels being so close to the terrain and getting them scuffed up. Like Smurfslayer said, if you run it off-road and air down, beware. If you just run your truck as a mall crawler on-road, aren't worried about scratching those expensive wheels up with rocks or mud, and don't mind people ribbing you about running 20's off-road, have at it; they look great! I myself, am glad I went to 17's. I'm heading to Colorado for some trails, Rocky Mtn Natl Park, Garden of the Gods, and Pike's Peak with the family soon. The 35's on my 17's are really plush on and off the road compared to the 20's. The 20's performed better on-road in turns, but the Raptor isn't a sports car.