As someone who has always had 20"+ on my trucks, I think Raptors look silly with wheels that big. It's not what it's meant for. Just looks wrong without a 17 or 18
For most people you aren't going to notice much of a different in handling and off road performance. Every day driving you are going to have a more cushiony ride with a 17/18 vs a 20 because you have more tire to eat up the bumps.
Another thing to factor in is tire prices. Some wheel sizes the price skyrockets for tires.