Mine does this horribly as well. Still trying to find out why.
All previous Fords I have had have exhibited this behavior after a period of time. It seemed that the worse this problem got, so did the doors not wanting to latch when it gets extremely cold.
I would assume seals, or as @EricM mentioned, maybe not keeping the vents in recirc?
Didn’t think about that w my beater ass 2000 Taurus but it makes sense. Doors were close to not wanting to latch on the ‘09 Edge today in -3* but reluctantly closed with a strong push.