Do you have a catch can? The oily goo if from your PCV system.
Short story is, under boost conditions the only low pressure area that crankcase pressure/fumes/vapors can be vented to and still be pulled into the engine to be burned is in front of the turbo inlet.
This causes oil (in the form of vapor) to be pulled in through the turbo into the intercooler system and condense back into a liquid, usually a pretty nasty liquid.
I have been running a catch can on my 7.3 for 15 years or so, on my wife's SHO for 8 years and never once since install have I seen any oil residue in the intercooler piping. My 7.3 does not suffer from oily swollen intercooler boots like most.
Because of my positive results, I had a catch can waiting in my garage before my Raptor arrived. Was first thing I did.