Nope you're screwed now! Lol.
FWIW I am pulling mine off again this week. It comes off when I am home and goes on when I leave since the wife hates driving with it off.
My wife hates driving without the sway bar too. But she will once in awhile.
But I removed mine a couple months ago and drive mainly on-road, but I love the "rock the boat" feel the truck has now. And I cannot report of any extra wear and tear, but I know the off-road handling is great, the steering wheel doesn't fight you and the ride is smoother.
Because this "mod" is SUPER easy, you really could just take it off for a day or two and see how you like the feel of the truck without it. If you decide you don't like it, it just takes 10 minutes to install it. Two nuts on the passenger frame rail and three nuts on the driver frame rail and then just undo the bolt on each lower control arm. Tip on removing the nut/bolts on the LCR, they are safety nuts (That's what she said), so they will just spin, you need to put a wrench to the nut and bolt to remove the connecting rods.