i don't think stephen colbert is very funny, myself. his schtick back in the day was impressive, but i don't think he's really transitioned well out of the persona and into a real person. i used to watch a lot of conan o'brien when i was a kid, and largely walked away from the entire concept of late night when he got screwed over. i don't keep up with any of them.
my read on the situation is that 11:30 is no longer a good time slot. young people today want to go to bed earlier, so they can wake up earlier and check their phones. i think that late night tv is misunderstanding this change in scheduling behaviour. people used to get up, fall out of bed, drag a comb across their head, find their way downstairs and drink a cup, look up, notice they were late, find their coat, grab their hat and make the bus in seconds flat. they don't that anymore. nowadays, they get up, roll over and stare at their phone for an hour before they fall out of bed. so, they have to go to bed early enough to get up to check their feeds.
that means that what colbert really lost out on was scheduling. it's too late at night, nowadays.
that doesn't mean that you just time-shift it like you're solving a relativity problem. the tv day is shorter, now, because people have phones; they used to spend 15 hours on tv, and now will spend 8 or 10. if you want to survive on actual tv (and not just youtube), you need a better time slot.