Well, I suspect it has all to do with peer pressure and the social contract, as you briefly touched on in your post. Guilt is how the social contract is enforced, and it is only necessary to enforce it when people are trying to do something they want but society doesn't want them to. Of course this changes over time, and different groups apply different pressures. For instance I think a random westerner would feel less guilty about smoking a cigarette 50 years ago than what they would today. Similarly some cultures used to associate obesity with beauty, while today we know it is generally unhealthy to be obese so society tries to make it clear that overeating is something you should feel ashamed of. Similarly with different sexual orientations, how much responsibility to take for your kith and kin, whether anything you do to your body should be your decision etc.
I feel that even when society says no to something as innocent as liking cartoons as a grown-up, it is a remnant of a time when grown-ups weren't supposed to have time to do much more than care for their family and jobs and hanging on to parts of childhood went contrary to this. And even then you have famous people in the past who clearly saw the value in hanging on to more lighthearted things. CS Lewis has a famous quote for instance:
But was his ability to grow up in this manner a privilege of his position in society? Would e.g. a contemporary coal miner have had the opportunity to hang on to their childhood, or would work and family have taken all their time? Tbh. I don't know enough about early 20th century England to answer that, but it is interesting to think about.
At any rate I agree that some balance between giving in to your urges and restraint needs to be found unless we want to live in a completely hedonistic society. And that doesn't even touch on how one's actions might affect others. Shouldn't we frown on that person wanting to drive their Porsche at 75 mph past the kindergarten? Expect them to feel bad about that? Of course this examples are codified in law, but law arises from what people in general agree one should feel guilty about.
Anyway to get back to the main drive of this thread, I think your example about watching cartoons as a grown-up is one of my guilty pleasures. Not something I feel all that bad about, but not something I'd bring up with most people I know either.