but, that isn't really what he's getting at - what he's really getting at is more in the question of sanction. that is, he skips a step and takes a big leap of logic to get there and then comes up with something ridiculous with a clear political motive. the more fundamental point - and the one that is actually cogent - that he's really concerned about is the freedom to be absurd without consequence. he then compresses that together into the unified idea of the freedom to be absurd - which is either irrigorous or, more likely, consciously disingenuous.
the answer to this is in mill, and really any writings by any liberal thinker: to separate what he's slyly compressed. the freedom to be absurd is one thing, and the freedom to be absurd without consequence is another. now, if one is absurd and harms nobody then there is no basis for any kind of action by anybody - individual, state, collective or not. but, if one is absurd and that affects others negatively, it does indeed stand to reason that they should be held liable for it. to conclude that one is not truly free to be absurd if they are to be held liable for the consequences of it is itself absurd.
so, berlin's true argument is really simply that true freedom means that the absurd should not be held liable for their behaviour - even when it affects others. you'll have to excuse me for not taking that particularly seriously.