Freedom means both freedom to associate and not to associate. From Eric Peters at ericpetersautos.com:
How do you know whether you are free – or not? Ask yourself whether you are free to associate – or not.
We often take this freedom for granted, even when it has been massively eroded. We are still free to associate – or not – on the personal level. We are not yet obliged to be friends with people we don’t especially like, or date people we’re not interested in that way – though both of these forms of free association are under attack in the form of heavy social pressure to be “open” to being friends with and dating people we’d rather not . . . associate with.
But outside of our private lives, freedom of association – which must include the right to not associate and for whatever reason – is essentially nonexistent. More precisely, it has been turned into an actionable offense to assert it.
If you doubt it, ask the Boy Scouts.
They did not wish to associate with girls for the evident reason implied by the name of their association. Boys not being girls – and vice-versa. Essentially for the same reason that girls generally do not want to associate with boys in certain contexts. It is why there is also an organization for them, the Girl Scouts. Each having a different focus and appeal. But that is not the relevant point because if one is free to associate then one need not explain why they wish to associate – or not.
You just do – or do not – and others are obliged to respect it, whether they like it or not.
The principle is, of course, both accepted and enforced in some contexts. Such as all-black fraternities, for instance. They are free to not associate with whites or Asians. But neither of the latter are free to associate – or not – on a similar basis.