Under a libertarian analysis, that would depend on whether the segregation is imposed by government or by the relevant property owner. If the government mandates segregation, it has initiated force. Violating a private property owner's rules would be initiation of force.
Note that in the US, we've never actually tried the system where private discrimination is neither required nor forbidden. We went straight from the mandatory discrimination of Jim Crow to the forbidden discrimination of the Civil Rights Act. So no one really knows how actual freedom of association would work out in practice. It is unfair therefore to try to taint libertarianism as equivalent to Jim Crow, but likewise one should be skeptical of libertarian claims that things would work out all right under such setup.
Note that in the US, we've never actually tried the system where private discrimination is neither required nor forbidden. We went straight from the mandatory discrimination of Jim Crow to the forbidden discrimination of the Civil Rights Act. So no one really knows how actual freedom of association would work out in practice. It is unfair therefore to try to taint libertarianism as equivalent to Jim Crow, but likewise one should be skeptical of libertarian claims that things would work out all right under such setup.