The thought of a trans person unable to use a service such as a public toilet while I am able to use that service makes me intensely uncomfortable and distresses me greatly.
I don't think it's a good idea to legally privilege someone else's discomfort over another person's rights.
