It depends on the client and the security implementations they support. For example IIRC no client support the last version of OMEMO (I think it was about OMEMO, I remember an article about it some time ago). Also are you sure that all the other people’s clients are on the same version and you’re not susceptible to a downgrade attack?

Unless you are ready to/want to control the whole environment (i.e. at least the clients and possibly the server), look into simplex.chat

Simplex.chat

No identifiers, pfp, FOSS, can route through tor.

Or host your own matrix or xmpp server.

First questio is: can you ask your home internet provider for ipv6?

Otherwise sign up to tailscale and connect your vps server (and your pc/devices) to it.