Valheim defaults to wrong port when using Custom Domains

[ShinyBlue 0]

I am using a domain I own to try and connect to my SurvivalServers Valheim server.  I've set up my DNS records as this wiki article explains.  However, the port assigned to my server (eg 12345) is not the default Valheim port (2456).  The Valheim client seems to make a DNS lookup for the appropriate IP, but then uses the default port (2456) if you haven't supplied one.  The result is that just using "my-domain.com" will make a request with the wrong port, skip my SRV record, and result in a "Failed to Connect" error.  I have to use "my-domain.com:12345" in order to connect to my server.  Is there any solution to this, such as changing my server's port to "2456" or forcing Valheim to not add the wrong port?