Exact same problem. Talking with some others I found on Valheim's steam discussions, seems to be related to survivalservers.
Control panel shows server is 'Ready!' but trying to connect directly through Steam it lists server as not responding. Have been playing since launch and just started having this issues this afternoon