Check if you can connect to tfs server collection through web portal or not.
Othrewise it's usually a proxy or a gateway issue HTTP Status 504
TFS plugin attempts to use the HTTP proxy to connect to the TFS server. Check you proxy settings or you could try to change the no proxy host config under Jenkins > Plugin Manager > Advanced, similar to the wildcard domain that tfs server resides on and then try again to reconnect.