Unstable connection causes training to freeze about clearml

[bmartinn]

Hi @OfirKP , the trains package was designed with always on network connection in mind, unfortunately there is no 'offline' option at the moment.

That said let me expand on the network issues you seem to run into:

If a connection is dropped (such as in your case), it will retry to connect to the server, these are the retry messages you keep getting.

If a connection was dropped after a socket was created (post handshake) it will use the socket request timeout (imagine a very very very slow download connection, this is essentially the same timeout). This timeout is defined to 10 minutes, after 10min passed and the request didn't return fully, a new connection will be established.

Setting retry limit to zero will not effect the socket timeout, and I suspect that this is was you observed, when you completely cut the network connections.

My quickest solution for you is to mark out the Task.init() line, whenever you are in unstable network environment (such as, commuting on a train 😃 ).

[bmartinn]

Closing, due to lack of activity.