[k8s] Profile & optimize `sky launch` a new K8s cluster about skypilot

[romilbhardwaj]

One low hanging fruit here to shave off 1-2s for each ssh connection would be to find a more robust fix for concurrent SSH connections.

We currently have a random sleep for 1-2s to avoid an issue with kubectl port-foward failing on concurrent calls.

skypilot/sky/templates/kubernetes-port-forward-proxy-command.sh.j2

Lines 52 to 55 in 823999a

	# To avoid errors when many concurrent requests are sent (see https://github.com/skypilot-org/skypilot/issues/2628),
	# we add a random delay before establishing the socat connection.
	# Empirically, this needs to be at least 1 second. We set this to be random between 1 and 2 seconds.
	sleep $(shuf -i 10-20 -n 1 | awk '{printf "%f", $1/10}')

See more investigation here.

[Michaelvll]

#3157 is a prototype for fixing this issue, which reduces the time for launch new cluster and launch existing clusters by 2x and exec on existing cluster by 3x.

We can separate the optimization to multiple separate PRs:

1. #3394: lazy import and avoid rsync for exec
2. TBD: avoid source bashrc for internal remote command execution
3. #3157: kubernetes command runner