[BUG] - MLFlow endpoint doesn't work in WSL2 about k3ai

[alefesta]

I can confirm that exposing MLFlow through port forward on port 5001 binding 0.0.0.0 works...I think I'll write a quick workaround ..and will mention this issue inside #3
I'm going to close the issue if you need to comment over please use #3 and again THANKS for trying K3ai!

[burntcarrot]

All resources:

❯ kubectl get all
NAME                                 READY   STATUS    RESTARTS   AGE
pod/k3ai-executor-76b4cb4686-bwscr   1/1     Running   0          30m
pod/minio-0                          1/1     Running   0          40m
pod/mlflow-5fff476c9f-t59bb          1/1     Running   0          37m
pod/postgres-0                       1/1     Running   0          38m

NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/kubernetes         ClusterIP   10.96.0.1       <none>        443/TCP          41m
service/minio-service      ClusterIP   10.96.183.209   <none>        9000/TCP         40m
service/mlflow-service     NodePort    10.96.150.194   <none>        5000:30500/TCP   36m
service/postgres-service   ClusterIP   10.96.90.90     <none>        5432/TCP         38m

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/k3ai-executor   1/1     1            1           30m
deployment.apps/mlflow          1/1     1            1           37m

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/k3ai-executor-76b4cb4686   1         1         1       30m
replicaset.apps/mlflow-5fff476c9f          1         1         1       37m

NAME                        READY   AGE
statefulset.apps/minio      1/1     40m
statefulset.apps/postgres   1/1     38m

Service and pod description:

❯ kubectl describe svc mlflow-service
Name:                     mlflow-service
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=mlflow
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.150.194
IPs:                      10.96.150.194
Port:                     http  5000/TCP
TargetPort:               5000/TCP
NodePort:                 http  30500/TCP
Endpoints:                10.244.0.7:5000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
❯ kubectl describe pod mlflow-5fff476c9f-t59bb
Name:         mlflow-5fff476c9f-t59bb
Namespace:    default
Priority:     0
Node:         myk3ai-control-plane/172.24.0.2
Start Time:   Fri, 05 Nov 2021 19:20:05 +0530
Labels:       app=mlflow
              pod-template-hash=5fff476c9f
Annotations:  <none>
Status:       Running
IP:           10.244.0.7
IPs:
  IP:           10.244.0.7
Controlled By:  ReplicaSet/mlflow-5fff476c9f

I might be wrong, but does this have anything to do with port forwarding?

[alefesta]

Hi, @burntcarrot first THANKS for your issue!
Second, what did you use to deploy the cluster on WSL2? KinD?
Finally, yep what did you observe is a well-known limitation on WSL2 😭 currently actually it lays on multiple problems there:

1. WSL2 uses dynamic IP that does not goes along with K8s very well. This is not up to K3ai but to the K8s providers. Docker addresses this natively with their proxy but..
2. NodePort stop working.. some like KinD project use extra-ports mapping but the limitation there is that the pod port and the nodePort have to be the same.

I should hack too many things that would become un-mantainable on the long run.. so port-forward may be a solution, something on the line k3ai plugin -n mlflow -port-forward but it would result in something like a terminal where you cannot operate otherwise it would stop the port-forward. Still would means the user would have to open another terminal to run any other command...

[alefesta]

I was thinking maybe get a complementary app to expose ports as port-forward ? Something like k3ai-expose that is invocated when the plugin is deployed and adds a port-forward for the service "on-demand"?