Unable to mount drives on Google GKE using the gcsfuse driver about zero-to-jupyterhub-k8s

[vizeit]

I have been using GCS bucket as a shared drive for a while now. You will need to include more mount options

"implicit-dirs,uid=1000,gid=100"

Also, make sure the workload identity has permission to access the bucket, ‘storage object admin’ role
You may also need the following settings

singleuser:
  networkPolicy:
    egressAllowRules:
      cloudMetadataServer: true

[consideRatio]

@jdbates wasn't this configuration enough for you? If not, are you also using Cilium (part of GCP's dataplane v2)?

singleuser:
  networkPolicy:
    egressAllowRules:
      cloudMetadataServer: true

[jdbates]

@consideRatio After further testing, it looks like just using

singleUser:
  networkPolicy:
    egressAllowRules:
      cloudMetadataServer: true

is sufficient. As far as my environment, I'm using a GKE Autopilot cluster with whatever the default configuration is.

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[jdbates]

Thank you for the help, @vizeit. It was the networkPolicy which was the culprit; apparently the default setting were blocking the gcsfuse sidecar container from connecting with the bucket. As a result, the sidecar never finished mounting the bucket, which in turn caused the notebook container to never finish spawning, which is what was responsible for the context deadline exceeded errors I was receiving.

I was able to connect successfully using:

singleUser:
  networkPolicy:
    enabled: false

Obviously this is not ideal; I'll need to figure out how to specify the exact egress rule I need (maybe just allow the IP of the bucket). Also, the additional mountOptions ("uid=1000,gid=100") did not seem to matter for my specific use case.

This issue can be closed, since it turned out to be a configuration issue and not a bug. However, it would be nice to have a little more documentation surrounding this, since none of the error messages I received were useful in diagnosing the problem.

[vizeit]

@jdbates I think you will need all the mount options I mentioned in my previous comment; try to write/save something to the shared drive without uid -gid mount options. If you browse through closed issues related to GKE Autopilot here in this repo, you may get more understanding of Dataplane V2

[jdbates]

As of right now, the following config seems to be working for me:

singleuser:
  image:
    name: jupyter/datascience-notebook
    tag: latest
  cmd: null
  startTimeout: 600
  storage:
    dynamic:
      storageClass: premium-rwo
    extraVolumes:
    - name: gcs-shared
      csi:
        driver: gcsfuse.csi.storage.gke.io
        volumeAttributes:
          bucketName: scg-datascience-shared
          mountOptions: "implicit-dirs"
    extraVolumeMounts:
    - name: gcs-shared
      mountPath: /home/shared
  extraFiles:
    jupyter_notebook_config.json:
      mountPath: /etc/jupyter/jupyter_notebook_config.json
      data:
        MappingKernelManager:
          cull_idle_timeout: 3600 # default: 0
          cull_interval: 300 # default: 300
          cull_connected: true # default: false
          cull_busy: false # default: false
  serviceAccountName: gcsfuse
  extraAnnotations:
    gke-gcsfuse/volumes: "true"
  cloudMetadata:
    blockWithIptables: false
  networkPolicy:
    egressAllowRules:
      cloudMetadataServer: true

I'll keep an eye on this and check whether or not the "uid=1000,gid=100" mountOptions matter, but as of right now they don't seem to be affecting anything.

Also, I can't believe I didn't see the issue you had posted @vizeit - would have saved me a weeks worth of trouble if I had. Closing this, since it's resolved and a duplicate.

[vizeit]

I have described detailed steps in my post if anyone wants to fully setup GCS bucket as a shared drive with zero-to-jupyterhub

https://www.vizeit.com/gcs-bucket-with-jupyterhub-on-gke/