This is a unusually long setup : "Waiting for group to become stable, current operations: creating: 1" about gcloudrig

[koderian]

EDIT: I found out this is because their is no resources available

[deekue]

I got this too. Poking around on the Cloud Console I found the instance group is failing to start with this error

Instance 'gcloudrig-n2q1' creation failed: Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 0.0 globally.

Each GPU type in each region has a quota of 1 for me but GPUS_ALL_REGIONS has a quota of 0.
I filed a support ticket requesting a quota increase, which said it would take 2 business days.

[deekue]

it seems GPUs are not available to Trial accounts. I upgraded to a Paid account but I still have 0 quota for GPUS_ALL_REGIONS

$ gcloud compute project-info describe --project gcloudrig-222205 --flatten="quotas" | grep -C1 GPUS_ALL_REGIONS  
  limit: 0.0
  metric: GPUS_ALL_REGIONS
  usage: 0.0

I filed a quota change ticket and it was completed the next day. Now I get

$ gcloud compute project-info describe --project gcloudrig-222205 --flatten="quotas" | grep -C1 GPUS_ALL_REGIONS  
  limit: 1.0
  metric: GPUS_ALL_REGIONS
  usage: 0.0

tried setup.sh again and now the instance is failing due to the zone not having resources (saw this on the Cloud Console, not sure how to detect this from gcloud).

[putty182]

Not all regions have GPUs, and trial accounts must be upgraded before you'll get any GPU quota; you still keep & use the free trial credits, but if you use them all up it will start charging you.

These scripts setup a multi-zone instance group, and pick all zones in the region that describe themselves as supporting the relevant GPU/accelerator. E.g. some regions don't have GPUs in 'c' regions, and others don't "officially" have P4/P100 GPUs, but actually do.

In theory, this means GCE will give you the next available instance from any zone in the region that has available resources & quota.

Quota issues aside, I've found some regions (australia-southeast1) have been a bit hit and miss without any published GCE outages. Usually launches within 5 minutes, but I have on occasion waited more than 30 minutes.

I'd recommend checking the Instance Group in the web console for actual warning/error messages.

[koderian]

@putty182 I get this too "**Permission denied (HTTP 403): Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 0.0 globally.**"

[deekue]

posted #14 to check for lack of GPUS_ALL_REGIONS quota, and provide info to fix it.