Comments (11)
from data-science-on-gcp.
Tried that - same error as before
from data-science-on-gcp.
from data-science-on-gcp.
That made it work, thanks Lak.
(By the way for anyone else at this stage, you might need to run export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
to get it to work.)
I then tried running the pipeline train_on_vertexai.py and it spent about 10 mins training before failing due to a memory error:
RuntimeError: Training failed with: code: 8 message: "The following quota metrics exceed quota limits: aiplatform.googleapis.com/custom_model_training_nvidia_t4_gpus"
I'm currently running the same pipeline using AutoML and it's been training for 2 hours so far - should it take that long?
from data-science-on-gcp.
from data-science-on-gcp.
-
Done, submitted a pull request
-
I do have an NVIDIA T4 GPU attached but it's still failing with the same error:
- AutoML pipeline ended up completing after 3 hours 40 mins.
from data-science-on-gcp.
I'm getting the same error when trying to run hyperparameter tuning:
google.api_core.exceptions.ResourceExhausted: 429 The following quota metrics exceed quota limits: aiplatform.googleapis.com/custom_model_training_nvidia_t4_gpus
from data-science-on-gcp.
from data-science-on-gcp.
from data-science-on-gcp.
When navigating to console.cloud.google.com/quotas it says that I need to upgrade to a paid account:
I'm still using a managed service (I tried creating a user-managed one a few days ago but it didn't work, something about not enough GPUs currently being available....I just put it down to bad timing and decided to try again later). I guess that's the root of the problem. Will try again.
from data-science-on-gcp.
So I still can't create a user-managed notebook with a GPU:
Have tried US-west, -east and -central. Sometimes I also get this error:
from data-science-on-gcp.
Related Issues (20)
- download.sh does not download the data HOT 1
- There is no Table Explorer in BQ HOT 1
- chapter 04: df07.py - Unable to open file: gs://BUCKETNAME/flights/staging/ch04timecorr.1656567385.996847/pipeline.pb. HOT 3
- hello
- problem extracting bigquery table data to cloud storage HOT 2
- Crash during Deployment of model on Vertex AI HOT 14
- Chapter 6 create_cluster.sh script problem
- Chapter 4: tz_correct KeyError: "CRS_ARR_TIME [while running 'flights:tzcorr-ptransform-157']" HOT 2
- I cannot make the table definition for the federated source in chapter 4 HOT 2
- AttributeError: tzinfo HOT 2
- README improvement for chapters 2 and 3 regarding upload to BQ HOT 4
- Ch.4 - Time Correction - input/output error when running df02.py HOT 3
- Ch7 - can't increase quota of Dataproc cluster; workaround not working HOT 1
- Ch.9 - can't export data to GCS - wrong project number? HOT 4
- Ch9 - training deep neural network - how to attach GPU? HOT 7
- Ch11 - error when running AutoML on full dataset HOT 1
- CH10 code to set TensorFlow level is broken now that there are 2 digits second level of current TensorFlow HOT 2
- Chapter 2 - ingest_flights.py - Access denied to gs file on bqload function HOT 4
- Ch9 - Call predictions using Flask webserver HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from data-science-on-gcp.