Comments (7)
For reference, logs from master node:
from initialization-actions.
Was able to resolve simply by removing the sourcing calls to /root/.bashrc
(no longer needed due to setting PATH env vars and other configs in global profiles). This properly allowed conda to be installed along with all dependencies, and also ensures updated path (exposing miniconda) for both root
and non-root users. Will submit patch shortly.
from initialization-actions.
OK, the PR above should resolve this. It has in my testing, including functionality contained in existing PRs depending on conda. Give it a try and we should get this merged asap. Thanks!
from initialization-actions.
Sorry for the breakage! We indeed rolled out some minor cleanup of the VM environment for newly created clusters, and didn't intend to change anything in a breaking way. We haven't quite standardized the testing of known initialization actions in the release process yet, but in the future we should be able to catch these before any public releases. I'll take a look at the PR.
from initialization-actions.
@dennishuo, thanks for the quick resolution. We look forward to contributing to the testing of init actions, and I believe a first step would be to expose some google compute engine resources to be for spinning up a minimal dataproc cluster and running a test suite. Internally, we will be implementing this soon, but believe it would be awesome if Google would contribute some resources to do the same.
Also, from our side, is there anything we can do to prevent such breakages in the future (any way to reference specific subminor dataproc versions)?
Thanks again!
from initialization-actions.
@nehalecky We're definitely hoping to provide a nice sandboxed CI/test setup for running minimal test suites, unfortunately can't guarantee a timeline at this point since we're still shuffling around various prioritizations.
It's indeed possible to reference specific subminor versions, but the long-term deprecation policies cover major.minor version "tracks" and not specific subminor versions, so they may be subject to shorter lifecycles of being officially supported compared to the lifecycles of major.minor tracks.
That said, in this case the previous subminor version that we'd expect to still work with the old conda init action would be version "1.0.0", and the current version is "1.0.1". Pinning to 1.0.1 now would ensure workloads are unaffected when 1.0.2 comes out, but it'd be strongly recommended to switch to newer subminor versions as soon as possible; previous subminor versions wouldn't get bug fixes or new features.
We'll probably add a nicer way to query for the list of supported and deprecated subminor versions, but in the meantime you can at least see the complete list by creating a cluster and specifying a bogus version like --image-version=notavalidversion
from initialization-actions.
@dennishuo, thank you for the comprehensive reply. We're excited about these future developments and will continue to work alongside you all to see this advanced! Have a great weekend!
from initialization-actions.
Related Issues (20)
- [hue] hive editor missing.
- [oozie] intermittent error writing to HDFS during init action HOT 1
- [gpu] ml-on-gcp repo (gpu metrics dependency) to be archived
- Missing linux headers on debian dataproc instances after update HOT 6
- Terraform provider does not offer a sequential ordering option - implement as init action HOT 2
- [bigtable] 2.1 clusters fail to come online with stock bigtable/bigtable.sh HOT 2
- [livy] update livy init action for 2.1 HOT 1
- [rapids] please update to work with latest dask-rapids v22.12 HOT 2
- [gpu] Driver does not install on 2.2 Rocky/Ubuntu images HOT 1
- [zeppelin] not supported on 2.1+ image versions HOT 1
- Error on wget livy binary naming HOT 5
- [spark-rapids] Drop Spark 2.x support in spark-rapids.sh
- [gpu] apt-get update Init script seeing broken repositories HOT 2
- [bigtable] apt-get update Init script seeing broken repositories
- [cloud-sql-proxy] Running the Cloud SQL Proxy as a persistent service
- Update initialization scripts to install latest RAPIDS `23.12` OR `24.02` HOT 2
- [gpu] Add tests for GPU agent HOT 1
- initialization actions which use apt-get update fail due to purged oldoldstable backports repository HOT 10
- rstudio.sh is unable to get the receive keys. Maybe due to invalid repo key. HOT 1
- Dataproc "apt-get update" failed on ubuntu20 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from initialization-actions.