Comments (18)
Need to move to latest mozprocess version sometime - https://pypi.org/project/mozprocess/#files. As you said, it supports Python 3 now.
from clusterfuzz.
FYI the URL for NDB on Python 3 seems to have changed to https://github.com/googleapis/python-ndb/blob/master/MIGRATION_NOTES.md
from clusterfuzz.
git ls-files | grep requirements.txt | xargs -n1 -t caniusepython3 -r
caniusepython3 -r docker/base/requirements.txt
Finding and checking dependencies ...
🎉 You have 0 projects blocking you from using Python 3!
caniusepython3 -r docker/ci/requirements.txt
Finding and checking dependencies ...
You need 1 project to transition to Python 3.
Of that 1 project, 1 has no direct dependencies blocking its transition:
googledatastore
caniusepython3 -r src/local/requirements.txt
Finding and checking dependencies ...
🎉 You have 0 projects blocking you from using Python 3!
caniusepython3 -r src/platform_requirements.txt
Finding and checking dependencies ...
🎉 You have 0 projects blocking you from using Python 3!
caniusepython3 -r src/requirements.txt
Finding and checking dependencies ...
🎉 You have 0 projects blocking you from using Python 3!
from clusterfuzz.
I think googledatastore should go away after moving to the new ndb package, right @oliverchang ?
from clusterfuzz.
right, googledatastore will no longer be required afterwards.
from clusterfuzz.
Step 1a completed:
git ls-files "*.py" | xargs futurize -w -n --stage1 -x libfuturize.fixes.fix_absolute_import
from clusterfuzz.
Need to enable --py3k in pylint here, but first need help on fixing absolute import things.
butler/lint.py: _execute_command_and_track_error('pylint ' + file_path)
from clusterfuzz.
Some notes for tracking
ml/rnn/generator.py: rnn_runner = new_process.ProcessRunner('python')
bot/tasks/ml_train_task.py: new_process.ProcessRunner('python')
protos/generate.sh: python2 -m grpc_tools.protoc --proto_path=$SCRIPT_DIR --python_out=$SCRIPT_DIR --grpc_python_out=$SCRIPT_DIR $SCRIPT_DIR/*.proto
from clusterfuzz.
appengine.mail dependency removed in 26bd818. we now provide support for sendgrid (also one of recommendations by appengine).
from clusterfuzz.
Futurize Stage 1 migration is now complete with 3016965
from clusterfuzz.
ndb still looks like it's in early development based on commits on https://github.com/googleapis/python-ndb ...
to replace the users module, firebase auth looks like the easiest and nicest way forward: https://cloud.google.com/appengine/docs/standard/python/authenticating-users-firebase-appengine
(from the suggested options on https://cloud.google.com/appengine/docs/standard/python/oauth/ )
from clusterfuzz.
It's unclear if firebase auth has a local emulator of sorts so that people can still do run_server without setting up any real projects though...
Perhaps local server instances could have auth disabled completely (i.e. no auth required) to keep things simple? @inferno-chromium what do you think? I think the current local server user system just lets you set a fake email... So we wouldn't be regressing much.
from clusterfuzz.
yes no auth for local sounds good. and i had the same idea on firebase, firebase support multiple other auth systems, so that will be great!
from clusterfuzz.
great, I can give firebase auth a shot next week :)
from clusterfuzz.
Rough plan for the remainder of the migration:
- Complete remaining futurize migrations for bot code.
- Add support for using the Python 3 ndb (dynamically based on if we're running on Python 2 or 3).
- Get all tests passing under Python 3, and more generally get our code in a state which makes it easy to run under both Python 2 and 3.
- Find a replacement for App Engine memcache.
- Migrate App Engine deployment to Python 3. Completely remove dependency on App Engine SDK and migrate to App Engine Python 3 Standard.
- Migrate preemptible bot deployment to Python 3, and gradual rollout to non-preemptible.
@mbarbella-chromium WDYT ?
from clusterfuzz.
The only thing I'd add is to evaluate packages in requirements for Python 3 compaitbility, but that will be part of getting all of the tests passing under Python 3. SGTM overall.
from clusterfuzz.
Current state is that most unit tests pass. All App Engine unit tests pass.
The next steps:
- Set up CI for Python 3 tests to make sure we don't regress anything.
- Pick a deployment (chromium?) to dogfood Python 3 appengine.
- Start staging of Python 3 bots.
from clusterfuzz.
Main part is done here.
from clusterfuzz.
Related Issues (20)
- Allow custom builds in nonprivileged execution HOT 1
- Ensure that impact tasks aren't run on non-chrome projects
- Fix issues with zero size thread pools. HOT 1
- clusterfuzz service crashing on fuzzbots HOT 2
- Investigate queue backlog HOT 8
- Rate limit task retries.
- Get rid of unwanted lines stat
- Fix uploading of custom builds.
- Unable to access clusterfuzz HOT 2
- Fix issue where global variables are causing tests to pass when they should fail HOT 1
- Figure out the future of peach in CF HOT 2
- Remove cap on weight based on fuzz targets HOT 1
- Add a retry for batch failures
- Decide if we want to keep email feature
- Investigate if chromium-tests-syncer works
- Lint src/python and src/scripts more often
- Fix linting errors in src/local/butler/scripts/setup.py HOT 2
- Deployment broken by python package upgrade
- Upgrade to python3.11 HOT 8
- Fix firebase login HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clusterfuzz.