Comments (9)
@pacospace do you think we could mitigate this issue by using "overlays" and having notebook specific dependencies? If so, we should look into using those here. This is a good first use case for a Elyra/KFpipeline that actually requires unique runtime images for individual notebooks. How does the aicoe-ci/build-check handle repo's with out a global Pipfile?
Thanks @MichaelClifford, I agree could be a very nice use case.
The tutorial we have shows how the repo will look like: https://github.com/thoth-station/elyra-aidevsecops-tutorial
Basically each notebook will have its own kernel and overlays/
repo matches kernel name selected for the notebook. If you use jupyterlab-requirements all kernels and overlays directory will be updated automatically. And the dependecies and kernel name will be present in notebook metadata as well.
So steps are:
- for each notebook/step have a software stack in
overlays/
directory: https://github.com/thoth-station/elyra-aidevsecops-tutorial/tree/master/overlays - update
.aicoe.yaml
to use overlays builds, you can have a look here: https://github.com/thoth-station/elyra-aidevsecops-tutorial/blob/master/.aicoe-ci.yaml
Then when you open a release the aicoe-ci pipeline will do the magic and each overlays will have an image on your registry depending on the name you specify in the .aicoe.yaml. Then your images are ready to be used in the Elyra Pipeline.
from mailing-list-analysis-toolkit.
@oindrillac why are you forcing scikit-learn==0.20.3
I'm trying to run this on the s2i-elyra-v6
image and it looks like both the language packages, hatesonar
and flair
require newer versions of sk-learn
from mailing-list-analysis-toolkit.
By not specifying the versions for hatesonar
or scikit-learn
in the Pipfile, running pipenv lock --pre
then pipenv install
seems to work.
from mailing-list-analysis-toolkit.
@MichaelClifford hatesonar with a scikit-learn > 0.20.3 was giving this error
The problem above is not only with installing scikit-learn==0.20.3.
Noticed the same when I tried
pipenv install --pre hatesonar
as well on top of the Pipfile on upstream.
I'm trying to run this on the s2i-elyra-v6 image
Doesnt the elyra v6 image support 3.8? The repo is on version python 3.6, and hence I've been trying this on the mailing-list image.
from mailing-list-analysis-toolkit.
@MichaelClifford on a version greater than 0.20.3
sonar = Sonar()
gives the following error
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-9-b474975b2fe6> in <module>
----> 1 sonar = Sonar()
/opt/app-root/lib/python3.6/site-packages/hatesonar/api.py in __init__(self)
19 model_file = os.path.join(BASE_DIR, 'model.joblib')
20 preprocessor_file = os.path.join(BASE_DIR, 'preprocess.joblib')
---> 21 self.estimator = joblib.load(model_file)
22 self.preprocessor = joblib.load(preprocessor_file)
23
/opt/app-root/lib/python3.6/site-packages/joblib/numpy_pickle.py in load(filename, mmap_mode)
583 return load_compatibility(fobj)
584
--> 585 obj = _unpickle(fobj, filename, mmap_mode)
586 return obj
/opt/app-root/lib/python3.6/site-packages/joblib/numpy_pickle.py in _unpickle(fobj, filename, mmap_mode)
502 obj = None
503 try:
--> 504 obj = unpickler.load()
505 if unpickler.compat_mode:
506 warnings.warn("The file '%s' has been generated with a "
/usr/lib64/python3.6/pickle.py in load(self)
1048 raise EOFError
1049 assert isinstance(key, bytes_types)
-> 1050 dispatch[key[0]](self)
1051 except _Stop as stopinst:
1052 return stopinst.value
/usr/lib64/python3.6/pickle.py in load_global(self)
1336 module = self.readline()[:-1].decode("utf-8")
1337 name = self.readline()[:-1].decode("utf-8")
-> 1338 klass = self.find_class(module, name)
1339 self.append(klass)
1340 dispatch[GLOBAL[0]] = load_global
/usr/lib64/python3.6/pickle.py in find_class(self, module, name)
1386 elif module in _compat_pickle.IMPORT_MAPPING:
1387 module = _compat_pickle.IMPORT_MAPPING[module]
-> 1388 __import__(module, level=0)
1389 if self.proto >= 4:
1390 return _getattribute(sys.modules[module], name)[0]
ModuleNotFoundError: No module named 'sklearn.linear_model.logistic'
from mailing-list-analysis-toolkit.
since the hatesonar
approach is experimental and is most likely not the final approach, is it a good idea not to track the dependencies in the pipfile and lock
from mailing-list-analysis-toolkit.
ERROR: ERROR: Could not find a version that matches black
Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1,18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
There are incompatible versions in the resolved dependencies.
@oindrillac From the error message, it looks like the issue is not hatesonar or sklearn, but rather black. What happens when you explicitly mention black version in the pipfile like this:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
ipython = "*"
[packages]
pandas = "*"
boto3 = "*"
wget = "*"
python-dotenv = "*"
sklearn = "*"
papermill = "*"
ibm-cloud-sdk-core = "*"
ibm-watson = "*"
scikit-learn = "==0.20.3"
hatesonar = "*"
black = "==20.8b1"
[requires]
python_version = "3.6"
[pipenv]
allow_prereleases = false
[thoth.allow_prereleases]
black = true
The above works fine for me on operate-first JH, on the mailing-list-analysis-toolkit
image.
from mailing-list-analysis-toolkit.
@chauhankaranraj thanks for the tip
black
seems to stop throwing an error after this.
however, i think the scikit-learn version is still throws a conflict when flair
is added
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
ipython = "*"
[packages]
pandas = "*"
boto3 = "*"
wget = "*"
python-dotenv = "*"
sklearn = "*"
papermill = "*"
ibm-cloud-sdk-core = "*"
ibm-watson = "*"
flair = "*"
scikit-learn = "==0.20.3"
hatesonar = "*"
black = "==20.8b1"
[requires]
python_version = "3.6"
[pipenv]
allow_prereleases = false
[thoth.allow_prereleases]
black = true
getting the version mismatch issue
✘ Locking Failed!
[ResolutionFailure]: File "/opt/app-root/lib64/python3.8/site-packages/pipenv/resolver.py", line 741, in _main
[ResolutionFailure]: resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]: File "/opt/app-root/lib64/python3.8/site-packages/pipenv/resolver.py", line 702, in resolve_packages
[ResolutionFailure]: results, resolver = resolve(
[ResolutionFailure]: File "/opt/app-root/lib64/python3.8/site-packages/pipenv/resolver.py", line 684, in resolve
[ResolutionFailure]: return resolve_deps(
[ResolutionFailure]: File "/opt/app-root/lib64/python3.8/site-packages/pipenv/utils.py", line 1395, in resolve_deps
[ResolutionFailure]: results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]: File "/opt/app-root/lib64/python3.8/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
[ResolutionFailure]: resolver.resolve()
[ResolutionFailure]: File "/opt/app-root/lib64/python3.8/site-packages/pipenv/utils.py", line 833, in resolve
[ResolutionFailure]: raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches scikit-learn==0.20.3,>=0.19.1,>=0.21.3 (from -r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 4))
Tried: 0.9, 0.10, 0.11, 0.12, 0.12.1, 0.13, 0.13.1, 0.14, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.17, 0.17.1, 0.18, 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.21.1, 0.21.2, 0.21.3, 0.22, 0.22, 0.22.1, 0.22.1, 0.22.2, 0.22.2.post1, 0.22.2.post1, 0.23.0, 0.23.0, 0.23.1, 0.23.1, 0.23.2, 0.23.2, 0.24.0, 0.24.0, 0.24.1, 0.24.1, 0.24.1
Skipped pre-versions: 0.15.0b1, 0.15.0b2, 0.16b1, 0.17b1, 0.19b2, 0.20rc1, 0.21rc2, 0.22rc2.post1, 0.22rc3, 0.22rc3, 0.23.0rc1, 0.23.0rc1, 0.24.dev0, 0.24.dev0, 0.24.0rc1, 0.24.0rc1
There are incompatible versions in the resolved dependencies:
scikit-learn==0.20.3 (from -r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 4))
scikit-learn (from sklearn==0.0->-r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 8))
scikit-learn>=0.19.1 (from hatesonar==0.0.7->-r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 9))
scikit-learn>=0.21.3 (from flair==0.8.0.post1->-r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 3))
and I am not sure how we can overcome this other than not tracking the scikit-learn pinned dependency in the Pipfile.
from mailing-list-analysis-toolkit.
@pacospace do you think we could mitigate this issue by using "overlays" and having notebook specific dependencies? If so, we should look into using those here. This is a good first use case for a Elyra/KFpipeline that actually requires unique runtime images for individual notebooks. How does the aicoe-ci/build-check handle repo's with out a global Pipfile?
from mailing-list-analysis-toolkit.
Related Issues (20)
- Fully deploy in MOC
- Failed to update dependencies to their latest version HOT 22
- Move all "stage" notebooks moved into correct directly locations.
- Add the Fedora User Mailing list HOT 3
- Recreate the "Keyword" section of dashboard with Watson tools.
- Move keyword analysis data cleaning steps into the raw_to_text notebook
- dependency error in repo HOT 7
- New minor release HOT 1
- Deliver Container Image HOT 1
- Pipenv install fails on trying to install ibm packages HOT 2
- Hatesonar dependency issue HOT 8
- Move Mailing list Argo Workflow to Operate 1st
- New minor release HOT 1
- Remove jupyter dependency
- Consolidate and update regex parsing in `raw_to_text.ipynb` notebook
- Move Mailing list Elyra/KFPipelines Workflow on Op1st
- edge cases for cleaning .mbox
- Apply overlays to manage notebook specific dependencies HOT 2
- Failed to update dependencies to their latest version for rhel:8 environment HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mailing-list-analysis-toolkit.