pipenv install fails about mailing-list-analysis-toolkit

[pacospace]

@pacospace do you think we could mitigate this issue by using "overlays" and having notebook specific dependencies? If so, we should look into using those here. This is a good first use case for a Elyra/KFpipeline that actually requires unique runtime images for individual notebooks. How does the aicoe-ci/build-check handle repo's with out a global Pipfile?

Thanks @MichaelClifford, I agree could be a very nice use case.

The tutorial we have shows how the repo will look like: https://github.com/thoth-station/elyra-aidevsecops-tutorial

Basically each notebook will have its own kernel and overlays/ repo matches kernel name selected for the notebook. If you use jupyterlab-requirements all kernels and overlays directory will be updated automatically. And the dependecies and kernel name will be present in notebook metadata as well.

So steps are:

• for each notebook/step have a software stack in overlays/ directory: https://github.com/thoth-station/elyra-aidevsecops-tutorial/tree/master/overlays
• update .aicoe.yaml to use overlays builds, you can have a look here: https://github.com/thoth-station/elyra-aidevsecops-tutorial/blob/master/.aicoe-ci.yaml

Then when you open a release the aicoe-ci pipeline will do the magic and each overlays will have an image on your registry depending on the name you specify in the .aicoe.yaml. Then your images are ready to be used in the Elyra Pipeline.

[MichaelClifford]

@oindrillac why are you forcing scikit-learn==0.20.3 I'm trying to run this on the s2i-elyra-v6 image and it looks like both the language packages, hatesonar and flair require newer versions of sk-learn

[MichaelClifford]

By not specifying the versions for hatesonar or scikit-learn in the Pipfile, running pipenv lock --pre then pipenv install seems to work.

[oindrillac]

@MichaelClifford hatesonar with a scikit-learn > 0.20.3 was giving this error

The problem above is not only with installing scikit-learn==0.20.3.

Noticed the same when I tried

pipenv install --pre hatesonar as well on top of the Pipfile on upstream.

I'm trying to run this on the s2i-elyra-v6 image

Doesnt the elyra v6 image support 3.8? The repo is on version python 3.6, and hence I've been trying this on the mailing-list image.

[oindrillac]

@MichaelClifford on a version greater than 0.20.3

sonar = Sonar() gives the following error

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-9-b474975b2fe6> in <module>
----> 1 sonar = Sonar()

/opt/app-root/lib/python3.6/site-packages/hatesonar/api.py in __init__(self)
     19         model_file = os.path.join(BASE_DIR, 'model.joblib')
     20         preprocessor_file = os.path.join(BASE_DIR, 'preprocess.joblib')
---> 21         self.estimator = joblib.load(model_file)
     22         self.preprocessor = joblib.load(preprocessor_file)
     23 

/opt/app-root/lib/python3.6/site-packages/joblib/numpy_pickle.py in load(filename, mmap_mode)
    583                     return load_compatibility(fobj)
    584 
--> 585                 obj = _unpickle(fobj, filename, mmap_mode)
    586     return obj

/opt/app-root/lib/python3.6/site-packages/joblib/numpy_pickle.py in _unpickle(fobj, filename, mmap_mode)
    502     obj = None
    503     try:
--> 504         obj = unpickler.load()
    505         if unpickler.compat_mode:
    506             warnings.warn("The file '%s' has been generated with a "

/usr/lib64/python3.6/pickle.py in load(self)
   1048                     raise EOFError
   1049                 assert isinstance(key, bytes_types)
-> 1050                 dispatch[key[0]](self)
   1051         except _Stop as stopinst:
   1052             return stopinst.value

/usr/lib64/python3.6/pickle.py in load_global(self)
   1336         module = self.readline()[:-1].decode("utf-8")
   1337         name = self.readline()[:-1].decode("utf-8")
-> 1338         klass = self.find_class(module, name)
   1339         self.append(klass)
   1340     dispatch[GLOBAL[0]] = load_global

/usr/lib64/python3.6/pickle.py in find_class(self, module, name)
   1386             elif module in _compat_pickle.IMPORT_MAPPING:
   1387                 module = _compat_pickle.IMPORT_MAPPING[module]
-> 1388         __import__(module, level=0)
   1389         if self.proto >= 4:
   1390             return _getattribute(sys.modules[module], name)[0]

ModuleNotFoundError: No module named 'sklearn.linear_model.logistic'

[oindrillac]

since the hatesonar approach is experimental and is most likely not the final approach, is it a good idea not to track the dependencies in the pipfile and lock

[chauhankaranraj]

ERROR: ERROR: Could not find a version that matches black
Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1,18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
There are incompatible versions in the resolved dependencies.

@oindrillac From the error message, it looks like the issue is not hatesonar or sklearn, but rather black. What happens when you explicitly mention black version in the pipfile like this:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
ipython = "*"

[packages]
pandas = "*"
boto3 = "*"
wget = "*"
python-dotenv = "*"
sklearn = "*"
papermill = "*"
ibm-cloud-sdk-core = "*"
ibm-watson = "*"
scikit-learn = "==0.20.3"
hatesonar = "*"
black = "==20.8b1"

[requires]
python_version = "3.6"

[pipenv]
allow_prereleases = false

[thoth.allow_prereleases]
black = true

The above works fine for me on operate-first JH, on the mailing-list-analysis-toolkit image.

[oindrillac]

@chauhankaranraj thanks for the tip

black seems to stop throwing an error after this.

however, i think the scikit-learn version is still throws a conflict when flair is added

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
ipython = "*"

[packages]
pandas = "*"
boto3 = "*"
wget = "*"
python-dotenv = "*"
sklearn = "*"
papermill = "*"
ibm-cloud-sdk-core = "*"
ibm-watson = "*"
flair = "*"
scikit-learn = "==0.20.3"
hatesonar = "*"
black = "==20.8b1"

[requires]
python_version = "3.6"

[pipenv]
allow_prereleases = false

[thoth.allow_prereleases]
black = true

getting the version mismatch issue

✘ Locking Failed! 
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/resolver.py", line 741, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/resolver.py", line 702, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/resolver.py", line 684, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/utils.py", line 1395, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/utils.py", line 833, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches scikit-learn==0.20.3,>=0.19.1,>=0.21.3 (from -r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 4))
Tried: 0.9, 0.10, 0.11, 0.12, 0.12.1, 0.13, 0.13.1, 0.14, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.17, 0.17.1, 0.18, 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.21.1, 0.21.2, 0.21.3, 0.22, 0.22, 0.22.1, 0.22.1, 0.22.2, 0.22.2.post1, 0.22.2.post1, 0.23.0, 0.23.0, 0.23.1, 0.23.1, 0.23.2, 0.23.2, 0.24.0, 0.24.0, 0.24.1, 0.24.1, 0.24.1
Skipped pre-versions: 0.15.0b1, 0.15.0b2, 0.16b1, 0.17b1, 0.19b2, 0.20rc1, 0.21rc2, 0.22rc2.post1, 0.22rc3, 0.22rc3, 0.23.0rc1, 0.23.0rc1, 0.24.dev0, 0.24.dev0, 0.24.0rc1, 0.24.0rc1
There are incompatible versions in the resolved dependencies:
  scikit-learn==0.20.3 (from -r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 4))
  scikit-learn (from sklearn==0.0->-r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 8))
  scikit-learn>=0.19.1 (from hatesonar==0.0.7->-r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 9))
  scikit-learn>=0.21.3 (from flair==0.8.0.post1->-r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 3))

and I am not sure how we can overcome this other than not tracking the scikit-learn pinned dependency in the Pipfile.

[MichaelClifford]

@pacospace do you think we could mitigate this issue by using "overlays" and having notebook specific dependencies? If so, we should look into using those here. This is a good first use case for a Elyra/KFpipeline that actually requires unique runtime images for individual notebooks. How does the aicoe-ci/build-check handle repo's with out a global Pipfile?