oracle / accelerated-data-science Goto Github PK

ADS is the Oracle Data Science Cloud Service's python SDK supporting, model ops (train/eval/deploy), along with running workloads on Jobs and Pipeline resources.

Home Page: https://accelerated-data-science.readthedocs.io/

License: Universal Permissive License v1.0

Makefile 0.02% Python 98.61% Dockerfile 0.02% Jinja 1.15% Shell 0.01% Jupyter Notebook 0.17% C 0.03% HTML 0.01%

cloud machine-learning oci oracle python3

accelerated-data-science's People

Contributors

Stargazers

Watchers

accelerated-data-science's Issues

[Bug]: forcing dependency on pyyaml 6< to 5.4.1 gives cython issue while building version 5.4.1

Oracle-ads version used

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of oracle-ads.
I have confirmed this bug exists on the main branch of oracle-ads.
I agree to follow Code of Conduct.

Description

installing latest version of oracle-ads, it has depency on pyyaml for version 6< (not =), and >5.4.1, so it takes 5.4.1 but with python 3.10 and 5.4.1 I'm getting a cython error :

Collecting pyyaml==5.4 (from -r requirements.txt (line 1))
Downloading PyYAML-5.4.tar.gz (174 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174.8/174.8 kB 4.4 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [62 lines of output]
/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in setup.cfg
!!

          ********************************************************************************
          The license_file parameter is deprecated, use license_files instead.
  
          By 2023-Oct-30, you need to update your project and remove deprecated calls
          or your builds will no longer be supported.
  
          See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
          ********************************************************************************
  
  !!
    parsed = self.parsers.get(option_name, lambda x: x)(value)
  running egg_info
  writing lib3/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib3/PyYAML.egg-info/top_level.txt
  Traceback (most recent call last):
    File "/Users/mac/.pyenv/versions/3.10.1/envs/mwaa/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/Users/mac/.pyenv/versions/3.10.1/envs/mwaa/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/mac/.pyenv/versions/3.10.1/envs/mwaa/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 271, in <module>
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1234, in run_command
      super().run_command(command)
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 314, in run
      self.find_sources()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
      mm.run()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 551, in run
      self.add_defaults()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
      sdist.add_defaults(self)
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
      super().add_defaults()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
      self._add_defaults_ext()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
    File "<string>", line 201, in get_source_files
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
(mwaa) mac@Mohameds-MacBook-Pro dpl-datapipelines-mwaa % pip install -r requirements.txt
Collecting pyyaml==5.3.1 (from -r requirements.txt (line 1))
Using cached PyYAML-5.3.1.tar.gz (269 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done

How to Reproduce

using python 3.10

pip install oracle-ads

What was Observed

          ********************************************************************************
          The license_file parameter is deprecated, use license_files instead.
  
          By 2023-Oct-30, you need to update your project and remove deprecated calls
          or your builds will no longer be supported.
  
          See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
          ********************************************************************************
  
  !!
    parsed = self.parsers.get(option_name, lambda x: x)(value)
  running egg_info
  writing lib3/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib3/PyYAML.egg-info/top_level.txt
  Traceback (most recent call last):
    File "/Users/mac/.pyenv/versions/3.10.1/envs/mwaa/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/Users/mac/.pyenv/versions/3.10.1/envs/mwaa/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/mac/.pyenv/versions/3.10.1/envs/mwaa/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 271, in <module>
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1234, in run_command
      super().run_command(command)
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 314, in run
      self.find_sources()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
      mm.run()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 551, in run
      self.add_defaults()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
      sdist.add_defaults(self)
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
      super().add_defaults()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
      self._add_defaults_ext()
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
    File "<string>", line 201, in get_source_files
    File "/private/var/folders/v8/z7_rv36j4pb8_hzv3nzvqhmm0000gn/T/pip-build-env-anvl__aw/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

What was Expected

successful installation of oracle ads with pyyaml 6

Version

2.8.7

I'm using a macbook pro M1 , but got the same issue on AWS linux based docker container. (MWAA)

Proposed solution :

loosen the pyyaml versions to allow higher than version 6 or equal to version 6

Allowing all parameters from `CreateApplicationDetails` to be passed through `prepare_app`

In particular, the Spark configuration properties in OCI dataflow can not be set using ads.dataflow, but it's able to be set with oci.data_flow_models.CreateApplicationDetails. Since prepare_app already calls CreateApplicationDetails it possible to add the rest of the CreateApplicationDetails parameters into prepare_app?

accelerated-data-science/ads/dataflow/dataflow.py

Line 171 in d594ed0

def prepare_app(

[FR]: Enable overwriting parameters in ads opctl run for Data Science Jobs, Pipelines and Data Flow Applications.

Willingness to contribute

Yes. I can contribute this feature independently.

Proposal Summary

Currently, the "ads opctl" command-line tool allows users to run existing Data Science jobs, pipelines and Data Flow applications by providing their OCIDs. However the tool lacks the capability to overwrite certain parameters associated with infrastructure or runtime settings.

This feature request aims to enhance the "ads opctl run" functionality by introducing the ability to overwrite parameters when running these jobs, applications or pipelines. By allowing users to modify infrastructure and runtime-related parameters, such as resource allocation, compute configurations, or environment variables, it would provide greater flexibility and control over the execution process.

Motivation

With the proposed changes, users will be able to specify custom parameter values the supersede the default or pre-configured settings during runtime. This will enable them to tailor the infrastructure and runtime aspects according to their specific requirements, ensuring optimal performance and resource utilization.

The ability to overwrite parameters will empower data scientists, data engineers, and other users of the ads tool to experiment, fine-tune, and optimize their workloads without having to create new job instances or modify existing configurations. This enhanced flexibility will significantly improve the productivity and efficiency of data-centric operations.

By implementing this feature, the "ads opctl run" command will become more versatile and adaptable to various scenarios, enabling seamless integration into different data processing pipelines and workflows.

Details

The enhancement could enable users to define an configuration YAML file that contains parameter values to the overridden during runtime. This approach simplifies the command execution process and allows for easier management and sharing of parameter configurations.

Example:

ads opctl run --ocid ocid1.datasciencejob.oc1... --file path/to/the/YAML

prepare_generic_model() throws error in Oracle Cloud Infrastructure DataScience Jobs

While using prepare_generic_model() with ADS in OCI Data Science Jobs, we experienced following problem:

"Traceback (most recent call last):"
" File "/usr/local/lib/python3.9/site-packages/ads/model/generic_model.py", line 610, in prepare"
" xgb_model.prepare("
"ValueError: inference_conda_env must be specified."
"During handling of the above exception, another exception occurred:"
"Exception: Could not locate manifest file in the provided conda environment: None.
" File "/usr/local/lib/python3.9/site-packages/ads/model/common/utils.py", line 82, in fetch_manifest_from_conda_location"
" manifest = fetch_manifest_from_conda_location(conda_prefix)"

It seems that ADS is not picking automatically the Python Version as well as the current Conda Environment used.

Workaround

to make this code work, specify inference_conda_env and inference_python_version manually
and while using .save specify introspection=False

More about prepare_generic_model() can be found here: ADS Model Catalog

... and here ads.common.model_export_util.prepare_generic_model

Inconsistency of the GenericModel.predict(data) in ADS 2.6.2 and 2.6.6 versions.

Generic Model's input data has not been serialized into data_dict in v2.6.2. However, in v2.6.6 and above, data passed in GenericModel.predict(data) will be wrapped in an object {"data": , "data_type": } . The score.py generated by v2.6.2 is not able to handle the data given by predict in v2.6.6 and above.

If you have a Model deployed with score.py generated by v2.6.2 and invoked by using predict in v2.6.6 and above, you will see this discrepancy.

AttributeError: type object 'JobInfrastructureConfigurationDetails' has no attribute 'JOB_INFRASTRUCTURE_TYPE_ME_STANDALONE'

When executing

from ads.jobs import Job
from ads.jobs import DataScienceJob, ScriptRuntime

job_name = 'attrition-model-training-job'

I'm getting :

AttributeError: type object 'JobInfrastructureConfigurationDetails' has no attribute 'JOB_INFRASTRUCTURE_TYPE_ME_STANDALONE'

in ADS 2.5.9.

ADS set_auth() is failing on resource_principal

in ADS version 2.7.0

ads.set_auth(auth='resource_principal')
fails and return:

ERROR:ads.common:ADS Exception Traceback (most recent call last): File "/home/datascience/conda/generalml_p38_cpu_v1/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3457, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_5566/331412029.py", line 1, in <module> ads.set_auth(auth="resource_principal") File "/home/datascience/conda/generalml_p38_cpu_v1/lib/python3.8/site-packages/ads/common/auth.py", line 156, in set_auth raise ValueError( ValueError: ~/.oci/config path does not exist, please provide existing path to config file.

Looks like ADS is expecting use principal here with config file in path.

Distributed training local test does not support source code from Git repo or OCI object storage

Following the documentation here, it allows users to specify source code from Git repo or OCI object storage. However if we try to debug the workload locally using command like ads opctl run -f train.yaml -b local, ADS will not do any git clone or copy code from remote to the docker image. Instead, it will still mount the local directory as code directory and run file specified as entry point. This only works if user already have a copy of the code locally (this copy might not be the same as the one from remote uri, as specified in the yaml). Otherwise there will be a FileNotFound error.

The behavior might be useful if user is actively developing the code. Then push the code to remote before running the training on data science jobs. However, users are likely to hit the error if they simply want to run the training using code from a remote git repository.

[Bug]: ModuleNotFoundError: No module named 'mlx'

Oracle-ads version used

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of oracle-ads.
I have confirmed this bug exists on the main branch of oracle-ads.
I agree to follow Code of Conduct.

Description

When importing

>>> from ads.explanations.explainer import ADSExplainer

I get

ModuleNotFoundError: No module named 'mlx'

I've searched around and can't find any way to get this resolved.

How to Reproduce

import ads
from ads.explanations.explainer import ADSExplainer

What was Observed

ModuleNotFoundError: No module named 'mlx'

What was Expected

The ADSExplainer to be imported

Version

'2.8.1'

Private reporting of a security issue

Dear Oracle team,

I may have identified a security vulnerability in the codebase of the latest version of accelerated-data-science (not 100% sure, I'm still checking... please).

I would like to report it privately. Could you please consider to opt-in this Github feature for private reporting.

Kind consideration
@Sim4n6

ads.to_sql with replace option not reading complete dataset

when using ads.to_sql with the if_exists="replace" option the resulting table is not created to reflect the maximum column lengths in the dataset. We only notice this in large datasets > 4m rows.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

oracle / accelerated-data-science Goto Github PK

accelerated-data-science's People

Contributors

Stargazers

Watchers

Forkers

accelerated-data-science's Issues

Oracle-ads version used

Description

How to Reproduce

What was Observed

What was Expected

Version

Proposed solution :

Willingness to contribute

Proposal Summary

Motivation

Details

Oracle-ads version used

Description

How to Reproduce

What was Observed

What was Expected

Version

Recommend Projects

Recommend Topics

Recommend Org