open-ce / dali-feedstock

This is the Open-CE project for feedstock collection, environment data, and build scripts

Welcome to the open-ce project. The project contains everything that is needed to build conda packages for a collection of machine learning and deep learning frameworks. All packages created for a specific version of Open-CE have been designed to be installed within a single conda environment. For more information on conda, please look at conda's official documentation.

This repository contains a collection of Open-CE files that can be used to create a conda channel. The conda channel will contain packages for every feedstock listed within the Open-CE files. Different variants of Python and CUDA can be specified at build time. Open-CE currently supports the following:

	Supported Versions
Architecture	Power, x86, s390x
Python	3.10, 3.11
CUDA	11.8, 12.2

The open-ce tool can also be used to build all or some of the packages provided by Open-CE. For more information on the open-ce tool, please see the open-ce-builder repository.

Open-CE Builder compatibility with various Open-CE releases

Open-CE version	Open-CE Builder version
1.7.8	12.0.1
1.8.1	12.0.1
1.9.1	12.0.3
1.10.0	12.0.3
1.11.0	>=13.0.1

GETTING STARTED

Requirements

• conda
  • The conda tool can either be installed through Anaconda or Miniconda.

Community Builds

Oregon State University hosts pre-built versions of the Open-CE conda channels here. These channels provide packages for both Power and x86 architectures. The latest version of Open-CE can be pulled down using the channel: https://ftp.osuosl.org/pub/open-ce/current/.

MIT hosts pre-built versions of Open-CE for the IBM Power architecture. Multiple versions of Open-CE are hosted within this single channel: https://opence.mit.edu/.

Builds with Enterprise Production Support

Rocket Software hosts pre-built versions of the Open-CE at conda channel here. This channel provides packages for Power architecture(ppc64le). The latest version of Open-CE can be pulled down using the mentioned channel.

In order to get notification you can register at https://community.rocketsoftware.com/forums/forum-home/community-members?communitykey=c7ece6e8-5a29-4a17-a2bc-68b65f89d29f. Please reach out to [email protected] for any questions and enquiries.

Installing Packages

Open-CE packages can be installed from one of the community builds. To install packages from one of the community channels, pass the channel's URL to the conda tool using the -c option.

The following command will install the tensorflow package from the most recent version of Open-CE hosted in within the OSU channel:

conda install -c https://ftp.osuosl.org/pub/open-ce/current/ tensorflow

For more information on installing conda packages, please see the official conda documentation.

For more information on installing conda packages created from using the open-ce tools locally, please see the open-ce-builder repository.

Contributions

For contribution information, please see the CONTRIBUTING.md page.

Slack Community

Join us on Slack! Use this link or ping the @open-ce/open-ce-dev-team for an invite.

dali-feedstock's Issues

Replace cudatoolkit-dev with nvcc package in DALI

Remove instances of cudatoolkit-dev from recipes. It should be replaced with {{ compiler('cuda')}} inside of the build section.

If CUDA_HOME is being set in the build.sh file, it will need to be removed.

[REQUEST]

Hello. Rafal from NVIDIA DALI team here.

Is your feature request related to a problem? Please describe.
We're including a new source dependency that requires GCC>=9.4 to compile and your build config uses GCC 7. Would it be a problem for you to bump the GCC version to 9.4?

Describe the solution you'd like
Bumping gcc version to 9.4

Describe alternatives you've considered
It would be very hard to decrease the compiler requirement for the project and we're quite convinced that we should build it from source.

Additional context
The new dependency is CV-CUDA which will replace some of the DALI operators' implementations. So, you could disable the cv-cuda build, but with time, as we port more operators, such a DALI build would loose some of the operators.

[1.3] Update DALI

DALI did hit 1.0 and is continuing on with subsequent releases

https://github.com/NVIDIA/DALI/releases

Update DALI for Open-CE 1.3 with a version that will match with TF 2.5.

DALI TF Plugin python files missing

The python files for DALI's tf_plugin don't appear to be installed properly. When attempting import nvidia.dali_tf_plugin, an error is returned: ModuleNotFoundError: No module named 'nvidia.dali_tf_plugin'.

I think that the recipe is just missing copying dali_tf_plugin/nvidia/dali_tf_plugin into PREFIX.

As part of this fix, DALI's tf_plugin tests should be added:

    nosetests --verbose test_dali_tf_plugin.py:TestDaliTfPluginLoadOk

    # DALI TF run
    nosetests --verbose test_dali_tf_plugin_run.py

    # DALI TF DATASET run
    nosetests --verbose test_dali_tf_dataset.py

    nosetests --verbose test_dali_tf_dataset_shape.py