nvidia / spark-rapids-container

Spark RAPIDS Container – Docker containers for Spark RAPIDS

RAPIDS Accelerator for Apache Spark Docker container

This repo provides the files to build the Docker conatiners with RAPIDS Accelerator for Apache Spark.

Docker containers

• Databricks container
• Kubenetes container

spark-rapids-container's Issues

[BUG] ANTLR Tool version 4.8 used for code generation does not match the current runtime version 4.9.3

Describe the bug
After following the step by step installation guide on Azure Databricks, I was met with a bug (ANTLR Tool version 4.8 used for code generation does not match the current runtime version 4.9.3). I can confirm that the cluster version is 10.4 LTS, with a worker type NC6_v3, and that the docker image is exactly the rapids-4-spark-databricks:23.02.0 from the repo. Is there an undocumented element to it ?

Steps/Code to reproduce bug
Follow the tutorial on running rapids with Docker on Databricks

Expected behavior
A running container

Environment details (please complete the following information)

• cloud : Azure
• Spark : 10.4 LTS
• Autoscaling : disabled
• Multi-node
• Photon : disabled
• auto-shutdown : disabled
• init script provided : file:/opt/spark-rapids/init.sh (from repo)

[BUG] Alluxio command reports error

Describe the bug

• Alluxio command reports logger not config error.
• Alluxio command reports PrometheusMetricsServlet ClassNotFoundException.
  It does not impact functionalities of Alluxio, but If we login to Web Terminal and run alluxio commans it shows errors.
  Since these errors do not show to end users, we can put this issue into low priority.

./alluxio fsadmin report

root@0224-020451-pugqs81d-10-59-182-117:/opt/alluxio/bin# ./alluxio fsadmin report
ERROR StatusLogger Reconfiguration failed: No configuration found for '7d4991ad' at 'null' in 'null'
06:44:21.573 [main] ERROR alluxio.metrics.MetricsSystem - Sink class alluxio.metrics.sink.PrometheusMetricsServlet cannot be instantiated
java.lang.ClassNotFoundException: alluxio.metrics.sink.PrometheusMetricsServlet
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_362]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_362]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) ~[?:1.8.0_362]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_362]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_362]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_362]
        at alluxio.metrics.MetricsSystem.startSinksFromConfig(MetricsSystem.java:273) ~[alluxio-client-2.9.0.jar:?]
        at alluxio.metrics.MetricsSystem.startSinks(MetricsSystem.java:210) ~[alluxio-client-2.9.0.jar:?]
        at alluxio.client.file.FileSystemContext.initContext(FileSystemContext.java:316) ~[alluxio-client-2.9.0.jar:?]
        at alluxio.client.file.FileSystemContext.init(FileSystemContext.java:305) ~[alluxio-client-2.9.0.jar:?]
        at alluxio.client.file.FileSystemContext.create(FileSystemContext.java:256) ~[alluxio-client-2.9.0.jar:?]
        at alluxio.client.file.FileSystemContext.create(FileSystemContext.java:225) ~[alluxio-client-2.9.0.jar:?]
        at alluxio.cli.fsadmin.FileSystemAdminShellUtils.checkMasterClientService(FileSystemAdminShellUtils.java:60) ~[alluxio-client-2.9.0.jar:?]
        at alluxio.cli.fsadmin.command.ReportCommand.run(ReportCommand.java:107) ~[alluxio-client-2.9.0.jar:?]
        at alluxio.cli.AbstractShell.run(AbstractShell.java:134) ~[alluxio-client-2.9.0.jar:?]
        at alluxio.cli.fsadmin.FileSystemAdminShell.main(FileSystemAdminShell.java:72) ~[alluxio-client-2.9.0.jar:?]


====================== The above is error msg================
====================== The below is output of this command ======

Alluxio cluster summary: 
    Master Address: 10.59.182.117:19998
    Web Port: 19999
    Rpc Port: 19998
    Started: 04-28-2023 06:39:15:645
    Uptime: 0 day(s), 0 hour(s), 5 minute(s), and 6 second(s)
    Version: 2.9.0
    Safe Mode: false
    Zookeeper Enabled: false
    Raft-based Journal: true
    Raft Journal Addresses: 
        10.59.182.117:19200
    Live Workers: 1
    Lost Workers: 0
    Total Capacity: 68.73GB
        Tier: SSD  Size: 68.73GB
    Used Capacity: 0B
        Tier: SSD  Size: 0B
    Free Capacity: 68.73GB

Steps/Code to reproduce bug
Create a Databricks cluster with:
Environment variables:
PROMETHEUS_COPY_DATA_PATH=/dbfs/chongg/dblogs-prometheus
ENABLE_ALLUXIO=1
Docker image URL: gaochong365/rapids-4-spark-databricks:23.04.0-rc1

After started the cluster, login to Web Terminal via Apps Tab.
cd /opt/alluxio/bin
./alluxio fsadmin report

[CICD] Enable the template for new issue & pull request

Enable the template for new issue & pull request

[DOC] build.sh should document what parameters are and what they do

Report incorrect documentation

Location of incorrect documentation
Databricks build.sh should document what parameters are and what they do

[BUG] Document overriding the spark rapids jar in docker image and name used

Describe the bug
Using the released docker image with the Job Workflow options in databricks, the Spark UI doesn't load. It first goes to the master UI and then if still try to click through to the application UI, it gives a bad gateway.

[FEA] Alluxio home settings should be pinned in Databricks Docker container

Is your feature request related to a problem? Please describe.
When using Alluxio in the Databricks Docker container, we require the user to set both ALLUXIO_HOME environment variables and the Spark configuration spark.rapids.alluxio.home. These should be unnecessary as the version of Alluxio is fixed inside the Docker container image.

Describe the solution you'd like
The ALLUXIO_HOME environment variable should be set in the Docker container's built-in init script. This can done during the Docker build process using a sed command inside the Dockerfile. The documentation should be updated to not require this setting. Also, for spark.rapids.alluxio.home, maybe the code itself can read ALLUXIO_HOME as a possible default value here (then we can file a corresponding issue in https://github.com/NVIDIA/spark-rapids`) or this setting can be added via sed command in the Dockerfile to update 00-custom-spark-driver-defaults.conf. Therefore the user should not be fully concerned with knowing the exact Alluxio paths, it should just work if ENABLE_ALLUXIO=1

[BUG] libpq.so.5: cannot open shared object file

User job failing using the databricks container: libpq.so.5: cannot open shared object file: No such file or directory

we likely need to install postgresql

[BUG] databricks docker missing numpy, pandas, pyarrow, and psycopg2

Describe the bug
our databricks docker container is missing basic python libraries numpy, pandas, pyarrow, and psycopg2. Since these are used a lot by many users we should include them. One specific customer tried our container and had failures because these missing.

Specific versions in 10.4 and I believe 11.3:

numpy==1.20.1 \
pandas==1.2.4 \
pyarrow==4.0.0 \
psycopg2==2.8.5 \

[CICD] Add signoff CI as required check

could simply copy from

https://github.com/NVIDIA/spark-rapids-jni/blob/branch-22.12/.github/workflows/signoff-check.yml
https://github.com/NVIDIA/spark-rapids-jni/tree/branch-22.12/.github/workflows/signoff-check

and make some repo-related updates

[DOC] we need release docs

Report incorrect documentation

we need release docs to point to released versions so peple can find our images from here

nvcr.io/nvidia/rapids-4-spark-databricks:22.10.0

[BUG] premerge action w/ invalid syntax

Describe the bug
e.g.
https://github.com/NVIDIA/spark-rapids-container/actions/runs/3511628027

Invalid workflow file: .github/workflows/premerge.yml#L26
The workflow is not valid. .github/workflows/premerge.yml (Line: 26, Col: 3): The identifier 'build docker image' is invalid. IDs may only contain alphanumeric characters, '', and '-'. IDs must start with a letter or '' and and must be less than 100 characters.

[CICD] Add premerge CI using github action

As container build does not require GPU resource, we could leverage github action resource to implement the premerge CI here

could refer to https://github.com/marketplace/actions/build-and-push-docker-images

and mark push: false

        name: Build
        uses: docker/build-push-action@v3
        with:
          push: false
          tags: <some PR info>

and also https://docs.github.com/en/actions/publishing-packages/publishing-docker-images

[FEA] Should use ubuntu 20.04+ from v23.02

We will drop ubuntu 18.04 support from v23.02 release.
So, for the K8s docker, we should use ubuntu 20.04+.