Comments (15)
Re-opening, there are several reports in the NVIDIA Developer forums
from cuda-repo-management.
Hi @justinokamoto and @Angel-Popa
I am investigating this now.
from cuda-repo-management.
I rolled back the repository metadata from the last posting, the signatures (i.e. Release.gpg
, InRelease
) failed to upload. Checked now apt-get update
and installation of packages is functional.
I have scheduled a re-posting job to run in a few hours and will verify the intermittent issue is resolved. I'll close the issue pending that verification for all affected repos.
Please let me know if you continue to see any errors, thank you!
from cuda-repo-management.
Did this get fixed? sudo apt update
runs without errors now on our end.
from cuda-repo-management.
The metadata for each repo passes repo-validate.sh
and manual testing, closing.
from cuda-repo-management.
The new release file seems to be error-ing for us:
E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/Packages.gz Hash Sum mismatch
Hashes of expected file:
- Filesize:815853 [weak]
- SHA256:66e891d82894f08ecedac13c4e058ce5ea21882032c66a5b4d981cc2552f94a1
- SHA1:7cc76dbdc9d32d50cf5a1d377c8c877bcef23dcd [weak]
- MD5Sum:dc953e306c3b53f23ccdbe433881a800 [weak]
Hashes of received file:
- SHA256:989834105adeb2987d44b3da1cf0bd621e4a86a260f5790dd2f865105d231efe
- SHA1:da0b54ecbb7f3dd76602bef508e9a8df901d1cd2 [weak]
- MD5Sum:aa04b5a6fb8ceb8c4c0676ce2c144903 [weak]
- Filesize:815853 [weak]
Last modification reported: Mon, 11 Jul 2022 23:01:57 +0000
Release file created at: Tue, 12 Jul 2022 15:01:44 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
did something go amiss in fixing this issue?
I see few other mentions of things going wrong else where too:
- google-deepmind/alphafold#535
- https://forums.developer.nvidia.com/t/hash-sum-mismatch-from-apt-get-update-in-cuda-images/220686
from cuda-repo-management.
any update on this? Still getting the error
from cuda-repo-management.
Any update on this error. I'm facing similar issues while running below commands
RUN apt-cache policy libcudnn8
RUN apt-get install libcudnn8=8.3.2.44-1+cuda11.5
error response:
Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/./libcudnn8_8.3.2.44-1+cuda11.5_amd64.deb Hash Sum mismatch
Hashes of expected file:�[0mFetched 423 MB in 5s (82.5 MB/s)
�[91m
- SHA512:10cf6e68aa4f65e23fa75b481fdc1ba3d45181f1d3d7004b4731ae3d1532cfbaf141dadc5342a8f4eb15fae047b2af956fb88bd2fd50c9a66991ffffcf6c34e5
- SHA256:a1f5eeab52bddb36e94fc933acda2170df4bdf6565edafce47cd03d7248720d9
- SHA1:67aa82cc11b9974533adee3f923aec85fd984640 [weak]
- MD5Sum:8f3bd1d122899edbc0a90d485c8f22d0 [weak]
- Filesize:422575544 [weak]
Hashes of received file:
- SHA512:998af4d5e69aa9de5ff040c6e97eafcaa6adac55ffef0d04114a53401d10dad6e6ddc34eb2787c4bf7b3c7ae5294573e91ecf522dd42ecc4eb7a336e555dd619
- SHA256:6c8830a5f58fc64b583f0b4af52d5b2acdbc066698deecc05195164028dcc7ad
- SHA1:5ffcba4f0e7d47c91acd0632a3d1ed1dfc7b42bb [weak]
- MD5Sum:25e31500c9014e4675cba4659d8542a5 [weak]
- Filesize:422575544 [weak]
Last modification reported: Thu, 06 Jan 2022 06:44:48 +0000
from cuda-repo-management.
Getting the same thing. Have tried everything and no luck.
from cuda-repo-management.
Sorry I missed the email notification @xmalina-aibuild / @xmalina, @Nrohlable
I'm not able to reproduce the mismatch.
Based on the timestamp and the RUN
commands, I'm guessing you are using a Dockerfile FROM:
an image that has not been updated in some time?
$ podman run -it ubuntu:20.04 /bin/bash -c "apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y wget sudo ca-certificates gnupg; bash"
$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
$ sudo dpkg -i cuda-keyring_1.0-1_all.deb
$ sudo apt-get update
$ apt-cache policy libcudnn8
libcudnn8:
Installed: (none)
Candidate: 8.5.0.96-1+cuda11.7
Version table:
8.5.0.96-1+cuda11.7 600
600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages
8.4.1.50-1+cuda11.6 600
600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages
8.4.0.27-1+cuda11.6 600
600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages
8.3.3.40-1+cuda11.5 600
600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 Packages
$ sudo apt-get install --verbose-versions libcudnn8=8.3.2.44-1+cuda11.5
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
libcudnn8 (8.3.2.44-1+cuda11.5)
0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded.
Need to get 423 MB of archives.
After this operation, 1270 MB of additional disk space will be used.
Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64
libcudnn8 8.3.2.44-1+cuda11.5 [423 MB]
Fetched 423 MB in 5s (91.9 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libcudnn8.
(Reading database ... 4885 files and directories currently installed.)
Preparing to unpack .../libcudnn8_8.3.2.44-1+cuda11.5_amd64.deb ...
Unpacking libcudnn8 (8.3.2.44-1+cuda11.5) ...
Setting up libcudnn8 (8.3.2.44-1+cuda11.5) ...
from cuda-repo-management.
Hey @kmittman, sorry for the delayed response.
I'm using the Dockerhub Image tag corresponding to tensorflow/tensorflow:2.8.0-gpu and creating a image for Computer vision application and training. I was able to update the cuda version from 11.2 to 11.7 as you see from here:
(Reading database ... 19687 files and directories currently installed.)
Preparing to unpack .../libcudnn8_8.5.0.96-1+cuda11.7_amd64.deb ...
Unpacking libcudnn8 (8.5.0.96-1+cuda11.7) over (8.1.0.77-1+cuda11.2) ...
Setting up libcudnn8 (8.5.0.96-1+cuda11.7) ...
But at the runtime training when I checked tensorflow was still running on cuda 11.2, where I'm receiving this error:
Node: 'model/conv2d/Conv2D'
DNN library is not found.
[[{{node model/conv2d/Conv2D}}]] [Op:__inference_predict_function_841]
This seems to be a gpu issue since, the training appears to be running just fine on CPU tho.
Note: The error is poping up while using MTCNN lib for face detection which i'm using just for inference.
from cuda-repo-management.
Hi @Nrohlable
The output you shared indicates that you installed cuDNN 8.5.0.96
compiled for CUDA 11.7.x
, not that the CUDA 11.7 toolkit is actually installed.
Looking at the Tensorflow 2.8.0-gpu docker image tag you mentioned, it was last updated 8 months ago, on February 2nd.
However, we performed a GPG key rotation to a new public key on ~ April 28th. All of the packages in the NVIDIA repository, including CUDA 11.2 packages were re-signed using the new GPG key.
The apt
package manager requires the public key to be enrolled in the environment, one method is to install the cuda-keyring
package, another method would be to fetch the pubkey with apt-key
(deprecated), or could wget
&& mv
the keyring file.
My suggestion would be one of the following
- (a.) perform a workaround to update the GPG key manually
- (b.) use an updated Docker image tag (i.e.
2.8.1-gpu
) - (c.) contact the maintainer of the Dockerfile to rebuild the
2.8.0-gpu
tag, see: tensorflow/tensorflow#56085
from cuda-repo-management.
Hi @kmittman,
2.8.1-gpu tensorflow image is working fine without any issues in my case.
Thanks for your help, really appreciate it
from cuda-repo-management.
Hey @kmittman,
I hope you could help me with this as well.
I'm training my Siamese Network for Face verification using TensorFlow, below are the time it is taking for training for comparison with CPU and GPU:
Siamese on CPU : 18/899 [..............................] - ETA: 4:41:43 - loss: 0.8343
Siamese on GPU: 18/899 [..............................] - ETA: 4:52:04 - loss: 0.9894
It seems both of them take exactly similar time, which shouldn't be the case.
Below i'm also attaching CPU and GPU details:
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 3939414009020902295
xla_global_id: -1
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 14258995200
locality {
bus_id: 1
links {
}
}
incarnation: 3474452779198683813
physical_device_desc: "device: 0, name: Tesla T4, pci bus id: 0000:00:1e.0, compute capability: 7.5"
xla_global_id: 416903419
]
These are following commands i'm running at the training time in order to make sure it is working on GPU:
from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())
print(tf.config.list_physical_devices('GPU'))
tf.debugging.set_log_device_placement(True)
As disccused earlier i'm using Tensorflow-2.8.1-gpu for training it on GPU.
Am i doing anything wrong here or do I need to run some other commands as well in order to fix this ?
from cuda-repo-management.
Closing this repository issue. Please follow up with Tensorflow team.
from cuda-repo-management.
Related Issues (12)
- [Closed] GPG key rotation HOT 15
- E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz HOT 5
- Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/sbsa/Packages.gz HOT 2
- apt update does not work on nvidia/cuda:12.3.2-devel-ubuntu22.04 HOT 1
- File has unexpected size 24 Apr 2024 HOT 1
- RHEL 8 x86_64 repository wrong package file size HOT 3
- Question! HOT 2
- GPG signing key rotation for CUDA repositories HOT 13
- Receiving error about Packages.gz being the incorrect size HOT 2
- Metadata issue with the CUDA repositories [CDN] HOT 2
- Install from debian remote repository with packages that reference relative urls (e.g. /./) fails, nvidia, cuda, dcgm HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cuda-repo-management.