melbahja / got Goto Github PK
View Code? Open in Web Editor NEWGot: Simple golang package and CLI tool to download large files faster π than cURL and Wget!
License: MIT License
Got: Simple golang package and CLI tool to download large files faster π than cURL and Wget!
License: MIT License
Hi, thanks for your great project, it is really great !!
But I met a problem when using your tool to download ImageNet-21k(https://image-net.org/data/winter21_whole.tar.gz). It is a single file and its size is about 1.2T.
After downoding 411GB of the file, it stucks with erorr saying "file too large". Could you please tell me how to avoid the mistake?
Thank you very much!
This is critical!
Hi! I've tried to install got
from the AUR and I received this error:
panic: runtime error: slice bounds out of range [:7] with length 4
goroutine 1 [running]:
main.main()
/home/mohamed/work/dev/go/src/opensource/got/cmd/got/main.go:43 +0x583
I wonder if disabling SSL cert check is possible? In some cases we use self-signed certificates and got
refuses to download from that servers.
pops an error like in the following and doesn't start to download.
Get "URL-REDACTED": x509: certificate signed by unknown authority
Hi melbahja!
I got 2 issue while using got
, maybe we can improve
save disk space
I use got
to download a file from http server support range feature.
the file size is about 8GB, so it take a moment to complete.
while downloading, I find in C:\Users\{username}\AppData\Local\Temp\GotChunks{some digits}
directory,
there are 40 chunk files, chunk-0 ~ chunk-39
. they are temp files to assemble the origin file,
this means, my computer should have 16GB free disk space to complete the download progress, is there any way to optimize?
I mean, since got
remove these chunk-i files after merge them together, but, if I don't have 16GB free disk space,
the 8GB file can't be download. what if the the file is 200GB? I should first have 400GB free disk space, that's unreasonable.
merge chunk files take a long time
while all chunk files are downloaded, got
didn't quit immediately, it take a while to merge these chunk files,
after all chunk files merged, got
quit, and we got the origin file, all chunk files removed also.
if the origin 8GB file, it take a momnet to merge, what if a 200GB file, how long will it take to merge chunk filesοΌ
I don't know if wget
or curl
will suspended a moment while download progress bar reach 100%
thanks anyway!
I use --chunk 5242880 to specify ChunkSize.
If the size of the target file is less then 5M, the content downloaded is not valid. Its sha256 is not right.
I check the source code, I find it has a bug. When d.info.Size is less than d.ChunkSize, then chunksLen is zero.
I fix it in my fork https://github.com/simonwu-os/got.
It works fine.
chunksLen := d.info.Size / d.ChunkSize
///added by simon wu
///fix if chunksLen = 0, then the result file content is not valid.
if chunksLen == 0 {
chunksLen += 1
}
///end of added
Hello,
Just an FYI - there's already a project named Got - unfortunate coincidence it seems.
Not sure if there's anything that can be done but thought I'd put it out there.
Regards
I tried to download the binary from GitHub using got
:
got --out got.tar.gz https://github.com/melbahja/got/releases/download/v0.1.1/got_0.1.1_Linux_amd64.tar.gz
But instead I got Response status code is not ok: 403
.
After a little search I found this issue cavaliergopher/grab#43.
Basically AWS returns 403 to HEAD
requests, changing the HEAD
request to a GET
at
Line 275 in 9098e5b
I would like to stream the output so I can pipe to tar.
e.g. curl -s "url" | tar -xz -C /tmp
Could we have an option to stream to stdout like: got -o - | tar -xz -C /tmp
Currently, there's no way to access version information. Adding -v, --version
or simply tacking it to got -h,--help
would be great.
Hello,
I wanted to ask whether setting a default download path was a wanted feature. so the out flag is optional.
For example when downloading a list of files (another feature perhaps?) it can be tedious to always provide the outfile. sure, its possible using xargs and some replacements but is that how it should be?
so I'd like to suggest the following:
If the outpath is undefined the filepath is parsed from the URL, or if empty replaced by index
. Meaning:
http://example.com/some/path/video.mp4?hash=deadbeef&expires=123456789 -> video.mp4
http://example.com/some/path/video.mp4 -> video.mp4
http://example.com/ -> index
http://example.com/index.html -> index.html
http://example.com/?page=about -> index // to keep it simple
http://example.com/about.php?session=asdf -> about.php
I would implement this if it is a wanted feature.
Hi!
The tools is really fast! π But it seems to produce corrupt files. E.g.
got --out job_jobse..mp4 https://arteconcert-a.akamaihd.net/am/concert/096000/096900/096905-054-A_SQ_0_VO_05149030_MP4-2200_AMM-CONCERT-NEXT_1NdTQsPyN0.mp4
The download itself worked without issues. If I play the downloaded file with mpv
I get this:
mpv --no-video job_jobse.mp4
Video --vid=1 (*) (h264 1280x720 25.000fps)
(+) Audio --aid=1 (*) (aac 2ch 48000Hz)
AO: [pulse] 48000Hz stereo 2ch float
A: 00:11:55 / 00:59:39 (19%)
[ffmpeg/audio] aac: channel element 3.8 is not allocated
Error decoding audio.
[ffmpeg/audio] aac: Reserved bit set.
[ffmpeg/audio] aac: Number of bands (32) exceeds limit (20).
Error decoding audio.
[ffmpeg/audio] aac: Multiple frames in a packet.
[ffmpeg/audio] aac: Input buffer exhausted before END element found
Error decoding audio.
[ffmpeg/audio] aac: Sample rate index in program config element does not match the sample rate index configured by the container.
[ffmpeg/audio] aac: Inconsistent channel configuration.
[ffmpeg/audio] aac: get_buffer() failed
Error decoding audio.
[ffmpeg/audio] aac: Input buffer exhausted before END element found
Error decoding audio.
[ffmpeg/audio] aac: Prediction is not allowed in AAC-LC.
Error decoding audio.
[ffmpeg/audio] aac: Sample rate index in program config element does not match the sample rate index configured by the container.
[ffmpeg/audio] aac: Too large remapped id is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[ffmpeg/audio] aac: If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. ([email protected])
...
So it plays without issues until 11:55 and then I get lots of errors. Downloading the same file with wget
everything works as expected.
Could it be that there are race conditions in the tool that causes the chunks to be assembled in the wrong order or something like that? I suspect if you do the same download now that it might work for you because you didn't trigger the race condition. But that's just guessing of course.
Team,
Whenever we are restarting for same file it downloads from scratch. Does it support the multipart download download and restart from where we paused earlier.
Hi,
I get the following
got -c 128 https://datashare.ed.ac.uk/download/DS_10283_3443.zip
99.34% βββββββββββββββββββββββββββ 12 GB/12 GB @ 0 B/s
I'm using a VM with 8 vCPUs, so it's 16 threads per core. It downloaded @40-90MB/s at the beginning and then once it reached 99.45% it slows down to 0 B/s. What's going on?
Schollz just released progressbar. Would make a great addition to got.
The default build script does not include support for arm64, even though binaries are available.
Hello,
Launching the command to a server with basic authorization (jenkins), I receive the following error (the file size is 3 gbs.):
2021/12/13 08:27:18 stream error: stream ID 21; INTERNAL_ERROR; received from peer
This happens over the first thirty seconds. In the following executions, the ID keeps changing but it always happens. I think that the problem is with the slow connection of the server but i don't know if it is possible to stabilize the connection so that it does not fall. Sadly, I cannot provide a repository because my project is not open source.
Thanks for your work!
There is this, but it's not hugely well-known, and only a partial port and hasn't been update in a while. This tool's name (got
) is really excellent, and I would love to see something like wget
written in Go. There are a few ports of well-known UNIX commands in modern languages (such as fd
; a fast alternative to find
). Would really be interested to see where this could go! (No pun intended).
Sometimes the Webserver provides a filename, maybe it's a good idea to use this if none is provided. If there is none, the default behavior of parsing it from the URL is probably good enough.
Hi, the latest version didn't build with the following error
user@host got % go build ./...
# golang.org/x/sys/unix
../../go/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_arm64.go:28:3: //go:linkname must refer to declared function or variable
Original(?)/related issue refers potential solution:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.