melbahja / got Goto Github PK

Sometimes (not always, not safely reproducible) I get unrealistic filesizes (see screenshot)

This happens from time to time on URLs where it works on other times (this one was with a github repo download about 10 MiB in size)

Hi, thanks for your great project, it is really great !!

But I met a problem when using your tool to download ImageNet-21k(https://image-net.org/data/winter21_whole.tar.gz). It is a single file and its size is about 1.2T.

After downoding 411GB of the file, it stucks with erorr saying "file too large". Could you please tell me how to avoid the mistake?

Thank you very much!

This is critical!

Hi! I've tried to install got from the AUR and I received this error:

panic: runtime error: slice bounds out of range [:7] with length 4

goroutine 1 [running]:
main.main()
	/home/mohamed/work/dev/go/src/opensource/got/cmd/got/main.go:43 +0x583

I wonder if disabling SSL cert check is possible? In some cases we use self-signed certificates and got refuses to download from that servers.

pops an error like in the following and doesn't start to download.

Get "URL-REDACTED": x509: certificate signed by unknown authority

Hi melbahja!
I got 2 issue while using got, maybe we can improve

1. save disk space
   I use got to download a file from http server support range feature.
   the file size is about 8GB, so it take a moment to complete.
   while downloading, I find in C:\Users\{username}\AppData\Local\Temp\GotChunks{some digits} directory,
   there are 40 chunk files, chunk-0 ~ chunk-39. they are temp files to assemble the origin file,
   this means, my computer should have 16GB free disk space to complete the download progress, is there any way to optimize?
   I mean, since got remove these chunk-i files after merge them together, but, if I don't have 16GB free disk space,
   the 8GB file can't be download. what if the the file is 200GB? I should first have 400GB free disk space, that's unreasonable.

2. merge chunk files take a long time
   while all chunk files are downloaded, got didn't quit immediately, it take a while to merge these chunk files,
   after all chunk files merged, got quit, and we got the origin file, all chunk files removed also.
   if the origin 8GB file, it take a momnet to merge, what if a 200GB file, how long will it take to merge chunk files？
   I don't know if wget or curl will suspended a moment while download progress bar reach 100%

thanks anyway!

I use --chunk 5242880 to specify ChunkSize.
If the size of the target file is less then 5M, the content downloaded is not valid. Its sha256 is not right.
I check the source code, I find it has a bug. When d.info.Size is less than d.ChunkSize, then chunksLen is zero.

I fix it in my fork https://github.com/simonwu-os/got.
It works fine.

chunksLen := d.info.Size / d.ChunkSize
///added by simon wu
///fix if chunksLen = 0, then the result file content is not valid.
if chunksLen == 0 {
chunksLen += 1
}
///end of added

Hello,

Just an FYI - there's already a project named Got - unfortunate coincidence it seems.

Not sure if there's anything that can be done but thought I'd put it out there.

Regards

I tried to download the binary from GitHub using got:
got --out got.tar.gz https://github.com/melbahja/got/releases/download/v0.1.1/got_0.1.1_Linux_amd64.tar.gz

But instead I got Response status code is not ok: 403.

After a little search I found this issue cavaliergopher/grab#43.

Basically AWS returns 403 to HEAD requests, changing the HEAD request to a GET at

I would like to stream the output so I can pipe to tar.

e.g. curl -s "url" | tar -xz -C /tmp

Could we have an option to stream to stdout like: got -o - | tar -xz -C /tmp

Currently, there's no way to access version information. Adding -v, --version or simply tacking it to got -h,--help would be great.

Hello,
I wanted to ask whether setting a default download path was a wanted feature. so the out flag is optional.

For example when downloading a list of files (another feature perhaps?) it can be tedious to always provide the outfile. sure, its possible using xargs and some replacements but is that how it should be?

so I'd like to suggest the following:

If the outpath is undefined the filepath is parsed from the URL, or if empty replaced by index. Meaning:

http://example.com/some/path/video.mp4?hash=deadbeef&expires=123456789 -> video.mp4
http://example.com/some/path/video.mp4 -> video.mp4
http://example.com/ -> index
http://example.com/index.html -> index.html
http://example.com/?page=about -> index // to keep it simple
http://example.com/about.php?session=asdf -> about.php

I would implement this if it is a wanted feature.

Hi!

The tools is really fast! 😃 But it seems to produce corrupt files. E.g.

got --out job_jobse..mp4 https://arteconcert-a.akamaihd.net/am/concert/096000/096900/096905-054-A_SQ_0_VO_05149030_MP4-2200_AMM-CONCERT-NEXT_1NdTQsPyN0.mp4

The download itself worked without issues. If I play the downloaded file with mpv I get this:

mpv --no-video job_jobse.mp4 
     Video --vid=1 (*) (h264 1280x720 25.000fps)
 (+) Audio --aid=1 (*) (aac 2ch 48000Hz)
AO: [pulse] 48000Hz stereo 2ch float
A: 00:11:55 / 00:59:39 (19%)
[ffmpeg/audio] aac: channel element 3.8 is not allocated
Error decoding audio.
[ffmpeg/audio] aac: Reserved bit set.
[ffmpeg/audio] aac: Number of bands (32) exceeds limit (20).
Error decoding audio.
[ffmpeg/audio] aac: Multiple frames in a packet.
[ffmpeg/audio] aac: Input buffer exhausted before END element found
Error decoding audio.
[ffmpeg/audio] aac: Sample rate index in program config element does not match the sample rate index configured by the container.
[ffmpeg/audio] aac: Inconsistent channel configuration.
[ffmpeg/audio] aac: get_buffer() failed
Error decoding audio.
[ffmpeg/audio] aac: Input buffer exhausted before END element found
Error decoding audio.
[ffmpeg/audio] aac: Prediction is not allowed in AAC-LC.
Error decoding audio.
[ffmpeg/audio] aac: Sample rate index in program config element does not match the sample rate index configured by the container.
[ffmpeg/audio] aac: Too large remapped id is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[ffmpeg/audio] aac: If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. ([email protected])
...

So it plays without issues until 11:55 and then I get lots of errors. Downloading the same file with wget everything works as expected.

Could it be that there are race conditions in the tool that causes the chunks to be assembled in the wrong order or something like that? I suspect if you do the same download now that it might work for you because you didn't trigger the race condition. But that's just guessing of course.

Team,

Whenever we are restarting for same file it downloads from scratch. Does it support the multipart download download and restart from where we paused earlier.

Hi,
I get the following

got -c 128 https://datashare.ed.ac.uk/download/DS_10283_3443.zip
  99.34% ▕████████████████████████▉▏ 12 GB/12 GB @ 0 B/s

I'm using a VM with 8 vCPUs, so it's 16 threads per core. It downloaded @40-90MB/s at the beginning and then once it reached 99.45% it slows down to 0 B/s. What's going on?

Schollz just released progressbar. Would make a great addition to got.

The default build script does not include support for arm64, even though binaries are available.

Hello,

Launching the command to a server with basic authorization (jenkins), I receive the following error (the file size is 3 gbs.):

2021/12/13 08:27:18 stream error: stream ID 21; INTERNAL_ERROR; received from peer

This happens over the first thirty seconds. In the following executions, the ID keeps changing but it always happens. I think that the problem is with the slow connection of the server but i don't know if it is possible to stabilize the connection so that it does not fall. Sadly, I cannot provide a repository because my project is not open source.

Thanks for your work!

There is this, but it's not hugely well-known, and only a partial port and hasn't been update in a while. This tool's name (got) is really excellent, and I would love to see something like wget written in Go. There are a few ports of well-known UNIX commands in modern languages (such as fd; a fast alternative to find). Would really be interested to see where this could go! (No pun intended).

Sometimes the Webserver provides a filename, maybe it's a good idea to use this if none is provided. If there is none, the default behavior of parsing it from the URL is probably good enough.

Hi, the latest version didn't build with the following error

user@host got % go build ./...
# golang.org/x/sys/unix
../../go/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_arm64.go:28:3: //go:linkname must refer to declared function or variable

Original(?)/related issue refers potential solution:

golang/go#51706 (comment)