mirror / wget Goto Github PK

root@build-server-8:/home/smore/WGET/wget-1.18# ./src/wget -4 -np --limit-rate=100k --timeout=1 --delete-after --tries=1 --no-dns-cache --dns-servers=8.8.4.4 http://cpp.sh --bind-address=150.1.1.111 --bind-dns-address=150.1.1.111
--2020-06-13 23:12:27-- http://cpp.sh/
Resolving cpp.sh (cpp.sh)...
661 Total count = al1=0x2193d40, al2=0x600000077
662 Total count = al1=0x2193d40, al2=0x600000077Segmentation fault (core dumped)

root@build-server-8:/home/smore/WGET/wget-1.18# !gdb
gdb ./src/wget core
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./src/wget...(no debugging symbols found)...done.
[New LWP 5232]
Core was generated by `./src/wget -4 -np --limit-rate=100k --timeout=1 --delete-after --tries=1 --no-d'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000416a3f in merge_address_lists ()
(gdb) bt
#0 0x0000000000416a3f in merge_address_lists ()
#1 0x000000000041717c in lookup_host ()
#2 0x000000000040451b in connect_to_host ()
#3 0x000000000041e1d9 in establish_connection ()
#4 0x000000000041f3a9 in gethttp ()
#5 0x000000000042180a in http_loop ()
#6 0x0000000000430795 in retrieve_url ()
#7 0x00000000004298ba in main ()
(gdb)

There are uninitialized struct address_list * in lookup_host function.
For v4 only queries on Debian Jessie system we are getting crash due to this.

Build-Steps:

./configure PKG_CONFIG_PATH="/usr/lib/x86_64-linux-gnu/pkgconfig/" GNUTLS_CFLAGS="-I/usr/include/" --without-ssl --with-cares
make clean;make

Patch has been sent with mail-header Patch: Segmentation fault in pure IPv4 wget #12

wget Read error at byte ((null)). Retrying.

WRONG GITHUB; SORRY.

Each video is saved into a separate folder which is already pre-existing with the right name. The names of all the recipes are stored in a text file.

File "fs_download_videos.py", line 93, in
file_name = wget.download(video_link_src, curr_file_path + "/")
File "/home/nsword/.local/lib/python3.8/site-packages/wget.py", line 534, in download
filename = filename_fix_existing(filename)
File "/home/nsword/.local/lib/python3.8/site-packages/wget.py", line 269, in filename_fix_existing
name,ext = filename.rsplit('.', 1)
ValueError: not enough values to unpack (expected 2, got 1)

Security Vulnerability Report

File: src/http.c

Functions: parse_content_range() and gethttp()

Vulnerability Type: Integer Overflow

Location: Lines 936, 942, 955 and 3739

Severity: High

Description:

In the parse_content_range() function, at lines 936, 942, 955, there exists a vulnerability related to an integer overflow. The vulnerability arises from the calculation of the variable num, which is assigned the value of

num = 10 * num + (*hdr - '0');

Both the multiplication and addition can lead to an integer overflow, and lead to unexpected behavior, due to the lack of validation.

Furthermore similarly to curl/curl#12983, at line 3739 of function gethttp(), the calculation of the contlen variable can also overflow:

contlen = last_byte_pos - first_byte_pos + 1;

Exploitation Scenario:

An attacker may craft a malicious request with carefully chosen values in the Content-Range header, triggering an integer overflow during the calculation of num and contlen. This could potentially lead to various security issues, such as memory corruption, buffer overflows, or unexpected behavior, depending on how the num and contlen variables is subsequently used.

Impact:

The impact of this vulnerability could be severe, potentially leading to:

Memory Corruption: If the calculated num and contlen value are used to allocate memory or perform operations such as copying data, an integer overflow could result in memory corruption, leading to crashes or arbitrary code execution.

Security Bypass: In scenarios where num and contlen value are used to enforce boundaries or permissions, an attacker may exploit the integer overflow to bypass security checks or gain unauthorized access to sensitive resources.

Denial of Service (DoS): A carefully crafted request exploiting the integer overflow could cause the application to enter an unexpected state or consume excessive resources, leading to a denial of service condition.

Recommendations:

Bounds Checking: Implement proper bounds checking to ensure that the values of num and contlen are within acceptable ranges before performing calculations.

Safe Arithmetic Operations: Consider using safer arithmetic operations or alternative calculation methods to prevent integer overflows, especially when dealing with potentially large or close-to-boundary values.

Input Validation: Validate input parameters to ensure they adhere to expected ranges and constraints before performing calculations.

Error Handling: Implement robust error handling mechanisms to gracefully handle scenarios where input parameters result in unexpected or invalid calculations.

Severity Justification:

The presence of an integer overflow vulnerability at lines 936, 942, 955 and 3739 poses a high risk to the security and stability of the application. Exploitation of this vulnerability could lead to severe consequences, including memory corruption, security bypass, or denial of service conditions.

Affected Versions:

This vulnerability affects all versions of the application that include the vulnerable parse_content_range() and gethttp() functions.

References:

OWASP Integer Overflow
CWE-190: Integer Overflow or Wraparound
CERT Secure Coding - INT32-C

Conclusion:

The presence of an integer overflow vulnerability at lines 936, 942, 955 in the parse_content_range() function and line 3739 of gethttp() poses a high risk to the security and stability of the application. It is imperative to address this vulnerability promptly by implementing appropriate bounds checking and error handling mechanisms to prevent potential exploitation and associated security risks.

Hey there!

I belong to an open source security research community, and a member (@Wninayyds) has found an issue, but doesn’t know the best way to disclose it.

If not a hassle, might you kindly add a SECURITY.md file with an email, or another contact method? GitHub recommends this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.

Thank you for your consideration, and I look forward to hearing from you!

(cc @huntr-helper)

so far, wget only support certificate file without password,
--certificate=FILE client certificate file

is it possible to add support client encrypted certificate file with password, such as like this: (as I known, curl support this feature now)
“wget --cert <certificate[:password]> Client certificate file and password” ?

this is the error code i keep getting and i dont know enough to correc this issue myself can anyone help this is what the issue looks like on my end when prompted excluding the url and other dataC:/OMITTED MORE DATA HERE o>wget https://download2.llamameta.net REST OF URL WAS HERE
Warning: wildcards not supported in HTTP.
--2024-02-11 03:23:09-- URL WAS HERE
Resolving download2.llamameta.net (download2.llamameta.net)... 3.163.115.24, 3.163.115.8, 3.163.115.112, ...
Connecting to download2.llamameta.net (download2.llamameta.net)|3.163.115.24|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-02-11 03:23:09 ERROR 403: Forbidden.

'Signature' is not recognized as an internal or external command,
operable program or batch file.
'Key-Pair-Id' is not recognized as an internal or external command,
operable program or batch file.
'Download-Request-ID' is not recognized as an internal or external command,
operable program or batch file.

I'm doing a wget --mirror operation on a fairly long URL (including the host folder ~ 242 chars) into another folder via --outdir adding another couple of chars.

This runs into the length limitation at

However I think that limitation is to aggressive/sensitive: It takes the entire quoted path (i.e. at least the 242 chars) and compares it against pathconf(..., _PC_NAME_MAX) when that is available. See

Note that the _PC_NAME_MAX returns the maximum length for a filename while PATH_MAX is the maximum length of a path. The former is 255 while the latter is 4096 on "usual" Linux systems. So the 2 code paths are not nearly identical!

Given that the "chomp buffer" size (19) is additionally subtracted any output path is truncated at 255-19=236 chars which isn't enough for use-cases such as mine mirroring a larger hierarchy of folders (files in depth of 10 folders)

page exists, but not downloaded... i tried to put headers but not success. what to do?
is that a bug?
I am downloading wget2...

LANG=EN; wget --random-wait -4 --max-redirect 100 --level 100 --limit-rate=30K -r -nc -p -U Mozilla 'www.literotica.com/average-joe-ch-01?page=2'

URL transformed to HTTPS due to an HSTS policy
--2022-07-11 13:49:09-- https://www.literotica.com/average-joe-ch-01?page=2
Resolving www.literotica.com (www.literotica.com)... 216.150.65.200, 216.150.65.190
Connecting to www.literotica.com (www.literotica.com)|216.150.65.200|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2022-07-11 13:49:10 ERROR 404: Not Found.

More often than not I try recursively downloading a webpage using wget, only to have it download a single index.html.gz then stop. Obviously wget can't read gzipped files so it fails to find any links for recursive downloading... I ended up using this wget fork that was last updated 10 years ago and it works fine, however I find it odd that such a basic feature never made it into mainline wget.

Please add a feature for automatically detecting and uncompressing gzipped webpages before crawling them.

Got the fix from https://lists.freedesktop.org/archives/modemmanager-devel/2017-February/003946.html, by changing ./configure.ac as below:

-AM_INIT_AUTOMAKE([subdir-objects dist-lzip])
+AM_INIT_AUTOMAKE([subdir-objects dist-lzip foreign])

Commit 32e26dc causes bootstrap to fail on OS X with:

• invoke gl_INIT in ./configure.ac.
  Creating lib/unicase/special-casing-table.h for gperf < 3.1
  sed: 1: "lib/unicase/special-cas ...": extra characters at the end of l command
  ./bootstrap: bootstrap_post_import_hook failed

Per the discussion at
https://stackoverflow.com/questions/5694228/sed-in-place-flag-that-works-both-on-mac-bsd-and-linux, the -i option is not part of POSIX Sed, and is apparently implemented differently on BSD/Apple OS X/macOS.

Year ago it worked fine on my previous pc windows 10. Now it saves some gibberish as filenames.
--restrict-file-names do nothing
--local-encoding=UTF-8 no works
unrecognised option `--local-encoding=UTF-8'

I'm using wget -m -np -r -R "index.html* http://apache_listing.com/folder/folder_in_cyrillic/