Comments (21)
The conclusion I've come to is that after one of curl_multi's sub-easy tasks finishes, calling curl_multi_remove_handle() or curl_easy_cleanup() destroys the proxy settings of the other sub-easy tasks that are still running
No, there is no such thing. That's a wild guess that cannot be correct. Also, I could reproduce this problem even with all the curl_multi_remove_handle() and curl_easy_cleanup() calls removed from the reproducer code.
In my testing in reproducing this issue, doing SOCKS5 seems shaky in 8.6.0 as well, compared to not using the proxy. Thus, I don't think 8.6.0 had this working correctly either, only that perhaps 8.7.x made it more visible.
I have not yet figured out what the problem is. The symptom is timed out transfers that otherwise would not timeout.
from curl.
I'm glad you reproduced the problem.
Some additional notes:
The "8.6.0-" version runs the same script for a long time without a similar problem. Or the chances of it happening are very low, or maybe it happened once but was assumed by me to be a problem with the VPS.
I tried to use https proxy and the problem didn't improve.
from curl.
I suspect I don't reproduce the problem you report. If you can see the same problem with HTTPS proxy, then you see something completely different. They use very different code paths.
Can you clarify exactly how the transfers fail? Does transfer 2-6 always fail once the first one is completed?
I figured out the reason why some of my proxy connections timed out: https://opensource.org
does not work over IPv6. When connecting to it over a SOCKS proxy, curl resolves the host name and uses only a single connect attempt to it with IPv6, which fails. It fails also without any proxy.
But when connecting directly to a host, without a proxy, curl does happy eyeballs which tries both IPv6 and IPv4 and thus the failed IPv6 attempt is not visible.
from curl.
Test results, there will be 10-20% normal downloads.
But my test download was a small 1.3M file. This could also be my test method I problem. I initially created only one task, and added the other 5 after getting the "content-length". In other words, task-1, using the request header "Range=bytes=0-", I would disconnect task-1 in the callback for received data based on the size of the received data. This means that task-1 is passively disconnected.
Problematic tests, 80% are 5 remaining tasks caught in a loop. Other times, sometimes it's 2 caught in a loop, sometimes it's 3...
from curl.
I found a way to download 100% properly. Task-1, even if it downloads the allocated data size, it doesn't disconnect and still downloads all the data. The other tasks then don't get stuck in a loop. That means the code that triggers this is, task-1
write_callback_function(){
if this.receive_size >= this.task_size
return 0
}
from curl.
Task-1, even if it downloads the allocated data size, it doesn't disconnect and still downloads all the data.
Come again? What do you mean that you do or don't do with the first transfer?
from curl.
According to the example of 10-at-a-time.html, if you use 6 chunks to download, the main thread needs to create these 6 tasks at once after getting the "content-length" (assuming the file size is 6):
task-1 Range=bytes=0-1
this_1.task.size =1
...
task-6 Range=bytes=5-6
this_6.task.size =1
Note that here task-1 has a normal task size.
The way I used it, I skipped the process of getting the "content-length" in the main thread, and created only one task, with "Range=bytes=0-" set in task-1.
So the event that creates the other 5 tasks happens in task-1 headerfunction
task-1 headerfunction => get content-length
this_1.task.size = content-length / 6
loop 5
Creation of 5 other tasks
This means that task-1 will download all the data (6) and I have to actively terminate it after it finishes downloading the specified content
write_callback_function(){
if this.receive_size >= this.task_size
return 0
}
And I found that task-1 write_callback_function() return 0 causes the other 5 tasks to get stuck in a loop. So the way to download 100% normally is to not actively end it and let task-1 download all the data (6).
from curl.
10-at-a-time.c is a single-threaded example, so referring to its thread as "the main thread" is a bit incongruous since there is no other thread. It sounds like you've converted that example into a multithreaded program. Improper use of multithreading in libcurl is a frequent source of problems. Did you abide by everything in https://curl.se/libcurl/c/threadsafe.html when you converted the example?
from curl.
It was my poor choice of words. I'm not using multithreading.
You can interpret the improper "threads" here as:
main thread = multi loop()
sub thread = task_callback()
from curl.
from curl.
I can't understand what "limitations" you're talking about, but I'm not operating the curl api in write_callback_function() other than using RtlMoveMemory() to read the data.
There are only two methods for "return":
return 0
return size * memb
Regarding task-1's header_callback_function, it also doesn't manipulate the curl api, it just creates 5 task objects with task information. In the multi loop{}, these objects are added to multi, consistent with '10-at-a-time.c'.
from curl.
from curl.
Unless someone explicitly says they need the AutoHotkey code, I can't provide it.
from curl.
I've summarized the current problem as follows:
In multi's multitasking download (using the same proxy), if one task aborts the connection with "return 0" at write_callback(), the proxy for the other tasks fails.
from curl.
I think maybe the odd terms in use makes it hard to speak the same language.
curl performs transfers. You can do multiple transfers with it. They are not multi-tasking.
By your description it sounds as if you add new transfers from within a callback, which sounds odd. libcurl tries to forbid that. So are you?
And again: you say repeatedly that "the other transfers fail", but I want to know more specifically and exactly how they fail. Details!
You say you do six transfers. Can you describe exactly what each transfer does?
from curl.
I searched all my statements and did not find the word "transfers".
If you look carefully, you will find it:
"
I'm not operating the curl api in write_callback_function()
Regarding task-1's header_callback_function, it also doesn't manipulate the curl api
"
from curl.
I don't know C at all, but I modified 10-at-a-time.c as a schematic. If you're still confused, close this issue, it's not worth the effort.
static size_t write_cb(char *data, size_t n, size_t l, void *userp)
{
// this = current task
/* take care of the data here, ignored in this example */
(void)data;
(void)userp;
this.receive_size += n * l
if this.receive_size >= this.task_size // Only task 1 requires this judgment
return 0
return n * l;
}
static task_list=[]
static size_t write_head(char *data, size_t n, size_t l, void *userp)
{
// this = task.1
if str=="\r\n"
&& if head has content-length
&& if head has content-range
{
this.task_size := content-length/6
offset = content-length/6
loop 5 //Create 5 objects containing task information
{
task_list.push({url: this.url, Range:"Range:bytes=%offset%-%offset+(content-length/6)%"})
}
}
/* take care of the data here, ignored in this example */
(void)data;
(void)userp;
return n * l;
}
int main(void)
{
CURLM *cm;
CURLMsg *msg;
unsigned int transfers = 0;
int msgs_left = -1;
int left = 0;
curl_global_init(CURL_GLOBAL_ALL);
cm = curl_multi_init();
/* Limit the amount of simultaneous connections curl should allow: */
curl_multi_setopt(cm, CURLMOPT_MAXCONNECTS, (long)MAX_PARALLEL);
// add task.1
CURL *eh = curl_easy_init();
curl_easy_setopt(eh, CURLOPT_WRITEFUNCTION, write_cb);
curl_easy_setopt(eh, CURLOPT_HEADERFUNCTION, write_head);
curl_easy_setopt(eh, CURLOPT_URL, "https://dl.google.com/drive-file-stream/GoogleDriveSetup.exe ");
curl_slist_append(xx,"Range=bytes=0-")
curl_multi_add_handle(cm, eh);
(*left)++;
do
{
int still_alive = 1;
curl_multi_perform(cm, &still_alive);
while ((msg = curl_multi_info_read(cm, &msgs_left)))
{
if (msg->msg == CURLMSG_DONE)
{
char *url;
CURL *e = msg->easy_handle;
curl_easy_getinfo(msg->easy_handle, CURLINFO_PRIVATE, &url);
fprintf(stderr, "R: %d - %s <%s>\n",
msg->data.result, curl_easy_strerror(msg->data.result), url);
curl_multi_remove_handle(cm, e);
curl_easy_cleanup(e);
left--;
}
else
{
fprintf(stderr, "E: CURLMsg (%d)\n", msg->msg);
}
if task_list
for x in task_list
{
CURL *eh = curl_easy_init();
curl_easy_setopt(eh, CURLOPT_WRITEFUNCTION, write_cb);
curl_easy_setopt(eh, CURLOPT_URL, x.url);
curl_slist_append(xx,x.Range)
curl_multi_add_handle(cm, eh);
(*left)++;
}
}
if (left)
curl_multi_wait(cm, NULL, 0, 1000, NULL);
} while (left);
curl_multi_cleanup(cm);
curl_global_cleanup();
return EXIT_SUCCESS;
}
from curl.
I searched all my statements and did not find the word "transfers".
That's exactly the problem: you refer to them as tasks but they are not tasks. They are transfers.
Your code refers to a this
as "current task" - but the same callback function is used for all transfers, so what is "the current task" ?
from curl.
AutoHotkey can attach data when creating callbacks." The "this" variable is just to record the task_size and nothing else. Like this new example
static size_t write_cb(this, char *data, size_t n, size_t l, void *userp)
{
/* take care of the data here, ignored in this example */
(void)data;
(void)userp;
this.receive_size += n * l
if this.receive_size >= this.task_size // Only task 1 requires this judgment
return 0
return n * l;
}
static task_list=[]
static size_t write_head(this, char *data, size_t n, size_t l, void *userp)
{
if str=="\r\n"
&& if head has content-length
&& if head has content-range
{
this.task_size := content-length/6
offset = content-length/6
loop 5 //Create 5 objects containing task information
{
task_list.push({url: this.url, task_size: content-length/6, Range:"Range:bytes=%offset%-%offset+(content-length/6)%"})
}
}
/* take care of the data here, ignored in this example */
(void)data;
(void)userp;
return n * l;
}
int main(void)
{
CURLM *cm;
CURLMsg *msg;
unsigned int transfers = 0;
int msgs_left = -1;
int left = 0;
curl_global_init(CURL_GLOBAL_ALL);
cm = curl_multi_init();
/* Limit the amount of simultaneous connections curl should allow: */
curl_multi_setopt(cm, CURLMOPT_MAXCONNECTS, (long)MAX_PARALLEL);
// add task.1
CURL *eh = curl_easy_init();
t_1 = {eh: eh, task_size:0, url: "https://dl.google.com/drive-file-stream/GoogleDriveSetup.exe"}
curl_easy_setopt(eh, CURLOPT_WRITEFUNCTION, write_cb.bind(t_1));
curl_easy_setopt(eh, CURLOPT_HEADERFUNCTION, write_head.bind(t_1));
curl_easy_setopt(eh, CURLOPT_URL, "https://dl.google.com/drive-file-stream/GoogleDriveSetup.exe");
curl_slist_append(xx,"Range=bytes=0-")
curl_multi_add_handle(cm, eh);
(*left)++;
do
{
int still_alive = 1;
curl_multi_perform(cm, &still_alive);
while ((msg = curl_multi_info_read(cm, &msgs_left)))
{
if (msg->msg == CURLMSG_DONE)
{
char *url;
CURL *e = msg->easy_handle;
curl_easy_getinfo(msg->easy_handle, CURLINFO_PRIVATE, &url);
fprintf(stderr, "R: %d - %s <%s>\n",
msg->data.result, curl_easy_strerror(msg->data.result), url);
curl_multi_remove_handle(cm, e);
curl_easy_cleanup(e);
left--;
}
else
{
fprintf(stderr, "E: CURLMsg (%d)\n", msg->msg);
}
if task_list
for x in task_list
{
CURL *eh = curl_easy_init();
curl_easy_setopt(eh, CURLOPT_WRITEFUNCTION, write_cb.bind(x));
curl_easy_setopt(eh, CURLOPT_URL, x.url);
curl_slist_append(xx,x.Range)
curl_multi_add_handle(cm, eh);
(*left)++;
}
}
if (left)
curl_multi_wait(cm, NULL, 0, 1000, NULL);
} while (left);
curl_multi_cleanup(cm);
curl_global_cleanup();
return EXIT_SUCCESS;
}
from curl.
I'm sorry but none of this brings me any closer to understanding the issue. I don't understand your program. I don't even know how the transfers fail. I can't reproduce the problem.
from curl.
Ok, thanks for the reply. I'm sure you didn't forget to use the SOCKS5 proxy in your test and you must have triggered the "return 0" in your test. I don't have any more information to provide, so it looks like I'm giving meaningless issues, so please ignore the problem.
from curl.
Related Issues (20)
- Change in behaviour for duplicate "Transfer-Encoding = chunked" headers HOT 3
- Please update tiny-curl HOT 1
- test 345 fails on AppVeyor HOT 2
- Unable to upload file with curl when --limit-rate is set to specific value HOT 3
- GitHub actions major outage/problem HOT 2
- `git clone` hangs with curl-8.7.1 HOT 7
- wolfSSL HTTP/3 peer verification is lacking HOT 4
- Ambiguous error message when decompressing brotli stream HOT 2
- CURL_VERSION_GSSAPI is not display on curl tool. HOT 4
- Bug in Debian (curl-8.7.1) leads to Failed writing received data to disk/application HOT 10
- Error 451 when transfering certain files using FTPS\TLS1.3 HOT 1
- Fail during configure if perl is missing but required HOT 5
- Windows DNS resolution: Curl crash when GetAddrInfoExW callback invoked on shutdown HOT 7
- CURLOPT_SSLVERSION: C++20 deprecation warning when values of two enums are combined HOT 1
- Sign the un-preprocessed `git archive` .tar.gz
- CMake: Fail to looking for symbols in BoringSSL HOT 6
- Compiling "threaded-ssl.c" with PThreads4W
- americanas.com.br immediately sends RST_STREAM HOT 16
- Missed to modify member variable name HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from curl.