High memory consumption and fetching random proxy from list problem about php-curl-class

[zachborboa]

The issue with high memory use should be resolved by using a pattern like this:
https://github.com/php-curl-class/php-curl-class/blob/master/examples/multi_curl_add_curl_from_url_list.php

To use a unique random proxy, something like this could work:
https://github.com/php-curl-class/php-curl-class/blob/master/examples/multi_curl_get_with_new_random_unique_proxy.php

[enseitankado]

The idea of destroying the once used proxy inside the beforeSend procedure is nice.
Unfortunately I did this by modifying the library itself, which would create additional work in updating the library. Now I'm familiar with the beforeSend function. Thanks for the illustrative sample program.

Funny thing is, I realized that I persisted in my question about memory consumption. When I looked a little more carefully, I decided that the example you gave worked correctly.

I see you are a few steps ahead of me. Thank you for your patience.

[zachborboa]

Hi. You can add additional requests after starting to reduce memory consumption. Take a look at this example:
https://github.com/php-curl-class/php-curl-class/blob/master/examples/multi_curl_add_curl_from_url_list.php

When using proxies, a desired proxy can be set on the individual request and a random proxy won't override the existing proxy:

php-curl-class/src/Curl/MultiCurl.php

Lines 1277 to 1280 in 3ff53bf

	if (is_array($this->proxies) && $curl->getOpt(CURLOPT_PROXY) === null) {
	$random_proxy = ArrayUtil::arrayRandom($this->proxies);
	$curl->setProxy($random_proxy);
	}

[enseitankado]

Hi,

As I mentioned at the beginning; Let's say I define 100 proxies and 100 get. The library does this randomly when choosing the proxy to use for get requests. This has two downsides. First, the same proxy can be selected again, and the other can include a proxy that has never been used when 100 requests are complete. An optional key is required to use the library random proxy.

Multithread requires a certain number of get requests to start at the same time. However, the code in the example moves on to the next request after one request is complete. So the example is sequential, not multithreaded.

 if (is_array($this->proxies) && $curl->getOpt(CURLOPT_PROXY) === null) { 
     $random_proxy = ArrayUtil::arrayRandom($this->proxies); 
     $curl->setProxy($random_proxy); 
 }