Mojo::UserAgent::Role::Queued - A role to process non-blocking requests in a rate-limiting queue.
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new->with_roles('+Queued');
$ua->max_redirects(3);
$ua->max_active(5); # process up to 5 requests at a time
for my $url (@big_list_of_urls) {
$ua->get($url, sub {
my ($ua, $tx) = @_;
if (! $tx->error) {
say "Page at $url is titled: ",
$tx->res->dom->at('title')->text;
}
});
};
Mojo::IOLoop->start unless Mojo::IOLoop->is_running;
# works with promises, too:
my @p = map {
$ua->get_p($_)->then(sub { pop->res->dom->at('title')->text })
->catch(sub { say "Error: ", @_ })
} @big_list_of_urls;
Mojo::Promise->all(@p)->wait;
Mojo::UserAgent::Role::Queued manages all non-blocking requests made through Mojo::UserAgent in a queue to limit the number of simultaneous requests.
Mojo::UserAgent can make multiple concurrent non-blocking HTTP requests using Mojo's event loop, but because there is only a single process handling all of them, you must take care to limit the number of simultaneous requests you make.
Some discussion of this issue is available here http://blogs.perl.org/users/stas/2013/01/web-scraping-with-modern-perl-part-1.html and in Joel Berger's answer here: http://stackoverflow.com/questions/15152633/perl-mojo-and-json-for-simultaneous-requests.
Mojo::UserAgent::Role::Queued tries to generalize the practice of managing a large number of requests using a queue, by embedding the queue inside Mojo::UserAgent itself.
Mojo::UserAgent::Role::Queued has the following attributes:
$ua->max_active(5); # execute no more than 5 transactions at a time.
print "Execute no more than ", $ua->max_active, " concurrent transactions"
Parameter controlling the maximum number of transactions that can be active at the same time.
Mojo::UserAgent::Role::Queued adds the following event to those emitted by Mojo::UserAgent:
$ua->on(queue_empty => sub { my ($ua) = @_; .... })
Emitted when the queue has been emptied of all pending jobs. In previous releases, this event was called stop_queue (this is a breaking change).
This software is Copyright (c) 2017-2019 by Dotan Dimet [email protected].
This library is free software; you can redistribute it and/or modify it under the terms of the Artistic License version 2.0.
Dotan Dimet [email protected]
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent