Spider framework about cfadmin

[CandyMi]

Hi, @gcclua
You can try using httpc:multi_request like this:

local Log = require("logging"):new()
local hc = http:new {}
local ok, response_array = hc:multi_request {
  [1] = {
    domain = "http/https://yourdomain/path",
    method = "get/post/json/file",
    headers = {{key1 = value1, key2 = value2, ...}},
    files/body = "your body"
  },
  [2] = { },
  [...] = { }
}
Log:DEBUG(ok, response_array)

This is a concurrent request method that will open multiple coroutines to concurrently request array items, and the return value is also in the array.

All usage examples can be found in script/test_httpc.lua. :)

[gcclua]

thanks , I will try it , and i have a new idea , copy multi instance of cf , and write logic circle inside main.lua

while true do 
    -- get spider new task from the center cf 
    local hc = http:new {} 
    local SpiderTaskList = hc:get(CenterUrl) 
    local SpiderTaskSet = ParseAndFetchURLFrom(SpiderTaskList) 
    if IsEmpty(SpiderTaskSet) == false then 
        -- do each task
        for k,v in pairs(SpiderTaskSet) do 
            local code, body = hc:get(v) 
            print(code, body) 
        end 
    else 
        Sleep(2) 
    end 
end

We will get distribute spider ? but all logic will inside core_sys_init, before core_sys_run,
potential problems?

[CandyMi]

Good idea, there is no problem with this use.

You may need to know that hc.class should close the connection when returning nil. : )

[gcclua]

OK, thank you

[CandyMi]

@gcclua
Add:
hc.class needs to be called actively when it is used: hc:close() method, otherwise it may cause fd to leak(Even if you actively call collectgarbage , it will not release fd).
Don't forget it. : )

[gcclua]

@gcclua
Add:
hc.class needs to be called actively when it is used: hc:close() method, otherwise it may cause fd to leak(Even if you actively call collectgarbage , it will not release fd).
Don't forget it. : )

Way1:

while true do 
    -- get spider new task from the center cf 
    local hc = http:new {} 
    local SpiderTaskList = hc:get(CenterUrl) 
    local SpiderTaskSet = ParseAndFetchURLFrom(SpiderTaskList) 
    if IsEmpty(SpiderTaskSet) == false then 
        -- do each task
        for k,v in pairs(SpiderTaskSet) do 
            local code, body = hc:get(v) 
            print(code, body) 
        end 
    else 
        Sleep(2) 
    end 
    hc:close() --Close fd 
end

Way2:

while true do 
    -- get spider new task from the center cf 
    local hc = http:new {} 
    local SpiderTaskList = hc:get(CenterUrl) 
    local SpiderTaskSet = ParseAndFetchURLFrom(SpiderTaskList) 
    if IsEmpty(SpiderTaskSet) == false then 
        -- do each task
        for k,v in pairs(SpiderTaskSet) do 
            local hc = http:new {} 
            local code, body = hc:get(v) 
            print(code, body) 
            hc:close() --Close fd 
        end 
    else 
        Sleep(2) 
    end 
    hc:close() --Close fd 
end

which is correct way?

[CandyMi]

According to how your code is written, you can write it like this:

local httpc = require "httpc"

local http_cls = require "httpc.class"

while true do 
    -- get spider new task from the center cf 
    local SpiderTaskList = httpc.get(CenterUrl) 
    local SpiderTaskSet = ParseAndFetchURLFrom(SpiderTaskList) 
    if noEmpty(SpiderTaskSet) then 
      local hc = http_cls:new {}  -- new httpc class
      for k,v in pairs(SpiderTaskSet) do 
          local hc = http_cls:new {} 
          local code, body = hc:get(v) 
          print(code, body) 
      end
      hc:close() -- close httpc 
    else 
      Sleep(2) 
    end 
end

And the internal domain must be guaranteed to be the same.

But using multi_request doesn't have this limitation, and you don't have to actively call the hc:close method. like this:

local httpc = require "httpc"

while true do 
  -- get spider new task from the center cf 
  local SpiderTaskList = httpc.get(CenterUrl) 
  local SpiderTaskSet = ParseAndFetchURLFrom(SpiderTaskList) 
  if noEmpty(SpiderTaskSet) then
    local requests = {}
    for k,v in pairs(SpiderTaskSet) do
      requests[#requests + 1] = { domain = v }
    end
    local ok, response_array = httpc.multi_request(requests)
    print(ok, response_array)
  else
    sleep(2)
  end
end

Because the internal will automatically close all fd. (Lua gc will automatically recycle the object) after each request is completed.

This is the difference between them.

[gcclua]

great, httpc.multi_request is simple to use, I want the result will return one by one and with timeout limit , timeout as the task fail flag, in one thread so difficult to implement, has cf some help ??

[CandyMi]

httpc.multi_request has a timeout parameter that controls the maximum timeout for each request to time out. like this:

httpc.multi_request {
  {
    domain = "your domain",
    timeout = 15
  }
}

You can refer to this.

[gcclua]

OK, I will use httpc.multi_request

[CandyMi]

@gcclua
Very happy to help you! : )
I suggest you use the latest version I just released, which fixes some bugs in the httpc library.