lpeg.common_log_format can not match nginx access log about lua_sandbox_extensions

[trink]

Why is the apache grammar builder being used? Also the quoted upstream_addr will give you a problem.

[liuyangc3]

sorry it is a typo, and unquoted upstream_addr it works, so I did a little dig
if logformat is "$upstream_addr - with a single quote

local log_format = '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$upstream_addr'
local grammar = clf.build_nginx_grammar(log_format)
local data = '10.100.27.90 - - [22/Dec/2016:14:32:22 +0800] "GET /index HTTP/1.1" 200 10057 "http://rid.p.nxin.com/login.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36" "-" "10.211.14.10:8080"'
local fields = grammar:match(data)
for _,u in pairs(fields.upstream_addr) do
    print(u)
end

console
----
10.211.14.10:8080"

can a upstream_addr contains a quote?

[liuyangc3]

my fix, this will get upstream_addr with quote

local l = require "lpeg"
l.locale(l)
local ip = require "lpeg.ip_address"


--[[ nginx upstream server config
config
    server unix:!"$%&'()*+,-./:<>=?@[\]^_`\;\{|\}~abc.123你好;
    server localhost;
    server [::1];
output in access log
    unix:!\x22$%&'()*+,-./:<>=?@[\x5C]^_`\x5C;\x5C{|\x5C}~abc.123\xE4\xBD\xA0\xE5\xA5\xBD
    127.0.0.1:80
    [::1]:80
]]--

-- 0 to 65535
local port = l.P"6553" * l.R"05"
           + l.P"655" * l.R"02" * l.digit
           + l.P"65" * l.R"04" * l.digit * l.digit
           + l.P"6" * l.R"04" * l.digit * l.digit * l.digit
           + l.R"15" * l.digit * l.digit * l.digit * l.digit
           + l.R"19" * l.digit^-3

local utf8 = l.P"\\x" * (l.R"09" + l.R"AF") * (l.R"09" + l.R"AF")
local safe_char = l.P"\\x5C" * l.S";{}" + l.P"\\x22" + l.P"\\x5C" + (l.R"\20\126" - l.S"\"\\;{}") + utf8
local unix_sock = l.P"unix:" * safe_char^1
local nginx_upstream_sep = ", "
local nginx_upstream_addr = l.C((unix_sock + (l.P"[" * ip.v6 * l.P"]" + ip.v4) * l.P":" * port) - l.S(nginx_upstream_sep))



-- test port
for i=1,65535 do
    local s = tostring(i)
    local m = l.C(port):match(s)
    assert(m == s, s .. "not match" .. m)
end

-- test nginx_upstream_addr
print(nginx_upstream_addr:match(
    "unix:!\x22$%&'()*+,-./:<>=?@[\\x5C]^_`\\x5C;\\x5C{|\\x5C}~abc.123\\xE4\\xBD\\xA0\\xE5\\xA5\\xBD"))
print(nginx_upstream_addr:match("192.168.0.1:80"))
print(nginx_upstream_addr:match("[2002:0:0:0:0:0:65e2:a259]:8888"))

I can pull a PR if needed

[liuyangc3]

https://github.com/nxintech/Charlie/blob/master/logs/common_log_format.lua.patch