peakwinter / python-nginx Goto Github PK

Thanks for this tool, it's great!
Just one question, is there any chance of an example of how to load a file and edit a key:value pair in the configuration?

Version: 1.4

Exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/nginx/__init__.py", line 1, in <module>
    from main import *
ImportError: No module named 'main'

Possible Fix:
from .main import *

If it finds:

 location / {
 return 301 $scheme://$host:$server_port${request_uri}bitbucket/;
 }

raises error:

   File "app.py", line 56, in readConfs
     c   = nginx.loads( txt )
   File "/usr/local/lib/python2.7/dist-packages/nginx.py", line 451, in loads
     if isinstance(lopen[0], Server):
 IndexError: list index out of range

Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import nginx
>>> nginx.loadf('/tmp/nginx.conf')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/nginx.py", line 492, in loadf
    return load(f)
  File "/Library/Python/2.7/site-packages/nginx.py", line 482, in load
    return loads(fobj.read())
  File "/Library/Python/2.7/site-packages/nginx.py", line 454, in loads
    if isinstance(lopen[0], Server):
IndexError: list index out of range

hi， how to use c.Remove() Method, can you give me a Example， thks！！！

After saving all comment string is doubled in config file.

"{}{}".format(x,y)
maybe should
"{0}{1}".format(x,y)

Hello!

I'm using 1.5.5
It failed when parsing this file

user  nginx;

Here is the error: nginx.ParseError: Config syntax, missing ';' at index: 12

But works fine when there is a new line at the end:

user  nginx;

Hello,

First, thanks for this project.
I submit the issue here, due to a connection issue with your Gitlab instance.

Context
The '}' char is inside a key/value state, inside a location of an existing config file

Steps to reproduce

1. Create a configuration file

$ cat /tmp/foo.conf
server {
    listen 80;
    location / {
        fastcgi_param MY_KEY "W*}4";
    }
}

2. Try to parse this file

 python-nginx  ~  22:38:31  python
Python 2.7.10 (default, Oct 14 2015, 16:09:02) 
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import nginx
>>> nginx.loadf('/tmp/foo.conf')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/alexandre/.local/share/virtualenvs/python-nginx/local/lib/python2.7/site-packages/nginx.py", line 487, in loadf
    return load(f)
  File "/home/alexandre/.local/share/virtualenvs/python-nginx/local/lib/python2.7/site-packages/nginx.py", line 477, in load
    return loads(fobj.read())
  File "/home/alexandre/.local/share/virtualenvs/python-nginx/local/lib/python2.7/site-packages/nginx.py", line 449, in loads
    if isinstance(lopen[0], Server):
IndexError: list index out of range
>>> 

Expected result
Parser see the '}' as a part on key value statement and not the end of a block

Version: 1.5.1

Nginx Config Key/Value:
add_header X-XSS-Protection "1;mode-block";

Parse Output:
{'conf': [{'add_header': 'X-XSS-Protection "1'}, {'mode-block"': ''}]}

How to reproduce:

1. Create nginx.conf with

server {
    listen 80;
    server_name test.example.com;

    location ~ "^/(test|[0-9a-zA-Z]{6})$" {
        if ($query_string ~ pid=(111)) {
            return 403;
        }

        proxy_pass http://127.0.0.1:81;
    }
}

2. Create a test.py script

import nginx
import os

conf_ast = nginx.loadf("./nginx.conf")

3. Run python test.py.

Expected result: config is successfully parsed.
Actual result:

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    conf_ast = nginx.loadf("./nginx.conf")
  File "/home/user/Downloads/tmp/parsing-debug/lib/python3.8/site-packages/nginx.py", line 556, in loadf
    return load(f)
  File "/home/user/Downloads/tmp/parsing-debug/lib/python3.8/site-packages/nginx.py", line 546, in load
    return loads(fobj.read())
  File "/home/user/Downloads/tmp/parsing-debug/lib/python3.8/site-packages/nginx.py", line 500, in loads
    if isinstance(lopen[0], Container):
IndexError: list index out of range

This happens because this regex https://github.com/peakwinter/python-nginx/blob/1.5.4/nginx.py#L447 incorrectly considers curly braces as a starting point of a location block.
I can think of a couple of ways to fix this like adding + so it would be ^\s*location\s*([^;]*?)\s+{ instead of ^\s*location\s*([^;]*?)\s*{ or adding \n at the end. But I'm not sure which one is better.

autoprov.py
#############
import nginx
c = nginx.loadf(confLocation)

Installed using pyinstaller

pyinstaller --onefile ../src/autoprov/autoprov.py

2556 INFO: Analyzing XXX/pybuild/autoprov.py
2574 INFO: Processing module hooks...
2574 INFO: Loading module hook "hook-encodings.py"...
2625 INFO: Loading module hook "hook-pydoc.py"...
2626 INFO: Loading module hook "hook-xml.py"...
2814 INFO: Looking for ctypes DLLs
2814 INFO: Analyzing run-time hooks ...
2819 INFO: Looking for dynamic libraries
3070 INFO: Looking for eggs
3070 INFO: Using Python library /lib64/libpython3.6m.so.1.0
3075 INFO: Warnings written to XXX/pybuild/build/autoprov/warn-autoprov.txt
3096 INFO: Graph cross-reference written to XXX/build/autoprov/xref-autoprov.html
3108 INFO: checking PYZ
3115 INFO: Building because toc changed
3115 INFO: Building PYZ (ZlibArchive) XXX/pybuild/build/autoprov/PYZ-00.pyz
3408 INFO: Building PYZ (ZlibArchive) XXX/pybuild/build/autoprov/PYZ-00.pyz completed successfully.
3410 INFO: checking PKG
3415 INFO: Building because toc changed
3415 INFO: Building PKG (CArchive) PKG-00.pkg
5589 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
5592 INFO: Bootloader /usr/local/lib/python3.6/site-packages/PyInstaller/bootloader/Linux-64bit/run
5592 INFO: checking EXE
5598 INFO: Building because toc changed
5598 INFO: Building EXE from EXE-00.toc
5600 INFO: Appending archive to ELF section in EXE XXX/pybuild/dist/autoprov
5633 INFO: Building EXE from EXE-00.toc completed successfully.

On runnng the exe

[temp@localhost rpmfiles]$ sudo XXX/pybuild/dist/autoprov
Traceback (most recent call last):
File "autoprov.py", line 1, in
ModuleNotFoundError: No module named 'nginx'
[15311] Failed to execute script autoprov
[virsec@localhost rpmfiles]$

uname -a
Linux localhost.localdomain 3.10.0-1160.24.1.el7.x86_64 #1 SMP Thu Apr 8 19:51:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

import nginx

TESTBLOCK="""
upstream test0 {
    ip_hash;
    server 127.0.0.1:8080;
    keepalive 16; 
}
upstream test1{
    server 127.0.0.2:8080;
    keepalive 16; 
}
upstream test2
{
    server 127.0.0.3:8080;
    keepalive 16; 
}

server {
    listen       80;
    server_name  example.com;

    location = / 
    {       
        root html;
    }   
}
"""


u1 = nginx.loads(TESTBLOCK)
print u1.children[0].value
print u1.children[1].value
print u1.children[2].value

nginx.dumpf(u1, '/tmp/example.conf')

For upstream test0 , ip_hash lost.
For upstream test1 and test2 , upstream name changed, result is test .
For location section, raise error:

Traceback (most recent call last):
  File "example.py", line 32, in <module>
    u1 = nginx.loads(TESTBLOCK)
  File "/usr/local/lib/python2.7/dist-packages/nginx.py", line 454, in loads
    if isinstance(lopen[0], Server):
IndexError: list index out of range

map $aa $bb { }

gives the error:

File "C:\Python27\lib\site-packages\nginx.py", line 515, in loadf
    return load(f)
  File "C:\Python27\lib\site-packages\nginx.py", line 505, in load
    return loads(fobj.read())
  File "C:\Python27\lib\site-packages\nginx.py", line 454, in loads
    if isinstance(lopen[0], Container):
IndexError: list index out of range

Exception raise if in configuration we have something like this (one instruction comment out):
location /media {
#alias /home/profile/project/media;
}
This is because of to_eval is empty. We can check this before
kname, kval = re.match(key_regex, to_eval).group(1, 2)

Directive documentation

Python version: 3.5.1
Python-nginx version: 1.2
OS: Linux (Ubuntu 16.04; x64)

I've loaded and edited the configuration using python-nginx. During this operation, the internal directive disappeared from the config.

Before:

location ~ /restapi/v1.0/account/[0-9]|~+/extension/[0-9]|~+/profile-image/[0-9]+x[0-9]+$ {
    internal;
    error_page 401 404 =200 /uas/generate?$args;
    proxy_intercept_errors on;
    proxy_set_header Authorization "SOME_TOKEN";
    proxy_pass http://intapi-rndvi3ams.lab.nordigy.ru:80;
}

location = /uas/generate {
    internal;
    proxy_pass http://127.0.0.1:8901;
}

After:

location ~ /restapi/v1.0/account/[0-9]|~+/extension/[0-9]|~+/profile-image/[0-9]+x[0-9]+$ {

    error_page 401 404 =200 /uas/generate?$args;
    proxy_intercept_errors on;
    proxy_set_header Authorization "SOME_TOKEN";
    proxy_pass http://intapi-rndvi3ams.lab.nordigy.ru:80;
}
location = /uas/generate {

    proxy_pass http://127.0.0.1:8901;
}

The following line (in my case in the beginning of the line:

include conf.d/pre/*.cfg;

gives the following error:

   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
     return self.view_functions[rule.endpoint](**req.view_args)
   File "app.py", line 89, in hostnames
     readConfs()
   File "app.py", line 27, in readConfs
     c = nginx.loadf( filepath )
   File "/usr/local/lib/python2.7/dist-packages/nginx.py", line 489, in loadf
     return load(f)
   File "/usr/local/lib/python2.7/dist-packages/nginx.py", line 479, in load
     return loads(fobj.read())
   File "/usr/local/lib/python2.7/dist-packages/nginx.py", line 447, in loads
     lopen[0].add(k)

This is my test code, check_http_send is a directive of popular nginx module nginx_upstream_check_module

import nginx

UPSTREAM_BLOCK = """
upstream test0 {
    server 1.1.1.1:8080;
    check interval=3000 rise=2 fall=3 timeout=3000 type=http;
    check_http_send "GET /alive.html  HTTP/1.0\r\n\r\n";
    check_http_expect_alive http_2xx http_3xx;
}

upstream test1 {
    ip_hash;
    server 2.2.2.2:9000;
    check_http_send 'GET /alive.html  HTTP/1.0\r\n\r\n';
}
"""

data = nginx.loads(UPSTREAM_BLOCK)
print nginx.dumps(data)

and I got this output:

upstream test0 {
    server 1.1.1.1:8080;
    check interval=3000 rise=2 fall=3 timeout=3000 type=http;
    check_http_send GET /alive.html HTTP/1.0

;
    check_http_expect_alive http_2xx http_3xx;
}

upstream test1 {
    None;
}

There are 2 bugs here:

• Lost double quote in check_http_send line of upstream test0 .
• The upstream tets1 is freaky.

This is a awesome project. It will be glade to see you fix this.

If in nginx.conf there is directive with a block, directives map, geo, upstream, etc. are not parsed correctly. For example:

http {
...
map_any_world anyvalue;
if ($slow) {
set $limit_rate 4k;
}
...
}

r'^\s*map\s*(.*?)\s*{' match:

map_any_world anyvalue;
if ($slow) {

expected - No matches

#32

when i use nginx.loadf('xxxxx').as_dict

• this is my nginx.conf

server{
    listen 80;
    #OPEN-PORT-443
    listen 443 ssl;
    server_name www.xxx.com;
    root /wwww/wwww;
    
    location ~ .*\.(js|css)?$ {
        expires 12h;
        error_log off;
        access_log /dev/null;  # it can work
    }
}

server{
    listen 80;
    #OPEN-PORT-443
    listen 443 ssl;
    server_name www.xxx.com;
    root /wwww/wwww;
    
    location ~ .*\.(js|css)?$ {
        expires 12h;
        error_log off;
        access_log /dev/null  # it can't work
    }
}

He'll be stuck here all the time,how can i fix it ?

---->index 
        error_log off;
        access_log /dev/null
    }
}

---->index 
        access_log /dev/null
    }
}

Hello!

Noticed strange delays, or even and seemingly endless loop while parsing a bunch of config files. Was able to determine that

in "loads" function (lines 540-554)
takes an exponentially long time if the end of config file has lot's of spaces (starting from 18 it can take up to 5 seconds end each next space increases time more)

Sadly I'm not a regexp ninja so I can't fix it by myself and just started to clean end of a file with strip, but maybe this knowledge will help someone somehow.

python-nginx==1.5.6
Python 3.6.12 (default, Dec 1 2020, 13:45:56)
[GCC 10.2.0] on linux

Example config file attached
example.conf.txt

Hi
Thank you for working on this useful library
A question: This line is last line of my existing nginx.conf, which adds 2 more configs for specific parameters. It is possible to parse line like this:
include ./conf.d/*.conf;

Best Regards,
Vahid

version 1.5.4
When using function 'as_strings', error show as this:

lib/python2.7/site-packages/nginx.py", line 402, in as_strings if '"' not in self.value and (';' in self.value or '#' in self.value): TypeError: argument of type 'int' is not iterable

This file could not be parsed:

user nginx;

http {
    server {
        listen 80;

        location ~* ^/portal {
            proxy_set_header Connection "";
            rewrite ^/portal(.*) $1 break;
        }
    }
}

Problem place is proxy_set_header Connection "";

Dear sir:
Unexpected results when parsing the limit_except block using the loadf method.
E.g.

location /M01 {
        proxy_pass http://backend;
        limit_except GET POST {deny all;}
}

The result of parsing using loadf is as follows
                   {
                        "location /M01": [
                            {
                                "proxy_pass": "http://backend"
                            },
                            {
                                "limit_except": "GET POST {\n                deny all"
                            }
                        ]
                    }

Obviously, there was an error in parsing the limit_except block. I guess it was caused by ";" after "deny all". Can you solve it? Thank you.