Coder Social home page Coder Social logo

alexberutils's People

Contributors

alex-ber avatar jayvdb avatar

Watchers

avatar

Forkers

jayvdb

alexberutils's Issues

Config file from another directory is not resolved

I have my configuration file in conf directory.
I am expecting this code to work, but I'm getting HiYaPyCoInvocationException.

from argparse import ArgumentParser
import alexber.utils.init_app_conf as init_app_conf

argumentParser = ArgumentParser(conflict_handler='resolve')
argumentParser.add_argument("--general.config.file", nargs='?', dest='config_file', default='conf/config.yml',
                            const='conf/config.yml')

dd = init_app_conf.parse_config(argumentParser=argumentParser, args=args)

This is the output:

ERROR MainProcess [hiyapyco.__init__] file not found: config.yml (C:\dev\work\example\config.yml)
Traceback (most recent call last):
  File "C:\dev\work\example\venv\lib\site-packages\hiyapyco\__init__.py", line 198, in __init__
    with io.open(fn, 'r', encoding=self.encoding) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\dev\\work\\example\\config.yml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\dev\work\example\runner.py", line 14, in run
    _run(args=args)
  File "C:\dev\work\example\app.py", line 44, in _run
    dd = init_app_conf.parse_config(argumentParser=argumentParser, args=args)
  File "C:\dev\work\example\venv\lib\site-packages\alexber\utils\init_app_conf.py", line 455, in parse_config
    dd = parser.parse_config(argumentParser=argumentParser, args=args)
  File "C:\dev\work\example\venv\lib\site-packages\alexber\utils\init_app_conf.py", line 260, in parse_config
    sys_d, profiles, white_list, list_ensure, config_file = self._parse_sys_args(argumentParser=argumentParser,
  File "C:\dev\work\example\venv\lib\site-packages\alexber\utils\init_app_conf.py", line 244, in _parse_sys_args
    default_d = ymlparsers.load([str(full_path)])
  File "C:\dev\work\example\venv\lib\site-packages\alexber\utils\ymlparsers.py", line 157, in load
    hiyapyco = HiYaPyCo(*args, **kwargs)
  File "C:\dev\work\example\venv\lib\site-packages\hiyapyco\__init__.py", line 210, in __init__
    raise HiYaPyCoInvocationException(
hiyapyco.HiYaPyCoInvocationException: yaml file not found: 'config.yml'

Process finished with exit code 1

Upgrade dependencies

As part of preparation of migrating to Python 3.8 I want to upgrade some dependencies.
Also pytest in use is old, so I will upgrade to the latest version.

Note, that all req-.txt remains unchanged. These files are used in setup.py.
requirements-.txt file are used to create reproducible environment, so if you are not using them directly their changes will not effect you.

Email formatting changed in Python 3.8

I have found regression in email.message.EmailMessage. This changes how e-mail is looks like when generated using EmailMessage.

I’m creating it as following:

from email.message import EmailMessage as _EmailMessage
from email.policy import SMTPUTF8 as _SMTPUTF8

msg = _EmailMessage(policy=_SMTPUTF8)

msg['Subject'] = self.get_subject()
msg.set_content(body.getvalue(), subtype='html')

Note: bodyText has lines that are longer than 78 not counting the end of line character(s).

Quote: max_line_length
The maximum length of any line in the serialized output, not counting the end of line character(s). Default is 78, per RFC 5322. A value of 0 or None indicates that no line wrapping should be done at all.

https://docs.python.org/3/library/email.policy.html#module-email.policy

At Python 3.7 this code generate msg with header that has

('Content-Type', 'text/html; charset="utf-8"'), ('Content-Transfer-Encoding', '7bit')

At Python 3.8 this same code generate msg with header that has

('Content-Type', 'text/html; charset="utf-8"'), ('Content-Transfer-Encoding', 'quoted-printable')

This effectively breaks the line apart.

I’m fixing the code to:

from email.message import EmailMessage as _EmailMessage
from email.policy import SMTPUTF8 as _SMTPUTF8

msg = _EmailMessage(policy=_SMTPUTF8)

msg['Subject'] = self.get_subject()
msg.set_content(body.getvalue(), subtype='html', cte='8bit')

I’m using 8bit and not 7bit because I want to support Unicode character (in my example, there is no surrogate pair, so it was resolved to 7bit). See the code below, it will be more clear.

The cause of the change is in the file email.contentmanager

This is the Python 3.7 function:

def _encode_text(string, charset, cte, policy):
    lines = string.encode(charset).splitlines()
    linesep = policy.linesep.encode('ascii')
    def embedded_body(lines): return linesep.join(lines) + linesep
    def normal_body(lines): return b'\n'.join(lines) + b'\n'
    if cte==None:
        # Use heuristics to decide on the "best" encoding.
        try:
            return '7bit', normal_body(lines).decode('ascii')
        except UnicodeDecodeError:
            pass
        if (policy.cte_type == '8bit' and
                max(len(x) for x in lines) <= policy.max_line_length):
            return '8bit', normal_body(lines).decode('ascii', 'surrogateescape')
        sniff = embedded_body(lines[:10])
        sniff_qp = quoprimime.body_encode(sniff.decode('latin-1'),
                                          policy.max_line_length)
        sniff_base64 = binascii.b2a_base64(sniff)
        # This is a little unfair to qp; it includes lineseps, base64 doesn't.
        if len(sniff_qp) > len(sniff_base64):
            cte = 'base64'
        else:
            cte = 'quoted-printable'
            if len(lines) <= 10:
                return cte, sniff_qp
    if cte == '7bit':
        data = normal_body(lines).decode('ascii')
    elif cte == '8bit':
        data = normal_body(lines).decode('ascii', 'surrogateescape')
    elif cte == 'quoted-printable':
        data = quoprimime.body_encode(normal_body(lines).decode('latin-1'),
                                      policy.max_line_length)
    elif cte == 'base64':
        data = _encode_base64(embedded_body(lines), policy.max_line_length)
    else:
        raise ValueError("Unknown content transfer encoding {}".format(cte))
    return cte, data

Note the lines

    if cte==None:
        # Use heuristics to decide on the "best" encoding.

This is Python 3.8 function:

def _encode_text(string, charset, cte, policy):
    lines = string.encode(charset).splitlines()
    linesep = policy.linesep.encode('ascii')
    def embedded_body(lines): return linesep.join(lines) + linesep
    def normal_body(lines): return b'\n'.join(lines) + b'\n'
    if cte==None:
        # Use heuristics to decide on the "best" encoding.
        if max((len(x) for x in lines), default=0) <= policy.max_line_length:
            try:
                return '7bit', normal_body(lines).decode('ascii')
            except UnicodeDecodeError:
                pass
            if policy.cte_type == '8bit':
                return '8bit', normal_body(lines).decode('ascii', 'surrogateescape')
        sniff = embedded_body(lines[:10])
        sniff_qp = quoprimime.body_encode(sniff.decode('latin-1'),
                                          policy.max_line_length)
        sniff_base64 = binascii.b2a_base64(sniff)
        # This is a little unfair to qp; it includes lineseps, base64 doesn't.
        if len(sniff_qp) > len(sniff_base64):
            cte = 'base64'
        else:
            cte = 'quoted-printable'
            if len(lines) <= 10:
                return cte, sniff_qp
    if cte == '7bit':
        data = normal_body(lines).decode('ascii')
    elif cte == '8bit':
        data = normal_body(lines).decode('ascii', 'surrogateescape')
    elif cte == 'quoted-printable':
        data = quoprimime.body_encode(normal_body(lines).decode('latin-1'),
                                      policy.max_line_length)
    elif cte == 'base64':
        data = _encode_base64(embedded_body(lines), policy.max_line_length)
    else:
        raise ValueError("Unknown content transfer encoding {}".format(cte))
    return cte, data

note, the lines:

    if cte==None:
        # Use heuristics to decide on the "best" encoding.
        if max((len(x) for x in lines), default=0) <= policy.max_line_length:

The line in bold was added in Python 3.8.
Because I do have long lines the try block that tries 7bit and 8bit is skipped in Python 3.8
And it is resolved as quoted-printable.

Fix importer

https://stackoverflow.com/questions/2699287/what-is-path-useful-for
https://docs.python.org/3/reference/import.html#package-path-rules

Changed in version 3.3: Parent packages are automatically imported.
https://docs.python.org/3/library/importlib.html#importlib.import_module

from importlib import import_module
from pkgutil import iter_modules

def walk_modules(path):
    """Loads a module and all its submodules from the given module path and
    returns them. If *any* module throws an exception while importing, that
    exception is thrown back.

    """

    mods = []
    mod = import_module(path)
    mods.append(mod)
    if hasattr(mod, '__path__'):
        for _, subpath, ispkg in iter_modules(mod.__path__):
            fullpath = path + '.' + subpath
            if ispkg:
                mods += walk_modules(fullpath)
            else:
                submod = import_module(fullpath)
                mods.append(submod)
    return mods

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.