Proposed feature:
It would be nice to have relative imports as in Python.

In, let's say 'modules.experiments.classA.module1'

def register(mf):
    mf.load('.module2')
    mf.load('..classB.module3')

In this way, module classes can be moved without destroying internal structures.

Things to consider for every MR

• adaption of documentation
• adaption of unit tests
• general structure

Here is a not working example (v1.35.2):

def setup():
    print("Not called")


def dummy_main():
    print("Called")


def register(mf):
    mf.register_event("main", dummy_main)
    mf.register_event("before_main", setup)

I guess the docs are not clear on the minimum requirements needed to use there transform events.

The reason is that this behavior would be easier to understand when starting to use miniflask.

In more detail:
In python, one expects redefinitions of functions such as

def foobar():
   ...
def foobar():
   ...
def foobar():
   ...

result in one single function, namely the last that has been defined in the scope.
In miniflask, however, events are by default non-unique, thus, defining multiple events of the same name results in running them all successively upon any event-call.

To make this feature intentional for the user, I suggest to redefine the default behavior.
If a user wants an event to be added to the event-list of a specific name, this should be stated explicitly (rather to specify if an event is unique).

This API-change may result in many code changes, however, the resulting behavior is more clear for newcomers of the framework.

Note: Maybe, state should only allow predefined variables?

Description
It would be a nice addition to preprocess/postprocess any event easily.
For every event, say myevent, miniflask could check wether any pre/post events exist.

API Proposals:

• _myevent and myevent_ (underscore defines if the event shall be executed before/after myevent
• myevent_pre and myevent_post
• myevent_before and myevent_after
• before_myevent and after_myevent

Note:
It would be easy to just add optional calls to pre and post events. However, the additional two ifs can be omitted, if the pre and post events are backed directly into the wrapper function.

Note:
Also take multiple steps into consideration.

It should be possiboe do instanciate of modules without making them fill the global event or state scope.

Used cases are:

• allow multiple instances of the same module coexist
• allow multiple modules ciexist that definde the same unique-events

After defining too many modules with many events, the events-module becomes useless.

Suggestion for Improvement:
Add

• a event matching and
• a module matching
  query option to this module.

Here are the (remaining) errors and warnings, which need to be solved the pass the CI.
Since they require a broader knowledge of the code, IMHO it would be best if you could look into it @da-h.

Module miniflask.event

• src/miniflask/event.py:4:0: R0903: Too few public methods (0/2) (too-few-public-methods)
• src/miniflask/event.py:8:0: R0903: Too few public methods (0/2) (too-few-public-methods)
• src/miniflask/event.py:21:4: W0231: __init__ method from base class dict' is not called (super-init-not-called)
• src/miniflask/event.py:63:41: W0621: Redefining name 'event' from outer scope (line 20) (redefined-outer-name)

Module miniflask.state

• src/miniflask/state.py:11:23: W0621: Redefining name 'state' from outer scope (line 37) (redefined-outer-name)
• src/miniflask/state.py:11:4: W0231: __init__ method from base class 'dict' is not called (super-init-not-called)
• src/miniflask/state.py:26:23: W0622: Redefining built-in 'type' (redefined-builtin)
• src/miniflask/state.py:38:36: W0621: Redefining name 'state' from outer scope (line 37) (redefined-outer-name)
• src/miniflask/state.py:38:4: W0231: __init__ method from base class 'dict' is not called (super-init-not-called)
• src/miniflask/state.py:194:4: W0235: Useless super delegation in method '__getattribute__' (useless-super-delegation)
• src/miniflask/state.py:220:23: W0621: Redefining name 'state' from outer scope (line 37) (redefined-outer-name)
• src/miniflask/state.py:225:12: W0621: Redefining name 'attr' from outer scope (line 4) (redefined-outer-name)

Module miniflask.miniflask

• src/miniflask/miniflask.py:440:13: W0143: Comparing against a callable, did you omit the parenthesis? (comparison-with-callable)
• src/miniflask/miniflask.py:452:11: W0143: Comparing against a callable, did you omit the parenthesis? (comparison-with-callable)
• src/miniflask/miniflask.py:503:87: W0640: Cell variable evt defined in loop (cell-var-from-loop)
• src/miniflask/miniflask.py:519:87: W0640: Cell variable glob defined in loop (cell-var-from-loop)
• src/miniflask/miniflask.py:594:8: C0200: Consider using enumerate instead of iterating with range and len (consider-using-enumerate)
• src/miniflask/miniflask.py:717:15: W0703: Catching too general exception Exception (broad-except)
• src/miniflask/miniflask.py:735:4: W0231: __init__ method from base class 'miniflask' is not called (super-init-not-called)
• src/miniflask/miniflask.py:758:26: W0621: Redefining name 'attr' from outer scope (line 11) (redefined-outer-name)

Module miniflask.modules.events

• src/miniflask/modules/events/__init__.py:57:0: R1721: Unnecessary use of a comprehension (unnecessary-comprehension)

introduce child modules to make it possible to collect / choose / merge several modules dynamically

API Proposal:

• when loading modules, call them with moduleA(strargument,42) to initialize a module with user-given arguments
• in case there is only one string-argument, you can use alternatively moduleA.strargument instead of moduleA(strargument)`.
• in case an argument-id is given, the dynamic module loads another module as a child module, meaning that no events are registered globally, but only locally
• defaults can be overwritten by moduleA during registration phase
• internally, it uses load(..., eventbind=False, as_id='.') (see #17)
• enable local event objects and the possibility to register methods to local miniflask event objects only

Problems so far with this plan:

• how to define that at least a module of type x/with event y shall be called?
• can we combine register_default_modules with that? (See #22)

Hi, I dont know if this is intentional, but the "mf.set_scope" function behavior was modified substantially. I previously used it to replace parts (submodules) of a module by different code (eg. a block of a resnet) in a separate module.

Example case: I want to replace grp.model.resnet.blocks.basicblock_pre by a different submodule. Before I could just use

mf.set_scope("grp.model.resnet")

in the replacement module in order to write state-variables in the same scope as the original submodule. Now, even when using the set_scope function, I cannot get rid of the scope of the new module (lola_arch here):

The following argument is required: --lola_arch.grp.model.resnet.first_conv_kernel_size

Is this intentional? How should I work with these situations?

API-proposal:

• (bf1e3e9) event-object should behave like a list of all events per module, e.g. events["moduleA"] should be a list of all defined events in that module
• (8d72615) (optional) introduce event.eventname.modules / event.eventname.fns for more granular control when using events directly
• introduce child modules to make it possible to collect / choose / merge events from other modules dynamically (see #21)

Postponed:

• redefine event-representation for better readability
  • although overloading __repr__ is indeed possible. In my experiments a object with a __call__ is slower than first-class wrapper-functions.

How can such a tool exist without any unit-testing? ;)

Todo:

• list all features ^^
• register_defaults registration of public settings / changeable using cli
  • basic types (int, float, bool, str)
  • enums (embed into test_argparse)
  • required arguments
  • lists of basic types
  • overwrite a variable using cli (also dynamic variable matching)
  • specialized versions
    • what does the option scope=... do?
    • register_globals
    • overwrite_globals
    • overwrite_defaults
    • register_helpers
• state (using module variables in an event)
  • fuzzy variable matching
  • state.temporary
  • like-expressions
  • lambda-expressions
• event
  • register_event
  • (dynamic) register_event
  • overwrite_event
  • unregister_event
  • optional events (and altfn)
  • before_ events
  • after_ events
  • outervar
• module dependencies
  • plain dependencies
  • relative module paths
  • automatic loading of parent modules (and how to turn that feature off)
  • default modules (by event-name)
  • default modules (by module name)
• exceptions + debug mode
  • during event (debug mode)
  • during event (non-debug mode)
  • during register (debug mode)
  • during register (non-debug mode)

API-Proposal:

mf.register_defaults({
   "var": optional(float)
})

This variable is None if the user does not specify any value and a float if he does.

Proposed change:

miniflask.init({
   'global': './link/to/global/modules',
   'set1': './sub/directory/set1_modules',
   'set2': './other/sub/directory/set2_modules',
})

Modules should then have the unique identifier global.dir.moduleA, etc.

Both commands sound very similar, however their used cases are pretty different.
What this actually does should be more visible in the method name.

Also, add unit-tests for both functions:

• set_scope
• register_as

• remove requirement to return args/kwargs
• include automatic argument matching (if argument list is inclomplete)

Using state.get() to retrieve state variables shows inconsistent behaviour, or often does not work at all.

As miniflask wraps CLI arguments, it may be helpful to let the module define units in which arguments are passed.

Example:

mf.register_defaults({
   "price": mf.unit({
      "base": "usd",
      "eur": 0.84
   })
})

Then the user can use these CLI-arguments:

• --price 42 or equivalently --price 42eur or even --price 42 eur

• --price 49.71usd or --price 49.71 usd

• Internally, miniflask would recalculate the numbers to the base unit if queried directly state["price"].

• Querying the unit given by the user could be done just as state.unit("price", "base")

• And recalculating to another unit by using state.unit("price", "eur")

Miniflask cannot find/use outervar variables in an event event_0 if a before_event_0 hook is used.

Hi all,

I am currently implementing multiple variants of a module which all work in the same scope.
Therefore, I use mf.set_scope("."). Which kind of bugs me.

Here is the module layout:

modules/
    ABC/defaults/__init__.py
    .../  ...   /.module
    ABC/variant1/__init__.py
    .../  ...   /.module
    ABC/variant2/__init__.py
    .../  ...   /.module

The module ABC has some default settings in defaults and two variants, namely variant1 and variant2.
Setting mf.set_scope(".") works fine, and they lived happily ever after.

But setting the scope to "." implies to me that we use the current directory/module as scope, which is either ABC/defaults, ABC/variant1 or ABC/variant2. Thus the mf.set_scope(".") command seems IMHO to be pointless.
Instead, I expected to set the scope to "..", which usually translates to the parent directory/module.
And this makes sense, since the module_id before set_scope is e.g. ABC.variant1, thus the parent is ABC.

What is the design choice behind using "." vs. ".."?

Class:

@dataclass
class MyData
    state: miniflask.state
    transforms: transforms
    mode: str

Registration:

mf.register_event("init_dataclass", MyData, unique=True)

Call:

return event.init_dataclass(state, transform, splitname)

argparse could modify sys.args to also query variables in a semi-fuzzy way as loads does

I suggest not to pollute objects with various different properties and attributes related to miniflask, such as fns, fns_args, mf_modules and subevents, but use one private (!) variable for this, e.g. _miniflask.

Using Enums created in other Modules does not work when including the same Enum using import statements.
A solution should at least be mentioned in the documentation.

See #28 (comment).

Enable scope-search for variables

Consider the following setup of a module named moduleA (having one variable dir), a submodule of moduleA named specialmoduleA (having one variable autostart) and a seperate module named moduleB.

moduleA│dir                                 = ./logs
              │specialmoduleA│autostart                    = True
moduleB | dir                           = ./somewhere

What should be intended behaviour when using the expression state["dir"] inside of specialmoduleA ?

Previously the code would exit with an error:
(When no submodules have been present in miniflask)

StateKeyError: Variable 'dir' not known. (Module moduleA.specialmoduleA attempted to access this variable.)

I tried the following interpretations:
	- as module variable: 'moduleA.specialmoduleA.dir'
	- as global Variable: 'dir'
	- finally, I tried any ordered selection that contains the keys: ['dir']. 

Proposed API Change:
Before trying the global variable and finally any variable containing the name dir, let the state also check all parent modules for such a variable.
I.e. in this case the expression would default to the variable of moduleA instead of exiting with an error.

state.scope lets the user work on multiple states simultaneously.

The API however does not match the default behaviour of miniflask v2 to automatically decide wether relative or absolute module ids are used as a parameter.

I encountered a bug when registering an after_"event" and adding "event" to the function parameters eg.

mf.register_event('after_backward', test, unique=False)

def test(event):
pass

I have no problems when including "state". I get the following stacktrace: https://pastebin.com/Q9v1ettP.

Hi all,

I just run across the issue of using an enum as default value, which is currently not supported.

Here is a small example of what I would like to do:

from enum import Enum
class Color(Enum):
    RED = 1
    GREEN = 2
    BLUE = 3

# ...
# and later use this enum as default value
mf.register_defaults({
    "color": Color.RED
})

This fails with the following error:

ValueError: Type '<enum 'Color'>' not supported.

Instead, what can be used is the value of the enum, like so:

# ...
mf.register_defaults({
    "color": Color.RED.value  # equals 1
})

However, using the value of the enum has no advantage, since I need to use the integer representation if I want to overwrite it (e.g. --color 2).

But if we can use the enum itself, the CLI call to override the value would be something like --color Color.GREEN, which IMHO is preferable and very clear.
(Note: I have no idea how the string can be converted to the enum. maybe this?)

What do you think? Is this a desired feature?

Restricting before/after events to have a specific method head and return format is confusing, especially if one changes an event from being called after to being called before, etc.

To improve the api, i suggest to move the fuctionality to the event variable.
In more detail, upon querying event.params, any event (including before/after events) allows to get the parameters of the caller).

For instance:

def before_event(event):
   event.params["var2"] *= 2
   ...

def main(var1, var2=1, var3=None):
   ...

(In this case, calling main would automatically transform the input of the parameter var2 as if it has been called using var2=2)

The thing I want to see is a list of events (and thus, their order) before their execution - before any execution at all to be precise.

Citing @da-h ideas:

• One possibility would require a bit of work and a introspection/inspection module in miniflask that analyses all calls to the event-object without calling the functions on its own. The downside is, that this will not work in all cases.
• Another solution would load/call the events normally, but also log the callstack (maybe as a file).

I suggest a with-statement, that changes variables in state only temporarily:

with state.temporary({
   "variable1": 42
}):
   # ... do something with the state, e.g.:
   event.call_something_that_uses_states()

When to original method returns nothing, after events fail, since hook["return"] is not set.

state could also query variables in a semi-fuzzy way as loads does

Pytest calls all scripts in one go.

Changing sys.path can have unwanted side-effects when defining separate module repositories.

This would allow lists of fixed sizes as arguments as well.

The API regarding event.optional, especially event.optional_unique seems a bit flawed.

The point of these two differences lies in the fact, that events themselfs can be optional and/or unique.
In that case the event-caller has to decide how the general api of the call looks, i.e.

• in case of optional events: multiple events can be called, thus, multiple results are returned in a list
• in case of optional and unique events: the input is transformed, thus, optional_unique returns the input to the calee

Is there maybe a way to improve the api to a more concise user experiance?

Remove the necessity to call
del event.myevent
to clear the cache.

In other words, any call to
mf.register_event (within an event?) should to this by itself.

I think it would even be better, if any call to mf.register_event would do that.

Note: Think also about a nicer way to pass the module's mf-object from the registration phase into an event.

Hi there,

using an unknown variable in a like statement hangs (indefinitely?!)

Here is a short example:

mf.register_defaults({"foobar": like("foobar", 0)})

where foobar was previously not set.

As eventnames are the main interface, these should be asked for when it comes to default modules.

API-Proposal:

• `register_default_module('module_id', 'eventname')
• `register_default_module('module_id', required_event='eventname')
• `register_default_module('module_id', required_module='regex')

When registering default values using the register_defaults method, I often find myself somewhat limited, especially when

• setting 'empty' values, but requiring a different type (e.g. string value, but None as default)
• using uncommon, but standard types (e.g. tuples)
• using custom types (e.g. classes)
• some types change their type (e.g. setting a default value as [(1, 3)] will turn into (1, 3) - but this is not what is intented?!)
• adding help messages
• providing a list of choices

Currently, to be able to support these usecases these things must be introduced by miniflask, and thus consequently need maintenance, and some may not even be realizable at all.
But so far, every (or at least almost every) issue I faced regarding argument parsing was somewhat solved by the python argparse package.
Therefore I would suggest to rely more on the capabilites of the argparse package by providing a more direct API.

This could look similar to the following (just an idea), where a dict of argparse parameters is passed instead of the actual value:

mf.add_cli_arguments({
    "integers": {metavar='N', type=int, nargs='+', help='an integer for the accumulator'}
    "--sum": {dest='accumulate', action='store_const', const=sum, default=max, help='sum the integers (default: find the max)'}
})

Of course, miniflask provides a few additional functionalities, but I think these can be provided by a custom argument parser and action classes.
This way, there would be no need to reinvent the wheel with argument parsing by miniflask.

IMHO it would be an improvement to the current state (code and usability wise) by pushing a few responsibilities to the user and relying more on the argparse package.

What do you think? Is this worth looking into?

Possibly, related to #53, we could group CLI-arguments using a syntax like

--module.name( --a 42 --b )
or
--module.name{ --a 42 --b }

to represent

--module.name.a 42 --module.name.b

Check if all features are present in documentation:

• debug-mode
  • on shows full traceback, also miniflask internal
  • turning off hides the internal traceback
• register_defaults registration of public settings / changeable using cli
  • basic types (int, float, bool, str)
  • enums (embed into test_argparse)
  • required arguments
  • lists of basic types
  • overwrite a variable using cli (also dynamic variable matching)
  • specialized versions
    • what does the option scope=... do?
    • register_globals
    • overwrite_defaults
    • register_helpers
• state (using module variables in an event)
  • fuzzy variable matching
  • state.temporary
  • like-expressions
  • lambda-expressions
• event
  • register_event (unique)
  • register_event (non-unique)
  • register_event (in event, unique)
  • register_event (in event, non-unique)
  • dynamic event registration
  • optional events (and altfn)
  • overwrite_event
  • unregister_event
  • before_ events
  • after_ events
  • outervar
• module dependencies
  • plain dependencies
  • relative module paths
  • automatic loading of parent modules (and how to turn that feature off)
  • default modules (by event-name)
  • default modules (by module name)
  • load_as_child
• integrate documentation text into api text and reference these parts
• event calling

Methods that do not have a event/state variable do not call before_ and after_ events at the moment.

Exception-Handling need to be improved.

• Exclude the miniflask.event wrap-funktions for typical exceptions
  (0a2a470)
• (Include a debug option that show them again)
  (94a873d)
• Due to defering some variable definitions (overwrite_defaults) the source of errors may not be displayed anymore in the exception list.
  (83a53f3)