vaaaaanquish / gokart-pipeliner

gokart pipeline project

I am your father...

gokart-pipeliner's Issues

override requires

class TaskD(gokart.TaskOnKart):
    foo = gokart.TaskInstanceParameter()
    bar = gokart.TaskInstanceParameter()

    def requires(self):
        return {'foo': self.foo, 'bar': self.bar}

    def run(self):
        x = self.load('foo')
        y = self.load('bar')
        self.dump(x + y)

👇

class TaskD(gokart.TaskOnKart):
    foo = gokart.TaskInstanceParameter()
    bar = gokart.TaskInstanceParameter()

    def run(self):
        x = self.load('foo')
        y = self.load('bar')
        self.dump(x + y)

settings don't override require task

Settings for using existing tasks.

add poetry

add poetry

retrun newest result data

image

hoge_task_result = gp.run([HogeTask], params=params, return_value=['HogeTask'])

result = gp.run([HogeTask, PiyoTask], params=params, return_value=['HogeTask', 'PiyoTask'])
print(result['HogeTask'], result['PiyoTask'])

add unittest

add unittest

add CI/CD

add CI/CD

Support dynamic params when running

running_params = {'TaskB': {'param1':'bar'}}
gp.run(predict, params=running_params)

bug load_data_frame

The task name may not be specified on exist task.

class Bar(gokart.TaskOnKart):
    target = goakrt.TaskInstanceParameter()

    def run(self):
        bar = self.load_data_frame(required_columns={'foo'})

Traceback (most recent call last):
  File "python3.7/site-packages/luigi/worker.py", line 191, in run
    new_deps = self._run_get_new_deps()
  File "python3.7/site-packages/luigi/worker.py", line 133, in _run_get_new_deps
    task_gen = self.task.run()
  File "python3.7/site-packages/foo/bar.py", line 31, in run
    bar = self.load_data_frame(required_columns={'foo'})
  File "python3.7/site-packages/gokart/task.py", line 221, in load_data_frame
    if data.empty and len(data.index) == 0 and len(required_columns - set(data.columns)) > 0:
AttributeError: 'dict' object has no attribute 'empty'

Output Dependence Tree

this one.
https://github.com/m3dev/gokart/blob/642a0aae7efd821b8ffea9dd38d21e8c5cb84eb2/gokart/run.py#L35

add ipynb examples

add ipynb examples

support list output

     36 
     37         if return_value:
---> 38             return task.output().load()
     39         return None
     40 

AttributeError: 'list' object has no attribute 'load'

setting don't output log

setting don't output log

Support gokart's read_environ method

this one.
https://github.com/m3dev/gokart/blob/642a0aae7efd821b8ffea9dd38d21e8c5cb84eb2/gokart/run.py#L20

poetry lock

-gokart = "==0.3.20"
+gokart = "^=0.3.20"

TypeError: 'module' object is not iterable

ERROR: Error in event callback for 'event.core.start'
Traceback (most recent call last):
  File "python3.6/site-packages/luigi/task.py", line 275, in trigger_event
    callback(*args, **kwargs)
  File "python3.6/site-packages/gokart/task.py", line 452, in _dump_module_versions
    self.dump(self._get_module_versions(),
  File "python3.6/site-packages/gokart/task.py", line 469, in _get_module_versions
    version = '.'.join([str(v) for v in module.__version__])
TypeError: 'module' object is not iterable

<module 'poetry.__version__' from 'python3.6/site-packages/poetry/__version__.py'>