nlppln / nlppln

NLP pipeline software using common workflow language

NLP Pipeline

nlppln is a python package for creating NLP pipelines using Common Workflow Language (CWL). It provides steps for (generic) NLP functionality, such as tokenization, lemmatization, and part of speech tagging, and helps users to construct workflows from these steps.

A text processing step consist of a (Python) command line tool and a CWL specification to use this tool. Most tools provided by nppln wrap existing NLP functionality. The command line tools are made with Click, a Python package for creating command line interfaces.

To create a workflow, you have to write a Python script:

from nlppln import WorkflowGenerator

with WorkflowGenerator() as wf:
  txt_dir = wf.add_input(txt_dir='Directory')

  frogout = wf.frog_dir(in_dir=txt_dir)
  saf = wf.frog_to_saf(in_files=frogout)
  ner_stats = wf.save_ner_data(in_files=saf)
  new_saf = wf.replace_ner(metadata=ner_stats, in_files=saf)
  txt = wf.saf_to_txt(in_files=new_saf)

  wf.add_outputs(ner_stats=ner_stats, txt=txt)

  wf.save('anonymize.cwl')

The resulting workflow can be run using a CWL runner, such as cwltool:

cwltool anonymize.cwl --txt_dir /path/to/directory/with/txt/files/

For creating new (e.g., project specific) NLP functionality, you can use nlppln-gen to generate boilerplate (i.e., empty) command line tools and CWL specifications.

The full documentation can be found on Read the Docs.

Installation

Install nlppln using pip:

pip install nlppln

Please check the installation guidelines for additional required software.

License

Copyright (c) 2016-2018, Netherlands eScience Center, University of Twente

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

nlppln's Issues

create_dirs only works when called with a filename (not a directory)

os.path.dirname removes the last part from the name, even if it is a directory.

Add badges

doc2txt in requirements.txt should be docx2txt

CWL files are not included when installing with pip

Make CWL types constants for tab-completion

When adding inputs, it would be nice if there was a list of CWL types available in nlppln, so that I could use tab-completion when entering them, e.g.

with nlppln.WorkflowGenerator() as wf:
  something = wf.add_input(something=nlppln.cwl_type. [TAB TAB TAB]

This way, I don't have to know all the CWL types by heart.

No output files in cwl template

If there is no meta_out file and no output files, the cwl template should specify this

Installation requirements failed on pyjq

Running "pip install -r requirements.txt" failed on package pyjq.
Apparently this package has its own, additional dependencies (which were not yet installed on my computer)
After running "sudo apt install -y autoconf automake build-essential libtool python-dev" installation was succesful

Make Docker Container for use on Windows

CWL on Windows requires a Docker container for command line tools to run in. So, make a Docker container that has nlppln preinstalled.

https://github.com/common-workflow-language/cwltool/blob/master/windowsdoc.md

Add tests for the CWL steps

Some test suite for commandline tools can be useful here: https://spin.atomicobject.com/2016/01/11/command-line-interface-testing-tools/

Give python commands meaningful name

Now all commands have (function) name 'command', perhaps it is better to give them a more meaningful name (useful for automatically generating cwl steps from click commands).

Generate script does not create boilerplate for opening the meta_in file

Add unit tests

Automatically generate CWL for click command

I already dislike the nlppln.generate command.

In https://github.com/common-workflow-language/gxargparse functionality for click2cwl already exists, but must be improved.

There is also https://github.com/erasche/argparse2tool and https://github.com/nlesc/click2cwl

Add documentation to cwl step and Python command

Allow user to (interactively) see the documentation

Add FAQ/tip/trick about SELinux

https://en.wikipedia.org/wiki/Security-Enhanced_Linux

An additional access control system that causes problems with Docker volume mappings (the command in the docker container thinks a mapped directory is empty).

Switch off SEL with: su -c "setenforce 0"
Switch on SEL with: su -c "setenforce 1"

Make CWL step for all files in directory

The python tool 'create_file_list' could be made into a cwl workflow step.

Check requirements

Not all packages used are in requirements.txt

Import scriptcwl workflow generator

So, workflows can be generated directly from nlppln.

Add CWL conformance test

https://pypi.python.org/pypi/cwltool

Make output directory an option for all commands

For each command, output files should be written to the current directory by default. The output directory can be changed by adding a -o option.

This is much more transparent than the current situation.

Use docopt to make beautiful CLI

Python docopt could be an alternative to click.

Error when running commands in Python 3

When I try running any command in Python 3.6, I get the following error:

python -m nlppln.commands.ls .
Traceback (most recent call last):
  File "/home/dafne/anaconda2/envs/adh/lib/python3.6/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/dafne/anaconda2/envs/adh/lib/python3.6/runpy.py", line 153, in _get_module_details
    code = loader.get_code(mod_name)
AttributeError: 'Py2Fixer' object has no attribute 'get_code'

Whitespace in template/command.py is incorrect

When there are no outputs or meta_out. This is due to the fact that the utility functions are not imported.

Fix installation instructions

pandas as a dependency is a problem. Not sure how to fix this.

Move textDNA-generate cwl step to textDNA repository

How to create ocr post correction from beginning?

I am an intern in one company and they gave me this ocr post correction project but i am new about machine learning and python. Your project exactly does what i want but i am getting lost while analyzing where to start and which steps should i follow. I know this takes your time but it is very important for me to understand and implement this project. I hope you will help. Thanks.

Remove option out_dir from cwl files

Setting a different directory to write output to using the --out_dir option in the cwl specifications doesn't work. The output directory should be changed using cwltool (--outdir).

The option --out_dir in the Python commands does work. So, it shouldn't be removed from the Python commands.

Automatically load cwl steps defined in nlppln

How to organize domain/project specific steps

Domain/project specific steps should be stored in a separate git repo. How should such a repo be organized and how can steps in a separate repo be used by nlppln?