tofunlp / lineflow Goto Github PK

View Code? Open in Web Editor NEW

180.0 180.0 9.0 863 KB

:zap:A Lightweight NLP Data Loader for All Deep Learning Frameworks in Python

Home Page: https://towardsdatascience.com/lineflow-introduction-1caf7851125e

License: MIT License

Makefile 0.32% Python 99.68%

deep-learning machine-learning natural-language-processing python

lineflow's People

Contributors

Stargazers

Watchers

Forkers

sobamchan crackerboy 0xqq dut3062796s tqce awesome-archive arita37 awoziji trellixvulnteam

lineflow's Issues

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Make Reducers

from lineflow import Dataset
from lineflow.reducers import FlatMap, Filter

ds = Dataset(range(100)).map(lambda x: [x] * 3)
_ = Filter(lambda x: x % 2 == 0)(ds)
_ = FlatMap(lambda x: x)(ds)

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Add README to lineflow/datasets

Support Dataset.len

class Dataset:
    def __len__(self):
        return len(self._dataset)

Include preprocessing to datasets

callback for before / after downloading dataset
- for example, unzip, parsing, etc.

Add lineflow.window for Language Model Training

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Add support for <SNLI and MLNLI>

Datasets

*** SNLI dataset *** : https://nlp.stanford.edu/projects/snli/
*** MLNLI dataset *** : https://www.nyu.edu/projects/bowman/multinli/

Please provide both of the datasets individually and the combined dataset as ALLNLI

Improve test cases for lineflow.datasets

Move api

lineflow.Dataset.load to lineflow.load

Allow a empty map_func for lineflow.flat_map

Fix bug in Microsoft Research Corpus

Should slice of IterableDataset return IterableDataset not List?

Is your feature request related to a problem? Please describe.

train = lfds.SciTLDR(split="train")  # IterableDataset
train_mini = train[:10]  # Now this is just a python list (List[Any])

If I make a subset of a dataset, it loses all the features such as .map.

Describe the solution you'd like
Return IterableDataset in stead of List.

Add Issue Template for Dataset

Dependabot couldn't authenticate with https://pypi.org/simple/

Dependabot couldn't authenticate with https://pypi.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

wmt14 google drive link is dead now.

Describe the bug
the google drive link to download wmt14 dataset is now unavailable.

To Reproduce

import lineflow.datasets as lfds
train_dataset = lfds.Wmt14("train")

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
I can try finding a working URL when I have some time.

TextDataset is now too complicated

Currently, TextDataset can handle single/multiple files and also combine two or more files with zip/concat method.
But it is implemented in only one class and hard to understand.
So I'd like to break into multiple classes.