alexlamson / datawrangler

Make quick and dirty data mining made easier in Sublime Text

alexlamson.com

Code for alexlamson.com

datawrangler's Issues

Detect separations (private function)

It tries to be a little smart and guess what kind of spacing is being used between the columns (ex. tabs vs spaces vs commas vs comma w/ spaces)

detect_separations doesn't work after aligning columns

Options to fix the problem

remember where the columns are when you align the columns (easier)
build detector function that will automatically detect that format (better if the data gets changed after aligning)

Align columns fails with this exact string

String to reproduce error:

ReportId, ActivityId, Name, GroupId, CommonGroupId, GroupListId, CommonGroupListId, IsActive, IsBillable, StartLocalTime, StartUtcTime, EndLocalTime, EndUtcTime, Notes, RelatedActivityId, SourceId, CurrentChangeSequence, CurrentChangeRandomValue, Other
(3, 611, 'ReportId, ActivityId, Name, GroupId, CommonGroupId • (Quantified Self, manictime sample) - Sublime Text (UNREGISTERED)', 62, 12, None, None, 1, None, '2018-06-27 13:39:31', '2018-06-27 17:39:31', '2018-06-27 13:39:36', '2018-06-27 17:39:36', None, None, None, 1500, 1218390469, '{}')

Stack trace:

  File "C:\Program Files\Sublime Text 3\sublime_plugin.py", line 1072, in run_
    return self.run(edit)
  File "C:\Users\Alex\AppData\Roaming\Sublime Text 3\Packages\User\DataWrangler.py", line 205, in run
    column_widths = detect_col_widths(self, sep, num_columns)
  File "C:\Users\Alex\AppData\Roaming\Sublime Text 3\Packages\User\DataWrangler.py", line 59, in detect_col_widths
    column_widths[i] = max(len(cell_string), column_widths[i])
IndexError: list index out of range

FlattenListOfLists should have a setting to disable ignoring non-list lines

AA
	BB
CC
DD
	EE

The above example doesn't retain the CC in the output. There should be a setting to enable this behavior.

line frequencies: ignore case when merging lines

This may have to exist in a setting or keybind to allow switching this on and off

align columns visually

 * add spaces to columns such that there is no vertical overlap between columns
 * maybe also re-order the columns such that the narrow column come first?

Example:

aaaa bb ccc
dd eeeeee ff

Becomes:

aaaa bb     ccc
dd   eeeeee ff

Function to remove stopwords

List of stopwords can be found here:
https://gist.github.com/sebleier/554280

Delete whitespace-only rows

For each line, remove it if the string matches any of the following regexes:
^$
^\t+$
^,+$
^ +$
^(, )+$

Ability to merge similar lines in word frequency file thing

outsource hard stuff to pandas

automatically detecting dtype and column separations could potentially be done by using some pandas libraries. If that's possible, it should be done

Automatic merging of similar lines

Not sure how it should be implemented, but take all lines that are uncommon but within a certain edit distance of a commonly occuring line, and replace the uncommon line with the common line.

Also, it may be wise to have some human oversight into this process while it happens.

Improve lines2words function

Make it use regex or something to separate the words

when aligning columns, word wrap isn't always off in the new tab

NLP-based word segmentation

Use NLTK to do word segmentation on the document. Spaces between words will be replaced with newlines.

standardize timestamps

convert dates and times to the format: YYY-MM-DD HH:MM:SS.SS (where hours is zero-padded 24-hour time)

Remove trailing tabs

when copying from google sheets, there are lines at the end of the doc that are empty sans tabs. this will remove those lines

would recommend creating a small screencast that captures plugin in action

For instance,
https://github.com/recite/re-cite.org/

The plugin seems useful, btw.

Is your feature request related to a problem? Please describe.
Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

delete column

Assuming that the document being viewed is a table, this command should delete the column that the cursor is currently in

Add support for table-izing raw lists of tuples printed from python

All pairs

Given a list of elements, list all pairs of elements for that list

ex.
Input
a,b,c,d
Output
a b
a c
a d
b c
b d
c d

Can be done with the following code

import itertools
list(itertools.combinations(['a','b','c','d'], 2))