kraymer / qifqif

Enrich your .QIF files with categories.

Yello there 👋

📖 Digest of my active open source repositories :

[chess] pawnhub: terminal listing of your online chess games results (2022) · [git] cesar: the radical compressor (2020) · [blog] Excommito: back-oriented jekyll template (2017) · [unixy] cronicle: time-spaced backups via cron (2017) · [unixy] F-dotfiles: opiniated dotfiles organization scheme based on stow (2016) · [media] flinck: sort your movies on filesystem using symlinks (2016) · [unixy] qifqif: enrich your .QIF files with categories. (2015) · [media] Replica: the id3 cloner (2012)
And also brack (2020 Dec), ublame (2020), imgimg (2019), etc.

🔥 Have a sneak-peek of future projects to come on my Discussions zone, and vote for the best one(s) ! ✊

qifqif's Issues

Match on Substring Fails for Part Word

I have a match 'Pharmacy' but it fails for 'STAY HEALTHYPHARMACY'. Match should not be restricted to whole words.

Authorize empty input for categories

meaning that no category line would be written in file

Search for matches using regex

And use word granularity for matches, so regex should be like \b<match>\b

Crash when bad match entered

STR: enter a string that doesn't figure in payee field

Traceback (most recent call last):
  File "/usr/local/bin/qifqif", line 9, in <module>
    load_entry_point('qifqif==0.5.1', 'console_scripts', 'qifqif')()
  File "/usr/local/lib/python2.7/site-packages/qifqif/__init__.py", line 350, in main
    transacs = process_file(transacs_orig, options=args)
  File "/usr/local/lib/python2.7/site-packages/qifqif/__init__.py", line 237, in process_file
    cat, match = process_transaction(t, options)
  File "/usr/local/lib/python2.7/site-packages/qifqif/__init__.py", line 224, in process_transaction
    ruler = query_ruler(t)
  File "/usr/local/lib/python2.7/site-packages/qifqif/__init__.py", line 164, in query_ruler
    ok, extras = check_ruler(ruler, t)
  File "/usr/local/lib/python2.7/site-packages/qifqif/__init__.py", line 136, in check_ruler
    extras[field_ko] = TERM.red('✖ %s' % field_ko.title())
AttributeError: 'dict' object has no attribute 'title'

Cleaner dry-run implementation

dump_to_file should be renamed dump_to_buffer and be written to dest file/stdout or both depending on batch and dry-run flags values.

<TAB> Completion

Completion function is not working for me. When it comes to introduce the category and I press tab it just shifts the cursor as usually happens when pressing tab in a text editor.

I am running the module on the Command Prompt of Windows 8.

I tried hard to find the cause of the non-working function without success. I am inexperienced on Python and programming in general.

Thanks

Include one config file from another

So that overidding a few matches while keeping the others is possible with minimum efforts.

Setup a blessed-less env on travis

To avoid issues like #42, Travis should have a second env configured with no blessed dependency.

Default json file is not created on Windows

reported by @pescortell

using ~/.qifqif.jsonis too specific for unix.

Make tab completion case insensitive

And auto-complete by keeping the input case (while applying a fixed case in internal json file)

Omission of final newline and ^ after rewriting .qif file

downstairs-study:Banking jrrk$ cp InformeA{,_old}.qif 
downstairs-study:Banking jrrk$ qifqif InformeA.qif 
Amount..: -186.30
Payee...: PURCHASE - DOMESTIC, FALLOWFIELD, WILLOW BANK HOTEL 
Memo....: ** 1466
Category: travel

---
Amount..: -5.50
Payee...: PURCHASE - DOMESTIC, TOLL CHARGE, M6 TOLL
Memo....: ** 1466
Category: travel

---
Amount..: -4.80
Payee...: PURCHASE - DOMESTIC, TOLL CHARGE, M6 TOLL
Memo....: ** 1466
Category: travel

---
Amount..: 38.23
Payee...: PURCHASE DOMESTIC REFUND, CAMBRIDGE, MARKS & SPENCER
Memo....: ** 1101
Category: clothing

---
downstairs-study:Banking jrrk$ diff InformeA{,_old}.qif 
2d1
< 
25c24,25
< Lclothing
\ No newline at end of file

---
> Lclothing
> ^
downstairs-study:Banking jrrk$ more InformeA.qif 
!Type:CCard    

D25/12/2014
T-186.30
PPURCHASE - DOMESTIC, FALLOWFIELD, WILLOW BANK HOTEL
M** 1466
Ltravel
^
D25/12/2014
T-5.50
PPURCHASE - DOMESTIC, TOLL CHARGE, M6 TOLL
M** 1466
Ltravel
^
D28/12/2014
T-4.80
PPURCHASE - DOMESTIC, TOLL CHARGE, M6 TOLL
M** 1466
Ltravel
^
D30/12/2014
T38.23
PPURCHASE DOMESTIC REFUND, CAMBRIDGE, MARKS & SPENCER
M** 1101
Lclothing
downstairs-study:Banking jrrk$ 

Clear screen after "Press any key to continue" message

Host a rss feed to communicate about qifqif releases

I don't want to maintain a twitter account just for that, so releases should be announced on http://kray.me/rss/projects/qifqif or something like that

Prefill category and match prompts when close match found

The fact that matching is done on full words limit the risk of unintentional matchings but can be frustrating when having to re-enter one letter change matches or stuff like that.
I'd like to keep asking user when no exact full word match found but prefill the prompt with an existing category and match when possible.
So it would make entering match variants as easy as typing Enter twice.

Don't stop on transactions with empty category

entering an empty category != no category

Use ESC to exit app cleanly

Ctrl+C should not be caught as it defeats the Principle of least astonishment (brutal interruption is expected).
Instead, ESC should be used to leave the app cleanly with a prompt to save the work done.

Tag search return first matched tag instead of longer one

if i have in cache

tag: BACON
match: foo

tag: SPAM
match: foo bar

and a transaction has crazy foo bar spank as payee line, SPAM should be the tag returned. Current behaviour return whatever is the first tag in the tree (BACON here as tree is alpha sorted)

Make a testable parse_args()

merge code from main() and build_parser into a testable parse_args()

Enable match on all transaction fields

By default qifqif matchings are based on the payee field, and are exact matches (a match is a substring of the payee field).
This assumption is what makes qifqif intuitive and easy to use but brings some limitations. Noticeably it could be useful to :

• categorize transactions by considering many fields : eg if payee field contains xxxx and amount > 30e
• define a match as a list of tokens present in payee field but that don't have to be sequential

Regexes to the rescue

As the current implementation covers most of the cases, adding a regex flag would allow to keep current simplicity by default while offering the power of regexes to power users

Classic mode :

Amount..: 16.80
Payee...: VIREMENT AXA FRANCE FRAIS DE SOINS
Enter category:

Expert mode (-e) :

Transaction..: 16.80 | VIREMENT AXA FRANCE FRAIS DE SOINS
Enter category:

So matching all equivalent transaction with amount between 10 and 30 e could be used by entering a match alike

^[12]\d\..*AXA FRANCE

diff coloring on payee line is one character off

The matching segment stop being green colored 1 char too soon

Empty line(s) get inserted after first line

like

!Type:Bank



D25/03/2015

Crash when processing files with no 'P' and 'L' lines

Traceback (most recent call last):
  File "/usr/local/bin/qifqif", line 9, in <module>
    load_entry_point('qifqif==0.2.1', 'console_scripts', 'qifqif')()
  File "/usr/local/lib/python2.7/site-packages/qifqif/__init__.py", line 125, in main
    result = process_file(lines, options=args)
  File "/usr/local/lib/python2.7/site-packages/qifqif/__init__.py", line 85, in process_file
    print ink(delimiter) if tag else delimiter
UnboundLocalError: local variable 'tag' referenced before assignment

File:

!Type:Bank
D11/04/15
T-21000.00
^
D15/04/15
T100.00
^

Install

Install instructions too terse.

Win7-32:
E:\gnucash>python pip install qifqif
python: can't open file 'pip': [Errno 2] No such file or directory

Windows console doesn't print unicode

http://bugs.python.org/issue1602
http://stackoverflow.com/questions/5419/python-unicode-and-the-windows-console

51d8b18, or installing win-unicode-console on client side may help, waiting for news from @flywire.

Whatever the solution it should be properly documented on repo homepage.

Use partial match for amount

Although we use match on full words for text fields as payee, makes sense to not limit the match on word (\b) for amount field.

Handle OFX files in input

Edit rst to make pypi page look good

cf https://pypi.python.org/pypi?:action=display&name=qifqif&version=0.3.0

Detect transactions with no payee field early and inform the user

Provide a --version flag

Matching indicators are not displayed for all fields

STR:

• enter a category
• press Enter at payee match prompt
• at guru prompt, enter a valid match on a field. Enter a failed match on a second field.

=> only failed field and category field have status chars. The valid field does not have ✔leading status char displayed

Provide auto-completion for matches

By generating the whole list of substrings composed of full words.

Cancel entered category when entering ' ' match

Sometimes a tag is entered with a typo. A match is queried and there is no way to revert the typo : entering nothing as match validate the tag with no match.
A match consisting only of spaces could be considered a special value forcing to discard entered tag and query it again.

Can't enter a '*' as match

Match: *
Traceback (most recent call last):
  File "./qifqif.py", line 11, in <module>
    qifqif.main()
  File "/home/flap/Dev/qifqif/qifqif/__init__.py", line 248, in main
    transacs = process_file(transacs_orig, options=args)
  File "/home/flap/Dev/qifqif/qifqif/__init__.py", line 112, in process_file
    options)
  File "/home/flap/Dev/qifqif/qifqif/__init__.py", line 99, in process_transaction
    match = query_match(cached_match, t['payee'])
  File "/home/flap/Dev/qifqif/qifqif/__init__.py", line 62, in query_match
    if not tags.is_match(match, payee):
  File "/home/flap/Dev/qifqif/qifqif/tags.py", line 17, in is_match
    return re.search(r'\b%s\b' % keyword, payee, re.I) is not None
  File "/usr/lib/python2.7/re.py", line 142, in search
    return _compile(pattern, flags).search(string)
  File "/usr/lib/python2.7/re.py", line 244, in _compile
    raise error, v # invalid expression
sre_constants.error: nothing to repeat

Remove special handling of 'no payee field' edge case

payee prompt should thus be skipped, straight to advanced one

Handle Ctrl+C gracefully

When pressing Ctrl+C a prompt should ask if user want to save or discard changes.

Related to #7 as it requires to be able to jump in file when saving the result (save result to dest then jump to last edited transaction in input file and append the rest to dest)

Improve coverage by mocking qifqif.query* functions

cf https://stackoverflow.com/questions/21046717/python-mocking-raw-input-in-unittests/21047132#21047132

Add feature to categorise ALL transactions not just those re-occurring

Love this tool. Makes categorising re-occurring transactions much easier.

I would like to be able to use this or something similar to categorise all my transactions. The way I envisage this to work is that another file defines your account structure (this can be exported from GNUCash). This file is read when partially entering account names for a transaction with auto-completion feature. The current functionality is almost there but currently the accounts available in the auto-completion only comes from the keyword matching file.

Let me know what you think?

Cached matchings get deleted

Load a qif file with some transactions with category field set.
=> these categories get their matchings deleted in .qifqif.json when exiting

Error trying to remove a tag not present in cache

Amount..: 190.00
Payee...: VIREMENT SEPA RECU AXA 
Edit 'dfs' category [y,N] ? y
Category: fsdfs
✔ Match accepted: <none>
Traceback (most recent call last):
  File "./qifqif.py", line 11, in <module>
    qifqif.main()
  File "/Users/flap/Dev/qifqif/qifqif/__init__.py", line 248, in main
    transacs = process_file(transacs_orig, options=args)
  File "/Users/flap/Dev/qifqif/qifqif/__init__.py", line 113, in process_file
    tags.edit(cached_tag, cached_match, tag, match, options)
  File "/Users/flap/Dev/qifqif/qifqif/tags.py", line 63, in edit
    tags[cached_tag].remove(cached_match)
KeyError: u'dfs'

Provide a way to edit match while keeping same category

If user makes the effort of re-entering the default existing category (instead of typing Enter), then it should trigger the edit match prompt.

blessed terminal functions are not properly emulated causing crash

Traceback (most recent call last):
File "C:\Python27\Scripts\qifqif-script.py", line 9, in 
load_entry_point('qifqif==0.5.1', 'console_scripts', 'qifqif')()
File "C:\Python27\lib\site-packages\qifqif__init__.py", line 350, in main
transacs = process_file(transacs_orig, options=args)
File "C:\Python27\lib\site-packages\qifqif__init__.py", line 237, in process_file
cat, match = process_transaction(t, options)
File "C:\Python27\lib\site-packages\qifqif__init__.py", line 207, in process_transaction
print('---\n' + TERM.clear_eol, end='')
TypeError: coercing to Unicode: need string or buffer, function found

Provide helpful error message when parsing error

Make blessed dependency optional

qifqif should work in degraded mode (no colors, no fullscreen ) when blessed is not available, as it is the case on Windows.

Only print file content on stdout on exit in batch mode

Not in normal mode :

---
  Date......: 16/03/2016
  Amount....: -43.00
  Payee.....: ACME Corp
  Category..: Misc

Press any key to continue (Ctrl+D to discard edits): 
!Type:Bank
D16/03/2016
T-43.00
PACME Corp
LMisc
^

Insert a pause before leaving edition screen

Add option to use another field than Payee as default

eg Memo field

Print index of transaction in the file and add '--jump' option

The current way to undo a bad input consist to rerun qifqif in audit mode.
Depending on file size it can be long to reach the transaction to edit.
An improvement could be to replace the current delimiter by

[XXX]---

so one can memorize a number to edit and then run `qifqif -a -j XXX``

Provide an option to always save transactions in case of interruption

To avoid a save prompt when leaving the app, we should restore the idea of a permanent updated temporary dest file :

• create empty temp file
• append each transaction to output file as soon as it is processed
• if early Ctrl+C :
  • if auto-save: append unprocessed left transactions and rename temp to dest
  • else: revert .json file

qifqif fails encoding raw_input() returned data to unicode

Traceback (most recent call last):
File "/usr/local/bin/qifqif", line 9, in
load_entry_point('qifqif==0.3.1', 'console_scripts', 'qifqif')()
File "/usr/local/lib/python2.7/site-packages/qifqif/init.py", line 245, in main
transacs = process_file(transacs_orig, options=args)
File "/usr/local/lib/python2.7/site-packages/qifqif/init.py", line 109, in process_file
options)
File "/usr/local/lib/python2.7/site-packages/qifqif/init.py", line 96, in process_transaction
match = query_match(cached_match, t['payee'])
File "/usr/local/lib/python2.7/site-packages/qifqif/init.py", line 65, in query_match
term.red('')))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2714' in position 15: ordinal not in range(128)

No match is asked when entering category