manishrjain / into-ledger Goto Github PK

View Code? Open in Web Editor NEW

105.0 105.0 19.0 988 KB

Most efficient and accurate tool to categorize and import expenses into ledger

License: MIT License

Go 100.00%

into-ledger's People

Contributors

Stargazers

Watchers

Forkers

silky ginabythebay connextrum dimagog mindreframer jcrben dougbeal cljoly backwardn agjohnson tpeacock19 marcelomata whrl jellelicht muehlburger saidaspen raeedcho

into-ledger's Issues

Unable to parse amount with $

If the CSV amount has a dollar sign in it, it can't be parsed by into-ledger.
Example CSV:
01/02/2016,$999.99,"description"

Result:

into-ledger -a test -j ledger.dat -d '01/02/2006' -csv dollar-sign.csv
ERROR           : Unable to parse transaction from the selected columns in CSV.
Selected CSV    : 01/02/2016, $999.99, description
Parsed Date     : 2016-01-02 00:00:00 +0000 UTC
Parsed Desc     : description
Parsed Currency : 0
2017/04/08 19:25:44 Please ensure that the above CSV contains ALL the 3 required fields.

currency not added to values in output file

I have recently updated into-ledger to latest version. After reviewing transactions , currency ( INR in my case) is not added to values in output ledger file.
Request your help.

Parsing CSV

I'm getting the following trying to parse the following CSV:

01/02/2015,PAY,,123.30,1234.13

I'm using the following command:

into-ledger -j coa2015.dat -csv combined.csv -o out.data -a tdbank -c CAD -debug -d MM/DD/YYYY

I get the following exception:

~/Accounting/Ledger/Personal $ Opening file: /home/brian/.into-ledger/shortcuts.yaml for reading key mappings
No config found in /home/brian/.into-ledger/config.yaml. Using all flags2016/11/27 22:47:31 %!(EXTRA []interface {}=[])
2016/11/27 22:47:31 Unable to parse txn for [[01/02/2015 PAY 123.30 1234.13]]. Got: {Date:0001-01-01 00:00:00 +0000 UTC Desc: To: From: Cur:3632.13 CurName: Key:[166 55 173 117 73 22 83 250 38 12 217 34 240 13 254 125] skipClassification:false}

main.check
/home/brian/go/src/github.com/manishrjain/into-ledger/main.go:82
main.parseTransactionsFromCSV
/home/brian/go/src/github.com/manishrjain/into-ledger/main.go:323
main.main
/home/brian/go/src/github.com/manishrjain/into-ledger/main.go:738
runtime.main
/usr/local/go/src/runtime/proc.go:188
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1998

error parsing 4 column CSV

This plugin looks quite helpful, thanks for building it.
I am trying to import a CSV file but I get the following error:

Opening file: .test/shortcuts.yaml for reading key mappings

ERROR           : Unable to parse transaction from the selected columns in CSV.
Selected CSV    : 01/24/2017, Some withdrawal, -$100.00, $500.00
Parsed Date     : 2017-01-24 00:00:00 +0000 UTC
Parsed Desc     : $500.00
Parsed Currency : 0
2017/01/25 07:53:17 Please ensure that the above CSV contains ALL the 3 required fields.

Here are the contents of my demo.csv file:

"date","description","amount","balance"
01/24/2017,"Some withdrawal",-$100.00,$500.00
01/23/2017,"A BANK 2032 TEXAS",$100.00,$600.00

The command I ran:

~/gocode/bin/into-ledger -a 'Test:Checking' \
		-c '$' \
		-debug \
		-conf .test \
		-csv demo.csv \
		-ic '4' \
		-j test.dat \
		-s 1

I am running into-ledger as of 870acb4.
Also, I lose input in my terminal window after the command fails. I can still enter commands but cannot see the cursor or any text. (that is separate issue).

There should be 2 spaces between account name and amount

First, thanks for into-ledger.

After successfully parsing my first CSV I checked the output and some of the entries contain only one space between the account name and the amount. I think Ledger requires 2 or more spaces. At first I thought it happened when I hit Return rather than a specific shortcut key but it doesn't seem consistent.

2016/12/19  Amazon.com
    Expenses:Personal                                    17.36
    Assets:Checking

2016/12/19  JTS PIZZA DEPOT
    Expenses:Food:Restaurants 25.89
    Assets:Checking

2016/12/19  BURGER KING
    Expenses:Food:Groceries 4.79
    Assets:Checking

gives obscure traceback

Thanks for sharing this.

It installed nicely on mac when I guessed brew install go and followed brew's advice to set up a GOPATH. (The readme could link to a go install how-to for new go-ers.)

I had problems running it. Here's what I tried:

$ into-ledger -j ~/src/hledger/data/sample.journal -csv ~/src/hledger/data/sample.csv -a assets:checking -o t.j
Opening file: /Users/simon/.into-ledger for reading key mappings
File /Users/simon/.into-ledger doesn't exist. Creating empty shortcuts
2016/11/02 16:38:21 strconv.ParseInt: parsing "": invalid syntax
main.check
	/Users/simon/.go/src/github.com/manishrjain/into-ledger/main.go:59
main.parseTransactionsFromCSV
	/Users/simon/.go/src/github.com/manishrjain/into-ledger/main.go:280
main.main
	/Users/simon/.go/src/github.com/manishrjain/into-ledger/main.go:673
runtime.main
	/usr/local/Cellar/go/1.7.3/libexec/src/runtime/proc.go:183
runtime.goexit
	/usr/local/Cellar/go/1.7.3/libexec/src/runtime/asm_amd64.s:2086
$ ls -l ~/.into-ledger
ls: /Users/simon/.into-ledger: No such file or directory
$ touch ~/.into-ledger
$ into-ledger -j ~/src/hledger/data/sample.journal -csv ~/src/hledger/data/sample.csv -a assets:checking -o t.j
Opening file: /Users/simon/.into-ledger for reading key mappings
2016/11/02 16:38:38 strconv.ParseInt: parsing "": invalid syntax
main.check
	/Users/simon/.go/src/github.com/manishrjain/into-ledger/main.go:59
main.parseTransactionsFromCSV
	/Users/simon/.go/src/github.com/manishrjain/into-ledger/main.go:280
main.main
	/Users/simon/.go/src/github.com/manishrjain/into-ledger/main.go:673
runtime.main
	/usr/local/Cellar/go/1.7.3/libexec/src/runtime/proc.go:183
runtime.goexit
	/usr/local/Cellar/go/1.7.3/libexec/src/runtime/asm_amd64.s:2086

Errors causes terminal to break

When an error occurs, for example a missing file for the -j option, the terminal is not usable anymore until reset is typed.

is an existing journal file required?

I'm new to ledger, and I want to start my journal with a csv downloaded from my bank website. I touched a journal file, but into-ledger is bailing out with a vague error message:

roughy:~ ben$ go/bin/into-ledger -a mybank -csv ~/Downloads/Transactions_25_06_2018.csv -debug
Using flags from config: map[o:/Users/ben/Documents/business/financial/ledger/mybank.out s:1 c:AUD j:/Users/ben/Documents/business/financial/ledger/journal.ldg d:02/01/2006 ic:0,1,4,7,8]
Opening file: /Users/ben/.into-ledger/shortcuts.yaml for reading key mappings
2018/06/23 21:39:03 Expected some categories. Found none.%!(EXTRA []interface {}=[])
2018/06/23 21:39:03 
2018/06/23 21:39:03 Should be true, but is false
main.assertf
	/Users/ben/go/src/github.com/manishrjain/into-ledger/main.go:100
main.(*parser).generateClasses
	/Users/ben/go/src/github.com/manishrjain/into-ledger/main.go:196
main.main
	/Users/ben/go/src/github.com/manishrjain/into-ledger/main.go:780
runtime.main
	/usr/local/Cellar/go/1.10.3/libexec/src/runtime/proc.go:198
runtime.goexit
	/usr/local/Cellar/go/1.10.3/libexec/src/runtime/asm_amd64.s:2361

and here's what my config.yaml looks like:

accounts:
  mybank:
    c: AUD
    j: /Users/ben/Documents/business/financial/ledger/journal.ldg
    d: 02/01/2006
    ic: "0,1,4,7,8"
    o: /Users/ben/Documents/business/financial/ledger/mybank.out
    s: 1

any ideas what "Should be true, but is false" is referring to?

Add option to specify column delimiter

My bank issues csv files with a semicolon ; as separator. into-ledger currently crashes with those files since it can't find a valid separator (I assume it just uses a comma ,)

Immediate crash: panic: provide at least two classes

Hi,

I wanted to try out into-ledger but can't get it to work. Am i missing some configuration?

> into-ledger -a testaccount -c EUR -csv test.csv -j in.ledger -o out.ledger -conf .into-ledger -debug -d "2016-02-01"
Opening file: .into-ledger for reading key mappings
panic: provide at least two classes

goroutine 1 [running]:
panic(0x5923c0, 0xc042150310)
        C:/Go/src/runtime/panic.go:500 +0x1af
github.com/jbrukh/bayesian.NewClassifierTfIdf(0xc042115c20, 0x0, 0xa, 0xc04202dbb8)
        C:/Users/ilo/dev/go/src/github.com/jbrukh/bayesian/bayesian.go:158 +0x506
main.(*parser).generateClasses(0xc04202ddd8)
        C:/Users/ilo/dev/go/src/github.com/manishrjain/into-ledger/main.go:161 +0x326
main.main()
        C:/Users/ilo/dev/go/src/github.com/manishrjain/into-ledger/main.go:672 +0x451

Doesn't reset terminal upon exit.

It's super annoying, even when running with --help, the terminal is in an unusable state, because the term isn't reset back to normal echo of the characters..

hledger journal support ?

Given some strategy for detecting them, and a hledger executable, hledger journals could be parsed like ledger ones. Here are the corresponding CSV commands and sample output:

~/src/hledger$ ledger -f data/sample.journal csv
"2008/01/01","","income","assets:bank:checking","$","1","",""
"2008/01/01","","income","income:salary","$","-1","",""
"2008/06/01","","gift","assets:bank:checking","$","1","",""
"2008/06/01","","gift","income:gifts","$","-1","",""
"2008/06/02","","save","assets:bank:saving","$","1","",""
"2008/06/02","","save","assets:bank:checking","$","-1","",""
"2008/06/03","","eat & shop","expenses:food","$","1","*",""
"2008/06/03","","eat & shop","expenses:supplies","$","1","*",""
"2008/06/03","","eat & shop","assets:cash","$","-2","*",""
"2008/12/31","","pay off","liabilities:debts","$","1","*",""
"2008/12/31","","pay off","assets:bank:checking","$","-1","*",""
~/src/hledger$ hledger -f data/sample.journal print -O csv
"txnidx","date","date2","status","code","description","comment","account","amount","commodity","credit","debit","status","posting-comment"
"1","2008/01/01","","","","income","","assets:bank:checking","1","$","","1","",""
"1","2008/01/01","","","","income","","income:salary","-1","$","1","","",""
"2","2008/06/01","","","","gift","","assets:bank:checking","1","$","","1","",""
"2","2008/06/01","","","","gift","","income:gifts","-1","$","1","","",""
"3","2008/06/02","","","","save","","assets:bank:saving","1","$","","1","",""
"3","2008/06/02","","","","save","","assets:bank:checking","-1","$","1","","",""
"4","2008/06/03","","*","","eat & shop","","expenses:food","1","$","","1","",""
"4","2008/06/03","","*","","eat & shop","","expenses:supplies","1","$","","1","",""
"4","2008/06/03","","*","","eat & shop","","assets:cash","-2","$","2","","",""
"5","2008/12/31","","*","","pay off","","liabilities:debts","1","$","","1","",""
"5","2008/12/31","","*","","pay off","","assets:bank:checking","-1","$","1","","",""

Unable to parse CSV transactions

I am very interesting in your program, but I can't get it to parse CSV transactions at all.

bash-4.3$ ./into-ledger -a "Chase Freedom" -csv /home/ryan/ledger/private/chasefreedom.csv
Opening file: /home/ryan/.into-ledger/shortcuts.yaml for reading key mappings
File /home/ryan/.into-ledger/shortcuts.yaml doesn't exist. Creating empty shortcuts
Using config: {Currency:USD Journal:/home/ryan/notes/ledger.dat DateFormat:01/02/2006 Ignore:2,4,6,7,8 Output:/home/ryan/ledger/private/chasefreedom.out Skip:1}
2017/01/17 09:51:14 %!(EXTRA []interface {}=[])
2017/01/17 09:51:14 
2017/01/17 09:51:14 Unable to parse txn for [9/30/2013 Hubbard Inn HUBBARD INN 23 debit Alcohol & Bars Chase Freedom CC  ]
. Got: {Date:0001-01-01 00:00:00 +0000 UTC Desc:Alcohol & Bars To: From: Cur:23 CurName: Key:[38 92 34 179 66 137 128 67 221 200 22 229 48 174 105 152] skipClassification:false Done:false}

main.check
	/home/ryan/go/src/src/github.com/manishrjain/into-ledger/main.go:86
main.parseTransactionsFromCSV
	/home/ryan/go/src/src/github.com/manishrjain/into-ledger/main.go:337
main.main
	/home/ryan/go/src/src/github.com/manishrjain/into-ledger/main.go:809
runtime.main
	/usr/lib/golang/src/runtime/proc.go:183
runtime.goexit
	/usr/lib/golang/src/runtime/asm_amd64.s:2086

Thank you!

Unable to parse negative amount in parenthesis (999)

When an amount is given in parenthesis to denote negative value (refund), it can't be parsed

$ into-ledger -a test -j ledger.dat -d '01/02/2006' -csv negative.csv 

ERROR           : Unable to parse transaction from the selected columns in CSV.
Selected CSV    : 01/02/2016, (999.99), refund
Parsed Date     : 2016-01-02 00:00:00 +0000 UTC
Parsed Desc     : refund
Parsed Currency : 0
2017/04/08 19:28:40 Please ensure that the above CSV contains ALL the 3 required fields.

When the negative amount is given with a dollar sign e.g. ($999.99), it also cannot be parsed - this is related to issue #11 as well.

More descriptive errors

Thanks for this tool! It's been making using ledger a lot easier. Although I think it's easy to use once you have it set up, getting it set up was a bit of a pain because there were several points where I couldn't figure out what I was doing wrong. I encountered the following issues during configuration of into-ledger, and I think the error messages could do a better job of helping the user along in each of these cases:

$ into-ledger -a citizens -csv ~/Downloads/EXPORT.CSV
Opening file: /home/david/.into-ledger/shortcuts.yaml for reading key mappings
No config found in /home/david/.into-ledger/config.yaml. Using all flags
  -a string
    	Name of bank account transactions belong to.
  -c string
    	Set currency if any.
  -conf string
    	Config directory to store various into-ledger configs in. (default "/home/david/.into-ledger")
  -csv string
    	File path of CSV file containing new transactions.
  -d string
    	Express your date format in numeric form w.r.t. Jan 02, 2006, separated by slashes (/). See: https://golang.org/pkg/time/ (default "01/02/2006")
  -debug
    	Additional debug information if set.
  -ic string
    	Comma separated list of columns to ignore in CSV.
  -j string
    	Existing journal to learn from.
  -o string
    	Journal file to write to. (default "out.ldg")
  -s int
    	Number of header lines in CSV to skip

	ERROR: Please specify the input ledger journal file 

Unchanged keyboard shortcuts. Skipping overwrite to /home/david/.into-ledger/shortcuts.yaml.

This was confusing because I had created /home/david/.into-ledger/config.yaml, but for some reason into-ledger was saying it couldn't find it. The problem turned out to be that I wrote "accouts" instead of "accounts" in config.yaml. An error like "Unrecognized key 'accouts' in config.yaml" would have made it immediately obvious what the problem was.

$ into-ledger -a citizens -csv ~/Downloads/EXPORT.CSV
Opening file: /home/david/.into-ledger/shortcuts.yaml for reading key mappings
Using config: {Currency:USD Journal:/home/david/Documents/finance/y2012-2016.txt DateFormat:1/2/16 Ignore:0,2,5,6,7 Output:/home/david/citizens.out Skip:1}
2016/12/30 23:24:13 %!(EXTRA []interface {}=[])
2016/12/30 23:24:13 
2016/12/30 23:24:13 Unable to parse txn for [DIRECT DEBIT 12/23/16 Checking JCPENNEY 2416     CONCORD      NH            1216 -350.00    $350.00]
. Got: {Date:0001-01-01 00:00:00 +0000 UTC Desc:JCPENNEY 2416     CONCORD      NH            1216 To: From: Cur:-350 CurName: Key:[59 22 128 102 70 48 233 220 134 3 63 100 103 90 221 81] skipClassification:false}

main.check
	/home/david/.go/src/github.com/manishrjain/into-ledger/main.go:85
main.parseTransactionsFromCSV
	/home/david/.go/src/github.com/manishrjain/into-ledger/main.go:336
main.main
	/home/david/.go/src/github.com/manishrjain/into-ledger/main.go:793
runtime.main
	/usr/lib/go/src/runtime/proc.go:183
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:2086

The problem here was that I had accidentally written "1/2/16" in config.yaml instead of "1/2/06" (as an aside, why does into-ledger require that the date format be specified WRT 1/2/2006? I don't know golang, is something like "%m/%d/%y" not straightforward to implement?), but the error seems to imply that the CSV file is mal-formatted. Would it be possible to instead generate an error like "Invalid dateformat for 'citizens' account in config.yaml"?

$ into-ledger -a citizens -csv ~/Downloads/EXPORT_edit.CSV
Opening file: /home/david/.into-ledger/shortcuts.yaml for reading key mappings
Using config: {Currency:USD Journal:/home/david/Documents/finance/y2012-2016.txt DateFormat:1/2/06 Ignore:0,2,5,6,7 Output:/home/david/citizens.out Skip:1}

Unchanged keyboard shortcuts. Skipping overwrite to /home/david/.into-ledger/shortcuts.yaml.
panic: provide at least two classes

goroutine 1 [running]:
panic(0x57f7c0, 0xc42011ec10)
	/usr/lib/go/src/runtime/panic.go:500 +0x1a1
github.com/jbrukh/bayesian.NewClassifierTfIdf(0xc4201060a0, 0x1, 0xa, 0x0)
	/home/david/.go/src/github.com/jbrukh/bayesian/bayesian.go:158 +0x4ff
main.(*parser).generateClasses(0xc42003bd88)
	/home/david/.go/src/github.com/manishrjain/into-ledger/main.go:185 +0x30c
main.main()
	/home/david/.go/src/github.com/manishrjain/into-ledger/main.go:789 +0xd23

This one still confuses me. After using the stack trace to debug this crash, it seems like when learning from an existing ledger file, into-ledger won't use accounts under "Equity" or "Assets" as input to the Bayesian classifier. It just so happens that the ledger file I was using for input only uses two accounts, one of which is under "Expenses", and one which is under "Assets". It appears that the Bayesian classifier requires at least two classes, but in this case it was being passed only one (the one account in my file under "Expenses"), so the whole thing crashed. I managed to get into-ledger to learn from my ledger file by adding a bogus transaction with a bogus Expenses account, but I do not understand why I got this error to begin with. Maybe it's as simple as changing the assertion on line 183 from "> 0" to "> 1" (in which case I would suggest the failure message be changed to something like "into-ledger requires at least two unique accounts to classify"), but in the first place I don't get why into-ledger doesn't input Asset and Equity accounts into the classifier.

Support comma as decimal separator

Some languages uses comma as decimal separator (3.14 is written as 3,14). A few of these are pretty much all of South America, Europe (not the UK), Russia, most parts of Africa.

This ties in with #13 as supporting comma decimal separator can only come after being able to use other CSV delimiters (my particular case would make semicolon my preferred delimiter).