moov-io / bai2 Goto Github PK

Bai2 Version: 0.1.0

03 (Account Identifier) and 16 (Transaction Detail) records contain three-digit "type codes" that denote the type(s) of data the record contains. These are key to interpreting the data in the BAI2 file; therefore it makes sense that this library should return relevant information about them.

Type codes come in three broad categories:

• Account Status: Pretty self explanatory, these describe things like ledger balance, available balance and float position. These show up in 03 records (and their continuations).
• Activity Summary: These are summaries of credit and debit activity on the account. Again, these appear in 03 records (and their continuations).
• Transaction Detail: These describe individual debits and credits on an account. As the name suggests, they show up on 16 records (and I assume their continuations, but the spec is mum on this).

Summary and Detail types also denote whether they refer to a debit or credit. The spec PDF has a large table detailing all the codes and their meanings. A kind person has helpfully transcribed them all into a CSV already.

We can probably convert that CSV into a hardcoded map without much difficulty. Something like

type TypeCode struct {
	TypeCode string // not int as leading zeros are significant
	Transaction string // "CR" "DB" or "NA"
	Level string // "Status" "Summary" or "Detail"
	Description string
}

var TypeCodes := map[string]TypeCode{
	"010": {
		TypeCode: "010",
		Transaction: "NA",
		Level: "Status",
		Description: "Opening Ledger",
	},
	// ...
}

Then it would be an easy lookup. We could also make type-aliased string enums for transaction and level, but unsure if it's really necessary.

In 03 records, we can also determine whether a type code should have item count and funds type by whether it's a status or summary type code, which may help in parsing that.

I am investigating how one might update the library to generate the trailer record aggregates. At first this seemed to me like it should be a relatively simple task by following some of the ACH library Create() patterns and iterating through the hierarchy.

However as I've come to understand the file specification more, and from reading through the bai2 library code, I can see counting the continuation records is becoming a bit of a foil. In particular, reading through string() in pkg/lib/record_transaction_detail.go#L114 and reading through util.WriteBuffer() pkg/util/write.go#L16, I can see that the total number of records is also directly tied to the file's Physical Record Length which follows the rules of the file spec.

I have some ideas of how I could count the number or records, which I can present both here and/or in a PR but I am curious, what are your thoughts on this issue, and why weren't the aggregate fields calculated in the design of the library to this point?

Bai2 Version: Main

What were you trying to do?
Run test suites in my fork

What did you expect to see?
Passing tests

What did you see?
Seed #0 of the fuzz test suite consistently failed

How can we reproduce the problem?
Run the test suite on windows

Bai2 Version: 0.1.0

What were you trying to do?

What did you expect to see?

What did you see?

How can we reproduce the problem?

03 (Account Identifier) records in BAI2 may contain sets of (type code, amount, item count, funds type) records multiple times in sequence for an arbitrary number of type codes. Currently, the library treats this sequence as though it can only occur once per record. (Frustratingly, this very important detail about the format is buried in a confusingly worded footnote on page 16 of the spec and is otherwise left unclear.) Once we're returning the "fully parsed" file model from #52 we can enhance the account object to contain a slice of these.

Bai2 Version: 0.1.0 (todo update this?)

What were you trying to do?

This is a follow-on from #49 and #51

What did you expect to see?
Ideally the parser should return a File object that contains one or more Groups. Groups contain one or more Accounts, and Accounts contain zero or more TransactionDetails. This is consistent with the semantics of the file format as defined in the spec.

What did you see?
Instead, the parser returns a file.Bai2 object (consider renaming this to File for consistency). This contains a slice of Groups as one might expected. However, the structure is flat from here on out. Instead, a group contains a flat slice of all of its subsequent "records" (i.e. structs representing individual rows in the file). This isn't optimal because it exposes lots of details of the file format to a user who really just wants the data from it, like the header and trailer records for a group/account, as well as continuation records.

How can we reproduce the problem?
Call bai2.Parse on any valid BAI2 file.

I think we can probably approach this in two distinct phases. First, a "scanning" phase where we read each row of the file into a record struct, building up a flat slice of records. Then we can have a "parsing" phase, where we build that richer structure from the records. The result of that second phase would be what we return to a user.

Copying from my comment on the previous PR, the final file/group/account objects can probably look something like

type File struct {
	// Metadata from file header
	Sender               string
	Receiver             string
	FileCreatedDate      string
	FileCreatedTime      string
	FileIdNumber         string
	PhysicalRecordLength int64 `json:",omitempty"`
	BlockSize            int64 `json:",omitempty"`
	VersionNumber        int64
	
	// Metadata from file trailer
	FileControlTotal string
	NumberOfGroups   int64
	NumberOfRecords  int64

	// Groups
	Groups []*Group
}

The consumer shouldn't need to be aware of the headers/trailers, so we can just collapse the data from those into the relevant model structs. We can do something similar for groups

type Group struct {
	// Metadata from group header
	Receiver         string `json:",omitempty"`
	Originator       string
	GroupStatus      int64
	AsOfDate         string
	AsOfTime         string `json:",omitempty"`
	CurrencyCode     string `json:",omitempty"`
	AsOfDateModifier int64  `json:",omitempty"`

	// Metadata from group trailer
	GroupControlTotal string
	NumberOfAccounts  int64
	NumberOfRecords   int64

	// Accounts
	Accounts []*Account
}

and accounts

type Account struct {
	// Metadata from account identifier
	AccountNumber string
	CurrencyCode  string       `json:",omitempty"`
	TypeCode      string       `json:",omitempty"`
	Amount        string       `json:",omitempty"`
	ItemCount     string       `json:",omitempty"`
	FundsType     FundsType    `json:",omitempty"` // this can be a string for first take, or we can ignore it until we get it right

	// Metadata from account trailer
	AccountControlTotal string
	NumberRecords       int64

	// Transaction Details
	TransactionDetails []*TransactionDetail
}

The transaction details struct would be similar.

For a first pass, I suggest we take a stab at proper handling of continuation records, but punt on handling funds type (and the fields that come after it in transaction details). For now we can return the last few fields in a string slice or something (similar to the Composite thing that exists now. Proper handling of that can come in a separate PR.

Bai2 Version: 0.1.0

What were you trying to do?

What did you expect to see?

What did you see?

How can we reproduce the problem?

This is a follow-on from #52. Right now the library doesn't properly handle funds types (see page 30 of [the spec]), instead coalescing all fields from funds type onward in a given record into a slice of strings.

Ideally we would have some kind of rich type here that captures all the detail in this. For a first pass just grabbing a slice of strings might be fine, as long as we can distinguish them from the fields that come after them in 16 (transaction detail) records.

Also I just discovered that 03 (account identifier) records can have more than one of these. It can have the sequence of (type code, amount, item count, funds type) once per type code, an arbitrary number of times. I'll open another issue to address that as well.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Other Branches

These updates are pending. To force PRs open, click the checkbox below.

• chore(deps): update actions/cache action to v4
• chore(deps): update actions/checkout action to v4
• chore(deps): update actions/setup-go action to v5
• chore(deps): update github artifact actions to v4 (major) (actions/download-artifact, actions/upload-artifact)
• chore(deps): update github/codeql-action action to v3

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update module github.com/moov-io/base to v0.49.1

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Bai2 Version: 0.1.0

What were you trying to do?
I tried to use the library (as an imported Go package, calling file.Parse) to parse a BAI II file defined in a variable length format (see page 10 of the spec).

What did you expect to see?
I expected to see a properly parsed file.Bai2 data structure with the fields populated appropriately.

What did you see?
Instead I saw a struct mostly filled with zero values. When looking at the source code I discovered that the library as currently written expects BAI files in the fixed-length format only. In fact, it's even more rigid than that, as the parser is expecting individual columns for each record type to begin at specific indices within the record, which is more rigid than the spec calls for. The correct behavior in either case would be to read the record until the next delimiter, no matter what index that appears at.

It seems I received silent failures rather than parse errors as well because the library simply swallows many errors that occur in parsing (see: nearly any call to util.EntryParser where the error is ignored). Many of the Validate methods also seem to expect specific hard-coded values, which means they aren't generally useful.

How can we reproduce the problem?
I can share a sample file here once I've verified it's free of any sensitive information. This should be enough to reproduce the behavior.

I'm on the Slack and am hoping to start a conversation there about making the parser more accepting as my team needs to be able to ingest variable-length BAI II files for work we're working on this quarter. We're happy to make the necessary changes and open a PR, but based on what I've heard it sounds like you may be looking to improve this library soon anyway. Let me know, thanks!