winstonsmith934 / fixedformat4j Goto Github PK

Sign overpunch is used quite heavily on mainframe systems, where they don't 
want to waste a character representing a negative number. 

Instead of a negative indicator (i.e. an initial "-" character), the final 
character is changed from a number to an associated letter).

See this link for a fuller / better description:

http://en.wikipedia.org/wiki/Signed_overpunch

What is the expected output? What do you see instead?

-1234.1234 would become 1234.123M


What version of the product are you using? 

Using current trunk version (Revision: 180)

On what Java version?

N/A


Please provide any additional information below.

I think this can be easily achieved by adding an additional Sign type (i.e. 
Sign.OVERPUNCH). 


Please find attached an enhanced Sign.java and an additional associated test 
class to prove some scenarios work (i.e. formatting using overpunch with 
short/integer/long/float/double/bigdecimal).

AbstractDecimalFormatter.FORMATTER is a static reference to a "known to be non 
thread-safe" class and then uses without synchronization.

Possible hideous side-effects may surface from this :-(

It is wrong to apply paddingChar as decimal fractions. It should be hard coded 
'0' as before.

The javadoc for com.ancientprogramming.fixedformat4j.annotation.Field says that 
the default value for align() is Align#RIGHT but it's Align#LEFT in the code.

Today the api looks for annotations on the concrete class that is to be loaded.

It would be nice if the manager could look further op the hierarchy 

What steps will reproduce the problem?
1. Define a bean with a BigDecimal field with the following annotations:

@Field(offset = 1, length = 16, align = Align.RIGHT, paddingChar = '0')
@FixedFormatNumber(sign = Sign.APPEND)

2. Define an instance of this bean and set this field to a negative value 
(e.g. new BigDecimal("-1")

3. Construct a FixedFormatManagerImpl and export the object.

What is the expected output?

000000000000100-

What do you see instead?

00000000000-100+

(FixedFormatManagerImpl will correctly import the text above)

What version of the product are you using? On what Java version?

fixed format version: 1.3.1
java: 1.6

Please provide any additional information below.

What steps will reproduce the problem?
1. Load a record with a numeric field and an empty string to read.
2. Export the same record
3. Compare the two strings

What is the expected output? What do you see instead?

Since there is no data to be read, I would expect the numeric variables to be 
null and not zero. When I export the same records, I find zeros where there was 
none in the initial data. There is no way to differenciate no data from zeros.

What version of the product are you using? On what Java version?
fixedformat4j 1.3.4, java 1.6

Please provide any additional information below.

The problem comes from the remove functions in the sign enum.

      if (StringUtils.isEmpty(result)) {
        result = "0";
      }

Currently, the line:

    String rawString = obj != null ? FORMATTER.format(obj) : ZERO_STRING;

in AbstractDecimalFormatter formats to a string without prior setting of 
fraction digits or a rounding mode. Later on, there is applied a clipping (or 
appending) of digits, according to the requested digits.

The format should do the rounding.

Change <T extends Number> back to <T>.
The restriction is unnecessary here and does nothing but break existing code :-(

This API breaking change (for those that extended the formatter in own code 
with something not a number) came out in 1.3.3.

What steps will reproduce the problem?
1. define a record with a BigDecimal field defined like this
    @Record
    public class TestRecord {
        private BigDecimal bd;

        @Field(offset = 1, length = 5, align = Align.RIGHT, paddingChar = '0')  
        @FixedFormatDecimal(decimals = 2, useDecimalDelimiter = false)
        @FixedFormatNumber(sign = Sign.NOSIGN)
        public BigDecimal getAmount()
        {
            return bd;
        }

        public void setAmount(BigDecimal amt)
        {
            this.bd = amt;
        }
    }

2. load a string with a field having a value of just cents (like 0.05) 

String data = "00005"; // value is 0.05
TestRecord tr = manager.load(TestRecord.class, data);

3. check the value of the field with tr.getAmount()

What is the expected output? What do you see instead?

the value of the variable should be 0.05 instead is 5.00

What version of the product are you using? On what Java version?

fixedformat4j-1.3.4 jdk 1.6

Please provide any additional information below.

The problem is within the Sign.NOSIGN.{...}.remove() and 
AbstractDecimalFormater.getStringToConvert the first method remove all padding 
chars from the string so the result is "5"

then the second method perform the following test:
      true (2)           false (1)
if (decimals > 0 && string.length() >= decimals) {
...
      } else {
        toConvert = string; // return just "5"
      }

with negative numbers is even worst because it throws exceptions

I attached a JUnit test file I used which should reproduce most of the issues

I did a small fix into AbstractDecimalFormater is not the best solution, more a 
workaround

regards
Angelo

Add this:

  public void testFormatWithSpecialPaddingChar() {
    assertEquals("+XXXX1005X", formatter.format(new BigDecimal(100.5), new FormatInstructions(10, Align.RIGHT, 'X', null, null, new FixedFormatNumberData(Sign.PREPEND, DEFAULT_POSITIVE_SIGN, DEFAULT_NEGATIVE_SIGN), new FixedFormatDecimalData(2, false, '.', RoundingMode.UNNECESSARY))));
  }


to TestBigDecimalFormatter as see it fails.

Some times, for me very often, fixed format files can have portions that
will have fixed data, example:

00100000000A20080101
001       - [3] Fixed 001 type of register
00000000A - [9] Spanish id card code
20080101  - [8] Data of request

I create a bean for this, String for document, Data for request time, so
001 need to be go in record, it's define the tipe of record, in the Java
world this will be me my bean.class.
Now i overcome this creating a annotated bean property with a default value
to the fixed data, this works, but despite developers looking extrange
properties (some times more that one fixed data can go in a line), so
annotating it at bean level would simply the thinks, and can do validation
on load data.
Another posibility that in my production beans don't arise is inner padding
on record, so a line can have spared fields, with a record padding so
actual padding only are done en the end of data and not at holes, it's
simple but breaks current implementation and not required at the moment.

I create two new annotations Fixed and Fixeds: 
Fixed need to specify offset, lenght, value, and optionally paddinfChar
that defaults to ' '.
I will put the value at offset and padding to length, simple :).
Fixeds is a array of Fixed annotations.

Pathed Formatter implementation is also in patch file.

Also i attach test cases for Junit for this implementation, i also run all
the tests in the project with the new implementation and runs fine, also i
run a lot of other tests over fixed format data for my production entities,
i remove my annottations from test and make it extend TestCase becouse i
see all your tests are done in this manner.

I spect that will usuful for anybody, so i'm happy to get this included in
trunk, or in a revision so for now i run my development with patched
fixedformat4j.

The fixeds and new develop on support hibernate validation for field and
bean can save great time on developers creating new file format manager,
for that we have a generic file that reads from stream or file, and loads
and export, have support for multi-header and multi-footers, it focus on
our need so i don't think will be suitable for public masses so if anybody
interested open a discusion about that, for good direction and a good
implementation for most people posible.

Regards.

In our flat file , the last column’s length is 30 (and offset is 120)
but the length of the values passed is sometimes less than 30.

In this case, the parser ignores the last column and shows the value as 
‘null’.

We tried paddingChar=’ ‘ to pad with spaces , but it didn’t help.

I have a flat file feed that contains a configuration of a product. It is 
possible for the user to add 0 or more options to this configuration. The way 
this is represented in our flat file is to have a code-description pair 
repeated 10 times. As far as I can tell, the only way to accomplish this with 
the current version is to create 10 individual fields (option1, option2, 
etc...) and make each of them a nested record. What would be great is if there 
was an annotation that would allow for a repeatable field or nested record and 
it would return an array of whatever the field was.

Could it be possible? I'm in a situation where every field is right aligned in 
my file. I'ts just a little annoying having to specify each alignment on each 
field of each record.

As the bean specification denotes that boolean getters starts with "is" it
would be nice to be able to annotate those as well.

Added onbehalf of Agusti:

The Record annotation could include a flag to tell how strict validation
should be; for example fail if actual record length does not exactly match
the expected record length or try to parse as fields as possible.

Added onbehalf of Agusti:
The message on parseexception contains lots of information, but I wished to
choose what information to show the user and the format of the message.

So it would be a great idea if you kept the data that you pass into the
constructor as class fields and make them available by getter methods. So
that client code would be able to catch the ParseException, retrieve the
pieces of information it has interest in and build a custom error message
in a convenient way.

The suggestion is to make readDataAccordingFieldAnnotation protected.

I am getting a Format exception when trying to export object having a null
date value. 

What steps will reproduce the problem?
1. Annotate getter method using the @FixedFormatNumber(sign=Sign.PREPEND, 
positiveSign='0') 

What is the expected output? What do you see instead?
You would expect the output from the formatter to use '0' as positive sign. 
0000100 for positive 100 and -000100 for -100 

What version of the product are you using? On what Java version?


Please provide any additional information below.
Sign.PREPEND should use FixedFormatNumberData.positive sign and not '+'. A 
propesed solution would be:


  PREPEND {
    public String apply(String value, FormatInstructions instructions) {
      String sign = StringUtils.substring(value, 0, 1);
      if ("-".equals(sign)) {
        value = StringUtils.substring(value, 1);
        sign = instructions.getFixedFormatNumberData().getNegativeSign().toString();
      } else {
        sign = instructions.getFixedFormatNumberData().getPositiveSign().toString();
      }
      String result = instructions.getAlignment().apply(value, instructions.getLength(), instructions.getPaddingChar());
      return sign + StringUtils.substring(result, 1);
    }

What steps will reproduce the problem?
1. NestedRecordUsage sample not working.Its throwing exception
2. Run above sample
3.

What is the expected output? What do you see instead?
Expected output is Nested class being parsed.
What I see is
Exception in thread "main" 
com.ancientprogramming.fixedformat4j.format.ParseException: failed to 
parse 'bar       001232008-10-21' at offset 6 as test.BasicRecord 
from 'foo  bar       001232008-10-21'. Got format instructions from 
test.NestedRecord.getRecord. See details{FormatContext{offset=6, 
dataType=test.BasicRecord, 
formatter=com.ancientprogramming.fixedformat4j.format.impl.ByTypeFormatter}
, FormatInstructions{length=25, alignment=LEFT, paddingChar=' ', 
fixedFormatPatternData=FixedFormatPatternData{pattern='yyyyMMdd'}, 
fixedFormatBooleanData=FixedFormatBooleanData{trueValue='T', 
falseValue='F'}, fixedFormatNumberData=FixedFormatNumberData
{signing=NOSIGN, positiveSign='+', negativeSign='-'}, 
fixedFormatDecimalData=FixedFormatDecimalData{decimals=2, 
useDecimalDelimiter=false, decimalDelimiter='.'}}}
    at 
com.ancientprogramming.fixedformat4j.format.impl.FixedFormatManagerImpl.rea
dDataAccordingFieldAnnotation(FixedFormatManagerImpl.java:206)
    at 
com.ancientprogramming.fixedformat4j.format.impl.FixedFormatManagerImpl.loa
d(FixedFormatManagerImpl.java:95)
    at test.NestedRecordUsage.main(NestedRecordUsage.java:23)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at com.intellij.rt.execution.application.AppMain.main
(AppMain.java:90)
Caused by: 
com.ancientprogramming.fixedformat4j.exception.FixedFormatException: 
com.ancientprogramming.fixedformat4j.format.impl.ByTypeFormatter cannot 
handle datatype[test.BasicRecord]. Provide your own custom FixedFormatter 
for this datatype.
    at 
com.ancientprogramming.fixedformat4j.format.impl.ByTypeFormatter.actualForm
atter(ByTypeFormatter.java:89)
    at 
com.ancientprogramming.fixedformat4j.format.impl.ByTypeFormatter.parse
(ByTypeFormatter.java:68)
    at 
com.ancientprogramming.fixedformat4j.format.impl.FixedFormatManagerImpl.rea
dDataAccordingFieldAnnotation(FixedFormatManagerImpl.java:204)
    ... 7 more

What version of the product are you using? On what Java version?
fixedformat4j-1.2.2
JDK 1.5

Please provide any additional information below.

I am using Groovy and don't have getter and setters in my source code.

It would be great to use the annotations on the fields.

What steps will reproduce the problem?

If a field is mapped like this:

@Field(offset = 42, length = 20, paddingChar = '0')
@FixedFormatPattern("yyyyMMddHHmmss")
public Date getDatumErstellung() {
     return datumErstellung;
}

then input like 20120912111100000000 will become 201209121111 and therefore not 
parseable to the format yyyyMMddHHmmss. This is because the DateFormatter 
removes all paddingChars including the seconds field in this example because 
they are 00.

What do you see instead?

I think that the DateFormatter should only remove the paddingChars up until the 
length of the date pattern. It comes down the what happens in this method of 
AbstractFixedFormatter:

String getRemovePadding(String value, FormatInstructions instructions) {
    return instructions.getAlignment().remove(value, instructions.getPaddingChar());
  }

(a method which, by the way, should be protected)

I would override that in DateFormatter and change it to :


protected String getRemovePadding(String value, FormatInstructions 
instructions) {
        String newValue = instructions.getAlignment().remove(value, instructions.getPaddingChar());
        return instructions.getAlignment().apply(newValue, instructions.getFixedFormatPatternData().getPattern().length(), instructions.getPaddingChar());
    }

What version of the product are you using? On what Java version?

1.3.1, JDK 6


Please provide any additional information below.

It should be possible to express signing for numbers.

This currently prevents the api for handling negative numbers correct

When primitive types are used in anotated clasess the parse breaks, so i
add this to add support for primitive in ByTypeFormatter:

  static {
    KNOWN_FORMATTERS.put(String.class, StringFormatter.class);
    KNOWN_FORMATTERS.put(Integer.class, IntegerFormatter.class);
    KNOWN_FORMATTERS.put(int.class, IntegerFormatter.class);
    KNOWN_FORMATTERS.put(Long.class, LongFormatter.class);
    KNOWN_FORMATTERS.put(long.class, LongFormatter.class);
    KNOWN_FORMATTERS.put(Date.class, DateFormatter.class);
    KNOWN_FORMATTERS.put(Character.class, CharacterFormatter.class);
    KNOWN_FORMATTERS.put(char.class, CharacterFormatter.class);
    KNOWN_FORMATTERS.put(Boolean.class, BooleanFormatter.class);
    KNOWN_FORMATTERS.put(boolean.class, BooleanFormatter.class);
    KNOWN_FORMATTERS.put(Double.class, DoubleFormatter.class);
    KNOWN_FORMATTERS.put(double.class, DoubleFormatter.class);
    KNOWN_FORMATTERS.put(Float.class, FloatFormatter.class);
    KNOWN_FORMATTERS.put(float.class, FloatFormatter.class);
    KNOWN_FORMATTERS.put(BigDecimal.class,  BigDecimalFormatter.class);
  }

So it's breaking becouse it can't find the method, so i recently send a
patch to use method type signature insead of data signature, this patch
will solve this, and open the posibility to extend anotated classes.

I attach the patch file.

What about making FixedFormatter a parameterized class using generics?

I hate it when I see these Object in, Object out apis.

http://code.google.com/p/fixedformat4j/source/browse/tags/1_1_0/fixedformat4j/sr
c/main/java/com/ancientprogramming/fixedformat4j/format/FixedFormatter.java

Currently there is no support for static nested or inner classes.
The formatter just throws an exception complaining about a missing default
constructor.

What steps will reproduce the problem?
1. When a negativ number -42 is formatted, the string "-42" is sent to 
Sign.APPEND.apply
2. First line in apply() will look at the END of the string to detect symbol
3. It will think it should be "+" as the minus in front of the string isn't seen
4. Result: "-42+"

What is the expected output? What do you see instead?

"42-"

Patch with test and fix attached.

What steps will reproduce the problem?

Create the following 2 files and run the test case.

DecimalRecord.java:

import java.math.BigDecimal;

import com.ancientprogramming.fixedformat4j.annotation.Align;
import com.ancientprogramming.fixedformat4j.annotation.Field;
import com.ancientprogramming.fixedformat4j.annotation.FixedFormatDecimal;
import com.ancientprogramming.fixedformat4j.annotation.Record;

@Record
public class DecimalRecord {

    private double doubleValue;
    private BigDecimal bigDecimalValue;

    public DecimalRecord(double doubleValue, BigDecimal bigDecimalValue) {
        this.doubleValue = doubleValue;
        this.bigDecimalValue = bigDecimalValue;
    }

    @Field(offset = 1, length=8, align=Align.RIGHT)
    @FixedFormatDecimal(decimals=4, useDecimalDelimiter=true)
    public double getDoubleValue() {
        return doubleValue;
    }

    @Field(offset = 9, length=8, align=Align.RIGHT)
    @FixedFormatDecimal(decimals = 4, useDecimalDelimiter=true)
    public BigDecimal getBigDecimalValue() {
        return bigDecimalValue;
    }

}


FixedFormatDecimalTest.java:

import static org.junit.Assert.assertEquals;

import java.math.BigDecimal;

import org.junit.Test;

import com.ancientprogramming.fixedformat4j.format.FixedFormatManager;
import com.ancientprogramming.fixedformat4j.format.impl.FixedFormatManagerImpl;

public class FixedFormatDecimalTest {

    @Test
    public void fixedFormatDoesntTrimLastDecimals() {

        String stringValue = "12.34567";        
        DecimalRecord record = new DecimalRecord(12.34567, new BigDecimal(stringValue));
        String expectedRecord = " 12.3457" + " 12.3457";
        FixedFormatManager fixedFormatManager = new FixedFormatManagerImpl();
        String formattedRecord = fixedFormatManager.export(record);
        assertEquals(expectedRecord, formattedRecord);

    }

}



What is the expected output? What do you see instead?
Expected:  12.3457 12.3457
Actual:  12.3460 12.3460


What version of the product are you using? On what Java version?
fixedformat4j-1.3.1.jar

Java on Mac:
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-10M3065)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)

Java on Windows (same problem):
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode)


Please provide any additional information below.

The problem seems to be that the first call in AbstractDecimalFormat.asString(T 
obj, FormatInstructions instructions):

    String rawString = obj != null ? FORMATTER.format(obj) : ZERO_STRING;

Returns a rawString that has been rounded to three decimals (12.346) and later 
in the method a trim to the specified number of decimals (4) is done, but by 
then the last decimal is already lost.

Looking at the junit test cases for fixedformat4j I notice that there doesn't 
seem to be any test cases that uses more than 3 decimals, so I added a couple 
to TestBigDecimalFormatter.java and TestDoubleFormatter.java that exposes the 
problem:

  public void testFormatBigDecimalWith4DecimalsKeepAllDecimalsNoDecimalDelimiter() {
    assertEquals("+00012345612", formatter.format(new BigDecimal(1234.5612), new FormatInstructions(12, Align.RIGHT, '0', null, null, new FixedFormatNumberData(Sign.PREPEND, DEFAULT_POSITIVE_SIGN, DEFAULT_NEGATIVE_SIGN), new FixedFormatDecimalData(4, false, '.'))));      
  }

  public void testFormatBigDecimalWith5DecimalsTo4DecimalsNoDecimalDelimiter() {
    assertEquals("+00012345612", formatter.format(new BigDecimal(1234.56121), new FormatInstructions(12, Align.RIGHT, '0', null, null, new FixedFormatNumberData(Sign.PREPEND, DEFAULT_POSITIVE_SIGN, DEFAULT_NEGATIVE_SIGN), new FixedFormatDecimalData(4, false, '.'))));
  }

  public void testFormatBigDecimalWith4DecimalsKeepAllDecimalsUseDecimalDelimiter () {
    assertEquals("+001234.5612", formatter.format(new BigDecimal(1234.5612), new FormatInstructions(12, Align.RIGHT, '0', null, null, new FixedFormatNumberData(Sign.PREPEND, DEFAULT_POSITIVE_SIGN, DEFAULT_NEGATIVE_SIGN), new FixedFormatDecimalData(4, true, '.'))));  
  }

  public void testFormatBigDecimalWith5DecimalsTo4DecimalsUseDecimalDelimiter () {
      assertEquals("+00123.4561", formatter.format(new BigDecimal(123.45612), new FormatInstructions(11, Align.RIGHT, '0', null, null, new FixedFormatNumberData(Sign.PREPEND, DEFAULT_POSITIVE_SIGN, DEFAULT_NEGATIVE_SIGN), new FixedFormatDecimalData(4, true, '.'))));  
  }


  public void testFormatDoubleWith4DecimalsKeepAllDecimals() {
    assertEquals("+001005055", formatter.format(new Double(100.5055), new FormatInstructions(10, Align.RIGHT, '0', null, null, new FixedFormatNumberData(Sign.PREPEND, DEFAULT_POSITIVE_SIGN, DEFAULT_NEGATIVE_SIGN), new FixedFormatDecimalData(4, false, '.'))));

  }

  public void testFormatDoubleWith5DecimalsTo4Decimals() {
    assertEquals("+001005556", formatter.format(new Double(100.55555), new FormatInstructions(10, Align.RIGHT, '0', null, null, new FixedFormatNumberData(Sign.PREPEND, DEFAULT_POSITIVE_SIGN, DEFAULT_NEGATIVE_SIGN), new FixedFormatDecimalData(4, false, '.'))));
  }

When method.invoke() in
FixedFormatManagerImpl.exportDataAccordingFieldAnnotation returns null it
bombs out with NullPointerException when it tries to follow referenced
annotated instances.

What steps will reproduce the problem?
1. if we use FixedFormatDecimal annotation for field
2. useDelimeter = true, decimals = 3
3. the string came without any delimiter

What is the expected output? What do you see instead?
expected output is the string itself without delimitter.
instead the formatter use two decimals after delimeter default

What version of the product are you using? On what Java version?


Please provide any additional information below.

What steps will reproduce the problem?
1. The load function will not accept the generic 
2.
3.

What is the expected output? What do you see instead?


What version of the product are you using? On what Java version?


Please provide any additional information below.

Sting str = Sting("filename");
FlatFileReader<IRPReader> reader = new FlatFileReader<IRPReader>(str);


import java.io.BufferedReader;
import java.io.DataInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStreamReader;

import com.ancientprogramming.fixedformat4j.exception.FixedFormatException;
import com.ancientprogramming.fixedformat4j.format.FixedFormatManager;
import com.ancientprogramming.fixedformat4j.format.impl.FixedFormatManagerImpl;

public class FlatFileReader<T> implements FlatFileReaderI<T> {
    private String dataFileName;
    private FixedFormatManager fixedFormatManager = new FixedFormatManagerImpl();
    private FileInputStream fileInputStream = null;
    private DataInputStream dataInputStream = null;
    private BufferedReader bufferedReader = null;
    private String strLine;
    private T dataClass;

    public FlatFileReader(final String dataFileName) {
        this.dataFileName = dataFileName;
        try {
            fileInputStream = new FileInputStream(dataFileName);
            dataInputStream = new DataInputStream(fileInputStream);
            bufferedReader = new BufferedReader(new InputStreamReader(
                    dataInputStream));
        } catch (Exception e) {// Catch exception if any
            System.err.println("Error: " + e.getMessage());
            e.printStackTrace();
        }

    }
    public void setDataClass(T dataClass){
        this.dataClass = dataClass;
    }
    public T getNext() {
        try {
            while ((strLine = bufferedReader.readLine()) != null) {
                return (T) fixedFormatManager.load(T , strLine);
            }
        } catch (FixedFormatException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
//      return V;
        return null;
    }
}

It would be nice to have a method to load data into an existing instance, i.e. 
provide a '<T> T load(Class<T> clazz, Object target, String data)' method.

What steps will reproduce the problem?
1. Try to parse a file with an invalid date field (2012-03-40, for instance).


What is the expected output? What do you see instead?
I wanted a parse exception. In this particular case, you will get APR-9th.

What version of the product are you using? On what Java version?
1.2.2; 1.5.

Please provide any additional information below.
This could be implemented by giving the developer the option to set the lenient 
property of simple data format to true.