myleshorton / go-charset Goto Github PK

Summary:
When using iconv to Tranlate UTF-8 to ASCII//TRANSLIT the 'é' is not 
translated to 'e' (like expected). Instead, a questionmark '?' is returned.
I have tried to debug this and I can see that after C.iconv is called, cScrach 
contains the '?' already. When not using //TRANSLIT, the 'é' is being 
translated to 'xx', where 'x' is the invalid char I used, and I guess it's 
placed twice because 'é' is a multibyte (2 bytes) UTF-8 character.

Used code (note that the invalid char is 'x', NOT '?'):
package main

import (
    "code.google.com/p/go-charset/charset/iconv"
    "encoding/hex"
    "log"
    "fmt"
)

func main() {
    input := "AéA"
    t, err := iconv.Translator("ASCII//TRANSLIT", "UTF-8", 'x')
    if err != nil {
        log.Fatalf("Coult not get charset translator from UTF-8 to ASCII. Got error: %s\n", err)
        return
    }
    fmt.Print(hex.Dump([]byte(input)))
    n, cdata, err := t.Translate([]byte(input), true)
    if err != nil {
        log.Fatalf("Could not translate string '%s' to ASCII. Got error: %s\n", input, err)
    }
    fmt.Print(hex.Dump(cdata))
    output := string(cdata)
    log.Printf("Translated %d characters from UTF-8 ('%s') to ASCII ('%s')\n", n, input, output)
}


Result when running:
geertjohan@VirtKubuntu:~$ iconvtesting 
00000000  41 c3 a9 41                                       |A..A|
00000000  41 3f 41                                          |A?A|
2012/06/20 09:47:29 Translated 4 characters from UTF-8 ('AéA') to ASCII ('A?A')

The biggest change to Go in regards to this package that wasn't fixable using 
gofix is the addition of a 'rune' type. With the attached patch the tests run.

Thanks,
Samuel

Used code:
package main

import (
    "code.google.com/p/go-charset/charset/iconv"
    "encoding/hex"
    "log"
    "fmt"
)

func main() {
    input := "é"
    t, err := iconv.Translator("ASCII//TRANSLIT", "UTF-8", 'x')
    if err != nil {
        log.Fatalf("Coult not get charset translator from UTF-8 to ASCII. Got error: %s\n", err)
        return
    }
    fmt.Print(hex.Dump([]byte(input)))
    n, cdata, err := t.Translate([]byte(input), true)
    if err != nil {
        log.Fatalf("Could not translate string '%s' to ASCII. Got error: %s\n", input, err)
    }
    fmt.Print(hex.Dump(cdata))
    output := string(cdata)
    log.Printf("Translated %d characters from UTF-8 ('%s') to ASCII ('%s')\n", n, input, output)
}


Go version:
go version go1.0.1


Error:
geertjohan@VirtKubuntu:~$ iconvtesting 
00000000  c3 a9                                             |..|
panic: runtime error: slice bounds out of range

goroutine 1 [running]:
code.google.com/p/go-charset/charset/iconv.(*iconvTranslator).Translate(0xf84003
e400, 0xf840045112, 0x0, 0x200000001, 0x0, ...)
        /tmp/go-build714705259/code.google.com/p/go-charset/charset/iconv/_obj/iconv.cgo1.go:157 +0x4ea
main.main()
        /home/geertjohan/Workspaces/WorkspaceGo/devpath/src/iconvtesting/iconvtesting.go:20 +0x24c

goroutine 2 [syscall]:
created by runtime.main
        /home/geertjohan/Applications/go/src/pkg/runtime/proc.c:221

What steps will reproduce the problem?
1. import go-charset with both charset and charset/data package
2. perform conversion on a string containing ASCII characters
3. check the result

What is the expected output? 
The ASCII characters should be there with the converted big5 characters.

What do you see instead?
The ASCII characters disappeared.

What version of the product are you using? On what operating system?
go1.1.1 on Windows XP without iconv


Please provide any additional information below.

Example (runnable example attached):
r, err := charset.NewReader("big5", strings.NewReader("123 English 
\xb4\xfa\xb8\xd5"))
if err != nil {
    log.Fatal(err)
}
result, err := ioutil.ReadAll(r)
if err != nil {
    log.Fatal(err)
}
fmt.Printf("%s\n", result)

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?
2d3a4e0fb600

Please provide any additional information below.

I want make an pull-request from my cloned hg-repository (in github way) but,
can't find how to. (Sorry, I'm not familiar with the hg)

Please pull it from my clone;

       https://code.google.com/r/ff4500-go-charset

I also attach a patch of my work.

Thanks,
Homin Lee.

If you use the example for the NewWriter provided in the docs and the string 
contains a '%', the % is used as a format string, because the signature of 
Fprintf is:

Fprintf(w io.Writer, format string, a ...interface{}) (n int, err error)

See:

https://code.google.com/p/go-charset/source/browse/charset/example_test.go#32

What is the expected output? What do you see instead?

If I want to convert e.g. the string '100% free', I get: 100%!f(MISSING)ree.

If I use fmt.Fprint(w, s) instead of fmt.Fprintf(w, s), the result is correct. 
I think, the docs / the example should be updated.

> What steps will reproduce the problem?

Creating a new reader with windows-1255 as the input charset.

> What is the expected output? What do you see instead?

Windows-1255 is a standard encoding (Hebrew), IMHO it should be supported :)

To reflect recent Go API changes. 

I'm unable to do a merge request, don't know how that works on Google Code, 
here's the commit:

http://code.google.com/r/saschapeilicke-go-charset/source/detail?r=de31d93b0c0cf
dc8b7a617430bf595a85af1d93c

What steps will reproduce the problem?
1. Install Go 1.2 on OS X Mavericks
2. Install the default Xcode command line tools (run "xcode-select --install" 
from Terminal)
3. Build go-charset

What is the expected output? What do you see instead?
I'd expect building to work, but instead it fails with:

# code.google.com/p/go-charset/charset/iconv
ld: warning: directory not found for option '-L/opt/local/lib'
ld: warning: directory not found for option '-L/opt/local/lib'
duplicate symbol _iconv_open_error in:
    $WORK/code.google.com/p/go-charset/charset/iconv/_obj/iconv.cgo2.o
    $WORK/code.google.com/p/go-charset/charset/iconv/_obj/list_query.cgo2.o
duplicate symbol _iconv_error in:
    $WORK/code.google.com/p/go-charset/charset/iconv/_obj/iconv.cgo2.o
    $WORK/code.google.com/p/go-charset/charset/iconv/_obj/list_query.cgo2.o
ld: 2 duplicate symbols for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

What version of the product are you using? On what operating system?
Fails with tip.

Please provide any additional information below.
This issue can be fixed by removing the exports of iconv_open_error and 
iconv_error from list_query.go. It looks like these symbols aren't used in this 
file at all, so removing the export should be fine. See the attached patch, 
which resolves this issue for me.

What steps will reproduce the problem?
 On Mac OS X, hg clone, cd charset/iconv, make -k

What is the expected output? What do you see instead?
make -k 
CGOPKGPATH=go-charset.googlecode.com/hg/charset cgo --  iconv.go 
error: 'EILSEQ' undeclared (first use in this function)
error: (Each undeclared identifier is reported only once
error: 'EINVAL' undeclared (first use in this function)
error: 'E2BIG' undeclared (first use in this function)
unresolved names
make: *** [_obj/_cgo_run] Error 2
make: Target `all' not remade because of errors.

What version of the product are you using? On what operating system?


Please provide any additional information below.

Add the following line to iconv/iconv.go will fix the problem:

//#include <errno.h>

Here, the library prints to stdout 
http://code.google.com/p/go-charset/source/browse/charset/codepage.go#122

It should be behind if DEBUG, I think.