evanmiller / fmptools Goto Github PK

Hi,
Thanks a lot for making this tool, it's amazing!

I'm running into the following issue when running the fmp2json command:

Version: File Maker Pro 12.0
 **** UNRECOGNIZED CODE 0x27 @ [6] *****
Error code: 11

I'm running v0.2.1 on ubuntu 18.04 amd64.

I read on #3 that you suggested using fmpdump to get more information about where this is failing but I'm getting the same when running fmpdump that I did with fmp2json. Uncommenting the code that you suggested in #3 gives me this error:

Version: File Maker Pro 12.0
 **** UNRECOGNIZED CODE 0x27 @ [6] *****
ERROR processing block, reporting partial results...
== 0 -> [ BLOCK 2 ] -> 3 ==
        [ Len: 4076 ]
-- data simple (0x1B): [1] --
Error code: 11

I can open the file in Filemaker Pro 19 with no problems. The database is password protected but I don't think that matters as it is working for other files that are also password protected.
I can't upload the file because it has sensitive information but I might be able to upload just the data where the process is failing if that would be useful.

Any help would be appreciated!

FP3 is assumed to be MacRoman. FP7 and later is assumed Windows-1252. Either locate the file bytes that indicate the encoding, or offer an option to the user to specify the encoding. (Note strings returned to the client are always UTF-8.)

I compiled and installed the application, but I can't find the tools installed, the installer tells me they should be in /usr/local/bin

I am using Ubuntu 21.10.

Where am I doing wrong?

Hello

I like test with an export to sqlite3 on a debian 11.

but it say checking for sqlite3_open_v2 in -lsqlite3... no

I dont't know where to loook to fix it

here the log from ./configure

checking for workbook_new in -lxlsxwriter... yes
checking for yajl_gen_alloc in -lyajl... no
checking for sqlite3_open_v2 in -lsqlite3... no
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: executing depfiles commands
config.status: executing libtool commands

with python3 , i have proper access to sqllite3 (v2 level).

root@debian11:/home/ladmin/patois/fmptools-master# python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.

import sqlite3
con = sqlite3.connect('example.db')
cur = con.cursor()
cur.execute('''CREATE TABLE stocks
... (date text, trans text, symbol text, qty real, price real)''')
<sqlite3.Cursor object at 0x7f8319324730>
cur.execute("INSERT INTO stocks VALUES ('2006-01-05','BUY','RHAT',100,35.14)")
<sqlite3.Cursor object at 0x7f8319324730>
con.commit()
any changes have been committed or they will be lost.
File "", line 1
any changes have been committed or they will be lost.
IndentationError: unexpected indent
con.close()

Hi,

I've downloaded the source code from the git today and compiled it on a Mac Min (Apple M1 chip) with 8 GB physical memory.

The data file is a 9.3GB .fmp12 data file. I have been running it for over a couple of hours now (still running). Is there anyway to speed up the performance?

Thanks.

Hello, thanks a lot for your work

while reading this, i'm wondering if this could be used to including deleted records i have a fp5 file and hoping it will include accidentialy deleted records.

In https://github.com/evanmiller/fmptools/blob/main/HACKING

fp5 sector layout:

    Offset  Length  Value
    0       1       Deleted? 1=Yes 0=No

in src/fmp.h it is declared as int deleted;

in src/block.c

src/block.c:    // block->deleted = sector[0];
src/block.c:    block->deleted = 0;

so my idea was forcing each block to be not deleted :-)
but after compiling i'll get the same amount of data as without the patch

thanks for any hints

Fuzz-testing indicates that the chunks are leaking, e.g.

Live Heap Allocations: 1270338425 bytes in 19105005 chunks; quarantined: 265282292 bytes in 256984 chunks; 11596 other chunks; total chunks: 19373585; showing top 95% (at most 8 unique contexts)
1062898432 byte(s) (83%) in 16607788 allocation(s)
    #0 0x10c8c86b7 in wrap_calloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5c6b7)
    #1 0x10b1ae5ca in process_block_v7 block.c:29
    #2 0x10b1b3ddf in process_block block.c:412
    #3 0x10b1b7d84 in process_blocks fmp.c:220
    #4 0x10b1bc3f2 in fmp_list_tables list_tables.c:48
    #5 0x10b1adfb3 in LLVMFuzzerTestOneInput fuzz_fmp.c:9
    #6 0x10b1ce6b8 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) FuzzerLoop.cpp:576
    #7 0x10b1cddf2 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) FuzzerLoop.cpp:485
    #8 0x10b1d011a in fuzzer::Fuzzer::MutateAndTestOne() FuzzerLoop.cpp:713
    #9 0x10b1d0de5 in fuzzer::Fuzzer::Loop(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) FuzzerLoop.cpp:844
    #10 0x10b1c6a83 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:765
    #11 0x10b1f07a2 in main FuzzerMain.cpp:20
    #12 0x7fff67ff0cc8 in start (libdyld.dylib:x86_64+0x1acc8)

44584960 byte(s) (3%) in 696640 allocation(s)
    #0 0x10c8c86b7 in wrap_calloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5c6b7)
    #1 0x10b1ae5ca in process_block_v7 block.c:29
    #2 0x10b1b3ddf in process_block block.c:412
    #3 0x10b1b7d84 in process_blocks fmp.c:220
    #4 0x10b1bb11d in fmp_list_columns list_columns.c:88
    #5 0x10b1ae0ba in LLVMFuzzerTestOneInput fuzz_fmp.c:13
    #6 0x10b1ce6b8 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) FuzzerLoop.cpp:576
    #7 0x10b1cddf2 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) FuzzerLoop.cpp:485
    #8 0x10b1d011a in fuzzer::Fuzzer::MutateAndTestOne() FuzzerLoop.cpp:713
    #9 0x10b1d0de5 in fuzzer::Fuzzer::Loop(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) FuzzerLoop.cpp:844
    #10 0x10b1c6a83 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) FuzzerDriver.cpp:765
    #11 0x10b1f07a2 in main FuzzerMain.cpp:20
    #12 0x7fff67ff0cc8 in start (libdyld.dylib:x86_64+0x1acc8)

Hi,

Thanks for these tools!

Working with git master at commit 4acc820. When trying to convert a .fmp12 file to JSON, I get the following error:

amoe@cslp019129 $ ./fmp2json foo.fmp12 out.json
**** UNRECOGNIZED CODE 0xc0 @ [1189] *****
Error code: 11

What would this mean? I think the file is valid. I can't provide the file itself but I can provide a hex dump of regions of the file if you need it.

Cheers!

Hi. I am trying to convert a .fp7 file containing about 2500 cooking recipes. Its structure is fairly simple (8 text fields + 1 image field), and I have the following issue with both fp2excel and fp2sqlite:

• Text fields with short contents (title, source URL, category, ...) seem correctly extracted
• Text fields with a longer content (ingredients, recipe itself) are either partially extracted, or most of the times empty.

I don't know if you still maintain these (very useful) tools, but if you are, I would be glad if you could fix this issue.
I can provide the original file (it's more than 4GB because of the embedded images), as well as the output files, but I'l have to create a share to do so - so I'll wait for an answer before doing so.

I wrote a small Dockerfile to install fmptools.

I though it could be useful for others:

FROM debian:12.1

RUN apt-get update && apt-get install -y \
    autoconf \
    libxlsxwriter-dev \
    gettext \
    libtool \
    libyajl-dev \
    sqlite3 \
    libsqlite3-0 \
    libsqlite3-dev \
    git \
    make \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

RUN git clone https://github.com/evanmiller/fmptools.git /fmptools

WORKDIR /fmptools

RUN autoreconf -i -f 
RUN ./configure
RUN make 
RUN make install

RUN ldconfig

It could probably be improved but it is working. 🙂

Hi,

Thank you for your great tools. I am trying to dump the content of an fmp12 file made with FileMaker Pro 7.0. Unfortunately, fmpdump exits with the following msg:
(I have uncommented the additional debug output lines mentioned in #3)

Version: File Maker Pro 7.0
Released: 1999-10-01
== 0 -> [ BLOCK 2 ] -> 3 ==
        [ Len: 4076 ]
-- data simple (0x1B): [1] --
-- field (0x04): [32] => "XzÝYYZ" --
 **** UNRECOGNIZED CODE 0xdb @ [64] *****
ERROR processing block, reporting partial results...
== 2 -> [ BLOCK 3 ] -> 0 ==
        [ Len: 4076 ]
-- PUSH 0x28 [ 0x80 0x03  ] --
-- PUSH 0x20 [ 0x01  ] --
-- PUSH 0xE0 [ 0x05  ] --
-- PUSH 0x20 [ 0x00  ] --
-- PUSH 0x38 [ 0x01 0x07 0xE5 0x06 0xCF 0x07 0xED 0x02 0x26 0x01 0x01 0x07 0xE5 0x06 0xCF 0x07 0xED 0x08 0x8D 0x06 0xCF 0x07 0xBD 0x07 0x8D 0x08 0x75 0x07 0x1F 0x08 0x35 0x01 0x09 0x02 0x3E 0x07 0xE5 0x08 0x05 0x07 0x1F 0x07 0x53 0x07 0xBD 0x07 0x8D 0x06 0xF7 0x07 0x73 0x02 0x3F  ] --
Error code: 11

Is there anything I can do about this?

EDIT:
Dump of block 2 can be found here: https://gist.github.com/sonovice/6902dc351daa8b938c02d2459ceb9db8

EDIT2:
An important information might be that the dataset consists of actually 3 fmp12 files that are somehow linked.

Hi,

I'm truing to do an implementation in Elixir, following the HACKING document, but I get unknown values for the first byte of some chunks. I tried to follow the C code but I have never learned C and it is hard!

I found this piece of code and it looks to me that the first block is ignored as you start on block index 1. Block 0 is not the header block.

    int next_block = 2;
    int *blocks_visited = calloc(file->num_blocks, sizeof(int));
    do {
        fmp_block_t *block = file->blocks[next_block-1];

Can you confirm?

Thank you