Incompatibility with EC-Lab>=11.50 - BioLogic.MPRfile issue about galvani

[ml-evs]

Thanks for all the suggestions + data sharing here! I think this shows how widely used galvani still is, so we should put in some effort to ensure its ongoing maintenance. Does anyone have a line on @chatcannon so we could potentially get some maintainers added to the repo? (I guess naturally that means I volunteer but would prefer not to do it alone...) To avoid hijacking this issue I will raise a new one based on this comment, but just wanted to encourage people from this discussion to think about it!

[fg-personal]

@PeterKraus, thanks for the interest in this topic. I would be happy to find a python-based open-source solution to parse potentiostat files and a user I know may be willing to share the files you mention with people that already have EC-Lab software on their pc.
Despite the kind reply from the BioLogic technical team (...they know that someone will work it out eventually anyway...), I do want to re-emphasize the importance of handling the data with care and respecting any potential restrictions.

EDIT: after the warning, the user may not be willing to share the data files anymore, I hope somebody else will find a solution.

[PeterKraus]

Thanks for the files, @fg-personal and @tk-4767. I might have someone to take this project up in the coming weeks. I'll keep you posted.

[tk-4767]

With files from version 11.48 I never had a problem

[fg-personal]

I have been trying to use the mpt reader function from galvani, after an mpr→mpt conversion in EC-Lab.
In the process, I found some new field names and I had to explicitely add them to the script.

I had to modify:

def fieldname_to_dtype(fieldname):
    """Converts a column header from the MPT file into a tuple of
    canonical name and appropriate numpy dtype"""

    ...
    elif fieldname in ("time/s", "P/W", "(Q-Qo)/mA.h", "x", "control/V",
                       "control/V/mA", "(Q-Qo)/C", "dQ/C", "freq/Hz",
                       "|Ewe|/V", "|I|/A", "Phase(Z)/deg", "|Z|/Ohm",
                       "Re(Z)/Ohm", "-Im(Z)/Ohm",  #new entries after this
                       "step time/s",  "Cs/uF", "Cp/uF", "Re(Y)/Ohm-1", "Im(Y)/Ohm-1",
                       "|Y|/Ohm-1", "Phase(Y)/deg", "Re(C)/nF", "Im(C)/nF",
                       "|C|/nF", "Phase(C)/deg", "Re(M)", "Im(M)", "|M|",
                       "Phase(M)/deg", "Re(Permittivity)", "Im(Permittivity)",
                       "|Permittivity|", "Phase(Permittivity)/deg", "Re(Resistivity)/Ohm.cm",
                       "Im(Resistivity)/Ohm.cm", "|Resistivity|/Ohm.cm", "Phase(Resistivity)/deg",
                       "Re(Conductivity)/mS/cm", "Im(Conductivity)/mS/cm", "|Conductivity|/mS/cm",
                       "Phase(Conductivity)/deg", "Tan(Delta)", "Loss Angle(Delta)/deg",
                       "Cs-2/uF-2", "Cp-2/uF-2"):
        return (fieldname, np.float_)
    ...
    else:
        raise ValueError("Invalid column header: %s" % fieldname)
    # added "step time/s", (present in CV files)
    # added "Cs/uF", "Cp/uF", "Re(Y)/Ohm-1", "Im(Y)/Ohm-1",... (present in EIS files)

I have also changed the line

fieldnames = next(mpt_file).decode(encoding).strip().split('\t')

to

fieldnames = next(mpt_file).replace(b'\xb5', b'u').decode(encoding).strip().split('\t')

to get rid of the μ, not easily decoded by the software.

As I try to import mpt files in python and to put them into dataframes

mpt = BL.MPTfile(filename_with_path)
df = pd.DataFrame(mpt[0])

I get slightly different headers depending on the EC-Lab version used to generate the original mpr file: EIS examples
I have the impression the mpt interpreter hasn't been updated in a while. I had errors while importing EIS files made with an EC-Lab version from 2019 ("Cs/uF", "Cp/uF" not recognized as valid entries) and I had to use the modified fieldname_to_dtype even for files this old.

I apologize if I'm stating the obvious inadvertently, but I hope this information can be of assistance to someone.

@PeterKraus, here is Data for a dummy resistor (2-electrode config) from the newest version (11.50) 15MOhm EC Lab 11.50 galvani.zip Some metadata has been removed in a text editor, now the files do not open in EC-Lab anymore for some reason, but all the measurement data should still be intact.

I can perform the mpr→mpt conversion for the files here upon request

[schaubhoffmann]

I am not working with EC-lab, but was asked by a collegue to tweak the Biologic.py to make it work for the data from EC-Lab 11.50. I have only used it on a few files so far and it definitely needs further testing. But as someone else might already be working on it, here is what I did so far:

I had to change the VMP_Module_hdr (similar to @dansteingart):

VMPmodule_hdr = np.dtype([('shortname', 'S10'),
                          ('longname', 'S25'),
                          ('something', '<u4'),
                          ('length', '<u4'),
                          ('somethingelse', '<u4'),
                          ('version', '<u4'),   
                          ('date', 'S8')])

Not sure if should be like this or like @dansteingart's - for the files I tested it does not make a difference.
Either way I end up with version numbers 10 for the Data Module and Log Module and 11 for the Data Module, which seems plausible for a new data format.

The initialization of the MPRfile class now of course needs to include the condition data_module['version'] == 11 in the if-statement for the Data Module version. The number of bytes of data before the main data array seems to have changed to 1007.
Like in Data Module version 3 there is a '\x01' before the start of the data.

So this is how it works for our data so far:

# this is for EC-LAB 11.50. data_module version == 11
elif data_module['version'] == 11: 
            column_types = np.frombuffer(data_module['data'][6:], dtype='<u2',
                                         count=n_columns)
            # There are bytes of data before the main array starts
            num_bytes_before = 1007  
            remaining_headers = data_module['data'][6 + 2 * n_columns:num_bytes_before-1]
            main_data = data_module['data'][num_bytes_before:]

I will now have the people actually working with EC-Lab test it.

[fg-personal]

I am not working with EC-lab, but was asked by a collegue to tweak the Biologic.py to make it work for the data from EC-Lab 11.50. I have only used it on a few files so far and it definitely needs further testing. But as someone else might already be working on it, here is what I did so far:

I had to change the VMP_Module_hdr (similar to @dansteingart):

VMPmodule_hdr = np.dtype([('shortname', 'S10'),
                          ('longname', 'S25'),
                          ('something', '<u4'),
                          ('length', '<u4'),
                          ('somethingelse', '<u4'),
                          ('version', '<u4'),   
                          ('date', 'S8')])

Not sure if should be like this or like @dansteingart's - for the files I tested it does not make a difference. Either way I end up with version numbers 10 for the Data Module and Log Module and 11 for the Data Module, which seems plausible for a new data format.

The initialization of the MPRfile class now of course needs to include the condition data_module['version'] == 11 in the if-statement for the Data Module version. The number of bytes of data before the main data array seems to have changed to 1007. Like in Data Module version 3 there is a '\x01' before the start of the data.

So this is how it works for our data so far:

# this is for EC-LAB 11.50. data_module version == 11
elif data_module['version'] == 11: 
            column_types = np.frombuffer(data_module['data'][6:], dtype='<u2',
                                         count=n_columns)
            # There are bytes of data before the main array starts
            num_bytes_before = 1007  
            remaining_headers = data_module['data'][6 + 2 * n_columns:num_bytes_before-1]
            main_data = data_module['data'][num_bytes_before:]

I will now have the people actually working with EC-Lab test it.

Seems promising but I still got some errors:
loop_module['version'] is not equal to 0 anymore and that raises the
ValueError("Unrecognised version for data module: %d" %data_module['version'])
That's a red flag, but allowing: loop_module['version'] to be 10 without raising the error seems to solve the problem. I haven't checked this extensively but the first tests I made look good.

With schaubhoffmann's changes to Biologic.py, the script doesn't work for files created with EC-Lab 11.46 or lower
ValueError: Found b'\x00\x00\x00\x00\x00\x00', expecting start of new VMP MODULE While in the read_VMP_modules function

For the moment, I will keep both a Biologic.py (original file) and a Biologic_SH.py (edits from schaubhoffmann) and switch from one to the other with try and except.

[Dongrun7]

I think the problem arose from the newest version of software

[d-cogswell]

I'm also seeing this error with mpr files from the newest version of EC-Lab.

[dansteingart]

It seems somewhat low-level: the VMPmodule_hdr length conditions in galvani and new ec-lab.mpr files. I pulled and added debug comments to BioLogic.py, so @Dongrun7's error above is reproduced

VMPmodule_hdr = np.dtype([('shortname', 'S10'),
                          ('longname', 'S25'),
                          ('length', '<u4'),
                          ('version', '<u4'),
                          ('date', 'S8')])

yields

b'VMP Set   VMP settings             \xff\xff\xff\xffa\x1a\x00\x00\x00\x00\x00\x00\n\x00\x00\x00'
{'shortname': b'VMP Set   ', 'longname': b'VMP settings             ', 'length': 4294967295, 'version': 6753, 'date': b'\x00\x00\x00\x00\n'}
109
Traceback (most recent call last):
  File "/Users/dan/Code/pithy/code/DS_galvani_futz_2.py", line 22, in 
    mpr_file = MPRfile(fns[0])

  File "/Users/dan/Code/pithy/code/DS_galvani_pull.py", line 403, in __init__
    modules = list(read_VMP_modules(mpr_file))
  File "/Users/dan/Code/pithy/code/DS_galvani_pull.py", line 362, in read_VMP_modules
    raise IOError("""Unexpected end of file while reading data
OSError: Unexpected end of file while reading data
                    current module: b'VMP settings             '
                    length read: 489056
                    length expected: 4294967295

Examining hdr_data we see the date is nonsensical, and it seems date is 8 bytes further out from longname

Through some trial and error in modifying VMPmodule_hdr I was able to get the code to step further...

VMPmodule_hdr = np.dtype([('shortname', 'S10'),
                          ('longname', 'S25'),
                          ('whoknows', 'S4'),
                          ('length', '<u8'),
                          ('version', '<u4'),
                          ('date', 'S8')])

but then gets hung up on versioning

b'VMP Set   VMP settings             \xff\xff\xff\xffa\x1a\x00\x00\x00\x00\x00\x00\n\x00\x00\x0006/21/23'
{'shortname': b'VMP Set   ', 'longname': b'VMP settings             ', 'holder': b'\xff\xff\xff\xff', 'length': 6753, 'version': 10, 'date': b'06/21/23'}
117
b'VMP data  VMP data                 \xff\xff\xff\xffW:\x07\x00\x00\x00\x00\x00\x0b\x00\x00\x0006/21/23'
{'shortname': b'VMP data  ', 'longname': b'VMP data                 ', 'holder': b'\xff\xff\xff\xff', 'length': 473687, 'version': 11, 'date': b'06/21/23'}
6935
b'VMP LOG   VMP LOG                  \xff\xff\xff\xff\xad\x1f\x00\x00\x00\x00\x00\x00\n\x00\x00\x0006/22/23'
{'shortname': b'VMP LOG   ', 'longname': b'VMP LOG                  ', 'holder': b'\xff\xff\xff\xff', 'length': 8109, 'version': 10, 'date': b'06/22/23'}
480687
b'VMP loop  VMP expt. loop indexes   \xff\xff\xff\xff0\x01\x00\x00\x00\x00\x00\x00\n\x00\x00\x0006/22/23'
{'shortname': b'VMP loop  ', 'longname': b'VMP expt. loop indexes   ', 'holder': b'\xff\xff\xff\xff', 'length': 304, 'version': 10, 'date': b'06/22/23'}
488861
Traceback (most recent call last):
  File "/Users/dan/Code/pithy/code/DS_galvani_futz_2.py", line 22, in 
    mpr_file = MPRfile(fns[0])
  File "/Users/dan/Code/pithy/code/DS_galvani_pull.py", line 430, in __init__
    raise ValueError("Unrecognised version for data module: %d" %
ValueError: Unrecognised version for data module: 11

I'm guessing a root cause is how VMPmodule_hdr is constructed, but I'm stumped after this

[d-cogswell]

Looks like the new version of EC-Lab might be outputting a new file version.

[dansteingart]

yeah, trying to figure out what the specific difference is. @d-cogswell do you have a link to the old version of EC-Lab? I want to downgrade until this gets licked.

[d-cogswell]

@dansteingart There is a list of old versions here, but no download links: https://www.biologic.net/support-software/ec-lab-software/

[BradyPlanden]

Has anyone found a workaround for this over the last few weeks?

[NuxitBrain]

yeah, trying to figure out what the specific difference is. @d-cogswell do you have a link to the old version of EC-Lab? I want to downgrade until this gets licked.

Hi ! i'm principally a user of the mpr module and i starded to see this error ... If your interested in, i have some old excutable of ec-lab (11.46 and 11.41) and bt-lab (1.73).
Also, if you only want data made with the older version, i think i can provide you data made with the version 11.46 (the one currently installed in our lab)

[BradyPlanden]

For confirmation, I reached out to BioLogic to confirm it is a MPR filetype change. Here is their response,

We have now heard back from our technical team and the product manager.

Apparently, they needed to change the mpr files structure a little so they can manage a larger amount of data.
Unfortunately, they are not allowed to provide the keys to the change (even though they know that someone will work it out eventually anyway) but you should be ok to continue using the earlier versions.

Rebuilding the keys manually everytime they decide to change the structure is quite the hassle. Does anyone have ideas or tools to reduce this overhead?

[ml-evs]

For confirmation, I reached out to BioLogic to confirm it is a MPR filetype change. Here is their response,

Thanks for this!

Rebuilding the keys manually everytime they decide to change the structure is quite the hassle. Does anyone have ideas or tools to reduce this overhead?

Just connecting some dots here from other issues I've seen, but it looks like some progress was made in #80 by analyzing the actual .exe and extracting structure. Might be more robust than manual updates. I'm happy to help out with this if there is interest.

[fg-personal]

Has anyone found a workaround over the last few weeks?
I tried the script below to quickly decode the exe
extract_info_hopefully_fromECLab_exe.py.txt (automatically generated script with minor adjustments)
I quickly found something by using
"grep -i -C 170 "Z'/ohm" extracted_raw_text.txt > exe_interesting_lines.txt" (the txt file is the output of the script above)
I checked in the neighbourhood of the memory location indicated here https://github.com/echemdata/galvani/issues/80/

Here's the result of my search:
exe_interesting_lines.txt
although I am not sure about how to implement this

Does anyone have ideas about how to proceed? I don't think I can go any further

[PeterKraus]

My plan for fixing this in yadg was to get a hand on a set of MPR and MPT files generated in 11.50 using the same experiments. I don't have access to any biologic hardware right now, and not much time to work on this. But getting such files, at least for the "basic methods" (constant current, constant voltage) might be a good start.

[tk-4767]

@PeterKraus, here is Data of a Dummy Cell from the newest version (11.50). DC_3_VMP3_02_CV.zip
I can provide you more data if needed.

[fg-personal]

@PeterKraus, here is Data for a dummy resistor (2-electrode config) from the newest version (11.50)
15MOhm EC Lab 11.50 galvani.zip
Some metadata has been removed in a text editor, now the files do not open in EC-Lab anymore for some reason, but all the measurement data should still be intact.

[fg-personal]

is this problem completely absent for files created in slightly older EC-Lab versions (e.g. 11.46 or 11.47)?