Comments (27)
Thanks for all the suggestions + data sharing here! I think this shows how widely used galvani still is, so we should put in some effort to ensure its ongoing maintenance. Does anyone have a line on @chatcannon so we could potentially get some maintainers added to the repo? (I guess naturally that means I volunteer but would prefer not to do it alone...) To avoid hijacking this issue I will raise a new one based on this comment, but just wanted to encourage people from this discussion to think about it!
from galvani.
@PeterKraus, thanks for the interest in this topic. I would be happy to find a python-based open-source solution to parse potentiostat files and a user I know may be willing to share the files you mention with people that already have EC-Lab software on their pc.
Despite the kind reply from the BioLogic technical team (...they know that someone will work it out eventually anyway...), I do want to re-emphasize the importance of handling the data with care and respecting any potential restrictions.
EDIT: after the warning, the user may not be willing to share the data files anymore, I hope somebody else will find a solution.
from galvani.
Thanks for the files, @fg-personal and @tk-4767. I might have someone to take this project up in the coming weeks. I'll keep you posted.
from galvani.
With files from version 11.48 I never had a problem
from galvani.
I have been trying to use the mpt reader function from galvani, after an mpr→mpt conversion in EC-Lab.
In the process, I found some new field names and I had to explicitely add them to the script.
I had to modify:
def fieldname_to_dtype(fieldname):
"""Converts a column header from the MPT file into a tuple of
canonical name and appropriate numpy dtype"""
...
elif fieldname in ("time/s", "P/W", "(Q-Qo)/mA.h", "x", "control/V",
"control/V/mA", "(Q-Qo)/C", "dQ/C", "freq/Hz",
"|Ewe|/V", "|I|/A", "Phase(Z)/deg", "|Z|/Ohm",
"Re(Z)/Ohm", "-Im(Z)/Ohm", #new entries after this
"step time/s", "Cs/uF", "Cp/uF", "Re(Y)/Ohm-1", "Im(Y)/Ohm-1",
"|Y|/Ohm-1", "Phase(Y)/deg", "Re(C)/nF", "Im(C)/nF",
"|C|/nF", "Phase(C)/deg", "Re(M)", "Im(M)", "|M|",
"Phase(M)/deg", "Re(Permittivity)", "Im(Permittivity)",
"|Permittivity|", "Phase(Permittivity)/deg", "Re(Resistivity)/Ohm.cm",
"Im(Resistivity)/Ohm.cm", "|Resistivity|/Ohm.cm", "Phase(Resistivity)/deg",
"Re(Conductivity)/mS/cm", "Im(Conductivity)/mS/cm", "|Conductivity|/mS/cm",
"Phase(Conductivity)/deg", "Tan(Delta)", "Loss Angle(Delta)/deg",
"Cs-2/uF-2", "Cp-2/uF-2"):
return (fieldname, np.float_)
...
else:
raise ValueError("Invalid column header: %s" % fieldname)
# added "step time/s", (present in CV files)
# added "Cs/uF", "Cp/uF", "Re(Y)/Ohm-1", "Im(Y)/Ohm-1",... (present in EIS files)
I have also changed the line
fieldnames = next(mpt_file).decode(encoding).strip().split('\t')
to
fieldnames = next(mpt_file).replace(b'\xb5', b'u').decode(encoding).strip().split('\t')
to get rid of the μ, not easily decoded by the software.
As I try to import mpt files in python and to put them into dataframes
mpt = BL.MPTfile(filename_with_path)
df = pd.DataFrame(mpt[0])
I get slightly different headers depending on the EC-Lab version used to generate the original mpr file: EIS examples
I have the impression the mpt interpreter hasn't been updated in a while. I had errors while importing EIS files made with an EC-Lab version from 2019 ("Cs/uF", "Cp/uF" not recognized as valid entries) and I had to use the modified fieldname_to_dtype even for files this old.
I apologize if I'm stating the obvious inadvertently, but I hope this information can be of assistance to someone.
@PeterKraus, here is Data for a dummy resistor (2-electrode config) from the newest version (11.50) 15MOhm EC Lab 11.50 galvani.zip Some metadata has been removed in a text editor, now the files do not open in EC-Lab anymore for some reason, but all the measurement data should still be intact.
I can perform the mpr→mpt conversion for the files here upon request
from galvani.
I am not working with EC-lab, but was asked by a collegue to tweak the Biologic.py to make it work for the data from EC-Lab 11.50. I have only used it on a few files so far and it definitely needs further testing. But as someone else might already be working on it, here is what I did so far:
I had to change the VMP_Module_hdr (similar to @dansteingart):
VMPmodule_hdr = np.dtype([('shortname', 'S10'),
('longname', 'S25'),
('something', '<u4'),
('length', '<u4'),
('somethingelse', '<u4'),
('version', '<u4'),
('date', 'S8')])
Not sure if should be like this or like @dansteingart's - for the files I tested it does not make a difference.
Either way I end up with version numbers 10 for the Data Module and Log Module and 11 for the Data Module, which seems plausible for a new data format.
The initialization of the MPRfile class now of course needs to include the condition data_module['version'] == 11
in the if-statement for the Data Module version. The number of bytes of data before the main data array seems to have changed to 1007.
Like in Data Module version 3 there is a '\x01' before the start of the data.
So this is how it works for our data so far:
# this is for EC-LAB 11.50. data_module version == 11
elif data_module['version'] == 11:
column_types = np.frombuffer(data_module['data'][6:], dtype='<u2',
count=n_columns)
# There are bytes of data before the main array starts
num_bytes_before = 1007
remaining_headers = data_module['data'][6 + 2 * n_columns:num_bytes_before-1]
main_data = data_module['data'][num_bytes_before:]
I will now have the people actually working with EC-Lab test it.
from galvani.
I am not working with EC-lab, but was asked by a collegue to tweak the Biologic.py to make it work for the data from EC-Lab 11.50. I have only used it on a few files so far and it definitely needs further testing. But as someone else might already be working on it, here is what I did so far:
I had to change the VMP_Module_hdr (similar to @dansteingart):
VMPmodule_hdr = np.dtype([('shortname', 'S10'), ('longname', 'S25'), ('something', '<u4'), ('length', '<u4'), ('somethingelse', '<u4'), ('version', '<u4'), ('date', 'S8')])
Not sure if should be like this or like @dansteingart's - for the files I tested it does not make a difference. Either way I end up with version numbers 10 for the Data Module and Log Module and 11 for the Data Module, which seems plausible for a new data format.
The initialization of the MPRfile class now of course needs to include the condition
data_module['version'] == 11
in the if-statement for the Data Module version. The number of bytes of data before the main data array seems to have changed to 1007. Like in Data Module version 3 there is a '\x01' before the start of the data.So this is how it works for our data so far:
# this is for EC-LAB 11.50. data_module version == 11 elif data_module['version'] == 11: column_types = np.frombuffer(data_module['data'][6:], dtype='<u2', count=n_columns) # There are bytes of data before the main array starts num_bytes_before = 1007 remaining_headers = data_module['data'][6 + 2 * n_columns:num_bytes_before-1] main_data = data_module['data'][num_bytes_before:]
I will now have the people actually working with EC-Lab test it.
Seems promising but I still got some errors:
loop_module['version'] is not equal to 0 anymore and that raises the
ValueError("Unrecognised version for data module: %d" %data_module['version'])
That's a red flag, but allowing: loop_module['version'] to be 10 without raising the error seems to solve the problem. I haven't checked this extensively but the first tests I made look good.
With schaubhoffmann's changes to Biologic.py, the script doesn't work for files created with EC-Lab 11.46 or lower
ValueError: Found b'\x00\x00\x00\x00\x00\x00', expecting start of new VMP MODULE While in the read_VMP_modules function
For the moment, I will keep both a Biologic.py (original file) and a Biologic_SH.py (edits from schaubhoffmann) and switch from one to the other with try and except.
from galvani.
I think the problem arose from the newest version of software
from galvani.
I'm also seeing this error with mpr files from the newest version of EC-Lab.
from galvani.
It seems somewhat low-level: the VMPmodule_hdr
length conditions in galvani and new ec-lab.mpr
files. I pulled and added debug comments to BioLogic.py
, so @Dongrun7's error above is reproduced
VMPmodule_hdr = np.dtype([('shortname', 'S10'),
('longname', 'S25'),
('length', '<u4'),
('version', '<u4'),
('date', 'S8')])
yields
b'VMP Set VMP settings \xff\xff\xff\xffa\x1a\x00\x00\x00\x00\x00\x00\n\x00\x00\x00'
{'shortname': b'VMP Set ', 'longname': b'VMP settings ', 'length': 4294967295, 'version': 6753, 'date': b'\x00\x00\x00\x00\n'}
109
Traceback (most recent call last):
File "/Users/dan/Code/pithy/code/DS_galvani_futz_2.py", line 22, in
mpr_file = MPRfile(fns[0])
File "/Users/dan/Code/pithy/code/DS_galvani_pull.py", line 403, in __init__
modules = list(read_VMP_modules(mpr_file))
File "/Users/dan/Code/pithy/code/DS_galvani_pull.py", line 362, in read_VMP_modules
raise IOError("""Unexpected end of file while reading data
OSError: Unexpected end of file while reading data
current module: b'VMP settings '
length read: 489056
length expected: 4294967295
Examining hdr_data
we see the date is nonsensical, and it seems date
is 8 bytes further out from longname
Through some trial and error in modifying VMPmodule_hdr
I was able to get the code to step further...
VMPmodule_hdr = np.dtype([('shortname', 'S10'),
('longname', 'S25'),
('whoknows', 'S4'),
('length', '<u8'),
('version', '<u4'),
('date', 'S8')])
but then gets hung up on versioning
b'VMP Set VMP settings \xff\xff\xff\xffa\x1a\x00\x00\x00\x00\x00\x00\n\x00\x00\x0006/21/23'
{'shortname': b'VMP Set ', 'longname': b'VMP settings ', 'holder': b'\xff\xff\xff\xff', 'length': 6753, 'version': 10, 'date': b'06/21/23'}
117
b'VMP data VMP data \xff\xff\xff\xffW:\x07\x00\x00\x00\x00\x00\x0b\x00\x00\x0006/21/23'
{'shortname': b'VMP data ', 'longname': b'VMP data ', 'holder': b'\xff\xff\xff\xff', 'length': 473687, 'version': 11, 'date': b'06/21/23'}
6935
b'VMP LOG VMP LOG \xff\xff\xff\xff\xad\x1f\x00\x00\x00\x00\x00\x00\n\x00\x00\x0006/22/23'
{'shortname': b'VMP LOG ', 'longname': b'VMP LOG ', 'holder': b'\xff\xff\xff\xff', 'length': 8109, 'version': 10, 'date': b'06/22/23'}
480687
b'VMP loop VMP expt. loop indexes \xff\xff\xff\xff0\x01\x00\x00\x00\x00\x00\x00\n\x00\x00\x0006/22/23'
{'shortname': b'VMP loop ', 'longname': b'VMP expt. loop indexes ', 'holder': b'\xff\xff\xff\xff', 'length': 304, 'version': 10, 'date': b'06/22/23'}
488861
Traceback (most recent call last):
File "/Users/dan/Code/pithy/code/DS_galvani_futz_2.py", line 22, in
mpr_file = MPRfile(fns[0])
File "/Users/dan/Code/pithy/code/DS_galvani_pull.py", line 430, in __init__
raise ValueError("Unrecognised version for data module: %d" %
ValueError: Unrecognised version for data module: 11
I'm guessing a root cause is how VMPmodule_hdr
is constructed, but I'm stumped after this
from galvani.
Looks like the new version of EC-Lab might be outputting a new file version.
from galvani.
yeah, trying to figure out what the specific difference is. @d-cogswell do you have a link to the old version of EC-Lab? I want to downgrade until this gets licked.
from galvani.
@dansteingart There is a list of old versions here, but no download links: https://www.biologic.net/support-software/ec-lab-software/
from galvani.
Has anyone found a workaround for this over the last few weeks?
from galvani.
yeah, trying to figure out what the specific difference is. @d-cogswell do you have a link to the old version of EC-Lab? I want to downgrade until this gets licked.
Hi ! i'm principally a user of the mpr module and i starded to see this error ... If your interested in, i have some old excutable of ec-lab (11.46 and 11.41) and bt-lab (1.73).
Also, if you only want data made with the older version, i think i can provide you data made with the version 11.46 (the one currently installed in our lab)
from galvani.
For confirmation, I reached out to BioLogic to confirm it is a MPR filetype change. Here is their response,
We have now heard back from our technical team and the product manager.
Apparently, they needed to change the mpr files structure a little so they can manage a larger amount of data.
Unfortunately, they are not allowed to provide the keys to the change (even though they know that someone will work it out eventually anyway) but you should be ok to continue using the earlier versions.
Rebuilding the keys manually everytime they decide to change the structure is quite the hassle. Does anyone have ideas or tools to reduce this overhead?
from galvani.
For confirmation, I reached out to BioLogic to confirm it is a MPR filetype change. Here is their response,
Thanks for this!
Rebuilding the keys manually everytime they decide to change the structure is quite the hassle. Does anyone have ideas or tools to reduce this overhead?
Just connecting some dots here from other issues I've seen, but it looks like some progress was made in #80 by analyzing the actual .exe and extracting structure. Might be more robust than manual updates. I'm happy to help out with this if there is interest.
from galvani.
Has anyone found a workaround over the last few weeks?
I tried the script below to quickly decode the exe
extract_info_hopefully_fromECLab_exe.py.txt (automatically generated script with minor adjustments)
I quickly found something by using
"grep -i -C 170 "Z'/ohm" extracted_raw_text.txt > exe_interesting_lines.txt" (the txt file is the output of the script above)
I checked in the neighbourhood of the memory location indicated here https://github.com/echemdata/galvani/issues/80/
Here's the result of my search:
exe_interesting_lines.txt
although I am not sure about how to implement this
Does anyone have ideas about how to proceed? I don't think I can go any further
from galvani.
My plan for fixing this in yadg
was to get a hand on a set of MPR and MPT files generated in 11.50 using the same experiments. I don't have access to any biologic hardware right now, and not much time to work on this. But getting such files, at least for the "basic methods" (constant current, constant voltage) might be a good start.
from galvani.
@PeterKraus, here is Data of a Dummy Cell from the newest version (11.50). DC_3_VMP3_02_CV.zip
I can provide you more data if needed.
from galvani.
@PeterKraus, here is Data for a dummy resistor (2-electrode config) from the newest version (11.50)
15MOhm EC Lab 11.50 galvani.zip
Some metadata has been removed in a text editor, now the files do not open in EC-Lab anymore for some reason, but all the measurement data should still be intact.
from galvani.
is this problem completely absent for files created in slightly older EC-Lab versions (e.g. 11.46 or 11.47)?
from galvani.
Related Issues (20)
- 'R/Ohm' missing in VMPdata_colID_dtype_map
- Test set-up info not available HOT 4
- NotImplementedError: Column ID 174 after column control/mA is unknown (similar to #64) HOT 1
- Similar issue to #27 and #48 HOT 1
- Refactor `fieldname_to_dtype` function
- Try parsing unknown columns as '<f4' with name "Unknown" HOT 1
- Error when opening *mpr file: 'NotImplementedError: Column ID 27 after column control/mA is unknown' HOT 8
- NotImplementedError: Column ID 509 after column Rdc/Ohm is unknown HOT 2
- use pyproject.toml
- Tests fail with "ValueError: Invalid magic for .mpr file: b'version https://git-lfs.github.com/spec/v1\noid sha25'" HOT 1
- I started analyzing ECLab.exe, here are my first results HOT 7
- parse file header HOT 1
- column 23 has wrong label HOT 2
- column 6 has wrong label HOT 1
- column 11 has wrong label
- Unable to read res files with mdbtools>=1.0 HOT 3
- Maintenance status + sustainability HOT 10
- Something to cite HOT 2
- OSError: Unexpected end of file while reading data might due incompatibility with EC-lab 11.50 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from galvani.