osplanning / omx Goto Github PK
View Code? Open in Web Editor NEWOpen Matrix (OMX)
Home Page: https://github.com/osPlanning/omx/wiki
License: Apache License 2.0
Open Matrix (OMX)
Home Page: https://github.com/osPlanning/omx/wiki
License: Apache License 2.0
Guys, we just had a user repro an overflow on zone number (he had zone #'s > 65535) when used in a mapping from Python. Looks like the unsigned short being used in createMapping() at omx/File.py line 173.
...
mymap = self.createArray(self.root.lookup, title, atom=tables.UInt16Atom(),
...
Right now the API assumes everything is an OMX matrix. Assuming this project is a wild success and future datatypes are coming down the pike, what do we need to do to generalize the API?
Has anyone tackled Javascript yet for reading OMX matrix files?
Looks like there's an hdf5 library https://www.npmjs.com/package/hdf5
We have this most awesome matrix viewer/clusterer that works in the browser written in multithreaded Javascript.
I remember a few months ago someone put together a python package that could be installed and maintained separately. Over the weekend I threw together an R package containing the API functions, adding some documentation files, etc.
I've got my work on a release branch in my gregmacfarlane/omxr repo. If anyone would like to help me check out the code, that would be great.
Also, I'd be happy to transfer ownership to osPlanning if you like.
I am using the R API to load a .omx
file I exported from Cube with the cube2omx.exe
tool. I can see the resulting tables from my matrix files in AMPKHWY16.omx
using HDFView for Ubuntu, so I think that step worked, which is why I'm filing this issue here.
But I cannot read these tables into R, using Ubuntu or OSX. For example,
library(rhdf5)
source("omx.r")
listOMX("AMPK16HWY.omx")
HDF5-DIAG: Error detected in HDF5 (1.8.7) thread 0:
#000: H5A.c line 550 in H5Aopen(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#001: H5Oattribute.c line 530 in H5O_attr_open_by_name(): can't locate attribute
major: Attribute
minor: Object not found
HDF5: unable to open attribute
Error: Error in h5checktype(). Argument not of class H5IdComponent.
I filed this here because I can look into the tables with HDFView. On the other hand, I can run through the examples in text_omx.r
with no difficulty, so the problem might be in cube2omx.exe
or my use of it.
Python api File.mapentries() always throws error because of undefined variable.
File.py line 144 is:
return (keymap,entries)
but should be:
return entries
variable keymap is not defined, so code throws misleading exception of type LookupError
I was trying to implement where() and it is getting really ugly sans a class for matrices...
Suggest subclassing CArray strictly, but open to others...
Makes loading the library in Excel via VBA possible
I'd like to propose that we evaluate the feasibility to support the faster Arrow-based data format.
Cube, Transcad, and Emme all use zone numbers that are 1-based, which is a huge pain in the ass.
How do we want to deal with this? A flag? Ignore it and leave it to the user?
@jeabraham - I’m trying to use the Java OMX library and it won’t compile against HDF5 1.10 because they’ve moved to long pointers. Has anyone tackled the conversion/update? I’m giving it a go but it’s a bit of work.
My motivation is actually to put the PECAS economic flow matrices (700 or so of them) into one file that can be read by a visualization program. That visualization program is currently in Javascript. Has anyone tackled reading OMX matrices from Javascript? There’s a Javascript HDF5 library. See #32
We need to update https://github.com/osPlanning/omx/wiki/Python since we changed the API.
It would be nice to wrap this feature since we are going from h5-->h5 and it should be seamless.
Traceback of error when trying to write CArray"
>>> myfile = omx.openFile(TEST_FILE, 'r')
>>> outFile = omx.openFile(r"testData/sfptam2.omx",'w')
>>> m1in=myfile["1"]
>>> m1in
/1 (CArray(3693, 3693), shuffle, zlib(7)) ''
atom := Float64Atom(shape=(), dflt=0.0)
maindim := 0
flavor := 'numpy'
byteorder := 'little'
chunkshape := (8, 3693)
>>> outFile["test1"]=m1in
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "omx\File.py", line 178, in __setitem__
return self.createMatrix(key, atom, shape, obj=dataset)
File "omx\File.py", line 35, in createMatrix
matrix[:] = obj
File "C:\Python27\Lib\site-packages\tables\array.py", line 672, in __setitem__
nparr = convertToNPAtom2(value, self.atom)
File "C:\Python27\Lib\site-packages\tables\utils.py", line 114, in convertToNP
Atom2
nparr = convertToNPAtom(object, atom, copy)
File "C:\Python27\Lib\site-packages\tables\utils.py", line 83, in convertToNPA
tom
nparr = array_of_flavor(arr, 'numpy')
File "C:\Python27\Lib\site-packages\tables\flavor.py", line 213, in array_of_f
lavor
return array_of_flavor2(array, flavor_of(array), dst_flavor)
File "C:\Python27\Lib\site-packages\tables\flavor.py", line 202, in flavor_of
"supported objects are: %s" % (type_name, supported_descs) )
TypeError: objects of type ``CArray`` are not supported in this context, sorry;
supported objects are: NumPy array, record or scalar; homogeneous list or tuple,
integer, float, complex or string
Solution:
outFile["test"] = numpy.array((m1in))
ActivitySim needs to read OMX files. Based on feedback from the community we'd like to have OMX available as a standalone Python library installable via pip and conda (on PyPI and binstar, respectively). This kind of release is required so we can easily install the OMX library for things like automated testing and so it can be listed as an explicit dependency of ActivitySim.
Related to making a release, I'd recommend splitting the directories for each OMX library into separate repositories. That's not strictly necessary, but it has some conveniences:
As an example, the official GitHub API wrappers for different languages are kept in separate repositories: https://github.com/octokit.
Whether you decide to split the the directories or not, I'm happy to help with the infrastructure for a release. We'll need this available soon for ActivitySim development and testing. (Though not until after the New Year.)
You may also be interested in the changes I made for the proposed integration of OMX with ActivitySim: ActivitySim/activitysim#7. The changes included PEP8 compliance, switch to pytest for running unit tests, and additional test coverage.
Version 3.3.0 of pytables expired deprecated camelCase function names in favor of underscore_delimited names thus breaking references in openmatrix to functions like getNode (now renamed get_node).
I think we probably would like capabilities on all fronts, but for trips (and other additive measures) we most likely want a summed 2D array while for something like a skim, we would probably like to be able to iterate through the list (simpler than a 3D array, although not all numpy functionality comes through).
Hello everyone and congratulations for this great project.
Standardising formats will definitely help everyone in the transport industry.
I just wanted to raise a concern regarding the selected approach. Although the transport industry has been using and storing OD matrices in the "matrix format" (e.g. rows and columns), I believe that this is not the most efficient approach. From my perspective as well as from quite a few other data analysts and programmers (e.g. https://vita.had.co.nz/papers/tidy-data.pdf ) storing data in a "list" or "database format" is more efficient. Following this format all ODs could be stored in a single file and the user will be able to make selections based on simple and standardised queries.
For instance:
Origin_Zone, Destination_Zone, Trip_Purpose, Time_Period, Trips
A, B, HBW, AM, 10
Z, X, HBO, IP, 12
...
I would really like to know the views of the development team regarding this comment.
Kind Regards
-Haris
python-omx-sample.py fails around line 52 calling:
myfile.listAllTags() # ['am','hwy','md','trips','trn']
This function appears to have been removed some time ago.
I tried changing it to call myfile.listAllAttributes()
instead but that is apparently not equivalent and furthermore it just fails a little further on.
from the SATURN team
One suggestion for the next revision to the specification is internal storage of the multiple tables within the OMX file. At the moment, they are stored alphabetically based on the table names so their relative position changes depending on the user-defined title. This causes some problems in implementation for our software as we use their level (ie 1st table, 2nd table, 3rd table) as the unique identifier not the table name (with the latter only optional). To get round this, we export the OMX tables (via UFM2OMX) with a Level position ‘Lxx’ prefix to the user-defined title name. When importing (via OMX2UFM), the levels are simply defined as the order they appear (ie alphabetically) and hence require some post import manipulation. The process works but it’s not as streamlined as it could be. Has anybody else flagged this as weakness?
The default matrix TITLE attribute is some funny character that is causing problems. In order to read our matrices into the OMX Viewer and VISUM, we had to override the TITLE after the matrix was created. We added the following bold line:
import openmatrix as omx
outfile = omx.openFile(os.path.join(dir_data, 'transit_od_demand_new_' + per + '.omx'), 'w')
outfile['metro'] = od_matrix[i+1][0]
outfile.createMapping('taz', taz_labels)
outfile['metro'].attrs.TITLE = 'metro'
outfile.close()
API lists some exception types, but none are implemented yet.
we need test cases to round trip validate OMX APIs. We can use this for the OMX APIs on GitHub, as well as for commercial third-party applications. We may also setup continuous integration.
Issue:
Traceback:
>>> outFile["test1"]=m1in
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "omx\File.py", line 178, in __setitem__
return self.createMatrix(key, atom, shape, obj=dataset)
File "omx\File.py", line 33, in createMatrix
chunkshape, byteorder, createparents)
File "C:\Python27\Lib\site-packages\tables\file.py", line 831, in createCArray
chunkshape=chunkshape, byteorder=byteorder)
File "C:\Python27\Lib\site-packages\tables\carray.py", line 203, in __init__
byteorder, _log)
File "C:\Python27\Lib\site-packages\tables\leaf.py", line 263, in __init__
super(Leaf, self).__init__(parentNode, name, _log)
File "C:\Python27\Lib\site-packages\tables\node.py", line 241, in __init__
parentNode._g_refNode(self, name, validate)
File "C:\Python27\Lib\site-packages\tables\group.py", line 494, in _g_refNode
% (self._v_pathname, childName))
tables.exceptions.NodeError: group ``/`` already has a child node named ``test1`
The AequilibraE plugin for QGIS also has an OMX viewer, which might be a good alternative.
I have not found a way to issue a pull request for the Wiki content on this repo, so that is why I am filling this issue.
The description of the capability and how to get it to work is documented here: https://www.xl-optim.com/displaying-omx-matrix-in-qgis/
Hi @mmilkovits, Can you check in your solution file on your branch and the resulting dll from the latest patch you made? Then I can pull it into the main code.
Issue Part 1 : call to pytables to open h5 file is returning an error that it isn't writable (FileModeError) despite there not being any permissions or read-onlyness of the file
Issue Part 2: why are we requiring write file privies if we are opening as read-only?
testTA.py:
import omx,numpy
TEST_FILE = r"testData/sfptam.h5"
myfile = omx.openFile(TEST_FILE, 'r')
Traceback:
C:\work\omx>python testTA.py
Traceback (most recent call last):
File "testTA.py", line 5, in <module>
myfile = omx.openFile(TEST_FILE, 'r')
File "C:\work\omx\omx\__init__.py", line 21, in openFile
f.root._v_attrs['omx_version'] = __version__
File "C:\Python27\Lib\site-packages\tables\attributeset.py", line 526, in __se
titem__
self.__setattr__(name, value)
File "C:\Python27\Lib\site-packages\tables\attributeset.py", line 427, in __se
tattr__
nodeFile._checkWritable()
File "C:\Python27\Lib\site-packages\tables\file.py", line 1620, in _checkWrita
ble
raise FileModeError("the file is not writable")
tables.exceptions.FileModeError: the file is not writable
Closing remaining open files: testData/sfptam.h5... done
C:\work\omx>python testTA.py > notwrited.log notwrite.log
@e-lo asked:
Where does API / Examples / Gists live?
Preferably in separate repositories, tagged with OMX version #. Currently the APIs of all code are in one place and move forward in commits together. I personally find this messy.
What do you all think about having:
• OMX.r
• OMX.py
• OMX.cpp
• OMX.java
• etc...
all as separate repositories? There are good and bad things about this, but the main issue i see is one of the APIs moving forward w/out volunteers to move forward the other APIs and ended up with stale parts of a repo. The major disadvantage I see is consolidated issue tracking/version control.
I got this error using the method createMapping() from the Python API:
--> 172 mymap = self.createArray(self.root.lookup, title, atom=tables.UInt32Atom(),AttributeError: 'File' object has no attribute 'createArray'`
It turns out that in my "tables" module (version 3.3.0) the method createArray() is called crate_array().
I fixed this bug localy by manually changing one line in the createMapping(self, title, entries, overwrite=False)
method of the File
class.
The original code was:
# Write the mapping!
mymap = self.createArray(self.root.lookup, title, atom=tables.UInt32Atom(),
shape=(len(entries),) )
I replaced it by:
# Write the mapping!
mymap = self.create_array(self.root.lookup, title, atom=tables.UInt32Atom(),
shape=(len(entries),) )
For information, I am using Python 2.7.3, and the openmatrix version 0.2
Transcad-style indexing is not yet supported. Design details:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.