heuer / mappa Goto Github PK

The CXTM test case is a hack, refactor it
Maybe move it to the Mappa subproject and use svn:externals

More detailed exception if a topic is not removable.
If a topic is not removable, the exception should name the problem and not
enumerate possible reasons why the topic is not removable

Either update the Java CXTM tests from TMAPI 1.0/tinyTiM 1.5 to TMAPI
2.x/tinyTiM 2.x or remove them

<http://code.google.com/p/ontopia/issues/detail?id=91>

MapHandler should check if the statement which should get the iid is a
duplicate

Implement the missing load / write operations to the connection 

Port the Java RTM impl. to Python using rdflib

<http://code.google.com/p/ontopia/issues/detail?id=84>
<http://code.google.com/p/ontopia/issues/detail?id=77>

The MapHandler should check if the statement which should be reified is a
duplicate.

During parsing XTM 1.0 the parser keeps a variable, 'seen_type' to remember
if an <instanceOf/> element was read. This variable is never resetted to False.

Spit out a warning if the role type is missing.

Currently all CXTM tests use only the syntax/in tests.
Integrate the syntax/invalid tests

Add a Distutils setup.py to the projects, c.f.
http://docs.python.org/distutils/

The tm.mio.create_deserializer function should try to look up the requested
deserializer in the dict of cached deserializers. If the deserializer is
not found, the function should reiterate through the available
deserializers and register newly added deserializers. If the deserializer
cannot be found return ``None``

CTM should be the default syntax for connection.loads(), so the format
parameter is not required for importing CTM.

lit.py assumes that "itemgetter" exists in the "operator" module, but in 
Python 2.3.5 it does not. In Python 2.5 it does.

Traceback (most recent call last):
  File "tmfs.py", line 4, in ?
    import mappa
  File "/Users/larsga/cvs-co/mappa/mappa/trunk/src/mappa/__init__.py", 
line 51, in ?
    from _internal.lit import Literal
  File "/Users/larsga/cvs-co/mappa/mappa/trunk/src/mappa/_internal/
lit.py", line 48, in ?
    from operator import itemgetter
ImportError: cannot import name itemgetter

Use a better naming scheme for the syntax readers. 

Currently: <SyntaxName>TopicMaps.

Maybe better:
mio.reader.<syntax-name>
tm.mio.reader.<syntax-name>

Check cheeseshop and register the names.

mergeutils.py uses at some places the 'old' Mappa model where topic.types
is a property. 

easy_install -U Mappa installs Mappa and the XTM package. Add the CTM
package as dependency, so easy_install -U Mappa installs Mappa, the XTM
package and the CTM package

Add a full text index to Mappa

Options:
* http://whoosh.ca/
  Pure Python
* http://lucene.apache.org/pylucene/
  Lucene port to Python
* http://nucular.sourceforge.net/
  Python, requires eTree

Need more merge tests, especially for topic maps.

Check where the code uses 
    sorted(iterable, cmp=comparator) 
instead of 
    sorted(iterable, key=k)

Replace cmp with key where possible.

Modules:
- CXTM writer
- CTM writer

Add an entry point to Mappa that is used by Topic Maps syntax writers

Write docs and examples

Think about the possibility to make the roles of an association immutable.
That means an API break and could mean that reification of roles and iids
on roles may not be allowed anymore. Reification of roles and iids on roles
suck anyway, but it breaks compatibility to TMDM.

Impacts:
- assoc.create_role(type, player) would go away
- role.player = new_player / role.type = new_type *may* go away (maybe just
the role.type = new_type)
- role.add_iid / role.remove_iid would go away
- role.reifier is not supported anymore

IIRC the ZTM supports just immutable associations; figure out how immutable
they are. :)

Importing my photo topic map (XTM 2.0) produces the following crash:

Traceback (most recent call last):
  File "tmfs.py", line 16, in <module>
    tm = import_xtm(sys.argv[1])
  File "tmfs.py", line 14, in import_xtm
    reader.parse(Source(file=open(filename)))
  File "/Users/larsga/cvs-co/mappa/tm/trunk/src/tm/mio/deserializer.py", 
line 107, in parse
    self._do_parse(source)
  File "/Users/larsga/cvs-co/mappa/mio-xtm/trunk/src/mio/reader/xtm/
__init__.py", line 76, in _do_parse
    parser.parse(as_inputsource(source))
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/xml/sax/expatreader.py", line 107, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/xml/sax/expatreader.py", line 207, in feed
    self._parser.Parse(data, isFinal)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/xml/sax/expatreader.py", line 338, in start_element_ns
    AttributesNSImpl(newattrs, qnames))
  File "/Users/larsga/cvs-co/mappa/mio-xtm/trunk/src/mio/reader/xtm/
__init__.py", line 93, in startElementNS
    self._content_handler.startElementNS(name, qname, attrs)
  File "/Users/larsga/cvs-co/mappa/mio-xtm/trunk/src/mio/reader/xtm/
xtm10.py", line 156, in startElementNS
    self._process_sid(href(attrs))
  File "/Users/larsga/cvs-co/mappa/mio-xtm/trunk/src/mio/reader/xtm/
xtm10.py", line 362, in _process_sid
    self.map_handler.subjectIdentifier(ref)
  File "/Users/larsga/cvs-co/mappa/tm/trunk/src/tm/mio/handler.py", line 
299, in subjectIdentifier
    self._handler.subjectIdentifier(iri)
  File "/Users/larsga/cvs-co/mappa/mappa/trunk/src/mappa/miohandler.py", 
line 142, in subjectIdentifier
    existing = self._tm.topic(sid=sid)
  File "/Users/larsga/cvs-co/mappa/mappa/trunk/src/mappa/_internal/
implhelper.py", line 89, in topic_by_identity
    raise ValueError('Expected either one "sid", "slo", or "iid" keyword 
argument, got: %r' % kw)
ValueError: Expected either one "sid", "slo", or "iid" keyword argument, 
got: {'sid': u'http://psi.ontopedia.net/Marc_Wilhelm_K%FCster'}

PLY generates the parse/lex tables and stores them in the current
directory. Ensure that PLY reads and writes them from the package
directory, use something like 

http://dev.se.wtb.tue.nl/projects/chi-tooling/browser/trunk/chinetics/chinetics/
core/plywrapper.py

for all PLY-based parsers

Use the project's Download section for distributing Mappa instead of
duplicating the distribution by uploading to Cheeseshop and to the Download
section.

Evaluate if url / download_url does the job.

Provide an additional keyword arg to write gzipped topic maps / read from
gzipped topic maps

The test "role-duplicate-reified4.xtm" fails:

miohandler.py line 464, in _are_mergable
AttributeError: 'NoneType' object has no attribute '__sig__'

<http://cxtm-tests.svn.sourceforge.net/viewvc/cxtm-tests/trunk/xtm2/in/role-dupl
icate-reified4.xtm?view=markup>

The following probably speaks for itself:

[larsga@Macintosh-2 mappa]$ python
Python 2.3.5 (#1, Mar 23 2006, 02:56:38) 
[GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mappa
>>> conn = mappa.connect()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "mappa/__init__.py", line 91, in connect
    backend = __import__('mappa.backend.%s' % backend_name, None, None, 
[''])
ImportError: No module named mem

Does this mean that there is no in-memory implementation at all? Or just 
that it's not checked in to Subversion?

Currently it is possible to set the variant's scope to the UCS. This should
result in a ModelContraintEx.

Ensure that the variant's scope is always the superset:

name.scope = ()
variant = name.create_variant('value', ()) # --> MCE

variant.scope = () # --> MCE

name.scope = [theme]
variant.scope = [theme] # --> MCE, no superset

name.scope = []
variant.scope = [theme] # Allowed, [theme] is a superset of the UCS

Mappa needs a normalization procedure to 
(a) detect duplicates 
(b) be conform to CXTM

For Mappa "+1"^^xsd:integer is different from "1"^^xsd:integer but ideally,
Mappa should normalize the "+1" value to "1". 

(b) is solvable if the CXTM writer does the normalization but this is a bad
idea since it does not solve (a)

Solution:
1. Implement literal normalization in Mappa
(maybe 2.) Move that implementation to the ``tm`` package?!?

Verify that r164 solves all merging problems.

The problem: The MapHandler keeps a stack of instances of _Construct.

A problem will occur if a _Construct keeps a reference to a topic which has
been merged. The reference could be used in the type, player, or scope
property. If topics are merged, the MapHandler should replace these
references with the topic which stays in the topic map.

Walking through the stack of constructs and checking if a topic is part of
the stack is not enough, each _Construct has to be checked.

- Handle LTM @"encoding"
- Test if Unicode BOM is detected

We need support for XTM 1.0 import. Support for mergemap is not necessary 
for our purposes.

Implement mergemap

Mappa should come with the memory-store but other stores should be plugable. 
Entry points etc. bla bla

easy_install mappa.store.durus
easy_install mappa.store.rdbms

Traceback (most recent call last):
  File "tmtest.py", line 165, in ?
    tm = make_topic_map(blog)
  File "tmtest.py", line 34, in make_topic_map
    tm = load_xtm("/Users/larsga/data/topicmaps/blog.xtm")
  File "tmtest.py", line 28, in load_xtm
    reader.parse(Source(file=open(file)))
  File "build/bdist.darwin-9.7.0-i386/egg/tm/mio/deserializer.py", line 
136, in parse
  File "/sw/lib/python2.3/site-packages/mio/reader/xtm/__init__.py", line 
76, in _do_parse
    parser.parse(as_inputsource(source))
  File "/sw/lib/python2.3/xml/sax/expatreader.py", line 107, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/sw/lib/python2.3/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/sw/lib/python2.3/xml/sax/expatreader.py", line 207, in feed
    self._parser.Parse(data, isFinal)
  File "/sw/lib/python2.3/xml/sax/expatreader.py", line 337, in 
start_element_ns
    AttributesNSImpl(newattrs, qnames))
  File "/sw/lib/python2.3/site-packages/mio/reader/xtm/__init__.py", line 
93, in startElementNS
    self._content_handler.startElementNS(name, qname, attrs)
  File "/sw/lib/python2.3/site-packages/mio/reader/xtm/xtm20.py", line 
187, in startElementNS
    handler.subjectIdentifier(href(attrs))
  File "build/bdist.darwin-9.7.0-i386/egg/tm/mio/handler.py", line 301, in 
subjectIdentifier
  File "build/bdist.darwin-9.7.0-i386/egg/mappa/miohandler.py", line 147, 
in subjectIdentifier
  File "build/bdist.darwin-9.7.0-i386/egg/mappa/miohandler.py", line 355, 
in _merge
  File "build/bdist.darwin-9.7.0-i386/egg/mappa/_internal/enhancer.py", 
line 228, in _merge
  File "build/bdist.darwin-9.7.0-i386/egg/mappa/_internal/mergeutils.py", 
line 315, in merge_topics
  File "build/bdist.darwin-9.7.0-i386/egg/mappa/backend/stub.py", line 
413, in remove
mappa._internal.exceptions.ModelConstraintViolation: The topic is not 
removable since it plays roles or it is used as type, player, reifier, or 
theme

I can attach the topic map, if desired.

If you delete a topic which is an instance of some topic type by calling 
topic.remove() the topic *appears* to be gone, but is in reality still 
present. Exporting to XTM 2.0 exports the type-instance association so 
that a reference to a stub of the topic is left behind in the XTM. 
Reimporting then recreates the topic with no characteristics except that 
it is still an instance of the same type.

Attaching a test file and a test script that reproduce the problem.

Move the plyutils/xmlutils from the mio package to another package, maybe
tm.utils

Do not forget to update mio.ltm and mio.ctm (and ql.tolog?!?)

- join  _urlutils with irilib and and ensure that it works
- normalize(iri) should return an unescaped IRI
- introduce externalize(iri) which encodes the IRI

Check-in the LTM on mio implementation

Implement LTM 1.3 mergemap

connection.load() relies on the "format" parameter. If the format is not
specified, connection.load() will fail.
Todo:
Try to extract the file extension from the source iff "format" is None

The XTM 1.0 keeps a variable to detect if the scope of a statement has been
read. This variable is never resetted. 

Implement mergemap for XTM 2.0

Implement an AsTMa= reader. Implies that CXTM tests have to be written.

Maybe it's worth to try Plex instead of the scanners generated by Ply since
Plex uses a DFA and seems to operate on a subset of a document while Ply
enforces that a document is read completely.

http://code.google.com/p/python-plex/

Implement the registration and discovery of deserializers in tm.mio

Implement include directive

Add a JSON Topic Maps (JTM) reader to the mio packages