Change PGN source from C to XML about canboat

[gb444]

A source of PGN infomation which was language generic would be a good resource. Seconded

[tkurki]

Should the c generation be part of the normal build or should the generated c be just checked into github?

If part of the build what tools should be used for the transformation?

[gb444]

I've converted the pgns and most of the Lookups etc in pgn.h into JSON for now, as the conversion was textually fairly easy, and JSON is easy to work with with python (my choice of language for this sort of thing). I can see problems if one wanted to convert to c easily as the two formats do not tie together that nicely (JSON has a small number of types etc). I'm a little confused, looking at the code why some of the definitions seem to use strings which are parsed on runtime and some non-static arrays of structs, appart from Cs lack of any other way to do it with low runtime overheads.

[woobagooba]

@tkurki Hi. As Canboat suggested, IMO best practice would be to generate the C (from XML) at build time. Using an XSLT transform invoked from the makefile. I'd prefer an XSD versus a DTD.

[tkurki]

How would you invoke the xslt, eg. what is the universally available tool that can be included in the build process? Pure C with Makefile is write once, build anywhere. I can build canboat on Raspberry, OS X and whatnot by just pulling from Github, would not want to lose that.

[keesverruijt]

At the moment the code has the data definitions in a C header file, but if you build the code (using make + (g)cc) you can have the analyzer produce a definition in XML. There is no XSD/DTD.

I did it this way so that I could start building this on a simple (very) small Linux board, with zero dependencies on libraries other than the standard C library.

If the source code flips over that would be cool. We could write a dedicated parser that only understands that single XML file, or use libxml2 or some other portable XML parser. But as @tkurki writes, we may lose the extremely simple build process we have now.

@gb444 Converting data by hand to JSON is clearly the wrong way forward, why don't you use the XML as source (by building -this- project first and then run

analyzer -explain-xml

You could then write a XML to JSON conversion XSLT and contribute that file to this project. Note that the current C to XML producer nicely unformats all C runtime strings for you. Here is an excerpt of the current XML that is produced:

     <Field>
       <Order>7</Order>
       <Name>Test ID</Name>
       <Description>See Airmar docs for table of IDs and failure codes; these lookup values are for DST200</Description>
       <BitLength>8</BitLength>
       <BitOffset>35</BitOffset>
       <BitStart>3</BitStart>
       <Type>Lookup table</Type>
       <Signed>false</Signed>
       <EnumValues>
         <EnumPair Value='1' Name='Format Code' />
         <EnumPair Value='2' Name='Factory EEPROM' />
         <EnumPair Value='3' Name='User EEPROM' />
         <EnumPair Value='4' Name='Water Temp Sensor' />
         <EnumPair Value='5' Name='Sonar Transceiver' />
         <EnumPair Value='6' Name='Speed sensor' />
         <EnumPair Value='7' Name='Internal temperature sensor' />
         <EnumPair Value='8' Name='Battery voltage sensor' />
       </EnumValues>
     </Field>

The C code is like that because the lookup tables started organically, with just a few values here and there. In the beginning of my reverse engineering project I did not realise how often certain lookups would be used. And of course C doesn't have the map like structures that python/php/java have. So yes, the code could really use in the code is a separate domain of lookup lists, with PGN fields referring to such lists. If anyone wants to come forward and implement this, please do. It would be especially cool if these new structures generated fixed lookup arrays that would be quicker. Maybe

[gb444]

Well I feel stupid! I think a runtime dependance on an xml parser would be best avoidable, as you said. Embeddability is clearly important for this project. I might look at generating pgn.h from the xml descriptions, if that seems like the best route. Parsing an xml file each time the process is started seems a bit over the top...

[woobagooba]

@gb444 I agree

[gb444]

Would python be an acceptable dependency for the build? I generally prefer to work in a high level language for doing text and xml manipulation and not having to compile would get rid of a step.

[tkurki]

Hmm. Is there really something to fix here? How about adding an optional postprocessing step (make docs) that uses the command line option to spit out the xml and transform it into json would get the PGN definition data out in a more usable format. Easy enough to tweak to get what you want, add other formats etc.

Maybe add a step (make publish) to automatically push PGNs in XML, text, JSON and Markup format to the canboat wiki here on Github?

Haven't really looked at the pgn.h C code, would adding new stuff be considerably easier by writing XML? Don't like to do that myself.

[gb444]

My wanting of JSON was simply because I forgot about the --explain-xml flag built in. JSON would be a good format but that could be generated from XML easily. The reason one might want to generate pgn.h is 1) a language independent storage format and 2) easy adding of pgns without any messing with source, which can be messy (I think adding something in XML would be easier and more robust). The chance to re-write might also lead to some memory reductions etc.

There is nothing particularly to fix, but I can see its utility and how it would enhance things.

[keesverruijt]

@tkurki: gee, I like your make publish target

analyzer --explain-xml > somewhere/pgn.xml
some-language-script < somewhere/pgn.xml > somewhere/pgn.json
git add -m"Release yyyymmdd" somewhere/pgn.*
git push

As long as it's not needed to build the C code I don't mind if some-language-script is in python, although I am more a Perl/PHP guy than Python. (I don't remember having to install multiple Perl versions, ever.)

[tkurki]

Just converted -explain-xml output with http://xmlsoft.org/XSLT/xsltproc.html and https://code.google.com/p/xml2json-xslt/ to json. Pretty straightforward, result is pretty much what you would expect:

*/{
    Date:"$Date: 2012-03-14 16:30:42 +0100 (wo, 14 mrt 2012) $",
    Comment:"See http://canboat.com for the full source code",
    CreatorCode:"Keversoft NMEA2000 Analyzer",
    License:"GPL v3",
    PGNs:[
      {
        PGN:59392,
        Description:"ISO Acknowledgement",
        Complete:true,
        RepeatingFields:0,
        Fields:[
          {
            Order:1,
            Name:"Control",
            BitLength:8,
            BitOffset:0,
            BitStart:0,
            Type:"Lookup table",
            Signed:false,
            EnumValues:[
              null,
              null,
              null,
              null]},
          {...

[tkurki]

Hmm, null EmunValues look less than perfect. With the changes below they come out ok as well:

Index: xml2js.xslt
===================================================================
--- xml2js.xslt (revision 3)
+++ xml2js.xslt (working copy)
@@ -38,18 +38,17 @@
   <!-- JS: ignore attributes (convert to comment) -->
   <xsl:template name="attrs">
     <xsl:if test="not(count(attribute::*)=0)">
-      <xsl:text>/*</xsl:text>
+      <xsl:text>{</xsl:text>
       <xsl:for-each select="attribute::*">
-        <xsl:text>@</xsl:text>
         <xsl:value-of select="local-name()"/>
-        <xsl:text>="</xsl:text>
+        <xsl:text>:'</xsl:text>
         <xsl:value-of select="."/>
-        <xsl:text>"</xsl:text>
+        <xsl:text>'</xsl:text>
         <xsl:if test="not(position()=last() or last()=1)">
-          <xsl:text> </xsl:text>
+          <xsl:text>,</xsl:text>
         </xsl:if>
       </xsl:for-each>
-      <xsl:text>*/</xsl:text>
+      <xsl:text>}</xsl:text>
     </xsl:if>
   </xsl:template>

@@ -298,7 +297,7 @@
     <xsl:call-template name="indent"/>
     <!-- JS: handle attributes -->
     <xsl:call-template name="attrs">
-      <xsl:with-param name="attrs" select="../@*"/>
+      <xsl:with-param name="attrs" select="./@*"/>
     </xsl:call-template>
     <xsl:call-template name="quote-property">
       <xsl:with-param name="name" select="name()"/>
@@ -330,7 +329,9 @@
     <xsl:call-template name="indent"/>
     <xsl:choose>
       <xsl:when test="not(child::node())">
-        <xsl:text>null</xsl:text>
+    <xsl:call-template name="attrs">
+      <xsl:with-param name="attrs" select="./@*"/>
+    </xsl:call-template>
       </xsl:when>
       <xsl:otherwise>
         <xsl:apply-templates select="child::node()"/>

PGN:59392,
        Description:"ISO Acknowledgement",
        Complete:true,
        RepeatingFields:0,
        Fields:[
          {
            Order:1,
            Name:"Control",
            BitLength:8,
            BitOffset:0,
            BitStart:0,
            Type:"Lookup table",
            Signed:false,
            EnumValues:[
              {Value:'0',Name:'ACK'},
              {Value:'1',Name:'NAK'},
              {Value:'2',Name:'Access Denied'},
              {Value:'3',Name:'Address Busy'}]},
          {

[timmathews]

@tkurki I like your approach for EnumValues, but instead of numbers-as-strings, how about just numbers?

{
  "EnumValues": [
    {"Value": 0, "Name": "ACK"},
    {"Value": 1, "Name": "NAK"},
    {"Value": 2, "Name": "Access Denied"},
    {"Value": 3, "Name": "Address Busy"}
  ]
}

[tkurki]

This was just a POC with a slightly tuned generic template, no doubt there is room for improvement.

[timmathews]

Well, since I'm bored at work, here's it is implemented: http://pyxis.openseasproject.org/api/v1/messages/1

[chgans]

Why not use the KCD format? https://github.com/julietkilo/kcd
Which is an open source alternative to http://socialledge.com/sjsu/index.php/DBC_Format
Other formats can be found here: https://github.com/ebroecker/canmatrix

[chgans]

As well this tool is pretty cool:
https://cantools.readthedocs.io/en/latest/

[TwoClocks]

There is a "bug" in the json file. Any PGN w/ only 1 field has an extra "Field" encapsulation. Like for isoRequest :

  {
        "PGN":59904,
        "Id":"isoRequest",
        "Description":"ISO Request",
        "Complete":true,
        "Length":3,
        "RepeatingFields":0,
        "Fields":{
          "Field":{
            "Order":1,
            "Id":"pgn",
            "Name":"PGN",
            "BitLength":24,
            "BitOffset":0,
            "BitStart":0,
            "Type":"Integer",
            "Resolution":1,
            "Signed":false}}},
      ```

[keesverruijt]

This is being continued else where, in issues #281 and #283.