Modular system for tools management for OSH documentation about movedo

[hoijui]

A sample OSH docu project could look like this:

build/docu/html/...
build/docu/md/...
build/docu/pdf/...
build/parts/case/model-tech-drawing.svg
build/parts/case/model-render.png
build/parts/electronics/circuit.png
build/parts/electronics/circuit.svg
build/parts/electronics/circuit-gerbers.zip
doc-tools/docu-compiler # git sub-module pointing at MoVeDo
doc-tools/free-cad-storage # git sub-module pointing at ReZipDoc
doc-tools/ki-cad-ploter # git sub-module pointing at a yet-to-be-created repo
doc-tools/manager # git sub-module pointing at a yet-to-be-created repo for doc-tools management
parts/case/docu.md
parts/case/model.fcstd
parts/electronics/docu.md
parts/electronics/circuit.pro
parts/electronics/circuit.kicad_pcb
parts/electronics/circuit.net
parts/electronics/circuit.sch
docu.md
LICENCE.md
README.md

or in tree style:

├── build
│   ├── docu
│   │   ├── html
│   │   ├── md
│   │   └── pdf
│   └── parts
│       ├── case
│       │   ├── model-render.png
│       │   └── model-tech-drawing.svg
│       └── electronics
│           ├── circuit-gerbers.zip
│           ├── circuit.png
│           └── circuit.svg
├── doc-tools
│   ├── docu-compiler
│   ├── free-cad-storage
│   ├── ki-cad-ploter
│   └── manager
├── parts
│   ├── case
│   │   ├── docu.md
│   │   └── model.fcstd
│   └── electronics
│       ├── circuit.kicad_pcb
│       ├── circuit.net
│       ├── circuit.pro
│       └── circuit.sch
├── LICENCE.md
└── README.md

[kasbah]

This issue/discussion surprises me somewhat as I thought this is a tool for standards writing and publishing rather than for hardware documentation. If the development focus branches out like this I worry that the standards writing use-case could be left behind and the tool will become more complicated and harder to use. It gives me second thoughts whether we should try use MoVeDo for the OKH standard going forward.

Nevertheless, some possible prior art and inspiration for the hardware documentation use-case could be:

1. My project: kitspace.org, specifically the kitspace YAML manifest which was also part of the the inspiration for the OKH manifest standard. In OKH we try and apply this approach to all hardware designs, not just electronics. The main difference to what you are describing is rather than impose a pre-defined structure on the user, let them use their own structure but add paths to the manifest so the tools know where to find things. In the kitspace.yaml we already have:

   • gerbers: a path to a gerbers folder
   • bom: a path to a BOM in csv, tsv, ods or excel formats with pre-defined possible headings
   • eda: a field with sub-fields
     • type: currently makes use of kicad or eagle
     • pcb: a path to the .kicad_pcb or .brd file

2. GitBuilding is a tool created by @julianstirling and it's very focused on the writing side, counting parts and collating the bill of materials. It produces nice documentation outputs for web and PDF. For the webview we have already integrated a preview view of STL and some other 3d formats. Maybe @julianstirling can say more about this.

[hoijui]

thank you for the feedback @kasbah , again! :-)

sorry for the confusion!
The issue is conceptually rather in a super-space of MoVeDo, not part of MoVeDo. I just opened the discussion here, because I did not know where else to put it.
MoVeDo might be used as a module in that system (in its current conceptualization),
but it should will be usable stand-alone as well.

I see MoVeDo as a general document management tool, not limited to standards, though.
Is there anything specific to standards, that is not in other documentations?

Regarding your two suggestions:

1. I did not think of meta-data files for specifying locations...
   nice! :-)
   We could use the best of both, as does Maven or Gradle with the Convention over configuration concept;
   meaning:

   If not further specified (in a meta-data file) the default structure can be assumed.

   This has the benefit of project owners having to configure less, and it will let OSH projects gravitate towards a common, optimal structure. Such a structure is necessarily usually more complex then what one would naturally use for a very simple project, but comes with many benefits for automatic processing of project, especially in a multi-tool environment, which is what I would like to encourage.

2. I had a look at GitBuilding, and it does not fulfill many of the items from my wish-list from above:

   • to prevent lock-ins - our whole system or parts of it should be replaceable with alternatives
   • to make it modular
   • (being worked on) to keep common functionality in a kind of library, which could be yet
     an other tool
   • platform independent - if not, it has to be Linux compatible at the
     very least as it will mostly run on CI jobs
   • multi-programming-language support - each tool could use a different
     language
   • plumbing through CLI interfaces
   • (does not apply because no sub-modules) to have a kind of comfortable managing interface to list, update, add
     and remove tools in any OSH documentation repo
   • tools being git repos, and added to the OSH doc repo as git sub-modules

It also introduces a new document format -> vendor-lock-in & bad for multi-tool support.
This document format mixes structure information (BOM) within unstructured data (Markdown like text),
which I think is not a good idea.

I have no good concept of how BOMs should be handled yet myself,
but I imagine them being generated from different, structured parts of the project data:

• KiCad can export BOMs into CSV or XML
• I don't know about FreeCAD, but maybe one can somehow get from FreeCAD to a CSV or XML BOM as well (in the future)?
• one could write CSV files directly

These different sources of BOM would all end up in one common, simple, text-based file, maybe CSV. From there we can again include it in Markdown documents, generate HTML or PDF views, reference parts of it, filter out just the electronics, or something else.

[hoijui]

while I have multi-programming-language support on my wish-list, and I think it should stay there, there should maybe be a best-practice suggestion though.
As of now, MoVeDo for example, is half BASH half Python. I personally do not like Python very much, but ... I should maybe at least try to write one CLI tool in it, and later maybe start rewriting part by part of MoVeDo's BASH scripts int Python.

Would anyone have a better suggestion then Python?
Maybe Julia?

Python (vs BASH) for this scenario (brackets mean: low importance):

• • cross-platform
• • faster
• • cleaner
• • more people know it/are fluent in it
• (+ unit testing is possible)
• • not as well adapted to this specific job
• • requires installing dependencies (while BASH is usually there by default on *NIX and BSD based systems)

Python compared to possible other languages:

• • many people know it
• • especially many people know and like it in the Open Source community
• • not the cleanest
• (- not the fastest)

I guess... it is optimal, right? (just trying to convince myself against my dislike of it ;-) )

[hoijui]

This document format mixes structure information (BOM) within unstructured data (Markdown like text),
which I think is not a good idea.

... though, if the part of GitBuilding that parses the document and extracts the BOM data could be used separately,
even such a document could be made to feed its data into the common BOM file.

[hoijui]

in my current concept, the build and the doc-tools dirs (and everything below them)
are specified by convention,
but the structure of the content (most of which I put into the parts dir in the example)
is still totally free.
It simply is mirrored in build/gen-src at build-time.

That is roughly also how MoVoDo does it.

[julianstirling]

The idea that we write the BOM in a separate file as a CSV is exactly the problem we are trying to solve. Having CSV files for your BOM stops you making you documentation modular and leads to documentation and BOM getting out of sync. Instead we embed the information about the parts used in the text for each page, this allows us to create total bills of materials and export the as CSV (this export has been merged to master but not released). Probably the best explanation of the structure is here

• I totally disagree with the assertion that introducing a new syntax is equivalent to vendor lock-in. Please don't randomly accuse us of vendor lock-in for trying to solve a problem, I find that exceptionally rude. We are looking document the format as an open standard. We have already been in contact with Oasis Open Standards about making the standard compatible with other similar standards. We also are having discussions with other tools about standard convergence. The underlying core functionality GitBuilding is markdown with some extra metadata -> plain markdown with the BOM already calculated. Thus if you ever want to move away from GitBuilding to plain markdown, then just use the output of the last time you ran it!
• The purpose of GitBuilding/BuildUp is to make the documentation itself modular. i.e. pages/sets of pages for sub-components that can then be reused in similar hardware to keep everything in sync across similar projects
• If you looked at the GitBuilding repo and issues we are in the process of splitting out the central underlying functionality (BuildUp) into a separate Python library from the UI. This would allow you to use the just build up to create plan markdown that you can pipe into MoVeDo, PanDoc, or any other markdown tool.
• Yes it is platform independent
• Currently GitBuilding is written in a single language but it is command line driven so it depends how you define multi-language support. Our CI scripts call it from BASH, our Windows CI test scripts call it from Powershell.
• The whole process has been designed from the ground up with CI in mind, we use it via CI to publish documentation (see https://gitlab.com/openflexure/openflexure-block-stage)
• I don't understand what you mean by "managing interface"
• I don't understand why you would want the tool to be included as a submodule? You could sub-module GitBuilding but then you would have the dev version not a stable release that you can get from PyPi. It also creates a higher barrier to entry for new users.

[hoijui]

If there is already BOM information in other tools (say FreeCAD or KiCad), then having it in your Markdown files is still duplication, even if you do not put it (manually) into a CSV file.
Thus I assume, your use-case is for projects that do not have any such things like mechanical-, electronics-, textile- or other model files, right?

When having it in a CSV file, one can generate markdown from that, and link to it, or include it in a generated output file in the same document.

I agree though, that in the absence of any other model containing BOM information, using your system might be a viable - or even optimal - solution, if it works nicely with tools that do not know about your special syntax. An example could be simple projects without electronics and 3D models.
It could be one source of BOM data, and as you explain it, it sounds like it would fit right into such a scenario, especially with your new developments.

---

• vendor lock-in - GitLab supports the Markdown command [[_TOC_]]. I don't know of anything else that supports this exact command (CodiMD supports [TOC] for example). GitHub does not have an equivalent thing. This is effectively a mini vendor lock-in for whomever uses that command. If GitLab writes this command into a standard, it is still, in practice, a vendor lock-in. This does not make GitLab evil or mischievous. It is not an accusation against GitLab, it is simply a fact regarding the actual, living, public IT eco-system. To me, vendor lock-in means: If you use X, then you are very limited in your choices of IT eco-system. This might not be the official definition (and I am happy to be told a better term for what I describe), but it is the most practically meaningful and useful way to see it, I believe.
  You may say, with your system, it will be different in the future, and maybe it will.. maybe GitHub and GitLab will integrate your library and support your syntax out of the box, and many other tools, but that is a very ambitious goal for the future, and it is not reality now.
• modularity - You are right, this should be an other point on the wish-list: support modular documentation (I will put it there)
  With this point though, I was talking about the documentation tool system its self. With modular I mean, something akin the UNIX philosophy, so in the way sed, grep, awk, find, ... can be seen as a set of mutually supporting modules of a bigger whole.
• library (of common base functionality) - nice you are working on that! :-) I will put a check-mark there.
• platform independent - check!
• multi-language support - Each sub-tool/-module could be written in a different language, is what I meant.
• CLI interfaces - This is about CLI (command-line interface), not CI (Continuous Integration). This point is very strongly tied to the last point and to modularity.
• managing interface - What I meant, was something like apt, or more closely Oh-My-ZSH or a vim plugin-manager. In the best case, we can use something that already exists (I never researched that).
• git sub-module - This is very closely tied to the last point. It is not limited to the latest dev versions at all, quite the contrary, it makes every version available, and would default to the stable ones. Making sure that modules can be used this way, does also not limit their use in an other way. Regarding easy-of-use for inexperienced users: This depends on the managing interface; how easy it is to install and use.

[julianstirling]

If there is already BOM information in other tools (say FreeCAD or KiCad), then having it in your Markdown files is still duplication, even if you do not put it (manually) into a CSV file.
Thus I assume, your use-case is for projects that do not have any such things like mechanical-, electronics-, textile- or other model files, right?

Yes and no. The thing is that even if the information is already in the CAD you are still going to be referring to the parts by name in the text.

Example

I have a FreeCAD assembly with 7 screws. I have assembly instructions which explain the toque to tighten these 7 screws. If I modify the assembly in FreeCAD and then export the BOM then the BOM (whether it is in CSV, Markdown, XML, YAML, whatever) is out of date with the text.

However, if I have tagged in the text:

Take seven of the [m3x16 hexagon cap screws](hexscrewlink.md){Qty:7} and tighten them into the following locations, to a torque of 1.3 Nm.

Now if can write a script that checks the BOM of the process I have described in my assembly instructions matches the BOM of that is in the FreeCAD file. As far as I can tell this is necessary because the software cannot competently adjust the assembly instructions. A human needs to go through when things change and check that it still make sense.

---

• vendor lock-in - The logical conclusion of your argument is that ANY new file format is lock in. Creating markdown was lock in, creating ODT was lock-in. Anything that was not a ASCII encoded plain text file is lock in. The unfortunate reality of the real world is that if no format exists that supports what you want to do you have to create one. From day 1 GitBuilding has had plain markdown as an output because it was written as a script that converted the meta data links in place. This means you can always get your data out, and we also checked from day 1 that the documentation was readable in GitLab/GitHub even if not perfect. The idea that we are trying "minor lock-ins" is absurd, offensive. If the formats that exist cannot support the needs of the open hardware community, we need to write our own. We could start from scratch, making it hard to replicate, or we could supplement a widely used standard.
• modularity - The unix BASH philosophy is wonderful. Yet unfortunately if you want a simple to install cross platform app, that is easy to debug and unit test you cannot do it a set of small bash scripts. I have learned this the hard way. The codebase is a python library. We are splitting out the core functionality from the more UI side of the project. The library functions are very fine grained small functions that try to adhere to the KISS principle. And being python libraries you can call them from other code. One of the main goals it to make software that can be used by everyone, not just nerds like us that use linux. So rag-tag-ball of BASH scripts was not going to cut it.
• library (of common base functionality) - This is already done, except as they are both in some flux due to the early stage of dev they are still in the same repo.
• multi-language support - If you think of GitBuilding/BuildUp as the step that takes you from markdown-with-part-metadata, to markdown or html etc. And then other tools as the ones that say create the STLs, publish, etc, etc. Then you can use what ever languages.
• managing interface - Assuming you have python 3.7+ you can run pip install gitbuilding on Windows/Mac/Linux to install the current stable version. As there are some external dependencies (especially on windows), and a lot of windows users don't like command line there is a plan to bundle the software up into stand alone downloadable packages. This is as well as, not instead of the ability to pip install as this is how I want to install it.
• git sub-module - We can agree to disagree on whether this is a good thing. But yes, seeing as GitBuilding is in a git repo you could do this if you so desired.

---

Taking a step back. Do you really feel that having one program that automates PDFs for Standards, assembly docs, user manuals is a good thing? I would say that the formatting needed fro a PDF of a standard is very formal sectioned text. For assembly docs you want to move towards something akin to big picture, minimal text side by side. Of course a single program could do all of this but it seems to be straying from the Unix principle you are championing of doing one thing and doing it well.

For OKH we are interested in trying MoVeDo for the standard. as this is what it can do now. If in the future we want to join efforts to make an all singing all dancing collection of tools for OSH documentation, then we should take a step back. The only sensible way to do this is for us to:

• use and understand each others current projects
• agree not to accuse each other of unethical practices such as lock-in if we disagree on how something is implemented (i.e suggest a specific and viable alternative rather than finger pointing)
• each make a list of ideal requirements
• work together to align these requirements to a single, group agreed, set of requirements.

Working together to set our objectives is far more productive than one person defining a personal manifesto of requirements and berating anyone that hasn't already lived up to their arbitrary standards.

---

I suggest that we close this issue and that we move the discussion of wider documentation tools and formats Hardware Documentation Consensus Groups' Tool Comparison Repo. We will also be having a call this Wednesday if you would like to join.

[hoijui]

... The thing is that even if the information is already in the CAD you are still going to be referring to the parts by name in the text.

Ahhhh.. click ... hmm :/
yeah, I did totally not think of that, thank you for taking the time to explain!

Example

I have a FreeCAD assembly with 7 screws. I have assembly instructions which explain the toque to tighten these 7 screws. If I modify the assembly in FreeCAD and then export the BOM then the BOM (whether it is in CSV, Markdown, XML, YAML, whatever) is out of date with the text.

However, if I have tagged in the text:

Take seven of the [m3x16 hexagon cap screws](hexscrewlink.md){Qty:7} and tighten them into the following locations, to a torque of 1.3 Nm.

Now if can write a script that checks the BOM of the process I have described in my assembly instructions matches the BOM of that is in the FreeCAD file. As far as I can tell this is necessary because the software cannot competently adjust the assembly instructions. A human needs to go through when things change and check that it still make sense.

So let me see if I got it right:

I change my FreeCAD model from something with 7 to sometihng with 8 screws, and I forget to adjust the text/docu.
The CI build extracts BOM info from both the FreeCAD model and the docu (in your format),
and sees that one has 8 screws, and the other 7, and then generates a warning/error,
so a human can make a fixing commit?

* **vendor lock-in** - The logical conclusion of your argument is that ANY new file format is lock in. Creating markdown was lock in, creating ODT was lock-in. Anything that was not a ASCII encoded plain text file is lock in. The unfortunate reality of the real world is that if no format exists that supports what you want to do you have to create one. From day 1 GitBuilding has had plain markdown as an output because it was written as a script that converted the meta data links in place. This means you can always get your data out, and we also checked from day 1 that the documentation was readable in GitLab/GitHub even if not perfect. The idea that we are trying "minor lock-ins" is absurd, offensive. If the formats that exist cannot support the needs of the open hardware community, we need to write our own. We could start from scratch, making it hard to replicate, or we could supplement a widely used standard.

I guess, lock-in fits better then vendor lock-in, but even that is not optimal.
either way, it is no offense, and nothig to be offended by. You seem to ignore what I said about this before. It is not about an ill-intention that I am accusing you of, but about a paractical situation that arises out of your design choices. Your choices might be the best/optimal ones in the scenario, but they are not perfect in every way. maybe, as you say, creating a new format is unavoidable or optimal, but making a new format has downsides, even if you also minimize them, with the tool that separates the BOM info and the pure markdown into separate files.
I am quite convinced by now though, that it is pprobably optimal.

* **modularity** - The unix BASH philosophy is wonderful. Yet unfortunately if you want a simple to install cross platform app, that is easy to debug and unit test you cannot do it a set of small bash scripts. I have learned this the hard way. The codebase is a python library. We are splitting out the core functionality from the more UI side of the project. The library functions are very fine grained small functions that try to adhere to the KISS principle. And being python libraries you can call them from other code. One of the main goals it to make software that can be used by everyone, not just nerds like us that use linux. So rag-tag-ball of BASH scripts was not going to cut it.

It is not the UNIX BASH philosophy, it is the UNIX philosophy. although BASH is the most common tool that does the plumbing, the important thing is the interface the tools support. any shell can do that plumbing, alos one implemented in python, or something not a shell can do it. the interface is basically: stdin, stdout, stderr, command line arguments, and a general gravitation towards text-based data, new-lines or '\0' separated units of text, with tabs or spaces as sub-dividers, ... and that's basically it.
If you define these tools as the units, then you can also do unit-testing on them.
Using this interface, we also get multi-language support with it.

Maybe though, an other interface would make sense for us, but I don't know one.
I guess this would be IPC,
which I only know by name, and seems to be a very general concept... but stuff reffering to itsself as IPC might be in the ballpark.

I am from the Java world, where I liked OSGi a lot,
which I think is similar to the Spring framework Core Technologies,
but that is of course Java only.

The "UNIX interface" is kind of very loose.. one can not define an abstract interface with meta-data or something like that, which is possible in OSGi, and I think in some functional languages or frameworks, there is evne better stuff.

Any replacement for the UNIX thing though, will be less widely known and more complicated to use for most people, I think.

* **multi-language support** - If you think of GitBuilding/BuildUp as the step that takes you from markdown-with-part-metadata, to markdown or html etc. And then other tools as the ones that say create the STLs, publish, etc, etc. Then you can use what ever languages.

cool, we are getting to mor and more common understanding! :-)

* **managing interface** - Assuming you have python 3.7+ you can run `pip install gitbuilding` on Windows/Mac/Linux to install the current stable version. As there are some external dependencies (especially on windows), and a lot of windows users don't like command line there is a plan to bundle the software up into stand alone downloadable packages. This is as well as, not instead of the ability to `pip install` as this is how I want to install it.

cool! :-)

* **git sub-module** - We can agree to disagree on whether this is a good thing. But yes, seeing as GitBuilding is in a git repo you could do this if you so desired.

+1

Taking a step back. Do you really feel that having one program that automates PDFs for Standards, assembly docs, user manuals is a good thing? I would say that the formatting needed fro a PDF of a standard is very formal sectioned text. For assembly docs you want to move towards something akin to big picture, minimal text side by side. Of course a single program could do all of this but it seems to be straying from the Unix principle you are championing of doing one thing and doing it well.

I never though of it like a thing specifically geared towards specfication/norm documentation. I actually used it from the start, and still do, also for a recipee blog.
That is a good aproach I think, because it automatically makes me implement eveerything in a way that is compatible for many things.
You should also keep in mind, that MoVeDo does not do any of the grunt work, but uses tools for all of that, and these tools themselfs are quite powerfull, and all of them are geared towards general documentaiton.
For example, right now we are in the proces sto improve the title-page of the DIN SPEC 3015. This will be done by allowing to supply a custom pandoc template oand/or live-patching the default one, if that proves feasible.

So there is really nothing that is norm-specific in MoVeDo as of now.

For OKH we are interested in trying MoVeDo for the standard. as this is what it can do now. If in the future we want to join efforts to make an all singing all dancing collection of tools for OSH documentation, then we should take a step back. The only sensible way to do this is for us to:

* use and understand each others current projects

* agree not to accuse each other of unethical practices such as lock-in if we disagree on how something is implemented (i.e suggest a specific and viable alternative rather than finger pointing)

* each make a list of ideal requirements

* work together to align these requirements to a single, group agreed, set of requirements.

sounds good!

Working together to set our objectives is far more productive than one person defining a personal manifesto of requirements and berating anyone that hasn't already lived up to their arbitrary standards.

I opened a discussion here, and supplied my wish-list, explicitly stating it as "my personal wish-list", suggesting that it is not god's unquestionable list; I guess that was not clear enough then, sorry. the list is not arbitrary, but comes from my experience. You are free to extend/adjust the list in this discussion, and I would like you to, as you seem well capable to do so.

I suggest that we close this issue and that we move the discussion of wider documentation tools and formats Hardware Documentation Consensus Groups' Tool Comparison Repo. We will also be having a call this Wednesday if you would like to join.

I am in favor of moving it, but would prefer for people not to require an IEEE account to take part. Do you have something for that as well?
It recently took me 4 days to get an account there. maybe it was just the wrong time of the year, but also if not.. it is a barrier.

Was the call yesterday or is it next week?

[julianstirling]

I change my FreeCAD model from something with 7 to something with 8 screws, and I forget to adjust the text/docu. The CI build extracts BOM info from both the FreeCAD model and the docu (in your format), and sees that one has 8 screws, and the other 7, and then generates a warning/error, so a human can make a fixing commit?

That is a use case I have had in mind. As FreeCAD has no default assembly workbench this is not something we are implementing in the near term. But we are making machine readable BOMs from the output so that you can do things like consistency check between CAD and docs. I am talking to a friend who uses Solidworks to generate CSV files from assemblies, for a proof of principle. Of course Solidworks will never run on the CI unless he has his own local windows runner, but for his use case this is less important.

I opened a discussion here, and supplied my wish-list, explicitly stating it as "my personal wish-list"

My fault, I got grumpy. Sorry

but would prefer for people not to require an IEEE account to take part

My preference would have been standard GitLab. I think Tobey's plan is to try to tap into some of the standards experience of IEEE, and also to trial the IEEE platform that seems to be taking open source hardware seriously. I think at first we should raise the problem of slow account creation as a barrier with IEEE, if it becomes a problem we can migrate.

Was the call yesterday or is it next week?

The call is next week. We are a bit disorganised arranging it, I will makes sure I let you know.