Questions on ossem_converter to convert markdown to yaml about ossem

[ashwin-patil]

Got it. Thanks for the swift and detailed response. This makes sense.
I can regenerate my files in YAML with bit of excel and notepad++ as I have structure ready. Regarding my other questions, i can make it work in yaml with either char(|) to handle newline, linebreaks or by quoting the dictionary/dynamic datatypes to escape those.
I will go ahead and close the issue.

[Cyb3rWard0g]

Hey @ashwin-patil thank you for all the information. i believe it would be good to loop @hxnoyd in this since he created the script. Hey @hxnoyd , whenever you have a chance would you mind taking a look at @ashwin-patil questions about regarding the ossem_coverter.py script? Thank you man in advance.

[hxnoyd]

Hi @ashwin-patil. ossem_converter.py option to convert from markdown was a temporary feature of the script, that we used when converting the initial OSSEM data sets from markdown to YAML.

The reason why is not working for you is because the markdown parser expects the 'old' OSSEM markdown structure. The new markdown structure is generated out of YAML, thus with a different structure, that ossem_converter.py does not know how to parse. This is the reason why the code was commented ;)

Ideally, everyone authoring new OSSEM data should do it in YAML. The reason for this is because YAML provides a cleaner data structure, due to is simplicity and readability. Another reason why ossem_converter.py conversion from markdown to YAML was commented, was to avoid writing back to YAML, specially from sources prone to structure errors (like markdown), that would pollute the YAML source.

That said, I would suggest that you re-create your AWS data dictionaries in YAML, under the /source folder. After creating the YAML data, you can use ossem_converter.py to generate the markdown. This will ensure that your PR will contain the YAML and the script generated markdown.

Going back to your questions:
Q: "Does the script accepts multi line markdowns for description field ?"
A: Yes, if you are using YAML.

Q: "Some of the fields in aws data sources are of dictionary/dynamic data type which can be seen in sample value. (e.g. UserIdentity, requestParameters, responseElements in CloudTrail). Are these supported in the markdown-to-yaml conversion ?"
A: Can you provide some event data as an example? I need more information to give you an informed answer :)

Thanks!