Proposed RFC: Documentation reporting on the Feature Grid about sig-docs-community

[chanmosq]

I've read and support this RFC proposal.

It's really important for docs to have a feature grid that can accurately portray the docs' health status across O3DE's various domains - I think this is a great idea. Additionally, it brings clarity to the overall state of O3DE Docs and can help identify any gaps.

Upon acceptance, work for this RFC should be supplemented with defining the levels of docs "quality". That would help sigs make a more informed judgement of the state of their feature docs.

[chanmosq]

Requesting feedback from @o3de/sig-release, and suggest bringing this up to TSC for awareness. Since this depends on other sigs to report on the docs health status of their own features, and is involved with the release process, we would like your review on this.

The last day for feedback is Friday, Oct. 28.

[vincent6767]

I've read and supported this proposal. I suggest asking individual SIGs for feedback as this adds another responsibility and coordination effort with the SIG Docs community.

[chanmosq]

In response to @vincent6767, we will reach out to other sigs and extend the date for processing this RFC.

[willihay]

I've read and support this proposal. Getting sig involvement in the evaluation of documentation relevant to their feature areas seems like a smart idea.

A couple questions on the definitions of the columns:

1. For tutorials you include "samples" in the description:

   Tutorials: Examples, samples, and walkthroughs designed to help users with the specified feature.

   "Samples" means something very specific, and has its own column. I'd remove that from the description.

2. Do you include as part of the tutorials category the how-to topics that are often embedded in the feature documentation? I am in agreement with this, because they both involve practical steps, but it's not 100% clear to me if this is the intention, and since this category is named "tutorials", the how-to topics might be overlooked when sigs evaluate the docs. You might want to explicitly state that tutorials can include any procedural topics in any of the various guides, not just the Tutorials ("learning guide") section. You could call this category "Procedural docs".

[amzn-rhhong]

Do we have a score/feedback system on each doc page? Something like "is this article helpful?" Or "on scale of 1-5 how do you think about this article"

I'm thinking pulling data like that and use it to generate the health matrix for docs is going to really reflect how customers think.

Could be as simple as this

[chanmosq]

@amzn-rhhong Having some sort of customer assessment on the docs' quality is a good idea and something the sig should investigate in. Perhaps this is a conversation for outside of the RFC, because the feature grid is intended for sigs to report on the status of docs. And at this time, the "quality" that we report on will be based on an internal assessment of "Do we have some learning content (docs, videos, or samples) such that a user can use the feature?"

[FiniteStateGit]

1. For tutorials you include "samples" in the description:
   > Tutorials: Examples, samples, and walkthroughs designed to help users with the specified feature.
   
   "Samples" means something very specific, and has its own column. I'd remove that from the description.

Agreed, "samples" in the definition of tutorials could be substituted with how-to's, topics that demonstrate a procedural workflow.

2. Do you include as part of the tutorials category the how-to topics that are often embedded in the feature documentation? I am in agreement with this, because they both involve practical steps, but it's not 100% clear to me if this is the intention, and since this category is named "tutorials", the how-to topics might be overlooked when sigs evaluate the docs. You might want to explicitly state that tutorials can include any procedural topics in any of the various guides, not just the [Tutorials](https://www.o3de.org/docs/learning-guide/) ("learning guide") section. You could call this category "Procedural docs".

It is intended that both how-to topics normally found embedded in the user-guide and the dedicated tutorials found in the tutorial section of the site are counted towards a feature's documentation.

[lmbr-pip]

I do agree with the sentiment, though I would like to understand how doc users find and make use of this information. Is there linkage to it from o3de.org? Its not really covered how folks looking for documentation are expected to work with this information. Is it just for potential docs contributors?

As for the proposal, my main problem is that ratings still seem highly subjective and require a lot of cognitive thought to set up for SIG Chair(s).

Secondly, did you consider having a state of docs section, ie "Missing - Not Planned", Planned, In Active Development, Delivered or something to convey state of feature with relation to docs

Especially interested in how do we relate docs and features to the "dev" documentation? I may mark a feature as having rich/wonderful/completed docs but they may only be in the dev docs branch and thus invisible to most users.

[sptramer]

Do we have a score/feedback system on each doc page? Something like "is this article helpful?" Or "on scale of 1-5 how do you think about this article"

I'm thinking pulling data like that and use it to generate the health matrix for docs is going to really reflect how customers think.

Could be as simple as this

👍/👎 rankings are outside of the discussion of the feature grid at this time. That would require a separate RFC; engaging in this kind of user study requires cooperation from legal departments.

[sptramer]

@lmbr-pip full comment

Hi pip I accidentally edited your comment instead of writing my own. Misclick that led to me overwriting some of your comments but I have left which ones I kept there, and moved my edits below.

---

I do agree with the sentiment, though I would like to understand how doc users find and make use of this information. Is there linkage to it from o3de.org? Its not really covered how folks looking for documentation are expected to work with this information. Is it just for potential docs contributors?

It would be part of the feature grid release notes (https://www.o3de.org/docs/release-notes/22-10-0/feature-state/). As for how they "make use" of this information - It's unclear. There are many ways that this can go wrong with under/over-evaluation (especially RED/GREEN), even if we were taking a data-driven approach to determine "quality" or "usefulness". We may need a more objective measurement system like NPP (negative sentiment evaluation).

As for the proposal, my main problem is that ratings still seem highly subjective and require a lot of cognitive thought to set up for SIG Chair(s).

Subjective evaluations would be performed by docs + the sig, under however they choose to do so. We understand that we will be consistently YELLOW or RED under this reporting rubric. It's about "sufficiency" of documentation for the product: Can a user onboard? Can they understand it? Is there full reference available?

Engineering teams who take dedicated time to focus on docs would be able to establish a baseline of quality ("is it better / worse than last time? Did we add stuff? Are things still missing? Do users report issues?"). We may need to hold this RFC until there's a way to perform better evaluations.

Secondly, did you consider having a state of docs section, ie "Missing - Not Planned", Planned, In Active Development, Delivered or something to convey state of feature with relation to docs

This would be the intent of a roadmap, not a feature grid. Feature grids are supposed to be for snapshots in time.

Especially interested in how do we relate docs and features to the "dev" documentation? I may mark a feature as having rich/wonderful/completed docs but they may only be in the dev docs branch and thus invisible to most users.

Do you report on the dev feature in the feature grid? If so, docs for the feature are also reported in the feature grid, on the same line item.

[chanmosq]

At sig-docs-meeting on 11/08/2022, we unanimously provisionally accepted this RFC, with the understanding that we need to establish a system of metrics for evaluating the quality, and address other concerns that are commented on this issue.