gchq / event-logging-schema Goto Github PK

The example for docs/schema-actions/process has an invalid value of "Split" for EventDetail/Process/Action which is an enumerated type

<File> allows <Hash> but it is app/system specific what that means.

Perhaps an enum MD5|SHA-1|SHA-256| (some others?)|Other

LocationComplextType should contain an additional three elements (xs:string types)

• Town - The location Town, e.g. Hollywood
• City - The location City, e.g. Los Angles
• State - The location State or Province, e.g. California

Summary
The descriptions for Door/SingleEntry and Door/RemoveAll suggest these elements may be omitted, however the constraints don't reflect this.

Proposal
Adjust constraints for Door/SingleEntry and Door/RemoveAll to have minOccurs=0

We should consider adding multiple Data elements for the Create, Delete, Network and View 'action' elements. All other 'actions' allow for multiple Data elements to carry additional information. These four are the only ones that don't. The only alternative is to place the 'action information' in Data elements of the 'subject of the action' (e.g. Object, Document, File, etc) which is NOT appropriate.

Ideally, all events are created on a locked down server and only ever transferred securely. But in some organisations / situations it might be appropriate or necessary to:

• Transfer via Syslog UDP,
• Transfer manually,
• Create on a client-side process under user control.
• Etc.

Therefore, it would be useful to record the level of veracity of the event, so it can be taken into account later, as necessary (e.g. treated with a degree of suspicion!)

Don't know the best way to do this, as ideally some information about the provenance of the event would be recorded. But it is probably sufficient to roll up into a single value (High/Medium/Low), at least for a first version.

Even when an operation does not fit nicely into the schema it can still be logged, as an <Unknown>. Such operations may fail, but there is not currently an <Outcome> element making it impossible to accurately record such an event.

Please add <Outcome> to <Unknown> to remedy this.

Consider adding unbounded Data elements to primary ComplexTypes. For example add to
EventTimeComplexType, SystemComplexType, DoorType, NetworkEventActionComplexType

If you have multiplte Meta blocks, e.g. using one for source data and another for something else then you need some way of distinguishing between them. Currently it has ContentType and Version attributes, so rather tan abusing them it would be better to have some other field such as Id, key or type.

The restriction/enumeration value attribute should change from 3.0.0 to 3.1.0 or remove the enumeration.
IE Change

 <xs:simpleType name="VersionSimpleType">
        <xs:annotation>
            <xs:documentation>Type for specifying version numbers in a common way, e.g. 2.4</xs:documentation>
        </xs:annotation>
        <xs:restriction base="xs:string">
            <xs:pattern value="[0-9]+(\.[0-9]+)*"/>
            <xs:enumeration value="3.0.0"/>
        </xs:restriction>
    </xs:simpleType>

to

 <xs:simpleType name="VersionSimpleType">
        <xs:annotation>
            <xs:documentation>Type for specifying version numbers in a common way, e.g. 2.4</xs:documentation>
        </xs:annotation>
        <xs:restriction base="xs:string">
            <xs:pattern value="[0-9]+(\.[0-9]+)*"/>
            <xs:enumeration value="3.1.0"/>
        </xs:restriction>
    </xs:simpleType>

Location.Floor is currently of type Integer. We have a site with a floor named "Ground". Please change Floor type to string.

Currently, the TimeZone element of LocationComplextType records

Timezone pair for location in RFC3339 time-numoffset format separated by a slash '/' character. The first offset is for the location's standard timezone and the second for the daylight savings timezone. For example, Bristol, UK would be +00:00/+01:00

It is recommended that this changes to

Timezone for location in ISO8601 Time zone designator format for the location at the time of the event. The format may be 'Z' for no offset from UTC, '+hh:mm', '+hhmm' or '+hh' for a positive offset from UTC in hours and minutes or just hours or '-hh:mm' or '-hhmm' or '-hh' for a negative offset from UTC in hours and minutes or just hours. Note that 'Z' can be represented as a positive zero offset from UTC i.e. '+00:00'

In situations where a user is viewing the results of a previous search, e.g. when the search results have been saved, the options available to model this with a View schema action are not ideal. Currently under View you have 1..many of the MultiObjectComplexType objects, one of which is SearchResult. SearchResult only really has Relevancy and Data, with no means of recording details of the items in that result set.

There is also ambiguity over whether SearchResult means a single items in a collection of results from a search, or the whole collection of items found in a single search.

Add:
CachedRemoteInteractive
CachedUnlock
Proxy

Sometimes events are received well after when there were created. The addition of a TimeReceived element can record when the event was received by the Event Repository (e.g Stroom proxy, Stroom Application, Syslog server, etc).

Some events have a duration as opposed to just a single point in time. The addition of a TimeEnded element can record when such events ended. (E.G. Connection events)

Following discussions on slack it was agreed that the way we model events that are part of some hierarchy, e.g. one event is the child of another, or events that form part of a session needs improving.

The proposed solution is

  <EventSource>
    <EventId>abcdefg</EventId> <!-- A unique id known to the source system that created the event -->
    <SessionId>JSESSION12345</SessionId> <!-- A session id if the event was created as part of a session (can be used to associate events by session) -->
  </EventSource>
  <EventDetail>
    <!-- leave alone -->
  </EventDetail>
  <EventParent> <!-- Can we change EventChain to EventParent - might be more obvious unless we think we are describing sibling rather than parent child relationships? -->
     <EventId>12344</EventId> <!-- the Id (EventSource/EventId) from the parent event -->
     <EventParent>
        <EventId>12343</EventId> <!-- the Id (EventSource/EventId)from the parent of the parent event -->
      </EventParent>
  </EventParent>
</Event>

EventParent would replace EventChain

In this location
https://github.com/gchq/event-logging-schema/tree/master/docs

the README.md has an incorrect link specified for the "Illustrative Examples"
https://github.com/gchq/event-logging-schema/blob/master/docs/completeExamples/completeExamples.md
(completeExamples.md does not exist"

Should use this URL
https://github.com/gchq/event-logging-schema/tree/master/docs/completeExamples

There is currently no sensible way of modelling a the approval/acceptance/rejection of an entity in the schema. For example being able to approve/reject a licence agreement or acceptable use policy.

Will allow modelling of CRUD operations on virtual devices.

Currently the PrintSettings element can record PaperSize, Orientation, Colour and DeviceFonts. There can be other print settings, for example

• Duplex Printing
• Double Sided
• etc

Rather than provide additional specific elements, allow of addition settings via generic Data elements.

Summary
Authenticate/Action enum values don't adequately describe the action of authenticating with a physical or virtual token. Some examples:

1. Smartcard
2. Door access card
3. MFA physical or virtual token

Likewise, Authenticate/LogonType doesn't cater for physical access events either.

Proposal

1. Add ReadToken to AuthenticateActionSimpleType enum
2. Add PhysicalAccess to AuthenticateLogonTypeSimpleType enum

This ensures we have a known class name in the event-logging library and also works around gchq/stroom#2134

This structure has a Name and a Value attribute, but these are both strings. There might be situations where complex data should be recorded.

Adding an xs:any element called Value could easily provide this capability.

Though maybe the name should be ComplexValue to make it less likely that the element is used where the attribute would be better.

Currently we have the following which limits the permission types granted to an enumerated list. We need to be able to support other types.

            <xs:element name="Permission" minOccurs="1" maxOccurs="unbounded">
              <xs:annotation>
                <xs:documentation>A permission rule associated with an object, e.g. read and write access being granted to a user.</xs:documentation>
              </xs:annotation>
              <xs:complexType>
                <xs:annotation>
                  <xs:documentation>A permission rule associated with an object, e.g. read and write access being granted to a user.</xs:documentation>
                </xs:annotation>
                <xs:sequence minOccurs="1" maxOccurs="1">
                  <xs:choice minOccurs="1" maxOccurs="1">
                    <xs:element name="User" type="evt:UserComplexType" minOccurs="1" maxOccurs="1">
                      <xs:annotation>
                        <xs:documentation>A user that has been granted (or is prevented from having) some form of permission.</xs:documentation>
                      </xs:annotation>
                    </xs:element>
                    <xs:element name="Group" type="evt:GroupComplexType" minOccurs="1" maxOccurs="1">
                      <xs:annotation>
                        <xs:documentation>A named group of users that has been granted (or is prevented from having) some form of permission.</xs:documentation>
                      </xs:annotation>
                    </xs:element>
                  </xs:choice>
                  <xs:element name="Allow" type="evt:PermissionAttributeSimpleType" minOccurs="0" maxOccurs="unbounded">
                    <xs:annotation>
                      <xs:documentation>The permission attributes that have been explicitly allowed.</xs:documentation>
                    </xs:annotation>
                  </xs:element>
                  <xs:element name="Deny" type="evt:PermissionAttributeSimpleType" minOccurs="0" maxOccurs="unbounded">
                    <xs:annotation>
                      <xs:documentation>The permission attributes that have been explicitly denied.</xs:documentation>
                    </xs:annotation>
                  </xs:element>
                </xs:sequence>
              </xs:complexType>
            </xs:element>

  <xs:simpleType name="PermissionAttributeSimpleType">
    <xs:annotation>
      <xs:documentation>The types of permission that can be assigned to an entity such as a document.</xs:documentation>
    </xs:annotation>
    <xs:restriction base="xs:string">
      <xs:enumeration value="Author"/>
      <xs:enumeration value="Owner"/>
      <xs:enumeration value="Read"/>
      <xs:enumeration value="Write"/>
      <xs:enumeration value="Execute"/>
    </xs:restriction>
  </xs:simpleType>

To avoid breaking the validation for existing documents the suggestion is to change the enum to regex pattern that will allow for other custom permission names. If we use the pattern ^(Author|Owner|Read|Write|Executor|Other: .*)$ then we can have values like:

<Allow>Owner</Allow>
<Allow>Other: Create Pipeline</Allow>
<Allow>Other: Create Folder</Allow>

The downside to this is that we lose the Java enum in the jaxb library, but it will still be enforced with schema validation.

Relates to gchq/stroom#2858

We need to migrate all the content over to hugo and create a versioned site so we have one version of the site for each schema release.

The ResourceComplexType sub-elements InboundSize, InboundContentSize, OutboundSize and OutboundContentSize which are transfer byte counts should be declared as xs:nonNegativeInteger rather than xs:positiveInteger to allow for zero byte counts which certain log sources return (e.g. Squid, Apache Httpd).

The RequestTime microsecond count should also be a xs:nonNegativeInteger for sometimes, certain log sources return 0 as the count due to resolution issues.

I realise that the absence of the element can indicate a zero size, but given the original log contains this value then perhaps the long term record should also contain the value. This is particularly important when exporting this information from Stroom to support an alternative capability that may expect a zero value given the original log contained them.

In many cases, an event source maintains a unique identifier (to the source capability). It is requested that we create an element to hold this source identifier.

It is suggest we use Event/EventSource/SourceEventId and this should be of type xs:string.

It is also suggested that users make use of an interim element Event/EventSource/Data[@Name='EventSourceId']/@Value and any future schema uplift (from one schema version to the next) translations apply this transform.

Currently the events are decorated on receipt with identifiers that identify the event within the instance of stroom. We do not want elements/attributes in the schema that are not intended for use by generating systems. Following discussions on slack, the proposed solution is to make use of the existing xs:any part of the Event/Meta element to hold an xml record belonging to a different namespace. This can then be used to hold any data specific to stroom's receipt/storage of the event.

<Event> <!-- Make sure there is never an Id attribute here (it did exist in some versions of the schema) -->
  <Meta>
      <stroom:meta>
         <stroom:id> <!-- the globally unique identifier for the event -->
         <stroom:sourceLocation>
             <stroom:streamId>12345</stroom:streamId>
             <stroom:recordNo>1</stroom:recordNo>
             <stroom:lineFrom>3</stroom:lineFrom>
             <stroom:colFrom>1</stroom:colFrom>
             <stroom:lineTo>4</stroom:lineTo>
             <stroom:colTo>1</stroom:colTo>
         </stroom:sourceLocation>
         <stroom:provenance>
              <!-- the pipeline id + version + input that spawned it, and so on up the chain -->
         </stroom:provenance>
      </stroom:meta>
  </Meta>

stroom:id would probably be a uuid generated when the event was received and translated into xml. This would provide a unique id for the event so it can be referred to or for identifying it if it is stored outside of stroom. This element would replace Event/Id.

This record would be controlled by a separate scheme controlled outside of this repo.

A system can reside in a Security Domain ... for example the the may be an instance of a System in a Secret security domain and another instance in a Official domain. Equally, one could have domains of Sales , Finance and Research.

This allows to categorize the System itself.

This proposal suggests adding an element called SecurityDomain as a string.

Annotation could be 'An optional element to define the domain that the system exists in.'

Please add between Organisation and Version.

Summary
With physical access control systems, there are cases where mechanisms like doors, are controlled by a computer systems. In cases where audit is being obtained from such systems, detail about both the system and the mechanism (e.g. door) needs to be reported. Logically, the EventSource is the place for that. The current schema release doesn't permit Device, Client or Server to be combined with Door.

Proposal
Make EventSource/Door an optional element and move it into the sequence that currently comprises Device, Client and Server.

Enumeration PermissionAttributeSimpleType may be Author, Owner, Read, Write, Execute.

Please add values Create, Update. Delete, Use

It should be unbounded like PermittedNationalities.

<xs:element name="PermittedOrganisations" minOccurs="0">
...
    <xs:complexType>
        <xs:sequence>
            <xs:element name="PermittedOrganisation" type="evt:OrganisationComplexType" maxOccurs="unbounded">
...
            </xs:element>
        </xs:sequence>
    </xs:complexType>
</xs:element>

Would save a bit of rework

There is a need to be able to tag/categorise objects/data described in the schema. It should be possible to associate an object under audit with multiple tags/categories and the tags/categories will likely be domain specific. This is akin to adding tags to photos in Flickr or the topics associated with GitHub repositories.

In order to produce sensible generated java code with the fluent builder xjc plugin the schema needs to be refactored to rename some types and factor anonymous complex types into named ones.

Some of the deprecated elements will also be removed.

The ClassificationComplexType is too simplistic for describing the protective marking of events and objects being referred to in an event. We need a richer classification scheme that will be appropriate for a variety of organisation's classification schemes.

Currently there is no way to model events relating to cloud services, e.g. when a user creates a Lambda in AWS. Currently you would need to model this as Create - Object and put Lambda as the type.

Suggest we add a new object type of CloudService to cover these cases.

We need a way to allow the events or objects described in the events to be extended or decorated using content/structures described outside of the event-logging schema.

For example #30 potentially requires events to be decorated with some concept of the veracity of an event. The veracity could be described by its own XML schema (or some other data format) and the content included in a Meta element within the event.

An element can't be of a type that is declared abstract

When a user elevates their privilege on Linux, it should be considered as an Authentication activity. Although it could be modeled as a Login, I believe it would be better if it had is own action.

This may also mean we need to add a GroupComplexType choice to the Authenticate/{Device,User} choice as one can elevate their group privileges as well as user - see #13

We should also add an Other value as a catch-all. One can then use the EventDetail/Description element to advise what the Other value means for the given event.

As one can perform an Authenticate action on a Linux Group, the objects that Authenticate can be applied to (User, Device) should be expanded to include Group - a evt:GroupComplexType object type.

We need some way of categorising the source system using a set of labels/tags that are not mutually exclusive. This would provide the means int stroom to easily filter data/processing based on the type of system.

This would the same structure as used for #31.

Not all emails have a parseable Date timestamp, but it is the timestamp recorded of the email. We should have a Date String element within the EmailComplextType to represent accurately the Date timestamp recorded in an email.

To cater for more printer actions it is suggested the PrintActionSimpleType enumeration to be extended to also hold
PauseJob, ResumeJob, FailedPrint and to cover for all other eventualities Other

Equivalent to the SQL IN clause and to support in dictionary in stroom.

The current UpdateComplexType supports the recording of the before and after states of an audited entity. Systems may not always have access to both states. E.g. if they are adding an extra permission to a user but do not have access to all the before or after states.

In these cases it may be preferable to be able to record a delta instead, i.e. just the information about what has changed, i.e. the addition of permission X.

The proposal is to add a choice to the existing UpdateComplexType like so

where DeltaType would be enumerated to Add|Remove.

If we add the Delta element then we could consider making Before mandatory, but that would introduce a breaking change.

In a number of scenarios, a file may have mutliple hashes, perhaps both cryptograpgic and fuzzy (similarity). As a result, the Hash element of BaseFileComplexType should be unbounded to support this. A'la

...
<xs:element name="Hash" type="evt:HashComplexType" minOccurs="0" maxOccurs="unbounded">
   <xs:annotation>
      <xs:documentation>Describes the output of a hash function and the type of has function used.</xs:documentation>
   </xs:annotation>
</xs:element>
...

The LocationComplexType currently has numerous fields for describing the location of an entity (e.g. a device) using fields like country, site, building, etc. It would be useful to be able to capture the latitude and longitude values where these are know. It would seem sensible to require these values in decimal degrees format, e.g. 51.4419646,-0.2760163