txn2 / tm

WIP: TXN2 types model API

tm is used for defining the strucutre and Elasticsearch indexing rules for Messages sent to Elasticsearch from rxtx through rtBeat with the key rxtxMsg. tm Models define the properties of the rxtx payload.

The tm library defines a type of generic nested meta-data Model. The tm server creates a services for the storage, retrieval and searching of Models associated with a txn2/provision Account.

A Model consists of a record stored in the Elasticsearch index ACCOUNT-models and a corresponding Elasticsearch template (_template/ACCOUNT-data-MODEL) representing the index pattern ACCOUNT-data-MODEL-*.

Configuration

Configuration is inherited from txn2/micro. The following configuration is specific to tm:

Flag	Environment Variable	Description
-esServer	ELASTIC_SERVER	Elasticsearch Server (default "http://elasticsearch:9200")
-mode	MODE	Protected or internal modes. ("internal" = token check bypass)

Routes

Method	Route Pattern	Description
POST	/model/:account	Upsert a model into an account.
GET	/model/:account/:id	Get a model by account and id.
POST	searchModels/:account	Search for models in an account with a Lucene query.

Local Development

The project includes a Docker Compose file with Elasticsearch, Kibana and Cerebro:

docker-compose up

Run the source in token bypass mode and pointed to Elasticsearch exposed on localhost port 9200:

go run ./cmd/tm.go --mode=internal --esServer=http://localhost:9200

Examples

The following examples assume mode is set to internal and will not check a Bearer token for proper permissions.

Upsert Model

Upserting a Model will result in an Ack with a Result payload.

The following creates a model called test and will result in a record with the id test in the xorg-models index. A mapping template will be also be generated and stored at _template/xorg-data-test:

curl -X POST \
  http://localhost:8080/model/xorg \
  -H 'Content-Type: application/json' \
  -d '{
    "machine_name": "test",
    "display_name": "",
    "description_brief": "",
    "description": "",
    "data_type": "",
    "format": "",
    "parsers": null,
    "type_class": "",
    "group": "",
    "parse": false,
    "index": 0,
    "fields": [
    	{
		    "machine_name": "event_type",
		    "display_name": "Event Type",
		    "description_brief": "",
		    "description": "",
		    "data_type": "keyword",
		    "format": "",
		    "parsers": null,
		    "type_class": "",
		    "group": "",
		    "parse": false,
		    "index": 0
		},
    	{
		    "machine_name": "gps_utc_time",
		    "display_name": "GPS UTC Time",
		    "description_brief": "",
		    "description": "",
		    "data_type": "date",
		    "format": "yyyyMMddHHmmss",
		    "parsers": null,
		    "type_class": "",
		    "group": "",
		    "parse": false,
		    "index": 0
		},
		{
		    "machine_name": "location",
		    "display_name": "",
		    "description_brief": "",
		    "description": "",
		    "data_type": "nested",
		    "format": "",
		    "parsers": null,
		    "type_class": "",
		    "group": "",
		    "parse": false,
		    "index": 0,
		    "fields": [
    	    	{
				    "machine_name": "lat",
				    "display_name": "",
				    "description_brief": "",
				    "description": "",
				    "data_type": "float",
				    "format": "",
				    "parsers": null,
				    "type_class": "",
				    "group": "",
				    "parse": false,
				    "index": 0
				},
    	    	{
				    "machine_name": "lon",
				    "display_name": "",
				    "description_brief": "",
				    "description": "",
				    "data_type": "float",
				    "format": "",
				    "parsers": null,
				    "type_class": "",
				    "group": "",
				    "parse": false,
				    "index": 0
				}				
		    ]
		}
	]
}'

Get Model

Getting a Model will result in a ModelResultAck.

curl http://localhost:8080/model/xorg/test

Search Models

Searching for Models will result in a ModelSearchResultsAck.

curl -X POST \
  http://localhost:8080/searchModels/xorg \
  -d '{
  "query": {
    "match_all": {}
  }
}'

Release Packaging

Build test release:

goreleaser --skip-publish --rm-dist --skip-validate

Build and release:

GITHUB_TOKEN=$GITHUB_TOKEN goreleaser --rm-dist