xmidt-org / codex-deploy

Codex provides a historical context about devices connected to XMiDT.

codex

Codex provides a historical context about devices connected to XMiDT.

Summary

Codex accepts incoming events, stores them in a database, and provides event information by device id. This repo is a library of packages used to implement codex.

Table of Contents

• Code of Conduct
• The Pieces
• Install
• Deploy
• Contributing

Code of Conduct

This project and everyone participating in it are governed by the XMiDT Code Of Conduct. By participating, you agree to this Code.

The Pieces

• Database: Any postgres or cassandra database will work. In deploy/, yugabyte is used. The services connect to the database using the codex-db library.
• Svalinn: Registers to an endpoint to receive events (Optional). Has an endpoint that receives events as WRP Messages, parses them, and inserts them into the database.
• Gungnir: Has endpoints that provide device information from the database.
• Fenrir (Deprecated): Deletes old records from the database at an interval.
• Heimdall: A consumer of Codex (sends requests to Gungnir) but also requests directly to the database to get a list of device IDs.

Install

This repo is a library of packages used for the codex project. There is no installation. To install each service, go to their respective READMEs.

Deploy

for deploying the project in Docker, refer to the deploy README.

Contributing

Refer to CONTRIBUTING.md.

codex-deploy's Issues

Add inserted and read record amount metrics

https://github.com/Comcast/codex/blob/bc5dbece9c78120c5c55336164ad5a8a08560529/db/metrics.go#L113

Similar to the number of records deleted, add a number of records inserted and number of records read metrics.

Remove Deprecated `webpa-common`

This ticket will outline a general todo list for the removal of webpa-common

TODO

• Review usage & outline replacements

Blacklist

Add a function within the db package to support getting the full blacklist from a table in the database.
Also, update docker-compose to create this table in the db-init script.

part of xmidt-org/svalinn#44

Add DB Fail Over List

If the database cluster the package is talking to goes down, there should be a back up list of clusters to try.

Think of it as a primary cluster and a backup list. So if the primary cluster comes back, the active connection should go from the backup list back to the primary.

Add symmetric key support for encryption/decryption

Add a metric for success rate

Upon ultimate success or failure when retrying, we add to this total metric so we can divide successes by this metric for a success rate that doesn't include failures that are rectified during retries.

Note: can conflict with #61

Improve Retry Logic

Currently, when an insert fails with n number of records. The exact same query will be rejected.

To improve upon this I recommend having a retry channel where records get added upon to try again in a batch and as a last result do an insert one record at a time.

Switch DB to yugabyte

codex-deploy/deploy/docker-compose/docker-compose.yml

Line 52 in b8ccebc

image: cockroachdb/cockroach:v2.1.4

both svalinn and gungnir are now using the cassandra api with yugabyte.

Use a set instead of a queue

https://github.com/Comcast/codex/blob/74d788fa0c00f6b33c836ef4145712292aa01868/db/batchDeleter/batchDeleter.go#L70

Use a set instead of a queue, so we don't store duplicates.

Add cucumber tests for each individual service

create mock db data and don't bring up other services when testing a specific service (svalinn/gungnir/fenrir)

Remove Event struct

wrp.SimpleEvent will be used instead

Update docker-compose process

Update the codex docker-compose process to be like xmidt-org/xmidt#12 implemented here xmidt-org/xmidt#15

Should we increase our docker-compose version?

codex-deploy/deploy/docker-compose/docker-compose.yml

Line 1 in c1c4d90

version: '3.4'

Is there any benefit to bumping the version?

change timestamp to use nanoseconds instead of seconds

Convert .Unix() to .UnixNano()

CVE-2022-28948 (Medium) detected in github.com/go-yaml/yaml-v2.4.0 - autoclosed

CVE-2022-28948 - Medium Severity Vulnerability

Vulnerable Library - github.com/go-yaml/yaml-v2.4.0

YAML support for the Go language.

Dependency Hierarchy:

• github.com/xmidt-org/webpa-common/xmetrics/xmetricstest-v1.11.4 (Root Library)
  • github.com/xmidt-org/webpa-common/xmetrics-v1.11.4
    • github.com/xmidt-org/webpa-common/logging-v1.11.4
      • github.com/spf13/viper-v1.7.1
        • ❌ github.com/go-yaml/yaml-v2.4.0 (Vulnerable Library)

Found in base branch: main

Vulnerability Details

An issue in the Unmarshal function in Go-Yaml v3 causes the program to crash when attempting to deserialize invalid input.

Publish Date: 2022-05-19

URL: CVE-2022-28948

CVSS 3 Score Details (5.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Local
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-fm53-mpmp-7qw2

Release Date: 2022-05-19

Fix Resolution: v3.0.0

---

Step up your Open Source Security Game with Mend here

Rename master branch to main

Also have to change references to the branch in .travis.yml, README, and CONTRIBUTING. Double check any other markdown files as well - sometimes links have the branch name in them.

Make type for enum

https://github.com/Comcast/codex/blob/58c6c25d3575e5dc964b78156c7684bb2ec76f94/db/event_types.go#L22

make a type that extends int for this enum set.

Fix code climate badge

Add Heimdall to README

github.com/xmidt-org/heimdall

Add Prometheus to local deployment

add a prometheus to scrape in the docker-compose file: https://github.com/Comcast/codex/blob/master/deploy/docker-compose/docker-compose.yml

a consul might also be needed for this to work correctly.

Add fenrir to deploy and docker-compose

remove records from log

https://github.com/Comcast/codex/blob/11332701ea635137cfe1eeb80ad40f7f285cb724/db/db.go#L317

Fix deploy repo to call yb-master yb-manager

codex-deploy/deploy/docker-compose/docker-compose.yml

Line 38 in 24c5544

yb-master:

Could be in other files too.

Extend the wrp to include the birthdate

Unable to start codex-deploy

Hi @joe94 and @kristinapathak ,

After running bash deploy.sh , we are getting following logs:
Creating network "docker-compose_back-tier" with the default driver
Creating prometheus ... done
Creating yb-manager-n1 ... done
Creating yb-tserver-n1 ... done
Creating svalinn ... done
Creating gungnir ... done
Connection error: ('Unable to connect to any servers', {'172.23.0.4': IndexError('list index out of range',)})

The error seems to be coming from yugabyte db instantiation. Can you please check on these?

Regards,
Utsav

invalid parameter

https://github.com/Comcast/codex/blob/ce949764b9adfc90a23a98c00d9c175865c00cd6/db/db.go#L166

ssl=true should not be part of the connection url.

Event List

Do you have a good starting point to figure out which events are sent from RDKB devices?

Add `kid` and `alg` to the Record struct

to indicate how the data is encrypted; if no encryption, alg should be none and kid should be empty