sw-samuraj / oci-sd

Prometheus service discovery for OCI (Oracle Cloud Infrastructure)

OCI Service Discovery

Prometheus service discovery for OCI (Oracle Cloud Infrastructure).

Content

1. How it works
2. How to use it
3. How to build it
4. Configuration
5. Metadata labels
6. Example
7. License

How it works

Unfortunately, Prometheus team doesn't accept new service discovery integrations in the Prometheus code base. Instead, they propose to use file SD integration. Therefore, OCI Service Discovery works in following manner:

As should be obvious from the diagram, synchronization of monitored targets (i.e. OCI instances) happens via shared oci-sd.json file (the file name can be arbitrary):

1. OCI-SD periodically provides this file
2. Prometheus periodically consumes it.

Please note, that those discoveries/scrapings are independent of each other.

How to use it

OCI-SD can be used in two ways: either as a standalone, CLI application, or as a Golang package which can be imported to your application.

Standalone application

You can build and run a command line application oci-sd. Once properly configured (see the Configuration section), the application starts to query OCI for list of instances and write those to the oci-sd.json file.

You can run the application with default values for configuration and output files:

./bin/oci-sd

Or you can provide specific values via command-line flags:

./bin/oci-sd -c conf/oci-sd.toml -o conf/prometheus-sd.json

See an Example section for a practical example and tutorial.

Golang package

If you don't want to have a separate standalone application for service discovery, you can include the oci (and probably the adapter) package(s) into your application.

Here is an example usage:

ociConfig := oci.SDConfig{
    // populate the config
}
logger := logrus.StandardLogger()

disc, err := oci.NewDiscovery(&ociConfig, logger)
if err != nil {
    logger.WithFields(logrus.Fields{"func": "main", "err": err}).Fatal("can't create OCI discovery")
}

sdOutputFile := "oci-sd.json"
adapter = adapter.NewAdapter(ctx, sdOutputFile, "ociSD", disc, *logger)
adapter.Run()

How to build a standalone application

Prerequisites for build of oci-sd as a standalone application is go1.11+ and make:

make build

Configuration

OCI-SD configuration

Command line flags

You can run oci-sd --help to see command line configuration flags:

$ ./bin/oci-sd --help
Usage of ./bin/oci-sd:
  -t, --compartment string   compartment for discovering targets
  -c, --config-file string   external config file (default "oci-sd.toml")
  -l, --log-directory string log file location (default "/var/log/oci-sd/")
  -i, --instance-principal   initialise with instance principal authentication
  -o, --output-file string   output file for file_sd compatible file (default "oci-sd.json")
  -s, --sanitise             sanitise instance tags to fit Prometheus requirements by removing special characters (:, -)

Configuration file

The configuration file (e.g. oci-sd.toml) should contain following mandatory values:

[SDConfig]
User = "ocid1.user.1"
FingerPrint = "42:42:42"
KeyFile = "/home/user/.oci/oci_api_key.pem"
PassPhrase = "secret"
Tenancy = "ocid1.tenancy.1"
Region = "us-phoenix-1"
Compartment = "ocid1.compartment.1"

Additionally, there can be following optional values:

Port = 4242               # default port is 9100
RefreshInterval = "120s"  # default interval is 60 seconds

Prometheus configuration

Here is Prometheus configuration for the file based service discovery:

scrape_configs:
  - job_name: "oci-sd"
    scrape_interval: 5s
    file_sd_configs:
      - files:
          - oci-sd.json
        refresh_interval: 1m
    relabel_configs:
      - source_labels: ["__meta_oci_public_ip"]
        target_label: "__address__"
        replacement: "${1}:9100"

Metadata labels

The following meta-labels are available on targets during re-labeling:

• __meta_oci_availability_domain: the availability domain in which the instance is running
• __meta_oci_compartment_id: OCID of the used compartment
• __meta_oci_defined_tag_<namespace>_<tagkey>: each defined tag value of the instance
• __meta_oci_freeform_tag_<tagkey>: each freeform tag value of the instance
• __meta_oci_instance_id: OCID of the instance
• __meta_oci_instance_name: the name of the instance
• __meta_oci_instance_state: the state of the instance
• __meta_oci_private_ip: the private IP address of the instance
• __meta_oci_public_ip: the public IP address of the instance, if available

Logs

The default log location is /var/log/oci-sd/oci-sd.log. It can be changed by passing in a custom --log-directory variable.

Example

You can find an example for standalone application in the example directory. It contains:

• oci-sd configuration file.
• Terraform scripts which create 3 OCI instances with running node_exporters.
• Prometheus configuration file with pre-configured file_sd scrape config and oci-sd specific re-labeling.

See example/README.md file for more details.

License

The oci-sd application and the oci package are published under BSD 3-Clause license.

The adapter package is published under Apache 2.0 license.

oci-sd's Issues

Terraform sample needs to be modernized

The terraform example needs to be updated to work with the latest version of terraform. In addition, need to update the version of prometheus, use modern OCI shapes (Flex), modern OS (OL8), and to handle starting the services under SELinux.

output file permissions

Hello,is there an easy way to modify the file permissions of the output file. since the output file always have 600 rights, Prometheus is not able to read the output file. I have to change the permissions manually to get prometheus working. Is there something I can do beside execute every x minutes a cron job? :(

oci-sd crashes when instance doesn't have public ip

We got a situation that a returned Vnic doesn't have public ip.
As oci-sd crashes.
The Vnic looks like:
Vnic={
AvailabilityDomain=CMiL:US-ASHBURN-AD-1
CompartmentId=ocid1.tenancy.oc1...
Id=ocid1.vnic.oc1....
LifecycleState=AVAILABLE
SubnetId=ocid1.subnet.oc1.iad....
TimeCreated=2019-11-20 08:47:44.691 +0000 UTC
DefinedTags=map[]
DisplayName=ohad-instance-20191120-1046
FreeformTags=map[]
HostnameLabel=ohad-instance-20191120-1046
IsPrimary=true
MacAddress=XXX
NsgIds=[]
PrivateIp=XXX
PublicIp=
SkipSourceDestCheck=false }
Etag=3bb52290
OpcRequestId=XXX }

Build Issues for Standalone

I'm trying to build the standalone version of oci-sd, but I'm getting build errors. I'm wondering if there was a dependency change that is breaking the build process.

make build
go: error loading module requirements

For reference, I'm building with go1.12.7

could oci-sd export mds metrics?

oci not support cloud exporter, so this could suuport export cloud service metrics like mysql?

"\"__meta_oci_defined_tag_Oracle-Tags_CreatedBy\" is not a valid label name

There was a log like this.

Aug 4 02:17:13 prometheus prometheus: level=error ts=2020-08-04T02:17:13.312Z caller=file.go:323 component="discovery manager scrape" discovery=file msg="Error reading file" path =/home/user/oci-sd/oci-sd.json err="\"__meta_oci_defined_tag_Oracle-Tags_CreatedBy\" is not a valid label name"

I replaced "__meta_oci_defined_tag_Oracle-Tags_CreatedBy" with "__meta_oci_defined_tag_Oracle_Tags_CreatedBy" in oci-sd.json and ran Prometheus.
Same for "Oracle-Tags_CreatedOn".