Google Cloud Platform (GCP) Stackdriver collects logs and metrics from your cloud services. You can use Google Cloud Pub/Sub to forward your logs from Stackdriver to Logz.io.
Before you begin, you'll need: Google Cloud SDK, a GCP project, a GCP Pub/Sub topic and subscribers to your GCP project
Set up a sink to export your logs to Stackdriver.
For more information, see Exporting with the Logs Viewer from Google Cloud.
Create a working directory for this step and cd
into it.
mkdir logzio-stackdriver && cd logzio-stackdriver
You'll need to build a credentials file so Pub/Sub can authenticate and get the right permissions.
This is simpler through the command line, but if you don't have the right permissions, you can use the Cloud Console.
Build your credentials file using your Google Cloud project ID.
wget https://raw.githubusercontent.com/logzio/logzio-pubsub/master/Makefile \
&& make PROJECT_ID=<project_id>
Run this command for each project you're working with.
Go to your project's page in GCP Console. In the left menu, select IAM & admin > Service accounts.
At the top of the Service accounts page, click + CREATE SERVICE ACCOUNT.
Give a descriptive Service account name, such as "credentials file". Click CREATE to continue to the IAM/permissions page.
Add these roles:
- "Pub/Sub Editor"
- "Pub/Sub Publisher"
- "Pub/Sub Subscriber"
Click CONTINUE to continue to Grant users access to this service account.
Click + CREATE KEY to open the Create key panel. Select JSON and click CREATE to save the private key to your machine.
Copy it to the logzio-stackdriver/
folder you created
at the beginning of this step.
Click DONE to return to teh Service accounts page.
Make pubsub-input.yml
, which will hold your Pub/Sub input configuration.
touch pubsub-input.yml
Open pubsub-input.yml
in your text editor, and paste this code block.
Complete configuration instructions are below the code block. 👇
listener: <<LISTENER-HOST>>
pubsubs:
- project_id: PROJECT-1_ID
topic_id: TOPIC-1_ID
token: <<SHIPPING-TOKEN>>
credentials_file: ./credentials-file.json
subscriptions: ["SUB1_ID", "SUB2_ID", "SUB3_ID"]
type: stackdriver
- project_id: PROJECT-1_ID
topic_id: TOPIC-2_ID
token: <<SHIPPING-TOKEN>>
credentials_file: ./credentials-file.json
subscriptions: ["SUB1_ID", "SUB2_ID", "SUB3_ID"]
type: stackdriver
- project_id: PROJECT-3_ID
topic_id: TOPIC-1_ID
token: <<SHIPPING-TOKEN>>
credentials_file: ./credentials-file.json
subscriptions: ["SUB1_ID", "SUB2_ID", "SUB3_ID"]
type: stackdriver
Parameter | Description |
---|---|
listener | The Logz.io listener host. Replace <<LISTENER-HOST>> with your region's listener host (for example, listener.logz.io ). For more information on finding your account's region, see Account region. |
pubsubs | This is an array of one or more GCP subscriptions. For each subscription, provide topic and subscriptions IDs, as given from Pub/Sub. |
token | Your Logz.io shipping token. Include this with each project under pubsubs . Replace <<SHIPPING-TOKEN>> with the token of the account you want to ship to. |
Download the logzio/logzio-pubsub image:
docker pull logzio/logzio-pubsub
Run this command from logzio-pubsub/
,
where you stored pubsub-input.yml
and credentials-file.json
.
docker run --name logzio-pubsub \
-v $(pwd)/pubsub-input.yml:/logzio-pubsub/pubsub-input.yml \
-v $(pwd)/credentials-file.json:/logzio-pubsub/credentials-file.json \
logzio/logzio-pubsub
Spin up your Docker containers if you haven’t done so already.
Give your logs some time to get from your system to ours, and then open Kibana.
0.0.5:
- Updated a new public SSL certificate.