Comments (6)
We have fixed our pipeline to alway reference the GCP project but, wanted to let the group know still.
from hush-house.
Thanks for letting us know @jkjell !
Assuming that this is coming through the metadata server (I might be wrong): as we're able to configure garden properties by making use of environment variables, we're able to configure google's internal metadata server to be blocked from the containers that run as
steps/checks in our machines:
worker:
replicas: 1
env:
- name: CONCOURSE_GARDEN_DENY_NETWORK
value: "169.254.169.254/32"
Regarding permissions granted to those VMs, we're not really able fully remove all of the current permissions as we need those in order to have the provisioning of new disks and
other administrative functions - we could perhaps reduce them, but not make them a full "no permissions granted".
I'll try to validate that soon and follow up with a PR
thx!
from hush-house.
Update: we added the deny rule to the workers - we'd need now to ensure that it's indeed blocking and it does what we expect (see
hush-house/deployments/with-creds/worker/values.yaml
Lines 34 to 35 in fb44c29
from hush-house.
We confirmed that the deny rule blocks the containers from being able to access the metadata server that the underlying host can access. That rule is applied now to all shared GCP workers on Hush House so this shouldn't happen anymore!
We provisioned a new worker without the rule turned on, and when we ran gcloud info
we could reproduce this behaviour, which explains why without the --project
flag, the gcloud CLI was falling through to the service account credentials used to provision the GKE cluster:
Account: [[email protected]]
Project: [cf-concourse-production]
Current Properties:
[core]
project: [cf-concourse-production]
account: [[email protected]]
disable_usage_reporting: [True]
After applying the rule, the gcloud info
output is much more locked down:
Account: [None]
Project: [None]
Current Properties:
[core]
disable_usage_reporting: [True]
from hush-house.
As of now, running fly execute
with
---
platform: linux
image_resource:
type: registry-image
source:
repository: platforminsightsteam/base-ci-image
run:
path: gcloud
args: ["info"]
shows Project: [cf-concourse-production]
. We intercepted a similar container and were able to curl 169.254.169.254
with no errors. We inspected /proc/$(pgrep gdn)/cmdline
on multiple worker pods, and saw that --deny-network169.254.169.254/32
does indeed appear. Something fishy is going on here.
from hush-house.
We have confirmed, as @cirocosta suspected, that concourse/concourse#5159 is the culprit here. We used the following docker-compose.yml:
version: '3'
services:
concourse-db:
image: postgres
environment:
POSTGRES_DB: concourse
POSTGRES_PASSWORD: concourse_pass
POSTGRES_USER: concourse_user
PGDATA: /database
concourse:
# digest:
# before PR sha256:488638b0651e1e6cc884876499970a181ef63f1b2b02b6b9718ca1383c51a0b4
# (https://ci.concourse-ci.org/teams/main/pipelines/concourse/jobs/build-rc-image/builds/86)
# after PR sha256:49837094a16050e64a02e8f100a1992084f89505fdddd98e48aae8aa5355b4b4
# (https://ci.concourse-ci.org/teams/main/pipelines/concourse/jobs/build-rc-image/builds/87)
image: concourse/concourse-rc@<digest>
command: quickstart
privileged: true
depends_on: [concourse-db]
ports: ["8080:8080"]
environment:
CONCOURSE_POSTGRES_HOST: concourse-db
CONCOURSE_POSTGRES_USER: concourse_user
CONCOURSE_POSTGRES_PASSWORD: concourse_pass
CONCOURSE_POSTGRES_DATABASE: concourse
CONCOURSE_EXTERNAL_URL: http://localhost:8080
CONCOURSE_ADD_LOCAL_USER: test:test
CONCOURSE_MAIN_TEAM_LOCAL_USER: test
CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: overlay
CONCOURSE_GARDEN_DENY_NETWORK: 172.217.1.174/32 # google.com
and ran fly execute
against both versions using this task.yml:
---
platform: linux
image_resource:
type: registry-image
source:
repository: appropriate/curl
run:
path: curl
args: ["google.com"]
Before the PR, we got curl: (7) Failed to connect to google.com port 80: Connection refused
, but after the PR we got
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
Given that the change in concourse/concourse#5159 causes Garden (via kawasaki) to prepend iptables rules, and setting --deny-network
appends them, we feel a bit discouraged when deciding how to address this "leaking GCP metadata" use case. Admittedly, neither @pivotal-jamie-klassen or I know a whole lot about iptables, so maybe there is some place we can configure garden to definitely block traffic to GCP's metadata server while still allowing outbound traffic from containers running in greenhouse (windows containers).
from hush-house.
Related Issues (20)
- upgrade stackdriver HOT 1
- Have log-based metric as code (terraform) HOT 2
- switch `vault` in nci to `vault-nci` HOT 3
- Update Concourse chart to use `concourse/concourse-chart`
- metrics: capture pod state change events
- metrics: node-pool error panel
- How to get secret from the old vault HOT 2
- Restrict access to the internal network on garden HOT 1
- access to the internal network on garden
- jaeger: increaese retention and use a pesistent storage type
- document force removal of nodes from the cluster
- deployments: have `cert-manager` in code
- Ensure jaeger-related pods don't land in worker node pools
- Scale Strabo Workers to 0
- Migration from helm2 to helm3 HOT 2
- Worker termination stuck for 47d HOT 1
- use datadog APM for traces HOT 1
- get rid of unnecessary metrics infrastructure for hush-house
- Investigate worker recovery for k8s ungracefully restarting pods out-of-band due to high memory consumption HOT 12
- Upgrade node pools to v1.15.4-gke.22 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hush-house.