Coder Social home page Coder Social logo

follow-me-install-kubernetes-cluster's Introduction

和我一步步部署 kubernetes 集群

dashboard-home

本系列文档介绍使用二进制部署 kubernetes v1.16.6 集群的所有步骤(Hard-Way 模式)。

在部署的过程中,将详细列出各组件的启动参数,它们的含义和可能遇到的问题。

部署完成后,你将理解系统各组件的交互原理,进而能快速解决实际问题。

所以本文档主要适合于那些有一定 kubernetes 基础,想通过一步步部署的方式来学习和了解系统配置、运行原理的人。

本系列系文档适用于 CentOS 7 及以上版本系统,随着各组件的更新而更新,有任何问题欢迎提 issue!

由于启用了 x509 证书双向认证、RBAC 授权等严格的安全机制,建议从头开始部署,否则可能会认证、授权等失败!

从 v1.16.x 版本开始,本文档做了如下调整:

  1. 容器运行时:用 containerd 替换 docker,更加简单、健壮;相应的命令行工具为 crictl;
  2. Pod 网络:用 calico 替换 flannel 实现 Pod 互通,支持更大规模的集群;

新增指标监控系统:使用主流的 Prometheus、Grafana 技术栈实现集群指标采集和监控;

如果想继续使用 docker 和 flannel,请参考附件文档。

历史版本

步骤列表

  1. 00.组件版本和配置策略
  2. 01.初始化系统和全局变量
  3. 02.创建CA根证书和秘钥
  4. 03.部署kubectl命令行工具
  5. 04.部署etcd集群
  6. 05-1.部署master节点.md
    1. 05-2.apiserver集群
    2. 05-3.controller-manager集群
    3. 05-4.scheduler集群
  7. 06-1.部署woker节点
    1. 06-2.apiserver高可用之nginx代理
    2. 06-3.containerd
    3. 06-4.kubelet
    4. 06-5.kube-proxy
    5. 06-6.部署calico网络
  8. 07.验证集群功能
  9. 08-1.部署集群插件
    1. 08-2.coredns插件
    2. 08-3.dashboard插件
    3. 08-4.kube-prometheus插件
    4. 08-5.EFK插件
  10. 09.部署Docker-Registry
  11. 10.清理集群
  12. A.浏览器访问apiserver安全端口
  13. B.校验TLS证书
  14. C.部署metrics-server插件
  15. D.部署Harbor-Registry

在线阅读

电子书

打赏

如果你觉得这份文档对你有帮助,请微信扫描下方的二维码进行捐赠,加油后的 opsnull 将会和你分享更多的原创教程,谢谢!

weixin_qr.jpg

广告位

版权

Copyright 2017-2020 zhangjun ([email protected])

知识共享 署名-非商业性使用-相同方式共享 4.0(CC BY-NC-SA 4.0),详情见 LICENSE 文件。

follow-me-install-kubernetes-cluster's People

Contributors

ci-jie avatar clouduol avatar crasshopper avatar doomzhou avatar duoyichen avatar hanaasagi avatar hylarucoder avatar iutx avatar jackviewhigh avatar jmgao1983 avatar kaiser1103 avatar kermit-ye avatar khs1994 avatar klbjlabs avatar leroy-chen avatar luhuisicnu avatar mdh67899 avatar morfies avatar okzhchy avatar onesafe avatar opsnull avatar oyb001 avatar pomelowang avatar radaren avatar resolvewang avatar tanmx avatar wenhuwang avatar wilhelmguo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

follow-me-install-kubernetes-cluster's Issues

关于Heapster被拒绝访问的问题

您好,我在部署完成Heapster后,出现以下画面中的被拒绝访问的问题。
我的Heapster的文件,完全是您的yaml文件的copy来进行复制的,请问是不是有什么地方需要修改?

image

token.csv 是什么?根据你写的部署MASTER,然后启动发现一个Forbidden问题

etcd 访问正常:
etcdctl --endpoint=https://192.168.8.130:2379 --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem cluster-health
2017-04-20 14:20:48.202497 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-04-20 14:20:48.203776 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member 261a27fe54f9e457 is healthy: got healthy result from https://192.168.8.199:2379
member 84166fb6649565fe is healthy: got healthy result from https://192.168.8.124:2379
member b6872d54339a0bdb is healthy: got healthy result from https://192.168.8.130:2379
cluster is healthy

kubectl get nodes
Error from server (Forbidden): User "admin" cannot list nodes at the cluster scope. (get nodes)

无法成功部署dns测试

我完全按照您的说明文档进行部署,但是到测试dns的时候,出现问题。
root@nginx:/# ping my-nginx
ping: unknown host
root@nginx:/# ping kubernetes
^C
root@nginx:/# ping kube-dns.kube-system.svc.cluster.local
ping: unknown host

请问我如何排查这个问题?

加入ingress的配置

没有cloud provider的情况下,对外暴露服务总不能全部用nodeport吧。
用ingress非常有必要

创建Kubernetes证书时错误

按照第一篇文档创建 kubernetes 各组件 TLS 加密通信的证书和秘钥中介绍,执行下面的命令,json文件与文档中相同,只不过是更改为我自己的IP地址。

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

报错信息

Failed to load config file: {"code":5200,"message":"failed to unmarshal configuration: invalid character '\"' after array element"}Failed to parse input: unexpected end of JSON input

以第二种方式执行

echo '{"CN":"kubernetes","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes -hostname="127.0.0.1,172.20.0.112,172.20.0.113,172.20.0.114,172.20.0.115,254.0.1,kubernetes,kubernetes.default" - | cfssljson -bare kubernetes

依然报上述错误。

环境信息

$cfssl version
Version: 1.2.0
Revision: dev
Runtime: go1.7.5

我使用go get方式安装的。

kubelet 无法正常启动, 谢谢

May 12 16:40:37 node82 kubelet: error: failed to run Kubelet: cannot create certificate signing request: the server has asked for the client to provide credentials (post certificatesigningrequests.certificates.k8s.io)
May 12 16:40:37 node82 systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
May 12 16:40:37 node82 systemd: Unit kubelet.service entered failed state.
May 12 16:40:37 node82 systemd: kubelet.service failed.
May 12 16:40:42 node82 systemd: kubelet.service holdoff time over, scheduling restart.
May 12 16:40:42 node82 systemd: Started Kubernetes Kubelet.
May 12 16:40:42 node82 systemd: Starting Kubernetes Kubelet...
May 12 16:40:42 node82 kubelet: I0512 16:40:42.530619 39763 feature_gate.go:144] feature gates: map[]
May 12 16:40:42 node82 kubelet: I0512 16:40:42.530701 39763 bootstrap.go:58] Using bootstrap kubeconfig to generate TLS client cert, key and kubeconfig file
May 12 16:40:42 node82 kubelet: error: failed to run Kubelet: cannot create certificate signing request: the server has asked for the client to provide credentials (post certificatesigningrequests.certificates.k8s.io)
May 12 16:40:42 node82 systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
May 12 16:40:42 node82 systemd: Unit kubelet.service entered failed state.
May 12 16:40:42 node82 systemd: kubelet.service failed.

获取etcd集群的信息的时候显示错误

感谢你的文档,目前正在按部就班的搭建中,遇到一个问题请教下:

etcd cluster 看上去搭建成功了:

[root@k8snode3 ~]# for ip in ${NODE_IPS}; do   ETCDCTL_API=3 /root/local/bin/etcdctl   --endp         oints=https://${ip}:2379    --cacert=/etc/kubernetes/ssl/ca.pem   --cert=/etc/kubernetes/ssl/         kubernetes.pem   --key=/etc/kubernetes/ssl/kubernetes-key.pem   endpoint health; done
2017-04-18 13:20:27.778205 I | warning: ignoring ServerName for user-provided CA for backward         s compatibility is deprecated
https://10.192.29.201:2379 is healthy: successfully committed proposal: took = 2.050628ms
2017-04-18 13:20:27.823011 I | warning: ignoring ServerName for user-provided CA for backward         s compatibility is deprecated
https://10.192.29.202:2379 is healthy: successfully committed proposal: took = 2.067567ms
2017-04-18 13:20:27.872696 I | warning: ignoring ServerName for user-provided CA for backward         s compatibility is deprecated
https://10.192.29.203:2379 is healthy: successfully committed proposal: took = 1.815505ms

但是我通过下面的命令去获取etcd的信息的时候显示错误

[root@k8scluster ~]# /root/local/bin/etcdctl \
>   --endpoints=${ETCD_ENDPOINTS} \
>   --ca-file=/etc/kubernetes/ssl/ca.pem \
>   --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
>   --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
>   member list
2017-04-18 13:16:01.459285 I | warning: ignoring ServerName for user-provided CA for backward         s compatibility is deprecated
client: etcd cluster is unavailable or misconfigured; error #0: Forbidden
; error #1: Forbidden
; error #2: Forbidden

[root@k8scluster ~]# /root/local/bin/etcdctl \
>   --endpoints=${ETCD_ENDPOINTS} \
>   --ca-file=/etc/kubernetes/ssl/ca.pem \
>   --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
>   --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
>   get ${FLANNEL_ETCD_PREFIX}/config
2017-04-18 13:18:03.020421 I | warning: ignoring ServerName for user-provided CA for backward         s compatibility is deprecated
Error:  client: etcd cluster is unavailable or misconfigured; error #0: Forbidden
; error #1: Forbidden
; error #2: Forbidden

error #0: Forbidden
error #1: Forbidden
error #2: Forbidden

[root@k8scluster ~]#

非常感谢!

heapster没有问题,但dashboard没有数据

dashboard日志:
Skipping Heapster metrics because of error: an error on the server ("Error: 'dial tcp 192.168.90.3:8082: getsockopt: connection timed out'\nTrying to reach: 'http://192.168.90.3:8082/api/v1/model/namespaces/kube-system/pod-list/heapster-3454743269-gmkcj,kube-dns-3574069718-9x3gw,kubernetes-dashboard-3239310776-kb4km,monitoring-influxdb-79946106-xkqpq/metrics/cpu/usage_rate'") has prevented the request from succeeding (get services heapster)

在相同主机的另一个容器是可以访问这个url的

配置和启动 kube-scheduler

$ cat > kube-scheduler.service <<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/root/local/bin/kube-scheduler \
--address=127.0.0.1 \
--master=http://{MASTER_IP}:8080 \
--leader-elect=true \
--v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

缺少$符号

some bug tips

01, 07-部署Node节点.md

ls -l /etc/kubernetes/kubelet.kubeconfig kubenet请求证书, 应该是master审批. client生成证书的目录是
ls -l /etc/kubernetes/ssl/ 是不是写错了?

02, 04-部署Kubectl命令行工具.md

admin.pem 证书 OU 字段值为 system:masters ... 这里OU应该是是O吧.

Node的日志一直报错

Failed to check if disk space is available on the root partition: failed to get fs info for "root": error trying to get filesystem Device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 36 in cached partitions map

Error creating: pods "kube-dns-699984412-" is forbidden: service account kube-system/kube-dns was not found, retry after the service account is created

你好 ZhangJun
那个报错都是通过去掉ServerAccount解决的。但是根据你的配置上面说“--admission-control 值必须包含 ServiceAccount;”
我找了好长时间还是没有找到解决方法,能帮帮我么?

[root@zxr dns]# kubectl get clusterrolebindings system:kube-dns -o yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: 2017-04-18T02:34:39Z
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-dns
  resourceVersion: "56"
  selfLink: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindingssystem%3Akube-dns
  uid: 9210911e-23df-11e7-a89f-408d5c380437
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-dns
subjects:
- kind: ServiceAccount
  name: kube-dns
  namespace: kube-system

请问还需要什么错误信息?我需要补充。

关于部署高可用 kubernetes master 集群的问题

环境

  • CentOS 7.2.1511
  • Docker 1.12.5
  • Kubernetes 1.6.0

问题描述

我按照部署高可用 kubernetes master 集群文档部署的master节点,不过改了一些配置文件,但是步骤是一样的,最后有如下几个现象,我认为可能是TLS还有问题。我看不出来究竟是哪里的问题,这是我的整个操作步骤

现象一

kubectl get componentstatuses时有有如下报错信息:

    etcd-2               Unhealthy   Get http://172.20.0.113:2379/health: malformed HTTP response "\x15\x03\x01\x00\x02\x02" 

与TLS认证有关。

现象二

kube-apiserver启动时有报错信息;

    Apr 11 18:06:25 sz-pg-oam-docker-test-001.tendcloud.com kube-apiserver[25718]: E0411 18:06:25.522715   25718 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *api.ResourceQuota: the server cannot complete the requested operation at this time, try again later (get resourcequotas)
    Apr 11 18:06:25 sz-pg-oam-docker-test-001.tendcloud.com kube-apiserver[25718]: E0411 18:06:25.951292   25718 storage_rbac.go:140] unable to initialize clusterroles: the server cannot complete the requested operation at this time, try again later (get clusterroles.rbac.authorization.k8s.io)

这个错误是否影响kubernentes运行?

现象三

kubectl get命令无法正常使用。

$kubectl --kubeconfig ~/.kube/config get all
Unable to connect to the server: unexpected EOF
The connection to the server 172.20.0.113:6443 was refused - did you specify the right host or port?
The connection to the server 172.20.0.113:6443 was refused - did you specify the right host or port?

查看config。

    apiVersion: v1
    clusters:
    - cluster:
        server: http://sz-pg-oam-docker-test-001:8080
      name: default-cluster
    - cluster:
        certificate-authority-data: REDACTED
        server: https://172.20.0.113:6443
      name: kubernetes
    contexts:
    - context:
        cluster: default-cluster
        user: default-admin
      name: default-context
    - context:
        cluster: kubernetes
        user: admin
      name: kubernetes
    current-context: kubernetes
    kind: Config
    preferences: {}
    users:
    - name: admin
      user:
        client-certificate-data: REDACTED
        client-key-data: REDACTED

而~/.kube/config文件也存在,6443端口也在,为什么无法访问?

部署 kubectl 命令行工具后执行kubectl get pods问题

Hi:

感谢您提供非常有价值的学习k8s的问题,但是我按照文档部署 kubectl 命令行工具后,执行kubectl get pods命令,依然会提示下面错误:

The connection to the server 172.26.10.85:6443 was refused - did you specify the right host or port?
说明:172.26.10.85:6443是KUBE_APISERVER地址

同时,使用netstat -lntp查看,没有6443端口启动
[root@k8smaster .kube]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 172.26.10.85:2379 0.0.0.0:* LISTEN 1202/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 1202/etcd
tcp 0 0 172.26.10.85:2380 0.0.0.0:* LISTEN 1202/etcd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1200/sshd
tcp6 0 0 :::22 :::* LISTEN 1200/sshd

kubectl配置如下
[root@k8smaster .kube]# kubectl config view
apiVersion: v1
clusters:

  • cluster:
    certificate-authority-data: REDACTED
    server: https://172.26.10.85:6443
    name: kubernetes
    contexts:
  • context:
    cluster: kubernetes
    user: admin
    name: kubernetes
    current-context: kubernetes
    kind: Config
    preferences: {}
    users:
  • name: admin
    user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED
    麻烦帮看下问题出现在哪里,万分感谢!

配置和启动 kube-controller-manager引入参数缺少$符号

$ cat > kube-controller-manager.service <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/root/local/binkube-controller-manager \
--address=127.0.0.1 \
--master=http://{MASTER_IP}:8080 \
--allocate-node-cidrs=true \
--service-cluster-ip-range=${SERVICE_CIDR} \
--cluster-cidr=${CLUSTER_CIDR} \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
--root-ca-file=/etc/kubernetes/ssl/ca.pem \
--leader-elect=true \
--v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

加粗地方引入参数时候,缺少$符号

碰到的几个问题

1、容器网络不通
原因:node 节点 iptables forward链设置不对
解决:iptables -P FORWARD ACCEPT
2、监控、elk 不能在dashboard展现
原因:master没装flannel,没法与pod通信
解决:master安装flannel

systemctl start kube-controller-manager 因为超时启动失败

环境

  • Kubernetes 版本:1.6.2
  • Etcd 版本:3.1.3
  • OS版本:CentOS-7.3.1611
  • Kernel 版本:4.10.10

现象
kube-apiserver成功启动:

ps -ef |grep kube |grep -v grep               
root       432     1  7 15:10 ?        00:00:03 /usr/bin/kube-apiserver --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota --advertise-address=192.168.140.60 --allow-privileged=true --apiserver-count=1 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/log/kubernetes/kube-apiserver.log --authorization-mode=RBAC --bind-address=192.168.140.60 --client-ca-file=/etc/kubernetes/ssl/ca.pem --enable-swagger-ui=true --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --etcd-prefix=/registry --etcd-servers=https://192.168.140.61:2379,https://192.168.140.62:2379,https://192.168.140.63:2379 --event-ttl=1h --experimental-bootstrap-token-auth --insecure-bind-address=192.168.140.60 --insecure-port=8080 --kubelet-https=true --runtime-config=rbac.authorization.k8s.io/v1alpha1 --secure-port=6443 --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --service-cluster-ip-range=10.20.0.0/16 --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --token-auth-file=/etc/kubernetes/token.csv --v=2

启动kube-controller-manager时失败:

systemctl start kube-controller-manager.service   
Job for kube-controller-manager.service failed because a timeout was exceeded. See "systemctl status kube-controller-manager.service" and "journalctl -xe" for details.
ps -ef |grep kube |grep -v grep                
root       432     1  2 15:10 ?        00:00:10 /usr/bin/kube-apiserver --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota --advertise-address=192.168.140.60 --allow-privileged=true --apiserver-count=1 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/log/kubernetes/kube-apiserver.log --authorization-mode=RBAC --bind-address=192.168.140.60 --client-ca-file=/etc/kubernetes/ssl/ca.pem --enable-swagger-ui=true --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --etcd-prefix=/registry --etcd-servers=https://192.168.140.61:2379,https://192.168.140.62:2379,https://192.168.140.63:2379 --event-ttl=1h --experimental-bootstrap-token-auth --insecure-bind-address=192.168.140.60 --insecure-port=8080 --kubelet-https=true --runtime-config=rbac.authorization.k8s.io/v1alpha1 --secure-port=6443 --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --service-cluster-ip-range=10.20.0.0/16 --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --token-auth-file=/etc/kubernetes/token.csv --v=2
root      3897     1  3 15:18 ?        00:00:01 /usr/bin/kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --cluster-cidr=10.10.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem --leader-elect=true --log-dir=/var/log/kubernetes --master=http://192.168.140.60:8080 --root-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem --service-cluster-ip-range=10.20.0.0/16 --v=2

停掉 kube-controller-manager,手动启动,会耗时特别长:

kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --cluster-cidr=10.10.0.0/16 --cluster-name
=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem --leader-el
ect=true --log-dir=/var/log/kubernetes --master=http://192.168.140.60:8080 --root-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-p
rivate-key-file=/etc/kubernetes/ssl/ca-key.pem --service-cluster-ip-range=10.20.0.0/16 --v=2
I0424 15:20:20.731227    4191 leaderelection.go:179] attempting to acquire leader lease...
I0424 15:20:20.749637    4191 leaderelection.go:189] successfully acquired lease kube-system/kube-controller-manager
I0424 15:20:20.749757    4191 event.go:217] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"7b43b5b8-28ba-11e7-83be-525400f684df", APIVersion:"v1", ResourceVersion:"1270", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' k8s-ssl-master became leader
I0424 15:20:20.776624    4191 controllermanager.go:427] Starting "horizontalpodautoscaling"
I0424 15:20:20.777744    4191 controllermanager.go:437] Started "horizontalpodautoscaling"
I0424 15:20:20.777777    4191 controllermanager.go:427] Starting "statefuleset"
I0424 15:20:20.777907    4191 horizontal.go:139] Starting HPA Controller
I0424 15:20:20.778505    4191 controllermanager.go:437] Started "statefuleset"
I0424 15:20:20.778535    4191 controllermanager.go:427] Starting "ttl"
I0424 15:20:20.778697    4191 stateful_set.go:144] Starting statefulset controller
I0424 15:20:20.779071    4191 controllermanager.go:437] Started "ttl"
I0424 15:20:20.779099    4191 controllermanager.go:427] Starting "podgc"
I0424 15:20:20.779221    4191 ttlcontroller.go:117] Starting TTL controller
I0424 15:20:20.779490    4191 controllermanager.go:437] Started "podgc"
I0424 15:20:20.779516    4191 controllermanager.go:427] Starting "job"
I0424 15:20:20.780271    4191 controllermanager.go:437] Started "job"
I0424 15:20:20.780300    4191 controllermanager.go:427] Starting "deployment"
I0424 15:20:20.781003    4191 controllermanager.go:437] Started "deployment"
I0424 15:20:20.781034    4191 controllermanager.go:427] Starting "replicationcontroller"
I0424 15:20:20.781147    4191 deployment_controller.go:151] Starting deployment controller
I0424 15:20:20.781729    4191 controllermanager.go:437] Started "replicationcontroller"
I0424 15:20:20.781757    4191 controllermanager.go:427] Starting "daemonset"
I0424 15:20:20.781891    4191 replication_controller.go:150] Starting RC Manager
I0424 15:20:20.782442    4191 controllermanager.go:437] Started "daemonset"
I0424 15:20:20.782473    4191 controllermanager.go:427] Starting "cronjob"
W0424 15:20:20.782493    4191 controllermanager.go:434] Skipping "cronjob"
I0424 15:20:20.782503    4191 controllermanager.go:427] Starting "certificatesigningrequests"
I0424 15:20:20.782608    4191 daemoncontroller.go:199] Starting Daemon Sets controller manager
I0424 15:20:20.784544    4191 controllermanager.go:437] Started "certificatesigningrequests"
W0424 15:20:20.784585    4191 controllermanager.go:421] "tokencleaner" is disabled
I0424 15:20:20.784598    4191 controllermanager.go:427] Starting "serviceaccount"
I0424 15:20:20.784744    4191 certificate_controller.go:120] Starting certificate controller manager
E0424 15:20:20.785214    4191 util.go:45] Metric for serviceaccount_controller already registered
I0424 15:20:20.785271    4191 controllermanager.go:437] Started "serviceaccount"
I0424 15:20:20.785292    4191 controllermanager.go:427] Starting "replicaset"
I0424 15:20:20.785399    4191 serviceaccounts_controller.go:122] Starting ServiceAccount controller
I0424 15:20:20.785934    4191 controllermanager.go:437] Started "replicaset"
I0424 15:20:20.785966    4191 controllermanager.go:427] Starting "namespace"
I0424 15:20:20.786052    4191 replica_set.go:155] Starting ReplicaSet controller
I0424 15:20:21.087924    4191 controllermanager.go:437] Started "namespace"
I0424 15:20:21.087961    4191 controllermanager.go:427] Starting "garbagecollector"
I0424 15:20:21.089551    4191 namespace_controller.go:189] Starting the NamespaceController
I0424 15:20:21.115655    4191 controllermanager.go:437] Started "garbagecollector"
I0424 15:20:21.115686    4191 controllermanager.go:427] Starting "disruption"
I0424 15:20:21.117268    4191 controllermanager.go:437] Started "disruption"
W0424 15:20:21.117298    4191 controllermanager.go:421] "bootstrapsigner" is disabled
I0424 15:20:21.117311    4191 controllermanager.go:427] Starting "endpoint"
I0424 15:20:21.117902    4191 controllermanager.go:437] Started "endpoint"
I0424 15:20:21.117920    4191 controllermanager.go:427] Starting "resourcequota"
I0424 15:20:21.118729    4191 controllermanager.go:437] Started "resourcequota"
I0424 15:20:21.118749    4191 plugins.go:101] No cloud provider specified.
I0424 15:20:21.119173    4191 nodecontroller.go:219] Sending events to api server.
I0424 15:20:21.119281    4191 garbagecollector.go:111] Garbage Collector: Initializing
I0424 15:20:21.119439    4191 disruption.go:269] Starting disruption controller
I0424 15:20:21.119571    4191 resource_quota_controller.go:240] Starting resource quota controller
I0424 15:20:21.219646    4191 garbagecollector.go:116] Garbage Collector: All resource monitors have synced. Proceeding to collect garbage
I0424 15:20:31.124992    4191 cidr_allocator.go:84] Sending events to api server.
I0424 15:20:31.125227    4191 taint_controller.go:157] Sending events to api server.
E0424 15:20:31.126001    4191 controllermanager.go:494] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail.
W0424 15:20:31.126058    4191 controllermanager.go:510] configure-cloud-routes is set, but no cloud provider specified. Will not configure cloud provider routes.
I0424 15:20:31.126898    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/host-path"
I0424 15:20:31.126955    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/nfs"
I0424 15:20:31.126979    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/glusterfs"
I0424 15:20:31.127003    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/rbd"
I0424 15:20:31.127029    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/quobyte"
I0424 15:20:31.127047    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/flocker"
I0424 15:20:31.127069    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/portworx-volume"
I0424 15:20:31.127089    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/scaleio"
I0424 15:20:31.127323    4191 pv_controller_base.go:277] starting PersistentVolumeController
I0424 15:20:31.127722    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/aws-ebs"
I0424 15:20:31.127757    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/gce-pd"
I0424 15:20:31.127801    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/cinder"
I0424 15:20:31.127828    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/portworx-volume"
I0424 15:20:31.127847    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/vsphere-volume"
I0424 15:20:31.127871    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/azure-disk"
I0424 15:20:31.127889    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/photon-pd"
I0424 15:20:31.127906    4191 plugins.go:363] Loaded volume plugin "kubernetes.io/scaleio"
I0424 15:20:31.128040    4191 attach_detach_controller.go:223] Starting Attach Detach Controller
I0424 15:20:31.219584    4191 disruption.go:277] Sending events to api server.
I0424 15:20:31.229165    4191 taint_controller.go:180] Starting NoExecuteTaintManager
W0424 15:26:45.195156    4191 reflector.go:323] k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:192: watch of <nil> ended with: etcdserver: mvcc: required revision has been compacted

看日志,好像卡在Starting NoExecuteTaintManager这一步,将 --enable-taint-manager设为false,还是一样。

配置项
etcd集群的配置项:

# [member]
ETCD_NAME="etcd-0"
ETCD_DATA_DIR="/var/lib/etcd/etcd-0"
#ETCD_WAL_DIR=""
#ETCD_SNAPSHOT_COUNT="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.140.61:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.140.61:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.140.61:2380"
# if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-0=https://192.168.140.61:2380,etcd-1=https://192.168.140.62:2380,etcd-2=https://192.168.140.63:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.140.61:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_SRV=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_STRICT_RECONFIG_CHECK="false"
ETCD_AUTO_COMPACTION_RETENTION="1"
#
#[proxy]
#ETCD_PROXY="off"
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
#
#[security]
ETCD_CERT_FILE="/etc/etcd/ssl/kubernetes.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/kubernetes-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.pem"
#ETCD_AUTO_TLS="false"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/kubernetes.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/kubernetes-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.pem"
#ETCD_PEER_AUTO_TLS="false"
#
#[logging]
ETCD_DEBUG="true"
# examples for -log-package-levels etcdserver=WARNING,security=DEBUG
ETCD_LOG_PACKAGE_LEVELS="DEBUG"

kube-apiserver.service 的配置项:

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/bin/kube-apiserver \
            --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
            --advertise-address=192.168.140.60 \
            --allow-privileged=true \
            --apiserver-count=1 \
            --audit-log-maxage=30 \
            --audit-log-maxbackup=3 \
            --audit-log-maxsize=100 \
            --audit-log-path=/var/log/kubernetes/kube-apiserver.log \
            --authorization-mode=RBAC \
            --bind-address=192.168.140.60 \
            --client-ca-file=/etc/kubernetes/ssl/ca.pem \
            --enable-swagger-ui=true \
            --etcd-cafile=/etc/kubernetes/ssl/ca.pem \
            --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem \
            --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem \
            --etcd-prefix=/registry \
            --etcd-servers=https://192.168.140.61:2379,https://192.168.140.62:2379,https://192.168.140.63:2379 \
            --event-ttl=1h \
            --experimental-bootstrap-token-auth \
            --insecure-bind-address=192.168.140.60 \
            --insecure-port=8080 \
            --kubelet-https=true \
            --runtime-config=rbac.authorization.k8s.io/v1alpha1 \
            --secure-port=6443 \
            --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
            --service-cluster-ip-range=10.20.0.0/16 \
            --service-node-port-range=30000-32767 \
            --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
            --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
            --token-auth-file=/etc/kubernetes/token.csv \
            --v=2
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

kube-controller-manager.service 的配置项:

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/bin/kube-controller-manager \
            --address=127.0.0.1 \
            --allocate-node-cidrs=true \
            --cluster-cidr=10.10.0.0/16 \
            --cluster-name=kubernetes \
            --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
            --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
            --leader-elect=true \
            --log-dir=/var/log/kubernetes \
            --master=http://192.168.140.60:8080 \
            --root-ca-file=/etc/kubernetes/ssl/ca.pem \
            --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
            --service-cluster-ip-range=10.20.0.0/16 \
            --v=2
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

我想通过nginx的upstream代理apiserver,如果通过6443方式不知道如何实现

目前我是在nginx中配置一个upstream

upstream k8s {
   server 192.168.6.212:6443 max_fails=3 fail_timeout=30s;
   server 192.168.6.222:6443 max_fails=3 fail_timeout=30s;

}
upstream k8s-8080 {
   server 192.168.6.212:8080 max_fails=3 fail_timeout=30s;
   server 192.168.6.222:8080 max_fails=3 fail_timeout=30s;
}
server {
	listen       6443;
	server_name  192.168.6.108;

	ssl on;
	ssl_certificate         /app/nginx/sslkey/kubernetes.pem;
	ssl_certificate_key     /app/nginx/sslkey/kubernetes-key.pem;

	ssl_protocols TLSv1 TLSv1.1 TLSv1.2; 
	ssl_ciphers ECDH:AESGCM:HIGH:!RC4:!DH:!MD5:!aNULL:!eNULL; 
	ssl_prefer_server_ciphers   on;  

location / {
	proxy_pass https://k8s;
}


}
server {
	listen       8080;
	server_name  192.168.6.108;

location / {
	proxy_pass http://k8s-8080;
}
}

但是在master节点通过kubectl get componentstatuses命令查看的时候提示

[root@master2 bin]# kubectl get componentstatuses
Error from server (Forbidden): User "system:anonymous" cannot list componentstatuses at the cluster scope. (get componentstatuses)
``
貌似还是认证相关的问题,想请教一下,该如何进行下去,谢谢

部署高可用 etcd 集群环境变量设置使用

创建etcd.service时,引用{NODE_NAME} 环境变量时候,缺少了$符号
$ sudo mkdir -p /var/lib/etcd # 必须先创建工作目录
$ cat > etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/root/local/bin/etcd \
--name={NODE_NAME} \
--cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
--peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
--initial-advertise-peer-urls=https://${NODE_IP}:2380 \
--listen-peer-urls=https://${NODE_IP}:2380 \
--listen-client-urls=https://${NODE_IP}:2379,http://127.0.0.1:2379 \
--advertise-client-urls=https://${NODE_IP}:2379 \
--initial-cluster-token=etcd-cluster-0 \
--initial-cluster=${ETCD_NODES} \
--initial-cluster-state=new \
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

【service】node无法访问

一台node上的service curl http://172.30.16.2:9090 可以访问

但是其他的node 执行报错

curl: (7) Failed connect to 172.30.16.2:9090; Connection refused

每台机器都可以ping的通。。

ping 172.30.16.2
PING 172.30.16.2 (172.30.16.2) 56(84) bytes of data.
64 bytes from 172.30.16.2: icmp_seq=1 ttl=64 time=0.204 ms

求助大神。。

目前有两台Master想实现down掉一台master集群仍然可以正常访问,不知道如何实现

我看到 https://github.com/opsnull/follow-me-install-kubernetes-cluster/blob/master/02-kubeconfig%E6%96%87%E4%BB%B6.md 中配置kubelet bootstrapping kubeconfig和kube-proxy kubeconfig文件的时候只用了一个master的IP地址,然后将生成的文件分发,我尝试down掉第一台master,在第二台上执行kubectl get componentstatuses,出现如下结果

[root@master2 .kube]# kubectl get componentstatuses
NAME                 STATUS      MESSAGE                                                                                            ERROR
controller-manager   Healthy     ok                                                                                                 
scheduler            Healthy     ok                                                                                                 
etcd-1               Unhealthy   HTTP probe failed with statuscode: 503                                                             
etcd-0               Unhealthy   Get https://192.168.6.212:2379/health: dial tcp 192.168.6.212:2379: getsockopt: no route to host   

求助:kubectl 证书不对

执行:kubectl get node 报如下错误
error: You must be logged in to the server (the server has asked for the client to provide credentials)

kube-apiserver 的日志报 “Unable to authenticate the request due to an error: x509: certificate specifies an incompatible key usage”

如下是admin-csr.json 内容,请帮看看有可能是哪里问题
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}

flannel 网络无法互通,dashboard无法访问

麻烦请教下,按照配置顺序配置下来,遇到两个问题

  1. 配置好flannel,部署了nginx,node之间无法ping通
    [root@server104 ~]# kubectl get pods -o wide
    NAME READY STATUS RESTARTS AGE IP NODE
    nginx-ds-fblq2 1/1 Running 2 5h 172.30.48.2 172.23.27.102
    nginx-ds-hq2jk 1/1 Running 2 5h 172.30.2.2 172.23.27.104
    nginx-ds-x4lrf 1/1 Running 1 5h 172.30.101.2 172.23.27.103
    [root@server104 ~]# ping 172.30.48.2
    PING 172.30.48.2 (172.30.48.2) 56(84) bytes of data.
    ^C
    --- 172.30.48.2 ping statistics ---
    3 packets transmitted, 0 received, 100% packet loss, time 1999ms
    [root@server104 ~]# ping 172.30.48.1
    PING 172.30.48.1 (172.30.48.1) 56(84) bytes of data.
    ^C
    --- 172.30.48.1 ping statistics ---
    6 packets transmitted, 0 received, 100% packet loss, time 4999ms
    [root@server104 ~]# ping 172.30.48.0
    PING 172.30.48.0 (172.30.48.0) 56(84) bytes of data.
    64 bytes from 172.30.48.0: icmp_seq=1 ttl=64 time=0.826 ms
    64 bytes from 172.30.48.0: icmp_seq=2 ttl=64 time=0.213 ms
    ^C
    --- 172.30.48.0 ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 999ms
    rtt min/avg/max/mdev = 0.213/0.519/0.826/0.307 ms

  2. dashboard 安装好之后,在对应node上面可以ping通dashboard的pod,但是web上面报错
    kubectl proxy --address='0.0.0.0' --port=8086 --accept-hosts='^*$'

http://172.23.27.104:8086/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/

Error: 'dial tcp 172.30.2.3:9090: getsockopt: connection timed out'
Trying to reach: 'http://172.30.2.3:9090/'

[root@server104 ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
default nginx-ds-fblq2 1/1 Running 2 5h 172.30.48.2 172.23.27.102
default nginx-ds-hq2jk 1/1 Running 2 5h 172.30.2.2 172.23.27.104
default nginx-ds-x4lrf 1/1 Running 1 5h 172.30.101.2 172.23.27.103
kube-system kubernetes-dashboard-1164048533-bdj3x 1/1 Running 15 4h 172.30.2.3 172.23.27.104
[root@server104 ~]# ping 172.30.2.3
PING 172.30.2.3 (172.30.2.3) 56(84) bytes of data.
64 bytes from 172.30.2.3: icmp_seq=1 ttl=64 time=0.058 ms
^C
--- 172.30.2.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.058/0.058/0.058/0.000 ms

求助浏览器无法访问dashboard

重新配置了下node还是无法访问,本地可以curl nodeip:nodeport,提示:

<!doctype html> <html ng-app="kubernetesDashboard"> <head> <meta charset="utf-8"> <title ng-controller="kdTitle as $ctrl" ng-bind="$ctrl.title()"></title> <link rel="icon" type="image/png" href="assets/images/kubernetes-logo.png"> <meta name="viewport" content="width=device-width"> <link rel="stylesheet" href="static/vendor.4f4b705f.css"> <link rel="stylesheet" href="static/app.93b90a74.css"> </head> <body> <!--[if lt IE 10]>
      <p class="browsehappy">You are using an <strong>outdated</strong> browser.
      Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your
      experience.</p>
    <![endif]--> <kd-chrome layout="column" layout-fill> </kd-chrome> <script src="static/vendor.6952e31e.js"></script> <script src="api/appConfig.json"></script> <script src="static/app.8a6b8127.js"></script> </body> </html>

浏览器访问

http://10.25.36.13:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/
Error: 'dial tcp 192.168.0.3:9090: getsockopt: no route to host'
Trying to reach: 'http://192.168.0.3:9090/'
iptables -nL --line-number
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0           
2    ACCEPT     tcp  --  10.0.0.0/8           0.0.0.0/0            tcp dpt:4194
3    ACCEPT     tcp  --  172.16.0.0/12        0.0.0.0/0            tcp dpt:4194
4    ACCEPT     tcp  --  192.168.0.0/16       0.0.0.0/0            tcp dpt:4194
5    DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:4194
6    ACCEPT     tcp  --  10.0.0.0/8           0.0.0.0/0            tcp dpt:4194
7    ACCEPT     tcp  --  172.16.0.0/12        0.0.0.0/0            tcp dpt:4194
8    ACCEPT     tcp  --  192.168.0.0/16       0.0.0.0/0            tcp dpt:4194
9    DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:4194

Chain FORWARD (policy DROP)
num  target     prot opt source               destination         
1    DOCKER-ISOLATION  all  --  0.0.0.0/0            0.0.0.0/0           
2    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
3    DOCKER     all  --  0.0.0.0/0            0.0.0.0/0           
4    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
5    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
2    KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0           

Chain DOCKER (1 references)
num  target     prot opt source               destination         

Chain DOCKER-ISOLATION (1 references)
num  target     prot opt source               destination         
1    RETURN     all  --  0.0.0.0/0            0.0.0.0/0           

Chain KUBE-FIREWALL (2 references)
num  target     prot opt source               destination         
1    DROP       all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-SERVICES (1 references)
num  target     prot opt source               destination
iptables -t nat -nL --line-number
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
2    DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
2    DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
2    MASQUERADE  all  --  192.168.0.0/20       0.0.0.0/0           

Chain DOCKER (2 references)
num  target     prot opt source               destination         
1    RETURN     all  --  0.0.0.0/0            0.0.0.0/0           

Chain KUBE-MARK-DROP (0 references)
num  target     prot opt source               destination         
1    MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x8000

Chain KUBE-MARK-MASQ (15 references)
num  target     prot opt source               destination         
1    MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
num  target     prot opt source               destination         
1    KUBE-MARK-MASQ  tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx-ds:http */ tcp dpt:8815
2    KUBE-SVC-MMVBGRO3JCFBHTP2  tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx-ds:http */ tcp dpt:8815
3    KUBE-MARK-MASQ  tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kubernetes-dashboard: */ tcp dpt:8959
4    KUBE-SVC-XGLOHA7QRQ3V22RZ  tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kubernetes-dashboard: */ tcp dpt:8959

Chain KUBE-POSTROUTING (1 references)
num  target     prot opt source               destination         
1    MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

Chain KUBE-SEP-22YW3O62YRV7OSGJ (1 references)
num  target     prot opt source               destination         
1    KUBE-MARK-MASQ  all  --  192.168.0.3          0.0.0.0/0            /* default/my-nginx: */
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/my-nginx: */ tcp to:192.168.0.3:80

Chain KUBE-SEP-3BXPWXBSOCOQ4XS4 (2 references)
num  target     prot opt source               destination         
1    KUBE-MARK-MASQ  all  --  10.25.36.13          0.0.0.0/0            /* default/kubernetes:https */
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ recent: SET name: KUBE-SEP-3BXPWXBSOCOQ4XS4 side: source mask: 255.255.255.255 tcp to:10.25.36.13:6443

Chain KUBE-SEP-AGF24HBSCHVRB2IM (1 references)
num  target     prot opt source               destination         
1    KUBE-MARK-MASQ  all  --  192.168.0.4          0.0.0.0/0            /* kube-system/kube-dns:dns */
2    DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:192.168.0.4:53

Chain KUBE-SEP-CF3CVUYBRRNMUR2I (1 references)
num  target     prot opt source               destination         
1    KUBE-MARK-MASQ  all  --  192.168.0.4          0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.0.4:53

Chain KUBE-SEP-D4DLK65XSSQNJBRU (1 references)
num  target     prot opt source               destination         
1    KUBE-MARK-MASQ  all  --  192.168.0.5          0.0.0.0/0            /* default/my-nginx: */
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/my-nginx: */ tcp to:192.168.0.5:80

Chain KUBE-SEP-FOETSOIESJMTAHBR (1 references)
num  target     prot opt source               destination         
1    KUBE-MARK-MASQ  all  --  192.168.0.6          0.0.0.0/0            /* kube-system/kubernetes-dashboard: */
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kubernetes-dashboard: */ tcp to:192.168.0.6:9090

Chain KUBE-SEP-HBCU25I33OSGI43V (1 references)
num  target     prot opt source               destination         
1    KUBE-MARK-MASQ  all  --  192.168.0.2          0.0.0.0/0            /* default/nginx-ds:http */
2    DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx-ds:http */ tcp to:192.168.0.2:80

Chain KUBE-SERVICES (2 references)
num  target     prot opt source               destination         
1    KUBE-MARK-MASQ  tcp  -- !10.254.0.0/16        10.254.82.249        /* default/nginx-ds:http cluster IP */ tcp dpt:80
2    KUBE-SVC-MMVBGRO3JCFBHTP2  tcp  --  0.0.0.0/0            10.254.82.249        /* default/nginx-ds:http cluster IP */ tcp dpt:80
3    KUBE-MARK-MASQ  udp  -- !10.254.0.0/16        10.254.0.2           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
4    KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  0.0.0.0/0            10.254.0.2           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
5    KUBE-MARK-MASQ  tcp  -- !10.254.0.0/16        10.254.0.2           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
6    KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  0.0.0.0/0            10.254.0.2           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
7    KUBE-MARK-MASQ  tcp  -- !10.254.0.0/16        10.254.136.12        /* default/my-nginx: cluster IP */ tcp dpt:80
8    KUBE-SVC-BEPXDJBUHFCSYIC3  tcp  --  0.0.0.0/0            10.254.136.12        /* default/my-nginx: cluster IP */ tcp dpt:80
9    KUBE-MARK-MASQ  tcp  -- !10.254.0.0/16        10.254.109.117       /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:80
10   KUBE-SVC-XGLOHA7QRQ3V22RZ  tcp  --  0.0.0.0/0            10.254.109.117       /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:80
11   KUBE-MARK-MASQ  tcp  -- !10.254.0.0/16        10.254.0.1           /* default/kubernetes:https cluster IP */ tcp dpt:443
12   KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  0.0.0.0/0            10.254.0.1           /* default/kubernetes:https cluster IP */ tcp dpt:443
13   KUBE-NODEPORTS  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-BEPXDJBUHFCSYIC3 (1 references)
num  target     prot opt source               destination         
1    KUBE-SEP-22YW3O62YRV7OSGJ  all  --  0.0.0.0/0            0.0.0.0/0            /* default/my-nginx: */ statistic mode random probability 0.50000000000
2    KUBE-SEP-D4DLK65XSSQNJBRU  all  --  0.0.0.0/0            0.0.0.0/0            /* default/my-nginx: */

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
num  target     prot opt source               destination         
1    KUBE-SEP-CF3CVUYBRRNMUR2I  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */

Chain KUBE-SVC-MMVBGRO3JCFBHTP2 (2 references)
num  target     prot opt source               destination         
1    KUBE-SEP-HBCU25I33OSGI43V  all  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx-ds:http */

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
num  target     prot opt source               destination         
1    KUBE-SEP-3BXPWXBSOCOQ4XS4  all  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-3BXPWXBSOCOQ4XS4 side: source mask: 255.255.255.255
2    KUBE-SEP-3BXPWXBSOCOQ4XS4  all  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
num  target     prot opt source               destination         
1    KUBE-SEP-AGF24HBSCHVRB2IM  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */

Chain KUBE-SVC-XGLOHA7QRQ3V22RZ (2 references)
num  target     prot opt source               destination         
1    KUBE-SEP-FOETSOIESJMTAHBR  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kubernetes-dashboard: */

求助kube-dns部署失败

Sync "kube-system/kube-dns-699984412" failed with unable to create pods: pods "kube-dns-699984412-" is forbidden: service account kube-system/kube-dns was not found, retry after the service account is created

请问master节点ServiceAccount需要特别配置吗?

node节点无法认证

配置了tls后,node节点的kubelet报错:
k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: the server has asked for the client to provide credentials (get services)

求解下,大神

通过 kube-apiserver 访问dashboard,提示User "system:anonymous" cannot proxy services in the namespace "kube-system".

[root@master1 ~]# kubectl cluster-info
Kubernetes master is running at https://192.168.6.212:6443
KubeDNS is running at https://192.168.6.212:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://192.168.6.212:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

使用浏览器访问https://192.168.6.212:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard,在浏览器钟提示

User "system:anonymous" cannot proxy services in the namespace "kube-system".

通过访问http://NodeIP:nodePort是能够访问的,通过kubectl proxy也可以访问,请问是不是还需要其他的配置什么的

Kubelet启动时失败

我的测试环境为
master安装在node1上,其中node1,node2和node3运行作为slave节点运行docker和kubelet。我机器上面都没有安装iptables并且关闭了firewalld。其中kube-proxy可以正常运行。

我严格按照您的方法进行创建,但是在运行kubelet的时候,总是提示以下错误信息

4月 16 13:39:16 v157-7-222-131.myvps.jp systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE 4月 16 13:39:16 v157-7-222-131.myvps.jp systemd[1]: Unit kubelet.service entered failed state. 4月 16 13:39:16 v157-7-222-131.myvps.jp systemd[1]: kubelet.service failed. 4月 16 13:39:16 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:16.230664 30983 wrap.go:75] GET /api/v1/namespaces/kube-system/endpoints/kube-controller-manager: (1.61163ms) 200 [[kube-controller-ma 4月 16 13:39:16 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:16.246697 30983 wrap.go:75] PUT /api/v1/namespaces/kube-system/endpoints/kube-controller-manager: (15.422977ms) 200 [[kube-controller- 4月 16 13:39:16 v157-7-222-131.myvps.jp kube-apiserver[30983]: E0416 13:39:16.276212 30983 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token 4月 16 13:39:16 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:16.276290 30983 wrap.go:75] POST /apis/certificates.k8s.io/v1beta1/certificatesigningrequests: (119.074µs) 401 [[kubelet/v1.6.1 (linux 4月 16 13:39:16 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:16.790427 30983 wrap.go:75] GET /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (1.4548ms) 200 [[kube-scheduler/v1.6.1 (linux 4月 16 13:39:16 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:16.795190 30983 wrap.go:75] PUT /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (4.145456ms) 200 [[kube-scheduler/v1.6.1 (lin 4月 16 13:39:17 v157-7-222-131.myvps.jp kube-apiserver[30983]: E0416 13:39:17.991728 30983 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token 4月 16 13:39:17 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:17.991783 30983 wrap.go:75] POST /apis/certificates.k8s.io/v1beta1/certificatesigningrequests: (101.855µs) 401 [[kubelet/v1.6.1 (linux 4月 16 13:39:18 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:18.249690 30983 wrap.go:75] GET /api/v1/namespaces/kube-system/endpoints/kube-controller-manager: (1.349974ms) 200 [[kube-controller-m 4月 16 13:39:18 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:18.255408 30983 wrap.go:75] PUT /api/v1/namespaces/kube-system/endpoints/kube-controller-manager: (4.660448ms) 200 [[kube-controller-m 4月 16 13:39:18 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:18.797634 30983 wrap.go:75] GET /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (1.497561ms) 200 [[kube-scheduler/v1.6.1 (lin 4月 16 13:39:18 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:18.803194 30983 wrap.go:75] PUT /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (5.049134ms) 200 [[kube-scheduler/v1.6.1 (lin 4月 16 13:39:19 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:19.126914 30983 wrap.go:75] GET /api/v1/nodes?resourceVersion=11426&timeoutSeconds=405&watch=true: (6m45.000742185s) 200 [[kube-contro 4月 16 13:39:19 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:19.127789 30983 rest.go:320] Starting watch for /api/v1/nodes, rv=11426 labels= fields= timeout=6m51s 4月 16 13:39:20 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:20.258700 30983 wrap.go:75] GET /api/v1/namespaces/kube-system/endpoints/kube-controller-manager: (1.853514ms) 200 [[kube-controller-m 4月 16 13:39:20 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:20.264227 30983 wrap.go:75] PUT /api/v1/namespaces/kube-system/endpoints/kube-controller-manager: (4.660183ms) 200 [[kube-controller-m 4月 16 13:39:20 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:20.806328 30983 wrap.go:75] GET /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (1.543562ms) 200 [[kube-scheduler/v1.6.1 (lin 4月 16 13:39:20 v157-7-222-131.myvps.jp kube-apiserver[30983]: I0416 13:39:20.832937 30983 wrap.go:75] PUT /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (25.436626ms) 200 [[kube-scheduler/v1.6.1 (li

Pod中无法访问外网

在集群部署完成后,出现Pod无法访问外网及集群内其它主机的情况。

/ # ping -c5 www.qq.com
ping: bad address 'www.qq.com'
/ # ping -c5 123.151.148.111
PING 123.151.148.111 (123.151.148.111): 56 data bytes

--- 123.151.148.111 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

/ # ping -c5 192.168.8.12
PING 192.168.8.12 (192.168.8.12): 56 data bytes

--- 192.168.8.12 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

集群部署配置:

master 192.168.8.6
node   192.168.8.9   192.168.8.10   192.168.8.11   192.168.8.12

所有Pod情况:

root@master6:~# kubectl get pods -o wide
NAME                       READY     STATUS    RESTARTS   AGE       IP            NODE
hello                      1/1       Running   0          2d        172.30.69.2   192.168.8.10
inet-3821096285-2zgzt      1/1       Running   0          17h       172.30.28.3   192.168.8.11
my-nginx-858393261-8d1zr   1/1       Running   1          2d        172.30.12.2   192.168.8.9
my-nginx-858393261-vfs30   1/1       Running   0          2d        172.30.79.3   192.168.8.12
my-nginx-858393261-x466f   1/1       Running   0          2d        172.30.28.2   192.168.8.11

Pod之间进行ping操作

# 从 172.30.28.3 进行ping操作
/ # ping -c3 172.30.69.2
PING 172.30.69.2 (172.30.69.2): 56 data bytes
64 bytes from 172.30.69.2: seq=0 ttl=62 time=2.616 ms
64 bytes from 172.30.69.2: seq=1 ttl=62 time=0.821 ms
64 bytes from 172.30.69.2: seq=2 ttl=62 time=0.670 ms

--- 172.30.69.2 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.670/1.369/2.616 ms
/ # ping -c3 172.30.28.2
PING 172.30.28.2 (172.30.28.2): 56 data bytes
64 bytes from 172.30.28.2: seq=0 ttl=64 time=0.614 ms
64 bytes from 172.30.28.2: seq=1 ttl=64 time=0.205 ms
64 bytes from 172.30.28.2: seq=2 ttl=64 time=0.142 ms

--- 172.30.28.2 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.142/0.320/0.614 ms

iptables规则:

root@minion12:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-FIREWALL  all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-SERVICES (1 references)
target     prot opt source               destination
REJECT     tcp  --  anywhere             anywhere             /* default/i1api: has no endpoints */ ADDRTYPE match dst-type LOCAL tcp dpt:8501 reject-with icmp-port-unreachable
REJECT     tcp  --  anywhere             10.254.238.108       /* default/i1api: has no endpoints */ tcp dpt:http reject-with icmp-port-unreachable

不能访问service页面

感谢您 的文档,目前服务都启动成功了,但是不能通过页面去访问:

Master/Node IPs:

...
master: 10.192.29.207
node: 10.192.29.208
...

Services status:

[root@localhost heapster]# /root/local/bin/kubectl get services --all-namespaces -s 10.192.29.207:8080
NAMESPACE     NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)                       AGE
default       kubernetes             10.254.0.1       <none>        443/TCP                       5h
kube-system   heapster               10.254.56.15     <none>        80/TCP                        21m
kube-system   kube-dns               10.254.0.2       <none>        53/UDP,53/TCP                 3h
kube-system   kubernetes-dashboard   10.254.172.131   <nodes>       80:8791/TCP                   1h
kube-system   monitoring-grafana     10.254.80.55     <none>        80/TCP                        21m
kube-system   monitoring-influxdb    10.254.223.124   <nodes>       8086:8686/TCP,8083:8614/TCP   21m

Pods Status

[root@localhost heapster]# /root/local/bin/kubectl get pods --all-namespaces -s 10.192.29.207:8080
NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE
kube-system   heapster-334572188-63gqs                1/1       Running   1          27m
kube-system   kube-dns-2298276164-frjpg               3/3       Running   6          3h
kube-system   kubernetes-dashboard-3377982832-r5906   1/1       Running   1          1h
kube-system   monitoring-grafana-854043867-6zddh      1/1       Running   1          27m
kube-system   monitoring-influxdb-340252977-n944m     1/1       Running   1          27m

Cluster info

[root@localhost heapster]# /root/local/bin/kubectl cluster-info -s 10.192.29.207:8080
Kubernetes master is running at 10.192.29.207:8080
Heapster is running at 10.192.29.207:8080/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at 10.192.29.207:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at 10.192.29.207:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
monitoring-grafana is running at 10.192.29.207:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
monitoring-influxdb is running at 10.192.29.207:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb

Browser - http://10.192.29.207:8080

{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/apps",
    "/apis/apps/v1beta1",
    "/apis/authentication.k8s.io",
    "/apis/authentication.k8s.io/v1",
    "/apis/authentication.k8s.io/v1beta1",
    "/apis/authorization.k8s.io",
    "/apis/authorization.k8s.io/v1",
    "/apis/authorization.k8s.io/v1beta1",
    "/apis/autoscaling",
    "/apis/autoscaling/v1",
    "/apis/autoscaling/v2alpha1",
    "/apis/batch",
    "/apis/batch/v1",
    "/apis/batch/v2alpha1",
    "/apis/certificates.k8s.io",
    "/apis/certificates.k8s.io/v1beta1",
    "/apis/extensions",
    "/apis/extensions/v1beta1",
    "/apis/policy",
    "/apis/policy/v1beta1",
    "/apis/rbac.authorization.k8s.io",
    "/apis/rbac.authorization.k8s.io/v1alpha1",
    "/apis/rbac.authorization.k8s.io/v1beta1",
    "/apis/settings.k8s.io",
    "/apis/settings.k8s.io/v1alpha1",
    "/apis/storage.k8s.io",
    "/apis/storage.k8s.io/v1",
    "/apis/storage.k8s.io/v1beta1",
    "/healthz",
    "/healthz/ping",
    "/healthz/poststarthook/bootstrap-controller",
    "/healthz/poststarthook/ca-registration",
    "/healthz/poststarthook/extensions/third-party-resources",
    "/healthz/poststarthook/rbac/bootstrap-roles",
    "/logs",
    "/metrics",
    "/swagger-ui/",
    "/swaggerapi/",
    "/ui/",
    "/version"
  ]
}

flannel的配置,PS 在master和node上都配置了flannel

[root@localhost ~]# /root/local/bin/etcdctl   --endpoints=${ETCD_ENDPOINTS}   --ca-file=/etc/kubernetes/ssl/ca.pem   --cert-file=/etc/kubernetes/ssl/kubernetes.pem   --key-file=/etc/kubernetes/ssl/kubernetes-key.pem   ls ${FLANNEL_ETCD_PREFIX}/subnets
2017-04-24 17:28:20.417878 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
/kubernetes/network/subnets/172.30.66.0-24
/kubernetes/network/subnets/172.30.38.0-24
[root@localhost ~]# /root/local/bin/etcdctl   --endpoints=${ETCD_ENDPOINTS}   --ca-file=/etc/kubernetes/ssl/ca.pem   --cert-file=/etc/kubernetes/ssl/kubernetes.pem   --key-file=/etc/kubernetes/ssl/kubernetes-key.pem   get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.38.0-24
2017-04-24 17:28:28.398278 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
{"PublicIP":"10.192.29.208","BackendType":"vxlan","BackendData":{"VtepMAC":"6a:06:02:bb:c2:21"}}

但是通过 浏览器都不能访问UI, 比如dashboard UI: http://10.192.29.207:8080/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/

Error: 'dial tcp 172.30.38.2:9090: getsockopt: connection timed out'
Trying to reach: 'http://172.30.38.2:9090/'

由于dashboard 还暴露了NodePort

kube-system   kubernetes-dashboard   10.254.172.131   <nodes>       80:8791/TCP                   1h

所以我通过浏览器访问http://10.192.29.208:8791, 也是报同样的错误

ERROR

The requested URL could not be retrieved

The following error was encountered while trying to retrieve the URL: http://10.192.29.208:8791/

Connection to 10.192.29.208 failed.

The system returned: (110) Connection timed out

The remote host or network may be down. Please try the request again.

Your cache administrator is webmaster.

然后我在Node中curl http://10.192.29.208:8791, 是有返回正常页面内容的(虽然是浏览器不兼容的信息)

[root@localhost ~]# curl 10.192.29.208:8791
 <!doctype html> <html ng-app="kubernetesDashboard"> <head> <meta charset="utf-8"> <title ng-controller="kdTitle as $ctrl" ng-bind="$ctrl.title()"></title> <link rel="icon" type="image/png" href="assets/images/kubernetes-logo.png"> <meta name="viewport" content="width=device-width"> <link rel="stylesheet" href="static/vendor.4f4b705f.css"> <link rel="stylesheet" href="static/app.93b90a74.css"> </head> <body> <!--[if lt IE 10]>
      <p class="browsehappy">You are using an <strong>outdated</strong> browser.
      Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your
      experience.</p>
    <![endif]--> <kd-chrome layout="column" layout-fill> </kd-chrome> <script src="static/vendor.6952e31e.js"></script> <script src="api/appConfig.json"></script> <script src="static/app.8a6b8127.js"></script> </body> </html> [root@localhost ~]#

想请教下,怎么通过master访问页面? thx.

配置和启动 kube-apiserver时候配置路径问题

创建 kube-apiserver 的 systemd unit 文件

$ cat > kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
ExecStart=/root/local/bin/kube-apiserver \
--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
--advertise-address=${MASTER_IP} \
--bind-address=${MASTER_IP} \
--insecure-bind-address=${MASTER_IP} \
--authorization-mode=RBAC \
--runtime-config=rbac.authorization.k8s.io/v1alpha1 \
--kubelet-https=true \
--experimental-bootstrap-token-auth \
--token-auth-file=/etc/kubernetes/token.csv \
--service-cluster-ip-range=${SERVICE_CIDR} \
--service-node-port-range=${NODE_PORT_RANGE} \
--tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--service-account-key-file=/etc/kubernetes/ca-key.pem \
--etcd-cafile=/etc/kubernetes/ssl/ca.pem \
--etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem \
--etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem \
--etcd-servers=${ETCD_ENDPOINTS} \

这个配置--service-account-key-file=/etc/kubernetes/ca-key.pem 指定的路径和
TLS 证书文件

$ sudo mkdir -p /etc/kubernetes/ssl
$ sudo cp token.csv /etc/kubernetes/ssl
$ sudo cp ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem /etc/kubernetes/ssl
这里指定的路径不一致

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.