Menu

Subscribe

AWS EC2安装Kubernetes (K8S)集群

环境说明：

10.0.0.27 node1 Amazon Linux 2 kernel 5.10
10.0.0.180 master Amazon Linux 2 kernel 5.10
10.0.0.106 node2 Amazon Linux 2 kernel 5.10

设置hostname

hostnamectl set-hostname master #只在master上执行
hostnamectl set-hostname node1 #只在node1上执行
hostnamectl set-hostname node2 #只在node2上执行

添加hosts

cat >> /etc/hosts << EOF
10.0.0.27 node1
10.0.0.180 master
10.0.0.106 node2
EOF

关闭防火墙（Amazon Linux 2 没有这个）：

systemctl stop firewalld
systemctl disable firewalld

关闭selinux：

sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
setenforce 0 # 临时

关闭swap：

swapoff -a # 临时
sed -i 's/.swap./#&/' /etc/fstab # 永久

将桥接的IPv4 流量传递到iptables 的链：

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720         
EOF
sysctl --system

安装docker

yum -y install epel-release wget
#wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
#yum list docker-ce --showduplicates | sort -r #查看yum仓库中可以安装的docker版本
#yum -y install docker-ce-18.06.1.ce-3.el7 #安装固定版本
#yum -y install docker-ce  #安装docker最新版
我在amazon linux2上安装，直接 amazon-linux-extras install docker -y
systemctl enable docker && systemctl start docker && systemctl status docker
docker --version

cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://qj799ren.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

systemctl restart docker &&  systemctl status docker
docker --version

下面的部分可以忽略了，直接跳到安装Kube系列

在执行yum -y install docker-ce的时候，遇到了报错

https://mirrors.aliyun.com/docker-ce/linux/centos/2/x86_64/stable/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.

查看阿里云镜像仓库路径 https://mirrors.aliyun.com/docker-ce/linux/centos/?spm=a2c6h.25603864.0.0.12d73da0rxLQlU
再对比docker-ce.repo里面具体的路径，如上面报错，$releasever对应的版本在安装的时候显示的是2，而阿里云仓库没有2这个版本。关于$releasever的值，可以通过这篇文档里的方法查看
https://unix.stackexchange.com/questions/19701/yum-how-can-i-view-variables-like-releasever-basearch-yum0
看了一下修改版本的方法 https://donghao.org/2015/04/30/the-value-of-releasever-for-yum/ ，Amazon Linux 并没有这个文件，因此还是在repo文件里直接写死版本2就完事了。例如当前我看到阿里云仓库最新的版本是9，我就用这个命令：

sed 's/$releasever/9/g' /etc/yum.repos.d/docker-ce.repo -i

然后就解决了这个报错。

安装Kube系列

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum clean all && yum makecache -y
yum list kubectl --showduplicates | sort -r #列出kubectl可用的版本

centos系统安装命令

yum -y install kubelet-1.23.9-0 kubeadm-1.23.9-0 kubectl-1.23.9-0
systemctl enable kubelet

只在master上执行，apiserver-advertise-address需要是master的IP

kubeadm init --apiserver-advertise-address=10.0.0.180  --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.23.9 --service-cidr=10.96.0.0/12  --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all

关键输出

[root@master ec2-user]# kubeadm init --apiserver-advertise-address=10.0.0.180  --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.23.9 --service-cidr=10.96.0.0/12  --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all
[init] Using Kubernetes version: v1.23.9
[preflight] Running pre-flight checks
        [WARNING FileExisting-tc]: tc not found in system path
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 10.0.0.180]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [10.0.0.180 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [10.0.0.180 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 7.503916 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: covumo.86j2il68oqp4qmto
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.0.180:6443 --token covumo.86j2il68oqp4qmto \
        --discovery-token-ca-cert-hash sha256:1864a0dd936f1e2d06748b522b1c42cebedb31550e1e0291bc1625c7c0e14fb7

（附）kubeadm停止流程 （跳到下面的命令启动集群）

  kubeadm reset -f
  rm -rf ~/.kube/
  rm -rf /etc/kubernetes/
  rm -rf /etc/systemd/system/kubelet.service.d
  rm -rf /etc/systemd/system/kubelet.service
  rm -rf /usr/bin/kube*
  yum clean all
  yum remove kube*

执行这些命令启动集群

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

之后就可以用kubectl了

[root@master ec2-user]# kubectl get nodes     #缺少网络插件所以才是NotReady状态
NAME     STATUS     ROLES                  AGE   VERSION
master   NotReady   control-plane,master   13m   v1.23.9

安装Pod 网络插件（CNI）（需要找到yaml，我用的是参考博客里的，太长就不放了。）

#在Master上操作
cat > calico.yaml << EOF
....
EOF

kubectl apply -f calico.yaml
kubectl get pods -n kube-system

[root@master ec2-user]# kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-677cd97c8d-bpncn   1/1     Running   0          2m6s
calico-node-4rqlg                          1/1     Running   0          2m6s
coredns-6d8c4cb4d-l75wm                    1/1     Running   0          42m
coredns-6d8c4cb4d-rt288                    1/1     Running   0          42m
etcd-master                                1/1     Running   2          42m
kube-apiserver-master                      1/1     Running   3          42m
kube-controller-manager-master             1/1     Running   1          42m
kube-proxy-dg8j9                           1/1     Running   0          42m
kube-scheduler-master                      1/1     Running   1          42m

Node节点加入Kubernetes

[root@node1 ec2-user]# kubeadm join 10.0.0.180:6443 --token covumo.86j2il68oqp4qmto \
>             --discovery-token-ca-cert-hash sha256:1864a0dd936f1e2d06748b522b1c42cebedb31550e1e0291bc1625c7c0e14fb7 
[preflight] Running pre-flight checks
        [WARNING FileExisting-tc]: tc not found in system path
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

检查集群的状态

kubectl get nodes
kubectl get pods -n kube-system

[root@master ec2-user]# kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-677cd97c8d-bpncn   1/1     Running   0          74m
calico-node-4rqlg                          1/1     Running   0          74m
calico-node-9r46p                          1/1     Running   0          68m
calico-node-pls87                          1/1     Running   0          68m
coredns-6d8c4cb4d-l75wm                    1/1     Running   0          114m
coredns-6d8c4cb4d-rt288                    1/1     Running   0          114m
etcd-master                                1/1     Running   2          115m
kube-apiserver-master                      1/1     Running   3          115m
kube-controller-manager-master             1/1     Running   1          115m
kube-proxy-5wstf                           1/1     Running   0          68m
kube-proxy-ccjw6                           1/1     Running   0          68m
kube-proxy-dg8j9                           1/1     Running   0          114m
kube-scheduler-master                      1/1     Running   1          115m

测试kubernetes 集群

##在kubernetes群集中创建Nginx:
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc -o wide

[root@master ec2-user]# kubectl get pod,svc -o wide
NAME                         READY   STATUS    RESTARTS   AGE    IP             NODE    NOMINATED NODE   READINESS GATES
pod/nginx-85b98978db-ztdjf   1/1     Running   0          9m9s   10.244.104.1   node2   <none>           <none>

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE     SELECTOR
service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        130m    <none>
service/nginx        NodePort    10.110.245.235   <none>        80:30600/TCP   3m15s   app=nginx

访问各个node的30600端口即可。
[ec2-user@ip-172-31-27-54 ~]$ curl http://node1:30600 -I
HTTP/1.1 200 OK
Server: nginx/1.21.5
Date: Mon, 18 Sep 2023 06:30:24 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 28 Dec 2021 15:28:38 GMT
Connection: keep-alive
ETag: "61cb2d26-267"
Accept-Ranges: bytes

[ec2-user@ip-172-31-27-54 ~]$ curl http://master:30600 -I
HTTP/1.1 200 OK
Server: nginx/1.21.5
Date: Mon, 18 Sep 2023 06:30:30 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 28 Dec 2021 15:28:38 GMT
Connection: keep-alive
ETag: "61cb2d26-267"
Accept-Ranges: bytes

创建Dashboard（非必须）

[root@master ec2-user]# kubectl apply -f recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
[root@master ec2-user]# kubectl get pods -n kubernetes-dashboard
NAME                                         READY   STATUS              RESTARTS   AGE
dashboard-metrics-scraper-6f669b9c9b-njwq8   0/1     ContainerCreating   0          5s
kubernetes-dashboard-758765f476-hk252        0/1     ContainerCreating   0          5s

创建service account并绑定默认cluster-admin管理员群集角色
创建用户
kubectl create serviceaccount dashboard-admin -n kube-system
用户授权
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
获取用户Token
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

使用输出的token登录Dashboard
然后访问网址，输入token登录。
https://10.0.0.27:30001
https://10.0.0.180:30001
https://10.0.0.106:30001

[root@master ec2-user]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
Name:         dashboard-admin-token-6xdkl
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: c841a8a7-9b09-4104-9d50-192542f462da

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1099 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6Ik44em44YnBJUFVwMWhmbkRwUFA5Z1hUa1JSVWlsZll0alRYTFpJWlczVjQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tNnhka2wiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYzg0MWE4YTctOWIwOS00MTA0LTlkNTAtMTkyNTQyZjQ2MmRhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.dLvX4PeYCNaw_95zAYywZXrMbIFb1iW8trBzVPsp7BSouY27rC6quLa0rC1LcspNhmN-GwsYWEXXKVSJEDP1OJOrG1M4wgyGZ99fbE8iqfJxUm7sw2R0saG8mt6DFhJZ5UI3Hr9dPOGpjsNpxFYx0qxOUnMOOmjaRxEqexnfmeHyf4QpOK5HaARY8etjijTByMr9ya2zPb8S_Wc6gdhSNFAfNDuf8OaSKSTysNRmC_KvVBS5CGQyu1Gmx-Mxm_cRiRBGVMGO6s1xHnm6SdkA1sKu4qKa1XHe7yKwuF0hs1fZAK_aCZmIZspvhSSYdZlbsY6L0HWbtPoKQTRvUFUC9w

部署metrics-server（非必须）

[root@master ec2-user]# kubectl apply -f components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

kubectl get pods -n kube-system|grep metrics
kubectl top nodes
kubectl top pods

补全命令的功能（非必须）

rpm -aq |grep completion
yum -y install bash-completion #安装补全命令的包
kubectl completion bash
source /usr/share/bash-completion/bash_completion
kubectl completion bash >/etc/profile.d/kubectl.sh
source /etc/profile.d/kubectl.sh
cat >> /root/.bashrc << EOF
source /etc/profile.d/kubectl.sh
EOF

参考文档

https://blog.csdn.net/qq_14910065/article/details/131404563
https://blog.csdn.net/jll126/article/details/125193274

默认分类 2

发表评论

  • OωO
  • |´・ω・)ノ
  • ヾ(≧∇≦*)ゝ
  • (☆ω☆)
  • （╯‵□′）╯︵┴─┴
  • ￣﹃￣
  • (/ω＼)
  • ∠(ᐛ」∠)＿
  • (๑•̀ㅁ•́ฅ)
  • →_→
  • ୧(๑•̀⌄•́๑)૭
  • ٩(ˊᗜˋ*)و
  • (ノ°ο°)ノ
  • (´இ皿இ｀)
  • ⌇●﹏●⌇
  • (ฅ´ω`ฅ)
  • (╯°A°)╯︵○○○
  • φ(￣∇￣o)
  • (งᵒ̌皿ᵒ̌)ง⁼³₌₃
  • (ó﹏ò｡)
  • Σ(っ°Д°;)っ
  • ╮(╯▽╰)╭
  • o(*
  • ＞﹏＜
  • (｡•ˇ‸ˇ•｡)
  • 泡泡
  • 颜文字

*