prometheus监控vmware操作文档

本文最后更新于0 天前，其中的信息可能已经过时，如有错误请发送邮件到big_fw@foxmail.com

一、采集vmware物理机信息

两种部署方式

一种是直接在虚拟机上部署vmware-exporter
另一种是直接部署到kubernetes中

方案一：虚拟机方式部署vmware-exporter

1.1 虚拟机配置信息参考

情况一：有vsphere center管理端

操作系统	CPU	内存	系统盘	数据盘
CentOS 7.9.2009	2	4	40G	200G

情况二：无vsphere center管理端

操作系统	CPU	内存	系统盘	数据盘
CentOS 7.9.2009	4	16	40G	200G

当没有vsphere center管理端的时候，需要针对每一个ESXI主机进行单独监控，因此会需要更多的资源

根据情况自行选择其中一种即可。

1.2 磁盘分区和挂载（如果有必要）

如果数据盘已完成分区、格式化、挂载等操作，此步骤可以省略

1）查看磁盘设备号

# 执行以下命令
lsblk

2）磁盘分区

# fdisk /dev/vdb        #执行此命令，进行分区
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xdf09eb6a.

Command (m for help): p   #在此输入p

Disk /dev/vdb: 1073.7 GB, 1073741824000 bytes, 2097152000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xdf09eb6a

    Device Boot      Start         End      Blocks   Id  System

Command (m for help): n   #在此输入n
Partition type:
    p   primary (0 primary, 0 extended, 4 free)
    e   extended
Select (default p): p
Partition number (1-4, default 1): 1    #在此输入1
First sector (2048-419430399, default 2048):       #回车
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-419430399, default 419430399):     #回车
Using default value 419430399
Partition 1 of type Linux and of size 200 GiB is set

Command (m for help): t   #在此输入t
Selected partition 1
Hex code (type L to list all codes): L     #在此输入L

  0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris        
  1  FAT12           27  Hidden NTFS Win 82  Linux swap / So c1  DRDOS/sec (FAT-
  2  XENIX root      39  Plan 9          83  Linux           c4  DRDOS/sec (FAT-
  3  XENIX usr       3c  PartitionMagic  84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
  4  FAT16 <32M      40  Venix 80286     85  Linux extended  c7  Syrinx         
  5  Extended        41  PPC PReP Boot   86  NTFS volume set da  Non-FS data    
  6  FAT16           42  SFS             87  NTFS volume set db  CP/M / CTOS / .
  7  HPFS/NTFS/exFAT 4d  QNX4.x          88  Linux plaintext de  Dell Utility   
  8  AIX             4e  QNX4.x 2nd part 8e  Linux LVM       df  BootIt         
  9  AIX bootable    4f  QNX4.x 3rd part 93  Amoeba          e1  DOS access     
  a  OS/2 Boot Manag 50  OnTrack DM      94  Amoeba BBT      e3  DOS R/O        
  b  W95 FAT32       51  OnTrack DM6 Aux 9f  BSD/OS          e4  SpeedStor      
  c  W95 FAT32 (LBA) 52  CP/M            a0  IBM Thinkpad hi eb  BeOS fs        
  e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a5  FreeBSD         ee  GPT            
  f  W95 Ext'd (LBA) 54  OnTrackDM6      a6  OpenBSD         ef  EFI (FAT-12/16/
10  OPUS            55  EZ-Drive        a7  NeXTSTEP        f0  Linux/PA-RISC b
11  Hidden FAT12    56  Golden Bow      a8  Darwin UFS      f1  SpeedStor      
12  Compaq diagnost 5c  Priam Edisk     a9  NetBSD          f4  SpeedStor      
14  Hidden FAT16 <3 61  SpeedStor       ab  Darwin boot     f2  DOS secondary  
16  Hidden FAT16    63  GNU HURD or Sys af  HFS / HFS+      fb  VMware VMFS    
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE 
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fe  LANstep        
1c  Hidden W95 FAT3 75  PC/IX           be  Solaris boot    ff  BBT            
1e  Hidden W95 FAT1 80  Old Minix      
Hex code (type L to list all codes): 8e      #输入lvm类型的hex代码
Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): p    #在此输入p

Disk /dev/sdb: 214.7 GB, 214748364800 bytes, 419430400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xdf09eb6a

    Device Boot      Start         End      Blocks   Id  System
/dev/vdb1            2048  2097151999  1048574976   8e  Linux LVM

Command (m for help): w     #保存
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

3）创建PV

pvcreate /dev/vdb1

创建好PV后，查看创建的PV

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/vdb1
  VG Name               vgdata
  PV Size               <1000.00 GiB / not usable 3.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              255999
  Free PE               255
  Allocated PE          255744
  PV UUID               dejeTn-cVJc-qWXM-EWRw-gKge-znNc-1a6VGH

4）创建VG

vgcreate vgdata /dev/vdb1

查看创建的VG

# vgdisplay
  --- Volume group ---
  VG Name               vgdata
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <1000.00 GiB
  PE Size               4.00 MiB
  Total PE              255999
  Alloc PE / Size       255744 / 999.00 GiB
  Free  PE / Size       255 / 1020.00 MiB
  VG UUID               R0tJm5-guE4-vy3Q-UgJX-jSkt-mLdo-T4bvxU

5）创建LV

lvcreate -l 100%FREE -n lvdata vgdata

6）磁盘格式化

mkfs.xfs /dev/vgdata/lvdata

mkdir /data
mount /dev/mapper/vgdata-lvdata /data

查看挂载后的分区

df -hT

8）设置开机自动挂载

vim /etc/fstab
/dev/mapper/vgdata-lvdata /data xfs defaults 0 0   #添加到最后一行即可

1.3 部署docker和docker-compose

1）安装必要的系统工具

yum install -y yum-utils device-mapper-persistent-data lvm2

2）添加docker的软件源

yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

3）更换软件源为阿里云软件源

sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo

4）安装docker

yum makecache fast

yum -y install docker-ce

5）配置docker

# 操作之前需要先提前进行磁盘分区，保证当前机器有/data数据盘
# 创建容器数据目录
mkdir /data/docker -p

# 修改docker配置
mkdir /etc/docker
vim /etc/docker/daemon.json
{
    "registry-mirrors": ["https://dockerhub.azk8s.cn", "https://docker.mirrors.ustc.edu.cn"],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "graph": "/data/docker",
    "storage-driver": "overlay2",
    "insecure-registries": [
        "registry.cn-beijing.aliyuncs.com",
        "registry.cn-hangzhou.aliyuncs.com"
        ],
    "max-concurrent-downloads": 10,
    "log-driver": "json-file",
    "log-level": "warn",
    "log-opts": {
        "max-size": "100m",
        "max-file": "3"
        }
}

6）启动docker

systemctl daemon-reload
systemctl start docker && systemctl enable docker

7）安装docker-compose

curl -SL https://github.com/docker/compose/releases/download/v2.18.1/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose

ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

1.4 docker-compose方式启动vmware-exporter

创建存放docker-compose启动文件的目录

mkdir /opt/docker-compose

1）情况一

如果有vsphere center，安装过程就比较简单，只需要执行以下命令即可

# vim /opt/docker-compose/vmware.yaml

# 将以下内容保存到/opt/docker-compose/vmware.yaml文件中
version: '3'
services:
  vmware_exporter:
    ports:
      - '0.0.0.0:9272:9272'
    restart: always
    environment:
      - VSPHERE_USER=修改为登录虚拟化平台的用户名
      - VSPHERE_PASSWORD=修改为登录虚拟化平台的密码
      - VSPHERE_HOST=修改为虚拟化平台的地址
      - VSPHERE_IGNORE_SSL=True
      - VSPHERE_SPECS_SIZE=2000
    image: pryorda/vmware_exporter
    volumes:
    - /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime

通过docker-compose进行启动

cd /opt/docker-compose

docker-compose -f vmware.yaml up -d

验证是否有监控数据

curl http://127.0.0.1:9272/metrics

2）情况二

如果没有vsphere center管理端工具，需要针对每一台物理机启动不同的容器进行监控，启动时需要设置不同的端口，如下所示：

# vim /opt/docker-compose/vmware.yaml

# 将以下内容保存到/opt/docker-compose/vmware.yaml文件中
version: '3'
services:
  # 修改此名称，不要有重复
  vmware_exporter01:
    ports:
      # 注意修改:前面的端口后，不要有重复
      - '0.0.0.0:9272:9272'
    restart: always
    environment:
      - VSPHERE_USER=修改为登录虚拟化平台的用户名
      - VSPHERE_PASSWORD=修改为登录虚拟化平台的密码
      - VSPHERE_HOST=修改为虚拟化平台的地址
      - VSPHERE_IGNORE_SSL=True
      - VSPHERE_SPECS_SIZE=2000
    image: pryorda/vmware_exporter
    volumes:
    - /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime
  # 修改此名称，不要有重复
  vmware_exporter02:
    ports:
      # 注意修改:前面的端口后，不要有重复
      - '0.0.0.0:9273:9272'
    restart: always
    environment:
      - VSPHERE_USER=修改为登录虚拟化平台的用户名
      - VSPHERE_PASSWORD=修改为登录虚拟化平台的密码
      - VSPHERE_HOST=修改为虚拟化平台的地址
      - VSPHERE_IGNORE_SSL=True
      - VSPHERE_SPECS_SIZE=2000
    image: pryorda/vmware_exporter
    volumes:
    - /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime
  # 修改此名称，不要有重复
  vmware_exporter03:
    ports:
      # 注意修改:前面的端口后，不要有重复
      - '0.0.0.0:9274:9272'
    restart: always
    environment:
      - VSPHERE_USER=修改为登录虚拟化平台的用户名
      - VSPHERE_PASSWORD=修改为登录虚拟化平台的密码
      - VSPHERE_HOST=修改为虚拟化平台的地址
      - VSPHERE_IGNORE_SSL=True
      - VSPHERE_SPECS_SIZE=2000
    image: pryorda/vmware_exporter
    volumes:
    - /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime
  # 修改此名称，不要有重复
  vmware_exporter04:
    ports:
      # 注意修改:前面的端口后，不要有重复
      - '0.0.0.0:9275:9272'
    restart: always
    environment:
      - VSPHERE_USER=修改为登录虚拟化平台的用户名
      - VSPHERE_PASSWORD=修改为登录虚拟化平台的密码
      - VSPHERE_HOST=修改为虚拟化平台的地址
      - VSPHERE_IGNORE_SSL=True
      - VSPHERE_SPECS_SIZE=2000
    image: pryorda/vmware_exporter
    volumes:
    - /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime

通过docker-compose进行启动

cd /opt/docker-compose

docker-compose -f vmware.yaml up -d

验证是否有监控数据

# 根据不同的端口进行请求验证
# 物理机A
curl http://127.0.0.1:9272/metrics
# 物理机B
curl http://127.0.0.1:9273/metrics
# 物理机C
curl http://127.0.0.1:9274/metrics
# 物理机D
curl http://127.0.0.1:9275/metrics

注意

0.0.0.0:9272:9272，分为两部分理解，其中0.0.0.0:9272是将虚拟机的9272映射到容器中，而9272是容器中服务的监听端口

方案二：k8s方式部署vmware-exporter

如果有vsphere center，则直接将配置地址配置成管理端的账号信息即可，如下所示

---
# 注意：
# 如果是虚拟机部署的vmware-exporter，则修改下面的地址为IP
# 这里默认是指定的service
apiVersion: v1
data:
  esx.yaml: |-
    ---
    - targets:
      - vmware-exporter:9272
      labels:
        job: vmware_esx
kind: ConfigMap
metadata:
  name: esx-host-cm
---
# Source: vmware-exporter/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: vmware-exporter-secret
type: Opaque
data:
  VSPHERE_USER: eG1nbA==
  VSPHERE_PASSWORD: d3d3LlhNZ2wuYzBt
  VSPHERE_HOST: MTAuMTI3LjkxLjMx
---
# Source: vmware-exporter/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vmware-exporter
spec:
  selector:
    app: vmware-exporter
  type: ClusterIP
  ports:
  - name: http
    protocol: TCP
    port: 9272
---
# Source: vmware-exporter/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vmware-exporter
spec:
  selector:
    matchLabels:
      app: vmware-exporter
  template:
    metadata:
      labels:
        app: vmware-exporter
        release: vmware-exporter
      annotations:
        prometheus.io/path: "/metrics"
        prometheus.io/port: "9272"
        prometheus.io/scrape: "true"
    spec:
      containers:
      - name: vmware-exporter
        image: "pryorda/vmware_exporter:latest"
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 2
            memory: 500Mi
        livenessProbe:
          tcpSocket:
            port: 9272
          initialDelaySeconds: 5
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
          periodSeconds: 10
        ports:
        - containerPort: 9272
          name: http
        env:
        - name: VSPHERE_IGNORE_SSL
          value: "TRUE"
        - name: VSPHERE_SPECS_SIZE
          value: "2000"
        - name: VSPHERE_HOST
          valueFrom:
            secretKeyRef:
              name: vmware-exporter-secret
              key: VSPHERE_HOST
        - name: VSPHERE_USER
          valueFrom:
            secretKeyRef:
              name: vmware-10-127-91-31
              key: VSPHERE_USER
        - name: VSPHERE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: vmware-exporter-secret
              key: VSPHERE_PASSWORD
        volumeMounts:
        - name: localtime
          mountPath: /etc/localtime
      volumes:
      - name: localtime
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai

如果没有vsphere center管理端，则需要对YAML中的name进行改名操作

二、接入prometheus-operator中

修改prometheus-additional.yaml, 由于之前在prometheus-operator中添加了集群外部署的node-exporter，因此在之前的基础上进行修改即可

# 可以先进行查看
kubectl get secret -n monitoring additional-configs -o yaml

# 结果是base64 encode后的内容，需要进行decode

例如下面的这段配置

- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name
- job_name: 'elasticsearch-hosts'
  file_sd_configs:
  - files: ['/hosts/es-file.yaml']
    refresh_interval: 5s

# 添加esxi相关的配置
- job_name: 'vmware_esx'
  metrics_path: '/metrics'
  file_sd_configs:
  - files:
    - /vmware/esx.yaml
  params:
    section: [esx]

先删除原有的secret

kubectl delete secret -n monitoring additional-configs

重新创建secret

kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring

修改prometheus-operator部署文件，添加相关挂载信息

  volumeMounts:
  # 新增的vmware挂载
  - mountPath: /vmware
    name: esx-host-cm
    readOnly: true
  volumes:
  - configMap:
      name: prometheus-external-node-exporter-hosts-cm
    name: prometheus-external-node-exporter-hosts-cm
  # 新增的vmware configmap挂载
  - configMap:
      name: esx-host-cm
    name: esx-host-cm

三、添加grafana的dashboard

https://raw.githubusercontent.com/pryorda/vmware_exporter/main/dashboards/esx.json

一、采集vmware物理机信息

方案一：虚拟机方式部署vmware-exporter

1.1 虚拟机配置信息参考

1.2 磁盘分区和挂载 （如果有必要）

1.3 部署docker和docker-compose

1.4 docker-compose方式启动vmware-exporter

方案二：k8s方式部署vmware-exporter

二、接入prometheus-operator中

三、添加grafana的dashboard

发送评论 编辑评论

推荐文章

1.2 磁盘分区和挂载（如果有必要）

发送评论编辑评论