ES

ElasticsearchGrafana Labs

1 数据源

1.1 ES 数据

"vm_doc": {
    "env_type": "online",                       # sandbox/online
    "pool": "xxx",                              # pool
    "region": "xxx",                            # region
    "zone": "AZONE-xxx",                        # 可用区
    "deployset_name": "xxx",                    # 部署集
    "host_ip": "xx.xx.xx.xx",                   # 机器 ip
    "is_reserve": "no",                         # 是否为预留机器

    "free_cpu_in_core": 5,                      # cpu 库存
    "free_mem_size_in_mb": 42021,               # mem 库存
    "free_disk_size_in_gb": 330,                # disk 库存

    "allocated_cpu_in_core": 3,                 # cpu 已分配量
    "allocated_mem_size_in_mb": 11979,          # mem 已分配量     
    "allocated_disk_size_in_gb": 120,           # disk 已分片量
    
    "used_mem_size_in_mb": 2048,                # mem 已使用量
    
    "total_cpu_in_core",                        # cpu 逻辑总量
    "total_mem_size_in_mb",                     # mem 逻辑总量
    "total_disk_size_in_gb",                    # disk 逻辑总量
    
    "physical_total_cpu_in_core",               # cpu 物理总量
    "physical_total_mem_size_in_mb",            # mem 物理总量

    "instance_count": 3,                        # 机器上容器个数
    "day_time": "2022-08-20",
    "timestamp": "2022-08-20T10:35:22.488559",
}

1.2 配置(add data source)

索引格式: vm-unit_2022-08-20

参数	值
Index name	[vm-unit_]YYYY-MM-DD
Pattern	Daily
Time field name	timestamp
ElasticSearch version	7.10+
Max concurrent Shard Requests	5
Min time interval	1d

2 仪表盘变量(dashboard/dashboard settings/Variables)

添加如下变量后，会在 dashboard 的上方出现下拉框

env_type        {"find": "terms", "field": "env_type.keyword", "size": 100}		
pool            {"find": "terms", "field": "pool.keyword", "size": 100}		
zone            {"find": "terms", "field": "zone.keyword", "size": 100}		
deployset_name  {"find": "terms", "field": "deployset_name.keyword", "size": 100, "query": "zone:$zone"}		

2.1 操作方法

General:
    Name:pool
    Type:Query
    -------其他项为空
Query Options:
    Data source: ${data_source}
    Refresh: On dashboard load
    Query: {"find": "terms", "field": "pool.keyword", "size": 100}
    Regex: 空
    Sort： Disable
Selection options：
    Multi-value: true
    Include All option: true
    Custom all value: 空

3 仪表盘(汇总图)

3.1 机器数量(折线图)

Query:
    pool:$pool AND env_type:online AND is_reserve:no
Alias
    无
Metric:
    count
Group By
    Terms, pool.keyword, Top 10, Min Doc Count: 1, Order by: Count
Then By
    Date Histogram, timestamp, Interval: 1d

• Terms: 表示查询具体的值

• Date Histogram：时间直方图

上面图会根据 pool 展示多条曲线，每条曲线代表一个 pool，横坐标是时间。

如果要对 pool 进行汇总，则可以点击 Transform，然后点击 Add field from calculation

Mode
    Reduce row
Field name
    pool1,pool2
Calculation
    Total
Alias
    Total
Replace all fields
    false

这样就会汇总 pool1 和 pool2 数据为 Total 了

3.2 内存

CPU 与内存指标类似

3.2.1 内存总分配量 or 物理总内存

Query
    pool:$pool AND env_type:online AND is_reserve:no
Alias
    无
Metric (1)
    Sum，allocated_mem_size_in_mb
Group By
    Terms, pool.keyword, Top 10, Min Doc Count: 1, Order by: Sum allocated_mem_size_in_mb
Then By
    Date Histogram, timestamp, Interval: 1d

再根据 3.1 生成下所有 pool 的总量曲线图。

3.2.2 内存分配率

内存分配率需要按照池单独生成图(如生成 pool1 的内存分配率)

Query

A----------------------------------------------
Query
    pool:pool1 AND env_type:online AND is_reserve:no
Alias
    allocated_mem
Metric (1)
    Sum, allocated_mem_size_in_mb
Group By
    Date Histogram, timestamp, Interval: 1d

B----------------------------------------------
Query
    pool:scs AND env_type:online AND is_reserve:no
Alias
    sum_mem
Metric (1)
    Sum, physical_total_mem_size_in_mb
Group By
    Date Histogram, timestamp, Interval: 1d

Transform

Add field from calculation

Mode
    Binary operation
Operation
    allocated_mem/sum_mem
Alias
    allocated_mem / sum_mem
Replace all fields
    true(这里需要设置为 true，只显示比例图)

4 仪表盘(按部署集区分)

4.1 机器数量(折线图)

每个部署集(deployset_name) 1 条曲线，横坐标是时间(每天一个点)

Query:
    zone:$zone AND pool:$pool AND deployset_name:$deployset_name AND env_type:$env_type

Metric:
    count

Group By:
    Terms, zone.keyword, Top 10, Min Doc Count: 1, Order by: Count
Then By:
    Date Histogram, timestamp, Interval: 1d

4.2 库存(折线图)

每个部署集(deployset_name) 1 条曲线，横坐标是时间(每天一个点)

Query:
    zone:$zone AND pool:$pool AND deployset_name:$deployset_name AND env_type:$env_type

Metric (1)
    Sum, free_cpu_in_core

Group By:
    Terms, deployset_name.keyword, Top 10, Min Doc Count: 1, Order by: Sum free_cpu_in_core
Then By
    Date Histogram, timestamp, Interval: 1d