引言

线上又出故障了——接口变慢、内存飙高、GC 频繁。但你的反应是：

• 接口多慢？不知道，没有 QPS 和响应时间监控
• 内存为啥飙？不知道，没有 JVM 指标
• GC 多频繁？不知道，没有 GC 日志采集

没有指标，排障就是"猜"。

本文从零搭建一套完整的指标体系：Micrometer 采集 → Prometheus 存储 → Grafana 可视化 → AlertManager 告警。全实战，附完整配置和仪表盘 JSON。

一、指标体系全景

1.1 为什么需要指标

1.2 技术选型

1.3 数据流

Spring Boot App (Micrometer)
  │
  │ /actuator/prometheus（暴露指标端点）
  ▼
Prometheus（定时抓取，15s 一次）
  │
  ├──→ Grafana（查询 PromQL，画图）
  │
  └──→ AlertManager（规则匹配，触发告警）
           │
           ▼
        钉钉/飞书/邮件

二、Spring Boot 接入 Micrometer

2.1 添加依赖

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<!-- Micrometer Prometheus Registry -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

2.2 配置暴露端点

# application.yml
management:
  endpoints:
    web:
      exposure:
        include: prometheus,health,info,metrics
  metrics:
    tags:
      application: order-service  # 给所有指标加上 application 标签
    distribution:
      percentiles-histogram:
        http.server.requests: true  # 启用直方图，支持 P95/P99 计算
      percentiles:
        http.server.requests: 0.5, 0.95, 0.99  # 计算 P50/P95/P99

2.3 验证

启动应用，访问 http://localhost:8080/actuator/prometheus：

# JVM 内存指标
jvm_memory_used_bytes{area="heap",id="G1 Eden Space",application="order-service"} 2.543E7
jvm_memory_used_bytes{area="heap",id="G1 Survivor Space",application="order-service"} 1.048E6
jvm_memory_used_bytes{area="nonheap",id="Metaspace",application="order-service"} 8.234E7

# GC 指标
jvm_gc_pause_seconds{action="end of minor GC",cause="G1 Evacuation Pause",application="order-service"} 0.023
jvm_gc_pause_seconds_count{action="end of minor GC",cause="G1 Evacuation Pause",application="order-service"} 15

# 线程指标
jvm_threads_states_threads{state="runnable",application="order-service"} 12
jvm_threads_states_threads{state="waiting",application="order-service"} 34

# HTTP 请求指标
http_server_requests_seconds_count{method="GET",status="200",uri="/api/orders/{id}",application="order-service"} 1543
http_server_requests_seconds_sum{method="GET",status="200",uri="/api/orders/{id}",application="order-service"} 12.345
http_server_requests_seconds_max{method="GET",status="200",uri="/api/orders/{id}",application="order-service"} 0.234

零代码，JVM 和 HTTP 指标全自动采集。

三、自动采集的指标详解

3.1 JVM 指标

3.2 HTTP 指标

Spring Boot 自动为每个 HTTP 请求采集：

标签：method（GET/POST）、status（200/500）、uri（接口路径）、exception（异常类型）

3.3 自定义标签

给特定接口加标签：

@Component
public class HttpRequestTagContributor implements WebMvcTagsContributor {

    @Override
    public void contribute(HandlerMethod handler, HttpServletRequest request,
                           HttpServletResponse response, Throwable exception,
                           Map<String, String> tags) {
        // 加业务模块标签
        String module = extractModule(request);
        tags.put("module", module);

        // 加用户类型标签
        String userType = request.getHeader("X-User-Type");
        if (userType != null) {
            tags.put("user_type", userType);
        }
    }
}

四、自定义业务指标

4.1 四种指标类型

4.2 Counter（计数器）

@Service
public class OrderService {

    private final Counter orderCreateCounter;
    private final Counter orderFailCounter;

    public OrderService(MeterRegistry meterRegistry) {
        // 订单创建计数器
        this.orderCreateCounter = Counter.builder("order.create.count")
                .description("订单创建总数")
                .tag("type", "total")
                .register(meterRegistry);

        // 订单创建失败计数器
        this.orderFailCounter = Counter.builder("order.create.count")
                .description("订单创建失败数")
                .tag("type", "failed")
                .register(meterRegistry);
    }

    public Order createOrder(CreateOrderRequest request) {
        try {
            Order order = doCreateOrder(request);
            orderCreateCounter.increment();  // 计数 +1
            return order;
        } catch (Exception e) {
            orderFailCounter.increment();   // 失败计数 +1
            throw e;
        }
    }
}

Prometheus 中查看：

order_create_count_total{type="total",application="order-service"} 1543
order_create_count_total{type="failed",application="order-service"} 12

PromQL 计算成功率：

# 订单创建成功率
1 - (order_create_count_total{type="failed"} / order_create_count_total{type="total"})

4.3 Gauge（瞬时值）

@Service
public class InventoryService {

    private final AtomicInteger activeUsers = new AtomicInteger(0);

    public InventoryService(MeterRegistry meterRegistry) {
        // 方式一：注册 Gauge（自动采集）
        Gauge.builder("inventory.active.users", activeUsers, AtomicInteger::doubleValue)
                .description("当前活跃用户数")
                .register(meterRegistry);

        // 方式二：直接采集集合大小
        Gauge.builder("inventory.queue.size", this::getQueueSize)
                .description("库存处理队列长度")
                .register(meterRegistry);
    }

    public void userLogin() {
        activeUsers.incrementAndGet();
    }

    public void userLogout() {
        activeUsers.decrementAndGet();
    }

    private int getQueueSize() {
        return processingQueue.size();
    }
}

4.4 Timer（耗时统计）

@Service
public class PaymentService {

    private final Timer paymentTimer;

    public PaymentService(MeterRegistry meterRegistry) {
        this.paymentTimer = Timer.builder("payment.process.duration")
                .description("支付处理耗时")
                .publishPercentiles(0.5, 0.95, 0.99)  // 发布 P50/P95/P99
                .publishPercentileHistogram()          // 发布直方图
                .register(meterRegistry);
    }

    public PaymentResult processPayment(Order order) {
        // 方式一：手动记录
        long start = System.nanoTime();
        try {
            return doPayment(order);
        } finally {
            paymentTimer.record(System.nanoTime() - start, TimeUnit.NANOSECONDS);
        }
    }

    // 方式二：用 Timer.Sample（更精确）
    public PaymentResult processPaymentV2(Order order) {
        Timer.Sample sample = Timer.start();
        try {
            return doPayment(order);
        } finally {
            sample.stop(Timer.builder("payment.process.duration")
                    .tag("method", "sample")
                    .register(meterRegistry));
        }
    }
}

Prometheus 中查看：

payment_process_duration_seconds_count{application="order-service"} 543
payment_process_duration_seconds_sum{application="order-service"} 12.34
payment_process_duration_seconds_max{application="order-service"} 0.45
payment_process_duration_seconds{quantile="0.5",application="order-service"} 0.012
payment_process_duration_seconds{quantile="0.95",application="order-service"} 0.089
payment_process_duration_seconds{quantile="0.99",application="order-service"} 0.234

4.5 @Timed 注解（声明式）

更简洁的方式，加注解即可：

@RestController
@Timed("order.api")  // 类级别：所有接口都采集
public class OrderController {

    @GetMapping("/orders/{id}")
    @Timed(value = "order.api.get", description = "查询订单耗时", percentiles = {0.5, 0.95, 0.99})
    public Order getOrder(@PathVariable Long id) {
        return orderService.getOrder(id);
    }

    @PostMapping("/orders")
    @Timed(value = "order.api.create", description = "创建订单耗时")
    public Order createOrder(@RequestBody CreateOrderRequest request) {
        return orderService.createOrder(request);
    }
}

需要配置注册 TimedAspect Bean：

@Configuration
public class MetricsConfig {

    @Bean
    public TimedAspect timedAspect(MeterRegistry registry) {
        return new TimedAspect(registry);
    }
}

4.6 DistributionSummary（分布统计）

@Service
public class OrderService {

    private final DistributionSummary amountSummary;

    public OrderService(MeterRegistry meterRegistry) {
        this.amountSummary = DistributionSummary.builder("order.amount.distribution")
                .description("订单金额分布")
                .baseUnit("yuan")
                .publishPercentiles(0.5, 0.95, 0.99)
                .register(meterRegistry);
    }

    public Order createOrder(BigDecimal amount) {
        amountSummary.record(amount.doubleValue());  // 记录订单金额
        // ...
    }
}

查看金额分布：

order_amount_distribution_yuan{quantile="0.5"} 99.0    # 50% 订单金额 ≤ 99 元
order_amount_distribution_yuan{quantile="0.95"} 899.0  # 95% 订单金额 ≤ 899 元
order_amount_distribution_yuan{quantile="0.99"} 2999.0 # 99% 订单金额 ≤ 2999 元

五、Prometheus 抓取配置

5.1 prometheus.yml

global:
  scrape_interval: 15s        # 抓取间隔
  evaluation_interval: 15s    # 规则评估间隔

# 告警规则文件
rule_files:
  - "rules/*.yml"

# AlertManager 配置
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

# 抓取目标
scrape_configs:
  # Spring Boot 应用
  - job_name: 'spring-boot-app'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 15s
    static_configs:
      - targets:
          - 'order-service:8080'
          - 'user-service:8081'
          - 'payment-service:8082'
        labels:
          group: 'production'

  # Prometheus 自身
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

5.2 服务发现（K8s 场景）

scrape_configs:
  - job_name: 'k8s-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # 只抓取有 prometheus.io/scrape 注解的 Pod
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # 使用注解中的 path
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      # 使用注解中的 port
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

K8s 中给 Pod 加注解：

apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/actuator/prometheus"

六、Grafana 仪表盘

6.1 数据源配置

Grafana → Configuration → Data Sources → Add Prometheus：

URL: http://prometheus:9090
Access: Server

6.2 JVM 监控面板

堆内存面板：

# 已用堆内存（按区域分）
jvm_memory_used_bytes{area="heap", instance="$instance"}

# 最大堆内存
jvm_memory_max_bytes{area="heap", instance="$instance"}

GC 频率面板：

# 每分钟 GC 次数（按 GC 类型分）
rate(jvm_gc_pause_seconds_count[1m]) * 60

# 每分钟 GC 耗时
rate(jvm_gc_pause_seconds_sum[1m]) * 60

6.3 业务 QPS 面板

QPS 面板（按接口分组）：

sum by(uri) (
  rate(http_server_requests_seconds_count{application="order-service"}[1m])
)

P99 响应时间面板：

histogram_quantile(0.99,
  sum by(uri) (
    rate(http_server_requests_seconds_bucket{application="order-service"}[5m])
  )
)

错误率面板：

sum(rate(http_server_requests_seconds_count{status=~"5.."}[1m]))
/
sum(rate(http_server_requests_seconds_count[1m]))

6.4 完整仪表盘 JSON

仪表盘 JSON 见示例工程 grafana/dashboards/order-service-dashboard.json，导入方式：

Grafana → Dashboards → New → Import → 上传 JSON 文件

七、AlertManager 告警

7.1 告警规则

# rules/alerts.yml
groups:
  - name: order-service-alerts
    rules:
      # 接口 P99 > 1 秒
      - alert: HighP99Latency
        expr: |
          histogram_quantile(0.99,
            sum by(uri) (
              rate(http_server_requests_seconds_bucket{application="order-service"}[5m])
            )
          ) > 1
        for: 2m  # 持续 2 分钟才告警
        labels:
          severity: warning
          team: order
        annotations:
          summary: "接口 P99 延迟过高"
          description: "接口 {{ $labels.uri }} 的 P99 延迟为 {{ $value }}s，超过 1 秒阈值"

      # 错误率 > 5%
      - alert: HighErrorRate
        expr: |
          sum(rate(http_server_requests_seconds_count{application="order-service", status=~"5.."}[5m]))
          /
          sum(rate(http_server_requests_seconds_count{application="order-service"}[5m]))
          > 0.05
        for: 1m
        labels:
          severity: critical
          team: order
        annotations:
          summary: "接口错误率过高"
          description: "错误率 {{ $value | humanizePercentage }}，超过 5% 阈值"

      # JVM 堆内存使用率 > 85%
      - alert: HighHeapUsage
        expr: |
          jvm_memory_used_bytes{area="heap", application="order-service"}
          /
          jvm_memory_max_bytes{area="heap", application="order-service"}
          > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "JVM 堆内存使用率过高"
          description: "堆内存使用率 {{ $value | humanizePercentage }}"

      # Full GC 频繁
      - alert: FrequentFullGC
        expr: |
          rate(jvm_gc_pause_seconds_count{action="end of major GC", application="order-service"}[5m]) > 0.5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Full GC 过于频繁"
          description: "Full GC 频率 {{ $value }} 次/秒"

      # 实例宕机
      - alert: ServiceDown
        expr: up{job="spring-boot-app"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "服务实例宕机"
          description: "实例 {{ $labels.instance }} 已离线超过 1 分钟"

7.2 AlertManager 配置

# alertmanager.yml
global:
  resolve_timeout: 5m

# 告警分组
route:
  group_by: ['alertname', 'team']
  group_wait: 10s        # 首次告警等待时间
  group_interval: 10s    # 同组告警间隔
  repeat_interval: 1h    # 重复告警间隔
  receiver: 'dingtalk'   # 默认接收者

  routes:
    # 严重告警 → 钉钉 + 电话
    - match:
        severity: critical
      receiver: 'dingtalk-critical'
      group_wait: 0s

    # 警告级别 → 钉钉
    - match:
        severity: warning
      receiver: 'dingtalk'

receivers:
  # 普通钉钉通知
  - name: 'dingtalk'
    webhook_configs:
      - url: 'https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN'
        send_resolved: true

  # 严重告警钉钉通知
  - name: 'dingtalk-critical'
    webhook_configs:
      - url: 'https://oapi.dingtalk.com/robot/send?access_token=YOUR_CRITICAL_TOKEN'
        send_resolved: true

# 告警抑制规则
inhibit_rules:
  # 如果服务宕机，抑制该服务的其他告警
  - source_match:
      alertname: ServiceDown
    target_match_re:
      alertname: '.*'
    equal: ['instance']

7.3 钉钉通知模板

AlertManager 原生不支持钉钉格式，需要用钉钉机器人 webhook 转发。可用 prometheus-webhook-dingtalk：

# docker-compose.yml 中添加
  dingtalk:
    image: timonwong/prometheus-webhook-dingtalk
    container_name: dingtalk
    ports:
      - "8060:8060"
    command:
      - '--ding.profile=dingtalk=https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN'

AlertManager 配置改为：

receivers:
  - name: 'dingtalk'
    webhook_configs:
      - url: 'http://dingtalk:8060/dingtalk/dingtalk/send'
        send_resolved: true

收到钉钉消息效果：

[告警] 接口 P99 延迟过高
状态：FIRING
严重程度：warning
描述：接口 /api/orders/{id} 的 P99 延迟为 1.23s，超过 1 秒阈值
开始时间：2026-08-01 14:30:15

八、Docker Compose 一键部署

# docker-compose.yml
version: '3.8'

services:
  # Spring Boot 应用
  order-service:
    build: .
    container_name: order-service
    ports:
      - "8080:8080"
    environment:
      - MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE=prometheus,health,info

  # Prometheus
  prometheus:
    image: prom/prometheus:v2.54.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/rules:/etc/prometheus/rules
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=30d'

  # AlertManager
  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml

  # 钉钉 Webhook 转发
  dingtalk:
    image: timonwong/prometheus-webhook-dingtalk:v2.1.0
    container_name: dingtalk
    ports:
      - "8060:8060"
    command:
      - '--ding.profile=dingtalk=https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN'

  # Grafana
  grafana:
    image: grafana/grafana:11.2.0
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
      - ./grafana/datasources:/etc/grafana/provisioning/datasources

volumes:
  prometheus_data:
  grafana_data:

九、实战场景：监控订单接口

9.1 需求

监控订单接口的：

1. QPS（每秒请求数）
2. P99 响应时间
3. 错误率
4. 订单创建量
5. 支付处理耗时

9.2 完整指标定义

@Configuration
public class OrderMetricsConfig {

    @Bean
    public Counter orderCreateCounter(MeterRegistry registry) {
        return Counter.builder("order.create.count")
                .description("订单创建总数")
                .tag("type", "total")
                .register(registry);
    }

    @Bean
    public Counter orderFailCounter(MeterRegistry registry) {
        return Counter.builder("order.create.count")
                .description("订单创建失败数")
                .tag("type", "failed")
                .register(registry);
    }

    @Bean
    public Timer paymentTimer(MeterRegistry registry) {
        return Timer.builder("payment.process.duration")
                .description("支付处理耗时")
                .publishPercentiles(0.5, 0.95, 0.99)
                .publishPercentileHistogram()
                .register(registry);
    }

    @Bean
    public DistributionSummary orderAmountSummary(MeterRegistry registry) {
        return DistributionSummary.builder("order.amount.distribution")
                .description("订单金额分布")
                .baseUnit("yuan")
                .publishPercentiles(0.5, 0.95, 0.99)
                .register(registry);
    }
}

9.3 Grafana 面板配置

面板 1：QPS（按接口）

sum by(uri) (
  rate(http_server_requests_seconds_count{application="order-service"}[1m])
)

图表类型：Time Series，Legend {{uri}}

面板 2：P99 响应时间

histogram_quantile(0.99,
  sum by(uri) (
    rate(http_server_requests_seconds_bucket{application="order-service"}[5m])
  )
)

图表类型：Time Series，单位 ms

面板 3：错误率

sum(rate(http_server_requests_seconds_count{application="order-service", status=~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_count{application="order-service"}[5m]))

图表类型：Stat，阈值 5% 变红

面板 4：订单创建 QPS

rate(order_create_count_total{type="total", application="order-service"}[1m])

面板 5：订单成功率

1 - (
  rate(order_create_count_total{type="failed", application="order-service"}[1m])
  /
  rate(order_create_count_total{type="total", application="order-service"}[1m])
)

面板 6：支付 P99 耗时

histogram_quantile(0.99,
  rate(payment_process_duration_seconds_bucket{application="order-service"}[5m])
)

十、最佳实践

10.1 指标命名规范

格式：<subsystem>_<metric_name>_<unit>

示例：
  ✅ order_create_count_total      订单创建计数
  ✅ payment_process_duration_seconds  支付耗时（秒）
  ✅ inventory_queue_size          库存队列大小
  ✅ jvm_memory_used_bytes         JVM 内存使用（字节）

  ❌ orderCount                    缺少单位
  ❌ payment_time                   不明确
  ❌ metric1                        无意义

10.2 标签使用原则

// ✅ 标签值是有限的枚举
Counter.builder("order.create.count")
    .tag("status", "success")  // success/failed
    .tag("type", "normal")     // normal/vip

// ❌ 标签值是无限的用户 ID
Counter.builder("order.create.count")
    .tag("userId", userId)     // 千万级用户 → 标签爆炸

原则：标签基数（cardinality）控制在 100 以内。

10.3 高基数标签的危害

# 每个用户一个标签 → 指标爆炸
http_requests_total{userId="user001"} 1
http_requests_total{userId="user002"} 1
...
http_requests_total{userId="user999999"} 1

# 100 万用户 → 100 万时间序列 → Prometheus OOM

10.4 性能影响

十一、总结

指标体系分层

Level 1：基础指标（开箱即用）
  → JVM 指标、HTTP 指标、进程指标

Level 2：业务指标（自定义）
  → Counter（订单数）+ Timer（耗时）+ Gauge（状态）

Level 3：可视化（Grafana）
  → QPS 面板 + P99 面板 + 错误率面板 + JVM 面板

Level 4：告警（AlertManager）
  → P99 > 1s 告警 + 错误率 > 5% 告警 + 宕机告警

指标类型选择

排障工具对照

互动话题：你的团队用什么做监控？有没有遇到过"线上挂了但监控没告警"的情况？欢迎留言讨论！

参考资料

• Micrometer 官方文档
• Prometheus 最佳实践
• Grafana 仪表盘市场
• Spring Boot Actuator 指标