kubelet为cadvisor添加namespace/pod/container标签示例详解
作者:a朋
kubelet内嵌了cadvisor
在kubernetes中,kubelet中内嵌了cadvisor,prometheus对cadvisor指标的拉取时,使用的是kubelet的endpoints,只是metric_path改为/metrics/cadvisor。
kubelet在集成cadvisor的同时,还其添加了namespace、pod、container标签,用于定位它在kubernetes资源中的具体位置。
container_spec_cpu_period{ container="prometheus", endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304f6c86_d20b_45aa_95ad_db5edc2e8b43.slice/cri-containerd-cbde185ec5c5d6ff6928b10e9dc21de7548aec0e0bcbdff25a9ebf80a9fd3adf.scope", image=“prometheus:latest", instance=“192.168.0.1:10250", job="kubelet", metrics_path="/metrics/cadvisor", name="cbde185ec5c5d6ff6928b10e9dc21de7548aec0e0bcbdff25a9ebf80a9fd3adf", namespace="monitoring", node=“node01", pod="prometheus-k8s-0", service="kubelet"} 100000
那么kubelet是如何为cadvisor添加这些标签的,cadvisor中有没有采集这些信息呢?
一.cadvisor的指标标签
为了验证cadvisor自身的采集能力,将cadvisor单独部署到kubernetes,配置其拉取的job=cadvisor,拉取到的指标如下:
container_cpu_usage_seconds_total{ container_env_HOSTNAME="prometheus-k8s-0”, container_env_KUBERNETES_PORT="tcp://10.233.0.1:443”, container_env_KUBERNETES_PORT_443_TCP="tcp://10.233.0.1:443”, container_env_KUBERNETES_PORT_443_TCP_ADDR="10.233.0.1”, container_env_KUBERNETES_PORT_443_TCP_PORT="443”, container_env_KUBERNETES_PORT_443_TCP_PROTO="tcp”, container_env_KUBERNETES_SERVICE_HOST="10.233.0.1”, … container_label_io_cri_containerd_kind="container”, container_label_io_kubernetes_container_name="prometheus”, // 容器名称 container_label_io_kubernetes_pod_name="prometheus-k8s-0”, // pod名称 container_label_io_kubernetes_pod_namespace="monitoring”, // namespace container_label_io_kubernetes_pod_uid="304f6c86-d20b-45aa-95ad-db5edc2e8b43”, container_label_maintainer="The Prometheus Authors <prometheus-developers@googlegroups.com>”, ... cpu="cpu00”, endpoint="http”, id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304f6c86_d20b_45aa_95ad_db5edc2e8b43.slice/cri-containerd-cbde185ec5c5d6ff6928b10e9dc21de7548aec0e0bcbdff25a9ebf80a9fd3adf.scope”, image=“prometheus:latest”, instance="10.233.123.70:8080”, job="cadvisor”, name="cbde185ec5c5d6ff6928b10e9dc21de7548aec0e0bcbdff25a9ebf80a9fd3adf”, service="cadvisor"} 709.899183253
可以看到拉取的指标中的label:
container_env_*:
- cadvisor采集了容器内的环境变量信息,并将其添加到指标的label中;
container_label_io_*:
- cadvisor采集了容器的label信息,并将其添加到指标的label中;
- 这些label信息包含了容器的container名称、所在的pod、所在的namespace信息;
也就是说,cadvisor可以采集到容器的container/pod/namespace信息。
那这些信息是从何而来的呢,看cadvisor的代码:
- 对cadvisor采集的指标,使用containerLabelsFunc函数添加额外的标签;
- containerLabelsFunc默认=DefaultContainerLabels;
// cadvisor/metrics/prometheus.go func NewPrometheusCollector(i infoProvider, f ContainerLabelsFunc, includedMetrics container.MetricSet, now clock.Clock, opts v2.RequestOptions) *PrometheusCollector { if f == nil { f = DefaultContainerLabels // 默认的添加标签的函数 } c := &PrometheusCollector{ infoProvider: i, containerLabelsFunc: f, ... } if includedMetrics.Has(container.CpuUsageMetrics) { c.containerMetrics = append(c.containerMetrics, []containerMetric{ { name: "container_cpu_user_seconds_total", help: "Cumulative user cpu time consumed in seconds.", valueType: prometheus.CounterValue, getValues: func(s *info.ContainerStats) metricValues { return metricValues{ { value: float64(s.Cpu.Usage.User) / float64(time.Second), timestamp: s.Timestamp, }, } }, }, ... } return c }
看一下DefaultContainerLabels函数的实现:
- 对container.spec中的label,添加了container_label_*前缀,这些label将被添加到最终的指标标签中;
- 对container.spec中的env,添加了container_label_env_*前缀,这些label将被添加到最终的指标标签中;
// cadvisor/metrics/prometheus.go const ( // ContainerLabelPrefix is the prefix added to all container labels. ContainerLabelPrefix = "container_label_" // ContainerEnvPrefix is the prefix added to all env variable labels. ContainerEnvPrefix = "container_env_" // LabelID is the name of the id label. LabelID = "id" // LabelName is the name of the name label. LabelName = "name" // LabelImage is the name of the image label. LabelImage = "image" ) func DefaultContainerLabels(container *info.ContainerInfo) map[string]string { set := map[string]string{LabelID: container.Name} if len(container.Aliases) > 0 { set[LabelName] = container.Aliases[0] } if image := container.Spec.Image; len(image) > 0 { set[LabelImage] = image } for k, v := range container.Spec.Labels { set[ContainerLabelPrefix+k] = v // container_label_* } for k, v := range container.Spec.Envs { set[ContainerEnvPrefix+k] = v // container_env_* } return set }
最后,再看一下containerInfo的spec中的label和env是什么样子:
"labels":{ "io.kubernetes.container.name":"node-exporter", "io.kubernetes.pod.name":"node-exporter-gw4qg", "io.kubernetes.pod.namespace":"monitoring", "io.kubernetes.pod.uid":"2ac69f2e-cb3c-4e4f-b890-183e457ef891" },
"env": [ "HOSTNAME=node01", "KUBERNETES_SERVICE_PORT=443", "KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443", "KUBERNETES_SERVICE_HOST=10.233.0.1", "KUBERNETES_PORT_443_TCP_PORT=443", "KUBERNETES_SERVICE_PORT_HTTPS=443", "KUBERNETES_PORT=tcp://10.233.0.1:443", "KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1", ... ],
cadvisor对原始的label和env添加了前缀,返回给prometheus:
对label:
- 原始的label=“io.kubernetes.container.name”,加上前缀,变成:container_label_io_kubernetes_container_name;
对env:
- 原始的env=HOSTNAME,加上前缀,变成:container_env_HOSTNAME;
二. kubelet集成cadvisor的指标标签
kubelet在集成cadvisor时,自定义了为指标额外添加标签的逻辑containerLabelsFunc,没有使用DefaultContainerLabels。
kubelet传入的函数=containerPrometheusLabelsFunc:
// kubernetes/pkg/kubelet/server/server.go r.RawMustRegister(metrics.NewPrometheusCollector(prometheusHostAdapter{s.host}, containerPrometheusLabelsFunc(s.host), includedMetrics, clock.RealClock{}, cadvisorOpts))
containerPrometheusLabelsFunc函数的代码实现如下:
- 从containerInfo.Spec.label中,提取io.kubernetes.pod.name的值,作为pod名称的label-value;
- 从containerInfo.Spec.label中,提取io.kubernetes.pod.namespace的值,作为namespace名称的label-value;
- 从containerInfo.Spec.label中,提取io.kubernetes.container.name的值,作为container名称的label-value;
// kubernetes/pkg/kubelet/server/server.go func containerPrometheusLabelsFunc(s stats.Provider) metrics.ContainerLabelsFunc { // containerPrometheusLabels maps cAdvisor labels to prometheus labels. return func(c *cadvisorapi.ContainerInfo) map[string]string { // Prometheus requires that all metrics in the same family have the same labels, // so we arrange to supply blank strings for missing labels var name, image, podName, namespace, containerName string if len(c.Aliases) > 0 { name = c.Aliases[0] } image = c.Spec.Image if v, ok := c.Spec.Labels[kubelettypes.KubernetesPodNameLabel]; ok { // io.kubernetes.pod.name podName = v } if v, ok := c.Spec.Labels[kubelettypes.KubernetesPodNamespaceLabel]; ok { // io.kubernetes.pod.namespace namespace = v } if v, ok := c.Spec.Labels[kubelettypes.KubernetesContainerNameLabel]; ok { // io.kubernetes.container.name containerName = v } // Associate pod cgroup with pod so we have an accurate accounting of sandbox if podName == "" && namespace == "" { if pod, found := s.GetPodByCgroupfs(c.Name); found { podName = pod.Name namespace = pod.Namespace } } set := map[string]string{ metrics.LabelID: c.Name, metrics.LabelName: name, metrics.LabelImage: image, "pod": podName, // pod名称 "namespace": namespace, // namespace "container": containerName, // container名称 } return set } }
三. 总结
cadvisor通过容器的接口,查询到spec.labels的信息(包含container/pod/namespace)后,
对于普通的cadvisor:
- 使用默认的DefaultContainerLabels函数添加标签,为label添加container_label_前缀;
对于kubelet集成的cadvisor:
- 使用自定义的containerPrometheusLablesFunc函数添加标签,将原始的标签变为container=/pod=/namespace=;
以上就是kubelet为cadvisor添加namespace/pod/container标签示例详解的详细内容,更多关于kubelet cadvisor标签添加的资料请关注脚本之家其它相关文章!