Flexible Log Retention in GKE

Application logs that require varying retention periods can be easily managed by assigning labels to Pods.

Terraform

Complete setup can be seen in the GKE-Infra project.

Terraform variables:

project_id - Google Cloud project ID.
logs_retention_bylabel_buckets - comma separated list of numbers, each number defines log-retention bucket (in days). The local.logs_retention_bylabel_buckets is a result of some post-processing.

Let’s create all the buckets:

resource "google_logging_project_bucket_config" "pod_label" {
  for_each = toset(local.logs_retention_bylabel_buckets)

  project        = var.project_id
  location       = "global"
  retention_days = each.value
  bucket_id      = "keep-logs-${each.value}-days"
}

Then all the accompanying sinks:

resource "google_logging_project_sink" "pod_label" {
  for_each = toset(local.logs_retention_bylabel_buckets)

  name                   = "keep-logs-${each.value}-days"
  destination            = "logging.googleapis.com/${google_logging_project_bucket_config.pod_label[each.value].id}"
  # `rkways.com` from Pod label changes into `rkways_com`
  filter                 = "resource.type = k8s_container labels.\"k8s-pod/rkways_com/gke-infra-logdays\" = \"${each.value}\" "
  unique_writer_identity = true
}

Please notice that the label in filter field is slightly changed: while Pod should be labelled with rkways.com/gke-infra-logdays – the matching is against k8s-pod/rkways_com/gke-infra-logdays (prefix and underscore).

The last part is an exclusion rule to avoid data duplication in the _Default bucket:

resource "google_logging_project_exclusion" "pod_label" {
  for_each = toset(local.logs_retention_bylabel_buckets)

  name        = "keep-logs-${each.value}-days"
  description = "Exclude Pod-labelled logs. Stored elsewhere."
  filter      = "resource.type = k8s_container labels.\"k8s-pod/rkways_com/gke-infra-logdays\" = \"${each.value}\" "
}

GKE

The YAML below creates a Pod whose logs are kept for a week - assuming the number 7 was in logs_retention_bylabel_buckets list.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: date10s
  namespace: default
spec:
  selector:
    matchLabels:
      app: date10s
  template:
    metadata:
      labels:
        app: date10s
        rkways.com/gke-infra-logdays: "7"
    spec:
      containers:
      - image: busybox
        name: busybox
        command: ['sh', '-c', 'while sleep 10; do date; done']

2021-04-21