problem

If you're templating your Grafana Alloy config file, you'll want to use *_file variables where possible to avoid the need to store sensitive variables and secrets as plain text in your source code repository.

Before you can use these variables, you'll need to store them somewhere secure, we'll be using GCP secrets manager for this example.

shh... secrets 🤫

Here's a quick way to get a secret value into GCP Secrets Manager. Make sure you wipe your history after you're done! history -c

printf '%s' "<secret-value>" | gcloud secrets create "prometheus-password" --project="$PROJECT_ID" --data-file=-

Or just the secret container in Terraform, and put the value in with gcloud or Console so the password isn't sitting in state:

resource "google_secret_manager_secret" "prometheus_password" {
  secret_id = "prometheus-password"
  replication {
    auto {}
  }
}

templating with _file variables

Using the *_file variables, we can now template our config file with the secret. config.alloy.j2

prometheus.remote_write "metrics" {
  endpoint {
    name = "remote"
    url  = "{{ prometheus_url }}"

    basic_auth {
      username      = "{{ prometheus_username }}"
      password_file = "{{ prometheus_password_file }}"
    }
  }
}

getting the secret on the box

This is a simple example of how to get a secret from GCP Secrets Manager and write it to a file on the box.

- name: Pull password from Secret Manager
  ansible.builtin.set_fact:
    prometheus_password: "{{ lookup('pipe', 'gcloud secrets versions access latest --secret=' ~ prometheus-password ~ ' --project=' ~ project_id) }}"
  no_log: true

- name: Refuse to write an empty secret
  ansible.builtin.assert:
    that: prometheus_password | length > 0

- name: Write password file
  ansible.builtin.copy:
    content: "{{ prometheus_password }}"
    dest: "{{ prometheus_password_file }}"
    owner: alloy
    group: alloy
    mode: "0400"
  become: true
  no_log: true

aaaand config!

Let's deploy the config to the box, this will use the *_file variables we created earlier to pull the secret from the file.

- name: Deploy Alloy configuration
  ansible.builtin.template:
    src: config.alloy.j2
    dest: /etc/alloy/config.alloy
    owner: alloy
    group: alloy
    mode: "0640"
  become: true
  no_log: true
  diff: false