Loki's own logs are stored with the "unit" labels instead of "juju_unit"

[sed-i]

Bug Description

The set of labels ingested by loki includes both "unit" and "juju_unit". Loki logs have "unit", related apps have "juju_unit".

To Reproduce

Deploy this bundle:

bundle: kubernetes
applications:
  flog:
    charm: flog-k8s
    channel: latest/edge
    revision: 8
    base: [email protected]/stable
    resources:
      workload-image: 2
    scale: 1
  ga:
    charm: grafana-agent-k8s
    channel: latest/edge
    revision: 80
    resources:
      agent-image: 43
    scale: 1
  loki:
    charm: loki-k8s
    channel: latest/edge
    revision: 158
    base: [email protected]/stable
    resources:
      loki-image: 97
      node-exporter-image: 2
    scale: 1
    trust: true
relations:
- - ga:logging-consumer
  - loki:logging
- - ga:logging-provider
  - flog:log-proxy

Environment

Model  Controller  Cloud/Region        Version  SLA          Timestamp
pg     j34         microk8s/localhost  3.4.2    unsupported  19:49:07-04:00

App   Version  Status  Scale  Charm              Channel      Rev  Address         Exposed  Message
flog           active      1  flog-k8s           latest/edge    8  10.152.183.33   no       
ga    0.40.4   active      1  grafana-agent-k8s  latest/edge   80  10.152.183.118  no       grafana-dashboards-provider: off
loki  2.9.6    active      1  loki-k8s           latest/edge  158  10.152.183.178  no       

Relevant log output

➜ curl 10.1.166.74:3100/loki/api/v1/labels
{"status":"success","data":["application","charm_name","container","filename","job","juju_application","juju_charm","juju_hook_name","juju_model","juju_model_uuid","juju_unit","logger","model","model_uuid","service_name","severity","unit"]}
➜ curl 10.1.166.74:3100/loki/api/v1/label/unit/values      
{"status":"success","data":["loki/0"]}
➜ curl 10.1.166.74:3100/loki/api/v1/label/juju_unit/values
{"status":"success","data":["flog/0"]}
➜ curl -sG 10.1.166.74:3100/loki/api/v1/query_range --data-urlencode 'query={unit="loki/0"}' | jq '.data.result[0]'
{
  "stream": {
    "application": "loki",
    "charm_name": "loki-k8s",
    "juju_hook_name": "config-changed",
    "logger": "__main__",
    "model": "pg",
    "model_uuid": "31887e0e-4c62-48df-85db-7f529bc25177",
    "service_name": "loki",
    "severity": "debug",
    "unit": "loki/0"
  },
  "values": [
    [
      "1718840174329739008",
      "No alertmanagers available"
    ]
  ]
}

Additional context

No response