Prometheus custom Alert annotations based on the pod

So we have a K8s cluster, where we run Pods with sort of the following spec:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/path: /metrics
    prometheus.io/port: "9115"
    prometheus.io/scrape: "true"
    custom.io/description: ToDO
    custom.io/logs: https://kibana.com
    custom.io/monitoring: https://grafana.com
    custom.io/owner: '@dev1'
    custom.io/runbook: https://github.com/
  labels:
    app: my-app
    environment: dev
    role: api
    tier: backend
  name: my-app
  namespace: dev
spec:
  containers:
  ...

We try to add as much information in the annotations to have it sort of centralized per app, so each developer can update those values accordingly and so it can be useful when someone is debugging issues. We had the idea that it would be useful if this information is added in the Alerts we get in Slack from AlertManager so it's even more easier to debug when a problem arises.

We considered the option of passing those annotations as metric labels, but it seemed like not the correct place to store that information. So we dropped that option.

Ideally, we would like to use these values (the custom.io annotations) when available in the alert, something like:

- alert: RestartsCountWarning
annotations:
  description: Warning - Pod has restarted at least 3 times
  summary: Pod restarting (>3)
  runbook: {{ $annotations.runbook }}
  logs: {{ $annotations.logs }}
  monitoring: {{ $annotations.monitoring }}
  owner: {{ $annotations.owner }}
expr: kube_pod_container_status_restarts_total{job="kube-state-metrics",namespace=~".*"} > 3
for: 5m
labels:
  severity: warning

Where $annotations is the hypothetical magic variable containing the info we need. So in the final alert sent to Slack we can show those values as {{ .Annotations.logs }} for example.

Among the many things we have tried, we tried passing those annotations to the target labels by using the conf:

relabel_configs:
- action: labelmap
  regex: __meta_kubernetes_pod_annotation_custom_io_(.+)

But haven't found how to use them from there.

So we are kind of stuck here, any ideas on how to achieve this? Or has anyone has been able to achieve this or something similar either using Prometheus or another tool? Or maybe we should be doing this with a totally different tool, but which one? Or any recommendations? Thanks a lot in advance.