KubeCrow reads metrics and logs from the backends already running in your clusters. There’s nothing to install and no port-forward to keep open: it finds them by their usual service names and labels and reaches them through the Kubernetes API server’s service proxy.
What it finds
- Metrics: any Prometheus-compatible API: Prometheus, Mimir, Thanos, VictoriaMetrics or Cortex. Gateways and query frontends are preferred, and Mimir’s
/prometheusand VictoriaMetrics’select/0/prometheusprefixes are tried. - Logs: Loki. The labels that hold namespace, pod and container are detected from Loki itself (Promtail, Alloy, Fluent Bit, Vector and OpenTelemetry naming), and can be changed in Settings → Observability.
Settings → Observability shows what was found for each cluster. If something isn’t found (it may just be restarting), KubeCrow checks again by itself (after two minutes, then less often) and says when.
Multi-tenant Mimir and Loki
A multi-tenant backend wants an X-Scope-OrgID tenant. When none is set, KubeCrow looks for the tenants your shippers send with in the configs of Alloy, Promtail, Fluent Bit, Vector, OpenTelemetry or Prometheus remote-write (tenant_id, X-Scope-OrgID), tries them, and shows the one in use and where it came from. Only tenant names are read from those configs. A tenant you type in Settings always wins.
What uses them
- Metrics tab for pods and workloads: CPU and memory against requests and limits, restarts, network and pod count over 1 hour to 7 days. For workloads, that covers every pod they’ve had, including replaced ones.
- Health: cluster CPU and memory over time.
- Right-sizing: requests and limits against real use.
- Logs → History and Search logs: lines from before a restart, from replaced or deleted pods, and one search across every pod in a namespace.
Without Prometheus
The Health chart still works: KubeCrow samples metrics-server every 30 seconds itself and keeps 24 hours per cluster on your Mac, so the history survives a restart. Longer ranges, per-pod history and right-sizing need Prometheus.
Permission needed
Reaching a backend through the API server needs get on services/proxy in the backend’s namespace. See permissions KubeCrow needs.
Questions
Do I need to port-forward Prometheus or Loki for KubeCrow?
No. KubeCrow reaches them through the Kubernetes API server’s service proxy, which needs get on services/proxy in their namespace.
Does KubeCrow work with Grafana Mimir and multi-tenant Loki?
Yes. It finds the tenant in your log and metrics shippers’ configs, or you can set it in Settings → Observability.