# Kubernetes Optimization
Learn how to effortlessly scale and right-size Kubernetes with granular control that ensures peak performance at the lowest possible cost
Kubernetes optimization tool - **PerfectScale by DoiT** empowers you to effortlessly fine-tune your environment, ensure optimal resource allocation, and achieve peak performance with comprehensive insights for the lowest possible cloud cost.
This documentation serves as your comprehensive guide to mastering the platform, ensuring you achieve maximum results efficiently.
{% hint style="success" %}
PerfectScale is compatible with and can be deployed in **any Kubernetes environment**, including **EKS/EKS Anywhere**, **GKE**, **AKS**, **KOPS**, **private cloud**, and other Kubernetes distributions.
{% endhint %}
## 🔑 [Starting a free account](https://app.perfectscale.io/account/sign-up?utm_source=referral\&utm_medium=ps-docs\&utm_campaign=open-docs\&utm_content=sign-up)
## Getting started with the K8s optimization tool
1. Deploy the **PerfectScale** stateless deployment into the cluster [with a single helm command](/getting-started/how-to-onboard-a-cluster).
2. Get a thorough analysis of risk and cost for the whole cluster within 5 minutes of onboarding.
3. Facilitate a productive collaboration by [inviting your team](/administration/user-management).
## Next to achieve
### [Cut your K8s cloud spending by up to 50%](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing)
Get a detailed multi-cloud, multi-cluster view of your K8s spending based on actual resource utilization, pinpoint problematic areas, prioritize the most impactful actions, and apply data-driven recommendations to reduce waste safely without affecting stability and performance.
### [Improve K8s performance](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing/understanding-at-risk-indicators)
Identify up to 30 different resilience types of risks, prioritize them by their impact on your services, and apply actionable recommendations to resolve them in just a few clicks.
### [Automate K8s optimization tasks](/enable-automation/introduction-to-automation)
Effortlessly and proactively optimize your Kubernetes environment by adjusting workloads' CPU and memory to achieve optimal values and ensure your environment is always perfectly tuned without manual intervention.
### [Get accurate cost visibility across your entire K8s stack](/visibility-and-optimization/trends-monitoring)
Analyze your K8s data across clusters, namespaces, and node groups to evaluate effectiveness, pinpoint anomalies, improve forecasting, and enable data-driven decisions based on long-term trends.
## Maximizing your optimization results
Create a tailored optimization ecosystem with diverse integration options to keep up with all your Kubernetes activities, streamline your optimization processes, and enhance your team's productivity.
1. [**Configure Alerts**](/customize-workflow/alerting/resiliency-alerts)**:** Receive real-time notifications about resilience risks or cost anomalies before they affect the performance or effectiveness of your environment.
2. [**Set Up Communication Channels**](/customize-workflow/communication-and-messaging): Manage alerts via Slack, MS Teams, or Datadog.
3. [**Connect Issue Management**](/customize-workflow/ticketing-and-bug-tracking): Integrate with Jira to handle issues efficiently within clusters.
4. [**Customize Pricing**](/customize-workflow/pricing): Tailor pricing to your needs with various pricing customization options
5. [**Integrate Observability**](/customize-workflow/observability): Enhance K8s visibility with tools like Grafana, Datadog, and more.
6. [**Set Up ArgoCD**](/getting-started/argocd-install-of-perfectscale-exporter): Connect PerfectScale recommendations to your standard GitOps flow.
## :handshake: Community
Join our [Сommunity](https://join.slack.com/t/perfectscalecommunity/shared_invite/zt-1tu9teu9e-Z9tGt4LpNI8tUC3j8obcmQ) to stay updated on what is trending in the atmosphere, our latest releases, upcoming events, technical discussions, and more.
## :woman\_superhero: Support
Feel free to ask any questions on Slack. You can also reach our support at .
# Overview
Get started with PerfectScale in minutes and optimize your Kubernetes environment effortlessly
PerfectScale can be deployed as a Managed SaaS or On-premise Self-hosted platform.
## Prerequisites
Before starting, there are some prerequisites you need to consider:
1. A Kubernetes cluster.
2. A fully configured [kubectl](https://kubernetes.io/docs/tasks/tools/) command-line interface with permissions on your local machine allows you to create a Namespace, ServiceAccount, ClusterRole, ClusterRoleBinding, Deployment, Service, and Secret.
3. Installed [Helm](https://helm.sh/docs/intro/install/) (minimal supported version **3.5**).
## How to start?
Getting started with PerfectScale is fast and seamless. Cluster onboarding takes just a few minutes, with flexible options to match your specific needs. Use one of the following guides to walk through the process and complete the setup smoothly:
:star: [Onboard a cluster using the UI](/getting-started/how-to-onboard-a-cluster) - simple, minimum-steps approach to start optimizing.
:robot: [Onboard clusters programmatically](/getting-started/onboarding-clusters-programmatically) - for those who want to provision clusters dynamically.
:woman\_singer: [Onboarding with ArgoCD](/getting-started/argocd-install-of-perfectscale-exporter) - for GitOps-driven deployments fans.
{% hint style="info" %}
The customers who have access to the DoiT Kubernetes Lens also have access to their PerfectScale tenant with an [**Admin role**](/administration/rbac-or-roles-and-permissions).
{% endhint %}
# How to onboard a cluster
Step-by-step guide on how to onboard your first cluster and start optimizing within a few minutes
{% hint style="success" %}
PerfectScale is compatible and can be deployed in any Kubernetes environment, including EKS/EKS Anywhere, GKE, AKS, KOPS, private cloud, and other Kubernetes distributions.
{% endhint %}
## Onboarding instructions
1. Login into the [PerfectScale](https://app.perfectscale.io/) app.
2. Once logged in, navigate to the **`Overview`** tab.
3. You can see a button to connect a new Kubernetes cluster. To connect a new cluster, click the **`Add Cluster`** button. A pop-up window with the following steps will appear.
Onboard first cluster
{% hint style="info" %}
To provision clusters dynamically, follow the instructions for [Onboarding clusters programmatically](/getting-started/onboarding-clusters-programmatically).
{% endhint %}
### PerfectScale Agent deployment
Onboarding PerfectScale to a cluster, starts with deploying the PerfectScale Agent.\
There are 2 deployment methods for doing so:
**Method 1: PerfectScale Operator**\
1\. Use helm to deploy the [PerfectScale Operator](/administration/perfectscale-operator).\
The full list of all possible [PerfectScale Operator helm (input) values can found here.](https://github.com/perfectscale-io/perfectscale-io.github.io/blob/main/charts/psc-agent-operator/values.yaml)\
2\. Use kubectl to deploy an Agents CR (Custom Resource of kind: Agents)\
3\. The PerfectScale Operator will then deploy the PerfectScale Agent based on the Agents CR.
**Method 2: Helm Deployment**\
1\. Use helm to deploy the PerfectScale Agent\
The full list of all possible [PerfectScale Agent helm (input) values can found here.](https://github.com/perfectscale-io/perfectscale-io.github.io/blob/main/charts/perfectscale-agent/values.yaml)
{% hint style="info" %}
We recommend you to install the [PerfectScale Operator](/administration/perfectscale-operator) to automate deployment and ensure your agents stay up to date with the latest features and product updates.
{% endhint %}
Connect cluster
{% hint style="info" %}
PerfectScale supports Windows nodes; however, the PerfectScale exporter itself must run on Linux nodes. In clusters that include both Windows and Linux nodes, you must configure a `nodeSelector` to ensure the exporter is scheduled only on Linux nodes. Please follow the [**additional instructions**](#mix-of-windows-and-linux-nodes) below to run the exporter on Linux nodes.
{% endhint %}
2. In a pop-up window, click **`Copy to Clipboard`** and clone the Helm chart provided on p.1.
3. Enter a name for your cluster and select the desired [**Optimization Policy**](/administration/cluster-settings#general-settings)**:**
* MaxSavings - maximum cost savings, the best for non-production environments
* Balanced (default) - optimally balances cost and resiliency
* ExtraHeadroom - the best fit for latency-sensitive environments
* MaxHeadroom - keeps the environment above the highest spikes
{% hint style="info" %}
The **Optimization Policy** feature allows you to specify how your resources should be allocated in order to support the individual needs of your workloads. Define the policies that best suit your environment and business goals, depending on whether you want to maximize cost savings or provide extra headroom to maintain the resilience of mission-critical services.\
\
The **Optimization Policy** can be set at a specific level. The workload's **Optimization Policy** takes precedence and will override the value defined at the cluster level.
Learn more about the optimization policy customization [here](/customize-workflow/optimization-policy-customization).
{% endhint %}
Discover more about customizing the Optimization policy [here](/customize-workflow/optimization-policy-customization).
#### Connecting cluster with PerfectScale Operator
Once you have named your cluster and selected the Optimization Policy, click **`Get Install Command`**, so PerfectScale will provide you with the needed installation components.
Connect cluster with PerfectScale Operator
Add Helm repo:
```
helm repo add perfectscale https://perfectscale-io.github.io --force-update
```
Deploy PerfectScale Operator:
```
helm upgrade --install psc-agent-operator perfectscale/psc-agent-operator \
-n perfectscale --create-namespace \
--set secret.create=true \
--set secret.clientId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \
--set secret.clientSecret=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \
--set clusterName={your-cluster-name}
```
Apply CRD:
{% hint style="info" %}
Learn more about PerfectScale Operator CRD [here](/administration/perfectscale-operator/understanding-perfectscale-operator-crd).
{% endhint %}
```yaml
kubectl apply -f - < [Mix of Windows and Linux nodes](#mix-of-windows-and-linux-nodes).
{% endhint %}
{% hint style="info" %}
PerfectScale supports Windows containers. To enable support for Windows containers, follow the instructions outlined here.
{% endhint %}
3. Click on the **`Finish and Close`** button.
{% hint style="info" %}
It is not supported to install multiple agents in a single cluster, as it may result in unexpected results.
{% endhint %}
The newly added cluster will appear under the **`Clusters`** list in the **`Overview`** tab. Once the [**`Agent Status`**](/visibility-and-optimization/clusters-metrics-overview#summary) becomes green, the cluster data will appear, which indicates successful cluster creation.
Agent status
:bulb: Discover additional information regarding the [**Overview**](/visibility-and-optimization/clusters-metrics-overview).
{% hint style="info" %}
Your cluster will become visible **only** after it starts transmitting data.
{% endhint %}
## Run PerfectScale agent on specific nodes
PerfectScale allows running the agent on particular nodes. To run the agent on the specific nodes, use one of the following options:
1. `nodeSelector` (in case there is no taint on the node).
2. `nodeSelector & toleration` (in case of taint on the node).
{% hint style="warning" %}
In case you are using **`zsh`** shell, make sure to add **`noglob`** before the helm command
{% endhint %}
```
helm upgrade --install -n perfectscale --create-namespace perfectscale \
--set secret.create=true \
--set secret.clientId=**** \
--set secret.clientSecret=**** \
--set settings.clusterName=name \
--set nodeSelector.type=monitoring \
--set tolerations[0].effect=NoSchedule,tolerations[0].key=monitoring,tolerations[0].operator=Exists \
--set kube-state-metrics.nodeSelector.type=monitoring \
--set kube-state-metrics.tolerations[0].effect=NoSchedule,kube-state-metrics.tolerations[0].key=monitoring,kube-state-metrics.tolerations[0].operator=Exists \
perfectscale/exporter
```
## Mix of Windows and Linux nodes
{% hint style="warning" %}
If the cluster contains both **Windows** and **Linux** nodes, it is necessary to add the following extra `nodeSelector` to run **PerfectScale** agent on the **Linux** nodes:\
\
\--set nodeSelector.beta\\\\.kubernetes\\\\.io/os=linux --set kube-state-metrics.nodeSelector.beta\\\\.kubernetes\\\\.io/os=linux
{% endhint %}
For Step 4 in the [**Cluster configuration**](#cluster-configuration) instructions above, utilize the subsequent command to designate the Windows nodes:
```
helm upgrade --install -n perfectscale --create-namespace perfectscale \
--set secret.create=true \
--set secret.clientId=**** \
--set secret.clientSecret=**** \
--set settings.clusterName=name \
--set nodeSelector.beta\\.kubernetes\\.io/os=linux \
--set kube-state-metrics.nodeSelector.beta\\.kubernetes\\.io/os=linux \
perfect scale/exporter
```
## GPU support
{% hint style="info" %}
**GPU** support is available starting with the **exporter version 1.0.55**.
**GPU memory** support is available starting with the **exporter version 1.1.11**. Learn more about requirements for GPU memory support [here](#required-dcgm-metrics-for-gpu-memory-support).
{% endhint %}
PerfectScale’s advanced GPU support feature helps teams optimize resource-intensive applications like AI, machine learning, cloud computing, etc., ensuring better performance, reducing costs, and improving overall efficiency across the entire Kubernetes stack.
{% hint style="info" %}
PerfectScale now only supports NVIDIA Data Center GPU Manager (DCGM).
{% endhint %}
In order to enable GPU support, in Step 4 in the [**Cluster configuration**](#cluster-configuration) instructions above, utilize the following command:
```
helm upgrade --install -n perfectscale --create-namespace perfectscale \
--set secret.create=true \
--set secret.clientId=**** \
--set secret.clientSecret=**** \
--set settings.clusterName=name \
--set settings.dcgm.exporterEnabled=true \
--set settings.dcgm.exporterNamespace="perfectscale" \
--set settings.dcgm.exporterPort="9400" \
--set settings.dcgm.exporterLabelSelector="app=nvidia-dcgm-exporter"
perfectscale/exporter
```
{% hint style="warning" %}
The **DCGM exporter is required**. Without it, PerfectScale will be unable to collect and display GPU utilization data.
{% endhint %}
### Required DCGM metrics for GPU memory support
PerfectScale requires the DCGM exporter to expose two GPU framebuffer metrics:
* `DCGM_FI_DEV_FB_TOTAL` - total framebuffer capacity
* `DCGM_FI_DEV_FB_RESERVED` - driver-reserved framebuffer
PerfectScale uses these metrics to calculate GPU memory utilization. If they are not exposed, GPU memory columns in InfraFit, PodFit, and node-group views will appear empty.
{% hint style="warning" %}
The default `dcgm-exporter` counters file does not enable `DCGM_FI_DEV_FB_TOTAL`.
`DCGM_FI_DEV_FB_RESERVED` was introduced in DCGM 3.0, so it may also be missing from older `dcgm-exporter` builds.
{% endhint %}
To confirm which metrics are currently exposed in your cluster, run:
```bash
DCGM_POD=$(kubectl get pod -n gpu-operator -l app=nvidia-dcgm-exporter \
-o jsonpath='{.items[0].metadata.name}')
kubectl exec -n gpu-operator "$DCGM_POD" -- \
curl -s localhost:9400/metrics | grep -E '^DCGM_FI_DEV_FB_(TOTAL|RESERVED)'
```
If the command returns no output, add the missing metrics to the DCGM exporter configuration. Use one of the following approaches to update the DCGM exporter configuration:
#### Patch the DCGM ConfigMap directly
Use this option if the GPU Operator is installed and managed manually, and the DCGM exporter ConfigMap is not controlled by GitOps or another reconciliation process.
{% stepper %}
{% step %}
**Find the ConfigMap that contains the DCGM exporter metrics list**
```bash
kubectl get configmap -n gpu-operator | grep -i metric
```
{% endstep %}
{% step %}
**Edit the relevant ConfigMap**
```bash
kubectl edit configmap -n gpu-operator
```
{% endstep %}
{% step %}
**Locate the dcgm-metrics.csv key**
This key contains the list of DCGM fields exposed by the exporter. Add the following metrics to that list and save the ConfigMap.
```bash
DCGM_FI_DEV_FB_TOTAL, gauge, Total framebuffer memory (in MiB).
DCGM_FI_DEV_FB_RESERVED, gauge, Reserved framebuffer memory (in MiB).
```
{% hint style="info" %}
If `DCGM_FI_DEV_FB_RESERVED` is already present, do not duplicate it. Add only `DCGM_FI_DEV_FB_TOTAL`.
{% endhint %}
{% endstep %}
{% step %}
**Restart the DCGM exporter**
```bash
kubectl rollout restart daemonset -n gpu-operator nvidia-dcgm-exporter
```
{% endstep %}
{% endstepper %}
#### Configure metrics through Helm values
Use this option if the GPU Operator is managed by ArgoCD, Flux, or another GitOps workflow.
In GitOps-managed clusters, manual edits to the ConfigMap are usually overwritten during the next reconciliation. Instead, define a custom metrics ConfigMap and reference it from the GPU Operator Helm values.
{% stepper %}
{% step %}
**Update the GPU Operator Helm values**
```bash
dcgmExporter:
config:
name: metrics-config
env:
- name: DCGM_EXPORTER_COLLECTORS
value: /etc/dcgm-exporter/dcgm-metrics.csv
```
This will point the DCGM exporter to your custom ConfigMap.
{% endstep %}
{% step %}
**Create a ConfigMap with the same name in the gpu-operator namespace**
The ConfigMap must include a `dcgm-metrics.csv` key that contains NVIDIA’s default counters list plus the required framebuffer metrics
```bash
kubectl create configmap metrics-config \
-n gpu-operator \
--from-file=dcgm-metrics.csv=/path/to/your/dcgm-metrics.csv
```
{% endstep %}
{% step %}
**Check if the `dcgm-metrics.csv` file includes the following entries**
```csv
DCGM_FI_DEV_FB_TOTAL, gauge, Total framebuffer memory (in MiB).
DCGM_FI_DEV_FB_RESERVED, gauge, Reserved framebuffer memory (in MiB).
```
{% hint style="info" %}
If `DCGM_FI_DEV_FB_RESERVED` is already present, do not duplicate it. Add only `DCGM_FI_DEV_FB_TOTAL`.
{% endhint %}
{% endstep %}
{% step %}
**Restart the DCGM exporter**
```bash
kubectl rollout restart daemonset -n gpu-operator nvidia-dcgm-exporter
```
{% endstep %}
{% endstepper %}
#### Propagation
After the DCGM exporter restarts, run the verify command again to make sure both metrics are exposed.
Once they are available, it may take up to 15 minutes for GPU memory data to appear in the PerfectScale UI.
## Java containers support
{% hint style="info" %}
The **Coroot agent** is disabled by default. Once enabled, PerfectScale automatically detects Java containers and starts collecting JVM metrics.
Add the following parameter when deploying the PerfectScale agent to enable this feature:
```yaml
--set settings.corootNodeAgent.enabled=true
```
{% endhint %}
{% hint style="info" %}
Coroot supports only Linux nodes.
{% endhint %}
PerfectScale automatically identifies Java containers running in your Kubernetes environment and collects JVM metrics from them. By continuously analyzing them, PerfectScale provides granular visibility into resource usage, identifying potential bottlenecks. Based on this analysis, PerfectScale generates tailored recommendations to help ensure that your services remain efficient and maintain consistent performance.
PerfectScale starts collecting JVM metrics automatically. However, if for some reason you do not want PerfectScale to collect this data, you can disable it by turning off the deployment of `psc-coroot-node-agent` during agent installation:
```
--set settings.corootNodeAgent.enabled=false
```
### Deploy coroot pods to specific nodes
If your cluster includes both Linux and Windows nodes, make sure to set a nodeSelector for Linux when deploying workloads that are not Windows-compatible.
YAML values file example:
```yaml
coroot:
nodeSelector:
kubernetes.io/os: linux
```
Helm command example:
```
helm upgrade --install perfectscale ..... \
--set coroot.nodeSelector."kubernetes\.io/os"=linux \
perfectscale/exporter
```
To restrict workloads to a specific set of nodes (for example, those labeled `component=java`), you can combine multiple node selectors.
YAML values file example:
```
coroot:
nodeSelector:
kubernetes.io/os: linux
component: java
```
Helm command example:
```
helm upgrade --install perfectscale ..... \
--set coroot.nodeSelector."kubernetes\.io/os"=linux \
--set coroot.nodeSelector.component=java \
perfectscale/exporter
```
## Windows containers support
{% hint style="info" %}
Make sure you are using PerfectScale exporter version 1.0.53 or later to enable Windows containers support.
{% endhint %}
PerfectScale supports Windows-based containers, allowing you to optimize and manage them seamlessly. To enable this feature, as a Step 4 in a [cluster configuration](#cluster-configuration), execute the following command:
{% hint style="info" %}
* Make sure the Helm parameter `windowsExporterEnabled` is set to **`true`**
* If the windows-exporter should be deployed by the PerfectScale Helm set `deployWindowsExporter=`**`true`**. In that case, no additional configurations are needed, as the default values will be sufficient.
* If `deployWindowsExporter` set to**`false`**, the additional parameters listed below need to be configured. \
\
Helm parameters should be configured according to the user’s environment:
* `windowsExporterNamespace`
* `windowsExporterPort`
* `windowsExporterLabelSelector`
{% endhint %}
## Size PerfectScale Agent
For large clusters, if you are not using automation, use PerfectScale’s recommendations for `psc-exporter` to properly size it.
## Setting cAdvisor scraping mode
If you prefer not to expose node/proxy permissions, execute the following command when [installing the exporter](/getting-started/how-to-onboard-a-cluster#onboarding-instructions):
```
helm upgrade --install -n perfectscale --create-namespace perfectscale \
--set secret.create=true \
--set secret.clientId=**** \
--set secret.clientSecret=**** \
--set settings.clusterName=name \
--set settings.cAdvisorScrapingMode=direct \
perfectscale/exporter
```
Depending on your security and access requirements, specify one of the scraping modes:
* `auto (default)`: Direct scraping with automatic failover to proxy if all nodes fail.
* `direct`: Force direct node scraping; excludes nodes/proxy RBAC permissions.
* `nodeProxy`: Force proxy scraping via the Kubernetes API.
## Uninstalling PerfectScale exporter
### If the exporter was installed via Helm
{% stepper %}
{% step %}
**To uninstall the PerfectScale exporter, run the following command**
```yaml
helm -n perfectscale uninstall psc-exporter
```
{% endstep %}
{% step %}
**Disconnect the cluster in Cluster Settings**
{% hint style="info" %}
Uninstall the exporter before disconnecting the cluster. If it remains installed, the cluster will continue transmitting data and reappear in the UI.
{% endhint %}
Disconnect a cluster
{% endstep %}
{% step %}
**Delete other PerfectScale agents (Optional)**
```bash
helm list -n perfectscale # returns the list of other helm releases
helm -n perfectscale uninstall
```
{% endstep %}
{% endstepper %}
### If the exporter was installed via PerfectScale Operator
{% stepper %}
{% step %}
**Delete the Agents CR**
Run the following command to uninstall the exporter and any other agents defined in the CR.
```bash
kubectl delete agents.perfectscale.io agents -n perfectscale # this follows this schema kubectl delete -n
```
{% endstep %}
{% step %}
**Uninstall Operator helm release (Optional)**
```bash
helm -n perfectscale uninstall psc-agent-operator
```
{% endstep %}
{% step %}
**Delete other PerfectScale agents (Optional)**
```bash
helm list -n perfectscale # returns the list of other helm releases
helm -n perfectscale uninstall
```
{% endstep %}
{% endstepper %}
## How to whitelist PerfectScale on GKE with binary authorization enabled
If your GKE cluster enforces Binary Authorization, you need to add PerfectScale images to the Binary Authorization policy whitelist before installation.
Add the following entries under `admissionWhitelistPatterns`:
```yaml
- namePattern: public.ecr.aws/perfectscale-io/**
- namePattern: registry.k8s.io/kube-state-metrics/kube-state-metrics*
```
**Example**:
```yaml
admissionWhitelistPatterns:
- namePattern: docker.io/library/alpine*
- namePattern: gcr.io/cloudsql-docker/gce-proxy*
- namePattern: gcr.io/cloud-sql-connectors/cloud-sql-proxy*
- namePattern: docker.io/justwatch/elasticsearch_exporter*
- namePattern: docker.io/otel/opentelemetry-collector-contrib*
- namePattern: us-central1-docker.pkg.dev/doit-glean/gcf-artifacts/**
- namePattern: public.ecr.aws/perfectscale-io/**
- namePattern: registry.k8s.io/kube-state-metrics/kube-state-metrics*
```
These entries allow:
* PerfectScale images from `public.ecr.aws/perfectscale-io`
* `kube-state-metrics` images from `registry.k8s.io`
Without this, PerfectScale components may be blocked from running in clusters where Binary Authorization is enforced.
# Onboarding clusters programmatically
Learn how to set up dynamic clusters provisioning with this step-by-step tutorial
PerfectScale enables the dynamic provisioning of clusters with ease, requiring only a few simple steps. Additionally, it supports mass provisioning, making it efficient for larger deployments. In this case, PerfectScale Helm needs to be included in the set of provisioned workloads of the cluster.
1. To provision clusters dynamically, you need the **`clientId`** and **`clientSecret`.** \
\
Follow steps **1 - 4** of the [Onboarding a cluster](/getting-started/how-to-onboard-a-cluster) instructions to get your **`clientId`** and **`clientSecret`**.
{% hint style="info" %}
[**Onboarding the initial cluster via UI**](/getting-started/how-to-onboard-a-cluster) **is a mandatory step**. In this phase, you will be given the secrets (**`clientId`** and **`clientSecret`**) that can be reused to dynamically onboard clusters or efficiently provision multiple clusters for larger deployments.
{% endhint %}
Credentials
{% hint style="success" %}
If you have already installed the agent and need to recall the **clusterName**, **clientID**, and **clientSecret**, you can retrieve them using the following commands accordingly:
`kubectl -n perfectscale describe deployment perfectscale-exporter | grep CLUSTER_NAME`
`kubectl -n perfectscale get secret perfectscale-exporter-secret -o jsonpath="{.data.clientId}" | base64 --decode`\
\
`kubectl -n perfectscale get secret perfectscale-exporter-secret -o jsonpath="{.data.clientSecret}" | base64 --decode`
{% endhint %}
2. Add **PerfectScale Helm** from the following [link](< https://perfectscale-io.github.io>).
3. Set dynamic variables per the environment that will include the following values:
**`--set settings.clusterName=`** the name of your cluster. To prevent any confusion, it is recommended that unique values be utilized for the cluster name.
{% hint style="danger" %}
**`clusterName`**is mandatory. PerfectScale Agent will not work if the cluster name is not provided.
{% endhint %}
4. Once the variables are identified, each cluster, provisioned with the **PerfectScale Agent,** will be automatically registered and transmit its data.
{% hint style="success" %}
If you have deleted the agent and want to reconnect the same cluster to PerfectScale, you can easily do so by using the same **`clusterName`**, **`clientID`**, and **`clientSecret`**.
{% endhint %}
# ArgoCD install of PerfectScale Exporter
Learn how to install PerfectScale exporter with ArgoCD with this comprehensive guide
## Prerequisites
### **Vocabulary**
{% hint style="info" %}
**PerfectScale Exporter** is a software agent that collects telemetry data related to scaling, aggregates it, and periodically exports the aggregated data to the PerfectScale SaaS Platform.
{% endhint %}
PerfectScale Exporter is a helm chart's helm repo that deploys a kubernetes deployment named perfectscale-exporter, along with other components.
```
helm search repo perfectscale | egrep "NAME|perfectscale/exporter"
```
{% code overflow="wrap" %}
```
NAME CHART V. APP V. DESCRIPTION
perfectscale/exporter v1.1.8 1.0.0 PerfectScale data exporter
```
{% endcode %}
PerfectScale Agent is the helm chart's upstream git repo.
{% embed url="" %}
{% hint style="info" %}
**App CR** is shorthand for ArgoCD Custom Resource (yaml object) of kind: Application.
{% endhint %}
**Helm Charts** are stored in **Helm Repos**. \
\
**Helm Chart**
1. A collection of files and folders on a file system.
2. A named-chart-x.y.z.tgz compressed archive of files and folders (a Gzip compressed tar archive).
**Helm Repo**
1. Helm Chart in a Git Repo, meaning helm chart is just a collection of files and folders that exist in a git repo. By convention, Git branches are commonly used to represent different versions of a Helm chart.
2. A classic Helm repository is a simple HTTP server that follows a convention-based file storage pattern, using an `index.yaml` file and packaged chart archives named `chart-name-x.y.z.tgz`.
3. A Helm chart in an OCI repository is stored in an Open Container Initiative repository, which is a REST API-based HTTP server that implements the OCI Distribution Specification. Helm chart packages, such as `chart-name-x.y.z.tgz`, can be stored and distributed through these repositories.
{% hint style="info" %}
The clarification above makes the following points easier to understand:
App CRs can reference Helm charts stored in Git repositories, classic Helm repositories, and OCI repositories. Because of this, an App CR’s `targetRevision` field, either `spec.sources.0.targetRevision` or `spec.source.targetRevision` , can have two distinct meanings:
* A Git branch or Git tag
* A Helm chart version
Installation Methods 1 and 2 refer to a classic Helm repository.
Installation Method 3 involves pulling a specific version of a packaged Helm chart, such as `chart-name-x.y.z.tgz`, from a classic Helm repository and converting it into a file-and-folder-based Helm chart stored in a private Git repository.
{% endhint %}
### Configuration info and tips
{% hint style="danger" %}
**The perfectscale-exporter helm chart's helm values require 3 mandatory input parameters.**
The PerfectScale Agent will not work if the following values are not provided:
1. clientId
2. clientSecret
3. clusterName
{% endhint %}
If the PerfectScale exporter was previously manually installed using helm, and you want to convert the manual deployment to be managed by ArgoCD, you need to look up the current live deployment's values.
```
kubectl -n perfectscale get secret perfectscale-exporter-secret -o jsonpath="{.data.clientId}" | base64 --decode
kubectl -n perfectscale get secret perfectscale-exporter-secret -o jsonpath="{.data.clientSecret}" | base64 --decode
kubectl -n perfectscale describe deployment perfectscale-exporter | grep CLUSTER_NAME
```
{% hint style="info" %}
Tip for deploying the PerfectScale exporter to multiple clusters
* If you only have a few clusters, each cluster can use unique values for `clientId`, `clientSecret`, and `clusterName`.
* If you need to deploy the PerfectScale Agent across multiple clusters, you can reuse the same `clientId` and `clientSecret` for each cluster, as long as each cluster has a unique `clusterName`.
{% endhint %}
{% hint style="info" %}
Helm has 2 common usage patterns:
1. Deployment tool
2. YAML templating engine
FluxCD uses helm as a deployment tool, while ArgoCD uses helm as a templating engine:
1. When FluxCD deploys a Helm chart, the resulting release is managed by both FluxCD and Helm. As a result, the release appears in the output of:
```
helm list --all-namespaces
```
2. When Argo CD deploys a Helm chart, it uses Helm as a YAML templating engine and then immediately applies the rendered manifests. As a result, the deployment is managed only by Argo CD and does not appear in the output of:
```
helm list -A
```
{% endhint %}
### Preparation
{% stepper %}
{% step %}
**Verify ArgoCD is installed and configured correct.**
{% endstep %}
{% step %}
**Check the latest PerfectScale exporter version**
**Method 1: Manual Lookup**
Look for the most recent release of perfectscale-agent-vx.y.z [here](https://github.com/perfectscale-io/perfectscale-io.github.io/releases)\
\
**Method 2: Run the following command:**
{% code overflow="wrap" %}
```bash
helm repo add perfectscale https://perfectscale-io.github.io --force-update && helm search repo perfectscale | egrep "NAME|perfectscale/exporter"
```
{% endcode %}
Example output, where the latest version is `v1.1.8`
{% code overflow="wrap" %}
```
NAME CHART V. APP V. DESCRIPTION
perfectscale/exporter v1.1.8 1.0.0 PerfectScale data exporter
```
{% endcode %}
{% endstep %}
{% step %}
**Identify the Helm values that need to be passed into the chart.**
Start by reviewing [Onboarding a cluster](/getting-started/how-to-onboard-a-cluster).
In addition to providing guidance, this page includes an example `helm upgrade --install ...` command with flags that can be converted into Helm values.
For example:
```bash
--set secret.create=true \
--set secret.clientId=**** \
--set secret.clientSecret=**** \
--set settings.clusterName=name
```
These Helm command flags are equivalent to the following values:
```yaml
secret:
create: true
clientId: ****
clientSecret: ****
settings:
clusterName: name
```
Verify that you have at least the three mandatory input parameters:
* `clientId`
* `clientSecret`
* `clusterName`
{% endstep %}
{% endstepper %}
### What handles the CR
As a general rule of thumb, you can expect an App CR to handle the following Kubernetes application deployment tasks:
* Create the Kubernetes namespace, if needed
* Fetch a cached copy of the Helm chart
* Deploy a Kubernetes Job to handle the initial installation or upgrade of any CRDs used by the Kubernetes application
* Use the provided Helm values with the Helm chart to deploy the Kubernetes application
This general logic applies to all three installation methods. However, each method begins with an **Overview of Method #** section that summarizes the individual nuances of that method.
## Installing PerfectScale Agent with ArgoCD
### Method 1: Single CR (Recommended)
{% hint style="info" %}
**Overview of Method 1:**
A single ArgoCD application is deployed, which:
1. Follows the normal App CR Handling logic.
2. Fetches the perfectscale-exporter helm chart from a classic helm repo.
3. Uses helm-values that exist within the App CR yaml object.
{% endhint %}
{% hint style="warning" %}
Since the resulting YAML file contains sensitive data, it must be stored securely. At a minimum, store it in a private Git repository.
{% endhint %}
1. Create a perfectscale-exporter.yaml file based on the example below.
{% hint style="info" %}
Ensure you replace and update values as needed.
{% endhint %}
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: perfectscale-exporter
namespace: argocd # Namespace where ArgoCD's Applications are stored
spec: # ^-- This is the conventional value, yours may differ.
project: default
syncPolicy:
automated:
enabled: true # <- Makes it so you don't need to sync in Web UI
syncOptions:
- CreateNamespace=true
- Validate=true
- ServerSideApply=true
destination:
server: https://kubernetes.default.svc
namespace: perfectscale
sources:
- repoURL: https://perfectscale-io.github.io # (Classic Helm Repo)
chart: exporter
targetRevision: v1.1.8 # Helm chart version, Replace with latest
helm:
valuesObject:
secret:
create: true
clientId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx #Replace
clientSecret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx #Replace
settings:
clusterName: your-clusters-name #Replace
corootNodeAgent: # <- JVM Metric collection
enabled: false
```
2. Apply the following command in the cluster application.
```
kubectl apply -f perfectscale-exporter.yaml
```
3. A new application should appear in the ArgoCD Web UI.
New application in ArgoCD
3. If you set `spec.syncPolicy.automated.enabled: false` in the App CR, you may need to access the Argo CD web UI to perform a manual sync.
Sync the application (if needed)
4. Ensure that all pods are running.
{% code expandable="true" %}
```
❯ kubectl get pods -n perfectscale
NAME READY STATUS RESTARTS AGE
perfectscale-exporter-6cbf556b58-6qz47 0/1 Pending 0 0s
perfectscale-exporter-92ee7f02-crd-upgrade-hook-jjrhn 0/1 Completed 0 22s
❯ kubectl get pods -n perfectscale
NAME READY STATUS RESTARTS AGE
perfectscale-exporter-6cbf556b58-6qz47 1/1 Running 0 41s
perfectscale-exporter-cr-manager-69bf9558b8-s4m9w 1/1 Running 0 41s
perfectscale-exporter-kube-state-metrics-5469fcf77c-p5trx 1/1 Running 0 41s
```
{% endcode %}
### Method 2: Two CRs (troubleshooting alternative)
{% hint style="info" %}
**Overview of Method 2:**
Method 1 may fail in rare cases, such as after an incomplete installation, due to CRD-related errors during the App CR sync. For example, the Kubernetes Job responsible for installing or upgrading the CRDs may fail to run correctly.
Method 2 is functionally equivalent to Method 1. The main difference is that Method 2 splits the logic across two Argo CD applications:
1. `perfectscale-exporter-crds`: installs or upgrades the CRDs from a Git repository branch.
2. `perfectscale-exporter`: deploys the Kubernetes application from the Helm chart.
{% endhint %}
{% hint style="danger" %}
Since the resulting YAML file contains sensitive data, it must be stored securely. At a minimum, store it in a private Git repository.
{% endhint %}
1. Create a perfectscale-exporter.yaml file based on the example below.
{% hint style="warning" %}
Ensure you replace and update values as needed.
{% endhint %}
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: perfectscale-exporter-crds
namespace: argocd # Namespace where ArgoCD's Applications are stored
spec: # ^-- This is the conventional value, yours may differ.
project: default
syncPolicy:
automated:
enabled: true # <- Makes it so you don't need to sync in Web UI
destination:
server: https://kubernetes.default.svc
source: #v-- a git repo
repoURL: https://github.com/perfectscale-io/perfectscale-io.github.io.git
targetRevision: perfectscale-agent-v1.1.8 # Git repo tag, Replace with latest
path: charts/perfectscale-agent/crds
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: perfectscale-exporter
namespace: argocd # Namespace where ArgoCD's Applications are stored
spec: # ^-- This is the conventional value, yours may differ.
project: default
syncPolicy:
automated:
enabled: true # <- Makes it so you don't need to sync in Web UI
syncOptions:
- CreateNamespace=true
- Validate=true
- ServerSideApply=true
destination:
server: https://kubernetes.default.svc
namespace: perfectscale
sources: #v-- a classic helm repo
- repoURL: https://perfectscale-io.github.io
chart: exporter
targetRevision: v1.1.8 # Helm chart version, Replace with latest
helm:
skipCrds: true # (The other Application takes care of the CRDs)
valuesObject:
secret:
create: true
clientId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx #Replace
clientSecret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx #Replace
settings:
clusterName: your-clusters-name #Replace
corootNodeAgent: # <- JVM Metrics collector
enabled: false
```
2. Apply both with a single command:
```
kubectl apply -f perfectscale-exporter.yaml
```
3. Ensure that all pods are running.
{% code expandable="true" %}
```
❯ kubectl get pods -n perfectscale
NAME READY STATUS RESTARTS AGE
perfectscale-exporter-6cbf556b58-c79wm 1/1 Running 0 54s
perfectscale-exporter-cr-manager-68d6bcbf5d-kp4zp 1/1 Running 0 54s
perfectscale-exporter-kube-state-metrics-5469fcf77c-ztfpw 1/1 Running 0 54s
```
{% endcode %}
### Method 3: Private git repo hosted helm chart and ArgoCD Web UI (Deprecated)
{% hint style="warning" %}
This method is deprecated because it takes longer to set up and is harder to maintain. Ensure that you replace and update the values as needed.
{% endhint %}
1. Use the unix shell commands below, to download from our helm repo, the most recent version of the perfectscale-exporter helm chart (as a .tgz file).
{% code overflow="wrap" %}
```bash
helm repo add perfectscale https://perfectscale-io.github.io --force-update
helm fetch perfectscale/exporter
tar -xf exporter-*.tgz && rm exporter-*.tgz
```
{% endcode %}
2. Upload the helm chart to a private git repo, and ensure ArgoCD can access the repo.
3. You'll need to create an additional helm values file, for each cluster that you plan to install the perfectscale-exporter application to.\
So a dev.yaml file could be used to represent custom helm values (helm chart input parameters), that would be used when installing into a dev cluster.\
Note: This file should be co-located next to the helm chart's default values.yaml
```
cat > exporter/dev.yaml <
5. Make sure ArgoCD is configured to be able to authenticate against the private git repo.\
(When done correctly, something like this `git@github.com:perfectscale/ps-kube.git` can become a valid repo target.)
6. Login to the ArgoCD Web UI interface and click the **`NEW APP`** button.
New app ArgoCD
7. Give the app a name.
New app form1
8. Scroll down and reference your git repo.\
(`git@github.com:perfectscale/ps-kube.git` represents ssh auth against a private git repo)\
(HEAD: refers to the latest revision)\
(Path: = exporter, refr
New app form2
9. Scroll down and configure a destination\
(kubernetes.default.svc represents the kubernetes cluster where ArgoCD is running)
New app form3
10. Configure input parameters as needed\
values.yaml represents default helm values of the helm chart.\
dev.yaml represents customized helm values that should be applied to a dev cluster.
New app form4
11. Once the creation process is completed, you will see an OutOFSync application.
OutOFSync application
12. Synchronize the application.
Application synchronization
13. Once things stabilize, you should see something like the image below.
Synchronized application
14. Ensure that all pods are running.
Check running pods
# Updating PerfectScale exporter resources
Learn how to update PerfectScale exporter resources
PerfectScale installation consists of two components:
* An agent
* kube-state-metrics.
Both components arrive with the defaults. However, one might want to update clusters that are either very small or extremely large (according to PerfectScale recommendations).
In order to update a cluster, a **`values.yaml`** customization file needs to be created, as shown in the example below.
{% hint style="info" %}
Make sure to replace the numbers from the example with your own.
{% endhint %}
**values.yaml:**
```yaml
resources:
requests:
cpu: 30m
memory: 50M
limits:
memory: 200M
kube-state-metrics:
resources:
requests:
cpu: 50m
memory: 50M
limits:
memory: 200
```
**Apply changes:**
```bash
helm repo update perfectscale
helm upgrade --install -n perfectscale \
-f values.yaml \
--set settings.clusterName=CLUSTER_NAME \
perfectscale perfectscale/exporter
```
{% hint style="info" %}
To view all the default helm values, visit the [values.yaml](https://github.com/perfectscale-io/perfectscale-io.github.io/blob/main/charts/perfectscale-agent/values.yaml) file.
{% endhint %}
# Re-onboarding a cluster
Learn how to re-upload your Kubernetes cluster, ensuring its data continuation
You may encounter situations where re-uploading a cluster is necessary. For example, if you disconnect a cluster, you might later need to upload it again to restore its visibility and management within the PerfectScale.
{% hint style="warning" %}
When you reconnect your cluster using the [standard configuration](/getting-started/how-to-onboard-a-cluster), PerfectScale will assign it a new ID and treat it as a new cluster. The historical data from the previous cluster will remain associated with the original cluster rather than the newly connected one. To restore your cluster while keeping it associated with its historical data, follow the instructions below.
{% endhint %}
## How to retrieve credentials
To retrieve the **clusterName**, **clientID**, and **clientSecret** by executing the following commands accordingly:
#### **clusterName**
```
kubectl -n perfectscale describe deployment perfectscale-exporter | grep CLUSTER_NAME
```
#### clientID
```
kubectl -n perfectscale get secret perfectscale-exporter-secret -o jsonpath="{.data.clientId}" | base64 --decode
```
#### clientSecret
```
kubectl -n perfectscale get secret perfectscale-exporter-secret -o jsonpath="{.data.clientSecret}" | base64 --decode
```
## Re-installing PerfectScale Agent
{% hint style="info" %}
While installing the PerfectScale Agent **(4)**, ensure that you use the same **clusterName** as previously used.\
\
{% endhint %}
Use the following command to re-upload the cluster:
```yaml
helm upgrade --install -n perfectscale --create-namespace perfectscale \
--set secret.create=true \
--set secret.clientId=your_clientId \
--set secret.clientSecret=your_clientSecret \
--set settings.clusterName=your_cluster_name \
perfectscale/exporter
```
{% hint style="info" %}
You can either reuse the clientId and ClientSecret already used for this cluster, or take an existing Id/Secret pair from another running cluster.
{% endhint %}
{% hint style="info" %}
If taints and tolerations are defined in the node pool, you must include them in the command too, so it will appear like this:
```
helm upgrade --install -n perfectscale --create-namespace perfectscale \
--set secret.create=true \
--set secret.clientId=your_clientId \
--set secret.clientSecret=your_clientSecret \
--set settings.clusterName=your_cluster_name \
--set tolerations[0].key=node_pool \
--set tolerations[0].operator=Equal \
--set tolerations[0].value=tooling \
--set kube-state-metrics.tolerations[0].key=node_pool,kube-state-metrics.tolerations[0].value=tooling,kube-state-metrics.tolerations[0].operator=Equal \
perfectscale/exporter
```
{% endhint %}
## Re-installing PerfectScale Automation Agent
If you've re-uploaded the cluster where PerfectScale Automation was previously operational and wish to resume using Automation for that cluster, it's important to re-install the Automation Agent. This step will ensure that your cluster remains continuously optimized, tailored to your previously configured CR.
When re-installing the Automation Agent, run the following command:
```
helm upgrade --install -n perfectscale psc-autoscaler \
--set secret.create=false \
perfectscale/psc-autoscaler
```
{% hint style="info" %}
If taints and tolerations are defined in the node pool, you must include them in the command too, so it will appear like this:
```
helm upgrade --install -n perfectscale psc-autoscaler \
--set tolerations[0].key=node_pool \
--set tolerations[0].operator=Equal \
--set tolerations[0].value=tooling \
--set secret.create=false \
perfectscale/psc-autoscaler
```
{% endhint %}
# Introduction to automation
Continuously enhance performance and resiliency, optimize resource utilization, minimize waste, and streamline the management of your Kubernetes infrastructure with PerfectScale Automation
{% hint style="warning" %}
**Automation is available for Expert Plan users. Users on the 30-day free trial also receive Expert access but can enable automation for only one cluster. To enable your trial access,** [**request automation here**](https://app.perfectscale.io/automation)**. Your request will be validated and, in most cases, approved promptly.**
**To upgrade your subscription level, contact** [**support@perfectscale.io**](mailto:support@perfectscale.io) **or use the** [**PerfectScale Slack Community**](https://perfectscalecommunity.slack.com/)**.**
{% endhint %}
## The fundamental principles of PerfectScale Automation
PerfectScale's PodFit Automation continuously optimizes your Kubernetes environment by autonomously adjusting workloads' CPU and memory to optimal values. Built to support highly dynamic production environments and HPA, PodFit Automations safely and accurately adjusts workload resources to improve stability and cost-effectiveness without the need for manual intervention from your team.
### What type of automated actions will be taken
#### :warning: **Resiliency Issues**
When a workload consumes more resources than allocated, PrefectScale's automation immediately provisions the necessary resources.
#### :money\_with\_wings: **Wasted Resources**
When PerfectScale identifies resource waste and its recommendations are significant, PerfectScale's automation will gradually reduce resources. The system carefully monitors the proper workload functioning after each resource-decreasing step to ensure the best availability.
### Effortless and flexible configuration
#### [**Cluster-level configuration**](/enable-automation/configuring-automation-with-crd/including-a-cluster-namespace-or-workload-to-the-automation/configuring-automation-for-a-cluster-with-crd)
Cluster-level automation configuration allows the definition of global settings that apply to all workloads within the cluster.
{% hint style="info" %}
Cluster-level automation configuration is a mandatory step.
{% endhint %}
#### [**Namespace-level Configuration**](/enable-automation/configuring-automation-with-crd/including-a-cluster-namespace-or-workload-to-the-automation/configuring-automation-for-a-namespace-with-crd)
Namespace-level automation configuration allows customization of automation settings for specific namespaces to meet application-specific needs. Automation configured on the namespace level overrides a global (Cluster-level) configuration.
#### [**Workload-level Configuration**](/enable-automation/configuring-automation-with-crd/including-a-cluster-namespace-or-workload-to-the-automation/configuring-automation-for-a-workload-with-crd)
Workload-level automation configuration allows for the fine-tuning of resource management for individual workloads. Automation configured on the workload level overrides cluster and namespace-level configuration.
### Optimization Policy
In order to support the needs of your workloads, it is important to select the proper **Optimization Policy**.
The optimization policies allow you to specify how your resources should be allocated in order to support the individual needs of your workloads. Define the policies that best suit your environment and business goals, depending on whether you want to maximize cost savings or provide extra headroom to maintain the resilience of mission-critical services:
* **MaxSavings** - maximum cost savings, the best for non-production environments
* **Balanced** (default) - optimally balances cost and resiliency
* **ExtraHeadroom** - the best fit for latency-sensitive environments
* **MaxHeadroom** - keeps the environment above the highest spikes
{% hint style="info" %}
Set up the **Optimization Policy** for the entire [cluster](/administration/cluster-settings#general-settings) or for a specific [workload](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#workload-optimization-policy). The workload's **Optimization Policy** takes precedence and will override the value defined at the cluster level.
{% endhint %}
{% hint style="info" %}
If a custom policy is set through the exporter when installing the PerfectScale Agent, it cannot be modified in the UI afterward. You can still change the policy by upgrading the exporter with the new value, or you can return it to the default by upgrading the exporter without specifying any value (this will also enable the option to change the custom time window through the UI).
{% endhint %}
Discover more about customizing the Optimization policy [here](/customize-workflow/optimization-policy-customization).
### Automation considerations and guardrails
#### **Revision Awareness**
PerfectScale does not contradict your development changes, and despite having automation turned on, PerfectScale immediately accepts any specific user's changes instead of the current recommendations. The system will increase or reduce resources as needed only after we have a clear understanding of how the changes compare with usage patterns.
{% hint style="info" %}
To ensure that automatic resource adjustments yield the best results for you, PerfectScale Automation will only take action once sufficient data is available. The data considered for automated actions are based on the frequency of revisions. In case of frequent revisions, Automation will take actions based on the data over 1 week. If revisions are infrequent and more time is required to gather sufficient data, the analysis will extend up to one month before any adjustments are made.
{% endhint %}
#### **HPA Awareness**
PerfectScale is aware of and respects the HPA settings of your workloads. When an HPA is configured, it takes CPU and memory HPA triggers into account when making optimization and scaling decisions.
#### **Impact Awareness**
We understand the impact that each automated change will make and will not initiate changes that will make a minimal impact on optimizing your environment.
{% hint style="info" %}
If you use CI/CD tools with auto-sync to avoid certificate regeneration for webhook configuration, you can create a static certificate and add it to your Helm override values file. Learn how to do it [here](#how-to-generate-static-certs-for-autoscaler).
{% endhint %}
{% hint style="info" %}
If no `fixResiliencyMaxAutomationFrequency` is specified, PerfectScale defaults to values based on the type of workload:
Deployment: "30m" \
Rollout: "30m"\
DaemonSet: "4h"\
StatefulSet: "24h"\
CronJob: "30m"\
Job: "30m"
{% endhint %}
## Revisions Audit
PerfectScale provides comprehensive revision history details with the built-up [Automation Audit Log](/enable-automation/exploring-automation-actions-and-impact).
## Automation capacity constraints
PerfectScale Automation can optimize up to three workloads at a time. In large clusters, as soon as one workload from the initial group is fully optimized, PerfectScale will automatically begin optimizing the next workload in the queue. This sequential approach ensures continuous optimization without overloading the system.
# Automation setup instructions
Set up PerfectScale Automation in minutes and start seeing instant optimization results with just a few simple steps
{% hint style="info" %}
Automation is currently available for workloads with type **`Deployment`**, **`DaemonSet`**, **`StatefulSet`**, **`CronJob`** and **`Job`**.
{% endhint %}
{% hint style="warning" %}
PerfectScale **does not support** automating workloads that use the **OnDelete update strategy**. The OnDelete strategy requires manual intervention to update pods, which falls outside the scope of PerfectScale's automated optimization capabilities.
{% endhint %}
To set up Automation, you can either follow the installation guide below or complete the setup directly within the platform. Both options make it easy to get started, so you can choose the one that best fits your workflow.
{% hint style="info" %}
To access the Automation enablement guide within the platform, go to the **Overview** screen, hover over the cluster you want to automate, and click the **Configure** button under the **Automated** column.
{% endhint %}
Configure button
## Step 1: Install PerfectScale Automation Agent
{% hint style="warning" %}
When deploying the **PerfectScale Automation Agent** in a **GKE Private Cluster**, you need to create an additional firewall rule. This enables the Control Plane address range to communicate with the Cluster Pod IPv4 address range on port 8443. If the webhook call fails, see the [troubleshooting guide](/enable-automation/troubleshooting#failed-to-call-webhook).
{% endhint %}
Deploy the PerfectScale automation agent. In order to install the PerfectScale automation agent, run the following commands:
```
helm repo add perfectscale https://perfectscale-io.github.io --force-update
```
and
```
helm upgrade --install -n perfectscale psc-autoscaler \
--set secret.create=false \
perfectscale/psc-autoscaler
```
{% hint style="info" %}
If taints and tolerations are defined in the node pool, you must include them in the command too, so it will appear like this:
```
helm upgrade --install -n perfectscale psc-autoscaler \
--set tolerations[0].key=node_pool \
--set tolerations[0].operator=Equal \
--set tolerations[0].value=tooling \
--set secret.create=false \
perfectscale/psc-autoscaler
```
{% endhint %}
{% hint style="warning" %}
If you utilize CI/CD tools with auto-sync, ensure to review the additional configuration steps. [Automation with ArgoCD](/enable-automation/automation-with-argocd)
{% endhint %}
{% hint style="info" %}
Follow the [**troubleshooting**](/enable-automation/troubleshooting) instructions provided, or [**contact support**](https://join.slack.com/t/perfectscalecommunity/shared_invite/zt-1tu9teu9e-Z9tGt4LpNI8tUC3j8obcmQ) if you encounter any issues.
{% endhint %}
### Verify Automation Agent Installation
To test and confirm that the Automation Agent was installed successfully, run the following command:
```
helm test psc-autoscaler -n perfectscale --logs
```
### Disabling Webhook for Specific Namespaces
Once the autoscaler is installed, all pods in the cluster go through the admission webhook. You can exclude specific namespaces from this process by disabling the webhook for them. For example:
## Step 2: Configure Automation
PerfectScale provides two ways to configure Automation and tailor it to your environment:
* [**Through the UI**](/enable-automation/configuring-automation-via-ui) \
A good starting point if you want to quickly explore and enable Automation, such as during a POC or in environments where a simple setup is sufficient.
{% hint style="warning" %}
UI automation provides a simplified experience and supports only part of the features. Full functionality and advanced configuration are available [via CR](/enable-automation/configuring-automation-with-crd).
{% endhint %}
* [**By using a Custom Resource (CR)**](/enable-automation/configuring-automation-with-crd) \
The best suits for teams that require more control and customization, as this option provides greater flexibility and configuration granularity.
The UI offers a simple and convenient way to set up and manage Automation directly in the platform, while the CR option enables flexible, GitOps-friendly configuration within your Kubernetes workflows. Explore how to configure Automation using one of the approaches described above by clicking the links above.
{% hint style="info" %}
**CR configuration takes priority** over UI settings. If both are enabled, PerfectScale will apply automation based on the CR configuration. You can still preview the list of automated workloads in the UI, but any changes should be made through the CR.
{% endhint %}
[^1]: Exclude a Namespace
# Configuring Automation via UI
Seamlessly setup Automation with the guided UI flow.
{% hint style="info" %}
Configuration automation via the UI is supported in **autoscaler version 1.0.43** and above.
{% endhint %}
UI-based automation is designed for teams who want to get started quickly without configuring CRs. It provides a simple, guided way to set up and manage automation directly in PerfectScale, removing friction for new users or teams less familiar with declarative setups.
{% hint style="warning" %}
UI automation provides a simplified experience and supports only part of the features. Full functionality and advanced configuration are available [via CR](/enable-automation/configuring-automation-with-crd).
{% endhint %}
{% hint style="success" %}
You can start with the UI to quickly set up and validate automations, and later move to CRDs for more advanced customization and granular control.
☝🏼 If automation is configured via CR in addition to UI-based settings, the UI-based automation will be disabled.
{% endhint %}
Once the [Automation Agent is installed](/enable-automation/automation-setup-instructions#step-1-install-perfectscale-automation-agent), you can easily configure automation for your clusters and workloads using a guided UI flow.
{% hint style="warning" %}
Only **Admin** users can configure automation via the UI. Other users can view the list of automated workloads but cannot create, edit, or disable automation.
{% endhint %}
To automate an entire cluster or specific workloads via the UI, go to the **Overview** screen, hover over the desired cluster, and click the **Automate** button in the **Automation** column.
Configuring automation via UI
Clicking the **Automate** button opens the Automation configuration screen. You have a few options:
* **Enable Cluster Automation** - enables the automation configuration, but no workloads are automated yet.
{% hint style="info" %}
This step is required to proceed with automation configuration.
{% endhint %}
* **Automate the entire cluster** - autonomous optimization is applied to all workloads within the cluster.
* **Automate specific workloads** - autonomous optimization is applied only to the workloads you select within the cluster.
## UI automation settings overview
This approach is designed to help teams quickly set up automation without the need to dig in and configure automation CR. While it offers less flexibility than a CR-based configuration, it provides everything you need to start optimizing your workloads autonomously.
### Automating a cluster
To automate the entire cluster, enable the **Cluster Automation** toggle. Once done, the **Enable** button next to the number of workloads in the cluster becomes active. Clicking **Enable** will turn on all toggles for the workloads automatically. Finally, click **Save Changes**. Within a few minutes, PerfectScale will begin optimizing the cluster autonomously.
Automating a cluster via UI
### Automating specific workloads
To enable automation for specific workloads, first turn on the **Cluster Automation** toggle. Once enabled, the toggles for individual workloads become active, allowing you to select the ones you want to automate. Click **Save Changes**, and within a few minutes, PerfectScale will start optimizing the selected workloads autonomously.
Automating workloads via UI
Alternatively, you can automate a workload directly from its zoom-in window. Click the workload to open the window, then toggle **Automation** on with a single click. Within a few minutes, PerfectScale will begin optimizing this workload autonomously.
Automating a workload via UI
{% hint style="info" %}
Automation is applied only to explicitly selected workloads. Newly created workloads are not automated by default. You need to select them in the UI and enable automation manually.
{% endhint %}
{% hint style="warning" %}
If any system or user-defined limitations prevent automation, or if you are trying to automate an [unsupported workload type](#supported-workload-types), a pop-up or [automation status](/enable-automation/verifying-automation-status) will be displayed to notify you. In this case, automation will not be applied.
{% endhint %}
### Additional configurations
Additionally, you can choose whether automation should respect your configured CPU limits. Generally, we recommend removing CPU limits to allow workloads to use available CPU resources more freely, improving performance and efficiency.
By default, PerfectScale respects your configured CPU limits.
{% hint style="info" %}
In most cases, **we recommend removing CPU limits** so workloads can use available CPU resources more freely, which can improve both performance and efficiency.
{% endhint %}
To let PerfectScale override configured CPU limits, open the **Automation** tab in a **Cluster Settings**, click the three-dot menu next to the cluster name, and clear **Keep CPU Limit**.
Additional configurations
To revert all automation changes and disable automation for the cluster, click the **Cleanup All Automation** button and confirm the action.
{% hint style="info" %}
**Keep CPU Limit** and **Cleanup All Automation** are cluster-level settings.
{% endhint %}
### Supported workload types
Automation via the UI supports the same workload types as CR-based automation, including **Deployments**, **DaemonSets**, **StatefulSets**, **CronJobs**, **Jobs**, **Rollouts,** and [**custom grouped workloads by label**](/customize-workflow/ephemeral-workloads-grouping). If a workload type is not supported, a corresponding status will be displayed to indicate this.
## Disabling automation via UI
You can disable automation in the UI only if it was originally configured through the UI. To do so, open the **Automation** window, click **Disable**, and save your changes.
Disabling automation via UI
# Configuring Automation with CRD
Seamlessly setup Automation with flexible CR configuration.
Use a Custom Resource (CR) to configure Automation through flexible configuration. This allows you to seamlessly integrate automation into your existing workflows and manage it using standard Kubernetes practices.
## Create Custom Resource (CR)
After the installation of the automation agent, it is important to create a **cluster automation configuration**, as this step **is mandatory:**
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: cluster-automation-config
spec:
automation:
operational:
stopAllAutomation: false # Global kill switch for automation (default: false)
cleanupAllAutomation: false # Stops automation and reverts all changes, restoring original resource specifications and settings
```
{% hint style="info" %}
Automation is only enabled in configurations where automationMode for the workload type is set to "Enabled". This can be configured in the `ClusterAutomationConfig`, `NamespaceAutomationConfig`, or `WorkloadAutomationConfig`.
**Example**:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: cluster-automation-config
spec:
automation:
operational:
stopAllAutomation: false # Global kill switch for automation (default: false)
cleanupAllAutomation: false # Stops automation and reverts all changes, restoring original resource specifications and settings
workloadTypes:
Deployment:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
```
This configuration enables automation for all workloads in a cluster with **type: Deployment**. Learn more about enabling automation for particular workload types [in a cluster](#cluster-level-configuration), [specific namespace](#namespace-level-configuration), or [specific workloads](#workload-level-configuration).
{% endhint %}
{% hint style="info" %}
**Only one cluster-level automation configuration can be applied per cluster**. To prevent errors, PerfectScale recommends using consistent naming for cluster-level configurations within the same cluster. This ensures that a new configuration will override the previous one without causing any conflicts.
{% endhint %}
To help you quickly start using PerfectScale Automation and streamline the optimization process, we provide pre-built Custom Resources (CRs) focused on desired optimization goals. You can easily implement the following CRs and customize them if needed.
{% hint style="info" %}
PerfectScale recommends you start with the cluster [cost-saving automation configuration](#recommended-adoption-path).
:point\_right: Explore more configuration examples at our [GitHub](https://github.com/perfectscale-io/perfectscale-io.github.io/tree/main/charts/psc-autoscaler/examples).
{% endhint %}
## :zap: Recommended adoption path
To minimize waste and achieve optimal savings, we recommend starting with a simple, cluster-level automation focused on cost reduction.
**Create** a YAML file with the recommended cluster-level automation configuration using the following template:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: cluster-automation-config
spec:
automation:
operational:
stopAllAutomation: false # Global kill switch for automation (default: false)
cleanupAllAutomation: false # Stops automation and reverts all changes, restoring original resource specifications and settings
workloadTypes:
Deployment:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
restrictions:
DaemonSet:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
restrictions:
```
**Apply** the created configuration by running the following command (where `cluster-automation-config.yaml` - the name of your file created in the previous step):
```
kubectl apply -f cluster-automation-config.yaml
```
### Combining cost savings and performance improvement configuration
The following example of a Cluster-level automation configuration allows you to automate the optimization of the entire cluster, enhancing its performance while maintaining optimal costs.
**Create** a YAML file with the cluster-level automation configuration using the following template:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: cluster-automation-config
spec:
automation:
operational:
stopAllAutomation: false # Global kill switch for automation (default: false)
cleanupAllAutomation: false # Stops automation and reverts all changes, restoring original resource specifications and settings
workloadTypes:
Deployment:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: true # Allows PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: true # Allows PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
limit:
increaseEnabled: true # Allows PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory limit
DaemonSet:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: true # Allows PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: true # Allows PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
limit:
increaseEnabled: true # Allows PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory limit
```
**Apply** the created configuration by running the following command (where `cluster-automation-config.yaml` - the name of your file created in the previous step):
```
kubectl apply -f cluster-automation-config.yaml
```
Let's explore additional examples of automation configurations that offer more granular and flexible control over Automation:
* [How to include a cluster, namespace, or workload in the Automation configuration](/enable-automation/configuring-automation-with-crd/including-a-cluster-namespace-or-workload-to-the-automation)
* [How to exclude a namespace or workload from the Automation configuration](/enable-automation/configuring-automation-with-crd/excluding-a-namespace-or-workload-from-the-automation-with-crd)
### Automation configuration defaults
Some automation configuration settings have predefined defaults. If you don’t specify a value, PerfectScale will automatically apply the default for that field. Below is the list of configurations and their default values.
# Including a cluster, namespace, or workload to the Automation
Learn how to effortlessly include a cluster, namespace, or workload into your Automation configuration
Advanced PerfectScale's Automation allows precise configuration, making it easy to adapt to your specific use cases and the operational needs of your applications. You can seamlessly automate optimization for the entire cluster, a specific namespace, or an individual workload while also having the flexibility to exclude particular entities from automation as needed.
# Configuring Automation for a cluster with CRD
Learn how to apply Automation to all workloads within the cluster effortlessly
Cluster-level automation configuration allows the definition of global settings that apply to all workloads within the cluster.
{% hint style="info" %}
You can effortlessly exclude specific namespaces and workloads from Automation. Learn more [here](/enable-automation/configuring-automation-with-crd/excluding-a-namespace-or-workload-from-the-automation-with-crd).
{% endhint %}
## Including a cluster in the Automation
{% hint style="success" %}
You can also scope automation to a specific workload or namespace. The workload or namespace-level settings take precedence over global (cluster-wide) automation settings. Learn more about how to automate a specific [namespace](/enable-automation/configuring-automation-with-crd/including-a-cluster-namespace-or-workload-to-the-automation/configuring-automation-for-a-namespace-with-crd) or [workload](/enable-automation/configuring-automation-with-crd/including-a-cluster-namespace-or-workload-to-the-automation/configuring-automation-for-a-workload-with-crd).
{% endhint %}
### Configuration with additional parameters
The following cluster-level automation configuration example represents global automation settings that apply to **all workloads with the type** **Deployment** within the cluster. Additionally, there is a specified frequency of the automation actions and maintenance window, which gives you control over the automation and minimizes the impact of changes on business operations.
{% hint style="info" %}
Uncover the full range of customization options, how to implement them and tailor Automation for your application's needs [here](/enable-automation/configuring-automation-with-crd/automation-customization-with-crd).
{% endhint %}
**Create** a YAML file with the cluster-level automation configuration using the following template:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: cluster-automation-config
spec:
automation:
operational:
stopAllAutomation: false # Global kill switch for automation (default: false)
cleanupAllAutomation: false # Stops automation and reverts all changes, restoring original resource specifications and settings
duringRolloutInProgress: pause # A default behavior that prevents Automation from applying new recommendations when concurrent ReplicaSets are identified
timeConstraints:
wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
maintenanceWindow:
monday:
- "00:00-23:59"
tuesday:
- "00:00-23:59"
wednesday:
- "00:00-23:59"
thursday:
- "00:00-23:59"
friday:
- "00:00-23:59"
saturday:
- "00:00-23:59"
sunday:
- "00:00-23:59"
maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week
workloadTypes:
Deployment:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
timeConstraints:
wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
maintenanceWindow:
monday:
- "00:00-23:59"
tuesday:
- "00:00-23:59"
wednesday:
- "00:00-23:59"
thursday:
- "00:00-23:59"
friday:
- "00:00-23:59"
saturday:
- "00:00-23:59"
sunday:
- "00:00-23:59"
maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week
Rollout:
operational:
automationMode: "Enabled" # Enables rollout support
```
**Apply** the created configuration by running the following command (where `cluster-automation-config.yaml` - the name of your file created in the previous step):
```
kubectl apply -f cluster-automation-config.yaml
```
Explore more about automation customization options [here](/enable-automation/configuring-automation-with-crd/automation-customization-with-crd).
# Configuring Automation for a namespace with CRD
Learn how to apply Automation to all workloads within the namespace effortlessly
## Including a namespace in the Automation
The following example of a namespace-level automation configuration allows you to customize automation settings for specific namespaces, catering to the needs of different applications. This example shows how to automate optimization for **all Deployment-type workloads in the namespace**.
**Create** a YAML file with the namespace-level automation configuration using the following template:
```yaml
apiVersion: perfectscale.io/v1
kind: NamespaceAutomationConfig
metadata:
name: namespace-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
automation:
operational:
automationMode: "Enabled" # Enables automation for a specific namespace
duringRolloutInProgress: pause # A default behavior that prevents Automation from applying new recommendations when concurrent ReplicaSets are identified
timeConstraints:
wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
maintenanceWindow:
monday:
- "00:00-23:59"
tuesday:
- "00:00-23:59"
wednesday:
- "00:00-23:59"
thursday:
- "00:00-23:59"
friday:
- "00:00-23:59"
saturday:
- "00:00-23:59"
sunday:
- "00:00-23:59"
maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum inst
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week
workloadTypes:
Deployment:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
timeConstraints:
wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
maintenanceWindow:
monday:
- "00:00-23:59"
tuesday:
- "00:00-23:59"
wednesday:
- "00:00-23:59"
thursday:
- "00:00-23:59"
friday:
- "00:00-23:59"
saturday:
- "00:00-23:59"
sunday:
- "00:00-23:59"
maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week
Rollout:
operational:
automationMode: "Enabled" # Enables rollout support
```
**Apply** the created configuration by running the following command (where `namespace-automation-config.yaml` - the name of your file created in the previous step)
```
kubectl apply -f namespace-automation-config.yaml
```
### Automating multiple namespaces with a single configuration
If you want to apply the same configuration to multiple namespaces, remove **`namespace: your-namespace`** from the configuration and apply the configuration with the following command for each namespace **`kubectl apply -f namespace-automation-config.yaml -n namespace`**, where **`namespace`** - the namespace where it will be applied.\
If a **`namespace`** is specified neither in the configuration nor in the command, the configuration will be applied to the default namespace.
### Automating kube-system namespace
{% hint style="warning" %}
If you need to automate the **`kube-system` namespace** in an **AKS cluster**, additional configuration is required [due to known limitations](https://learn.microsoft.com/en-us/azure/aks/faq#can-admission-controller-webhooks-impact-kube-system-and-internal-aks-namespaces-). \
\
Learn more about the configuration [here](#automating-the-kube-system-namespace-in-aks-clusters).
{% endhint %}
By default, PerfectScale does not automate the `kube-system` namespace. To enable automation for this namespace, follow these simple steps.
**Remove** the `kube-system` namespace from the `excludedNamespaces` (is only needed if you are using **Autoscaler v1.0.6 and older**).
```yaml
helm upgrade --install -n perfectscale --reset-then-reuse-values psc-autoscaler --set settings.excludedNamespaces=[] ./helm
```
**Create** a YAML file with the namespace-level automation configuration using the following template:
```yaml
apiVersion: perfectscale.io/v1
kind: NamespaceAutomationConfig
metadata:
name: kube-system-ns-config
namespace: kube-system # The namespace where the configuration should be applied
spec:
automation:
operational:
automationMode: "Enabled" # Enables automation for a specific namespace
duringRolloutInProgress: pause # A default behavior that prevents Automation from applying new recommendations when concurrent ReplicaSets are identified
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: true # Allows PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
limit:
keepLimit: false # Automation can set the CPU limit to zero
memoryManagement:
request:
increaseEnabled: true # Allows PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
limit:
increaseEnabled: true # Allows PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory limit
```
**Apply** the created configuration by running the following command (where `kube-system-ns-config.yaml` - the name of your file created in the previous step)
```yaml
kubectl apply -f kube-system-ns-config.yaml
```
Explore more about automation customization options [here](broken://pages/AoWNruTdYbpYb7aSotCr).
#### :exclamation: Automating kube-system namespace in AKS clusters
Azure Kubernetes Service has a [known limitation](https://learn.microsoft.com/en-us/azure/aks/faq#can-admission-controller-webhooks-impact-kube-system-and-internal-aks-namespaces-) that prevents tools from managing resources within the `kube-system` namespace by default. To enable automation in this namespace, it is necessary to add the following annotation to allow Admissions Enforcer to ignore it:
```
helm upgrade --install -n perfectscale psc-autoscaler \
--set-string admission.annotations."admissions\.enforcer/disabled"=true \
perfectscale/psc-autoscaler
```
# Configuring Automation for a workload with CRD
Learn how to effortlessly automate the right-sizing of the specific workloads
## Including a workload in the Automation
Workload-level automation allows fine-tuning resource management of individual workloads. Automation configured on the workload level overrides cluster and namespace-level configuration.
{% hint style="info" %}
Workload automation config should specify a namespace, and it should match the namespace of the associated workload.
{% endhint %}
**Create** a YAML file with the workload-level automation configuration using the following template:
```yaml
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
targetRef:
kind: Deployment
name: workload
automation:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
duringRolloutInProgress: pause # A default behavior that prevents Automation from applying new recommendations when concurrent ReplicaSets are identified
timeConstraints:
wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
maintenanceWindow:
monday:
- "00:00-23:59"
tuesday:
- "00:00-23:59"
wednesday:
- "00:00-23:59"
thursday:
- "00:00-23:59"
friday:
- "00:00-23:59"
saturday:
- "00:00-23:59"
sunday:
- "00:00-23:59"
maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead maximumCores: -1
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead maximumGiB: -1
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week
containers:
container1:
operational:
restrictions:
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week
```
**Apply** the created configuration by running the following command (where `workload-automation-config.yaml` - the name of your file created in the previous step)
```
kubectl apply -f workload-automation-config.yaml
```
## Automating workloads with a specific type
### DaemonSet
{% hint style="info" %}
DaemonSet workloads are automated in **decrease-only mode** to prevent over-provisioning. This avoids excessive resource recommendations caused by spikes on one node, reducing waste and ensuring pods can still be scheduled on smaller nodes.
{% endhint %}
**Create** a YAML file with the workload-level automation configuration using the following template:
```yaml
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: your-workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
targetRef:
kind: DaemonSet #Case sensitive, must be DaemonSet, not Daemonset
name: workload
automation:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
duringRolloutInProgress: pause # A default behavior that prevents Automation from applying new recommendations when concurrent ReplicaSets are identified
timeConstraints:
wasteMaxAutomationFrequency: "4h" # Applies recommendations to decrease resources every 4 hours
fixResiliencyMaxAutomationFrequency: "4h" # Applies recommendations to increase resources every 4 hours
maintenanceWindow:
monday:
- "00:00-23:59"
tuesday:
- "00:00-23:59"
wednesday:
- "00:00-23:59"
thursday:
- "00:00-23:59"
friday:
- "00:00-23:59"
saturday:
- "00:00-23:59"
sunday:
- "00:00-23:59"
maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week
```
**Apply** the created configuration by running the following command (where `your-workload-automation-config.yaml` - the name of your file created in the previous step)
```
kubectl apply -f your-workload-automation-config.yaml
```
### StatefulSet
{% hint style="info" %}
Automating StatefulSets requires careful consideration:
1. You may want someone to manage production database resource changes, including any restart. Pods with persistent volumes are more sensitive.
2. In many cases, a StatefulSet runs on dedicated nodes. Consequently, reducing resources will not lead to a reduction in the actual cost.
{% endhint %}
**Create** a YAML file with the workload-level automation configuration using the following template:
```yaml
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: your-workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
targetRef:
kind: StatefulSet #Case sensitive, must be StatefulSet, not Statefulset
name: workload
automation:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
duringRolloutInProgress: pause # A default behavior that prevents Automation from applying new recommendations when concurrent ReplicaSets are identified
timeConstraints:
wasteMaxAutomationFrequency: "24h" # Applies recommendations to decrease resources every 4 hours
fixResiliencyMaxAutomationFrequency: "24h" # Applies recommendations to increase resources every 4 hours
maintenanceWindow:
monday:
- "00:00-23:59"
tuesday:
- "00:00-23:59"
wednesday:
- "00:00-23:59"
thursday:
- "00:00-23:59"
friday:
- "00:00-23:59"
saturday:
- "00:00-23:59"
sunday:
- "00:00-23:59"
maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week
```
**Apply** the created configuration by running the following command (where `your-workload-automation-config.yaml` - the name of your file created in the previous step)
```
kubectl apply -f your-workload-automation-config.yaml
```
### CronJob and Job
{% hint style="info" %}
PerfectScale doesn't restart workloads with the CronJob or Job type and automates only new workloads.
{% endhint %}
**Create** a YAML file with the workload-level automation configuration using one of the following templates:
#### **CronJob configuration**
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: your-workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
targetRef:
kind: CronJob #Case sensitive, must be CronJob, not Cronjob
name: workload
automation:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
duringRolloutInProgress: pause # A default behavior that prevents Automation from applying new recommendations when concurrent ReplicaSets are identified
timeConstraints:
wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 4 hours
fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 4 hours
maintenanceWindow:
monday:
- "00:00-23:59"
tuesday:
- "00:00-23:59"
wednesday:
- "00:00-23:59"
thursday:
- "00:00-23:59"
friday:
- "00:00-23:59"
saturday:
- "00:00-23:59"
sunday:
- "00:00-23:59"
maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week
### **Job configuration**
```yaml
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: your-workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
targetRef:
kind: Job
name: workload
automation:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
duringRolloutInProgress: pause # A default behavior that prevents Automation from applying new recommendations when concurrent ReplicaSets are identified
timeConstraints:
wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 4 hours
fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 4 hours
maintenanceWindow:
monday:
- "00:00-23:59"
tuesday:
- "00:00-23:59"
wednesday:
- "00:00-23:59"
thursday:
- "00:00-23:59"
friday:
- "00:00-23:59"
saturday:
- "00:00-23:59"
sunday:
- "00:00-23:59"
maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week
```
**Apply** the created configuration by running the following command (where `your-workload-automation-config.yaml` - the name of your file created in the previous step)
```
kubectl apply -f your-workload-automation-config.yaml
```
## **Automating workloads by label**
You can easily configure automation for specific workloads by using labels. To do so, add the workloadLabelSelectors section to your cluster-level or namespace-level configuration. Specify the label key and value, then choose whether to enable or disable automation for workloads with the given label.
{% hint style="info" %}
If `workloadLabelSelectors` are specified in both `ClusterAutomationConfig` and `NamespaceAutomationConfig`configurations, PerfectScale will merge them.
In cases of a conflict on the key and value between the namespace and cluster configurations, the `allowAutomation` value from the namespace configuration will take precedence.
{% endhint %}
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: your-cluster-automation-config
spec:
automation:
operational:
workloadLabelSelectors:
- key: "environment"
value: "production"
allowAutomation: false # Disables Automation for workload with the specified label
- key: "tier"
value: "frontend"
allowAutomation: true # Enables Automation for workload with the specified label
```
**Expected behavior:**
1. If `workloadLabelSelectors` is empty or not set, automation configurations will be applied according to the other configurations in place.
2. If `workloadLabelSelectors` only contains labels with `allowAutomation: true`, automation configurations will be applied only to the workloads that match at least one label specified in this configuration.
3. If `workloadLabelSelectors` only contains labels with `allowAutomation: false`, automation configurations will be applied solely to workloads that do not match any specified labels in this configuration.
4. If `workloadLabelSelectors` contains a mix of labels with `allowAutomation: false` and `allowAutomation: true`, automation configurations will be applied to workloads that match the labels with `allowAutomation: true` and do not match the labels with `allowAutomation: false` in this configuration.
{% hint style="info" %}
If a workload has a label selector `allowAutomation: true`, indicating it should be automated, but `workloadType.X.operational.automationMode` is set to **Disabled**, the `automationMode` setting takes precedence.
{% endhint %}
{% hint style="info" %}
If you enable automation for a custom workload type, the `WorkloadLabelsSelector` in a cluster or namespace configuration will not be applied. All workloads of that custom type will be automated despite the label's configuration.
{% endhint %}
### Label-based automation execution logic
When autonomous optimization is configured using labels, PerfectScale applies the following logic:
1. Check the `labels` section in the **Automation CR**.
2. Check the configurations applied to workloads that have these labels, including the `automationMode` value.
{% hint style="warning" %}
To automate workloads using labels, ensure an exact match between the labels defined in the Automation CR and the labels applied to the target workloads. If the labels do not match exactly, the workload will not be automated.
{% endhint %}
#### Expected results example
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: your-cluster-automation-config
spec:
automation:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
workloadLabelSelectors:
- key: "environment"
value: "production"
allowAutomation: false # Disables Automation for workload with the specified label
- key: "env"
value: "dev"
allowAutomation: true # Enables Automation for workload with the specified label
- key: "tier"
value: "frontend"
allowAutomation: true # Enables Automation for workload with the specified label
```
:white\_check\_mark: **Automated workloads**
As a result of the Automation CR configuration above, PerfectScale verifies that `automationMode` is set to **Enabled** and checks the `workloadLabelSelectors`. Automation will be applied to all workloads in the specified namespace that match at least one configured label with `allowAutomation: true`, and do not match label with `allowAutomation:false`.
In this case, the automation mode is applied according to the default merge logic based on the cluster- or namespace-level configuration.
:red\_circle: **Non-automated workloads**
As a result of the Automation CR configuration above, PerfectScale verifies that `automationMode` is set to **Enabled** and checks the `workloadLabelSelectors`. Workloads with `allowAutomation: false`, as well as workloads in the specified namespace that do not have at least one label with `allowAutomation: true`, will **not** be automated.
## Automating ephemeral workloads
{% hint style="info" %}
Ephemeral workload automation is supported, starting with autoscaler version 1.0.17.
{% endhint %}
PerfectScale's advanced Automation empowers you to effortlessly optimize K8s workloads, even in highly dynamic and complex environments. By optimizing workloads with a transient nature, you can ensure that you’re not over-provisioning or paying for unused capacity, significantly reduce cloud costs, and meet performance demands without latency or bottlenecks.
By grouping ephemeral workloads with specific workload labels, PerfectScale automates the optimization of ephemeral workloads such as:
* Airflow
* Knative
* Spark jobs
* Temporal
* Github actions
* and more
### Step 1: Group ephemeral workloads by labels
To automate the optimization of ephemeral workloads, you first need to group them by applying specific labels. This is done by adding predefined labels to each workload. For detailed instructions on how to group ephemeral workloads, refer to [this page](/customize-workflow/ephemeral-workloads-grouping#grouping-by-labels).
{% hint style="info" %}
PerfectScale doesn't restart grouped workloads and will only apply recommendations to newly created pods.
{% endhint %}
### Step 2: Apply CR that contains these labels
For workloads with a custom type, automation is enabled by setting up a Custom Resource (CR), where you can specify all the necessary configurations.
#### Cluster automation configuration
**Create** a YAML file with the automation configuration and indicate the custom workload type:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: cluster-automation-config
spec:
automation:
operational:
workloadTypes:
CustomWorkloadType: # The custom workload type specified when grouping ephemeral workloads in Step 1
operational:
automationMode: "Enabled" # Enables automation for the workloads with the custom workload type
```
**Apply** the created configuration by running the following command (where `cluster-automation-config.yaml` - the name of your file created in the previous step)
```
kubectl apply -f cluster-automation-config.yaml
```
#### Namespace automation configuration
**Create** a YAML file with the automation configuration and indicate the custom workload type:
```yaml
apiVersion: perfectscale.io/v1
kind: NamespaceAutomationConfig
metadata:
name: namespace-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
automation:
operational:
workloadTypes:
CustomWorkloadType: # The custom workload type specified when grouping ephemeral workloads in Step 1
operational:
automationMode: "Enabled" # Enables automation for the workloads with the custom workload type
```
**Apply** the created configuration by running the following command (where `namespace-automation-config.yaml` - the name of your file created in the previous step)
```
kubectl apply -f namespace-automation-config.yaml
```
#### Workload automation configuration
**Create** a YAML file with the automation configuration and indicate the custom workload type:
```yaml
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
targetRef:
kind: CustomSparkJob # Custom workload type
name: customWorkload
automation:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
```
**Apply** the created configuration by running the following command (where `workload-automation-config.yaml` - the name of your file created in the previous step)
```
kubectl apply -f workload-automation-config.yaml
```
# Excluding a namespace or workload from the Automation with CRD
Learn how to effortlessly exclude a cluster, namespace, or workload from your Automation configuration
In some cases, you might want to exclude specific namespaces or workloads from automation. Use the following configurations in order to **exclude** the particular **namespace** or **workload** from automation in the cluster
## **Exclude a namespace**
```yaml
apiVersion: perfectscale.io/v1
kind: NamespaceAutomationConfig
metadata:
name: your-namespace-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
automation:
operational:
automationMode: Disabled # Disables Automation for the specified namespace
```
### Exclude kube-system namespace
By default, PerfectScale does not automate the `kube-system` namespace. However, in some cases, GCP users may encounter warning banners. If you notice this warning and want to prevent PerfectScale from intercepting the pods in this namespace (meaning you don't want to automate the kube-system namespace at all), further configuration is needed.
There are two ways to exclude the kube-system namespace completely:
1. Set `excludeNamespaces` to `kube-system` [here](https://github.com/perfectscale-io/perfectscale-io.github.io/blob/f39759c64cb64b357f05e4ac314f963f6cd1d61a/charts/psc-autoscaler/values.yaml#L22).
```
excludedNamespaces: ["kube-system"]
```
2. Upgrade the Automation Agent by running the following command:
```
helm upgrade --install -n perfectscale psc-autoscaler \
--set secret.create=false \
--reset-then-reuse-values \
--set "settings.excludedNamespaces[0]=kube-system" \
perfectscale/psc-autoscaler
```
## **Exclude a workload**
{% hint style="info" %}
Workload automation config should specify a namespace, and it should match the namespace of the associated workload.
{% endhint %}
```yaml
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: your-workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
automation:
operational:
automationMode: Disabled # Disables Automation for the workload with the specified parameters
targetRef:
kind: Deployment
name: deployment-name-here
```
### **Exclude workload(s) by label**
{% hint style="info" %}
Workload automation config should specify a namespace, and it should match the namespace of the associated workload.
{% endhint %}
```yaml
apiVersion: perfectscale.io/v1
kind: NamespaceAutomationConfig
metadata:
name: your-workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
automation:
operational:
workloadLabelSelectors:
- key: "environment"
value: "production"
allowAutomation: false # Disables Automation for the workload(s) with the specified label
```
# Automation customization with CRD
Learn how to customize your Automation setup
{% hint style="info" %}
* All example YAML HashMaps within `timeConstraints` and `restrictions`, are applicable to 3 different kinds of yaml objects:
* `kind: ClusterAutomationConfig`
* `kind: NamespaceAutomationConfig`
* `kind: WorkloadAutomationConfig`\
* All example YAML HashMaps within `timeConstraints` , are applicable to multiple YAML locations:
* `kind: ClusterAutomationConfig` and\
`kind: NamespaceAutomationConfig`
* `spec.automation.operational.timeConstraints:`
* `spec.automation.workloadTypes.(StatefulSet|Deployment|DaemonSet|CronJob|Job).operational.timeConstraints:`
* `kind: WorkloadAutomationConfig`
* `spec.automation.operational.timeConstraints:`
* `spec.automation.operational.containers.(a-specific-container-name).operational.timeConstraints:` \
* All example YAML HashMaps within `restrictions` , are applicable to multiple YAML locations:
* `kind: ClusterAutomationConfig` and\
`kind: NamespaceAutomationConfig`
* `spec.automation.operational.restrictions:`
* `spec.automation.workloadTypes.(StatefulSet|Deployment|DaemonSet|CronJob|Job).operational.restrictions:`
* `kind: WorkloadAutomationConfig`
* `spec.automation.operational.restrictions:`
* `spec.automation.operational.containers.(a-specific-container-name).operational.restrictions:` \
* `kind: NamespaceAutomationConfig` and\
`kind: WorkloadAutomationConfig`
* Can have an optional value: `metadata.namespace: example-namespace`. Most examples omit this for brevity.\
* All restrictions are compared to the original spec values of the workload.
* `kind: ClusterAutomationConfig` is used to set the original spec value.
* If a minimal config is used several implicit default values will be used.
* To determine the implicit default values
1. Login to
2. Navigate to the Overview tab
3. Select a cluster that has automation enabled
4. Select any running workload by name
5. Click in the top left of the pop-up window to see Automation Config CR mergedConfig YAML manifest.
6. The mergedConfig view makes explicit the end-result of merged values and implicit default values.
* `kind: NamespaceAutomationConfig` can be used to override ClusterAutomationConfig.
* `kind: WorkloadAutomationConfig` can be used to both ClusterAutomationConfig and NamespaceAutomationConfig.
{% endhint %}
Customize your automation with the following settings:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig #, NamespaceAutomationConfig, or WorkloadAutomationConfig
metadata:
name: example
spec:
automation:
operational:
timeConstraints:
wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
maintenanceWindow: #UTC
monday: ["00:00-23:59"]
tuesday: ["00:00-23:59"]
wednesday: ["00:00-23:59"]
thursday: ["00:00-23:59"]
friday: ["00:00-23:59"]
saturday: ["00:00-23:59"]
sunday: ["00:00-23:59"]
maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # default false
decreaseEnabled: true # default true
minimumCores: 0.1 # default unset, to override to unset, use -1
maximumCores: 7.5 # default unset, to override to unset, use -1
limit:
keepLimit: true # default true
memoryManagement:
request:
increaseEnabled: true # default false
decreaseEnabled: true # default true
minimumGib: 0.1 # default unset, to override to unset, use -1
maximumGib: 15.1 # default unset, to override to unset, use -1
limit:
increaseEnabled: true # default false
decreaseEnabled: false # default false
minimumGib: 2 # default unset, to override to unset, use -1
maximumGib: 31.1 # default unset, to override to unset, use -1
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 4 # default 3
weekly: 7 # default 7
```
## Time constraints
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: example
spec:
automation:
operational:
timeConstraints:
wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
maintenanceWindow:
sunday: [00:00-23:59]
maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
```
### :wastebasket: Maximum automation frequency for waste
{% hint style="danger" %}
`maxAutomationFrequency` field is deprecated as of autoscaler version **v1.0.16**. If `maxAutomationFrequency` is present and `wasteMaxAutomationFrequency` is not specified, `maxAutomationFrequency` will override `wasteMaxAutomationFrequency`.
{% endhint %}
This optional field defines the maximum frequency at which automated actions can be executed to apply recommendations when waste is detected, and resource reduction is needed.
{% hint style="info" %}
If no `wasteMaxAutomationFrequency` is specified, PerfectScale defaults to values based on the type of workload:
Deployment: "30m" \
Rollout: "30m"\
DaemonSet: "4h"\
StatefulSet: "24h"\
CronJob: "30m"\
Job: "30m"
{% endhint %}
### 🚨 Maximum automation frequency for waste resiliency
{% hint style="danger" %}
`maxAutomationFrequency` field is deprecated as of autoscaler version **v1.0.16**.
{% endhint %}
This optional field defines the maximum frequency at which automated actions can be executed to apply recommendations when resiliency risks are detected, and resource increase is needed.
{% hint style="info" %}
If no `fixResiliencyMaxAutomationFrequency` is specified, PerfectScale defaults to values based on the type of workload:
Deployment: "30m" \
Rollout: "30m"\
DaemonSet: "4h"\
StatefulSet: "24h"\
CronJob: "30m"\
Job: "30m"
{% endhint %}
### :clock10: **Maintenance window**
The maintenance window defines the timeframe for PerfectScale to implement the recommendations.
The maintenance window is particularly helpful in minimizing the impact of changes on business operations and provides a structured approach to system updates and changes.
{% hint style="info" %}
Time is indicated in UTC.
{% endhint %}
**Example 1**
The automation actions are scheduled daily between 00:00 and 23:59 UTC:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: example1
spec:
automation:
operational:
timeConstraints:
maintenanceWindow: #UTC
monday: ["00:00-23:59"] # <-- Implicit Default Value
tuesday: ["00:00-23:59"]
wednesday: ["00:00-23:59"]
thursday: ["00:00-23:59"]
friday: ["00:00-23:59"]
saturday: ["00:00-23:59"]
sunday: ["00:00-23:59"]
```
**Example 2**
Excluding specific days from the automation maintenanceWindow. \
To exclude a specific day(s) from an automation maintenanceWindow, set the timeframe to `00:00-00:00` (e.g., Monday - Friday in the example above).
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: example2
spec:
automation:
operational:
timeConstraints:
maintenanceWindow: #UTC
monday: ["00:00-00:00"]
tuesday: ["00:00-00:00"]
wednesday: ["00:00-00:00"]
thursday: ["00:00-00:00"]
friday: ["00:00-00:00"]
```
{% hint style="info" %}
In the example above, the maintenanceWindow does not specify a timeframe for Saturday and Sunday. As a result, these days default to the implicit default value of `00:00-23:59` .
{% endhint %}
**Example 3**
The automation actions are scheduled for the end of one day and the beginning of the next.
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: example3
spec:
automation:
operational:
timeConstraints:
maintenanceWindow:
monday:
- "00:00-02:30"
- "23:30-23:59"
tuesday:
- "00:00-02:30"
- "23:30-23:59"
wednesday:
- "00:00-02:30"
- "23:30-23:59"
thursday:
- "00:00-02:30"
- "23:30-23:59"
friday:
- "00:00-02:30"
- "23:30-23:59"
saturday:
- "00:00-02:30"
- "23:30-23:59"
sunday:
- "00:00-02:30"
- "23:30-23:59"
```
{% hint style="danger" %}
**Example of a Potential Syntax Error:**\
The following configuration is invalid because time intervals must be contained within a single day. The example below ranges from late Monday into early Tuesday:
```
spec:
automation:
operational:
timeConstraints:
maintenanceWindow:
monday:
- "23:30-02:30"
```
{% endhint %}
### :person\_gesturing\_no: Minimum resiliency level to override maintenance window
{% hint style="danger" %}
`maintenanceWindowIgnoredForResiliency` is deprecated.
{% endhint %}
In some cases, you may want Automation to override the maintenance window to ensure that critical resiliency issues, such as CPU throttling or OOM occurring outside the maintenance window, are properly addressed by an immediate resource increase.
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: example
spec:
automation:
operational:
timeConstraints:
maintenanceWindowIgnoredMinResiliencyLevel: None
```
This field indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue. When set to `None` (default), the maintenance window constraints are enforced. However, when one of the resiliency levels is selected (`Low`, `Medium`, `High`), PerfectScale Automation will ignore maintenance window constraints for risks with the indicated resiliency level and above, applying changes to resolve the issue.
**Example 1**
The maintenance window will be ignored for risks at any level.
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: example1
spec:
automation:
operational:
timeConstraints:
maintenanceWindowIgnoredMinResiliencyLevel: Low
```
**Example 2**
The maintenance window will be ignored for risks with `Medium` and `High` risk levels
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: example2
spec:
automation:
operational:
timeConstraints:
maintenanceWindowIgnoredMinResiliencyLevel: Medium
```
**Example 3**
The maintenance window will be ignored for `High` risks only.
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: example3
spec:
automation:
operational:
timeConstraints:
maintenanceWindowIgnoredMinResiliencyLevel: High
```
## 💡Keep CPU limit
In general, we recommend removing CPU limits when possible. Doing so can improve the performance and efficiency of your workloads by allowing them to use available CPU resources more freely.
However, be aware that removing misconfigured CPU limits can increase memory consumption in many cases. This requires **adjusting** both **memory requests and limits** to ensure **optimal performance** and **efficient use** of resources.
{% hint style="success" %}
We **highly recommend** that removing CPU limits always be combined with **enabling `increase allowed` memory settings**. This allows the system to automatically adjust memory resources to meet the increased demands, maintaining the stability and performance of your workloads.
{% endhint %}
## :arrows\_counterclockwise: Memory increase iteration limit
`maxMemoryIncreaseIterations` sets the maximum allowed number of daily and weekly memory limit increase iterations.
{% hint style="info" %}
`max_memory_increase_iterations` is a mandatory field. Make sure it exists and doesn't equal to 0.
{% endhint %}
Once the number of daily or weekly memory increase iterations reaches the maximum value, to prevent potential memory leaks, PerfectScale Automation will stop recursive memory increase.
In order to notify the customer, PerfectScale will raise a relevant indicator and create an [Alert](/configure-alerts/alerts-overview).
## ⚙️ Optimization policy
PerfectScale offers flexible policy customization, giving you greater control over how optimization is applied.
With this feature, you can:
* Configure independent optimization policies for CPU and Memory.
* Apply a Memory Request = Limit policy for memory-sensitive workloads.
* Define custom time windows when the default setup is not sufficient due to known and predictable resource spikes.
These options enable you to tailor optimization behavior to your workload requirements, reliability constraints, and operational preferences while maintaining efficient and predictable optimization.
Learn more about optimization policy customization [here](/customize-workflow/optimization-policy-customization).\
Additional YAML Examples can be found [here](https://github.com/perfectscale-io/perfectscale-io.github.io/tree/main/charts/psc-autoscaler/examples).
# Disable automation with CRD
Learn how to disable Automation
PerfectScale Automation is a powerful feature that streamlines and optimizes your infrastructure management. However, there may be scenarios where you need to temporarily or permanently disable automation. Choose one of the methods provided to deactivate Automation smoothly, depending on your needs.
## Disabling automation for the namespace/workload
This method enables you to selectively exclude specific namespaces and workloads within your cluster from automation.
Set a flag `spec.automation.operational.automationMode =`` `**`Disabled`** in the namespace/workload automation configuration.
{% hint style="info" %}
The **workload** settings **take precedence over** the settings of the **namespace** or **cluster** (the **namespace** settings **take precedence over** the **cluster's**). This feature allows enabling automation for the entire namespace or cluster while allowing for specific workloads to be disabled within this namespace/cluster.
{% endhint %}
Use one of the following configurations to **exclude** the particular **namespace** or **workload** from automation in the cluster.
### **Exclude namespace**
```yaml
apiVersion: perfectscale.io/v1
kind: NamespaceAutomationConfig
metadata:
name: your-namespace-automation-config
spec:
automation:
operational:
automationMode: Disabled
```
### **Exclude workload**
```yaml
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: your-workload-automation-config
spec:
automation:
operational:
automationMode: Disabled
targetRef:
kind: Deployment
name: deployment-name-here
```
### **Exclude workload by label**
```yaml
apiVersion: perfectscale.io/v1
kind: NamespaceAutomationConfig
metadata:
name: your-namespace-automation-config
spec:
automation:
operational:
workloadLabelSelectors:
- key: "environment"
value: "production"
allowAutomation: false # Disables Automation for workload with the specified label
```
## Disable automation for the cluster
To disable automation for the entire cluster, use one of the following options:
* Set a flag `spec.automation.operational.StopAllAutomation =`` `**`true`** in the cluster automation configuration.
* Manually delete cluster automation configuration.
* Uninstall PerfectScale Automation Agent.
### StopAllAutomation
This method allows you to fully disable automation across the entire cluster, overriding any existing automation settings for specific namespaces or workloads.
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: cluster-automation-config
spec:
automation:
operational:
stopAllAutomation: true
```
## Cleanup automation
In certain scenarios, you may need to disable automation and revert any changes it has applied, returning your cluster to its original state based on the specified configuration.
To achieve this, set the `cleanupAllAutomation` parameter to `true`. This will not only stop automation but also roll back any modifications made by the automation processes, restoring the original resource specifications and settings.
{% hint style="info" %}
`cleanupAllAutomation` overrides any existing automation settings for specific namespaces or workloads.
{% endhint %}
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: cluster-automation-config
spec:
automation:
operational:
cleanupAllAutomation: true
```
## Uninstall the automation agent
To uninstall the PerfectScale Automation Agent, execute the following command
```yaml
helm -n perfectscale uninstall psc-autoscaler
```
# Verifying Automation status
Verify the status of Automation to ensure it is running properly
Visit the [Overview](/visibility-and-optimization/clusters-metrics-overview#automated) screen to verify the Automation status of each cluster. This section provides a dedicated column that displays the cluster's Automation status and the number of automated workloads.
For more detailed information on specific workloads, navigate to [Podfit](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#automation-status), where you can check the Automation status of individual workload(s).
Status
Indicator
Description
Active
Automation is up and running.
Limited by User
Indicates that automation is active but limited by a user-defined rule. Learn more about resource allocation constraints here.
Limited by System
Indicates that automation is active but limited by a system safeguard. Learn more about resource allocation constraints here.
Limited by System and User
Indicates that automation is active but limited by both system and user-defined rules. Learn more about resource allocation constraints here.
Not Enabled
-
The workload is not enabled for automation, but can be automated.
Disabled by User
Indicates that the automation is available but intentionally disabled for the workload by the user.
Example: if automation is enabled at the cluster level but disabled at the namespace level, the namespace setting takes precedence, and automation is disabled for those workloads.
Not Supported
Automation is not supported for the given workload type.
# Rollout-aware Automation
Explore how PerfectScale’s rollout-aware automation syncs continuous optimization with continuous deployment, ensuring safe optimization across advanced K8s
{% hint style="info" %}
Autoscaler version v1.0.32 or later is required.
{% endhint %}
PerfectScale automatically detects your Argo Rollouts strategy, whether Blue-Green, Canary, or A/B, and aligns optimization with your deployment approach to ensure continuous feature delivery without compromising K8s efficiency or stability.
Rollout-aware automation enables instant, safe optimization across even the most advanced and diverse Kubernetes environments, ensuring your clusters stay stable and reliable.
## Automation behaviour policy for rollout ReplicaSets
The Automation behavior policy provides you control over the optimization process, enabling you to define how Automation should act if new recommendations arrive during an active rollout. This ensures that Automation consistently drives desired outcomes without contradicting the strategies you use to innovate and seamlessly deliver value.
{% hint style="success" %}
Automation behavior settings can be configured at the cluster, namespace, or workload level.
{% endhint %}
To set the automation behavior policy when rollout is in the active phase, specify the desired policy in the Automation Configuration CR:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: cluster-automation-config
spec:
automation:
operational:
duringRolloutInProgress: pause # A default behavior that prevents Automation from applying new recommendations when concurrent ReplicaSets are identified
workloadTypes:
Rollout:
operational:
automationMode: "Enabled" # Enables rollout support
```
### Aggregate **ReplicaSets**
When the **merge** rollout Automation behaviour is set, PerfectScale calculates recommendations based on the merged utilization of all ReplicaSets with the same R1, and applies new recommendations for all the ReplicaSets of the same R1.
```yaml
duringRolloutInProgress: aggregate
```
### **Pause Automation**
**Pause** (default) behavior prevents Automation from applying new recommendations when concurrent ReplicaSets are identified.
```yaml
rolloutAutomationBehavior: pause
```
When `rolloutAutomationBehavior: pause` and multiple concurrent ReplicaSets are detected, Automation will not take any action, and a corresponding tooltip will be displayed:
{% hint style="warning" %}
**Automation Limited by User-Defined Rule**. Automation restricted: Resource changes are not allowed when multiple concurrent ReplicaSets are detected.
{% endhint %}
# In-place workload right-sizing
Autonomous workload optimization without restarts
{% hint style="info" %}
Kubernetes version **1.33 or later**, with the **beta feature gate enabled**, is required to support in-place changes.
{% endhint %}
## In-place concept
The standard pod scaling approach requires pods to restart when applying new resource requests and limits. While this can work for stateless applications, it may still introduce latency and service disruptions, making it less suitable for workloads that demand continuous uptime. In addition, this method may add operational overhead for teams to manage it effectively.
To address these challenges, **In-place Workload Right-sizing** is available. This feature is ideal for services where stability is critical, allowing you to improve performance seamlessly, reduce disruption risks due to restarts, and make the optimization process even more flexible and efficient.
With in-place changes enabled, you can:
1. **Keep infrastructure reliable** by eliminating the risks associated with restarts.
2. **Stay continuously optimized** without service disruption.
3. **Maintain operational productivity** without the overhead of time-consuming manual investigation and optimization.
You can find more details about in-place changes on the official Kubernetes website:
{% embed url="" %}
{% @github-files/github-code-block url="" %}
## Prerequisites
To start leveraging in-place workload right-sizing, your cluster must support in-place changes.
🎯 Update Kubernetes to **version 1.33 or later** and **enable the beta** `InPlacePodVerticalScaling` [**feature gate**](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) for your control plane and for all nodes in your cluster. After that, PerfectScale will automatically perform in-place workload right-sizing without additional effort from your side.
{% hint style="success" %}
No additional configurations are needed, as PerfectScale automatically applies changes in place, delivering instant optimization.
{% endhint %}
## In-place resizing limitations
Limitation
Description
Supported workload types
Deployment
StatefulSet
DaemonSet Rollout
Resizable resources
CPU
request
limit
Memory
request
limit (increase only)
Operating system
Windows pods do not support in-place
Kubernetes version
1.33 or later
Container resize policy
restartPolicy: NotRequired (apply the resource change to the running container without restarting it). If restartPolicy: RestartContainer - in-place changes will not apply.
QoS Class
The Pod's original QoS class is determined at creation and cannot be changed by a resize.
## In-place right-sizing with PerfectScale Automation
In-place pod resizing combined with PerfectScale Automation provides a complete solution for graceful and effective workload right-sizing. It eliminates manual effort associated with identifying inefficiencies and risks due to guestimated allocations, enabling continuous safe cost-reduction.
{% hint style="info" %}
If in-place resizing is not supported, PerfectScale’s safe automation algorithm will apply changes with a traditional scaling approach, ensuring your clusters remain reliable and stable.
{% endhint %}
For automated pods that support in-place changes, when an over-provisioned or under-provisioned workload is detected, PerfectScale instantly applies data-driven recommendations without pod restart, ensuring safe K8s optimization.
{% hint style="success" %}
When PerfectScale applies in-place changes to right-size workloads, **it unifies pod resources, guaranteeing consistency of the resources** across the cluster.
{% endhint %}
### PodResizePending handling
You may encounter a situation where it is impossible to apply new resources to a pod. In this case, the pod will get the **PodResizePending** status. This can happen due to:
1. The requested resize **is impossible** on the current node (for example, requesting more resources than the node can provide).
2. The requested resize **is not currently possible**, but may become possible later (for example, if another pod is removed and resources become available).
To handle such cases, you can configure the `PodResizePending` strategy in the automation configuration CR by specifying one of the following values:
Value
Description
rollingRestart (default)
All pods will be restarted in order to update resources.
deleteRecreatePendingPod
Only pods in the PodResizePending status will be deleted to trigger rescheduling.
Conditions: :circle-right: The number of pods in the PodResizePending state is below the 20% threshold. :circle-right: At least 5 pods were successfully updated
If any condition fails, a rollingRestart will be applied.
**CR example:**
```yaml
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: workload-automation-config
namespace: your-namespace
spec:
targetRef:
kind: Deployment
name: pacmem-inplace-resize-burstable-qos
automation:
operational:
automationMode: Enabled
automationStrategy: inPlace
podResizePendingStrategy: rollingRestart # configured PodResizePending strategy
duringRolloutInProgress: pause # a default behavior that prevents Automation from applying new recommendations when concurrent ReplicaSets are identified
```
{% hint style="success" %}
The **Automation Strategy** can be configured at the **cluster**, **namespace**, or **workload** level.
{% endhint %}
## Reverting to the evictMutate strategy
Once the agent is updated to **v1.0.26 or later**, PerfectScale **defaults to the In-Place** automation strategy. If a customer wants to opt out of In-Place automation and revert to the **evict + mutate** behavior, this must be explicitly configured in the Automation CR at the desired level (cluster, namespace, or workload):
```yaml
automationStrategy: evictMutate
```
# Injected containers handling
Explore PerfectScale’s approach to handling injected (immutable) containers
PerfectScale identifies sidecar containers within pods. Due to the immutable nature of sidecar container resource allocations, Automation does not apply changes to sidecar containers. Instead, PerfectScale focuses on pod-level recommendations, ensuring safe and effective optimization without impacting sidecar functionality.
When the injected sidecar is detected:
1. PerfectScale will indicate such container as **`injected sidecar`**.
2. **Limited by Rule** Automation status will be raised for the automated workloads.
3. Automation executes pod-level recommendations only.
4. Automation won’t make changes that could trigger HPA (if such is introduced) due to under-provisioned sidecars. Current values are kept to maintain stability.
## Injected sidecar is detected
When PerfectScale identifies the immutable container within the pod, the Automation status will be updated to the **Limited by Rule**, and `injected sidecar` badge will appear for this container.
Injected sidecar badge
In this case, PerfectScale Automation will not apply the recommendations, and a tooltip will appear next to the resource utilization chart and recommendations section.
## HPA and Sidecar Detected
When an immutable container is detected in a pod with HPA enabled, Automation will also be restricted for the other containers in that pod. This prevents triggering the HPA due to under-provisioned sidecar allocations and ensures workload stability.
In this case, PerfectScale retains the current values in order to maintain stability and avoid potential risks.
Automation is restricted
# Exploring Automation actions and impact
Automation performance evaluation offers insights into automation-driven actions, highlighting their impact on resource optimization and overall efficiency
The Automation screen offers a detailed overview of all actions PerfectScale Automation performs across your environment, highlighting their impact on resource optimization. This view provides clear insights into automation-driven optimization results and helps seamlessly evaluate its efficiency.
With the provided level of information granularity, you can effortlessly review a detailed breakdown of the actions taken by Automation. This includes insights into the specific tasks executed, the containers affected, and the precise timestamps indicating when each action occurred. Such comprehensive visibility ensures thorough oversight and facilitates further analysis.
{% hint style="info" %}
The automation screen can only be accessed by users with the [Automation feature enabled](/enable-automation/automation-setup-instructions). Request access to Automation directly from the platform or contact us through [**support@perfectscale.io**](mailto:support@perfectscale.io) or [**PerfectScale Slack Community**](https://perfectscalecommunity.slack.com/)**.**
{% endhint %}
Automation KPIs screen
### Exploring automation performance
PerfectScale provides a detailed and transparent view of automation performance, enabling you to evaluate the progress of automation-driven optimization effectively. This comprehensive insight enables you to monitor key metrics and understand how automation improves resource allocation, enhancing the stability and cost-effectiveness of your environment.
#### Addressed Resiliency Issues
Resiliency issues widget
This widget highlights the resiliency risks that PerfectScale's automation has addressed within the selected time frame. In the top right corner, you can see the total count of risks addressed during this period.
To view the specific number of risks resolved on any given date, hover over the corresponding bar in the chart.
This view allows you to seamlessly evaluate the optimization actions performed by automation to enhance the stability and resiliency of your Kubernetes clusters.
#### CPU optimization
CPU optimization widget
The CPU optimization widget displays changes in CPU requests over time, giving you a detailed view of how Automation has optimized CPU allocation for workloads and their replicas. It provides a detailed visual comparison of the adjustments made against the original configurations, offering transparent insight into the efficiency improvements achieved during the selected period.
To check the details of CPU optimization performed by Automation on a specific date, hover over the corresponding bar in the chart.
**Original Requests**
The original requests reflect the sum of CPU requests from the initial configurations of all workloads and their replicas at the selected time.
**Optimized Requests**
The optimized requests represent the sum of CPU requests allocated by Automation for all workloads and their replicas at the selected time.
**Impact**
Impact quantifies the difference between the CPU requests from the original configuration and those allocated by Automation, providing clear visibility into how effectively Automation optimizes CPU allocation for your clusters.
#### Memory optimization
Memory optimization widget
The Memory optimization widget displays changes in Memory requests over time, giving you a detailed view of how Automation has optimized Memory allocation for workloads and their replicas. It provides a detailed visual comparison of the adjustments made against the original configurations, offering transparent insight into the efficiency improvements achieved during the selected period.
To check the details of Memory optimization performed by Automation on a specific date, hover over the corresponding bar in the chart.
**Original Requests**
The original requests reflect the sum of Memory requests from the initial configurations of all workloads and their replicas at the selected time.
**Optimized Requests**
The optimized requests represent the sum of Memory requests allocated by Automation for all workloads and their replicas at the selected time.
**Impact**
Impact quantifies the difference between the Memory requests from the original configuration and those allocated by Automation, providing clear visibility into how effectively Automation optimizes Memory allocation for your clusters.
### Automation Actions
Name
Description
Started at
The timestamp indicates when the automated action was initiated.
Cluster
Indicates which cluster the automated action was taken in.
Namespace
Indicates which namespace the automated action was taken in.
Workload
Indicates which workload the automated action was taken in.
Type
Indicates the type of workload in which the automated action was taken. 💡Automation is currently available for workloads of the following types: Deployment, DaemonSet, StatefulSet, CronJob, and Job.
Container
Indicates which container the automated action was taken in.
Executed
Indicates automation execution type: ➣ Via Restart - regular eviction with a restart. ➣ In-Place - in-place resizing without a restart. ➣ Cleanup - automation action caused by enabling the cleanupAllAutomation in a cluster automation configuration
CPU Request
Displays the CPU request change if any action was taken.
CPU Limit
Displays the CPU limit change if any action was taken.
Memory Request
Displays the Memory request change if any action was taken.
Memory Limit
Displays the Memory limit change from its previous value to the new one if any action was taken.
{% hint style="info" %}
To access full data and examine the details, click the log and navigate to the [PodFit Zoom-in](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#zoom-in-window) window.
{% endhint %}
{% hint style="warning" %}
Actions associated with deleted or disconnected clusters remain visible for historical reference. Rows for these clusters are grayed out, and navigation to the Zoom In view is disabled because the cluster is no longer being observed by PerfectScale.\
{% endhint %}
### Personalize Automation Audit Log view
PerfectScale empowers you to customize your Automation Audit Log view, enabling you to focus on what matters to you.
{% hint style="info" %}
To customize tables and focus on the data that matters most, use the Columns menu. Learn more about table customization [here](/customize-workflow/tables-customization).
{% endhint %}
:top: To sort the data in the desired order, simply click on the title of the column.
📌 To filter the data by a specific value, use a drop-down list at the top of the columns.
Available filters:
* Cluster
* Namespace
* Workload
* Type
* Container
📐 Choose your preferred format for displaying changes to resources with the advanced filters
Recommendations formats
* Detailed - to display the changes made to resources (shows both the previous and new values).
* Total Changes in Units - to display changes made to resources as an absolute number, factoring in replica count.
* Single Instance Changes in Units - to display changes made to resources as an absolute number.
* Single Instance Change in % - to display changes made to resources in a percentage format.
{% hint style="info" %}
Hover over the results to get more details on the resource changes
{% endhint %}
### Export the report
The export feature gives you the ability to export your data into a `.csv` file for seamless analysis and effortless sharing. Click the **`Export`** button, and the data will be exported in a second.
Report export
# Automation with ArgoCD
A quick guide on how to seamlessly integrate PerfectScale with your GitOps solutions
PerfectScale Automation can be seamlessly integrated with GitOps solutions such as ArgoCD and Flux. However, to avoid any potential issues due to different configurations, we recommend reviewing the documentation below.
## Certificates regeneration loop
In the Autoscaler Helm chart, certificate generation occurs whenever ArgoCD executes the `helm template`. This process may lead to temporary outages of the Autoscaler due to the caBundle and the certificates/keys being out of sync. To prevent this issue, we recommend generating a static caBundle and certs using the following instructions.
### How to generate static Certs for Autoscaler
To create a static certificate, follow a few simple steps:
1. Generate Certs using the usual helm template command.
{% hint style="info" %}
The certificate is valid for 1825 days.
{% endhint %}
```
helm template -n perfectscale psc-autoscaler perfectscale/psc-autoscaler > psc-autoscaler.yaml
```
2. Get Ca Certs from the generated template.
```
cat psc-autoscaler.yaml | grep caBundle | head -n 1 | awk '{print $NF}'
exemple output: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJekNDQWd1Z0F3SUJBZ0lRTWllcDBjbVNuWk1GRjE0SXRuQ3E0ekFOQmdrcWhraUc5dzBCQVFzRkFEQWMKTVJvd0dBWURWUVFERXhGd2MyTXRZWFYwYjNOallXeGxjaTFqWVRBZUZ3MHlOREE1TWpZd016SXlNVEZhRncweQpPVEE1TWpVd016SXlNVEZhTUJ3eEdqQVlCZ05WQkFNVEVYQnpZeTFoZFhSdmMyTmhiR1Z5TFdOaE1JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTdlbTlSMFlmckRpVmtEdXR4RUpEeXllQ0lFQW4KdllZZVNYb2RKNU80Q1VQakRwZHNvdEc3WlRuL00wd0lqSVVMSUszSnhjYmJPUkFBU3haTHJFU1lTM010MlZKZQpSL3pEU2lRM0ROUEV4Z1k2NDArdjVmWmwrN2xnM2dUd2tPRms4U1A1VEpjN3ViTUxFZFJGWGtkOTlhS2F6VDJKCldia3N3bEx0YTUzbmZQSUNmLzFnd2xKZVlZYTRMZUxHY1dQclRKUWorcXpYME9CdWg0cjhQNzRCT3lrOEhGUUgKTlk4L05mczZJQ2c1d2lJcjRJV2pvd05SZS9NUW14SVhwaWtJV21IallQUUV1R1BMWElUYnc4cGNvVS8wajJrYQpZN1c3UDBsdWlRWDV6Yzd1TlRrN3NlTE85N2xLNWY5eWJ4QnJUcHRaeXM3azA2MllvYm9hamZCbkh3SURBUUFCCm8yRXdYekFPQmdOVkhROEJBZjhFQkFNQ0FxUXdIUVlEVlIwbEJCWXdGQVlJS3dZQkJRVUhBd0VHQ0NzR0FRVUYKQndNQ01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZDNXNQTVpQUDFYV2Y1cmZZM0dGVVhuTgpZSDRGTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFER0NvTDNhdTZRS2F3ZjZnYmRCdGprVFJLb1pPMzBzZkpVCnRJUUVpb1loZlBPaGdiZm1IbFVHS2NYS2NYdFF4dWJXY2dpYUNjeUpqMk9sb2MwWXpnbHVETGJONHVpVGlNcnEKcUkvMWU2N213YUJvQmtBdkU2UWVkRmxYNmlMeVhpM0wxNEFEUDQ1eHhIRm13RzRmRlRzQktvU09uMDF5YW10ZwpwR3lzeWN4QXpET0pYQTYvL0pmd1grT3p1c0lUREVEUXdDWWxZdEZGbnlvdjZtRlFKWjZtbFROMDdhVng1STEzClVXRUlSUTJDVEJkM08zOGc5dkJKYUYvUHQ4YzZqUGZFQW43ZjRsSVM3dVBoU0NJc0FYbS93Q1Fnc29OTStKRHEKOWIrUTZDUGRma1dPdXo5M0RBVmN0T3B6bmJFR0JHcW1oc29GMjRseUJ5dVUzQXVTa2QzOQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
```
3. Get Crt Certs from the generated template.
```
cat psc-autoscaler.yaml | grep "tls.crt:" | awk '{print $NF}'
exemple output: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURhekNDQWxPZ0F3SUJBZ0lSQU1xRlFBVDRtcmJhcWNqR1lwWHVjcUV3RFFZSktvWklodmNOQVFFTEJRQXcKSERFYU1CZ0dBMVVFQXhNUmNITmpMV0YxZEc5elkyRnNaWEl0WTJFd0hoY05NalF3T1RJMk1ETXlNakV4V2hjTgpNamt3T1RJMU1ETXlNakV4V2pBWk1SY3dGUVlEVlFRREV3NXdjMk10WVhWMGIzTmpZV3hsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxZeXFTSEVLRExHdmV3T0NBTmdLRUt4cm1uTXlSaEYKTzZuRUJHdFluOTNwMXovd0pJVHZ4cEpJNmRwamVWMzZyREREU0R5cnlYbndkYXBjcHU4WUdMWHhxd0xrUVpkSgpkRExlNVc0MVQrZXNOMGpyUWZqaFk1L2xRODdkemFlY1RleEFYNkFiUG9Uc1h5Q0N0OTU3aEdCdWN5STdoSGoyCmxiMm4yRlREVi93czRwbFRwVW56dmw0QWJvUS9HTTZWdmRBdUZUV3B4SDRmOXliYjYrL0pxSVFXWkJqL0ZIK3YKS3l3NVY5QWVWY3BkME9JMVhyd3lyampvLzJjRy9ONUNKUnN2T0l0WXhnZy9mek94cjZVWDlibmgweVdWSk1uTAoxUGdxU0tONzk1S3J0ckI0dzVPYlMvVmo3c0J3WllSRGk1VENiTUNtUm5JMHZzTWY0dXE2cms4Q0F3RUFBYU9CCnFqQ0JwekFPQmdOVkhROEJBZjhFQkFNQ0JhQXdIUVlEVlIwbEJCWXdGQVlJS3dZQkJRVUhBd0VHQ0NzR0FRVUYKQndNQ01Bd0dBMVVkRXdFQi93UUNNQUF3SHdZRFZSMGpCQmd3Rm9BVUxtdzh4azgvVmRaL210OWpjWVZSZWMxZwpmZ1V3UndZRFZSMFJCRUF3UG9JYmNITmpMV0YxZEc5elkyRnNaWEl1Y0dWeVptVmpkSE5qWVd4bGdoOXdjMk10CllYVjBiM05qWVd4bGNpNXdaWEptWldOMGMyTmhiR1V1YzNaak1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ2YKMnQvTVJucU1BMW1BTzkyZGtpQzFIQXh4N2RyWUxwUnE3amVCZVFHMGQrMURvYmRhakhiZzNTc3RFTG53L1hUZAo4V0hIcytPckNQL2lBbkpDaVlCOFVJK2F3NU9jY1lUeUR2eEZod3hrZzFndEdXcGQvczMreThQYU9EVHAyVS9hCkhjMXQ2U3FUSWF2WXE2UXhESzNVc2lYanVBa2orQk5PMGZ0NW9CY1lDZzJpZERtcXpKS1dzdjQwM0RHWmw0S3gKazNPRWNqVUY1YTZFaURRTTZ2VEQwQVFGVzRHeXprZS9ua29LdlJSZHJyUytibWFCSEwyeS9BQkxOdG9DOXZKVQpnU2N2K2hjL0VvdktZeFg4cDZ4dkd0RjMzeXhOb3RjdjV2WGJSeW1iYTFLQUg5cDZ4aFFDRGpyUVFEUHVSZElUCktETDI5d3FjdFJCc2pZUzdjN3R4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
```
4. Get Key Certs from the generated template.
```
cat psc-autoscaler.yaml | grep "tls.key:" | awk '{print $NF}'
exemple output: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdGpLcEljUW9Nc2E5N0E0SUEyQW9Rckd1YWN6SkdFVTdxY1FFYTFpZjNlblhQL0FrCmhPL0dra2pwMm1ONVhmcXNNTU5JUEt2SmVmQjFxbHltN3hnWXRmR3JBdVJCbDBsME10N2xialZQNTZ3M1NPdEIKK09Gam4rVkR6dDNOcDV4TjdFQmZvQnMraE94ZklJSzMzbnVFWUc1eklqdUVlUGFWdmFmWVZNTlgvQ3ppbVZPbApTZk8rWGdCdWhEOFl6cFc5MEM0Vk5hbkVmaC8zSnR2cjc4bW9oQlprR1A4VWY2OHJMRGxYMEI1VnlsM1E0alZlCnZES3VPT2ovWndiODNrSWxHeTg0aTFqR0NEOS9NN0d2cFJmMXVlSFRKWlVreWN2VStDcElvM3Yza3F1MnNIakQKazV0TDlXUHV3SEJsaEVPTGxNSnN3S1pHY2pTK3d4L2k2cnF1VHdJREFRQUJBb0lCQUZ6SVE3VDhUWFErSFMxKwpuN29JVjdoTy80REZWb3JUaHVTVys2RW15cEtacUpIWTJ1QVBzdEYvN0FiWXJVK2NmZDkvTXVRaW4wRkhvTVFXCjFyUGZVNFlrQVhCQmlOMWYvSnFEUTlieHN6VkpVQVlFTjk1aitQWE9MTS9GeUJyWTBCM0ZVbTBZTXFoQ0x0ZEIKZWgwRVRZOUY3Nys1QzJjR080OFBVRzBwNVQ1M1NMNXdBNWN3UFNkakRVdGxlNVpCZ2kvNU1UMVNmU1owV1g0SAozOVIrdm1LTElsZHpkS0ZYWnRTOFNlbS9qTHcwU2s5dTl4VXdsRDZ5OUJXb0QwSm5DOUtYdDJTS1Jpekk4SUNoCkIvU3ZaMkhXblVVTUVxSkhxYXYvaGs2amJRR1p1eUJTWmNkSFV4WXJtTUl2L3VhdmcyRHNrekVYNDI2R016aTMKRmNUUkhURUNnWUVBNVFIVHdacjBtSUo4bmdJQVVqUjduOW80aUtPRmVCZmN0YmFFLythTnF4ZkN1bHdRT2RJTgoxQW03SFl6M1JGWGtwQkJSc09ERkd5OHlOaCtYS2lRTnJ5TzUvNGNmdHNtVENLTXBRMUhVTjV5UXg2VDZMemtJCkNrQWJrOHhLWVk5OXpPZnFKYThueFlrWFRoWXJ6eWR2ODJxT1JrZ1FqSFdHeE1vVlJmaEZxdFVDZ1lFQXk2eGoKZWlSRVNkWHQyU1I1TENuTG9Ld29NbHcwTkNjcStJVFBSb1FoQTJHTHF0S2dqVHExWEFNZ2ZNUTVFenlIK2JoZAo5ZVlMK2pxYzRsZVBORk0zYjI4YnJQNmdPV1VBeE5xK0JXY2JGblNYMmc5Y2FrOVk2TllsMnFQbzZzUGdKSVpaCkIyV1o0Q0NkazVDNENabmh0VGZIOW5DbEorT0p2Z2NEMkFieVBwTUNnWUVBNGMySGFtTEx3b1VITXhWdUU2QkUKTlQ0S2VGQWtEbUlZOGVOaUNLRGdGMDk2Wmg3WXNPcDVVQlc0Tms4RWtKRFF2U1NLS0FtNDBERWRKVmpJeUhoOApOYkRZZVNoWEVQaDR6Zzl6WkRtUE1yWmgyempFcDBOb3RoSzRIbWRWQUJWb1NWNXk5VzFibmQ2cTZBZmNjb25FClRSTkg3ZXlveEpZdHFHUkp6WkFid1hFQ2dZRUFzZ1djOFk2dmphWUtma2l1RFJiL0lCSTZGRUtaY3BobHE5RlIKNXYzb2FTOGl5QTNRUFlGNk1kR2QzTkFhWHlFQ2JSaGtNR0dDU0paZ3VFbjJKck14K2htd3BCWHZjQnREZjh6KwpXSFNWclp2OWVxRUFrditGNUFZQ0VraEFkcXI4eUpBUlV6RFFQYUhrZWEzOTNvNTlSVXRiWWFGMkNoYU5OK2FqCnluZlAxemNDZ1lBM3hrYWlvNkhpMmVjUkp6RGNWd1dvQ1VycWF0bHhCSVJjQmNtZ1h6WTZKNVVvSG9nWVZNcjkKRGs0Y3B1YXZnNklmRXM5Y0NPTnNEOC9YUDNZRmM4Mm4vUy9IdW10N0ZUdmJQcmc5eHdWWEFxcEExOVUxUXVkVApkSkZ1cWZWWjVzTXdCRXUyUjFjckJYNWNVQzJzVUE1eVhmaUpZb3FwTkFIOTZwQTJna010Vmc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
```
5. In values.yaml set **admission.staticCerts.enabled** to **true** and **copy/past certs**
```yaml
admission:
failurePolicy: "Ignore"
staticCerts:
enabled: true
caCert: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJekNDQWd1Z0F3SUJBZ0lRTWllcDBjbVNuWk1GRjE0SXRuQ3E0ekFOQmdrcWhraUc5dzBCQVFzRkFEQWMKTVJvd0dBWURWUVFERXhGd2MyTXRZWFYwYjNOallXeGxjaTFqWVRBZUZ3MHlOREE1TWpZd016SXlNVEZhRncweQpPVEE1TWpVd016SXlNVEZhTUJ3eEdqQVlCZ05WQkFNVEVYQnpZeTFoZFhSdmMyTmhiR1Z5TFdOaE1JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTdlbTlSMFlmckRpVmtEdXR4RUpEeXllQ0lFQW4KdllZZVNYb2RKNU80Q1VQakRwZHNvdEc3WlRuL00wd0lqSVVMSUszSnhjYmJPUkFBU3haTHJFU1lTM010MlZKZQpSL3pEU2lRM0ROUEV4Z1k2NDArdjVmWmwrN2xnM2dUd2tPRms4U1A1VEpjN3ViTUxFZFJGWGtkOTlhS2F6VDJKCldia3N3bEx0YTUzbmZQSUNmLzFnd2xKZVlZYTRMZUxHY1dQclRKUWorcXpYME9CdWg0cjhQNzRCT3lrOEhGUUgKTlk4L05mczZJQ2c1d2lJcjRJV2pvd05SZS9NUW14SVhwaWtJV21IallQUUV1R1BMWElUYnc4cGNvVS8wajJrYQpZN1c3UDBsdWlRWDV6Yzd1TlRrN3NlTE85N2xLNWY5eWJ4QnJUcHRaeXM3azA2MllvYm9hamZCbkh3SURBUUFCCm8yRXdYekFPQmdOVkhROEJBZjhFQkFNQ0FxUXdIUVlEVlIwbEJCWXdGQVlJS3dZQkJRVUhBd0VHQ0NzR0FRVUYKQndNQ01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZDNXNQTVpQUDFYV2Y1cmZZM0dGVVhuTgpZSDRGTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFER0NvTDNhdTZRS2F3ZjZnYmRCdGprVFJLb1pPMzBzZkpVCnRJUUVpb1loZlBPaGdiZm1IbFVHS2NYS2NYdFF4dWJXY2dpYUNjeUpqMk9sb2MwWXpnbHVETGJONHVpVGlNcnEKcUkvMWU2N213YUJvQmtBdkU2UWVkRmxYNmlMeVhpM0wxNEFEUDQ1eHhIRm13RzRmRlRzQktvU09uMDF5YW10ZwpwR3lzeWN4QXpET0pYQTYvL0pmd1grT3p1c0lUREVEUXdDWWxZdEZGbnlvdjZtRlFKWjZtbFROMDdhVng1STEzClVXRUlSUTJDVEJkM08zOGc5dkJKYUYvUHQ4YzZqUGZFQW43ZjRsSVM3dVBoU0NJc0FYbS93Q1Fnc29OTStKRHEKOWIrUTZDUGRma1dPdXo5M0RBVmN0T3B6bmJFR0JHcW1oc29GMjRseUJ5dVUzQXVTa2QzOQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==aaaaaaaaaaaaaaaaaa"
tlsCert: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURhekNDQWxPZ0F3SUJBZ0lSQU1xRlFBVDRtcmJhcWNqR1lwWHVjcUV3RFFZSktvWklodmNOQVFFTEJRQXcKSERFYU1CZ0dBMVVFQXhNUmNITmpMV0YxZEc5elkyRnNaWEl0WTJFd0hoY05NalF3T1RJMk1ETXlNakV4V2hjTgpNamt3T1RJMU1ETXlNakV4V2pBWk1SY3dGUVlEVlFRREV3NXdjMk10WVhWMGIzTmpZV3hsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxZeXFTSEVLRExHdmV3T0NBTmdLRUt4cm1uTXlSaEYKTzZuRUJHdFluOTNwMXovd0pJVHZ4cEpJNmRwamVWMzZyREREU0R5cnlYbndkYXBjcHU4WUdMWHhxd0xrUVpkSgpkRExlNVc0MVQrZXNOMGpyUWZqaFk1L2xRODdkemFlY1RleEFYNkFiUG9Uc1h5Q0N0OTU3aEdCdWN5STdoSGoyCmxiMm4yRlREVi93czRwbFRwVW56dmw0QWJvUS9HTTZWdmRBdUZUV3B4SDRmOXliYjYrL0pxSVFXWkJqL0ZIK3YKS3l3NVY5QWVWY3BkME9JMVhyd3lyampvLzJjRy9ONUNKUnN2T0l0WXhnZy9mek94cjZVWDlibmgweVdWSk1uTAoxUGdxU0tONzk1S3J0ckI0dzVPYlMvVmo3c0J3WllSRGk1VENiTUNtUm5JMHZzTWY0dXE2cms4Q0F3RUFBYU9CCnFqQ0JwekFPQmdOVkhROEJBZjhFQkFNQ0JhQXdIUVlEVlIwbEJCWXdGQVlJS3dZQkJRVUhBd0VHQ0NzR0FRVUYKQndNQ01Bd0dBMVVkRXdFQi93UUNNQUF3SHdZRFZSMGpCQmd3Rm9BVUxtdzh4azgvVmRaL210OWpjWVZSZWMxZwpmZ1V3UndZRFZSMFJCRUF3UG9JYmNITmpMV0YxZEc5elkyRnNaWEl1Y0dWeVptVmpkSE5qWVd4bGdoOXdjMk10CllYVjBiM05qWVd4bGNpNXdaWEptWldOMGMyTmhiR1V1YzNaak1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ2YKMnQvTVJucU1BMW1BTzkyZGtpQzFIQXh4N2RyWUxwUnE3amVCZVFHMGQrMURvYmRhakhiZzNTc3RFTG53L1hUZAo4V0hIcytPckNQL2lBbkpDaVlCOFVJK2F3NU9jY1lUeUR2eEZod3hrZzFndEdXcGQvczMreThQYU9EVHAyVS9hCkhjMXQ2U3FUSWF2WXE2UXhESzNVc2lYanVBa2orQk5PMGZ0NW9CY1lDZzJpZERtcXpKS1dzdjQwM0RHWmw0S3gKazNPRWNqVUY1YTZFaURRTTZ2VEQwQVFGVzRHeXprZS9ua29LdlJSZHJyUytibWFCSEwyeS9BQkxOdG9DOXZKVQpnU2N2K2hjL0VvdktZeFg4cDZ4dkd0RjMzeXhOb3RjdjV2WGJSeW1iYTFLQUg5cDZ4aFFDRGpyUVFEUHVSZElUCktETDI5d3FjdFJCc2pZUzdjN3R4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
tlsKey: "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdGpLcEljUW9Nc2E5N0E0SUEyQW9Rckd1YWN6SkdFVTdxY1FFYTFpZjNlblhQL0FrCmhPL0dra2pwMm1ONVhmcXNNTU5JUEt2SmVmQjFxbHltN3hnWXRmR3JBdVJCbDBsME10N2xialZQNTZ3M1NPdEIKK09Gam4rVkR6dDNOcDV4TjdFQmZvQnMraE94ZklJSzMzbnVFWUc1eklqdUVlUGFWdmFmWVZNTlgvQ3ppbVZPbApTZk8rWGdCdWhEOFl6cFc5MEM0Vk5hbkVmaC8zSnR2cjc4bW9oQlprR1A4VWY2OHJMRGxYMEI1VnlsM1E0alZlCnZES3VPT2ovWndiODNrSWxHeTg0aTFqR0NEOS9NN0d2cFJmMXVlSFRKWlVreWN2VStDcElvM3Yza3F1MnNIakQKazV0TDlXUHV3SEJsaEVPTGxNSnN3S1pHY2pTK3d4L2k2cnF1VHdJREFRQUJBb0lCQUZ6SVE3VDhUWFErSFMxKwpuN29JVjdoTy80REZWb3JUaHVTVys2RW15cEtacUpIWTJ1QVBzdEYvN0FiWXJVK2NmZDkvTXVRaW4wRkhvTVFXCjFyUGZVNFlrQVhCQmlOMWYvSnFEUTlieHN6VkpVQVlFTjk1aitQWE9MTS9GeUJyWTBCM0ZVbTBZTXFoQ0x0ZEIKZWgwRVRZOUY3Nys1QzJjR080OFBVRzBwNVQ1M1NMNXdBNWN3UFNkakRVdGxlNVpCZ2kvNU1UMVNmU1owV1g0SAozOVIrdm1LTElsZHpkS0ZYWnRTOFNlbS9qTHcwU2s5dTl4VXdsRDZ5OUJXb0QwSm5DOUtYdDJTS1Jpekk4SUNoCkIvU3ZaMkhXblVVTUVxSkhxYXYvaGs2amJRR1p1eUJTWmNkSFV4WXJtTUl2L3VhdmcyRHNrekVYNDI2R016aTMKRmNUUkhURUNnWUVBNVFIVHdacjBtSUo4bmdJQVVqUjduOW80aUtPRmVCZmN0YmFFLythTnF4ZkN1bHdRT2RJTgoxQW03SFl6M1JGWGtwQkJSc09ERkd5OHlOaCtYS2lRTnJ5TzUvNGNmdHNtVENLTXBRMUhVTjV5UXg2VDZMemtJCkNrQWJrOHhLWVk5OXpPZnFKYThueFlrWFRoWXJ6eWR2ODJxT1JrZ1FqSFdHeE1vVlJmaEZxdFVDZ1lFQXk2eGoKZWlSRVNkWHQyU1I1TENuTG9Ld29NbHcwTkNjcStJVFBSb1FoQTJHTHF0S2dqVHExWEFNZ2ZNUTVFenlIK2JoZAo5ZVlMK2pxYzRsZVBORk0zYjI4YnJQNmdPV1VBeE5xK0JXY2JGblNYMmc5Y2FrOVk2TllsMnFQbzZzUGdKSVpaCkIyV1o0Q0NkazVDNENabmh0VGZIOW5DbEorT0p2Z2NEMkFieVBwTUNnWUVBNGMySGFtTEx3b1VITXhWdUU2QkUKTlQ0S2VGQWtEbUlZOGVOaUNLRGdGMDk2Wmg3WXNPcDVVQlc0Tms4RWtKRFF2U1NLS0FtNDBERWRKVmpJeUhoOApOYkRZZVNoWEVQaDR6Zzl6WkRtUE1yWmgyempFcDBOb3RoSzRIbWRWQUJWb1NWNXk5VzFibmQ2cTZBZmNjb25FClRSTkg3ZXlveEpZdHFHUkp6WkFid1hFQ2dZRUFzZ1djOFk2dmphWUtma2l1RFJiL0lCSTZGRUtaY3BobHE5RlIKNXYzb2FTOGl5QTNRUFlGNk1kR2QzTkFhWHlFQ2JSaGtNR0dDU0paZ3VFbjJKck14K2htd3BCWHZjQnREZjh6KwpXSFNWclp2OWVxRUFrditGNUFZQ0VraEFkcXI4eUpBUlV6RFFQYUhrZWEzOTNvNTlSVXRiWWFGMkNoYU5OK2FqCnluZlAxemNDZ1lBM3hrYWlvNkhpMmVjUkp6RGNWd1dvQ1VycWF0bHhCSVJjQmNtZ1h6WTZKNVVvSG9nWVZNcjkKRGs0Y3B1YXZnNklmRXM5Y0NPTnNEOC9YUDNZRmM4Mm4vUy9IdW10N0ZUdmJQcmc5eHdWWEFxcEExOVUxUXVkVApkSkZ1cWZWWjVzTXdCRXUyUjFjckJYNWNVQzJzVUE1eVhmaUpZb3FwTkFIOTZwQTJna010Vmc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo="
```
## ArgoCD ignore diffs
If you use ArgoCD with Autosync enabled, you may encounter issues with the Autoscaler. Automation changes resources at the pod level without affecting the parent resources spec (Deployment, StatefulSet, etc.), so ArgoCD doesn't detect any changes and will not attempt to revert them.
Autoscaler doesn't patch the resources section in the parent manifest. However, for the evict+mutate flow, it patches the annotations section of parent objects by adding two annotations:
```yaml
annotations:
automation.perfectscale.io/originatingRevision: ...
automation.perfectscale.io/restartedAt: ...
```
This might lead to pod restarts. If Autoscaler is not in a *healthy* state at this time, the applied recommendations might be missed.
To ignore these annotations in ArgoCD, configure the ignoreDifferences key at the installation level by adding it to the `argocd-cm` ConfigMap. In our patches, we utilize the Manager field, so you can seamlessly exclude it by specifying our manager name, `PerfectScaleAutoscaler`.
```yaml
data:
resource.customizations.ignoreDifferences.all: |
managedFieldsManagers:
- PerfectScaleAutoscaler
```
Or by adding the following configuration to your ArgoCD App manifest:
```yaml
spec:
ignoreDifferences:
- group: "*"
kind: "*"
managedFieldsManagers:
- PerfectScaleAutoscaler
```
## How to enable cert-manager for Automation admission webhooks
With [`cert-manager`](https://cert-manager.io/) installed in your Kubernetes cluster, TLS certificates for Admission Webhooks can be automatically provisioned and managed.
### Enabling cert-manager Support
There are two options for enabling cert-manager integration:
1. Setting the following value in your `values.yaml` file:
```
admission:
certManager:
enabled: true
```
2. Via Helm CLI using `--set`:
```
helm upgrade --set admission.certManager.enabled=true .....
```
When `admission.certManager.enabled` is set to `true`, the chart will:
1. Create a cert-manager `Issue`.
2. Create a `Certificate` resource for the webhook service.
3. Mount the generated TLS certificate in the webhook deployment.
4. Automatically configure `caBundle` for the webhook configuration using the cert-manager-generated CA.
# Troubleshooting
Guidance for diagnosing and resolving common issues
## failed to call webhook
When operating within a GKE Private Cluster, an additional firewall rule is required when [installing PerfectScale Automation Agent](#install-perfectscale-automation-agent).
A new firewall rule will permit the Control Plane address range to communicate with the Cluster Pod IPv4 address range on port 8443.
Refer to the official documentation for detailed instructions:
{% embed url="" %}
## Autoscaler debugging
To diagnose issues with the PerfectScale Autoscaler, start by checking the logs of the serve pod. If none of the following logs are present: `patched`, `empty patch`, or `failed to marshal patch:`, it may indicate that the API server is unable to reach the PerfectScale Autoscaler serve pod, and most likely it is an infrastructure-related issue.
Here are some common scenarios where this might happen:
1. Network policies at the Kubernetes level (e.g., Calico) are blocking access.
2. Network policies blocking traffic from the API server to Kubernetes pods (e.g., GCP custom security groups).
3. An interceptor (such as Istio) intercepts requests but does not forward them to the serve pod.
If you see error logs from the serve pod, such as messages starting with `failed to marshal patch:` or `empty patch`, please contact support, and we will assist you.
# About cloud billing integration
Unlock a precise optimization and comprehensive analysis of K8s expenses across your entire environment by integrating your cloud billing
By default, PerfectScale pulls the "Shelf" prices from the public cloud providers' pricing API. However, for a higher level of granularity, more accurate tracking of resource consumption, and the actual cost of your K8s, PerfectScale allows pulling usage and billing information directly from your cloud provider through the integration.
Leverage the most detailed source of information about the costs and usage of cloud computing services (like AWS CUR or Azure Cost Management), including billing, discounts, reserved resources, and free credits.
{% hint style="warning" %}
**This feature is only available for Advanced and Expert Plan users. To upgrade your subscription level, contact** [**support@perfectscale.io**](mailto:support@perfectscale.io) **or use the** [**PerfectScale Slack Community**](https://perfectscalecommunity.slack.com/)**.**
{% endhint %}
With the cloud billing integration, PerfectScale will provide you with a clear picture of resource allocation based on your unique billing conditions and give you accurate pricing of your Reserved Instances, Savings Plans, and more. This will give you a comprehensive view of the K8s cost optimization process and help prioritize the actions to ensure they make the most significant impact.
Use the following guides to effortlessly integrate your usage and billing information depending on your cloud provider:
* [AWS](/cloud-billing-integration/connecting-aws-cur)
* [Azure](/cloud-billing-integration/connecting-azure-cost-management)
* [GCP](/cloud-billing-integration/connecting-gcp-cloud-billing)
* 💡 In addition, DoiT customers can pull [DoiT pricing](/cloud-billing-integration/doit-pricing) data to PerfectScale directly from DCI.
After enabling cloud billing, the `Pricing source` on the `Overview` screen updates to Cloud Provider along with the connection status indication (green for connected or red if the connection failed). If PerfectScale experiences an issue with the cloud billing connectivity, it will inform you with a banner that the connection was lost. PerfectScale will then use Shelf Price until the issue is resolved.
{% hint style="success" %}
If you have a global discount for on-demand nodes that isn’t included in your custom cloud billing, you can apply it in addition to your AWS CUR, Azure Cost Management, and other integrations. Learn more on the dedicated cloud billing integration pages.
:bulb: For [**DoiT customers**](/cloud-billing-integration/doit-pricing), **this discount is applied automatically** without any additional configuration.
{% endhint %}
# Connecting AWS CUR
Step-by-step guide on how to enable AWS CUR
PerfectScale has created a detailed guide to help you effortlessly enable AWS CUR.
In order to start using AWS CUR, follow the steps provided below:
1. [Create Cost and Usage Reports](#to-create-cost-and-usage-reports)
2. [Set up Amazon Athena](#to-create-cost-and-usage-reports-1)
3. [Run Amazon Athena queries](#to-create-cost-and-usage-reports-1)
## Create Cost and Usage Reports
1. Sign in to the [Billing and Cost Management console](http://console.aws.amazon.com/billing).
2. Go to `Data Exports`in the navigation pane and choose `Create`.
3. In the **Export type** choose `Legacy CUR export.`
4. Enter a `Report name` for your report (for example: `cur-perfectscale-example`).
5. Select `Refresh automatically` in the **Export Content** / **Data refresh settings**.
6. In the **Data export delivery options** for **Report data time granularity** choose one of the following options:
* `Hourly` to aggregate the line items in the report by hour.
* `Daily` to aggregate the line items in the report by day.
* `Monthly` to aggregate the line items in the report by month.
To get more detailed data, PerfectScale recommends aggregating the line items in the report Hourly.
7. For **Report data integration** choose Amazon Athena.
8. In the **Data export storage settings**, configure the section following one of these steps:
* Select the existing bucket (if you already have S3 bucket).
* Select **Create a bucket**, enter a bucket name, and choose the Region for the new bucket.
9. For the **S3 path prefix**, enter the report path prefix you want to prepend to the name of your report.
10. Add **Tags** if needed.
11. Review the settings for your report and click `Create report`.
{% hint style="info" %}
It can take up to 24 hours for AWS to deliver your first report to your Amazon S3 bucket.
{% endhint %}
### CUR master account support
If your AWS account is a part of an AWS Organization or if you manage multiple AWS accounts within the same organization, you can centralize billing data using the management (primary) account of the AWS Organization. This setup allows you to configure AWS CUR only once and get a detailed view of costs and usage across all accounts, enabling better cost management across your entire organization.
When CUR is enabled at the organization level, it automatically collects cost and usage data from all member accounts, simplifying expense tracking and management across multiple accounts in a centralized way. This setup allows you to utilize a single AWS CUR for all pricing profiles, requiring only a change in the region where your Kubernetes cluster is installed.
{% hint style="info" %}
If you are setting up CUR in the master account, ensure that all subsequent configurations (CUR, Athena, and IAM) are also completed in this account.
{% endhint %}
## Set up Amazon Athena
Set up Amazon Athena using AWS CloudFormation templates.
{% hint style="info" %}
AWS CloudFormation doesn't support cross-region resources. In order to use an AWS CloudFormation template, all the resources should be created in the same AWS Region. The Region must support the following services:
* AWS Lambda
* Amazon Simple Storage Service (Amazon S3)
* AWS Glue
* Amazon Athena
{% endhint %}
1. Go to the [Amazon S3 console](https://console.aws.amazon.com/s3/).
2. In the S3 bucket that you opted to receive the AWS CUR report (cur-perfectscale-example) in the folder **report path prefix**(your-report-path-prefix)**/report name** (your-report-name) you will find template file [**crawler-cfn.yml**](https://s3.console.aws.amazon.com/s3/object/cur-perfectscale-example?region=eu-central-1\&bucketType=general\&prefix=example/cur-perfectscale-example/crawler-cfn.yml)**.**
AWS generate automatically template file and it can take up to 24h to generate from creating CUR
3. Go to **Object actions**, and click `Download as`.
4. Navigate to the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/).
5. In case of using AWS CloudFormation for the first time, select `Create New Stack` and click `With new resources (standard)` in the dropdown list.
Create new stack
Otherwise - `Create Stack`.
6. Select `Choose an existing template` in **Prerequisite - Prepare template**.
7. Select `Upload a template file` in **Specify template**.
8. After clicking `Choose file` select the downloaded `.yml` template, and click `Open`.
9. As a next step, enter the **Stack name** for your template, name `cur-perfectscale-example`, add tags, if needed, and go to the next page.
10. Tick the check box at the bottom of the page and click `Submit`. \
:white\_check\_mark: **I acknowledge that AWS CloudFormation might create IAM resources**
The template creates the following resources:
* Three IAM roles
* An AWS Glue database
* An AWS Glue crawler
* Two Lambda functions
* An Amazon S3 notification
11. To ensure the template is configured properly, check the `status` in `stack info`.
Template status
## Run Amazon Athena queries
1. Go to the Amazon **Athena service** and select [**Query editor**](https://eu-central-1.console.aws.amazon.com/athena/home?region=eu-central-1#/query-editor).
2. Click `Edit settings` on the top right corner
Amazon Athena settings1
3. In the `Query result location and encryption` write down the path to the created for the CUR bucket, or use another bucket (for example, `s3://cur-perfectscale-example/`**`athena`**`/` ) and click `Save`.
Amazon Athena settings2
4. Return to the [**Query editor**](https://eu-central-1.console.aws.amazon.com/athena/home?region=eu-central-1#/query-editor) and run the following query to ensure the configuration works properly:\
`select status from cost_and_usage_data_status`
Amazon Athena settings3
## Configure the authentication method
There are two ways to grant us access to your CUR:
1. Delegate access across AWS accounts using IAM roles (**recommended**).
2. Create a separate IAM user.
### How to delegate access with IAM roles
1. Go to **AWS IAM** service
2. Chose `Policy` and click `Create Policy`.
3. Add the following JSON permissions to the policy.
{% hint style="info" %}
Replace **{{S3\_BUCKET\_NAME}} line 62** and **{{ATHENA\_DATABASE\_NAME}} line 12,44,45** with your values in the JSON policy below, where **ATHENA\_DATABASE\_NAME** is the name of your Athena database created by the CloudFront script.
\
**Example:**
{% endhint %}
{% hint style="warning" %}
:bulb: **NOTE:** There is no “-“ in the name of Athena. AWS replaces all “-“with “\_“.
{% endhint %}
Add JSON permissions to the policy
{% code lineNumbers="true" %}
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BaseAthenaPermissions",
"Effect": "Allow",
"Action": [
"athena:*"
],
"Resource": [
"arn:aws:athena:*:*:workgroup/primary",
"arn:aws:athena:*:*:datacatalog/{{ATHENA_DATABASE_NAME}}"
]
},
{
"Sid": "BaseGluePermissions",
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:GetDatabases",
"glue:GetTable",
"glue:GetTables",
"glue:GetPartition",
"glue:GetPartitions",
"glue:BatchGetPartition",
"glue:StartColumnStatisticsTaskRun",
"glue:GetColumnStatisticsTaskRun",
"glue:GetColumnStatisticsTaskRuns"
],
"Resource": [
"arn:aws:glue:*:*:catalog",
"arn:aws:glue:*:*:database/{{ATHENA_DATABASE_NAME}}",
"arn:aws:glue:*:*:table/{{ATHENA_DATABASE_NAME}}/*"
]
},
{
"Sid": "BaseQueryResultsPermissions",
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts",
"s3:AbortMultipartUpload",
"s3:PutObject",
"s3:PutBucketPublicAccessBlock"
],
"Resource": [
"arn:aws:s3:::{{S3_BUCKET_NAME}}",
"arn:aws:s3:::{{S3_BUCKET_NAME}}/*"
]
},
{
"Sid": "BaseS3BucketPermissions",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": [
"*"
]
},
{
"Sid": "BasePricingPermissions",
"Effect": "Allow",
"Action": [
"pricing:GetProducts"
],
"Resource": [
"*"
]
}
]
}
```
{% endcode %}
4. Choose `Roles` and then choose to `Create a Role` in the navigation pane.
5. Choose the `An AWS account` role type.
6. Select `Another AWS account`.
7. For `Account ID`, enter **888061904880**.
8. In the Options, choose `Require external ID` and write down your **External ID**
External ID is a unique, user-defined string used when setting up cross-account access in AWS Identity and Access Management (IAM). This additional security measure ensures that only trusted third-party entities can assume a specific role.
The external ID can be any string you define (a combination of random numbers, letters, or both).
9. Choose `Next: Permissions` to set the permissions associated with the role.
10. Select the check box next to the policy you created before.
11. Name the user and click `Next`.
12. **(Optional)** Add description and metadata to the role by attaching tags as key-value pairs (you may need it for more information about using tags in IAM).
13. After reviewing the role, choose `Create role`.
14. You should now obtain the role's Amazon Resource Name (ARN), a unique identifier for the role you need to add to the PerfectScale [AWS CUR Profile](#aws-cur-pricing-profile-configuration).\
**Example**: *arn:aws:iam::989068116150:role/access-perfectscale-account-to-cur*).
### How to create IAM user for CUR integration
1. Go to **AWS IAM** service.
2. Go to `Policy` and click `Create Policy`.
3. Add the following JSON permissions to the policy and move to the next step.
{% hint style="info" %}
Replace **{{S3\_BUCKET\_NAME}} line 62** and **{{ATHENA\_DATABASE\_NAME}} line 12,44,45** with your values in the JSON policy below, where **ATHENA\_DATABASE\_NAME** is the name of your Athena database created by the CloudFront script.
\
**Example:**
{% endhint %}
{% hint style="warning" %}
:bulb: **NOTE:** There is no “-“ in the name of Athena. AWS replaces all “-“with “\_“.
{% endhint %}
Add JSON permissions to the policy
{% code lineNumbers="true" %}
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BaseAthenaPermissions",
"Effect": "Allow",
"Action": [
"athena:*"
],
"Resource": [
"arn:aws:athena:*:*:workgroup/primary",
"arn:aws:athena:*:*:datacatalog/{{ATHENA_DATABASE_NAME}}"
]
},
{
"Sid": "BaseGluePermissions",
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:GetDatabases",
"glue:GetTable",
"glue:GetTables",
"glue:GetPartition",
"glue:GetPartitions",
"glue:BatchGetPartition",
"glue:StartColumnStatisticsTaskRun",
"glue:GetColumnStatisticsTaskRun",
"glue:GetColumnStatisticsTaskRuns"
],
"Resource": [
"arn:aws:glue:*:*:catalog",
"arn:aws:glue:*:*:database/{{ATHENA_DATABASE_NAME}}",
"arn:aws:glue:*:*:table/{{ATHENA_DATABASE_NAME}}/*"
]
},
{
"Sid": "BaseQueryResultsPermissions",
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts",
"s3:AbortMultipartUpload",
"s3:PutObject",
"s3:PutBucketPublicAccessBlock"
],
"Resource": [
"arn:aws:s3:::{{S3_BUCKET_NAME}}",
"arn:aws:s3:::{{S3_BUCKET_NAME}}/*"
]
},
{
"Sid": "BaseS3BucketPermissions",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListAllMyBuckets"
],
"Resource": [
"*"
]
},
{
"Sid": "BasePricingPermissions",
"Effect": "Allow",
"Action": [
"pricing:GetProducts"
],
"Resource": [
"*"
]
}
]
}
```
{% endcode %}
4. Name your policy and click `Create Policy`.
5. Go to **User** and click `Create User`.
6. Name the user and click `Next`.
7. Select your policy in the `Attach policies directly` and move to the next step.
8. **Create User**
9. Select just created user, go to **Security\_credentials,** and click `Create access key`.
10. Copy the generated **Access key/Secret access key** and paste it into the CUR profile.
***
**Visit the official AWS documentation for more details**
***
## Configuring AWS pricing integration with AWS CUR profile
After enabling CUR, in order to start using the integration, you need to create an AWS CUR Pricing Profile and apply it to the desired cluster.
### :tools: **How to create** AWS **CUR profile**
There are two options for creating a Profile: [from the Settings tab](#from-the-settings-tab) or directly [from the Overview](#from-the-overview-tab).
#### From the **Settings** tab
Go to the **`Settings`** tab on the left panel -> select **`Pricing`** -> click the **`+Add Profile`** button -> select **`AWS CUR`** -> name the profile and put the needed values in the relevant fields -> click **`Save`** button.
AWS CUR profile from the settings
{% hint style="info" %}
To verify the accuracy of your AWS CUR Profile configuration, simply click on the **`Test Integration`** button. When the configuration is correct, you will see the message **`AWS CUR Configured Correctly`** . In case of **`AWS CUR Configured Wrongly`** response, check, and ensure the correctness of the data.
{% endhint %}
#### From the **Overview** tab
Go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the custom **`Pricing Profile`** and click **`three dots`** button -> select **`Cluster Settings`** -> go to **`Customizations`** -> click on **`Add New Profile`** in the **`Pricing Profile`** drop-down list -> select **`AWS CUR`** type -> name and configure your profile -> click the **`Save And Apply`** button -> click the **`Save Changes`** button.
AWS CUR profile from the overview
#### **AWS CUR profile configuration**
AWS CUR profile configuration
`athena_result_bucket`: the S3 bucket, where Athena stores query results.
`athena_region`: the AWS region where Athena is running.
`athena_database`: the name of the database created on Athena setup.
`athena_table`: the name of the table, created on Athena setup.
`aws_account_id:` AWS account, where the cluster is running.
{% hint style="warning" %}
`aws_account_id` refers to the account where the cluster is running, **NOT** the AWS billing account ID.
{% endhint %}
`aws_external_id`: the ID for cross-account access in AWS Identity and Access Management (IAM).
{% hint style="info" %}
**`asw_external_id`** is a unique, user-defined string used when setting up cross-account access in AWS Identity and Access Management (IAM). This additional security measure ensures that only trusted third-party entities can assume a specific role.
The external ID can be any string you define (a combination of random numbers, letters, or both).
**Examples**: \
YourCompanyName-Partner-2024-UniqueString \
3JdpNfwvkpw4rs1sGsdrF0rM1R2\
f47ac10b-58cc-4372-a567-0e02b2c3d479
{% endhint %}
`role_arn`: the Amazon Resource Name associated with the role possessing the necessary credentials to execute calls on your behalf.
The complete configuration might look like the following **example**:
```yaml
athena_database: athenacurcfn_access-perfectscale-account-to-cur
athena_region: us-east-1
athena_result_bucket: s3://access-perfectscale-account-to-cur/
athena_table: access-perfectscale-account-to-cur
aws_account_id: '888061904880'
aws_external_id: f47ac10b-58cc-4372-a567-0e02b2c3d479
role_arn: arn:aws:iam::693587426073:role/access-perfectscale-account-to-cur
```
{% hint style="info" %}
**NOTE**: Alternatively, you can authenticate using **credentials**. To authenticate with the credentials, replace `role_arn` with the two following fields in the configuration above:
`access_key_id`: the ID of a long-term credential for a specific user in AWS (IAM, root).
`secret_access_key`: the Secret Key for an Access Key (can only be retrieved upon creation). If the access is lost, the secret key must be recreated.
As a result, your configuration will look like this:
```yaml
aws_account_id: ''
aws_external_id: ''
athena_result_bucket: ''
athena_region: ''
athena_database: ''
athena_table: ''
access_key_id: ''
secret_access_key: ''
```
{% endhint %}
#### Global discount for on-demand nodes
If you have a global discount for on-demand nodes (from a cloud provider or cloud reseller) that isn’t included in your custom cloud billing, you can apply it in addition to your AWS CUR integration.
Add the following parameters to your AWS CUR profile to enable applying the globalDiscount on top of your cloud billing integration:
`global_discount` (optional block):
* `percentage` - the percentage (0–100, floats supported) of the discount to apply on top of standard on-demand node pricing. This field is required when `global_discount` exists.
* `start_date` (optional) - the effective start date for applying the `global_discount`. If this field is empty, it defaults to today.
{% hint style="info" %}
If you change the `percentage` or `start_date`, only newly generated [Trends report](/visibility-and-optimization/trends-monitoring) will reflect the update. Historical reports will not update automatically. Contact if you need them recalculated.
{% endhint %}
The complete configuration might look like the following **example**:
```yaml
global_discount:
percentage: 15
start_sate: 2025-31-12
athena_database: athenacurcfn_access-perfectscale-account-to-cur
athena_region: us-east-1
athena_result_bucket: s3://access-perfectscale-account-to-cur/
athena_table: access-perfectscale-account-to-cur
aws_account_id: '888061904880'
aws_external_id: f47ac10b-58cc-4372-a567-0e02b2c3d479
role_arn: arn:aws:iam::693587426073:role/access-perfectscale-account-to-cur
```
#### AWS CUR configuration verification
To verify if AWS CUR is configured properly, click the **Test Integration** button.
:green\_circle: When the configuration is correct, you will see the message **AWS CUR Configured Correctly**.
:red\_circle: In case of **AWS CUR Configured Wrongly** response, check the **`Show integration failure details`** dropdown and fix the displayed issues.
Wrong AWS CUR profile configuration
### :tools: **How to apply AWS CUR** **profile**
#### Apply to a single cluster
To apply **`AWS CUR Pricing Profile`** to the cluster, go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`AWS CUR Pricing Profile`** and click **`three dots`** button -> select **`Cluster Settings`** -> go to **`Customizations`** -> select the needed profile in the **`Pricing Profile`** drop-down list.
Applying a single AWS CUR profile to the cluster
#### Apply to multiple clusters
To apply the profile to **multiple clusters** from a single view, use the **`Manage Assignments`** feature.
Go to the **`Settings`** tab on the left panel -> select the **`Pricing`** -> click the **`Manage Assignments`** button -> apply the profiles for the needed clusters -> click the **`Save Changes`** button.
Applying AWS CUR profile to multiple clusters
## Configuring AWS pricing integration with CR
To configure AWS CUR pricing, you need to define and apply a Custom Resource (CR) that specifies your unique billing conditions. This approach allows you to manage accurate pricing directly through Kubernetes manifests.
{% hint style="warning" %}
The Custom Resource (CR) must be created in the `perfectscale` namespace.
{% endhint %}
### Setup instructions
1. Enable the [AWS CUR](#to-create-cost-and-usage-reports) in your AWS account.
2. Set up the report to export data to an S3 bucket and integrate it with [AWS Athena](#to-create-cost-and-usage-reports-2).
3. Create an [IAM Role](#create-iam-user-for-cur-integration-1) for cross-account access (recommended) or an [IAM User](#create-iam-user-for-cur-integration-2) with the required permissions.
4. Configure the [CR](#recommended-cr-configuration).
#### Recommended CR configuration
We recommend configuring a CR using the IAM Role authentication method.
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
pricing:
- type: aws_cur
name: aws-cur-integration
assigned: true
value:
global_discount:
percentage: 7.5
start_date: 2025-12-31
# Authentication - IAM Role (recommended)
role_arn: "arn:aws:iam::your-account-id:role/your-role-name"
aws_external_id: "your-unique-external-id-here"
# Athena Configuration
athena_database: athenacurcfn_perfectscale_cur_report
athena_region: us-east-1
athena_result_bucket: s3://perfectscale-cur-results/
athena_table: perfectscale_cur_report_hourly
# AWS Account ID (where cluster is running, NOT billing account)
aws_account_id: "your-aws-account-id-here"
```
#### ⚙️ **CR parameters:**
Parameter
Description
global_discount (optional block)
The discount to apply on top of standard on-demand node pricing. percentage - the percentage (0–100, floats supported) of the discount. This field is required when global_discount exists. start_date (optional) - the effective start date for applying the global_discount.
role_arn
The Amazon Resource Name associated with the role possessing the necessary credentials to execute calls on your behalf.
aws_external_id
The ID for cross-account access in AWS Identity and Access Management (IAM).
athena_database
Name of the database that was created during the Athena setup.
athena_region
AWS region where Athena is running.
athena_result_bucket
S3 bucket where Athena stores query results.
athena_table
Name of the table that was created on the Athena setup
aws_account_id
AWS account where cluster is running, NOT billing account
{% hint style="warning" %}
If a `global_discount` percentage is provided without a `start_date` in the CR profile, we won’t default to today’s date as we do for profiles configured in the UI, and the discount will not be applied. The pricing profile appears broken in the UI.
{% endhint %}
#### Alternative CR configuration
Alternatively, you can configure a CR using the IAM User authentication method.
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
pricing:
- type: aws_cur
name: aws-cur-integration-user
assigned: true
value:
global_discount:
percentage: 7.5
start_date: 2025-12-31
# Authentication - IAM User (alternative)
access_key_id: "your-aws-access-key-id-here"
secret_access_key_from:
secretKeyRef:
name: aws-credentials
key: secret-access-key
# Athena Configuration
athena_database: your_athena_database_name
athena_region: us-east-1
athena_result_bucket: s3://your-athena-results-bucket/
athena_table: your_athena_table_name
# AWS Account ID (where cluster is running, NOT billing account)
aws_account_id: "your-aws-account-id-here"
```
{% hint style="warning" %}
If a `global_discount` percentage is provided without a `start_date` in the CR profile, we won’t default to today’s date as we do for profiles configured in the UI, and the discount will not be applied. The pricing profile appears broken in the UI.
{% endhint %}
To configure AWS CUR pricing with IAM User authentication, a secret is required:
```yaml
piVersion: v1
kind: Secret
metadata:
name: aws-credentials
namespace: perfectscale
type: Opaque
stringData:
secret-access-key: "your-aws-secret-access-key-here"
```
{% hint style="info" %}
Ensure that the secret is created in the same namespace as the exporter - `perfectscale`.
{% endhint %}
# Connecting Azure Cost Management
Step-by-step guide on how to enable Azure Cost Management
In order to provide PerfectScale with access to your Azure pricing using default authentication with the Azure SDK, the following data should be provided:
`azure_client_id`
`azure_tenant_id`
`azure_client_subscription`
PerfectScale has created a detailed guide to help you access the credentials effortlessly.
## How to get client\_id and tenant\_id
1. Sign in to the [Azure Portal](https://portal.azure.com/).
2. In the left-hand menu, navigate to **Microsoft Entra ID**.
3. Under **Manage**, select **App registrations**.
4. Click **+ New registration** at the top.
New app registration
5. Enter a name for your application, for example, MyAppPerfectScale.
6. Choose the supported account type. Typically, a **Single tenant** is used.
7. Click **Register**.
New app registration form
8. After registering, you’ll be redirected to the application overview page.
1. Copy the **Application (client) ID** – this is your `azure_client_id`.
2. Copy the **Directory (tenant) ID** – this is your `azure_tenant_id`.
Credentials
## How to get client\_subscription
1. In the [Azure Portal](https://portal.azure.com/), navigate to **Subscriptions** in the left-hand menu.
2. Select the subscription you want to use.
3. Copy your **Subscription ID** (`azure_client_subscription`) from the overview page.
Subscription ID
## **How to configure access to Azure without secrets**
1. In the application’s **Certificates & secrets** section, go to **Federated credentials** and click **+Add credential.**
## How to get client\_secret
client\_secret is the secret key that allows the application to authenticate to Azure AD and get an access token to Azure resources. Follow the instructions below to get the credentials.
1. In the application’s **Certificates & secrets** section, go to **Client secrets**.
2. Click **+ New client secret**.
New client secret
3. Add a description, for example, Secret for PerfectScale, and set an expiry period as needed.
4. Click **Add**.
Add a client secret
5. :bangbang:Copy the **Value** (your azure\_client\_secret) immediately, as it will only be displayed once:bangbang:
## How to grant permissions to your application
{% hint style="info" %}
Ensure you provide PerfectScale with the necessary permissions, as this step is mandatory.
{% endhint %}
1. In the [Azure Portal](https://portal.azure.com/), navigate to **Subscriptions** in the left-hand menu.
2. Select the subscription you want to use.
3. In the left-hand menu, choose **Access Control (IAM)**.
Access control
4. Navigate to **Role assignments**.
5. Press **+ Add** and choose **Add role assignment**.
Add role assignment
6. In the **Role** section, select **Reader** role and click **Next**.
Reader role
7. In the **Members** section, click **Select Members** and use the right-hand search menu to find your application (e.g., “MyApp”).
* You can also search the application using **Object ID**.
Application search
8. Click **Review + Assign**.
Once you have all the credentials needed, you need to create an Azure Profile and apply it to the desired cluster.
## Configuring Azure pricing integration with Azure profile
### :tools: **How to create** Azure Billing p**rofile**
There are two options for creating a Profile: [from the Settings tab](#from-the-settings-tab) or directly [from the Overview](#from-the-overview-tab).
#### From the **Settings** tab
Go to the **`Settings`** tab on the left panel -> select **`Pricing`** -> click the **`+Add Profile`** button -> select **`Azure Billing`** -> name the profile and put the needed values in the relevant fields -> click **`Save`** button.
Azure profile from the settings
#### From the **Overview** tab
Go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Azure Profile`** and click **`gear`** button -> go to **`Customizations`** -> click on **`Add New Profile`** in the **`Pricing Profile`** drop-down list -> select **`Azure Billing`** type -> name and configure your profile -> click the **`Save And Apply`** button -> click the **`Save Changes`** button.
Azure profile from the overview
#### **Azure Billing profile configuration**
Azure profila configuration
`azure_subscription_id`: The unique identifier assigned to your Azure subscription that allows PerfectScale to interact with Azure resources within your subscription. [Click here to learn how to get it](#how-to-get-client_subscription).
`azure_tenant_id` (Directory (tenant) ID): The unique identifier of the organization in Azure Active Directory that allows Azure to manage authentication within the specific directory. [Click here to learn how to get it.](#how-to-get-client_id-and-tenant_id)
`azure_client_id` (Application (client) ID): The unique identifier assigned to your application that allows it to access Azure resources. [Click here to learn how to get it.](#how-to-get-client_id-and-tenant_id)
`azure_auth_type:` The Azure authentication method . By default, PerfectScale specifies `workload_identity` .
{% hint style="info" %}
We recommend using `workload_identity` for `azure_auth_type`. Alternatively, you can authenticate using `client_secret`, but you'll also need to specify `azure_client_secret`. [Click here to learn how to get it.](#how-to-get-client_secret)
{% endhint %}
#### Global discount for on-demand nodes
If you have a global discount for on-demand nodes (from a cloud provider or cloud reseller) that isn’t included in your custom cloud billing, you can apply it in addition to your Azure Cost Management integration.
Add the following parameters to your Azure Billing profile to enable applying the globalDiscount on top of your cloud billing integration:
`global_discount` (optional block):
* `percentage` - the percentage (0–100, floats supported) of the discount to apply on top of standard on-demand node pricing. This field is required when `global_discount` exists.
* `start_date` (optional) - the effective start date for applying the `global_discount`. If this field is empty, it defaults to today.
{% hint style="info" %}
If you change the `percentage` or `start_date`, only newly generated [Trends report](/visibility-and-optimization/trends-monitoring) will reflect the update. Historical reports will not update automatically. Contact if you need them recalculated.
{% endhint %}
The complete configuration might look like the following **example**:
```yaml
global_discount:
percentage: 15
start_date: 2025-12-31
azure_auth_type: workload_identity
azure_client_id: ata4ed50-0d03-4752-g6r7-b4bbc5h32e8u
azure_subscription_id: df2d6148-6bdc-4a65-a0b0-3fa54678bb04
azure_tenant_id: c8aed8ae-6a60-4f42-a6bh-aee92bf2e432
```
#### Azure Billing configuration verification
To verify if Azure Cost Management is configured properly, click the **Test Integration** button.
:green\_circle: When the configuration is correct, you will see the message **Azure Configured Correctly**.
:red\_circle: In case of **Azure Configured Wrongly** response, check the **`Show integration failure details`** dropdown and fix the displayed issues.
Wrong Azure profile configuration
### :tools: **How to apply Azure Billing profile**
#### Apply to a single cluster
To apply **`Azure Billing Profile`** to the cluster, go to the **`Overview`** tab on the left-hand panel -> find the cluster to which you want to apply the **`Azure Billing Profile`** and click **`gear dots`** button -> go to **`Customizations`** -> select the needed profile in the **`Pricing Profile`** drop-down list.
Applying a single Azure profile to the cluster
#### Apply to multiple clusters
To apply the profile to **multiple clusters** from a single view, use the **`Manage Assignments`** feature.
Go to the **`Settings`** tab on the left-hand panel -> select the **`Pricing`** -> click the **`Manage Assignments`** button -> apply the profile for the needed clusters -> click the **`Save Changes`** button.
Applying Azure profile to multiple clusters
## Configuring Azure pricing integration with CR
To configure Azure pricing, you need to define and apply a Custom Resource (CR) that specifies your unique billing conditions. This approach allows you to manage accurate pricing directly through Kubernetes manifests.
{% hint style="warning" %}
The Custom Resource (CR) must be created in the `perfectscale` namespace.
{% endhint %}
### Setup instructions
1. Set up Azure billing export in your subscription.
2. Create an Azure AD application and service principal.
3. Grant the application the [necessary permissions](#how-to-grant-permissions-to-your-application) to access billing data.
4. Generate a [client secret](#how-to-get-client_secret) for the application (if using client\_secret authentication).
5. Copy the [client ID](#how-to-get-client_id-and-tenant_id), [tenant ID](#how-to-get-client_id-and-tenant_id), [subscription ID](#how-to-get-client_subscription), and [client secret](#how-to-get-client_secret) to the configuration below.
#### Recommended CR configuration
We recommend configuring a CR using `workload_identity` for `azure_auth_type`.
Configure the CR:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
pricing:
- type: azure_billing
name: azure-billing-export
assigned: true
value:
global_discount:
percentage: 7.5
start_date: 2025-12-31
azure_auth_type: workload_identity
azure_client_id: "your-azure-client-id-here"
azure_subscription_id: "your-azure-subscription-id-here"
azure_tenant_id: "your-azure-tenant-id-here"
```
#### ⚙️ **CR parameters:**
Parameter
Description
global_discount (optional block)
The discount to apply on top of standard on-demand node pricing. percentage - the percentage (0–100, floats supported) of the discount. This field is required when global_discount exists. start_date (optional) - the effective start date for applying the global_discount.
azure_auth_type
The Azure authentication method.
azure_client_id
The application (client) ID assigned to your application when it was registered in Azure Active Directory.
azure_subscription_id
The unique identifier for your Azure subscription.
azure_tenant_id
The unique identifier for your Azure Active Directory (AAD) tenant.
{% hint style="warning" %}
If a `global_discount` percentage is provided without a `start_date` in the CR profile, we won’t default to today’s date as we do for profiles configured in the UI, and the discount will not be applied. The pricing profile appears broken in the UI.
{% endhint %}
#### Alternative CR configuration (not recommended)
Alternatively, you can configure a CR using the `client_secret`.
Apply the **Azure Client Secret** first (only needed for authentication with `client_secret`):
```yaml
apiVersion: v1
kind: Secret
metadata:
name: azure-credentials
namespace: perfectscale
type: Opaque
stringData:
client-secret: "your-azure-client-secret-here"
```
{% hint style="info" %}
Ensure that the secret is created in the same namespace as the exporter - `perfectscale`.
{% endhint %}
Configure the CR:
```yaml
piVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
pricing:
- type: azure_billing
name: azure-billing-export
assigned: true
value:
global_discount:
percentage: 7.5
start_date: 2025-12-31
azure_auth_type: client_secret
azure_client_id: "your-azure-client-id-here"
azure_client_secret_from:
secretKeyRef:
name: azure-credentials
key: client-secret
azure_subscription_id: "your-azure-subscription-id-here"
azure_tenant_id: "your-azure-tenant-id-here"
```
#### ⚙️ **CR parameters:**
Parameter
Description
global_discount (optional block)
The discount to apply on top of standard on-demand node pricing. percentage - the percentage (0–100, floats supported) of the discount. This field is required when global_discount exists. start_date (optional) - the effective start date for applying the global_discount.
azure_auth_type
The Azure authentication method.
azure_client_id
The application (client) ID assigned to your application when it was registered in Azure Active Directory.
azure_subscription_id
The unique identifier for your Azure subscription.
azure_tenant_id
The unique identifier for your Azure Active Directory (AAD) tenant.
azure_client_secret_from
The client secret for authentication (when using client_secret authentication type).
{% hint style="warning" %}
If a `global_discount` percentage is provided without a `start_date` in the CR profile, we won’t default to today’s date as we do for profiles configured in the UI, and the discount will not be applied. The pricing profile appears broken in the UI.
{% endhint %}
# Connecting GCP Cloud Billing
Step-by-step guide on how to export GCP Cloud Billing data
Use the guide below to easily configure GCP Cloud Billing integration.
## GCP Setup
### Prerequisites
* GCP project with billing export enabled. Follow the official documentation to set up Cloud Billing data export to BigQuery
{% embed url="" %}
{% hint style="info" %}
Standard usage cost, Detailed usage cost, and Pricing should be enabled.
{% endhint %}
* BigQuery dataset with billing export data
* Service account with the relevant permissions
### Service Account Required permissions
* BigQuery Data Viewer on the billing export dataset
* BigQuery Job User permission
* Service Account Token Creator (for impersonation)
### Variables
{% hint style="info" %}
These variables will be used in the next steps to complete your [GCP Billing profile setup](#gcp-billing-profile-configuration).
{% endhint %}
```yaml
gcp_project_id=your-project-id
gcp_dataset_project_id=your-dataset-project-id # optional
gcp_billing_account=your-billing-account
gcp_service_account=your-service-account@project.iam.gserviceaccount.com
gcp_dataset_id=your-gcp-dataset-id
```
## Configuring GCP access
Follow these steps to grant PerfectScale the required permissions.
1. Create Service Account
2. Assign the roles required to enable BigQuery access:
* `BigQuery Data Viewer`
* `BigQuery Job User`
3. On the tab **Principals with access**, grant permissions to the principal `perfectscale-gcp-cur-accessor@psc-dev-1-000001.iam.gserviceaccount.com` with the role `Service Account Token Creator`.
Principals with access
{% hint style="info" %}
The customer should provide their **Service Account email** (e.g., `for-ps-access@ps-demo.iam.gserviceaccount.com`). This account will be used by PerfectScale to access the necessary billing and usage data.
{% endhint %}
## Configuring GCP Billing Export integration with GCP profile
Once everything above is in place, the next step is to create a GCP Billing profile and connect it to the cluster(s) you need.
### :tools: **How to create** GCP Billing p**rofile**
There are two options for creating a Profile: [from the Settings tab](#from-the-settings-tab) or directly [from the Overview](#from-the-overview-tab).
GCP Billing profile from the settings
Go to the **`Settings`** tab on the left panel -> select **`Pricing`** -> click the **`+Add Profile`** button -> select **`GCP Billing`** -> name the profile and put the needed values in the relevant fields -> click **`Save`** button.
#### From the **Overview** tab
Go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Azure Profile`** and click **`gear`** button -> go to **`Customizations`** -> click on **`Add New Profile`** in the **`Pricing Profile`** drop-down list -> select **`GCP Billing`** type -> name and configure your profile -> click the **`Save And Apply`** button -> click the **`Save Changes`** button.
GCP Billing profile from the overview
#### **GCP Billing profile configuration**
GCP Billing profile configuration
`gcp_project_id:` The unique identifier for your GCP project used to interact with it across services and APIs.
`gcp_dataset_project_id:` (optional) The project where the billing export dataset is located. Requires read access to the dataset. If omitted, defaults to `gcp_project_id`.
{% hint style="warning" %}
Set `gcp_dataset_project_id` when your BigQuery billing export dataset is in a different GCP project than the one used by the service account running queries. This is common in locked-down, multi-project GCP environments where billing data access and query execution permissions are managed separately.
{% endhint %}
`gcp_billing_account:` The unique identifier for your GCP billing account to manage billing across projects.
{% hint style="info" %}
For `gcp_billing_account`, all dashes (`-`) should be replaced with underscores (`_`).
**Example:**
gcp\_billing\_account: '015255\_0E128B\_3125S8'
{% endhint %}
`gcp_service_account:`The service account name used to authenticate and securely interact with Google Cloud APIs on your behalf.
{% hint style="info" %}
PerfectScale uses a principal with access to the service account, and not a service account JSON key
{% endhint %}
`gcp_dataset_id:`The unique identifier assigned to a BigQuery dataset within a specific Google Cloud project.
#### Global discount for on-demand nodes
If you have a global discount for on-demand nodes (from a cloud provider or cloud reseller) that isn’t included in your custom cloud billing, you can apply it in addition to your GCP Cloud Billing integration.
Add the following parameters to your GCP Billing profile to enable applying the globalDiscount on top of your cloud billing integration:
`global_discount` (optional block):
* `percentage` - the percentage (0–100, floats supported) of the discount to apply on top of standard on-demand node pricing. This field is required when `global_discount` exists.
* `start_date` (optional) - the effective start date for applying the `global_discount`. If this field is empty, it defaults to today.
{% hint style="info" %}
If you change the `percentage` or `start_date`, only newly generated [Trends report](/visibility-and-optimization/trends-monitoring) will reflect the update. Historical reports will not update automatically. Contact if you need them recalculated.
{% endhint %}
The complete configuration might look like the following **example**:
```yaml
global_discount:
percentage: 15
start_date: 2025-12-31
gcp_billing_account: 02DC4B_6001C0_B6C05B
gcp_dataset_id: with_random_name_us
gcp_project_id: test-4-000004
gcp_service_account: for-ps-access@test-4-000004.iam.gserviceaccount.com
```
#### Profile configuration verification
To verify if the profile is configured properly, click the **Test Integration** button.
:green\_circle: When the configuration is correct, you will see the message **GCP Billing configured correctly**.
:red\_circle: In case of **GCP Billing configured wrongly** response, check the **`Show integration failure details`** dropdown and fix the displayed issue(s).
### :tools: **How to apply GCP Billing profile**
#### Apply to a single cluster
To apply **`GCP Billing profile`** to the cluster, go to the **`Overview`** tab on the left-hand panel -> find the cluster to which you want to apply the **`GCP Billing Profile`** and click **`gear dots`** button -> go to **`Customizations`** -> select the needed profile in the **`Pricing Profile`** drop-down list.
Applying GCP Billing profile to a single cluster
#### Apply to multiple clusters
To apply the profile to **multiple clusters** from a single view, use the **`Manage Assignments`** feature.
Go to the **`Settings`** tab on the left-hand panel -> select the **`Pricing`** -> click the **`Manage Assignments`** button -> apply the profile for the needed clusters -> click the **`Save Changes`** button.
Applying GCP Billing profile to multiple clusters
## Configuring GCP pricing integration with CR
To configure GCP pricing, you need to define and apply a Custom Resource (CR) that specifies your unique billing conditions. This approach allows you to manage accurate pricing directly through Kubernetes manifests.
{% hint style="warning" %}
The Custom Resource (CR) must be created in the `perfectscale` namespace.
{% endhint %}
### Setup instructions
1. Enable [BigQuery billing export](#prerequisites) in your GCP project
2. Create a service account with [BigQuery Data Viewer](#configuring-gcp-access) permissions
3. Generate a JSON key for the service account
4. Copy the service account details and billing configuration to the [CR](#cr-configuration).
#### CR configuration
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
pricing:
- type: gcp_billing
name: gcp-billing-export
assigned: true
value:
global_discount:
percentage: 7.5
start_date: 2025-12-31
gcp_billing_account: "your-gcp-billing-account-id-here"
gcp_dataset_id: "billing_export_dataset"
gcp_project_id: "your-gcp-project-id-here"
gcp_service_account: "your-service-account@your-project.iam.gserviceaccount.com"
```
#### ⚙️ **CR parameters:**
Parameter
Description
global_discount (optional block)
The discount to apply on top of standard on-demand node pricing. percentage - the percentage (0–100, floats supported) of the discount. This field is required when global_discount exists. start_date (optional) - the effective start date for applying the global_discount.
gcp_billing_account
The unique identifier for your GCP billing account to manage billing across projects.
gcp_dataset_id
The unique identifier assigned to a BigQuery dataset within a specific Google Cloud project.
gcp_project_id
The unique identifier for your GCP project used to interact with it across services and APIs.
gcp_service_account
The service account principal used to authenticate and securely interact with Google Cloud APIs on your behalf.
{% hint style="warning" %}
If a `global_discount` percentage is provided without a `start_date` in the CR profile, we won’t default to today’s date as we do for profiles configured in the UI, and the discount will not be applied. The pricing profile appears broken in the UI.
{% endhint %}
# DoiT pricing
Leverage DoiT pricing as your cloud pricing source
For DoiT customers leveraging PerfectScale for Kubernetes optimization, PerfectScale pulls the actual cloud pricing data directly from DoiT. This integration benefits customers of both solutions by unifying pricing data across platforms, providing accurate and granular insights into Kubernetes costs that reflect the discounting customers receive as a DoiT customer. It takes into account each customer’s specific billing conditions, delivering clarity and transparency without extra effort.
{% hint style="warning" %}
DoiT pricing data is available for clusters running on **Amazon EKS** and **Google GKE**.
{% endhint %}
## Connecting DoiT pricing
To start leveraging DoiT pricing as a cloud pricing source, only two steps are needed:
### Step 1
Connect DoiT Cloud Intelligence to your cloud provider via DoiT connector. This will grant DoiT permissions to access relevant information in your environment. You can find the instructions in the [official DoiT documentation](https://help.doit.com/docs/kubernetes/kubernetes-intelligence/get-started#enable-kubernetes-core).
{% hint style="info" %}
If you’re already using DoiT Kubernetes Lens, this step is already complete.
{% endhint %}
### Step 2
Deploy the PerfectScale Agent. Learn how to do it [here](/getting-started/how-to-onboard-a-cluster#onboarding-instructions).
Once these steps are completed, PerfectScale begins pulling pricing data from DoiT, indicating the relevant source name on the Overview page.
DoiT Pricing
# Configuring multi-cloud billing integration
How to configure multi-cloud billing integration using a CRD
You can configure cloud billing integrations for multiple cloud providers using CR. This allows PerfectScale to pull cost data directly from each source, taking into account your unique billing conditions, ensuring accurate pricing calculations for your multi-cloud environment.
In the following example, we’ll guide you through configuring cloud billing integrations for both GCP and Azure clusters.
## Step 1: Enable GCP Cloud Billing
1. Enable [BigQuery billing export](/cloud-billing-integration/connecting-gcp-cloud-billing#gcp-setup) in your GCP project.
2. Create a service account with [BigQuery Data Viewer](/cloud-billing-integration/connecting-gcp-cloud-billing#service-account-required-permissions) permissions.
3. Generate a JSON key for the service account.
4. Copy the service account details and billing configuration (you will need them for [CR configuration](#step-3-configure-cr)).
## Step 2: Enable Azure Cost Management
1. Set up Azure billing export in your subscription.
2. Create an Azure AD application and service principal.
3. Grant the application the [necessary permissions](#how-to-grant-permissions-to-your-application) to access billing data.
4. Generate a [client secret](#how-to-get-client_secret) for the application (if using client\_secret authentication).
5. Copy the [client ID](#how-to-get-client_id-and-tenant_id), [tenant ID](#how-to-get-client_id-and-tenant_id), [subscription ID](#how-to-get-client_subscription), and [client secret](#how-to-get-client_secret) to the configuration below.
## Step 3: Configure CR
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
pricing:
# GCP Billing Export Integration
- type: gcp_billing
name: gcp-billing-export
assigned: true
value:
global_discount:
percentage: 4
start_date: 2025-12-31
gcp_billing_account: "your-gcp-billing-account-id"
gcp_dataset_id: "billing_export_dataset"
gcp_project_id: "my-gcp-project"
gcp_service_account: "your-billing@my-project.iam.gserviceaccount.com"
# Azure Billing Integration
- type: azure_billing
name: azure-billing-export
assigned: false
value:
global_discount:
percentage: 15
start_date: 2025-12-31
azure_auth_type: workload_identity
azure_client_id: "your-azure-client-id-here"
azure_subscription_id: "your-azure-subscription-id-here"
azure_tenant_id: "your-azure-tenant-id-here"
```
{% hint style="info" %}
We recommend configuring a CR using `workload_identity` for `azure_auth_type`. However, alternatively, you can also configure a CR using `client_secret`. Learn more [here](/cloud-billing-integration/connecting-azure-cost-management#alternative-cr-configuration-not-recommended).
{% endhint %}
{% hint style="warning" %}
If a `global_discount` percentage is provided without a `start_date` in the CR profile, we won’t default to today’s date as we do for profiles configured in the UI, and the discount will not be applied. The pricing profile appears broken in the UI.
{% endhint %}
# Clusters' metrics overview
Overview provides advanced visibility to the most important metrics per each cluster, allowing you to prioritize which cluster needs to focus on
[**`Overview`**](https://app.perfectscale.io/overview) provides a summary of cost and health metrics for the entire environment, giving you better visibility on cloud spending, waste, risks, and carbon emissions for each onboarded cluster and helping to focus on the particular clusters that need attention.
Overview screen
## **Upper Panel**
#### **Tenant**
Displays the account name. In the case of multiple accounts, it enables the user to switch between them seamlessly.
#### **Timeframe**
By default, the Overview dashboard displays data for the past 30 days, as specified in the Timeframe.
If a cluster has been running for less than 30 days, PerfectScale will calculate and display the **projected values for Cost, Potential Savings, Pod Waste, Node Idle, and CO₂** based on the data collected so far. Learn more about the projection logic [here](#projected-vs-actual-values).
## **Account Summary**
Account Summary offers a comprehensive view of aggregated data across your entire account, facilitating a seamless evaluation of the health and efficiency of your K8s environment while highlighting identified optimization opportunities that can be achieved with PerfectScale.
Account summary
{% hint style="info" %}
For clusters running for less than 30 days, PerfectScale displays projected values based on the data collected so far. For example, if a cluster has been running for 7 days, the projections will be calculated using those 7 days of data. Learn more [here](#projected-vs-actual-values).
{% endhint %}
#### **Current High Risks**
The widget summarizes the total number of resilience issues in your environment, allowing you to quickly assess overall health and effortlessly drill down into the most critical risks for resolution.
{% hint style="info" %}
The widget defaults to showing the number of high resilience risks. Use the [Risks](#risks) filter in the [Clusters Table](#summary) below to adjust the risk level and display the desired data.
Clicking on the widget will take you to the [Alert](/configure-alerts/alerts-overview) page, where you can dive deeper into the specific issues, review detailed information, and take appropriate actions to resolve them.
{% endhint %}
#### Pod Waste
The widget summarizes the total cost of wasted resources across your environment, helping you evaluate the cost-effectiveness of your clusters and providing one-click access to details and data-driven recommendations.
{% hint style="info" %}
Clicking on the widget will navigate you to the Waste Trends report for more insights and further analysis.
{% endhint %}
#### Node Idle
The widget summarizes the total cost of unused node space in your environment, highlighting inefficiencies in K8s setup.
{% hint style="info" %}
Clicking on the widget will navigate you to the Infrafit view for more insights and further analysis.
{% endhint %}
#### **Carbon Footprint**
* **Emissions**\
Displays the Carbon emission of all onboarded clusters within an account in the selected timeframe.
{% hint style="warning" %}
***Carbon Footprint*** is available for K8s environments hosted on **AWS**, **GCP**, and **Azure**. Stay updated on feature availability for your cloud provider.
{% endhint %}
#### **Cost and Savings**
Cost and Savings widget
This comprehensive widget shows the total cost of clusters across your K8s environment, compared to the expected optimized cost achievable through data-driven recommendations based on actual resource consumption (usage) and node-level optimization.
This view enables you to easily evaluate cost-saving opportunities and get a clear understanding of the optimization potential across your clusters.
{% hint style="info" %}
Clicking on the widget will navigate you to the Cost Trends report for more insights and further analysis.
{% endhint %}
{% hint style="info" %}
Negative savings indicate an under-provisioned environment.
{% endhint %}
#### **Impact Tracker**
Get clear insights into the advantages of your optimization effort with the Impact Tracker widget. This powerful feature highlights the savings and mitigated risks over time, providing you with an easy-access visual representation of your progress and impact. Whether you are tracking cost savings or improved performance, the Impact Tracker ensures that your efforts are accurately measured.
Impact Tracker widget
{% hint style="info" %}
The Impact Tracker calculates overall cost savings across your environment, capturing results from both manual actions and automation. However, risk mitigation is measured exclusively for issues resolved through automation.
To get additional insights, simply hover over the widget tooltips.
{% endhint %}
## **Clusters Table**
Cluster summary provides a detailed breakdown of cloud spending, waste, risks, and carbon emissions for each onboarded cluster.
{% hint style="info" %}
For clusters running for less than 30 days, PerfectScale displays projected values based on the data collected so far. For example, if a cluster has been running for 7 days, the projections will be calculated using those 7 days of data. Learn more [here](#projected-vs-actual-values).
{% endhint %}
At the top of the table, you'll find the total number of onboarded clusters.
Clusters table
Use the **+ Add Cluster** button to add more clusters with a single click, enabling a thorough analysis of your entire Kubernetes environment. Learn more about cluster onboarding [here](/getting-started/how-to-onboard-a-cluster).
Working together with your team proves to be far more effective. Invite your team by clicking the **Invite Users** button.
Use **Manage Agents** menu to seamlessly access and manage the PerfectScale agents for all your clusters. Learn more about agents [here](/product-overview/perfectscale-agent).
To customize tables and focus on the data that matters most, use the Columns menu. Learn more about table customization [here](/customize-workflow/tables-customization).
#### **Agent Status**
Displays the status of the agent. Choose All or filter by specific values from the drop-down list.
Status and description:
* *Stopped Transmitting Data* - a red indicator indicates that communication with the Agent was stopped.
* *Transmitting Data* - a green indicator indicates that communication with the Agent works.
* *Never Transmitted Data* - a grey indicator shows that communication with the Agent was not set.
* *Upgrade Available* - an Agent status indicates that a new version is available for the upgrade, and PerfectScale continues data processing with the current.
* *Upgrade Required* - an Agent status indicates that the current version is incompatible and an upgrade is required to continue data processing.
{% hint style="info" %}
Hover over the particular cluster to display its **Agent Status** and access additional settings:
* Check Upgrade Instructions for the agent
* Navigate to Cluster Settings
* Review additional actions applicable to a cluster
{% endhint %}
#### Cluster name
Displays the names of the onboarded clusters. Use the search bar to quickly find a specific cluster, or sort the clusters by clicking on the column header.
#### Pricing Source
Indicates whether the pricing source is derived from a cloud provider or a standard shelf price. You can select `All`, filter by specific value from the drop-down menu, or sort the data by clicking on the column header.
#### **Cloud Provider**
Displays the cluster's provider. You can select `All`, filter by specific provider from the drop-down menu, or sort the data by clicking on the column header.
#### Region and Zone
Displays the region where the cluster is running. You can select `All`, filter by specific value from the drop-down menu, or sort the data by clicking on the column header.
#### Optimization Policy
Displays the cluster's optimization policy. Optimization policy allows you to specify how your resources should be allocated in order to support the individual needs of your workloads. Define the policies that best suit your environment and business goals, depending on whether you want to maximize cost savings or provide extra headroom to maintain the resilience of mission-critical services.
* MaxSavings - maximum cost savings, the best for non-production environments
* Balanced (default) - optimally balances cost and resiliency
* ExtraHeadroom - the best fit for latency-sensitive environments
* MaxHeadroom - keeps the environment above the highest spikes
{% hint style="info" %}
The **Optimization Policy** can be set for the entire [cluster](/administration/cluster-settings#general-settings) and a specific [workload](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#workload-optimization-policy). The workload's **Optimization Policy** takes precedence and will override the value defined at the cluster level.
{% endhint %}
{% hint style="info" %}
If a custom policy is set through the exporter when installing the PerfectScale Agent, it cannot be modified in the UI afterward. You can still change the policy by upgrading the exporter with the new value, or you can return it to the default by upgrading the exporter without specifying any value (this will also enable the option to change the custom time window through the UI).
{% endhint %}
Discover more about customizing the Optimization policy [here](/customize-workflow/optimization-policy-customization).
#### Automated
Displays the number of automated workloads compared to the total automatable (eligible for automation) workloads within the cluster. You can sort the data by clicking on the column header. Clicking the **`CRD`** button opens the configured [Automation CR](/enable-automation/introduction-to-automation#cluster-level-configuration).
{% hint style="success" %}
When you hover over a non-automated cluster, a **Configure** button will appear. Click it to open a modal with **step-by-step instructions** for setting up Automation.
{% endhint %}
Configure automation button
**Automatable workloads** represent the number of non-deleted workloads with supported types (Deployments, DaemonSets, StatefulSets, CronJobs, Jobs, Rollouts, and custom grouped workloads).
{% hint style="warning" %}
StatefulSets and DaemonSets configured with the **OnDelete** update strategy are not supported.
{% endhint %}
**Automated workloads** represent the number of non-deleted workloads that are not in the following automation statuses: `Not Supported`, `Not Enabled`, or `Disabled by User`. Learn more about automation statuses [here](/enable-automation/verifying-automation-status).
{% hint style="success" %}
**Example**
In a cluster, where:
* Total workloads number: 615
* Deleted workloads: 81
* Workloads with the automation status Active: 98
* Workloads with the automation status DisabledByUser: 46
* Workloads with the automation status MixedLimitReached: 19
* Workloads with the automation status NotEnabled: 14
* Workloads with the automation status NotSupported: 132
* Workloads with the automation status SystemLimitReached:1 37
* Workloads with the automation status UserLimitReached: 88
You will se the following results:
* Automatable workloads: 615 - 81 - 132 = 402
* Automated workloads: 402 - 14 - 46 = 342
{% endhint %}
#### Risks
Displays the number of resilience issues at the selected risk level for the chosen period. You can select `All`, filter by specific value from the drop-down menu, or sort the data by clicking on the column header.
#### Days seen
Displays the number of days the cluster has been connected and transmitting data. This value is shown only when the cluster has been seen for less than 30 days, indicating that the estimated monthly values are based on less than 30 days of analyzed data.
{% hint style="info" %}
If the cluster has been seen for less than 30 days, PerfectScale displays [projected monthly values](#projected-vs-actual-values) calculated from the available data.
{% endhint %}
#### Expected Max Savings
Displays the achievable cost savings for the cluster through PerfectScale [recommendations](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#recommendations-labels-and-policies-view) or [Automation](/enable-automation/introduction-to-automation), ensuring your services remain high-performing, reliable, and cost-effective.
#### Cost
Displays the cost of the cluster for the selected period.
#### Pod Waste
Displays wasted resources in the cluster, allowing you to evaluate its cost-effectiveness easily. By sorting the clusters, you can quickly identify the most wasteful areas with a single click on the column header.
#### Node Idle
Displays the total cost of provisioned but unutilized resources that could be reduced without impacting the performance or reliability of the system. By sorting the clusters, you can quickly identify the areas that need attention with a single click on the column header.
#### CO2
Displays the carbon impact emission of the cluster for the selected period.
{% hint style="warning" %}
**CO2** is available for K8s environments hosted on **AWS**, **GCP**, and **Azure**. Stay updated on feature availability for your cloud provider.
{% endhint %}
#### Labels
Displays the cluster labels. This helps organize clusters and makes navigation and filtering easier across PerfectScale dashboards for large K8s environments. To display cluster labels, you should add them first. [Learn more](/administration/cluster-settings#cluster-labels).
{% hint style="info" %}
If no labels are defined, the Labels column on the Overview screen is hidden.
{% endhint %}
#### Kubernetes version
Displays the Kubernetes version currently used by the cluster.
#### Table summary
Summary
This section summarizes the total cost of clusters and the achievable cluster costs through PerfectScale’s recommendations and provides insights into reducible pod waste and idle node costs. These metrics highlight potential savings and opportunities to minimize waste, helping to guide data-driven decisions for more efficient resource allocation.
## Projected vs Actual values
The Overview screen displays data using the default 30-day timeframe. If a cluster has been running for less than 30 days, PerfectScale will calculate and display the **projected values for Cost, Potential Savings, Pod Waste, Node Idle, and CO₂** based on the data collected so far.
**Example**: If a cluster ran for 14 days and cost $100, the projected 30-day cost would be $214.
When projected data is shown instead of actuals, a relevant tooltip will be displayed:
Projected values tooltip
### Excluding clusters from projections
In some situations, you may want to exclude some clusters from the projection logic. This is especially useful for ephemeral clusters that run for less than 30 days/month, where a 30-day projection may not be accurate.
For these cases, you can seamlessly exclude such clusters from the projection logic, allowing PerfectScale to display the actual data for the period during which the cluster was running.
To exclude a cluster from projections, add the following cluster label. Learn more about cluster labels [here](/administration/cluster-settings#cluster-labels).
```yaml
exclude-from-projections: true
```
When a cluster is excluded from projections, a relevant tooltip will be displayed:
Cluster excluded from projections
# Podfit | vertical pod right-sizing
Explore a granular, comprehensive view of your clusters' health and costs, identify and prioritize areas that need attention while autonomously optimizing workloads
PerfectScale Podfit provides comprehensive insights into the health and costs of your cluster and its components, helping you quickly pinpoint areas requiring attention along with data-driven, actionable recommendations to streamline and enhance your optimization process.
Podfit screen
## Cluster overview and telemetry
The overview and telemetry section delivers a comprehensive summary of performance risks, costs, and waste insights for the selected cluster, along with identified optimization opportunities you can quickly achieve with PerfectScale. This view enables a quick evaluation of your cluster's overall health and efficiency, pinpointing configuration issues and empowering you to streamline and enhance your optimization process effectively.
### Overview section
Podfit overview
**Cluster selector** allows for dynamic switching between clusters, enabling seamless management and monitoring of a multi-cluster environment.
**Tenant** - the account name (PerfectScale in the example above).
**Automation** – Indicates whether the cluster is automated and shows the current automation status.
* If automation is enabled and active, the corresponding status will be displayed.\
Clicking the **``** button opens the configured Automation Custom Resource (CR).
Automation is active
* If automation CR has never been configured, a **Configure** button will appear. Clicking it opens a modal with a **step-by-step guide** for enabling automation.
Automation is not configured
**Optimization Policy** - displays the optimization policy of the selected cluster. Optimization Policy allows you to specify how your resources should be allocated in order to support the individual needs of your workloads. Define the policies that best suit your environment and business goals, depending on whether you want to maximize cost savings or provide extra headroom to maintain the resilience of mission-critical services.
* MaxSavings - maximum cost savings, the best for non-production environments
* Balanced (default) - optimally balances cost and resiliency
* ExtraHeadroom - the best fit for latency-sensitive environments
* MaxHeadroom - keeps the environment above the highest spikes
{% hint style="info" %}
The **Optimization Policy** can be set for the entire [cluster](/administration/cluster-settings#general-settings) and for a specific [workload](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#workload-optimization-policy). The workload's **Optimization Policy** takes precedence and will override the value defined at the cluster level.
{% endhint %}
{% hint style="info" %}
If a custom policy is set through the exporter when installing the PerfectScale Agent, it cannot be modified in the UI afterward. You can still change the policy by upgrading the exporter with the new value, or you can return it to the default by upgrading the exporter without specifying any value (this will also enable the option to change the custom time window through the UI).
{% endhint %}
Discover more about customizing the Optimization policy [here](/customize-workflow/optimization-policy-customization).
**Timeframe** allows you to adjust the period for reviewing metrics, enabling a focused analysis for a specific time range.
**Export** allows you to easily download your data as a .csv file, enabling smooth analysis and effortless sharing.
### Telemetry section
The telemetry section provides a comprehensive overview of aggregated data for the selected cluster, offering key insights into the cluster's health and efficiency. This helps you evaluate the cluster's performance easily and identifies opportunities for optimization, giving you a clear view of its overall status.
Telemetry section
**Current Risks** shows the total [risks](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing/understanding-at-risk-indicators) identified within the cluster for the selected period. This value is dynamic and updates based on the filters applied in the [workload table](#workloads-table).
**Unused Resources** provides insights into the resources within the cluster that are not being effectively utilized:
* **Pod Waste** displays the total cost of wasted resources within the cluster. Clicking on this metric will direct you to the workload waste report, offering a detailed visual breakdown of the workloads contributing to the waste. This allows you to quickly identify the most impactful areas requiring attention, enabling more efficient optimization.
* **Node Idle** indicates the total cost of unutilized node space. By clicking on this metric, you'll be navigated to a comprehensive view of the cluster at the infrastructure level. This view provides valuable insights into the behavior of different node groups and types, enabling you to optimize the underlying infrastructure for your workloads effectively.
**Cost & Expected Optimized Cost** is a powerful widget that offers insights into the total costs incurred compared to the actual resource utilization. This information helps you evaluate whether the cluster is well-balanced, over-provisioned, or under-provisioned. Additionally, the widget provides a Recommended Cost, reflecting the potential savings achievable through PerfectScale's recommendations, ensuring your cluster operates efficiently and cost-effectively. Clicking on this metric will direct you to the cluster cost report for further investigation.
{% hint style="info" %}
Negative savings indicate an under-provisioned environment.
{% endhint %}
**CPU/Memory Utilization Over Time** provides a comprehensive visual representation of resource allocation, requests, and usage trends within your cluster. Tracking these metrics over a specified timeframe allows you to analyze historical data to understand how resource dynamics have changed, compare actual usage with allocated and requested resources, and identify utilization patterns.
* Used - p99 of utilization
* Requested - p99 of the combined requests of all the workloads
* Allocated - p99 of available cluster compute
## Workloads table
{% hint style="info" %}
PerfectScale provides comprehensive GPU visibility when GPU nodes are detected in the cluster. This visibility enables you to monitor GPU usage in real time, identify underutilized or idle resources, and make informed decisions to optimize GPU allocation and reduce waste. Learn more about GPU optimization [here](/visibility-and-optimization/gpu-optimization).
{% endhint %}
The Workload table provides a detailed overview of all the workloads running in your cluster. Each row represents a specific workload and its containers, including critical metrics like cost, waste, and potential cost increase due to under-provisioned resources. This view will help you quickly identify workloads that are misaligned with resource demands, highlighting optimization opportunities and areas at risk that require attention. With dynamic filtering and sorting options, you can easily focus on specific namespaces, labels, or workloads, making it easier to prioritize optimization tasks and run clusters efficiently.
> Workloads are sets of pods of a `Deployment`, `StatefulSet`, `DaemonSet`, `Job,` or custom resource CR (for example - `Runner`, `SparkJob,` etc)
{% hint style="info" %}
Hover over the column name to view hints.
{% endhint %}
Workloads table
{% hint style="info" %}
To customize tables and focus on the data that matters most, use the Columns menu. Learn more about table customization [here](/customize-workflow/tables-customization).
{% endhint %}
#### **Filtering resiliency issues**
Status indicates workloads at risk. Workloads could be easily filtered by the resiliency risk level or particular [indicator](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing/understanding-at-risk-indicators).\
Risk indicators are dynamic, i.e., the presence of **`OOM`** indicator in the list means that at least one workload experienced an out-of-memory event in a given `timeframe`.
The dot count is a visual indicator of risk levels, with three levels: Low, Medium, and High (three dots represent the High-risk level).
Hollow dots indicate a muted workload, while shaded dots indicate the presence of a workflow ticket in progress.
Status
#### **Automation status**
This column shows the current automation status of each workload. You can quickly filter the data by automation status, prioritizing and focusing on the most relevant workloads for further investigation. Learn more about statuses [here](/enable-automation/verifying-automation-status).
{% hint style="info" %}
Multiselect is available.
{% endhint %}
Automation statuses
#### **Type**
This column identifies the workload type (e.g., Deployment, StatefulSet). You can use filtering, sorting, and multi-select options to tailor the data display, making focusing on specific workload types easier.
#### **Namespace**
The namespace column shows the namespace of each workload. You can apply filtering, sorting, and multi-select options to customize the data display, allowing you to focus on specific namespaces.
{% hint style="info" %}
If PerfectScale does not detect any workload in the Namespaces for 7 consecutive days, those Namespaces will be consolidated into a separate Namespace `__deleted-namespaces__`.
{% endhint %}
#### **Running Hours**
The workload running hours column indicates the total duration each workload, including its replicas, has been actively running in the cluster during the selected period. You can use the sorting option to arrange the data in your preferred order.
#### **Cost/h**
The workload cost per hour column indicates the total hourly expense of the workload. You can use the sorting option to arrange the data in your preferred order.
#### **Cost**
The workload total cost column shows the total expense of the workload for the selected period, considering both its hourly cost and the duration it has been actively running. You can easily identify the most costly workloads in the cluster with a single click using the sorting option.
#### **Waste**
The workload waste column indicates the historical waste caused by over-provisioned resources allocated to a workload. You can easily identify the most wasteful workloads in the cluster with a single click using the sorting option.
#### **Savings Opportunity**
The savings opportunity displays the expected total savings achievable by applying data-driven workload right-sizing recommendations.
#### **Risk Mitigation**
The risk mitigation column shows the projected rise in workload cost based on PerfectScale’s recommendations, indicating that the workload is under-provisioned. This helps you predict the cost adjustments required to maintain cluster stability.
#### Container
The container column lists the containers associated with each workload. You can use filtering options to display the data for a specific container(s). Multi-select is available.
### View Customization
Easily jump between [Recommendations](#recommendations), [Labels and Policies](#labels-and-policies-view), and [HPA](#hpa-view) views using the switcher above the table.
Podfit view customization
#### Recommendations view
The **Recommendations Table** offers clear insights into necessary workload resource adjustments to maintain the cluster's stability and cost efficiency. To access more information, click on the workload. This will open up a [Zoom-in window](#zoom-in-window) that provides a comprehensive breakdown.
| Name | Description |
| ---------------------------------------- | ------------------------------------------------------- |
|
CPU Request
| PerfectScale guidelines for CPU Request. |
|
CPU Limit
| PerfectScale guidelines for CPU Limit. |
|
Memory Request
| PerfectScale guidelines for Memory Request. |
|
Memory Limit
|
PerfectScale guidelines for Memory Limit.
|
{% hint style="info" %}
If one or more resources have reached their CRD-defined size constraints, the recommendations will not be executed. In this case, the **Limited by Rule** indicator, along with an explanatory tooltip, will be displayed near the recommendations.\
Learn more about resource allocation constraints [here](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing/limitrange-and-resourcequota).
{% endhint %}
To customize your recommendations view, use the `Resource Change View` drop-down menu.
Recommendations formats
* Detailed - to display the changes made to resources (shows both the previous and new values).
* Total Impact in Units - to display changes made to resources as an absolute number, factoring in replica count.
* Single Instance Impact in Units - to display changes made to resources as an absolute number.
* Single Instance Impact in % - to display changes made to resources in a percentage format.
When the recommendation view is set to `Total Impact in Units`, the resource change impact summary is available. This view provides a clear understanding of the effect of total resource adjustments, enabling seamless evaluation of the optimization process.
Resource change summary
#### Labels and Policies view
Using your existing labels can help you manage the workloads more effectively by allowing you to focus on the most important ones.
PerfectScale collects and supports Workloads and Namespaces labels.
To customize the Labels View, PerfectScale allows you to choose **two** label keys. Each column in the Labels Table corresponds to a selected key and displays its relevant data for each workload.
To configure the label, click on the gear button. Then, choose the desired labels to be displayed and click the **`Apply`** button. Once the changes are applied, the values that correspond to the selected keys for workloads will be displayed.
When configuring the label view, it is possible to operate with the labels of Workloads and Namespaces. All the labels appear in the same list.
{% hint style="warning" %}
The Workload labels have higher precedence than Namespace labels. If the Workload label and Namespace label have the same name, only the Workload label will be displayed.
{% endhint %}
Labels view
{% hint style="success" %}
**`Podfit Labels Profile`** enables users to create and save sets of labels, which can then be applied to clusters. The Label set listed in the Podfit Labels Profile will be applied to the clusters attached to this profile. [Learn here](/customize-workflow/managing-workloads-with-labels) how to configure the profile.
{% endhint %}
{% hint style="info" %}
The Label set listed in the attached to the cluster **`Podfit Labels Profile`** takes precedence over any manually applied labels. If the cluster has a **`Podfit Labels Profile`** attached, it will always revert to its label set. However, if no such profile is attached, any manual label changes will be saved.
{% endhint %}
**Optimization Policy** outlines how resources should be allocated to meet the unique requirements of each workload. The **Optimization Policy** can be set for the entire [cluster](/administration/cluster-settings#general-settings) and a specific [workload](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#workload-optimization-policy).
Optimization policy
#### HPA view
The HPA view provides a clear overview of workloads utilizing Horizontal Pod Autoscaler (HPA). This feature enables users to quickly identify the workloads where HPA has been introduced and adjust HPA thresholds with provided informative tooltips that offer tailored recommendations. These recommendations are particularly helpful in optimizing scaling decisions, minimizing resource waste, and ensuring efficient operation of workloads.
HPA view
Column
Description
HPA
Indicates whether HPA has been introduced for the workload. You can easily sort the column by clicking the header or apply specific filters.
CPU (%)
Displays the trigger for HPA by CPU. For insights on threshold recommendations, simply hover over the warning tooltip. You can easily sort the column by its values by clicking the header.
There are two types of indicators to be aware of:
A red indication signifies that the threshold is below 60%, indicating potential significant CPU waste.
A yellow indication suggests that the threshold falls between 60% and 80%, pointing to potential moderate CPU waste.
Memory (%)
Displays the trigger for HPA by Memory. For insights on threshold recommendations, simply hover over the warning tooltip. You can easily sort the column by its values by clicking on the header.
There are two types of indicators to be aware of:
A red indication signifies that the threshold is below 60%, indicating potential significant Memory waste.
A yellow indication suggests that the threshold falls between 60% and 80%, pointing to potential moderate Memory waste.
Custom metric
Indicates whether a Custom metric has been detected. You can easily sort the column by clicking the header or apply specific filters.
#### GPU view
This view delivers exceptional GPU and GPU memory utilization visibility to monitor and optimize GPU resources within your Kubernetes clusters, helping to identify optimization opportunities, reduce resource waste, and improve overall K8s efficiency.
GPU view
Learn more about GPU visibility [here](/visibility-and-optimization/gpu-optimization).
### Collapse ephemeral workloads
Ephemeral workloads, such as CI pipelines, jobs, and short-lived runners, can create significant noise in Podfit. To improve visibility and reduce noise, PerfectScale groups ephemeral workloads into a single collapsed row (ephemeral bin), helping you keep the Podfit view focused on the workloads that matter most.
Collapse ephemeral workloads
{% hint style="success" %}
When PerfectScale identifies a cluster with a large number of ephemeral workloads, it displays a banner in Podfit with guidance on how to group them to improve visibility and automate optimization. Learn more how to group ephemeral workloads [here](/customize-workflow/ephemeral-workloads-grouping).
{% endhint %}
When the **Collapse Ephemeral Workloads** toggle is enabled, all ephemeral workloads identified are grouped into a single row (ephemeral bin) in the Podfit workloads table.
{% hint style="info" %}
Collapse Ephemeral Workloads does not change workloads in the cluster, modify labels, or apply grouping recommendations automatically. It only reduces the amount of data loaded and displayed in the main Podfit view, while still allowing users to inspect the underlying workloads when needed.
{% endhint %}
This toggle is enabled by default, but you can review the workloads included in the ephemeral bin at any time. To view grouped workloads, turn off **Collapse Ephemeral Workloads**, click the filter button on the left, and select **Ephemeral**. This will show all grouped ephemeral workloads and the total number of workloads included.
Ephemeral workloads list
To view only ephemeral workloads, apply the **`Ephemeral Workload = Yes`** filter to the table.
Customize table
{% hint style="info" %}
If the **Ephemeral Workload** column is not visible, open the **Customize table** menu and make sure the corresponding checkbox is selected.
{% endhint %}
## Detailed workload analysis
The zoom-in window provides comprehensive details of the workload's current state and behavior, along with historical data over time, delivering detailed metrics and unmatched visibility on resource utilization efficiency and performance risks. It provides actionable recommendations for adjusting resource allocations to enhance performance and minimize waste, and emphasizes the impact once they are implemented. Additionally, users can explore the Revisions Timeline, which displays all updates and changes, including automated or manual adjustments made to the workload, simplifying further analysis and helping track optimization progress over time.
By clicking on the workload, you will be directed to its zoom-in window:
Zoom-in window
1. [Top panel ](#top-panel)
2. [Workload summary panel](#workload-summary-panel)
3. [Workload details panel](#workload-details-panel)
### Top panel
The top panel shows the name of the selected workload and provides easy access to the Workload Optimization Policy settings and actions menu. If there's an open ticket associated with the workload, it’s also indicated here, with one-click access to view its details.
#### Workload Optimization Policy
Workload optimization policy
This displays the optimization policy of the selected workload. The optimization policy specifies how resources should be allocated to achieve the desired level of resiliency and meet application demand. This ensures that your system maintains optimal performance and stability according to your predefined standards.
* MaxEconomy - the best fit for non-production environments (Low Resiliency)
* Balanced (default) - optimally balances cost and resiliency (Medium Resiliency)
* ExtraHeadroom - the best fit for latency-sensitive environments (High Resiliency)
* MaxHeadroom - keeps the environment above the highest spikes (Highest Resiliency)
{% hint style="info" %}
Set **ExtraHeadroom** or **MaxHeadroom** Optimization Policy with just a few clicks for your mission-critical production services, ensuring continuous optimal performance.
{% endhint %}
To change the policy for the workload, select the desired one from the drop-down list and click **`Save`** button to apply the changes.
{% hint style="info" %}
The **Optimization Policy** can be set for the entire cluster and for a specific workload. The workload's **Optimization Policy** takes precedence and will override the value defined at the cluster level. If the **Optimization Policy** is not specified for the workload, PerfectScale will use the default policy set for the cluster.
{% endhint %}
Discover more about customizing the Optimization policy [here](/customize-workflow/optimization-policy-customization).
#### Actions
Actions
The actions menu provides quick access to various tasks for streamlined workload management.
Clicking `View in Observability` will be directed you to the observability tool connected to the cluster. [Learn more here](/customize-workflow/observability) about how to integrate your preferred observability tool and receive exceptional insights from PerfectScale directly to your dashboard.
Create a ticket with all the details about needed changes in the defined project and assign it to the relevant engineer (team) automatically by clicking `Create Ticket`. Learn how to integrate your Jira with PerfectScale smoothly [here](/customize-workflow/ticketing-and-bug-tracking).
{% hint style="info" %}
If the ticket already exists, you can use one of the following options: **View Task** or **Delete Task**. \
\
{% endhint %}
`Mute Workload` is a useful feature when you want to stop receiving notifications for a specific workload. By muting it, you'll no longer get alerts related to that workload, even if there’s an [Alert Profile](/customize-workflow/alerting/resiliency-alerts) linked to it. If you want to start receiving alerts for the previously muted workload, click `Un-Mute Workload` in the same menu.
{% hint style="info" %}
**Muted workloads** may still appear in certain metrics while being excluded from others. [Check the details here](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing/muted-workload).
{% endhint %}
Clicking `Revert to Default Layout` will reset the order of the widgets in the [Workload details panel](#workload-details-panel).
### Workload summary panel
This panel provides a comprehensive overview of key cost metrics, highlighting potential savings and identifying existing performance risks. Additionally, it shows the average number of observed workload replicas and indicates whether HPA has been introduced, along with its associated thresholds.
Workload automation status
**Automation status** displays whether the selected workload is automated and shows its current automation status. Learn more about statuses [here](#automation-status).
Workload summary
At the top of the panel, you can see the **type** of the selected workload, along with the corresponding **namespace** and **cluster**. Running hours (**Running Hrs**) refers to the total duration the workload, including its replicas, has been actively running in the cluster over the last 30 days.
**Cost** section reflects the total expenses associated with the workload over the past 30 days.
**Waste** section reflects the total price of unutilized resources associated with the workload over the past 30 days.
**Savings opportunity** section shows the reducible workload cost through PerfectScale's recommendations, all while maintaining peak performance. This section has 3 states:
* :clock2: **Learning** – Displayed when the maturity threshold has not yet been reached.
* :heavy\_dollar\_sign: **Amount** – Displayed when potential savings are detected (estimated savings from workload right-sizing only).
* :white\_check\_mark: **All applied** – Displayed when all recommendations have been applied, and no further cost reduction opportunities are available.
#### Automation status and configuration
Workload automation status
The Automation section shows whether the workload is automated, displays the current Automation status ([more about automation statuses](#automation-status)), and provides easy access to the Automation Custom Resource (CR) configuration.
If the automation is not configured yet, you can seamlessly configure it. This will allow you to actively maintain your environment in prime condition and ensure peak K8s performance at minimal cost.
To get quick access to the automation configuration associated with the workload, click the `Automation Config CR` button.
Historical versions of specific Config CRs can be easily accessed, allowing for a comprehensive review of their changes over time.
Automation CR history
Clicking **`Show History`** will open an additional panel with the list of historical CR versions, enabling you to review all the changes that were made to the CR configuration over time. Select any previous version to preview it with highlighted changes, showing the differences between the selected version and the current one.
#### Risks breakdown
Current risks
The Revision **Risks section** provides an overview of the risks per container associated with this specific revision. Hover over the `View All Risks` to access the full list of risks.
This view is particularly helpful for quickly evaluating the workload's health at any given moment over the past 30 days.
**Risk Mitigation Cost** indicates the expected cost increase required to address performance risks caused by resource underprovisioning. This value is displayed only when resource increase recommendations exist, and the maturity threshold is met.
#### Recommendations
{% hint style="warning" %}
The recommendations are only available for the current revision.
{% endhint %}
Recommendations
The recommendations widget provides [policy-driven](#workload-optimization-policy) recommendations for workload right-sizing. With this comprehensive view, you can effortlessly review current resource requests and limits per container, followed by the recommended values based on the actual resource consumption.
To apply the recommendations manually, click the `View REcommendations YAML` button and deploy the recommendations to your cluster.
You can also seamlessly configure autonomous workload optimization to actively maintain your environment in prime condition and ensure peak K8s performance at minimal cost. Learn more about Automation configuration [here](/enable-automation/introduction-to-automation).
### Workload details panel
The workload details panel offers unparalleled visibility into workload resource utilization at any moment over the past 30 days, identifying inefficiencies in real-time and uncovering new optimization opportunities. Providing data-driven optimization recommendations empowers you to take action proactively, enhance efficiency, and reduce costs while maintaining high performance.
{% hint style="info" %}
You can easily **manage the order of widgets** in the workload details panel. Grab the widget up or down by clicking on the widget name and moving it to the desired place on the panel.\ \
\
To reset the widgets to the **default order**, select `Revert to Default Layout` from the [Actions menu](#actions).\
{% endhint %}
#### CPU and Memory over time
These widgets provide a granular historical view of CPU and Memory utilization per container over the past 30 days, including the p90, p95, and p99.9 utilization percentiles, enabling you to seamlessly evaluate the efficiency of resource distribution by leveraging a comprehensive visual comparison of these values with set resource requests and limits.
CPU & Memory over time
1. **Container selector** allows you to easily switch between containers to display the data for the specific container. Click the drop-down menu and select the needed container from the list.
2. **Control panel** includes toggles that allow you to easily add or remove quantile lines from the chart. This feature is especially useful for managing data display, enabling smooth workload analysis with just a few clicks.\
\
Clicking on the toggles will either include or exclude the corresponding quantile lines from the chart:
* **p90**
* **p95**
* **p99.9**
* **p100** - based on working set data collected at the metrics scraping interval (short-lived spikes between scrapes may not be captured).
* **Max** - the highest observed memory usage (including OS page cache, buffers, and spikes), used **to define the memory limit**.
Percentiles
{% hint style="info" %}
Use the gear button to define the custom percentile.\
\
{% endhint %}
3. **Recommendations** section displays the [policy-driven](#workload-optimization-policy) resource requests and limits recommendations for the selected container compared to the current values.
#### GPU and GPU memory over time
Clicking on a specific workload opens the detailed workload view - Zoom-in window. This panel provides in-depth information about the workload’s current state and behavior, along with historical data on resource allocation and utilization over time. It includes GPU and GPU memory utilization metrics as well as other key workload metrics. Learn more about zoom-in capabilities [here](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#detailed-workload-analysis).
Workload GPU utilization widgets
Clicking on the toggles will either include or exclude the corresponding quantile lines from the chart:
* **p90**
* **p95**
* **p99.9**
* **p100** - based on working set data collected at the metrics scraping interval (short-lived spikes between scrapes may not be captured).
#### Cost vs Waste
This widget provides a comprehensive historical cost and waste overview across all containers within the workload. It is particularly helpful for understanding cost and waste trends as well as identifying anomalies and spikes.
{% hint style="info" %}
Cost is determined by the maximum of resources allocated or used (p90) on each machine. Any remaining machine headroom is not distributed across multiple workloads.
{% endhint %}
Cost vs waste widget
#### Replicas
This widget provides a comprehensive view of workload replicas, allowing you to track scaling trends over time. This view supports both scaling scenarios: a static replica count when HPA is not used and dynamic scaling when HPA has been introduced.
When the number of replicas is static, PerfectScale shows the average amount of replicas captured.
Static replicas
When HPA is enabled for a workload, the replicas widget provides a comprehensive view of all key configured parameters alongside the actual scaling values over time. On the right side of the widget, you’ll also find the configured HPA triggers and an indicator showing whether the maximum replica count has been reached, helping you monitor scaling behavior, detect spikes, and ensure efficient resource allocation.
Dynamic scaling
{% hint style="info" %}
Click the controls above the graph to display or conceal specific parameters.\
{% endhint %}
#### Revisions Timeline
Effective optimization of the environment requires understanding the release content and its impact on cost and resilience. PerfectScale built an advanced solution to address such issues and provide users with a comprehensive breakdown of every revision for each container. This enables easy comparison of versions to track issues and remediation effectiveness.
Revisions timeline
This view is particularly helpful for tracking when, why, and how resource allocations have changed over time. It displays the source of each revision, whether it was applied manually or through PerfectScale Automation, and, in the case of automated changes, explains the trigger of the action. It also highlights revisions where no resource changes occurred, helping users quickly identify relevant updates. This level of transparency streamlines investigations and keeps teams informed about the reason behind every change and its impact.
**The revisions timeline** chart displays all the revisions for the last 30 days, where each section corresponds to a particular revision and indicates the number of pods running under it.
By hovering over the revision, the following details will be shown:
* Revision ID
* Revision date and time
* Revision trigger
Use the **revision selector** to display specific revisions by including or excluding them. Click on the corresponding indicators to select or deselect them
Revisions selector
**Clicking the revision** will highlight this revision on the other Zoom-in window charts and display the corresponding data on the [Workload summary panel](#workload-summary-panel), enabling you to access all the needed data and streamline the analysis with a single click.
Revision on the charts
When you select a specific revision, you’ll see details about what triggered it, what container(s) were affected, which resources were changed, and how. If it was done by Automation, you’ll also see what issue the change intended to fix.
Revision details
# Understanding 'At Risk' indicators
Explore wide range of 'At Risk' indicators that PerfectScale provides
Risks
## **Resilience indicators**
#### **OOM**
Out-of-Memory events usually occur in the following situations:
* The memory limit for a pod is set too low. An event will be triggered when the memory usage of the pod reaches a defined limit.
* Node is experiencing memory pressure and tries to evict some pods.\
\
[Official documentation](https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/#exceed-a-container-s-memory-limit)
#### **CPU Throttling**
CPU Throttling occurs when the pod reaches its defined CPU limit and could create latency in application response.
{% hint style="info" %}
Kubernetes use [CFS’s quota mechanism](https://www.kernel.org/doc/Documentation/scheduler/sched-design-CFS.txt) to implement the limit. The quota is based on the time period and not based on available CPU power. `cfs_period_us` is used to define the time period, it’s always 100000us (100ms).\
\
For example, a container with 1 core limit will be throttled after 50ms when running on 2 cores node and after 25ms when running on 4 cores node regardless of the number of consumed CPU cores.
{% endhint %}
#### **RestartsObserved**
Frequent restarts indicate the presence of a problem with a high potential of harming the desired SLA.
#### Eviction
Eviction indicates forcefully terminating and removing a running pod from a node. Eviction events usually occur due to memory or CPU pressure on a node.
[Official documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/)
When eviction is observed, an [alert](/configure-alerts/alerts-overview) will be triggered immediately to inform the users. Make sure that you have configured and assigned the [integration profile](/customize-workflow/communication-and-messaging) to the cluster to receive timely notifications on [Slack](/customize-workflow/communication-and-messaging/slack-integration) or [MS Teams](/customize-workflow/communication-and-messaging/ms-teams-integration) channel.
#### HPAAtMaxReplicasObserved
As demand for a service or application increases, HPA will scale the system to handle the additional load by dynamically adding more replicas. Once the maximum configured limit of replicas is reached, PerfectScale will raise the HPAAtMaxReplicasObserved indicator, which means the system cannot scale further based on the existing settings.
Depending on a workload's running time at maximum replicas, the severity of the indicator will vary. For example, the longer a workload runs at maximum replicas, the higher the severity indicator.
#### SuspectedMemoryLeak
Indicates repeated attempts to increase memory limits while the workload continues to experience insufficient memory. This behavior signals abnormal memory consumption patterns and may point to a memory leak or misconfiguration.
When SuspectedMemoryLeak is detected, PerfectScale Automation stops attempting to increase memory, raises the relevant indicator, and alerts you.
## Limit/Request not set indicators
#### **CpuRequestNotSet**
Setting proper `CPU requests` helps the Kubernetes scheduler to allocate the right amount of CPU for each container, making sure that the nodes' capacity meets the demand.
#### **MemRequestNotSet**
Setting proper `MEMORY requests` helps the Kubernetes scheduler to allocate the right amount of memory for each container, making sure that the nodes' capacity meets the demand.
#### **MemLimitNotSet**
Setting proper `MEMORY limit` helps to protect your worker node from OOM, preventing the risk of memory over-allocation.
{% hint style="info" %}
Unlike compressible CPU (new cycle every 100ms), MEMORY is incompressible and cannot be over-allocated.
{% endhint %}
## UnderProvisioning indicators
#### **UnderProvisionedCpuRequest**
Setting proper `CPU requests` helps the Kubernetes scheduler to allocate the right amount of CPU for each container, making sure that the cluster's nodes capacity meets the demand.
#### **UnderProvisionedMemRequest**
Setting proper `MEMORY requests` helps the Kubernetes scheduler to allocate the right amount of memory for each container, making sure that the cluster's nodes capacity meets the demand.
#### **UnderProvisionedMemLimit**
Setting proper `MEMORY limit` helps to protect your worker node from OOM, preventing the risk of memory over-allocation. However, under-provisioned `MEMORY limit` could cause unwanted OOM events on a pod level, potentially harming the desired SLA.
## Waste indicators
#### **OverProvisionedCpuRequest**
Setting proper `CPU requests` helps the Kubernetes scheduler allocate the right amount of CPU for each container, ensuring that the cluster's nodes capacity meets the demand. In cases of over-provisioned CPU requests, cloud resources are unnecessarily wasted due to allocation without utilization.
#### **OverProvisionedMemoryRequest**
Setting proper `MEMORY requests` helps the Kubernetes scheduler allocate the right amount of memory for each container, ensuring that the cluster's nodes capacity meets the demand. However, when a memory request is over-provisioned, it wastes cloud resources, which are allocated but never used.
# LimitRange and ResourceQuota
This page explains how PerfectScale handles LimitRange and ResourceQuota in your clusters, ensuring compliance with your specified resource allocation constraints
{% hint style="info" %}
For LimitRange and ResourceQuota support, a minimum exporter version of v1.0.42 is required.
{% endhint %}
## LimitRange
In some cases, you may want to constrain minimum and maximum computing resource usage per pod or container in a namespace by configuring LimitRange. PerfectScale respects your policy to constrain the resource allocations that you specified for your pods and containers in the namespace and doesn't contradict it.
When PerfectScale detects that its recommended resource allocation either exceeds or falls below the specified LimitRange for your pods and containers, the relevant indicator will be raised.
LimitRange detected
Hover over the indicator or click on the workload to view detailed information.
LimitRange details
{% hint style="info" %}
For automated workloads, if the recommended resource allocation exceeds or falls below the specified LimitRange, PerfectScale will not apply the recommendation. Instead, it will automatically adjust the resource allocation to the maximum or minimum value the LimitRange allows, ensuring compliance with your predefined constraints.
{% endhint %}
{% hint style="warning" %}
When introducing or updating a LimitRange into an automated cluster, it is necessary to pause automation for at least 30 minutes to ensure that resource constraints are applied correctly before resuming automated operations.
{% endhint %}
## ResoureQuota
In some scenarios, you might want to constrain the overall resource consumption within a namespace by ResourceQuota to ensure that the total usage across all workloads stays within specific limitations.
When PerfectScale detects the ResourceQuota configured, the relevant indicator will be raised.
ResourceQuota detected
Hover over the indicator or click on the workload to access the details.
{% hint style="info" %}
For **automated workloads**, if a ResourceQuota is detected, PerfectScale will apply recommendations in **decrease-only** mode to ensure constraints are respected. However, recommendations will still be available.
{% endhint %}
{% hint style="warning" %}
When introducing or updating a ResourceQuota into an automated cluster, it is necessary to pause automation for at least 30 minutes to ensure that resource constraints are applied correctly before resuming automated operations.
{% endhint %}
# Muted workload
Understand the impact of muted workloads on various metric calculations
| Platform Section | Status |
| ---------------------------------------------------- | -------- |
| Podfit cluster cost | Included |
| Podfit workloads count | Included |
| Podfit summary - resilience issues count | Excluded |
| Podfit summary - pod waste | Excluded |
| Podfit summary - total CPU | Included |
| Podfit summary - total Memory | Included |
| Overview page cluster entry - risks count | Excluded |
| Overview page cluster entry - pod waste (historical) | Included |
| Overview page cluster entry - potential savings | Excluded |
| Overview page cluster entry - cost | Included |
| Overview page summary - cost | Included |
| Overview page summary - pod waste (historical) | Included |
| Overview page summary - potential savings | Excluded |
| Overview page summary - curent risks count | Excluded |
# Infrafit | node right-sizing
InfraFit is an advanced feature that provides a granular view of resource allocation across the various nodes supporting your Kubernetes clusters
## **Infrafit overview**
Infrafit is an advanced feature that provides comprehensive visibility of the entire Kubernetes environment at the infrastructure level. It helps in understanding the behavior of specific node groups and node types, and provides actionable data-driven insights, optimizing the underlying infrastructure of the workloads. Infrafit offers a clear historical view of node utilization across the entire environment, enhancing the optimization process.
This level of visibility allows you to identify areas with idle space and gain insights into adjusting node sizes based on actual resource utilization and selecting the optimal node types to support the needs of your workloads. Additionally, Infrafit provides a unique view into scheduling results, showing what workloads are being scheduled to each node group, what is preventing scaling down, and what causes overcommitment over time.
## Infrafit & Podfit connectivity
**Infrafit** and [**Podfit**](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing) together offer a comprehensive solution that provides a multidimensional approach to reduce wasted resources in your nodes. When drilling down into individual [node types](#view-by-node-group) or [node groups](#view-by-node-type), InfraFit provides a unique view into scheduling results, showing which workloads are being scheduled to the node group, what is preventing scaling down, and what causes overcommitting over time.
While Podfit helps you right-size your workloads based on their actual resource utilization, Infrafit provides data-driven recommendations that help you choose the optimal node type to best serve the needs of your workloads.
This provides insights that can help you more efficiently bin-pack and size your node to maximize resource utilization. It also helps you lower your overall node count while improving the efficiency of your node-autoscalers (like Karpenter or Cluster Autoscaler) to keep your costs optimized. [Learn more about Karpenter optimization](/visibility-and-optimization/karpenter-optimization).
By combining the capabilities of both features, you gain a comprehensive understanding of your infrastructure and workloads, enabling more efficient management and cost reduction.
### **Key benefits**
* **Integrated Insights:** Infrafit provides detailed visibility into node groups and node types, while Podfit offers insights into workload scheduling and resource allocation. Together, they give you a holistic view of your environment.
* **Optimized Resource Allocation:** By analyzing the data from Infrafit and Podfit, you can make informed decisions on adjusting node sizes, selecting the best nodes for your workloads, improving bin-packing, and optimizing the performance of node-autoscalers (e.g., Karpenter or Cluster Autoscaler).
* **Enhanced Efficiency:** Identify workloads that prevent scaling down or cause overcommitment and adjust resource allocations accordingly. This helps in lowering the overall node count while maximizing resource utilization.
* **Cost Reduction:** With a clear understanding of resource usage and waste, you can implement strategies to reduce costs and improve the efficiency of your infrastructure.
* **Maximize Karpenter**: Get the most value from Karpenter by combining workload-level insights with node-level visibility. By right-sizing workloads and analyzing scheduling patterns, node utilization, and NodePool configurations, PerfectScale identifies opportunities and provides actionable recommendations to improve bin-packing, expand instance flexibility, reduce unused capacity, and enhance consolidation efficiency. [Learn more about Karpenter optimization](/visibility-and-optimization/karpenter-optimization).
### **How it works**
* **Workload Scheduling Visibility:** Infrafit provides a unique view into scheduling results, showing which workloads are being scheduled to each node group, what prevents scaling down, and what causes overcommitment over time.
* **Resource Utilization Insights:** Podfit complements this by offering detailed insights into workload runtime and resource usage, helping you identify areas of improvement.
* **Data-Driven Recommendations:** Together, Infrafit and Podfit provide actionable recommendations to optimize your Kubernetes environment, ensuring that resources are allocated efficiently and costs are minimized.
By leveraging the combined power of Infrafit and Podfit, you can achieve a more efficient, cost-effective, and well-optimized Kubernetes infrastructure.\
\
We will explore how to leverage these capabilities throughout the documentation.
## **Upper Panel**
Infrafit upper panel
#### **1. Tenant name**
Displays the name of the account and enables you to switch between different accounts quickly.
#### **2. Clusters drop-down**
This menu allows you to switch between clusters and displays the associated data.
#### **3. In-app path**
Shows your current location within the app, helping you easily navigate and understand where you are in the interface.
#### 4. Optimization Policy
Displays the cluster's optimization policy. Optimization policy allows you to specify how your resources should be allocated in order to support the individual needs of your workloads. Define the policies that best suit your environment and business goals, depending on whether you want to maximize cost savings or provide extra headroom to maintain the resilience of mission-critical services.
* MaxSavings - maximum cost savings, the best for non-production environments
* Balanced (default) - optimally balances cost and resiliency
* ExtraHeadroom - the best fit for latency-sensitive environments
* MaxHeadroom - keeps the environment above the highest spikes
{% hint style="info" %}
The **Optimization Policy** can be set for the entire [cluster](/administration/cluster-settings#general-settings) and a specific [workload](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#workload-optimization-policy). The workload's **Optimization Policy** takes precedence and will override the value defined at the cluster level.
{% endhint %}
{% hint style="info" %}
If a custom policy is set through the exporter when installing the PerfectScale Agent, it cannot be modified in the UI afterward. You can still change the policy by upgrading the exporter with the new value, or you can return it to the default by upgrading the exporter without specifying any value (this will also enable the option to change the custom time window through the UI).
{% endhint %}
Discover more about customizing the Optimization policy [here](/customize-workflow/optimization-policy-customization).
#### **5. Timeframe**
Allows you to check the data for a specific time period: click on the drop-down list in the upper right corner and select one of the options.
#### **6. Export**
This feature allows you to seamlessly analyze and effortlessly share your data by exporting it into a .csv file. Click the **`Export`** button, and the data will be exported to your local machine in a few seconds.
## **Nodes Resource Utilization Over Time Panel**
{% hint style="info" %}
PerfectScale provides comprehensive GPU visibility when GPU nodes are detected in the cluster. This visibility enables you to monitor GPU usage in real time, identify underutilized or idle resources, and make informed decisions to optimize GPU allocation and reduce waste. Learn more about GPU optimization [here](/visibility-and-optimization/gpu-optimization).
{% endhint %}
Nodes utilization
{% hint style="info" %}
Hover over the specific time point on the chart to view data for that time
{% endhint %}
#### **1. Node Group**s/Node Types selector
This selector allows easy switching between node groups and node types, giving you different perspectives of your infrastructure for more comprehensive analysis.
#### **2. Utilization**
A drop-down selector allows you to choose the resource utilization percentile (affects graphs as well as the Utilization chart in the table below), helping you better understand the resource utilization patterns.
#### 3. Cost per Node Group/Node Type
Displays the cost of the Node Groups or Node Types over the selected timeframe.
{% hint style="info" %}
The diagram displays the top 10 node groups or node types with the highest costs
{% endhint %}
#### 4. CPU Over Time
Displays the allocated, requested, and used amount of CPU (cores) with the selected [utilization percentile](#id-2.-utilization) over the selected timeframe.
#### 5. Memory Over Time
Displays the allocated, requested, and used amount of Memory (GB) with the selected [utilization percentile](#id-2.-utilization) over the selected timeframe.
{% hint style="info" %}
**GPU** and **GPU Memory Over Time**
When PerfectScale detects active GPU resources within a cluster, it automatically enables GPU and GPU memory widgets and utilization insights in the UI. Learn more about GPU resources optimization [here](/visibility-and-optimization/gpu-optimization).
{% endhint %}
## Data Table Display Options
Use the [**Node Groups**](/visibility-and-optimization/infrafit-or-node-right-sizing/node-group-view)**/**[**Node Types**](/visibility-and-optimization/infrafit-or-node-right-sizing/node-type-view) switcher to change the desired view easily.
Data display switch
# Node group view
Gain insights and identify the most impactful optimization opportunities within the node groups
{% hint style="info" %}
To customize tables and focus on the data that matters most, use the Columns menu. Learn more about table customization [here](/customize-workflow/tables-customization).
{% endhint %}
This view provides a comprehensive and granular breakdown of all node groups within a cluster. It facilitates immediate identification of resource inefficiencies, enables quick prioritization of optimization tasks for optimal results, and unlocks further deep investigation.
Node group view
## Data table
{% hint style="info" %}
Node group **``** contains nodes whose labels were not identified (for example, custom labels). In that case, we recommend you group such nodes manually by configuring the [**`Label Customizations Profile`**](/customize-workflow/label-customizations#node_group_labels).\
\
{% endhint %}
Column
Description
Node Group
This displays the Node Group Name. To sort or filter data, click on a column title or use a drop-down list.
Architecture
This displays the Node Architecture (ARM, x86). You can sort or filter data by clicking on a column title or using a drop-down list.
Nodes (avg & max)
This displays the average and maximum number of nodes in a specific node group. Click on a column title to sort the data.
Reservation
Display the reservation type of nodes in the group. To sort or filter data, click on a column title or use a drop-down list.
Avg Cost/h
Displays the average node group cost per hour. Click on a column title to sort data.
Total Cost
Displays the total node group cost. Click on a column title to sort data.
Idle Cost
Displays the cost of the space in a node group that has never been used. Click on a column title to sort the data.
Displays the last time PerfectScale observed the Node Group.
Utilization
A visual representation of CPU, Memory, GPU, and GPU memory Utilization (allocation, request, and usage) based on the selected usage percentile. Use a drop-down list to filter data with the needed value.
**Node Groups Total**
Shows the total cost and idle of the nodes.
## Diving into the node group
By clicking on the node group, you will be navigated to the Node Group Details screen. Here, you can get a detailed view of the running workloads within this group or review the distribution of node types along with relevant data for further analysis.
Once you are in the Node Group, you can easily explore the data with various levels of granularity based on your needs:
* [Node Types](#node-recommendations)
* [Workloads](#workloads-view)
* [Workloads Chart](#workloads-chart-view)
* [Unevictable Pods](#unevictable-pods-view)
{% hint style="info" %}
PerfectScale automatically detects if a cluster autoscaler, for example, Karpenter, is running within a specific node group. When detected, a corresponding label is displayed next to the node group name in the top panel:
* Karpenter
* ClusterAutoscaler
* Unknown
{% endhint %}
### ⚡️ **Node recommendations**
{% hint style="info" %}
Node recommendations **are not available when Karpenter is installed** in the cluster.
:bulb: If the current setup is cost-optimal, PerfectScale will not display recommendations, as the environment is considered well-balanced. In this case, a **`No new recommendations available`** banner will be shown. However, you can still access node recommendations by clicking **`Compare All`**.
{% endhint %}
Node recommendations
**Node Recommendations** is a powerful feature that provides actionable, data-driven insights to help you choose the optimal node type for your workloads and streamline the overall optimization process. Also, this view delivers the estimated savings you can achieve by applying the recommendations, enabling better forecasting and budget planning.
PerfectScale continuously analyzes your node groups and recommends alternative instance types that can run the same workloads at a lower cost. It takes into account workload resource requests, DaemonSet overhead, scheduling constraints, and instance compatibility to ensure accurate and reliable recommendations.
{% hint style="warning" %}
Node recommendations are optimized for steady-state workloads (*e.g., Deployments, StatefulSets*). Bursty workloads like *Jobs* or *CI pipelines* may yield less accurate results due to their ephemeral nature.
{% endhint %}
Click the `Show` button to expand the view and see the top four best-fit node type recommendations, ranked by achievable savings, with the best match listed first.
Top ranked node recommendations
#### **Understanding recommendations**
PerfectScale provides tailored node recommendations to help you optimize costs while ensuring your workloads run efficiently. The following outlines how these recommendations are calculated:
* Only instance types from compatible families are considered (for example, general-purpose remains within general-purpose, etc.).
* Each instance takes into account the largest single workload, including DaemonSet overhead.
* PerfectScale simulates scheduling before providing the recommendation, ensuring all workloads can be successfully placed on the recommended number of nodes.
{% hint style="warning" %}
GPU-aware recommendations are not currently supported. If a GPU workload is detected, a disclaimer will be displayed to indicate this limitation.
{% endhint %}
Node recommendation - best match
The recommendations widget provides a recommended node type as well as node details, including instance type, resource allocation, average number of nodes, and the listed hourly node cost.
PerfectScale also displays the projected cost impact, both hourly and monthly, after implementing the recommendations, along with the percentage difference compared to current costs. These values are calculated over a 30-day period using list pricing.
{% hint style="warning" %}
The price comparison between the original and recommended nodes is performed on a like-for-like basis: on-demand to on-demand and spot to spot. PerfectScale does not factor in any discounts the customer currently has on their existing nodes, as we cannot predict what discounts will apply to the recommended nodes if the change is made.
{% endhint %}
{% hint style="success" %}
PerfectScale highly values your feedback on node recommendations — don’t forget to click the **"Is this helpful?"** button to let us know!
{% endhint %}
Feedback button
#### **Accessing the full recommendations list**
To view all available node recommendations, click the **`Compare All`** button. This will open the complete list, which you can search by instance family or type and sort by potential savings, projected cost, and other criteria to quickly find the nodes that best fit your needs.
Compare all recommendations
#### **Muting the recommendations**
To hide recommendations for a specific node group, click the **`Mute`** button for that node group. PerfectScale will stop displaying recommendations for it. You can restore them at any time by clicking **`Unmute`**.
Mute recommendations
### **Node group details**
#### **Upper panel**
**Node Group** indicates the name of the node group associated with the displayed data.
**Optimization Policy** indicates the cluster policy that specifies how your resources should be allocated to support the individual needs of your workloads. Learn more about policies [here](/visibility-and-optimization/clusters-metrics-overview#optimization-policy).
**Timeframe** allows you to check the data for a specific time period: click on the drop-down list in the upper right corner and select one of the existing options.
Seamlessly **Export** and effortlessly share your data by exporting it into a .csv file for further analysis.
**Node Group Resources Utilization**
To view data at a specific percentile, use the Utilization filter. The filter will impact both the charts and the table.
Percentile
#### **Charts**
Charts
**Cost per Node Type** displays the cost trend of the Node Types over the selected timeframe.
**CPU** displays the allocated, requested, and used amount of CPU (cores) with the selected usage percentile in the group over time.
**Memory** displays the allocated, requested, and used amount of Memory (GB) with the selected usage percentile in the group over time.
{% hint style="info" %}
**GPU** and **GPU Memory Over Time**
When PerfectScale detects active GPU resources within a cluster, it automatically enables GPU and GPU memory widgets and utilization insights in the UI. Learn more about GPU resources optimization [here](/visibility-and-optimization/gpu-optimization).
{% endhint %}
#### **Node group data table**
Column
Description
Node Type
Instance Type Name. Click on a column title or use a drop-down list to sort or filter data.
Architecture
Node Architecture. Click on a column title or use a drop-down list to sort or filter data.
Reservation
Node reservation type. Click on a column title or use a drop-down list to sort or filter data.
CPU/Mem (node)
Node size. Click on a column title to sort data.
Nodes avg/max
Average and maximum number of nodes with a specific type. Click on a column title to sort data.
Avg Pods per Node
Current average number of pods per the node in the node group.
Max Pods per Node
The maximum possible pods that can be scheduled on the node.
Running Hours
Total instance running hours.
Avg Cost/h
Average cost per hour of the instance with the specific type. Click on a column title to sort data.
Total Cost
Total cost of nodes with the specific type. Click on a column title to sort data.
Idle Cost
Cost of the space in nodes with the specific types that has never been used. Click on a column title to sort the data.
Last Seen
Last time PerfectScale observed node with a specific type.
Utilization
CPU, Memory, GPU, and GPU memory Utilization (allocation, request, and usage) based on the selected usage percentile. Use a drop-down list to filter data with the needed value.
{% hint style="info" %}
Clicking on a specific node type will navigate you to the workloads table, displaying the workloads associated with that node type. \
\
This view is particularly helpful for identifying the most wasteful workload running on the node with the selected type and providing data-driven recommendations on how to eliminate it in a few clicks.\
To explore the workload details at a deeper level, click on the specific workload to open a [Zoom-in](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#zoom-in-window) window that provides an in-depth view.
{% endhint %}
## Workloads view
Diving into the workloads running on a specific node enables you to seamlessly identify which workloads contribute the most to resource waste due to over-provisioning and adjust resource allocations with data-driven recommendations, creating new opportunities to optimize your underlying Kubernetes infrastructure.
Workloads view
Column
Description
Workload
Indicates the name of the workload.
Automation
Displays the automation status of a particular workload. You can easily sort the data by automation status to focus on the most relevant information for further investigation.
Type
Indicates the type of the workload.
Namespace
Indicates the workloads' namespace.
Running Hours
The workload running hours.
Total Cost
The total cost of allocated resources of the workload.
Pod Waste
The total cost of reducible workload resources.
Increase Needed
An estimated cost increase associated with implementing the recommendation to resolve resiliency risks caused by underprovisioned resources.
Potential Savings
An estimated dollar savings by applying workload sizing recommendations.
Container
Indicates the container of the workload.
**Labels and Policies view**
Labels and policies view
Column
Description
Optimization Policy
Displays the Optimization policy associated with the workload:
MaxSavings - maximum cost savings, the best for non-production environments
Balanced (default) - optimally balances cost and resiliency
ExtraHeadroom - the best fit for latency-sensitive environments
MaxHeadroom - keeps the environment above the highest spikes
Labels
Displays the label associated with the workload. You can select up to two labels to display.
{% hint style="info" %}
To configure the label, click on the gear button. Then, choose the desired labels to be displayed and click the **`Apply`** button. Once the changes are applied, the values corresponding to the selected workload keys will be displayed.
{% endhint %}
{% hint style="warning" %}
The Workload labels have higher precedence than Namespace labels. If the Workload label and Namespace label have the same name, only the Workload label will display.
{% endhint %}
**HPA view**
The HPA view provides a clear overview of workloads utilizing Horizontal Pod Autoscaler (HPA). This feature enables users to quickly identify the workloads where HPA has been introduced and adjust HPA thresholds with provided informative tooltips that offer tailored recommendations. These recommendations are particularly helpful in optimizing scaling decisions, minimizing resource waste, and ensuring efficient operation of workloads.
HPA view
Column
Description
HPA
Indicates whether HPA has been introduced for the workload. You can easily sort the column by clicking the header or apply specific filters.
CPU (%)
Displays the trigger for HPA by CPU. For insights on threshold recommendations, simply hover over the warning tooltip. You can easily sort the column by its values by clicking the header.
There are two types of indicators to be aware of:
A red indication signifies that the threshold is below 60%, indicating potential significant CPU waste.
A yellow indication suggests that the threshold falls between 60% and 80%, pointing to potential moderate CPU waste.
Memory (%)
Displays the trigger for HPA by Memory. For insights on threshold recommendations, simply hover over the warning tooltip. You can easily sort the column by its values by clicking on the header.
There are two types of indicators to be aware of:
A red indication signifies that the threshold is below 60%, indicating potential significant Memory waste.
A yellow indication suggests that the threshold falls between 60% and 80%, pointing to potential moderate Memory waste.
Custom metric
Indicates whether a Custom metric has been detected. You can easily sort the column by clicking the header or apply specific filters.
## Workloads chart view
The workloads trend chart helps you identify waste and cost trends within the Node Group, directing your focus to the most critical and valuable aspects.
Workloads chart
#### 1. Scope
Select whether you want to display the workload data by waste or cost using this selector.
#### 2. Workload Type
Filter the data by the workload type.
#### 3. Interval
Select the interval for data display based on your desired data granularity.
#### 4. Legend
Use the workloads legend to include or exclude the particular workloads from the chart.
{% hint style="info" %}
Maximize your productivity by utilizing **Shortcuts**.
* **`Ctrl`**/**`Command`**+ **`Click`** legend entity = Select/Deselect the entity. Allows to do a multi-select.
* **`Click`** chart/legend entity = select one or all entities.
{% endhint %}
#### 5. Limit Selector
Choose a limit from the dropdown (from 1 to 15) to exhibit only the top N entities
{% hint style="info" %}
The entities not falling within the specified limit will be consolidated to **`Other`** on the diagram. To view entities beyond the limit, use the **`Previous`** and **`Next`** buttons.
{% endhint %}
## Unevictable pods
Unevictable Pods provides you clear visibility into the workloads preventing nodes from being removed from a cluster. In the Unevictable Pods view, you can explore nodes and their pods, quickly identify pods that cannot be evicted, and access the detailed pod and node data for further investigation. By combining a cluster-wide view with node-level detail, it helps you seamlessly identify and address scale-down blockers, improve bin-packing, maximize the efficiency of your cluster autoscaling solution (like Karpenter, or Cluster Autoscaler), and ultimately optimize your K8s cloud costs.
Unevictable pods view
### Getting started
In Infrafit, select the desired node group and switch to the **Unevictable Pods** view. The chart displays all nodes from the currently selected node group, using the latest available snapshot by default.
#### Snapshot selector
Use the date picker in the top-right corner to select a specific snapshot (up to one month back).
Snapshot
#### Top panel summary
The **top panel** shows a summary of the selected node group across node types, including total running pods, as well as the number of unevictable pods. This helps you identify whether unevictable pods are present and could affect scale-down, highlighting savings opportunities.
Unevictable Pods - top panel
### Instance charts
Use the **Instances charts** for deeper analysis. This view shows which pods are running on each instance, their share of resources, and whether any of them are unevictable, helping you quickly understand if pods may impact scale-down.
Instances charts
#### View by resource type
Switch between **CPU**, **Memory**, and **GPU** views using the view selector to display data by a specific resource type.
#### Node cards
Node cards
Each node is represented as a card displaying the following information:
* **Resource bars:** Horizontal bars showing requested vs. total capacity for CPU, RAM, and GPU.
* **Pod treemap:** Pods inside each node are displayed as a treemap. Unevictable pods are highlighted with a red border for quick identification.
* **Bottom labels:** Each card shows the node's taints and a count of unevictable pods.
By hovering over a pod, you can see its details in the tooltip. However, if you need more extensive view, click on a pod or a specific node, so you can get a full details in the sidebar.
By clicking on any node or pod, you can open the sidebar detail view, which includes:
* **Item info** — name, namespace, resource requests and limits, and other key metadata
* **Resources utilized** — a breakdown of actual resource usage
* **Labels** — all Kubernetes labels applied to the node or pod
* **Annotations** — all Kubernetes annotations associated with the item
* and more
Pod details
#### Filters and sorting
Use the **Filter** sidebar to narrow down the view by node-level or pod-level criteria. Filters allow you to focus on specific subsets of your cluster, such as nodes with taints or pods in a particular namespace, etc.
Use the **Sort** dropdown to reorder nodes based on different criteria, such as idle, resource utilization or unevictable pod count.
### ⚡️ Unevictable pods recommendations
When PerfectScale detects unevictable pods, it provides precise, data-driven YAML diff recommendations that compare the current configuration with the recommended changes, helping you troubleshoot bin-packing issues and resolve them in just a few clicks.
To access Unevictable Pods recommendations, select the relevant NodeGroup in InfraFit, open the **Unevictable Pods** view, and click **See Recommendations** or the **Unevictable Pods** widget.
Unevictable pods recommendations
{% hint style="info" %}
Recommendations are automatically prioritized and ranked by cost impact, helping you focus on the highest-value optimization opportunities first and maximize savings.
{% endhint %}
Click **See Recommendation** next to a recommendation to open its details. You'll find a clear explanation of the recommended change, along with the corresponding YAML configuration that you can copy with a single click and apply directly to your manifest.
{% hint style="success" %}
To help you optimize with confidence, PerfectScale assigns a **confidence level** and **risk level** to every recommendation. These indicators help you assess the expected impact of each change, enabling safe optimization while maintaining application reliability and performance.
{% endhint %}
Recommendation YAML
Some findings may be intentional and do not require any action. If that's the case, click **Mute for Workload**. PerfectScale will automatically hide the same finding for all unevictable pods in that workload, so it won't appear in future scans. You can still view muted findings in the **Muted** tab and unmute them at any time.
Mute finding
{% hint style="info" %}
The **Active** and **Muted** tabs give you visibility into all findings, making it easy to see which ones still in a backlog and require attention, and which are muted.
{% endhint %}
# Node type view
Explore the node type view of Infrafit
Identify wasted node resources and gain optimization insights to select the right nodes to better serve your environment's needs with the Node Types view.
{% hint style="info" %}
To customize tables and focus on the data that matters most, use the Columns menu. Learn more about table customization [here](/customize-workflow/tables-customization).
{% endhint %}
Node type view
{% hint style="info" %}
**GPU** and **GPU Memory Over Time**
When PerfectScale detects active GPU resources within a cluster, it automatically enables GPU and GPU memory widgets and utilization insights in the UI. Learn more about GPU resources optimization [here](/visibility-and-optimization/gpu-optimization).
{% endhint %}
Column
Descripti
Node Type
Displays the Instance Type Name. Click on a column title or use a drop-down list to sort or filter data.
Architecture
Displays the Node Architecture. Click on a column title or use a drop-down list to sort or filter data.
Reservation
Display the reservation type of nodes. Click on a column title or use a drop-down list to sort or filter data.
CPU/Mem (node)
Refers to the size of the instance type (memory and CPU).
Nodes (avg & max)
Displays the average and maximum number of nodes with the specific instance type. Click on a column title to sort data.
Avg Pods/node
Current average number of pods per the node in the node group.
Max Pods/node
The maximum possible pods that can be scheduled on the node.
Node Group
Displays which group nodes with a specific instance type belong to. Click on a column title or use a drop-down list to sort or filter data.
Running Hours
Displays the total instance running hours.
Avg Cost/h
Displays the average cost per hour of the instance with the specific type. Click on a column title to sort data.
Total Cost
Displays the total cost of nodes with the specific type. Click on a column title to sort data.
Idle Cost
Displays the cost of the space in nodes with the specific types that has never been used. Click on a column title to sort the data.
Last Seen
Displays the last time PerfectScale observed the node with the specific instance type.
Utilization
A visual representation of CPU, Memory, GPU, and GPU memory Utilization (allocation, request, and usage) based on the selected usage percentile. Use a drop-down list to filter data with the needed value.
Clicking on a specific node type will navigate you to the workloads table, displaying the workloads associated with that node type. This view is particularly helpful for identifying the most wasteful workload running on the node with the selected type and providing data-driven recommendations on how to eliminate it in a few clicks. To explore the workload details at a deeper level, click on the specific workload to open a Zoom-in window that provides an in-depth view.
# GPU optimization
Reduce cloud GPU costs with real-time utilization visibility and insights
{% hint style="info" %}
PerfectScale now only supports NVIDIA Data Center GPU Manager (DCGM).
{% endhint %}
{% hint style="info" %}
**GPU** support is available starting with the **exporter version 1.0.55**.\
**GPU memory** support is available starting with the **exporter version 1.1.11**.
{% endhint %}
PerfectScale delivers exceptional **GPU** and **GPU memory** utilization visibility to monitor and optimize GPU resources within your Kubernetes clusters. This feature helps teams identify optimization opportunities, reduce resource waste, and improve overall K8s efficiency.
{% hint style="warning" %}
To enable GPU visibility support in PerfectScale, the **NVIDIA DCGM exporter** should be installed. Additionally, specific configuration parameters should be set when deploying or upgrading the PerfectScale agent. Learn more [here](/getting-started/how-to-onboard-a-cluster#gpu-support).
{% endhint %}
When PerfectScale detects active GPU resources within a cluster, it automatically enables GPU and GPU memory widgets and utilization insights in the UI.
GPU widgets
This view provides detailed GPU usage and allocation efficiency metrics, helping you identify savings opportunities and drive further data-driven Kubernetes optimization.
## Podfit GPU visibility
GPU view
To quickly identify GPU-allocated workloads in **PodFit**, switch to the **GPU view** by clicking the GPU tab from the view selector, as shown below. Once there, you will see the GPU and GPU memory utilization metrics per container, as well as the GPU scheduler type, for example, KAI, and the sharing type, for example, TimeSlicing. Sort the table by GPU usage to bring all GPU-consuming workloads to the top.
Clicking on a specific workload opens the detailed workload view - Zoom-in window. This panel provides in-depth information about the workload’s current state and behavior, along with historical data on resource allocation and utilization over time. It includes GPU and GPU memory utilization metrics as well as other key workload metrics. Learn more about zoom-in capabilities [here](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#detailed-workload-analysis).
Workload GPU utilization widgets
## Infrafit GPU visibility
To see detailed GPU usage across your infrastructure, go to **InfraFit**. The GPU chart shows how much of your GPUs are being used versus how much was requested, making it easy to spot inefficiencies and find ways to optimize.
Node group GPU utilization
This view helps you quickly evaluate the difference between requested GPU and GPU memory resources and actual usage, making it easy to pinpoint underutilized or idle GPU capacity across your clusters.
By clicking on the specific node group, you will get a granular breakdown of individual instances within that group, along with key metrics for each one.
Node type GPU utilization
Clicking on a specific instance will display a list of workloads running on that machine, allowing for deeper investigation and analysis.
# Java containers
Optimize Java containers with granular visibility and tailored recommendations
## Enabling JVM metrics collection
{% hint style="info" %}
The **Coroot agent** is disabled by default. Once enabled, PerfectScale automatically detects Java containers and starts collecting JVM metrics.
Add the following parameter when deploying the PerfectScale agent to enable this feature:
```yaml
--set settings.corootNodeAgent.enabled=true
```
{% endhint %}
{% hint style="warning" %}
Coroot supports only Linux nodes.
{% endhint %}
Once the Coroot agent is installed, PerfectScale starts identifying Java containers running in your Kubernetes environment and collecting JVM metrics from them. By continuously analyzing them, PerfectScale provides granular visibility into resource usage, identifying potential bottlenecks. Based on this analysis, PerfectScale generates tailored recommendations to help ensure that your services remain efficient and maintain consistent performance.
### Deploy coroot pods to specific nodes
If your cluster includes both Linux and Windows nodes, make sure to set a nodeSelector for Linux when deploying workloads that are not Windows-compatible.
YAML values file example:
```yaml
coroot:
nodeSelector:
kubernetes.io/os: linux
```
Helm command example:
```
helm upgrade --install perfectscale ..... \
--set coroot.nodeSelector."kubernetes\.io/os"=linux \
perfectscale/exporter
```
To restrict workloads to a specific set of nodes (for example, those labeled `component=java`), you can combine multiple node selectors.
YAML values file example:
```
coroot:
nodeSelector:
kubernetes.io/os: linux
component: java
```
Helm command example:
```
helm upgrade --install perfectscale ..... \
--set coroot.nodeSelector."kubernetes\.io/os"=linux \
--set coroot.nodeSelector.component=java \
perfectscale/exporter
```
## Java usage visibility
When a Java container is detected, PerfectScale automatically starts collecting key JVM metrics. By continuously analyzing this data, PerfectScale delivers deep visibility into Heap, Non-Heap, and Garbage Collector usage patterns over time. By providing tailored, data-driven optimization recommendations, teams can seamlessly optimize resource utilization and cut cloud costs safely, ensuring their services remain efficient and maintain consistent performance.
Java containers resource utilization
When heap parameters are not set, the relevant indicator will appear under container flags, along with the calculated `-Xms` and `-Xmx` values.
Heap size is not set
{% hint style="info" %}
PerfectScale respects explicitly set heap parameters when providing recommendations or applying automation.
{% endhint %}
When heap parameters are explicitly set, PerfectScale always respects them for resource recommendations and automation. The configured values are also displayed under the Container Flags section.
MaxHeapSize set
{% hint style="warning" %}
When recommendations **exceed the MaxHeapSize** or **fall below the MinHeapSize**, Automation will not apply the recommendation to ensure that explicitly set values are respected. The relevant tooltip will be displayed.
{% endhint %}
PerfectScale also provides detailed visibility into Garbage Collector time, helping teams better understand trends such as rising GC activity, correlate spikes with performance or latency issues, validate the impact of optimization changes, and plan capacity more effectively. By monitoring GC behavior alongside memory usage, teams can ensure Java workloads remain stable and efficient.
GC time
On the right side, you can also view additional Garbage Collector parameters, including the ActiveProcessorCount and GC type:
* ZGC
* ShenandoahGC
* G1GC
* ParallelGC
* ParallelOldGC
* ConcMarkSweepGC
* SerialGC
{% hint style="info" %}
The JVM maps millicores using the following logic: 1000m = 1 CPU, and 1001m = 2 CPUs
{% endhint %}
## Disabling JVM metrics collection
To limit JVM metrics collection, you need to disable the deployment `psc-coroot-node-agent` when [installing the PerfectScale Agent](/getting-started/how-to-onboard-a-cluster#cluster-configuration) by adding the following parameter to the installation helm:
```
--set settings.corootNodeAgent.enabled=false
```
# Karpenter optimization
Maximize Karpenter efficiency with granular visibility and data-driven recommendations
{% hint style="info" %}
Karpenter recommendations require exporter version **v1.1.11** or later.
{% endhint %}
Karpenter helps Kubernetes teams improve infrastructure efficiency by dynamically provisioning the right compute capacity for their workloads. However, to achieve the full value of node autoscaling, teams need to ensure that Karpenter NodePools are well configured and aligned with workload requirements.
PerfectScale provides granular visibility into Karpenter configurations and delivers actionable recommendations to improve them. By evaluating Karpenter NodePool settings and workload behavior, it identifies suboptimal configurations and generates recommendations that help teams maximize autoscaling outcomes.
When Karpenter is detected in the cluster and associated with a specific node group, a dedicated label and the Karpenter version are displayed in the InfraFit screen header and next to the relevant node group.
Karpenter label
## Explore your Karpenter configuration
Hover over the node group with the Karpenter label and click the `` button to access the Karpenter configuration CR.
Karpenter configuration
This view provides visibility into the NodePool and EC2 configuration, as well as a history of configuration changes.
## Node group view
When node autoscaler, such as Karpenter or Cluster Autoscaler, is detected in the cluster, additional visibility capabilities become available.
You can easily filter node groups by the detected cluster autoscaling solution. Click the **Node Group** drop-down and select the relevant value.
Cluster autoscaler filter
Also, you can filter the data by consolidation policy, whether it is `WhenEmpty` or `WhenEmptyOrUnderutilized`. Click the **Consolidation** drop-down and select the relevant value from the list.
### Karpenter recommendations
When Karpenter is detected, PerfectScale analyzes its current configuration and evaluates its efficiency against the resource requests of running workloads. If inefficiencies are identified, PerfectScale provides data-driven recommendations to help improve the configuration and maximize autoscaling outcomes.
{% hint style="info" %}
Karpenter recommendations are based on the current workload requests and limits. For more accurate and effective recommendations, we recommend right-sizing workloads first, either by enabling [autonomous PerfectScale optimization](/enable-automation/automation-setup-instructions) or by manually applying [PerfectScale recommendations](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#detailed-workload-analysis). This allows Karpenter recommendations to be calibrated based on actual resource utilization, ensuring the best outcome.
{% endhint %}
When Karpenter recommendations are available, they are displayed in the **Node Recommendation** column in the following format: **Recommendation Name → Recommended Value**. The recommended value may include removing an existing configuration, modifying the current value, or adding a new one.
Karpenter recommendations
PerfectScale provides the following recommendation types:
Recommendation type
Description
Consolidation Policy Optimization
Provides recommendations for the consolidation policy to optimize underutilized nodes and increase cost-saving opportunities.
Consolidation Delay Enablement
Provides recommendations for the consolidation delay to enable timely consolidation while avoiding overly aggressive disruption.
Static NodePool Detection
Provides recommendations for NodePool flexibility to improve Karpenter’s ability to scale dynamically based on workload demand and increase consolidation opportunities.
Instance Generation Flexibility
Provides recommendations for Karpenter flexibility to expand instance selection options, helping Karpenter choose cheaper, newer, or more available instances.
Instance Family Modernization
Provides recommendations for instance family flexibility by suggesting better generations of the same instance family.
Instance Category Recommendation
Provides recommendations for instance category selection by evaluating the aggregate CPU-to-memory profile of workloads in the NodePool and suggesting an instance category that better matches the workload shape.
Instance Type Pinning Prevention
Provides recommendations for switching to broader instance selection criteria to increase scheduling flexibility and avoid capacity shortages.
NodePool Limits Near Exhaustion
Provides recommendations to increase NodePool resource limits when they are near exhaustion, creating additional headroom for burst workloads, scaling needs, and node consolidation.
{% hint style="success" %}
PerfectScale highly values your feedback on node recommendations. Don’t forget to share your feedback and let us know what you think.
{% endhint %}
Click a specific node group to drill down and access the full recommendations view. This view provides a centralized overview of the node pool’s current configuration and optimization recommendations, including a detailed breakdown of the suggested actions and the factors influencing them. It also displays the node pool CPU-to-memory ratio, detected constraints, and any identified unevictable pods that may prevent nodes from being safely scaled down.
Detailed Karpenter recommendations
Click the `` button to open the **YAML Recommendations Configuration** popup.
This view displays the full current Karpenter configuration along with the recommended values, highlighting the areas PerfectScale recommends changing to improve efficiency. Click **Copy to Clipboard** to copy the final configuration with the recommended values, so you can seamlessly apply the updated configuration.
Apply recommended configuration YAML
Clicking the `View pods` button next to the **detected unevictable pods** opens the granular **Unevictable Pods** dashboard. This dashboard shows the pod distribution within the node group, highlights the pods that cannot be evicted, and provides detailed pod and node data to support further investigation. [Learn more about Unevictable pods](/visibility-and-optimization/infrafit-or-node-right-sizing/node-group-view#unevictable-pods-view).
Unevictable pods view
#### **Muting the recommendations**
To hide recommendations for a specific node group, click the **`Mute`** button for that node group. PerfectScale will stop displaying recommendations for it. You can restore them at any time by clicking **`Unmute`**.
Mute recommendations
# Trends monitoring
Unlocks granular visibility and in-depth insight into costs, waste, and risk metrics over time
To navigate to the **Trend Reports** page, simply click on the **`Trends`** tab located on the left panel.
Trends screen
{% hint style="info" %}
Each bar in the chart contains slices representing entities defined in the [Stack By](#stack-by) section.
{% endhint %}
Trends report
## Trends Report
Reporting highlights trends and seasonality across your Kubernetes environment, directing focus to the most valuable aspects.
You can customize the view for your particular needs and get a clear understanding of your optimization progress, which components of your environment are well-optimized, and what requires attention.
To build your perfect report, we recommend following a few simple steps:
1. Select the [scope](#scope) that you would like to focus on;
2. Select the desirable [entity to stack](#stack-by);
3. Select the [interval](#interval) for the data representation;
4. Include or exclude the particular entities if needed with [filters](#filters) or [entities legend](#entities-legend).
The following example illustrates the trend of cost for the clusters. The bars in the chart are arranged in a stack, each representing the clusters' weekly cost.
1️⃣ - the total costs of the clusters from the lowest to the highlighted ones.
2️⃣ - the cost of the highlighted cluster.
Cost by cluster
## **Scoping parameters**
The display data selector defines which data to show. Get the data you want in just a few clicks. Choose the [entities](#stack-by), [scope](#scope), and [timeframe](#interval) to display stacked data.
Scoping parameters
#### Scope
A drop-down list enables a quick selection of displayed data scope. It includes the following options:
* Cost
* Risk
* Waste
* vCPU/Hours
* vCPU/Month
{% hint style="info" %}
**Risk Level** (available with **`Risk`** scope only)
Allows filtering and displaying the data related to the entities with the relevant risk level:
* All
* Low
* Medium
* High
{% endhint %}
Risks chart
#### **Stack By**
A drop-down list enables one to define how the data should be stacked. It includes the following options:
* Cluster
* Namespace
* [Workload](#dive-into-the-workload-details)
* [Node Group/Type/Reservation](#drill-down-into-nodes)
{% hint style="success" %}
When selecting Cost stacked by Workload or Namespace, the **Show Idle Cost** feature is available.
**Idle Cost** identifies a spare cluster capacity on top of what was requested by particular workloads.
Select the relevant value in the **Scope By** list (Workload or Namespace), and **`Show Idle Cost`**toggle will appear. By default, the toggle is in the **on** position.
{% endhint %}
Idle cost
### :mag: Dive into the workload details
To explore identified anomalies or specific trends in depth, seamlessly access detailed information on their causes, and/or obtain actionable recommendations for eliminating them, simply jump into the workload [Zoom-in](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing#zoom-in-window) window, which provides all the necessary information directly from the Trends Report. This allows you to effortlessly switch from questions to answers, enabling you to make quick, data-driven decisions.
Workload details
Stack the data by **`workload`** to get the overtime cost, waste, or risk trend insights for each workload in your environment. Right-clicking on a workload in the chart will open a **Workload Details** tooltip, displaying a list of the namespaces where this workload exists.
By clicking on a specific namespace from the list, a Zoom-in window of the workload in that namespace will open.
Workloads in a namespace
{% hint style="info" %}
If there are duplicate workload names inside the same namespace, a workload type will be displayed to differentiate them.\
\
Workloads last seen over 4 weeks ago are displayed in gray.
{% endhint %}
{% hint style="info" %}
If you have multiple clusters, filter a specific one to access the **`View Workload Details`** tooltip.
{% endhint %}
{% hint style="warning" %}
If the cluster is no longer connected to PerfectScale, workload details are not available. In such cases, a tooltip will indicate that the cluster has been disconnected and the workload details are not available.\
{% endhint %}
### :mag: Drill down into node cost analysis
Seamlessly access detailed views of your K8s instances from different perspectives (Node Type, Node Group, or Node Reservation) and quickly identify anomalies or specific trends with just a few clicks.
Stack the data by `Node Type`, **`Node Group`**, or **`Node Reservation`** to get the overtime cost insights. Once the data is stacked, new filters will be available to apply. These filters will provide a more granular and customized view of your infrastructure, enabling you to precisely monitor and manage different aspects of your K8s instances.
Node cost insights
#### Interval
You can use a drop-down list to change the interval that each bar in a chart represents. The following options are available:
* **Hour** - data for the last week is presented, with each bar representing a 2-hour data aggregation.
* **Day** - data from the past 60 days are displayed, with each bar depicting a single day.
* **Week** - data spanning 52 weeks is exhibited, with each bar representing a calendar week.
* **Month** - data from the past 13 months are displayed, with each bar representing a calendar month.
### Filters
Trends filters
You can customize your Trends Report by choosing which clusters, namespaces, and workloads you want to include or exclude.
{% hint style="info" %}
* Click the entity to add/remove it from the chart
* Click **`Select All`** to select or deselect all the entities from the chart
* Use the search bar to find the needed entity quickly
* Multiselect is available
* Use the ✖️ button to clear filters in a single click
{% endhint %}
{% hint style="warning" %}
If PerfectScale does not detect any workload in the Namespaces for 7 consecutive days, those Namespaces will be consolidated into separate Namespace `__deleted-namespaces__`
{% endhint %}
### Entities legend
Trends legend
The list of the entities that have been defined under [Stack By](#stack-by). Each entity is clickable and is assigned a distinct color that corresponds to the chart.
{% hint style="info" %}
Maximize your productivity by utilizing **Shortcuts**.
* **`Ctrl`**/**`Command`**+ **`Click`** legend entity = Select/Deselect the entity. Allows to do a multi-select.
* **`Click`** chart/legend entity = select one or all entities.
{% endhint %}
#### **Disconnected clusters**
If a cluster is disconnected, it will still appear in the Trends report for the period when it was actively reporting data to PerfectScale. This preserves historical accuracy and allows you to analyze past trends even after the cluster is no longer connected.
Disconnected cluster
To indicate its status, a **`disconnected`** badge is displayed when hovering over the cluster and in the cluster legend. Since the cluster is no longer connected, the workloads' details are not available.
### **Limit selector**
Choose a limit from the dropdown (from 1 to 15) to exhibit only the top N entities selected in [Stack By](#stack-by).
Limit selector
The report's scope defines what qualifies for the set limit using the following logic:
* Cost scope displays the top N most expensive entities, where N is the limit per page;
* Waste scope displays the top N most wasteful entities, where N is the limit per page;
* Risk scope displays the top N entities with the most risks, where N is the limit per page;
* vCPU/hours displays the top N entities with the highest vCPU/h value, where N is the limit per page.
Trends scrolling
{% hint style="info" %}
The entities not falling within the specified limit will be consolidated to **`Other`** on the diagram. To view entities beyond the limit, use the **`Previous`** and **`Next`** buttons.
{% endhint %}
## **Save report**
Once your report is ready, you can save the view for future reference using the **`Save Report`** feature. This will help you avoid repetitive actions and streamline your investigation process. Saved reports are visible and accessible to every authorized user in your tenant.
{% hint style="info" %}
Use the **`Set as my default`** checkbox to load the report as the default view the next time you open Trends.
{% endhint %}
Save report
{% hint style="info" %}
If you apply changes to a previously saved report and save it with those changes, it will create a new report instead of affecting the initial one.
{% endhint %}
## **Load report**
Seamlessly load your saved views with just a few clicks. Clicking the **`Load Report`** displays all saved reports across your tenant, allowing you to sort the list by report name, creator, or creation date. Select the desired report from the list and click `Load Report` button to recreate the saved view.
Load report
{% hint style="info" %}
The **`Delete`** option allows you to remove saved views from the list. Each user can only delete their own saved reports.
{% endhint %}
## **Export report**
The export feature gives you the ability to export your data into a `.csv` file for seamless analysis and sharing effortlessly. Click the **`Export`** button, and the data will be exported to your local machine in a few seconds.
Export report
# Alerts overview
Get real-time resilience and financial anomalies alerts across your environment and address them before they impact your services
PerfectScale delivers timely and reliable alerts to keep you informed about unusual system activity and detected anomalies. These alerts are specifically designed to highlight changes in key indicators, enabling you to take quick action to address potential issues before they impact your system's efficiency, performance, or stability.
PerfectScale allows you to easily set up resiliency and financial alerts and manage them efficiently by configuring the corresponding alert profile:
* [Financial alerts configuration](/customize-workflow/alerting/financial-alerts)
* [Resiliency alerts configuration](/customize-workflow/alerting/resiliency-alerts)
{% hint style="info" %}
For faster updates, utilize [Slack](/customize-workflow/communication-and-messaging/slack-integration) or [MS Teams](/customize-workflow/communication-and-messaging/ms-teams-integration) integrations to receive immediate notification when an alert is generated.
{% endhint %}
Alerts screen
## Alerts summary panel
This panel provides a summarized view of alerts across the account, showing the total number of active alerts along with a breakdown by category, including resilience, cost, and waste alerts.
Alerts summary
## Alerts details panel
This panel delivers a granular breakdown of alerts, including their status, trigger details (time and location), and whether they have been acknowledged.
{% hint style="info" %}
To customize tables and focus on the data that matters most, use the Columns menu. Learn more about table customization [here](/customize-workflow/tables-customization).
{% endhint %}
Alerts details
Use the switcher to effortlessly toggle between the [resilience](#resilience-alerts) and the financial alerts dashboards, ensuring you have quick access to both tools.
Resilience/Financial alerts switcher
## Workload-level alerts
Workload-level alerts are designed to trigger alerts based on a workload's specific state. Unlike standard alerts, which depend on particular conditions within a container, workload-level alerts are triggered by changes in specific workload indicators. This feature enhances alerting capabilities and introduces new types of alerts, allowing for more comprehensive monitoring and response to resiliency risks identified before they impact users.
* [**Max HPA Replicas**](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing/understanding-at-risk-indicators#hpaatmaxreplicasobserved) - a workload-level alert indicating that the system can not scale further because the maximum configured HPA replica limit has been reached.
## The algorithm for inactive/deleting alerts
To improve visibility and highlight only relevant issues, PerfectScale uses the following algorithm to deactivate or delete irrelevant Alerts.
#### Inactive Alert
PerfectScales checks the `last seen` date of the **`Alert`**. If the `last seen > 3 days`, the **`Alert`** will be considered inactive, and its marker will be changed to grey.
Inactive alert indicator
#### Alert deleting
With the same scenario mentioned earlier, PerfectScales checks the `last seen` date of the **`Alert`**. If the `last seen > 29 days`, the **`Alert`** will be deleted.
## Available notification channels
For faster updates, utilize [Slack](/customize-workflow/communication-and-messaging/slack-integration), [MS Teams](/customize-workflow/communication-and-messaging/ms-teams-integration), or [Datadog](/customize-workflow/communication-and-messaging/datadog-alerts-integration) Integration Profiles to receive notifications when an alert is generated.
# Alerts acknowledgement
Ensure each critical issue is acknowledged and properly addressed
The Alerts Acknowledgement feature helps to indicate to team members that the ownership of a specific alert has been taken, so every user in the system will be aware that the alert is not being ignored.
Alert acknowledgment
To acknowledge an alert, hover over it and click `Acknowledge`.
Acknowledge the alert
Once someone takes ownership of the alert, their name will be displayed in the `Acknowledge by` column. If the integration with [Slack](/customize-workflow/communication-and-messaging/slack-integration) or [MS Teams](/customize-workflow/communication-and-messaging/ms-teams-integration) is configured, other system users will be notified about it via the selected communication channel
Any user can unacknowledge an alert and then acknowledge it to take ownership of it.\
To override the ownership of the Alert, click the :heavy\_multiplication\_x: icon against the Alert and then click `Acknowledge`.
UnAcknowledge the alert
{% hint style="info" %}
In order to start receiving notification of acknowledged or unacknowledged alerts, a [Slack](/customize-workflow/communication-and-messaging/slack-integration) or [MS Teams](/customize-workflow/communication-and-messaging/ms-teams-integration) integration should be set up, and the relevant profile should be connected to the cluster.
{% endhint %}
# Financial alerts
Get real-time financial anomalies alerts across your environment and keep your cloud bill in check
PerfectScale Financial Alerts allow users to set up cost and waste alerts, notifying them when exceeding thresholds. This enables proactive cost management and resource optimization, helping teams stay within budget and reduce unnecessary spending.
#### How to configure financial alerts
To start receiving financial alerts, create a Financial Alerts Rule. Check the following [instructions](/customize-workflow/alerting/financial-alerts) to complete the setup.
#### Financial alerts dashboard
{% hint style="info" %}
Clicking on a specific alert will navigate you to the trend report, providing deeper insights for further analysis.
{% endhint %}
Financial alerts dashhboard
| Column | Description |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Status** | Indicates the alert status (Active or Inactive). Use filters to focus on the most important aspects. |
| **Alert Details** | Displays the name of the financial alert profile. Filter and display of alerts by specific profile name. Multiselect is available. |
| **Scope** | Indicates the metric (Cost or Waste) for triggering an alert. Use filters to focus on the most important aspects. |
| **Period** | Indicates the timeframe over which the metric will be evaluated. Filter and display of alerts by specific period. Multiselect is available. |
| **Target** | Indicates the object for triggering an alert. Filter and display of alerts by specific target. Multiselect is available. |
| **Value** | Displays the reached $ amount. Sortable in ascending/descending order. |
| **Threshold** | Displays the defined condition for triggering the alert. Sortable in ascending/descending order. |
| **Cluster** | Displays clusters associated with alerts. Customize the view by selecting a cluster to see relevant alerts. Multiselect is available. |
| **Namespace** | Displays namespaces associated with alerts. Customize the view by selecting a namespace to see relevant alerts. Multiselect is available. |
| **Workload** | Displays workloads associated with alerts. Customize the view by selecting a workload to see relevant alerts. Multiselect is available. |
| **Acknowledged by** | Indicates whether a team member has taken ownership of a particular alert and displays the user's name. Filter and display acknowledged alerts, acknowledged alerts by specific users, or unacknowledged alerts. Multiselect is available. Learn more about Alerts Acknowledgement [here](/configure-alerts/alerts-overview/alerts-acknowledgement). |
| **Date** | Displays the alert creation date. |
| **Notification** | Displays the channel through which the notification about the alert was sent. Filter and display of alerts by specific notification channels. Multiselect is available. |
# Resiliency alerts
Get real-time resilience alerts across your environment and address them before they impact your services
PerfectScale provides a flexible, impact-driven issue prioritization, allowing you to configure alerting tailored to your needs. Resilience Alerts are designed to notify users of potential risks that could impact workload stability and availability, inform them about issues such as resource constraints, misconfigurations, scaling limitations, etc., and help teams proactively address risks and maintain optimal cluster performance.
Learn more about resiliency indicators [here](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing/understanding-at-risk-indicators).
#### How to configure resilience alerts
{% hint style="info" %}
By default, PerfectScale generates alerts for every cluster where a [resilience indicator](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing/understanding-at-risk-indicators) with a [min\_risk\_level: high](/customize-workflow/alerting/resiliency-alerts#customize-the-alerts-by) was identified.
{% endhint %}
To start receiving resilience alerts, create a dedicated profile and link it to the target cluster. Check the following [instructions](/customize-workflow/alerting/resiliency-alerts) to complete the setup.
#### Resilience alerts dashboard
{% hint style="info" %}
Clicking on a specific alert opens the alert investigation window, providing deeper insights for further analysis.
{% endhint %}
Resilience alerts dashboard
Column
Description
Status
Indicates the risk level (High, Medium, or Low), and alert status (Active or Inactive). Use filters to prioritize issues and focus on the most important alerts seamlessly. Multiselect is available.
Alert Details
Displays the indicator that triggered the alert. Filter the data by selecting specific risk types for a more focused analysis. Multiselect is available.
Cluster
Displays clusters associated with alerts. Customize the view by selecting a cluster to see relevant alerts. Multiselect is available.
Namespace
Displays namespaces associated with alerts. Customize the view by selecting a namespace to see relevant alerts. Multiselect is available.
Workload
Displays workloads associated with alerts. Customize the view by selecting a workload to see relevant alerts. Multiselect is available.
Automation status
Indicates whether the workload is under Automation and its status at the time of the Alert. Learn more about Automation status here.
Type
Displays workload types associated with alerts. Customize the view by selecting a workload type to see relevant alerts. Multiselect is available.
Container
Displays containers associated with alerts. Customize the view by selecting a container to see relevant alerts. Multiselect is available.
First Seen
Displays the date the alert appeared for the first time. Sortable in ascending/descending order.
Last Seen
Displays the date of the alert appeared for the last time. Sortable in ascending/descending order.
Acknowledged by
Indicates whether a team member has taken ownership of a particular alert and displays the user's name. Filter and display acknowledged alerts, acknowledged alerts by specific users, or unacknowledged alerts. Multiselect is available. Learn more about Alerts Acknowledgement here.
Notified via
Displays the channel through which the notification about the alert was sent. Filter and display of alerts by specific notification channels. Multiselect is available.
# Understanding K8s scaling
PerfectScale simplifies the right-sizing and right-scaling of Kubernetes environments
PerfectScale's agent (stateless deployment, one per K8s cluster) continuously observes scaling, resiliency, and performance metrics across the entire K8s stack, including workloads, infrastructure, etc.
The agent is constantly looking for two types of issues:
* Risky under-provisioning or missing definitions that could harm uptime and SLA.
* Over-provisioning, where compute resources are constantly underutilized, results in waste and excessive CO2 emissions.
Those issues could affect Vertical Scaling, Horizontal Scaling, or both.
## Vertical scaling
Vertical scaling refers to the amount of resources (such as CPU and memory) allocated to a particular instance, like a Container, Pod, Node, etc. When we scale up or down vertically, we modify the amount of resources available for each instance.
\
To achieve proper scheduling and runtime, all Pods' containers should have defined resources (requests, and in many cases, limits), and underlying infrastructure (worker nodes) should have the proper resource capacity to host the Pods (containers).
## Horizontal scaling
Horizontal scaling refers to parallelization. To achieve the desired level of performance or resiliency, we might need multiple instances of the same workload or infrastructure (to ensure High Availability, processing parallelization, etc.)
Kubernetes and the open-source community provide multiple tools to control horizontal scaling. However, those should be tuned appropriately to bring the desired values.
PerfectScale supports the following horizontal scaling products/solutions :
* HorizontalPodAutoscaler
* Keda
* ClusterAutoscaler
* AWS Karpenter
* GCP Autopilot
# System & user-defined restrictions
An overview of system and user-defined rules and restrictions
## 🤖 **System-defined safeguards**
System-defined rules automatically apply restrictions due to technical limitations or when Automation cannot execute safely, to prevent risks and instability.
| Rule type | Tooltip (automated workload) | Tooltip (non-automated workload) |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| ResourceQuota | Automation is restricted to avoid exceeding ResourceQuota constraints. | Recommendation is restricted to honor ResourceQuota constraints. Current values are kept for stability. |
| LimitRange | Recommended value is outside the defined LimitRange. Current values were kept for stability. | Recommendation is restricted to honor LimitRange constraints. Current values are kept for stability. |
| NodeAware Recommendation | Automation is restricted because the recommended resources exceed node capacity, preventing scheduling issues. | Recommendation is capped at node capacity. Current values are kept for stability. |
| HPA with Sidecar | Automation is restricted to avoid triggering HPA due to sidecar underprovisioning. | Recommendation is constrained to avoid triggering HPA. Current values are kept for stability. |
| Sidecar | Sidecar allocations are immutable. Automation will execute only pod-level recommendations. | Sidecar allocations are immutable |
| OnDelete Strategy | Automation is restricted because OnDelete strategy requires pods to be manually deleted for updates to apply. | - |
| kube-system Namespace | Automation is restricted because kube-system workloads must be explicitly enabled. | - |
| DaemonSet | Automation is restricted to decrease-only mode for DaemonSets to prevent over-provisioning. | - |
| `CPULimitRemovalRecommended` | Memory resources are not reduced because of the recommendation to remove the CPU limit. Removing the CPU limit might increase memory usage. | Memory resources are not reduced because of the recommendation to remove the CPU limit. Removing the CPU limit might increase memory usage. |
| `CPULimitRemovalRecommended` | CPU resources are not reduced because of the recommendation to remove the CPU limit. Removing the CPU limit might increase CPU usage. | CPU resources are not reduced because of the recommendation to remove the CPU limit. Removing the CPU limit might increase CPU usage. |
| `CPULimitRemovedRecently` | Memory resources could not be reduced because of the recent CPU limit removal. Removing the CPU limit might increase memory usage. | Memory resources could not be reduced because of the recent CPU limit removal. Removing the CPU limit might increase memory usage. |
| `CPULimitRemovedRecently` | CPU resources could not be reduced because of the recent CPU limit removal. Removing the CPU limit might increase CPU usage. | CPU resources could not be reduced because of the recent CPU limit removal. Removing the CPU limit might increase CPU usage. |
| Xmx, Xms MaxHeapSize above recommendation | Automation restricted: MaxHeapSize is explicitly set to {{.ThresholdFormatted}} {{.Unit}}. Automation will not reduce below the configured value. | MaxHeapSize is explicitly set to {{.ThresholdFormatted}} {{.Unit}}. Recommendation will not reduce below the configured value. |
| Insignificant resource change | Resource change is insignificant, keeping recommendation at {{.ThresholdFormatted}} {{.Unit}} | Resource change is insignificant, keeping recommendation at {{.ThresholdFormatted}} {{.Unit}} |
## 👨🏻💻 **User-defined rules**
Restrictions triggered by user preference configurations.
| Rule type | Tooltip |
| ---------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
Automation mode Disabled
(automationMode)
| Automation mode is set to “Disabled" |
| Global kill‑switch (`stopAllAutomation`) |
Cluster kill-switch (stopAllAutomation) is active, halting all automated actions.
|
|
Minimum resiliency level to ignore maintenance window
(maintenanceWindowIgnoredMinResiliencyLevel)
|
Resiliency risk is below the configured threshold outside the maintenance window.
Monthly waste is below the configured threshold amount of \.
|
|
CPU request increase not allowed (increaseEnabled: false)
| Increasing CPU requests is disabled by configuration. |
|
CPU request decrease not allowed (decreaseEnabled: false)
| Decreasing CPU requests is disabled by configuration. |
| Minimum CPU requests (`minimumCores`) |
Recommended CPU request is below the configured minimum \ cores, using the minimum value.
|
| Maximum CPU requests (`maximumCores`) |
Recommended CPU request exceeds the configured maximum \ cores, using the maximum value.
|
| Keep CPU limit (`keepLimit`) | CPU limit removal is disabled by configuration, retaining the existing limit. |
| Keep CPU limit (`keepLimit`) | CPU request could not be reduced because keep CPU limit configured and workload experiences CPU throttling. Reducing CPU request could cause a higher throttlingю |
| Memory request increase not allowed (`increaseEnabled: false`) | Increasing memory requests is disabled by configuration. |
| Memory request decrease not allowed (`decreaseEnabled: false`) | Decreasing memory requests is disabled by configuration. |
| Minimum memory request (`minimumGiB`) |
Recommended memory request is below the configured minimum \ GiB, using the minimum value.
|
| Maximum memory request (`maximumGiB`) |
Recommended memory request exceeds the configured maximum \ GiB, using the maximum value.
|
|
Memory limit increase not allowed
(increaseEnabled: false)
| Increasing memory limits is disabled by configuration. |
|
Memory limit decrease not allowed
(decreaseEnabled: false)
| Decreasing memory limits is disabled by configuration. |
| Minimum memory limit (`minimumGiB`) | Recommended memory limit is below the configured minimum `` GiB, using the minimum value. |
| Maximum memory limit (`maximumGiB`) | Recommended memory limit exceeds the configured maximum \ GiB, using the maximum value. |
| Daily max memory limit increases (`maxMemoryIncreaseIterations.daily`) | Daily memory increase limit reached, waiting until tomorrow. |
| Weekly max memory limit increases (`maxMemoryIncreaseIterations.weekly`) | Weekly memory increase limit reached; waiting until next week. |
|
Maintenance window
(maintenanceWindow)
| Currently outside the maintenance window, the recommendation will be applied in the next maintenance window. |
| Concurrent ReplicaSets (`duringRolloutInProgress: pause`) | Resource changes are not allowed when multiple concurrent ReplicaSets are detected |
# PerfectScale Agent
PerfectScale psc\_exporter metrics and associated alerts overview
{% hint style="info" %}
For large clusters, if you are not using automation, use PerfectScale’s recommendations for `psc-exporter` to properly size it.
{% endhint %}
PerfectScale Exporter installed in your Kubernetes cluster provides helpful [metrics](#metrics) for monitoring, reviewing, and alerting on PerfectScale platform behavior. These metrics will instantly detect abnormal behavior and notify you through [alerts](#alerts).
## Metrics
The Exporter presents metrics in the Prometheus format and utilizes two categories of metrics:
1. Counter - the increasing metric
* [ksm\_scraping\_errors\_total](#ksm_scraping_errors_total)
* [cadvisor\_scraping\_errors\_total](#cadvisor_scraping_errors_total)
* [time\_windows\_upload\_errors\_total](#time_windows_upload_errors_total)
* [auth\_errors\_total](#auth_errors_total)
* [upload\_policy\_errors\_total](#upload_policy_errors_total)
2. Gauge - system state at a specific time point. This metric can go down and can go up.
* [ksm\_instances\_scraped](#ksm_instances_scraped)
* [cadvisor\_instances\_scraped](#cadvisor_instances_scraped)
PerfectScale Exporter takes [Kube State Metrics (KSM)](#ksm-metrics) and [cAdvisor Metrics](#cadvisor-metrics) as inputs and uploads [compressed metrics to PerfectScale SaaS](#perfectscale-metrics) as output.
### KSM Metrics
These metrics monitor the communication between the Exporter and the KSM in your cluster.
#### ksm\_instances\_scraped
* **Metric Name**: `psc_exporter_ksm_instances_scraped`
* **Description**: The number of KSM instances the exporter tried to scrape during the latest scraping round. Usually, you have only 1.
* **Type**: Gauge
#### ksm\_scraping\_errors\_total
* **Metric Name**: `psc_exporter_ksm_scraping_errors_total`
* **Description**: A counter that grows incrementally to keep track of errors during KSM scraping.
* **Type**: Counter
### cAdvisor Metrics
cAdvisor component provides the Exporter with information on specific containers and their resource usage.
#### cadvisor\_instances\_scraped
* **Metric Name**: `psc_exporter_cadvisor_instances_scraped`
* **Description**: The number of cAdvisor instances the exporter tried to scrape during the latest scraping round.
* **Type**: Gauge
#### cadvisor\_scraping\_errors\_total
* **Metric Name**: `psc_exporter_cadvisor_scraping_errors_total`
* **Description**: A counter that grows incrementally to keep track of errors during cAdvisor scraping.
* **Type**: Counter
### PerfectScale Metrics
PerfectScale provides an additional group of metrics that pertains to exporters communicating with the PerfectScale SaaS platform.
#### time\_windows\_upload\_errors\_total
* **Metric Name**: `psc_exporter_time_windows_upload_errors_total`
* **Description**: Total number of failed attempts to upload time windows to PerfectScale.
* **Type**: Counter
#### auth\_errors\_total
* **Metric Name**: `psc_exporter_auth_errors_total`
* **Description**: Total number of PerfectScale exporter authorization errors.
* **Type**: Counter
#### upload\_policy\_errors\_total
* **Metric Name**: `psc_exporter_upload_policy_errors_total`
* **Description**: Total number of errors when updating the upload policy.
* **Type**: Counter
## Alerts
Alerts are useful in immediately informing about the abnormal behavior of the Exporter based on the [metrics](#metrics) described above.
### How to configure the Alerts
To enable or configure the Alerts, update the Helm values according to your requirements.
#### Helm Values example
```yaml
serviceMonitor:
enable: true
prometheusRule:
enable: true
labels:
customLabel: "value"
annotations:
customAnnotation: "value"
team: "operations"
severity: "critical"
cAdvisorScraping:
timeRange: "15m"
threshold: 0.5
```
The [PerferfectScale Agent's upstream helm values.yaml](https://github.com/perfectscale-io/perfectscale-io.github.io/blob/main/charts/perfectscale-agent/values.yaml#L231-L261) file, can be referenced for a more complete example of available options.
### Alerts overview
#### PerfectScale Exporter High KSM Scraping Error Rate
* **Alert Name**: PerfectScale Exporter High KSM Scraping Error Rate
* **Description**: Within the last 5 minutes, over 30% of kube-state-metrics scraping attempts have failed.
#### PerfectScale Exporter High cAdvisor Scraping Error Rate
* **Alert Name**: PerfectScale Exporter High cAdvisor Scraping Error Rate
* **Description**: Over a specified percentage of cAdvisor scraping attempts have failed within the specified time range.
#### PerfectScale Exporter Time Windows Upload Error Rate
* **Alert Name**: PerfectScale exporter Time Windows Upload Error Rate
* **Description**: Within 1 hour, three or more time windows upload errors to PerfectScale occurred.
#### PerfectScale Exporter Authorization Errors
* **Alert Name**: PerfectScale exporter Authorization Errors
* **Description**: Within 1 hour, two or more PerfectScale Exporter authorization errors occurred.
#### PerfectScale Exporter Upload Policy Refresh Errors
* **Alert Name**: PerfectScale exporter Upload Policy Refresh Errors
* **Description**: Within 1 hour, two or more errors occurred when updating the upload policy.
# PerfectScale agent skill
Explore how to bring agentic workflows into your Kubernetes optimization process with the PerfectScale agent skill
## What is PerfectScale CLI
PerfectScale CLI is a command-line tool that uses PerfectScale’s public APIs to access cluster cost and optimization data. It lets you work with AI agents such as Claude, Codex, and Antigravity to query clusters, workloads, namespaces, costs, waste, risks, and other metrics, and combine this data with information from other sources to get a more complete operational context.
## When to use
PerfectScale CLI introduces various use cases to improve your optimization workflows.
* Query clusters, workloads, namespaces, costs, waste, and risks from the command line
* Aggregate and rank data, group by namespace, team label, workload type, risk\
severity
* Export to CSV, JSON, or JSONL for downstream consumption
* Review automation audit logs with filtering by cluster, namespace, and time range
* Pipe results into any workflow: dashboards, Slack bots, CI/CD pipelines, custom\
reports
## Getting started
{% stepper %}
{% step %}
**Download the skill**
Use your Agent to install PerfectScale CLI skill with a single prompt:
```
Install the skill from https://github.com/doitintl/perfectscale-cli
```
Alternatively, you can do it manually. Download and install the latest release for your platform from:
{% embed url="" %}
{% endstep %}
{% step %}
**Generate an API Token**
1. Open [app.perfectscale.io](app.perfectscale.iohttps://app.perfectscale.io/)
2. Click the user icon in the bottom-left corner
3. Go to Org Settings -> API Tokens
4. Click Generate Token and assign a Read Only role
5. Copy the `client_id` and `client_secret`
{% endstep %}
{% step %}
**Authenticate**
```bash
pscli auth login --client-id 'YOUR_CLIENT_ID' --client-secret 'YOUR_CLIENT_SECRET'
```
The CLI validates the token and saves a local profile. Credentials are stored securely in your OS config directory with restricted file permissions.
{% endstep %}
{% step %}
**Start Querying**
```bash
pscli clusters list # List your clusters
pscli workloads list -c prod-a -s waste -r desc -T 10 # Top 10 wasteful workloads
pscli workloads summary -c prod-a # Cost summary for a cluster
pscli workloads export -c prod-a -F workloads.csv # Export to CSV
```
{% endstep %}
{% endstepper %}
{% hint style="warning" %}
Known limitations
* Only service-token auth is supported
* Only the public API is supported
* Workloads are fixed to a `30d` period because the public endpoint is fixed-window
* Namespace and many workload filters are client-side
* There is no first-class public nodegroup command
* CSV is the only export format
{% endhint %}
# PerfectScale data collected
Explore what the data PerfectScale collects in order to provide service
In order to provide service, PerfectScale collects the following data:
1. cAdvisor metrics:
```yaml
container_cpu_cfs_periods_total
container_cpu_cfs_throttled_periods_total
container_cpu_usage_seconds_total
container_memory_working_set_bytes
container_network_receive_bytes_total
container_network_transmit_bytes_total
```
2. kube-state-metrics metrics:
```
kube_pod_labels
kube_pod_annotations
kube_pod_info
kube_pod_container_status_running
kube_pod_container_status_terminated
kube_pod_container_status_terminated_reason
kube_pod_container_status_last_terminated_reason
kube_pod_container_resource_requests
kube_pod_container_resource_limits
kube_pod_container_status_restarts_total
kube_pod_start_time
kube_pod_completion_time
kube_node_info
kube_node_status_capacity
kube_node_status_allocatable
kube_node_status_condition
kube_node_labels
kube_pod_status_phase
kube_job_owner
kube_replicaset_owner
kube_horizontalpodautoscaler_info
kube_horizontalpodautoscaler_labels
kube_horizontalpodautoscaler_spec_target_metric
kube_horizontalpodautoscaler_spec_min_replicas
kube_horizontalpodautoscaler_spec_max_replicas
kube_horizontalpodautoscaler_status_desired_replicas
kube_horizontalpodautoscaler_status_current_replicas
kube_pod_container_info
kube_namespace_labels
kube_pod_status_reason
kube_job_status_failed
```
3. The data about containers’ spec (original) resources (spec CPU\&Memory Requests\&Limits) from `kubeapiserver`.
4. container-jvm metrics collected from coroot-node-agent:
```
container_jvm_heap_used_bytes
container_jvm_heap_size_bytes
container_jvm_gc_time_seconds
container_jvm_info with labels extended with java VM.flags values
```
# PerfectScale Autoscaler Objects' Events
Explore PerfectScale autoscaler objects' events
Event name
Object
Event type
Event message
Description
RefObjectNotFound
WorkloadAutomationConfig
Warning
Workload specified in TargetRef not found.
The event indicates that the K8s object specified in the TargetRef of the Workload Automation Config does not exist. The event could appear only in the config creation stage.
UpdatingWorkload
WorkloadAutomationStatus (k8s object associated with the workload, e.g. Job/CronJob)
Normal
Received new resource recommendations for {objectNamespace}/{objectName}
This event is applicable only to non-restartable workloads, such as Jobs and CronJobs.
The event indicates that new resource recommendations have been applied to WorkloadAutomationStatus (moved from target to current section).
Since the autoscaler cannot restart such workloads, new recommendations will be applied to pods during the next pod creation scheduled by K8s.
RestartingWorkload
WorkloadAutomationStatus (k8s object associated with the workload, e.g. Deployment/DaemonSet/StatefulSet)
Normal
Received new resource recommendations for {objectNamespace}/{objectName}
The event is only relevant for the restartable workloads (e.g., Deployment/DaemonSet/StatefulSet).
The event indicates that new resource recommendations have been applied to WorkloadAutomationStatus (moved from target to current section), AND autoscaler started rollout-restart for the workload (to apply the new recommendations).
Restart is needed for pod re-creation to apply the new recommendations.
RestartFailed
WorkloadAutomationStatus (k8s object associated with the workload, e.g. Deployment/DaemonSet/StatefulSet)
Warning
Failed to restart {objectNamespace}/{objectName}: {errorMessage}
The event is only relevant for restartable workloads (e.g., Deployment/DaemonSet/StatefulSet).
The event indicates that the restart failed from the autoscaler perspective. Possible reasons:
We failed to start the restart process (the k8s client returned an error on the http patch request).
The deadline for restarting all pods has been exceeded (means not all pods have our latest restartedAt annotation and have running state).
MutatingPod
WorkloadAutomationStatus
Normal
Applying PerfectScale recommendations to {podNamespace}/{podName}
The event indicates that recommendations for the specified pod exist and can be applied (no R1 happened, etc.).
MutatedPod
WorkloadAutomationStatus
Normal
Successfully applied PerfectScale recommendations to {podNamespace}/{podName}
The event indicates that recommendations for the specified pod were properly converted to JSON and returned to k8s. Therefore, recommendations should be applied to the pod.
MutatePodFailed
WorkloadAutomationStatus
Warning
Failed to apply PerfectScale recommendations to {podNamespace}/{podName}
The event indicates that recommendations for the specified pod can not be properly converted to JSON and, as a result - were not returned to K8s. Therefore, recommendations will not be applied to the pod.
# Outbound Request Ports used by the PerfectScale Agent and Autoscaler
Explore outbound request ports used by the exporter and autoscaler
## API
Port 443, TCP
```
api.app.perfectscale.io:443 (https)
```
## S3
1. If you **can** enable egress to AWS S3 in **eu-central-1** by path style:
**Port 443, TCP**
*
*
2. If you **can not** enable egress to AWS S3 in **eu-central-1** by path style:
**Port 443, TCP**
*
*
{% hint style="info" %}
:bulb:**NOTE**: the additional configuration on the PerfectScale side is required. If the **path style is not allowed**, please contact customer support via [support@perfectscale.io](mailo:support@perfectscale.io) or our [Slack Community](https://join.slack.com/t/perfectscalecommunity/shared_invite/zt-1tu9teu9e-Z9tGt4LpNI8tUC3j8obcmQ), indicating that you need to use a virtual hosted style endpoint.
{% endhint %}
## Images Pull
To allow your kubelet to pull the PerfectScale Exporter and Autoscaler images, make sure the following domain is whitelisted for the worker node:
```
https://public.ecr.aws/perfectscale-io
```
# PerfectScale Weekly Report
Receive PerfectScale insights into the cost and waste of your Kubernetes directly into your inbox
PerfectScale automatically sends a Weekly Report summarizing key metrics from the past week directly to your inbox. This report offers an overview of the largest changes across your account, helping you track and review past events. This feature ensures you stay on top of your Kubernetes changes, unlocking quick access to further details.
## Weekly Report content
### Weekly summary
The weekly summary provides high-level insights into the cost and waste of your Kubernetes environment running on PerfectScale. It highlights how these metrics have changed compared to the previous period, enabling you to quickly access key cost-related trends and proceed with further analysis with one click.
With this single view, you will clearly understand your clusters' weekly costs and pod waste, seeing exactly how they have increased or decreased compared to the previous period. Additionally, you can review the number of alerts generated and actions taken by Automation, helping you stay informed about recent events and maintain visibility into your clusters' performance and efficiency.
Weekly report summary
Click the **`Open Overview`** button to access the PerfectScale app directly, where you can view more details and continue your analysis.
### Detailed metrics overview
In this section, PerfectScale collects important metrics to help you effortlessly evaluate key trends in your environment. These insights reveal data-driven optimization opportunities that could significantly enhance your Kubernetes efficiency while highlighting areas that may require further improvement.
It includes **Cost by Namespace**, offering a daily breakdown of expenses associated with each namespace, highlighting the idle costs, and making it easier to understand and manage resource spending. The **Pod Waste by Namespace** metric identifies the cost of unused resources for each workload, helping to pinpoint inefficiencies. In the **Most Wasteful Workloads** section, you will find the workloads that waste the most resources, highlighting significant optimization opportunities. Additionally, **Top New Alerts Observed This Week** summarizes the latest and most critical alerts, helping to identify and prioritize areas needing immediate attention.
## How to get the Weekly Report
Users with the [**WeeklyReportReceiver role**](/administration/rbac-or-roles-and-permissions) automatically receive the Weekly Report at the email address specified when their account was created.
A user with **Admin** permissions can manage this role in the **User Management** section. To access it, click the **Account** **button** in the bottom-left corner, then go to **Organization Settings → Users**.
From there, you can invite a new user and assign the **WeeklyReportReceiver** role, or grant this role to an existing user by selecting **Edit roles** from the menu next to their name.
# Theme Settings
Learn how to switch between Light, Dark, and System modes to customize the platform’s appearance to your preference
You can seamlessly switch between Light, Dark, and System modes. To do so, click the **Account button** in the bottom-left corner, hover over **UI Theme**, and select your preferred option.
Theme settings
# Customization overview
Explore how to customize your Kubernetes optimization journey
PerfectScale allows easy cluster management through profiles, increasing productivity with minimal effort. PerfectScale provides various customizations to help you create your own K8s optimization ecosystem, enhancing observability across your environment and simplifying alerting, ticketing, pricing management, etc., to ensure you achieve the best possible results.
To make your optimization process seamless and personalized, PerfectScale provides various out-of-the-box integrations, helping you to build your optimization ecosystem tailored to your business needs.
{% hint style="success" %}
You can configure integrations using either UI profiles or CR-based inline profiles. However, **we strongly recommend you configure the integrations via PerfectScale UI**.
**Profiles referenced via CR take precedence over profiles assigned in UI.**
{% endhint %}
You can quickly configure integrations using **Customization Profiles** (YAML files) via UI, or customize your environment by configuring profiles with **CRs**.
{% hint style="info" %}
You can apply multiple profiles to a single cluster, but only one profile of each type (Slack, Jira, etc.).
{% endhint %}
## :sparkles: Recommended path
1. Create and test profiles in the UI first.
2. Use refByID to reference the profile by its ID (case-sensitive).
3. Ensure the profile type you are referencing matches the existing profile type.
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
pricing:
# Reference an existing pricing profile by ID
# No assigned field needed - refByID is always assigned
- type: custom
refByID: "pricing-1"
```
## Configuring integration profile via UI
You can seamlessly create an integration profile directly from the PerfectScale UI. Simply go to **Settings**, select the **integration type**, and click **Add profile**. This approach allows you to set up and manage integrations quickly and accurately. Detailed step-by-step instructions for each integration type are provided on their dedicated integration pages.
Settings
{% hint style="info" %}
When a profile is created in the UI, its ID has the following format: `{type|class}-N`. Underscores in type/class names are converted to hyphens in IDs.\
\
**Examples**: `jira-1`, `slack-2`, `resiliency-alerts-1`, etc.
{% endhint %}
Created profiles in the UI can be referenced in a CR using `refByID`.
**Example:**
```
- type: jira
refByID: "jira-1"
```
## Configuring inline integration profile via CR
Alternatively, you can configure an inline integration profile using a CR. This approach is ideal for teams that prefer GitOps workflows. Detailed step-by-step instructions for each integration type are provided on their dedicated integration pages.
The inline profile configured via CR is visible in UI and can be tested before activation with a single click. You cannot modify or delete a profile configured via CR in the UI. You can make temporary changes for testing, but these changes will not be saved.
{% hint style="info" %}
Inline profiles will be displayed in the UI using the following name format: `{your profile name}-{type/class}-{clusterUID}`
**Example:** `dev-team` profile with the type `Jira` \
`dev-team-jira-1522a0a0-b363-4e6c-bc0d-cfd82c56402e`
{% endhint %}
Inline profiles will be displayed in the UI using the following name format:
```
{your profile name}-{type/class}-{clusterUID}
```
You can seamlessly set `assigned: false` to create profiles without activating them immediately (for example, for testing). In that case, the profile will be visible in the UI but inactive. Only one profile per type (Slack, Jira, etc.) can have `assigned: true`.
{% hint style="warning" %}
Profiles created via CRD are cluster-specific and cannot be referenced by other clusters.
{% endhint %}
### Prerequisites for configuring the integration profile via CR
{% hint style="info" %}
The feature is supported starting from the exporter **v1.0.80**.
{% endhint %}
If **psc-exporter (v1.0.80+)** is newly installed, the required **CRDs are installed automatically**, allowing you to manage cluster settings using custom resources (CRs). However, CRDs are **not installed automatically** when running a `helm upgrade`. If you are upgrading, you must install the CRDs manually by running:
```
kubectl apply -f https://github.com/perfectscale-io/perfectscale-io.github.io/blob/main/charts/perfectscale-agent/crds/clustersettings.yaml
```
## Combining refByID and inline integrations
{% hint style="success" %}
We highly recommend creating profiles via UI and referencing them using `refByID` rather than inline profiles. If you prefer using inline profiles, you can test them in UI before activation, even when `assigned: false`.
{% endhint %}
You have the option to configure a CR that combines multiple integrations, allowing you to manage them with a single resource. Additionally, you can include existing profiles that were previously configured via the PerfectScale UI, giving you flexibility to mix and match inline and UI-based configurations in one setup.
You can combine both approaches by referencing profiles created in the PerfectScale UI directly in your CR using **refByID (case-sensitive)**. However, refByID cannot reference profiles created via CR, and a CR cannot have both `refByID` and an inline profile with `assigned: true` for the same type.
**Example:**
* ✅ **Valid:** `refByID` + inline profiles with `assigned: false` (for testing)
* ❌ **Invalid:** `refByID` + inline profile with `assigned: true`
{% hint style="info" %}
When using **refByID**, ensure the profile type matches the existing profile's type: jira, slack, teams, etc.
{% endhint %}
#### Example
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
# Pricing: Create inline custom pricing profile and assign it
pricing:
- type: custom
name: production-pricing
assigned: true
value:
nodeTypes:
- instanceType: c5.large
pricing:
cpuCoreHourPrice: 0.085
memGBHourPrice: 0.0095
- instanceType: c5.xlarge
pricing:
cpuCoreHourPrice: 0.085
memGBHourPrice: 0.0095
- instanceType: m5.large
pricing:
cpuCoreHourPrice: 0.096
memGBHourPrice: 0.0096
# Create AWS CUR profile but don't assign it yet (for testing)
- type: aws_cur
name: aws-cur-backup
assigned: false
value:
role_arn: "arn:aws:iam::your-account-id:role/your-role-name"
aws_external_id: "your-unique-external-id-here"
athena_database: athenacurcfn_perfectscale_cur
athena_region: us-east-1
athena_result_bucket: s3://perfectscale-cur-results/
athena_table: perfectscale_cur_hourly
aws_account_id: "your-aws-account-id-here"
# Integrations: Mix inline Slack and referenced Jira
integrations:
# Create and assign inline Slack integration
- type: slack
name: team-alerts
assigned: true
value:
channel: "perfectscale-production"
routings:
- label_alert
- cost_waste_alert
slack_token_from:
secretKeyRef:
name: slack-credentials
key: bot-token
# Reference existing Jira integration (created in UI)
- type: jira
refByID: "jira-1"
# Resiliency Alerts: Create inline profile
resiliency_alerts:
- name: production-alerts
assigned: true
value:
min_risk_level: high
ignore_workload: "^(test-.*|dev-.*)"
ignore_namespace: "^(kube-system|kube-public)$"
ignore_container: "^(istio-proxy|envoy)$"
ignore_indicator: ""
# PodFit Labels: Reference existing profile
podfit_labels:
- refByID: "label-1"
# Customization: Create inline profile
customization:
- name: app-labels
assigned: true
value:
workload_labels:
- app.kubernetes.io/name
- app.kubernetes.io/component
- team
- cost-center
node:
spot_labels:
label_cloud_google_com_gke_nodepool: pool-2
label_spotinst_io_node_lifecycle: spot
node_group_labels:
- eks.amazonaws.com/nodegroup
- kops.k8s.io/instancegroup
architecture_labels:
- kubernetes.io/arch
os_labels:
- kubernetes.io/os
instance_type_labels:
- node.kubernetes.io/instance-type
region_labels:
- topology.kubernetes.io/region
availability_zone_labels:
- topology.kubernetes.io/zone
ignored_pod_labels:
- uid
- namespace
ignored_node_labels:
- uid
```
{% hint style="info" %}
CR profiles (both refByID and inline) take precedence over UI-assigned profiles and will override them.
{% endhint %}
## Deleting a profile
{% hint style="danger" %}
Profiles assigned to clusters cannot be deleted.
{% endhint %}
To delete a profile, it **must first be unassigned** from all clusters.\
If the profile is still assigned to one or more clusters, deletion is blocked, and a pop-up message will inform you of the reason with the following instructions.
Unassign a cluster befor deleting
To unassign a profile from all clusters, navigate to **Manage Assignments** and deselect the relevant profile.
{% hint style="warning" %}
Profiles created via CR cannot be deleted in the UI, so the delete option is disabled.
{% endhint %}
## Troubleshooting
#### CR creation failed
* Check if ClusterSettings already exists: `kubectl get clustersettings -n perfectscale`
* Review logs: `kubectl logs -n perfectscale -l app=perfectscale-exporter`
#### Profile not applied
* Check logs for sync errors
* Verify secrets exist: `kubectl get secrets -n perfectscale`
* Check profile format matches the type
#### Secret not found
* Verify secret exists: `kubectl get secret -n perfectscale`
* Ensure the secret is in `perfectscale` namespace
#### Profile deletion failed
* Ensure the profile is unassigned from all clusters
* If it is still assigned, navigate to **Manage Assignments** and deselect the relevant profile.
# Alerting
Get instant alerts across your environment, and focus and address the most impactful issues through advanced auto-prioritization
PerfectScale delivers timely and reliable alerts to keep you informed about unusual system activity and detected anomalies. These alerts are specifically designed to highlight changes in key indicators, enabling you to take quick action to address potential issues before they impact your system's efficiency, performance, or stability.
PerfectScale allows you to easily set up resiliency and financial alerts and manage them efficiently by configuring the corresponding alert profile:
* [Resiliency alerts configuration](/customize-workflow/alerting/resiliency-alerts)
* [Financial alerts configuration](/customize-workflow/alerting/financial-alerts)
# Resiliency alerts
Resolve resilience risks proactively before they affect your users with impact-driven prioritization
## What are resiliency alerts
Alerts are designed to quickly identify and notify of relevant indicator changes, ensuring they can be eliminated before impacting the system.
{% hint style="info" %}
For faster updates, utilize [Slack](/customize-workflow/communication-and-messaging/slack-integration), [MS Teams](/customize-workflow/communication-and-messaging/ms-teams-integration), or [DataDog Alerts](/customize-workflow/communication-and-messaging/datadog-alerts-integration) Integrations to receive notifications when an Alert is generated.
{% endhint %}
{% hint style="success" %}
By default, PerfectScale generates alerts for every cluster where a [resilience indicator](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing/understanding-at-risk-indicators) with a [min\_risk\_level: high](/customize-workflow/alerting/resiliency-alerts#customize-the-alerts-by) was identified.
In order to customize your alerts, an [Alerts Profile](/customize-workflow/alerting/resiliency-alerts) must be set up and applied to the cluster. It will override the default alert generation process, ensuring identification and notification about relevant indicator changes.
{% endhint %}
Resiliency alerts profile
{% hint style="info" %}
You can apply only one Resiliency Alert Profile per cluster.
{% endhint %}
{% hint style="info" %}
A cluster that transmits data is required to start receiving alerts configured with an Alert Profile.
{% endhint %}
## Configuring resiliency alerts with Alerts Profile
### :tools: **How to create** **Alerts Profile**
There are two options for creating a Profile: [from the Settings tab](#from-the-settings-tab) or directly [from the Overview](#from-the-overview-tab).
#### **From the Settings tab.**
Go to the **`Settings`** tab on the left panel -> select **`Alerts`** -> click the **`+Add Profile`** button -> name the profile in the corresponding row -> specify the `min_risk_level` (low, medium, high) and ignored entities (if needed) -> click **`Save`** button.
Resiliency alerts profile from settings
{% hint style="info" %}
If the `min_risk_level` is set to medium, only indicators with `high` or `medium` severity will trigger alerts.
{% endhint %}
#### From the **Overview** tab
Go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Alert Profile`** and click **`gear`** button -> go to **`Customizations`** -> click on **`Add New Profile`** in the **`Alerts`** drop-down list -> name the profile in the corresponding row -> specify the `min_risk_level` (low, medium, high) and ignored entities (if needed) -> click the **`Save And Apply`** button.
Resiliency alerts profile from overview
#### Alerts Profile Configuration
Resiliency alerts profile configuration
1. Name the profile.
2. Configure the profile:
* `min_risk_level` - the minimum risk level of the issue to trigger the alert.\
\
:bulb:**NOTE**: if the `min_risk_level` is set to medium, only indicators with `high` or `medium` severity will trigger alerts.
* `ignore_namespace` - excludes specific namespaces from triggering resiliency alerts..
* `ignore_workload` - excludes specific workloads from triggering resiliency alerts..
* `ignore_container` - excludes specific containers from triggering resiliency alerts.
* `ignore_indicator` - excludes specific [resiliency issues indicators](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing/understanding-at-risk-indicators) from triggering alerts.
* `active_notification_resend` - enables the configuration of the active alert resend feature at a specified interval to ensure critical alerts remain visible until addressed.\
\
:bulb: **NOTE**: `active_notification_resend` is set to `off` by default.\
\
:point\_right: **Examples**: \
Set `active_notification_resend: 5h` to resend notifications for active alerts every 5 hours.\
Set `active_notification_resend: 1d` to resend notifications for active alerts every day.
### :tools: **How to apply** **Alerts Profile**
#### Apply to a single cluster
To apply **`Alert Profile`** to the cluster, go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Alert Profile`** and click **`gear`** button -> go to **`Customizations`** -> select the needed profile in the **`Alerts`** drop-down list.
Applying resiliency alerts profile to a single cluster
#### Apply to multiple clusters
To apply the profile to **multiple clusters** from a single view, use the **`Manage Assignments`** feature.
Go to the **`Settings`** tab on the left panel -> select the **`Alerts`** -> click the **`Manage Assignments`** button -> apply the profiles for the needed clusters -> click the **`Save Changes`** button.
:bulb:***NOTE***: You can easily manage your profiles (create, delete), but deleting such profiles is impossible if the profile is already connected to the cluster. Change the profile to default or any other, and only after that, remove the current one. If you change the Alert Profile to 'None', it will use the default action for alert generation (detected [resilience indicator](/visibility-and-optimization/podfit-or-vertical-pod-right-sizing/understanding-at-risk-indicators) with a [min\_risk\_level: high](/customize-workflow/alerting/resiliency-alerts#customize-the-alerts-by)).
{% hint style="info" %}
If you want to stop generating alerts to your communication channels, disconnect the messaging profile from the cluster ([Slack](/customize-workflow/communication-and-messaging/slack-integration), [MS Teams](/customize-workflow/communication-and-messaging/ms-teams-integration)).
{% endhint %}
## Configuring resiliency alerts integration with CR
To enable alerting for resiliency risks, you need to define and apply a Custom Resource (CR) that specifies your alert parameters. This approach allows you to manage alert configurations directly through Kubernetes manifests.
{% hint style="warning" %}
The Custom Resource (CR) must be created in the `perfectscale` namespace.
{% endhint %}
Here is an example of the CR configuration:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
resiliency_alerts:
- name: production-alerts
assigned: true
value:
min_risk_level: high
ignore_workload: "^(test-.*|dev-.*)"
ignore_namespace: "^(kube-system|kube-public|kube-node-lease)$"
ignore_container: "^(istio-proxy|envoy|linkerd-proxy)$"
ignore_indicator: "CpuThrottling, CpuRequestNotSet"
active_notification_resend: 'off'
```
#### ⚙️ **CR parameters:**
Parameter
Description
min_risk_level
Specifies the minimum risk level that triggers resiliency alerts.
Values: high, medium, low.
Example: if set to medium, alerts will be generated for both medium and high risk levels.
ignore_workload
ignore_namespace
ignore_container
Specifies the namespaces, workloads, or containers for which alerts are disabled. Use the regex pattern.
💡To ignore multiple entities in one row, list them separated by commas.
ignore_indicator
Specifies resiliency issues for which alerts will not be generated. Examples: OOM, CpuThrottling, CpuRequestNotSet, MemRequestNotSet, MemLimitNotSet, UnderProvisionedMemRequest, UnderProvisionedCpuRequest, UnderProvisionedMemLimit, UnderProvisionedCpuLimit, OverProvisionedCpuRequest, OverProvisionedMemRequest, RestartsObserved, EvictionsObserved, HPAAtMaxReplicasObserved
💡 Leave empty to not ignore any
active_notification_resend
Specifies the interval for re-sending notifications for active alerts. Values: off (default) - disables re-sending notifications; h - sets hourly interval d - sets daily interval Example: active_notification_resend: '2h' - re-sends notification for active alerts every 2 hours.
# Financial alerts
Instantly identify and address cost spikes and anomalies to keep your cloud bill in check
## What are financial alerts
PerfectScale Financial Alerts allow users to set up cost and waste alerts, notifying them when exceeding specific thresholds. This enables proactive cost management and resource optimization, helping teams stay within budget and reduce unnecessary spending.
To start receiving financial alerts, create a Financial Alerts Rule.
Financial alerts rule
### :tools: **How to create** Financial **Alerts Rule**
Go to the **`Settings`** tab on the left panel -> select **`Financial Alerts`** -> click the **`Create Alert Rule`** button -> configure the rule -> click **`Save`** button.
#### Financial Alerts profile configuration
Financial alerts rule configuration
{% hint style="info" %}
When more than one cluster, namespace, etc., is selected in Target, the combined total (SUM) values will trigger alerts.
{% endhint %}
**Alert Name**
Name your financial alert rule to seamlessly organize, filter, and analyze alerts on the [financial alerts dashboard](/configure-alerts/alerts-overview#financial-alerts).
**Target**
Defines the scope of monitoring for alerts:
* **Organization** - monitors metrics across the entire account;
* **Cluster** - monitors metrics of the specific cluster(s). Select cluster(s) from the cluster selector.
Financial alerts rule for cluster
* **Namespace** - monitors metrics of the specific namespace(s). Select the specific cluster and then the namespace(s) from the selector that appears. Use the **`+`** button to select multiple clusters.
Financial alerts rule for namespace
* **Workload** - monitors metrics of the specific workload(s). Select the specific cluster and then specify the namespace(s) and workload(s) from the selector that appears. Use the **`+`** button to select multiple clusters.
Financial alerts rule for workload
**Scope**
Defines the metric to monitor for triggering the alerts
* Cost - track the cost of specified objects in Target. When the defined metric exceeds the defined threshold, an alert will be triggered.
* Waste - track the waste of specified objects in Target. When the defined metric exceeds the defined threshold, an alert will be triggered.
**Period**
Specifies the metric evaluation window (Hour, Day, Week, Month).\
**Example**: If selected `Day` - PerfectScale will daily evaluate the metric and trigger an alert if the threshold is exceeded.
**Starting At**
Specifies the start date for the metric evaluation period.
**Threshold**
Specifies the conditions that will trigger an alert.
* **Amount** - defines a specific monetary value exceeding what will trigger an alert.
Financial alerts rule threshold1
* **Compared to the previous period** - specifies the max deviation of the value to trigger the alert compared to the previous period based on the selected evaluation window. Set a maximum allowed percentage increase and a minimal trigger value as thresholds.
Financial alerts rule threshold2
**Notification**
{% hint style="info" %}
If the notification channel is not specified, financial alerts will be only displayed on the [Financial Alerts dashboard](/configure-alerts/alerts-overview#financial-alerts).
{% endhint %}
By default, PerfectScale automatically publishes alerts to your [Financial Alerts dashboard](/configure-alerts/alerts-overview#financial-alerts). Additionally, you can configure alert routing directly to the preferred communication channel (Slack, MS Teams, etc.) in just two simple steps:
1. Create the profile. Learn more about available communication integrations [here](/customize-workflow/communication-and-messaging).
2. Select the relevant profile in the Notification drop-down list when configuring the Financial Alerts Rule.
Notification channel for financial alerts rule
{% hint style="warning" %}
PerfectScale does not support Slack profiles with the configured routing by label or by ID for financial alerts. In that case, alerts will be sent the default
{% endhint %}
# Pricing
Configure accurate pricing by leveraging a range of pricing customization options
PerfectScale offers various options for customizing pricing to meet your needs. Check the following instructions for pricing configurations
* [Custom Pricing configuration](/customize-workflow/pricing/custom-pricing-configuration) - a configuration set for manually setting up billing parameters.
* [AWS CUR Pricing configuration](/customize-workflow/pricing/aws-cur-configuration) - a configuration set to start using AWS CUR pricing.
* [Azure CUR Pricing configuration](/customize-workflow/pricing/azure-cost-management-configuration) - a configuration set to start using Azure CUR pricing.
* [GCP Cloud Billing configuration](/customize-workflow/pricing/gcp-cloud-billing-configuration) - a configuration set to start using GCP Cloud Billing.
* [Multi-cloud billing integration](/cloud-billing-integration/configuring-multi-cloud-billing-integration) - integrating multiple cloud billing sources with one configuration set.
* [DoiT pricing ](/cloud-billing-integration/doit-pricing)- how to start using the DoiT pricing source.
# Custom Pricing configuration
Learn how to configure custom pricing profile
PerfectScale **`Pricing profile`** feature allows you to customize how your cluster's price is calculated for more accurate data. This can be especially helpful if you have special pricing conditions from your cloud provider.
{% hint style="info" %}
PerfectScale allows you to create unlimited **Custom Pricing** profiles, but only one can be applied to each cluster.
{% endhint %}
Pricing profile
## Configuring Custom pricing with the profile
### :tools: **How to create** Custom p**ricing profile**
There are two options for creating a Profile: [from the Settings tab](#from-the-settings-tab) or directly [from the Overview](#from-the-overview-tab).
#### From the **Settings** tab
Go to the **`Settings`** tab on the left panel -> select **`Pricing`** -> click the **`+Add Profile`** button -> select **`Custom`** -> name the profile and put the needed values in the relevant fields -> click **`Save`** button.
Pricing profile from settings
{% hint style="success" %}
Tick **`Set as default`** checkbox to apply the profile as a default to all the clusters that have no specific profile attached.
{% endhint %}
#### From the **Overview** tab
Go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the custom **`Pricing Profile`** and click **`gear`** button -> go to **`Customizations`** -> click on **`Add New Profile`** in the **`Pricing Profile`** drop-down list -> name and configure your profile -> click the **`Save And Apply`** button -> click the **`Save Changes`** button.
Pricing profile from overview
#### Custom pricing profile configuration
Pricing profile configuration
`global_discount` (optional block):
* `percentage` - the percentage (0–100, floats supported) of the discount to apply on top of standard on-demand node pricing. This field is required when `global_discount` exists.
* `start_date` (optional) - the effective start date for applying the `global_discount`. If this field is empty, it defaults to today.
{% hint style="info" %}
If you change the `percentage` or `start_date`, only newly generated [Trends report](/visibility-and-optimization/trends-monitoring) will reflect the update. Historical reports will not update automatically. Contact if you need them recalculated.
{% endhint %}
`instanceType` - the name of the instance that is given by the provider.
`memGBHourPrice` - the price ($) of 1 GB of memory per hour.
`cpuCoreHourPrice` - the price ($) of 1 core per hour.
`nodeHourPrice` - the price ($) of a node per hour.
{% hint style="success" %}
To incorporate a new instance, duplicate the element and input appropriate values.
{% endhint %}
PerfectScale provides two options for specifying custom pricing in the profile:
* Specify the cost of the resources (`memGBHourPrice` and `cpuCoreHourPrice`)
* Specify the cost of the node (`nodeHourPrice`)
{% hint style="info" %}
`nodeHourPrice`**takes precedence over** `memGBHourPrice` and `cpuCoreHourPrice`. If all the values are specified in the PricingProfile, PerfectScale will take `nodeHourPrice` into account only.
{% endhint %}
### :tools: **How to apply** Custom p**ricing profile**
#### Apply to a single cluster
To apply **`Pricing Profile`** to the cluster, go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the custom **`Pricing Profile`** and click **`gear`** button -> go to **`Customizations`** -> select the needed profile in the **`Pricing Profile`** drop-down list.
Applying pricing profile to a single cluster
#### Apply to multiple clusters
To apply the profile to **multiple clusters** from a single view, use the **`Manage Assignments`** feature.
Go to the **`Settings`** tab on the left panel -> select the **`Pricing`** -> click the **`Manage Assignments`** button -> apply the profiles for the needed clusters -> click the **`Save Changes`** button.
Applying pricing profile to multiple clusters
{% hint style="info" %}
You can easily manage your profiles (create, delete), but deleting such profiles is impossible if the profile is already connected to the cluster. Change the profile to default or any other, and only after that, remove the current one.
{% endhint %}
## Configuring Custom pricing with the CR
To configure Custom pricing, you need to define and apply a Custom Resource (CR) that specifies your unique billing conditions. This approach allows you to manage pricing for specific instances directly through Kubernetes manifests.
{% hint style="warning" %}
The Custom Resource (CR) must be created in the `perfectscale` namespace.
{% endhint %}
Here is an example of the CR configuration:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
pricing:
# Create and assign a custom pricing profile inline
- type: custom
name: my-custom-pricing
assigned: true
value:
global_discount:
percentage: 5
start_date: 2025-31-12
nodeTypes:
- instanceType: c5.large
pricing:
cpuCoreHourPrice: 0.085
memGBHourPrice: 0.0095
nodeHourPrice: 0.0
- instanceType: c5.xlarge
pricing:
cpuCoreHourPrice: 0.085
memGBHourPrice: 0.0095
nodeHourPrice: 0.0
- instanceType: m5.large
pricing:
cpuCoreHourPrice: 0.096
memGBHourPrice: 0.0096
nodeHourPrice: 0.0
- instanceType: m5.xlarge
pricing:
cpuCoreHourPrice: 0.096
memGBHourPrice: 0.0096
nodeHourPrice: 0.0
```
#### ⚙️ **CR parameters:**
Parameter
Description
global_discount (optional block)
The discount to apply on top of standard on-demand node pricing. percentage - the percentage (0–100, floats supported) of the discount. This field is required when global_discount exists. start_date (optional) - the effective start date for applying the global_discount.
instanceType
The instance name assigned by the provider. Example: m5.large, m5.xlarge, etc.
cpuCoreHourPrice
The hourly cost ($) for each GB of memory.
memGBHourPrice
The hourly cost ($) for 1 CPU core.
nodeHourPrice
The hourly cost ($) of a node.
{% hint style="warning" %}
If a `global_discount` percentage is provided without a `start_date` in the CR profile, we won’t default to today’s date as we do for profiles configured in the UI, and the discount will not be applied. The pricing profile appears broken in the UI.
{% endhint %}
# AWS CUR configuration
Unlock a high level of granularity and precise resource consumption tracking with AWS CUR integration
{% hint style="warning" %}
**AWS CUR integration is only available for Advanced and Expert Plan users. To upgrade your subscription level, contact** [**support@perfectscale.io**](mailto:support@perfectscale.io) **or use the** [**PerfectScale Slack Community**](https://perfectscalecommunity.slack.com/)**.**
{% endhint %}
By default, PerfectScale pulls the "Shelf" prices from the public cloud providers' pricing API. However, for a higher level of granularity, more accurate tracking of resource consumption, and the actual cost of your K8s, PerfectScale allows pulling usage and billing information directly from your cloud provider through the integration.
To leverage the most detailed source of information about the costs and usage of cloud computing services (like AWS CUR), including billing, discounts, reserved resources, and free credits, follow this [step-by-step guide](/cloud-billing-integration/connecting-aws-cur).
# Azure Cost Management configuration
Unlock a high level of granularity and precise resource consumption tracking with Azure Cost Management integration
{% hint style="warning" %}
**Azure Cost Management integration is only available for Advanced and Expert Plan users. To upgrade your subscription level, contact** [**support@perfectscale.io**](mailto:support@perfectscale.io) **or use the** [**PerfectScale Slack Community**](https://perfectscalecommunity.slack.com/)**.**
{% endhint %}
By default, PerfectScale pulls the "Shelf" prices from the public cloud providers' pricing API. However, for a higher level of granularity, more accurate tracking of resource consumption, and the actual cost of your K8s, PerfectScale allows pulling usage and billing information directly from your cloud provider through the integration.
To leverage the most detailed source of information about the costs and usage of cloud computing services (like Azure Cost Management), including billing, discounts, reserved resources, and free credits, follow this [step-by-step guide](/cloud-billing-integration/connecting-azure-cost-management).
# GCP Cloud Billing configuration
Unlock a high level of granularity and precise resource consumption tracking with GCP Cloud Billing integration
{% hint style="warning" %}
**GCP Cloud Billing integration is only available for Advanced and Expert Plan users. To upgrade your subscription level, contact** [**support@perfectscale.io**](mailto:support@perfectscale.io) **or use the** [**PerfectScale Slack Community**](https://perfectscalecommunity.slack.com/)**.**
{% endhint %}
By default, PerfectScale pulls the "Shelf" prices from the public cloud providers' pricing API. However, for a higher level of granularity, more accurate tracking of resource consumption, and the actual cost of your K8s, PerfectScale allows pulling usage and billing information directly from your cloud provider through the integration.
To leverage the most detailed source of information about the costs and usage of cloud computing services (like GCP Cloud Billing), including billing, discounts, reserved resources, and free credits, follow this [step-by-step guide](/cloud-billing-integration/connecting-gcp-cloud-billing).
# Ticketing & Bug Tracking
Easily collaborate with your team by integrating Jira for efficient ticketing and bug tracking, ensuring effective communication and issue management
Jira integration simplifies issue management within clusters and helps you ensure every issue gets properly addressed. With just a few clicks, tickets can be opened and assigned to the proper teams, with the details needed to streamline resolutions and configuration changes.
{% hint style="info" %}
PerfectScale only supports the SaaS Jira integration.
{% endhint %}
## Configuring Jira integration with the profile
### :tools: **How to create Jira Profile**
There are two options for creating a Profile: [from the Settings](#from-the-settings-tab) tab or directly [from the Overview](#from-the-overview-tab).
#### From the **Settings** tab
Go to the **`Settings`** tab on the left panel -> select **`Integrations`** -> click the **`+Add Integration`** button and select **`Jira`** from the drop-down -> insert the required data.
Jira profile from settings
#### From the **Overview** tab
Go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Profile`** and click **`gear`** button -> go to **`Customizations`** -> find **`Integrations`** and click the **`Ticketing & Bug Tracking`** drop-down list -> click the **`Add New Profile`** button -> insert the required data.
Jira profile from overview
#### Jira Profile Configuration
Jira profile configuration
1. Name your profile
2. Configure the profile according to the following instructions:
`jira_user`: Email address of the user creating the profile. We recommend creating a separate JIRA user for PerfectScale.
`jira_secret`: API token to authenticate with an Atlassian cloud product.
{% hint style="info" %}
The API token in the profile must have the **Jira Administrator global permission**.
{% endhint %}
* Go to and click **`Create API token`** button.
New API token
* Insert the label and click **`Create`** button.
Create an API token
* Copy the created API token and put it in the `jira_secret` field.\
\
:bulb: **NOTE**: To generate the API token, you need to use the Jira account that is associated with the **`Integration Profile`**(account specified in `jira_user`).
`jira_url`: Corporate Jira URL. Example: , where **xxx** - a corporate domain.
`jira_project`: Jira project where tickets will be created.
`jira_issue_type`: The type of a Jira ticket.
`jira_sumary` : The summary of a Jira ticket.
`jira_assignee` : The default assignee (member ID) of a Jira ticket.
Member id
`jira_resolution`(optional): Set **true** to assign the resolution reason when closing a ticket in the specified **jira\_project**.
`jira_custom_fields`(optional): Specify the ID of the required Jira ticket field that is not included in the default setup. Use the following format for the custom field ID:
```
jira_custom_fields:
customfield_XXXXX: 'DevOps Team'
```
Learn how to get a field ID [here](#how-to-get-jira-custom-field-id).
3. Use **`Set as default`** checkbox to automatically assign the profile to the clusters.\
\
:bulb: **NOTE**: The manually assigned profile takes priority over the default one.
4. To verify the accuracy of your Jira configuration, simply click on the **`Test Integration`** button. When the configuration is correct, you will see the message **`Jira Configured Correctly`** . In case of **`Jira Configured Wrongly`** response, check and ensure the correctness of the data.
5. Click the **`Save`** button to save the progress.
#### How to get Jira custom field ID
A simple method to obtain a Jira custom field ID without requiring extra permissions is to inspect the relevant Jira issue that includes this field. Follow the next simple steps:
1. Open an issue (Jira ticket) with the custom field for which you want to find the ID.
2. Right-click on the field name and select `Inspect`.
3. On the Developer Tools panel, you will find the name and ID for your custom field.
Jira custom field idApplying resiliency alerts profile to a single cluster
{% hint style="info" %}
You can easily set up multiple custom fields within one profile by listing them under `jira_custom_fields`. For example:
```yaml
jira_custom_fields:
customfield_00000: 'DevOps Team'
customfield_00001: 'Resource anomaly fix'
customfield_00002: 'Under-provisioning'
```
{% endhint %}
### :tools: **How to apply Jira Profile**
#### Apply to a single cluster
To apply the **`Profile`** to the cluster, go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Profile`** and click **`gear`** button -> go to **`Customizations`** -> find **`Integrations`** and click the **`Ticketing & Bug Tracking`** drop-down list -> select the needed profile.
Applying Jira profile to a single cluster
#### Apply to multiple clusters
To apply a profile to multiple clusters from a single view, use the **`Manage Assignments`** feature. \
\
Go to the **`Settings`** tab on the left panel -> select **`Integrations`** -> click the **`Manage Assignments`** button -> apply the profile for the needed clusters in the **`Ticketing & Bug Tracking`** section -> click the **`Save Changes`** button.
Applying Jira profile to multiple clusters
## Configuring Jira integration with CR
To enable Jira ticketing, you need to define and apply a Custom Resource (CR) with the required Jira parameters. This approach allows you to manage Jira ticketing directly through Kubernetes manifests.
{% hint style="warning" %}
The Custom Resource (CR) must be created in the `perfectscale` namespace.
{% endhint %}
### Setup instructions
1. Navigate to your Jira instance (e.g., )
2. Create a bot user or use an existing user account.
{% hint style="info" %}
We recommend you create a separate JIRA user for PerfectScale.
{% endhint %}
3. Generate an API token for the user.
4. [Copy the API token](https://id.atlassian.com/manage-profile/security/api-tokens) (you will need it for CR configuration).
5. Apply the Jira API token secret.
6. Configure a CR.
{% hint style="info" %}
You may need to use custom fields for the integration. This configuration is optional, and you can learn more about it [here](#how-to-get-jira-custom-field-id).
{% endhint %}
Apply the **Jira API token secret** first:
```yaml
Jira API Token Secret (apply this first)
apiVersion: v1
kind: Secret
metadata:
name: jira-credentials
namespace: perfectscale
type: Opaque
stringData:
api-token: "your-jira-api-token-here"
```
{% hint style="info" %}
Ensure that the secret is created in the same namespace as the exporter - `perfectscale`.
{% endhint %}
Configure the CR:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
integrations:
- type: jira
name: infrastructure-tickets
assigned: true
value:
jira_url: "https://mycompany.atlassian.net"
jira_user: "perfectscale-bot@mycompany.com"
jira_secret_from:
secretKeyRef:
name: jira-credentials
key: api-token
jira_project: "INFRA"
jira_issue_type: "Bug"
jira_summary: "PerfectScale discovered issue(s) with workload"
jira_assignee: ""
jira_resolution: false
# Optional: custom fields (not included in the default setup)
# jira_custom_fields:
# customfield_10001: "DevOps Team"
# customfield_10002: "Resource anomaly fix"
```
⚙️ **CR parameters:**
API token to authenticate with an Atlassian cloud product.
jira_project
Jira project where you want tickets to be created.
jira_issue_type
The desired type of a Jira ticket.
jira_summary
The desired summary of a Jira ticket.
jira_assignee
The default assignee (member ID) of a Jira ticket.
jira_resolution
(optional) Set true to assign the resolution reason when closing a ticket in the specified jira_project.
jira_custom_fields
(optional) Specify additional Jira ticket field (field ID) that is not included in the default setup.
# Communication & Messaging
Learn how to stay on top of alerts and receive them through your preferred communication channel for timely and effective responses
For faster updates, utilize communication channels by configuring Integration Profiles. Efficiently manage the [alerts](/configure-alerts/alerts-overview), receive timely notifications via preferred channels such as [Slack](/customize-workflow/communication-and-messaging/slack-integration), and [MS Teams](/customize-workflow/communication-and-messaging/ms-teams-integration), and/or receive them directly to your [Datadog](/customize-workflow/communication-and-messaging/datadog-alerts-integration).
# Slack Integration
Streamline alerts management with Slack integration
Integrating Slack streamlines the management of **`Alerts`** in clusters, allowing users to receive timely notifications in the defined Slack channel when alerts are detected.
## Configuring Slack integration with the profile
### :tools: **How to create Slack Profile**
There are two options for creating a Profile: [from the Settings](#from-the-settings-tab) tab or directly [from the Overview](#from-the-overview-tab).
#### From the **Settings** tab
Go to the **`Settings`** tab on the left panel -> select **`Integrations`** -> click the **`+Add Integration`** button and select **`Slack`** from the drop-down -> insert the required data.
Slack profile from settings
#### From the **Overview** tab
Go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Profile`** and click **`gear`** button -> go to **`Customizations`** -> find **`Integrations`** and click the **`Communication & Messaging`** drop-down list -> click the **`Add New Profile`** button and select **`Slack`** -> configure the profile.
Slack profile from overview
#### Slack profile configuration
1. Name the profile.
2. Configure the profile:
* It is important to include relevant data in the **`slack_token`** field as it is a **mandatory** one. \
Learn how to configure [slack\_token here](/customize-workflow/communication-and-messaging/slack-integration/how-to-configure-slack_token).
* Indicate the name of the Slack channel where you want to receive the alerts. This field is **mandatory,** and the indicated channel will be the default channel if Advanced Alert Routing fails. To specify the Slack channel, enter its name in the **`channel`** field.
* Set up [Advanced Alert Routings](#advanced-alerts-routing) to send alerts to different Slack channels for various workloads within a cluster.
{% hint style="warning" %}
The channel\_id is deprecated. Please use the channel name instead.
{% endhint %}
{% hint style="success" %}
You can create as many as needed Slack Integration Profiles with only one [slack\_token](/customize-workflow/communication-and-messaging/slack-integration/how-to-configure-slack_token) by changing the channel name.
{% endhint %}
3. Use **`Set as default`** checkbox to automatically assign the profile to the clusters.\
\
:bulb: **NOTE**: The manually assigned profile takes priority over the default one.
4. Verify if your Slack is configured properly with the [**`Test Integration`**](#slack-integration-verification) button.
5. Click **`Save`** button
### :tools: **How to apply** **Slack Profile**
#### Apply to a single cluster
To apply **`Slack Profile`** to the cluster, go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Slack Profile`** and click **`gear`** button -> go to **`Customizations`** -> select the needed profile in the **`Communication & Messaging`** drop-down list.
Applying Slack profile to a single cluster
#### Apply to multiple clusters
To apply the profile to **multiple clusters** from a single view, use the **`Manage Assignments`** feature.
Go to the **`Settings`** tab on the left panel -> select the **`Integrations`** -> click the **`Manage Assignments`** button -> go to **`Communication & Messaging`** section and apply the profiles for the needed clusters -> click the **`Save Changes`** button.
Applying Slack profile to multiple clusters
## Advanced Alerts Routing
If you need to configure routing for Slack alerts in a more advanced way, PerfectScale provides an option for setting it up with labels.
This approach enables sending alerts to different Slack channels for various workloads within a cluster.
Learn [how to configure routings here](/customize-workflow/communication-and-messaging/slack-integration/how-to-configure-routings).
{% hint style="info" %}
If the value of a label with key `label_key` is wrong or empty, PerfectScale will send Alerts to the default Slack channel specified in the `channel` row.
{% endhint %}
## Configuring Slack integration with CR
To enable Slack alerts, you need to define and apply a Custom Resource (CR) with the required parameters. This approach allows you to manage Slack alerting directly through Kubernetes manifests.
{% hint style="warning" %}
The Custom Resource (CR) must be created in the `perfectscale` namespace.
{% endhint %}
### Setup instructions
1. Create the app at .
2. Go to the **OAuth & Permissions** tab and specify the following scope:
* chat:write.public
* chat:write
* channels:read
3. Click the **Install to Workspace** button.
4. Copy the **Bot User OAuth Token** (you will need it for the **Slack token secret** in the next steps).
Apply the **Slack token secret** first:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: slack-credentials
namespace: perfectscale
type: Opaque
stringData:
bot-token: "xoxb-your-slack-bot-token-here"
```
{% hint style="info" %}
Ensure that the secret is created in the same namespace as the exporter - `perfectscale`.
{% endhint %}
Configure the CR:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
integrations:
- type: slack
name: team-slack-alerts
assigned: true
value:
channel: "perfectscale-alerts"
slack_token_from:
secretKeyRef:
name: slack-credentials
key: bot-token
#Optional: Advanced alert routing
#routings:
#- label_alert
#- resiliency_alert
```
⚙️ **CR parameters:**
Parameter
Description
channel
Slack channel name that indicates where to receive PerfectScale alerts.
slack_token_from
A secret that permits PerfectScale to interact with your Slack. Learn how to configure slack_token here.
routings
Advanced alerts routing that enables sending alerts to different Slack channels for various workloads within a cluster based on labels. Learn more here.
## Slack Integration Verification
To verify the accuracy of your Slack configuration, simply click on the **`Test Integration`** button. When the configuration is correct, you will see the message **`Slack Configured Correctly`** . In case of **`Slack Configured Wrongly`** response, check and ensure the correctness of the data.
{% hint style="warning" %}
**`channel`** and **`slack_token`** are mandatory. Verification will fail if **`channel`** and/or **`slack_token`** are missing or incorrect.
{% endhint %}
Slack integration verification
# How to configure slack\_token
Step-by-step guide on how to configure slack\_token
Setting up slack\_token is a simple process that can be completed in a few steps.
1. Go to and click **`Create an App`** button.\
Setting up Slack token
{% hint style="info" %}
By default, only workspace owners and administrators can deploy apps. However, specific users and user groups can be authorized in your workspace settings. Administrators can also grant this permission to any non-guest team member.
{% endhint %}
2. In case `From scratch` option is selected, name the App, select the relevant namespace, and click **`Create App`** button.
Create new app
3. Navigate to the `OAuth & Permissions` tab.
4. Go to Scopes and click on the `Add an OAuth Scope` button.
5. Select the following options from the drop-down list:
* `chat:write.public`
* `chat:write`
* `channels:read`
6. Click on the `Add Scopes` button.
7. Scroll up and click the `Install to Workspace`. Once this step is done, slack\_token will appear.
OAuth Scope
# How to configure routings
Learn how to configure Slack routings
Routings for Slack notifications enable sending alerts to different Slack channels for various workloads within a cluster based on labels.
```yaml
slack_token: 'slack-token' # replace with your slack_token
channel: 'slack-test' # replace with your channel name
routings:
- label_key_3
- label_key_4
```
To maximize your effectiveness, Slack routing is available not only on the workload level but also on the namespace level. This means you can configure routing for specific workloads as well as for the entire namespace.
{% hint style="info" %}
Workload labels take precedence over namespace labels. If you set a routing label for a workload that differs from the label for its namespace, PerfectScale will send Alerts to the channel specified for the workload.
{% endhint %}
Example of a label:
`label_some_key : label_some_value`, where `label_some_value` = Slack channel name
## Slack Routing Configuration
1. Name the profile.
2. Ensure to configure `slack_token` and include the relevant data in the [**`slack_token`**](/customize-workflow/communication-and-messaging/slack-integration/how-to-configure-slack_token) field. This field is **mandatory**.
3. Indicate the `channel`. This field is **mandatory,** and the indicated channel will be the default channel if Advanced Alert Routing fails.
4. List `label_key`(s).
5. Verify if your Slack is configured properly with the `Test Integration` button.
6. Click **`Save`** button.
{% hint style="warning" %}
The channel\_id is deprecated.
{% endhint %}
# MS Teams Integration
Streamline alerts management with MS Teams integration
By integrating Teams, the process of managing Alerts in clusters becomes more efficient. This feature enables users to promptly receive notifications on the designated Teams channel whenever alerts are produced.
MS Teams profile
## Configuring Teams integration with the profile
### :tools: **How to create Teams Profile**
There are two options for creating a Profile: [from the Settings tab](#from-the-settings-tab) or directly from the Overview.
#### From the **Settings** tab
Go to the **`Settings`** tab on the left panel -> select **`Integrations`** -> click the **`+Add Integration`** button and select **`Teams`** from the drop-down -> insert [teams\_webhook](/customize-workflow/communication-and-messaging/ms-teams-integration/how-to-configure-teams_webhook) -> click **`Save`** button.
MS Teams profile from settings
#### From the **Overview** tab
Go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Profile`** and click **`gear`** button -> go to **`Customizations`** -> find **`Integrations`** and click the **`Communication & Messaging`** drop-down list -> click the **`Add New Profile`** button and select **`Teams`** -> insert [teams\_webhook](/customize-workflow/communication-and-messaging/ms-teams-integration/how-to-configure-teams_webhook) -> click **`Save And Apply`** button.
MS Teams profile from overview
### :tools: **How to apply** Teams **Profile**
#### Apply to a single cluster
To apply **`Teams Profile`** to the cluster, go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Teams Profile`** and click **`gear`** button -> go to **`Customizations`** -> select the needed profile in the **`Communication & Messaging`** drop-down list.
Applying MS Teams profile to a single cluster
#### Apply to multiple clusters
To apply the profile to **multiple clusters** from a single view, use the **`Manage Assignments`** feature.
Go to the **`Settings`** tab on the left panel -> select the **`Integrations`** -> click the **`Manage Assignments`** button -> go to **`Communication & Messaging`** section and apply the profiles for the needed clusters -> click the **`Save Changes`** button.
Applying MS Teams profile to multiple clusters
## Configuring Teams integration with CR
To enable Teams alerts, you need to define and apply a Custom Resource (CR) with the required parameters. This approach allows you to manage Teams alerting directly through Kubernetes manifests.
{% hint style="warning" %}
The Custom Resource (CR) must be created in the `perfectscale` namespace.
{% endhint %}
### Setup instructions
{% stepper %}
{% step %}
**Configure teams\_webhook**
If you haven't completed this configuration before, follow the detailed instructions [here](/customize-workflow/communication-and-messaging/ms-teams-integration/how-to-configure-teams_webhook).
{% endstep %}
{% step %}
**Apply teams\_webhook**
```yaml
apiVersion: v1
kind: Secret
metadata:
name: teams-credentials
namespace: perfectscale
type: Opaque
stringData:
webhook-url: "https://defaultabc.ab.environment.api.powerplatform.com/abc"
```
{% hint style="info" %}
Ensure that the secret is created in the same namespace as the exporter - `perfectscale`.
{% endhint %}
{% endstep %}
{% step %}
**Configure the CR**
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
integrations:
- type: teams
name: ops-team-notifications
assigned: true
value:
teams_webhook_from:
secretKeyRef:
name: teams-credentials
key: webhook-url
```
⚙️ **CR parameters:**
Parameter
Description
teams_webhook_from
A secret that permits PerfectScale to interact with your Teams. Learn how to configure teams_webhook here.
{% endstep %}
{% endstepper %}
## Teams Integration Verification
To verify the accuracy of your MS Teams configuration, simply click on the **`Test Integration`** button. When the configuration is correct, you will see the message **`Teams Configured Correctly`**. In case of **`Teams Configured Wrongly`** response, check and ensure that the [`teams_webhook`](/customize-workflow/communication-and-messaging/ms-teams-integration/how-to-configure-teams_webhook) URL is correct.
MS Teams integrations verification
# How to configure teams\_webhook
Step-by-step guide on how to configure teams\_webhook
{% hint style="info" %}
**teams\_webhook** is a mandatory field to configure MS Teams integration.
{% endhint %}
{% hint style="warning" %}
starting **May 18, 2026**, old webhooks may stop working. To avoid disruptions, all users need to migrate to the new workflow. [Learn more](https://devblogs.microsoft.com/microsoft365dev/retirement-of-office-365-connectors-within-microsoft-teams/).
{% endhint %}
Setting up teams\_webhook is a simple process that can be completed in a few steps.
1. Go to .
2. In the target Teams channel, open Workflows and create a new flow using the `When a Teams webhook request is received` trigger. Set `Anyone` in the `Who can trigger the flow`.
3. Parse JSON wit hthe following parameters:
* Content - “Teams” Body
* Schema:
```json
{
"type": "object",
"properties": {
"type": {
"type": "string"
},
"attachments": {
"type": "array",
"items": {
"type": "object",
"properties": {
"contentType": {
"type": "string"
},
"contentUrl": {
"type": [
"string",
"null"
]
},
"content": {
"type": "object"
}
}
}
}
}
}
```
4. Select `Apply to each`, then set `attachments` as the **Select an output from previous steps** value.
5. Set `content` value for the **Compose -> Inputs** (inside `Apply to each`).
Configuring MS Teams webhook
6. Click **Post card in chat or channel** and set the **Adaptive Card** field to `Outputs` (inside `Apply to each`).
Post card config
# Datadog Alerts Integration
Streamline alerts management with Datadog alerts integration
Datadog Alerts Integration enables you to receive PerfectScale Alerts directly into your Datadog dashboard, ensuring you stay informed about any resilience risks identified in your Kubernetes environment, and helping you prioritize tasks to eliminate the issues before they impact performance and user experience.
Datadog Alerts Integration could be enabled in a few simple steps:
1. [Create an API Key](#how-to-create-api-kye-in-datadog) in Datadog.
2. Configure [Datadog Alerts Profile](#how-to-create-datadog-alerts-profile).
3. [Apply the profile](#how-to-apply-datadog-alerts-profile) to the desired cluster(s).
## :tools: **How to create API Kye in Datadog**
1. Log in to your Datadog profile.
2. Hover over your account icon and click API Keys
Datadog API key
3. Click **`New Key`** button, name your key, and click **`Create Key`**.
New key1
4. Copy the **API Key.** You will need it for the [Datadog Alerts Profile configuration](#datadog-alerts-profile-configuration).
New key2
## Configuring Datadog integration with the profile
### :tools: **How to create Datadog Alerts Profile**
There are two options for creating a Profile: [from the Settings](#from-the-settings-tab) tab or directly [from the Overview](#from-the-overview-tab).
#### From the **Settings** tab
Go to the **`Settings`** tab on the left panel -> select **`Integrations`** -> click the **`+Add Integration`** button and select **`Datadog`** from the drop-down -> insert the required data.
Datadog alerts profile from settings
#### From the **Overview** tab
Go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Profile`** and click **`gear`** button -> go to **`Customizations`** -> find **`Integrations`** and click the **`Monitoring`** drop-down list -> click the **`Add New Profile`** button -> insert the required data.
Datadog alerts profile from overview
#### Datadog alerts Profile Configuration
Datadog alerts profile configuration
1. Name the Profile
2. `datadog_api_key` - Datadog API Key. See the details [here](#how-to-create-api-kye-in-datadog).\
\
`datadog_site_region` - specifies the Datadog regional domain where your data is sent and managed (example: datadoghq.com).
The available datadog\_site\_region:
| Code | Domain |
| ------- | ----------------- |
| US1 | datadoghq.com |
| US3 | us3.datadoghq.com |
| US5 | datadoghq.com |
| EU | datadoghq.eu |
| AP1 | ap1.datadoghq.com |
| US1-FED | ddog-gov.com |
3. Use **`Set as default`** checkbox to automatically assign the profile to the clusters.\
\
:bulb: **NOTE**: The manually assigned profile takes priority over the default one.
### :tools: **How to apply** **Datadog Alerts Profile**
#### Apply to a single cluster
To apply **`Datadog Alerts Profile`** to the cluster, go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Datadog Alerts Profile`** and click **`gear`** button -> go to **`Customizations`** -> select the needed profile in the **`Monitoring`** drop-down list.
Applying Datadog alerts profile to a single cluster
#### Apply to multiple clusters
To apply the profile to **multiple clusters** from a single view, use the **`Manage Assignments`** feature.
Go to the **`Settings`** tab on the left panel -> select the **`Integrations`** -> click the **`Manage Assignments`** button -> go to **`Monitoring`** section and apply the profiles for the needed clusters -> click the **`Save Changes`** button.
Applying Datadog alerts profile to multiple clusters
### **How to uninstall Datadog Alerts Integration**
In order to stop submitting Alerts to Datadog, the profile should be unassigned from the cluster.
Go to the **`Settings`** tab on the left panel -> select the **`Integrations`** -> click the **`Manage Assignments`** button -> go to **`Monitoring`** section and select **`None`** for the needed clusters -> click the **`Save Changes`** button.
Uninstall Datadog alerts integration
## Configuring Datadog integration with CR
To enable Datadog alerts, you need to define and apply a Custom Resource (CR) with the required parameters. This approach allows you to manage Datadog alerting directly through Kubernetes manifests.
{% hint style="warning" %}
The Custom Resource (CR) must be created in the `perfectscale` namespace.
{% endhint %}
### Setup instructions
1. Log in to your Datadog account.
2. Navigate to Organization Settings > API Keys.
3. Create or copy an API key.
4. Copy the API key (you will need it for the **Datadog API Key secret** in the next steps).
Apply the **Datadog API Key secret** first:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: datadog-credentials
namespace: perfectscale
type: Opaque
stringData:
api-key: "your-datadog-api-key-here"
```
{% hint style="info" %}
Ensure that the secret is created in the same namespace as the exporter - `perfectscale`.
{% endhint %}
Configure the CR:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
integrations:
- type: datadog
name: production-monitoring
assigned: true
value:
datadog_api_key_from:
secretKeyRef:
name: datadog-credentials
key: api-key
datadog_site_region: "datadoghq.com"
```
⚙️ **CR parameters:**
Parameter
Description
datadog_api_key_from
A secret that contains a Datadog API Key. Learn more how to create it here.
datadog_site_region
Specifies the Datadog regional domain where your data is sent and managed.
{% hint style="info" %}
Available datadog\_site\_region values:
* **US1**: datadoghq.com
* **US3**: us3.datadoghq.com
* **US5**: us5.datadoghq.com
* **EU**: datadoghq.eu
* **AP1**: ap1.datadoghq.com
* **US1-FED**: ddog-gov.com
{% endhint %}
# Label customizations
Explore advanced label configuration
PerfectScale can recognize some labels automatically. However, it can be set up manually for advanced label configuration. This is particularly helpful when you are using custom labels to organize your K8s resources.
You can customize your labels for the workloads as well as for nodes. Currently, there are available the following options:
Label
Description
workload_labels
Defines labels that can be applied to Kubernetes resources (pods, deployments, or stateful sets) to identify and manage workloads within a cluster.
spot_labels
Identifies spot nodes.
node_group_labels
Identifies the nodes group in a cluster.
architecture_labels
Specifies the architecture of the node (x86_64, arm64, etc.)
os_labels
Specifies the operating system running on a node within the Kubernetes cluster.
instance_type_labels
Specifies the type or configuration of VMs (instances) within a cloud infrastructure.
region_labels
Specifies the geographical region where a particular resource is located.
availability_zone_labels
Specify the availability zone where a particular resource is located.
ignored_pod_labels
Defines labels that Kubernetes components should disregard when making pod decisions.
ignored_node_labels
Defines labels that Kubernetes components should disregard when making node decisions.
## Label Formatting
Kube-state-metrics exposes labels following Prometheus conventions:
1. Add the `label_` prefix to the label
2. Replace non-alphabetic characters with underscores
3. Convert camel case to snake case by using underscores to separate words
{% hint style="info" %}
**Example**
kubernetes.azure.com/scalesetpriority: spot -> label\_kubernetes\_azure\_com\_scalesetpriority: spot
```
node:
spot_labels:
label_kubernetes_azure_com_scalesetpriority: spot
```
{% endhint %}
## Configuring Custom Labels with the profile
### :tools: **How to create** **Customization Profile**
There are two options for creating a Profile: [from the Settings tab](#from-the-settings-tab) or directly [from the Overview](#from-the-overview-tab).
#### From the **Settings** tab
Go to the **`Settings`** tab on the left panel -> select **`Customizations`** -> click the **`+Add Profile`** button -> name and configure the profile -> click **`Save`** button.
Customization profile from settings
#### From the **Overview** tab
Go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Customization Profile`** and click **`gear`** button -> go to **`Customizations`** -> click on **`Add New Profile`** in the **`Customization Profile`** drop-down list -> name and configure your profile -> click the **`Save And Apply`** button -> click the **`Save Changes`** button.
Customization profile from overview
{% hint style="info" %}
Use **`Set as default`** checkbox to automatically assign the profile to the clusters.\
\
:bulb: **NOTE**: The manually assigned profile takes priority over the default one.
{% endhint %}
### :tools: **How to apply** Customization **Profile**
#### Apply to a single cluster
To apply **`Customization Profile`** to the cluster, go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Customization Profile`** and click **`gear`** button -> go to **`Customizations`** -> select the needed profile in the **`Customization Profile`** drop-down list.
Applying customization profile to a single cluster
#### Apply to multiple clusters
To apply the profile to **multiple clusters** from a single view, use the **`Manage Assignments`** feature.
Go to the **`Settings`** tab on the left panel -> select the **`Customizations`** -> click the **`Manage Assignments`** button -> apply the profiles for the needed clusters -> click the **`Save Changes`** button.
Applying customization profile to multiple clusters
## Configuring Custom Labels with CR
To enable custom labels and node identification, you need to define and apply a Custom Resource (CR) with the required parameters. This approach allows you to manage Custom Labels directly through Kubernetes manifests.
{% hint style="warning" %}
The Custom Resource (CR) must be created in the `perfectscale` namespace.
{% endhint %}
### Setup instructions
1. Identify the labels used in your Kubernetes cluster to distinguish workloads.
2. Identify the labels used for node properties such as spot instances, node groups, architecture, etc.
3. Configure the fields below to match your cluster’s label structure.
4. Configure the CR.
{% hint style="info" %}
Ensure labels follow Prometheus conventions. Learn more about the format [here](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
{% endhint %}
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
customization:
- name: custom-workload-labels
assigned: true
value:
workload_labels:
- io.codefresh.pipelineName
- app.kubernetes.io/name
- app.kubernetes.io/component
- app.kubernetes.io/version
- app.kubernetes.io/part-of
node:
spot_labels:
label_cloud_google_com_gke_nodepool: pool-2
label_spotinst_io_node_lifecycle: spot
node_group_labels:
- eks.amazonaws.com/nodegroup
- kops.k8s.io/instancegroup
architecture_labels:
- kubernetes.io/arch
- beta.kubernetes.io/arch
os_labels:
- kubernetes.io/os
- beta.kubernetes.io/os
instance_type_labels:
- node.kubernetes.io/instance-type
- beta.kubernetes.io/instance-type
region_labels:
- topology.kubernetes.io/region
- failure-domain.beta.kubernetes.io/region
availability_zone_labels:
- topology.kubernetes.io/zone
- failure-domain.beta.kubernetes.io/zone
ignored_pod_labels:
- uid
- namespace
- pod
ignored_node_labels:
- uid
```
⚙️ **CR parameters:**
Parameter
Description
workload_labels
Defines labels that can be applied to Kubernetes resources (pods, deployments, or stateful sets) to identify and manage workloads within a cluster.
spot_labels
Identifies spot nodes (key-value pairs).
node_group_labels
Identifies the node groups in a cluster.
architecture_labels
Specifies the architecture of the node (x86_64, arm64, etc.).
os_labels
Specifies the operating system running on a node within the Kubernetes cluster.
instance_type_labels
Specifies the type or configuration of VMs (instances) within a cloud infrastructure
region_labels
Specifies the geographical region where a particular resource is located.
availability_zone_labels
Specifies the availability zone where a particular resource is located.
ignored_pod_labels
Defines labels that Kubernetes components should disregard when making pod decisions.
ignored_node_labels
Defines labels that Kubernetes components should disregard when making node decisions.
# Ephemeral workloads grouping
Leverage advanced PerfectScale grouping to streamline your optimization process even for highly dynamic environments
The PerfectScale grouping configuration is a powerful tool that enables the aggregation of redundant workloads exhibiting similar patterns. By preventing overload through batches of identical information, it enables you to focus on what matters, thereby streamlining analysis processes.
Custom grouping is a rule that aggregates multiple workloads into a single entity (for example, all GitLab runners are aggregated into a single entity—the GitLab workload), merging their data (resource usage, limits, requests, etc.).
This feature is especially convenient when using Spark, GitLab, Airflow or any other operators that produce short-lived, small workloads that often reflect just a single pod in the cluster, or when automating the ungrouped workloads.
## Grouping by labels
{% hint style="success" %}
This grouping approach is specifically helpful for **automating dynamic, short-lived workloads**. Learn more about how to automate ephemeral workloads [here](/enable-automation/configuring-automation-with-crd/including-a-cluster-namespace-or-workload-to-the-automation/configuring-automation-for-a-workload-with-crd#automating-ephemeral-workloads).
{% endhint %}
To group the workloads, two predefined labels should be added to this workload:
| Key | Value | Description |
| ----------------------------------------------- | -------------------- | ---------------------------------- |
| perfectscale.io/workload-grouping-workload-name | custom-workload-name | Specifies the target workload name |
| perfectscale.io/workload-grouping-workload-type | custom-workload-type | Specifies the target workload type |
{% hint style="danger" %}
For the `perfectscale.io/workload-grouping-workload-type` label, the value must be a **custom workload type**. Do not use an existing Kubernetes workload type such as `Deployment`, `StatefulSet`, `DaemonSet`, `CronJob`, and `Job` this conflicts with PerfectScale’s workload grouping automation logic.
{% endhint %}
{% hint style="info" %}
Labels `perfectscale.io/workload-grouping-workload-name` and `perfectscale.io/workload-grouping-workload-type` are required to configure automation for ephemeral workloads. After applying the labels, a new workload will appear in PerfectScale. However, automation will only reduce resources after sufficient data has been collected.
{% endhint %}
To ensure PerfectScale considers all revisions, including those not made by Automation, and to drive better results, you can optionally specify the following labels:
Key
Value
Description
perfectscale.io/workload-grouping-honor-spec
true
false (default)
Allows PerfectScale to consider the resource changes in the original spec and changes to current resources.
When to use? Set to true if:
You are manually changing spec resources (set by the customer in the parent object) and want PerfectScale to respect those changes. ❗Note: To ensure predictable automation behavior, use this label with the value true only when all workloads in the group have equal resources.
You want every manual resource change revision to appear in the Revisions Timeline.
perfectscale.io/workload-grouping-honor-image
true
false (default)
Allows PerfectScale to consider the image name in the calculated hash.
When to use? Set to true if:
If you want to see the deployment history in the timeline when the application version (image) changes. ❗Note: Each update will create a new revision in the Revisions Timeline, which may result in a large number of entries.
{% hint style="warning" %}
For label-grouped workloads, PerfectScale ignores Automation CR settings that limit recommendations based on the current resource spec. In this case, these fields under `automation.containers..operational.restrictions` are not applied:
```bash
cpuManagement.request.increaseEnabled: false
cpuManagement.request.decreaseEnabled: false
memoryManagement.request.increaseEnabled: false
memoryManagement.request.decreaseEnabled: false
memoryManagement.limit.increaseEnabled: false
memoryManagement.limit.decreaseEnabled: false
```
These fields are not applied because label-grouped workloads often run multiple revisions with different specs. Having a single spec for such workloads would may lead to inaccurate recommendations. Instead, PerfectScale derives recommendations from **aggregated usage data** across all revisions.
**What still works:**
Absolute bounds are always respected, regardless of grouping:
* `cpuManagement.request.minimumCores` / `maximumCores`
* `memoryManagement.request.minimumGiB` / `maximumGiB`
* `memoryManagement.limit.minimumGiB` / `maximumGiB`
**How to restore spec-relative behavior:**
If you need spec-relative restrictions for a specific grouped workload, add this label:
```bash
perfectscale.io/workload-grouping-honor-spec: true
```
{% endhint %}
{% hint style="info" %}
If you enable automation for a custom workload type, the WorkloadLabelsSelector in a cluster or namespace configuration will not be applied. All workloads of that custom type will be automated despite the label's configuration.
{% endhint %}
### Cross-namespace grouping
{% hint style="info" %}
This feature is supported in autoscaler version **1.0.43** and later.
{% endhint %}
In environments with dynamic namespace structure, it can be difficult to apply consistent automation policies across related workloads. PerfectScale addresses this by enabling centralized automation for workloads across multiple namespaces using labels.
To group and automate such workloads, you need to proceed with a few simple steps:
1. Create a target namespace (the grouping destination).
2. Apply grouping labels to relevant workloads, including `perfectscale.io/workload-grouping-workload-namespace` with the target namespace value.
3. In the target namespace, define a [namespace](/enable-automation/configuring-automation-with-crd/including-a-cluster-namespace-or-workload-to-the-automation/configuring-automation-for-a-namespace-with-crd) or [workload-level](/enable-automation/configuring-automation-with-crd/including-a-cluster-namespace-or-workload-to-the-automation/configuring-automation-for-a-workload-with-crd) automation configuration.
4. Once done, PerfectScale groups workloads from different namespaces under the target namespace and applies the configured automation.
{% hint style="info" %}
This label is optional. If the label is not set, workloads remain grouped within their original namespace.
{% endhint %}
Specifies the target namespace for cross-namespace grouping.
When this label is set, workloads from multiple namespaces are grouped under the specified target namespace. Automation configured in the target namespace is applied to all grouped workloads, regardless of their original namespace.
{% hint style="info" %}
When `perfectscale.io/workload-grouping-workload-namespace` is specified, automation configs from original workload namespaces will not be considered. In order to override the configuration, you need a namespace or workload-level config in the targeted namespace.
{% endhint %}
#### How this works in practice
Pod Example:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-name
namespace: {original-namespace}
labels:
perfectscale.io/workload-grouping-workload-name: {target-name}
perfectscale.io/workload-grouping-workload-type: {target-type}
perfectscale.io/workload-grouping-workload-namespace: {target-namespace}
```
Original namespace configuration:
```yaml
apiVersion: perfectscale.io/v1
kind: NamespaceAutomationConfig
metadata:
name: original-namespace-config
namespace: {original-namespace}
spec:
automation:
operational:
restrictions:
...
```
Target namespace configuration:
```yaml
apiVersion: perfectscale.io/v1
kind: NamespaceAutomationConfig
metadata:
name: target-namespace-config
namespace: {target-namespace}
spec:
automation:
operational:
restrictions:
...
```
{% hint style="info" %}
When target namespace label\
`perfectscale.io/workload-grouping-workload-namespace: {target-namespace}` is specified, automation configs from the `{original-namespace}` namespace(s) will not be considered. Only the configs from `{target-namespace}` will be applied to such workloads.
{% endhint %}
# Observability
Improve resource utilization clarity with Observability profiles
With observability profiles, users can easily access the observability tool to view comprehensive data and usage patterns over a period of time.
PerfectScale **Observability** integration supports the following tools:
* Grafana
* DataDog
## Configuring Observability with the profile
### :tools: How to create Observability Profile
There are two options for creating a Profile: [from the Settings tab](#from-the-settings-tab) or directly [from the Overview](#from-the-overview-tab).
#### From the **Settings** tab
Go to the **`Settings`** tab on the left panel -> select **`Integrations`** -> click the **`+Add Integration`** button and select **`Observability`** from the drop-down -> insert the required data.
Observability profile from settings
#### From the **Overview** tab
Go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Profile`** and click **`gear`** button -> go to **`Customizations`** -> find **`Integrations`** and click the **`Observability`** drop-down list -> click the **`Add New Profile`** button -> insert the required data.
Observability profile from overview
#### Observability Profile Configuration
Create **Grafana** or **DataDog URL** in a few steps:
* Go to PerfectScale [GitHub](https://github.com/perfectscale-io/observability/tree/main/dashboards).
* Download either the Grafana or DataDog dashboard.
* Import the dashboard into your Grafana or DataDog instance.
* Copy the link to the dashboard (**Grafana URL, DataDog URL**).
Observability provile configuration
{% hint style="info" %}
Use **`Set as default`** checkbox to automatically assign the profile to the clusters.\
\
:bulb: The manually assigned profile takes priority over the default one.
{% endhint %}
### :tools: **How to apply Observability Profile**
#### Apply to a single cluster
To apply the **`Profile`** to the cluster, go to the **`Overview`** tab on the left panel -> find the cluster to which you want to apply the **`Profile`** and click **`gear`** button -> go to **`Customizations`** -> find **`Integrations`** and click the **`Observability`** drop-down list -> select the needed profile.
Applying observability profile to a single cluster
#### Apply to multiple clusters
To apply a profile to multiple clusters from a single view, use the **`Manage Assignments`** feature. \
\
Go to the **`Settings`** tab on the left panel -> select **`Integrations`** -> click the **`Manage Assignments`** button -> apply the profile for the needed clusters in the **`Observability`** section -> click the **`Save Changes`** button.
Applying observability profile to multiple clusters
## Configuring Observability with CR
To configure Observability integration, you need to define and apply a Custom Resource (CR) with the required parameters. This approach allows you to manage Observability integration directly through Kubernetes manifests.
{% hint style="warning" %}
The Custom Resource (CR) must be created in the `perfectscale` namespace.
{% endhint %}
### Setup instructions
1. Go to the repository:
2. Download the dashboard (Grafana or Datadog).
3. Import the downloaded dashboard into your Grafana or Datadog instance.
4. Configure the CR.
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings-main
namespace: perfectscale
spec:
profiles:
integrations:
- type: observability
name: custom-observability-platform
assigned: true
value:
# URL to your observability dashboard (Grafana or DataDog)
url: "https://observability.mycompany.com/dashboard"
```
⚙️ **CR parameters:**
Parameter
Description
url
The link to your observability dashboard (Grafana or Datadog).
## How to use Observability
To access the detailed data of a specific cluster using an observability tool (for example, Grafana), navigate to the **`Overview`** tab on the left panel and click on the needed cluster.
You will navigate directly to the list of cluster workloads. Select any workload to bring up the workload zoom-in window.
View in observability
Click on the `Actions` menu and select `View in Observability` to be directly taken to the relevant dashboard of your observability tool.
# Managing workloads with labels
Customize Podfit Label view to seamlessly access the data for the specific workloads
Using your existing labels, you can effectively manage workloads within PerfectScale, seamlessly focusing on the most important areas.
PerfectScale allows you to customize the Labels View in the Podfit and configure the labels to tailor to your specific needs. To personalize your Label View, you have the option to select two labels.
Podfit labels view
{% hint style="warning" %}
If the cluster has a **Podfit Labels profile** applied, manually changing the label view in Podfit remains an option, but changes will not be saved. It will always revert to the labels from the manually set **Podfit Labels profile** or the [**Default Labels profile**](#default-label-profile).
{% endhint %}
{% hint style="info" %}
Each cluster can have only one **Podfit Labels profile**.
{% endhint %}
## Configuring Labels with the profile
### :tools: ***How to***** Create** **Podfit Label profile**
Go to the **`Settings`** tab on the left panel -> select **`Podfit Labels`** -> click the **`+Add Profile`** button -> name the profile and put the needed labels in the relevant fields -> click **`Save`** button.
Creating label profile
### :tools: **How to Apply** **Podfit Label profile**
With the **`Manage Assignments`** feature, you can easily connect the **`Podfit Label Profile`** to the cluster(s) from a single view.\
\
Go to the **`Settings`** tab on the left panel -> select **`Podfit Labels`** -> click the **`Manage Assignments`** button -> select the profile for the needed cluster -> click the **`Save Changes`** button.
Applying label profile
### Default Label profile
The Default Label Profile feature helps reduce repetitive actions and customize the Podfit Label view per your priorities in a few seconds.
To easily apply the label set specified in the **`Podfit Label profile`** to all clusters in one click, tick the profile as a Default while creating or viewing it.
{% hint style="warning" %}
The default profile takes priority over any changes made manually. Manually changing the label view in Podfit remains an option, but changes will not be saved.
{% endhint %}
Default label profile
# Optimization Policy customization
Learn how to customize the optimization policy to tailor recommendations and automation to your services' specific needs
PerfectScale provides flexible customization options for configuring optimization policies to achieve greater control over optimization. With this feature, you can seamlessly align your optimization policy with your optimization goals to achieve the desired results efficiently.
{% hint style="warning" %}
To prevent the risk of CPU throttling, the resource `Request = Limit` setup is not available for the CPU resource.
{% endhint %}
You can seamlessly apply the optimization policy [via UI](#setting-optimization-policy-via-ui) or directly through Kubernetes manifests [with CR](#setting-optimization-policy-with-crd).
{% hint style="info" %}
While the UI allows you to configure an optimization policy at the **cluster** and **workload** levels, using a CR also enables you to set an optimization policy for a **specific namespace**.
{% endhint %}
When an Optimization Policy is managed with a **Custom Resource (CR)**, the UI reflects the CR as the source of truth, and the relevant `CRD` badge will appear:
Optimization policy configured with a CR
**Cluster-level CR**
* If a cluster-level optimization policy CR is applied, the UI will not allow changes to the cluster optimization policy.
**Workload-level CR**
* If a workload-level optimization policy CR is applied, the UI will not allow saving changes to that workload’s optimization policy.
* Users can preview recommendations from other policies even when a CR is applied.
Different approaches for setting an optimization policy may take precedence over others. **Review the priority order** to ensure your configuration leads to the desired outcome:
**Workload-level (Highest Priority)**
1. Workload CR -> `Custom CPU/Custom Memory Policy`
2. Workload CR -> `Global`
3. Workload UI -> `Custom CPU/Custom Memory Policy`
4. Workload UI -> `Global`
**Namespace-level4$**
5. Namespace CR -> `Custom CPU/Custom Memory Policy`
6. Namespace CR -> `Global`
**Cluster-level**
7. Cluster CR -> `Custom CPU/Custom Memory Policy`
8. Cluster CR -> `Global`
9. Cluster UI -> `Custom CPU/Custom Memory Policy`
10. Cluster UI-> `Global`
## Independent optimization policies for CPU and Memory
PerfectScale enables independent optimization policies for **CPU** and **Memory**, ensuring recommendations and automation are precisely aligned with your optimization goals while meeting reliability requirements.
## Memory request = memory limit policy
For memory-sensitive workloads, PerfectScale allows you to apply a **Memory Request = Memory Limit** policy on top of your optimization policy. This helps reduce the risk of OOM events and prevents noisy-neighbor effects by ensuring guaranteed memory allocation for your critical services.
{% hint style="info" %}
In some cases, higher-priority constraints may affect recommendations and take precedence over the **Memory Request = Limit** setting, including:
* ResourceQuota
* LimitRange
* Automation CR restrictions
* Avoid setting limits for young workloads
{% endhint %}
{% hint style="danger" %}
A conflict may occur between the **Request = Limit** configuration and the Automation CR.
For example, if **Request = Limit** is enabled while the CR automation configuration sets `memoryManagement.request.increaseEnabled=false`, recommendations will be restricted accordingly.
{% endhint %}
## Custom optimization policy time window
PerfectScale uses the default data maturity time window for the recommendations, depending on the selected Optimization Policy. This algorithm allows PerfectScale to collect sufficient data to ensure reliable recommendations and accurately adjust resources through automated actions.
However, there may be situations when the default setup is not efficient enough due to known and predictable resource spikes within a known timeframe. PerfectScale allows you to set a **Custom time window** for the optimization policy to tailor recommendations and automation to your services' specific needs.
You can adjust the time window for the optimization policy for both the entire cluster and a specific workload.
## Setting optimization policy via UI
#### Setting independent policies for the cluster via UI
You can seamlessly set the independent CPU and Memory optimization policies for the already running cluster.
From the Overview screen, navigate to the **Cluster Settings** by clicking the **gear icon** next to the cluster. Open the Optimization Policy dropdown, and select **`Custom CPU & Mem Policies`**. Then select the desired policies for both CPU and Memory, and click the `Save changes` button.
Setting independent policies for the cluster
#### Setting independent policies for the workload via UI
Click the specific workload to open the Zoom-in window. In the top-right corner, open the Optimization Policy dropdown, select **`Custom CPU & Mem Policies`**, elect the desired policies for both CPU and Memory, and click the `Save` button.
Setting independent policies for the workload
#### Setting memoryRequestEqualsLimit for the cluster via UI
You can seamlessly set the memoryRequestEqualsLimit policy for the already running cluster.
From the Overview screen, navigate to the **Cluster Settings** by clicking the **gear icon** next to the cluster. Open the Optimization Policy dropdown, and select **`Memory Request = Limit`**. Then click the `Save changes` button.
Setting Memory Request = Limit policy for a cluster
#### Setting memoryRequestEqualsLimit for the workload via UI
Click the specific workload to open the Zoom-in window. In the top-right corner, open the Optimization Policy dropdown, select **`Memory Request = Limit`**, and click the `Save` button.
Setting Memory Request = Limit policy for a workload
{% hint style="info" %}
Once **`Memory Request = Limit`** is enabled, PerfectScale automatically applies i[ndependent optimization policies](#independent-optimization-policies-for-cpu-and-memory) for CPU and memory for this workload.
{% endhint %}
#### Setting a custom time window for the cluster via UI
You can easily specify the custom time window for the optimization policy when onboarding the cluster, or modify it for an already running cluster.
When onboarding a new cluster, enable the toggle `Custom time window` in the Optimization Policy drop-down and select the policy and the desired time window.
{% hint style="info" %}
If the exporter was installed with a specified custom policy, the subsequent upgrades should also include it. PerfectScale recommends using the upgrade command in the UI, as it keeps your fields from the installation. Find out more about upgrading the agent [here](/administration/cluster-settings#upgrade-available).
{% endhint %}
Custom time window
If you want to modify the custom time window for the already onboarded cluster, navigate to the **Cluster Settings** by clicking the **gear icon** next to the cluster. Open the Optimization Policy dropdown, and select the desired policy and custom time window.
{% hint style="info" %}
If a custom policy is set through the exporter when installing the PerfectScale Agent, it cannot be modified in the UI afterward. You can still change the custom policy by upgrading the exporter with the new value, or you can return it to the default by upgrading the exporter without specifying any value (this will also enable the option to change the custom time window through the UI).
{% endhint %}
Setting custom time window for the cluster
Once the custom time window is applied, it will be displayed next to the optimization policy in UI.
#### Setting a custom time window for the workload via UI
To set the custom time window for the optimization policy of the particular workload, navigate to the Zoom-in window by clicking the workload, select the desired optimization policy and custom time window, and click `Save` button.
Setting custom time window for the workload
## Setting optimization policy with CR
Custom Resource (CR) provides a cloud-native way to define and manage optimization policies directly through Kubernetes manifests.
This advanced approach is specifically helpful if you want to manage policies as code (versioning, reviews, GitOps workflows) and apply consistent configurations across clusters and environments with full control.
{% hint style="info" %}
CRs support optimization policies at the cluster, namespace, and workload levels.
{% endhint %}
#### Setting optimization policy for the cluster with CR
To apply a cluster-wide optimization policy, create a cluster-level custom resource. Here is an example:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterRecommendationsConfig
metadata:
name: cluster-recommendations-config
spec:
policies:
workloadPolicy: Balanced # General optimization policy applied to all workloads by default
cpuPolicy: MaxSavings # Override the general policy for CPU specifically
memoryPolicy: MaxHeadroom # Override the general policy for memory specifically
customTimeWindowDays: 30 # Use 30 days of historical data for recommendations
memoryRequestEqualsLimit: false # Ensure memory request equals limit for stability
```
When an optimization policy is set at the cluster level via a CR, a corresponding badge appears next to the policy name. Clicking the badge opens a pop-up with the configured CRD.
Cluster optimization policy with CR
#### Setting optimization policy for the namespace with CR
To apply an optimization policy to the specific namespace, create a namespace-level custom resource. Here is an example:
```yaml
apiVersion: perfectscale.io/v1
kind: NamespaceRecommendationsConfig
metadata:
name: demo-recommendations-config
namespace: demo
spec:
policies:
workloadPolicy: MaxSavings # General optimization policy applied to all workloads by default
cpuPolicy: MaxSavings # Override the general policy for CPU specifically
memoryPolicy: Balanced # Override the general policy for memory specifically
customTimeWindowDays: 14 # Use 14 days of historical data for recommendations
memoryRequestEqualsLimit: true # Ensure memory request equals limit for stability
```
#### Setting optimization policy for the workload with CR
To apply an optimization policy to the specific workload/workload type, create a workload-level custom resource. Here is an example:
```yaml
apiVersion: perfectscale.io/v1
kind: WorkloadRecommendationsConfig
metadata:
name: rabbitmq-recommendations-config
namespace: apps
spec:
targetRef:
kind: StatefulSet
name: rabbitmq
policies:
workloadPolicy: MaxHeadroom # General optimization policy
customTimeWindowDays: 30 # Use 30 days of historical data for recommendations
memoryRequestEqualsLimit: true # Ensure memory request equals limit for stability
```
When an optimization policy is set at the workload with a CR, a corresponding badge appears next to the policy name. Clicking the `View CRD` button opens a pop-up with the configured CR.
Workload optimization policy with CR
The drop-down menu includes a **Default** policy option to reset the optimization policy to the inherited configuration. This means that if a namespace-level CR is present, you can return to the namespace configuration, or if only a cluster-level policy exists (via CRD or UI), you can return to the cluster default.
{% hint style="info" %}
If a workload-level CR exists, you cannot save changes, but you can preview how different policies affect recommendations.
{% endhint %}
{% hint style="warning" %}
When a CR is applied, you can view its configuration. However, if the current state differs from the CR, you must first return to the default to access it.
{% endhint %}
## Reverting to default optimization policy settings
To revert to the default optimization policy settings **configured via UI**, simply upgrade the exporter without specifying any value. This will also enable modifying the custom time window through the UI.
```yaml
helm upgrade --install -n some-release-name --create-namespace some-namespace \
--set secret.create=true \
--set secret.clientId=CLIENT_ID \
--set secret.clientSecret=CLIENT_SECRET \
--set settings.clusterName=CLUSTER_NAME \
perfectscale/exporter
```
To revert to the default Optimization Policy settings **configured with CR**, delete the corresponding Custom Resource. If a workload-level CR is deleted, the namespace-level configuration will be applied (if available). If the namespace-level CR is deleted, the cluster-level configuration will be used. If the cluster-level CR is also removed, the system defaults will apply.
# Tables customization
Explore how to customize your view to tailor it to your priorities and goals
PerfectScale allows you to customize tables through the platform, helping you tailor each view to the data you care about most. Instead of working with a fixed table layout, you can choose which columns to show, how they are ordered, and how much space each column uses. This makes it easier to scan large data sets, reduce visual noise, optimize screen space, and work more efficiently.
Table customization is available across all the views:
* **Overview**
* **PodFit**
* **InfraFit**
* **Automation**
* **Alerts**
To customize your view, use the **Customize Table** menu on the relevant page.
Customize table menu
### Show or hide columns
You can seamlessly hide specific columns from the table. This helps remove information that is not relevant to your workflow and frees up screen space. For example, if your account does not use labels on the Clusters page, you can hide the **Labels** column to keep the table cleaner and easier to scan.
To do so, simply uncheck irrelevant columns in the **Customize Table** menu.
### Change column order
You can also reorder columns from the **Customize Table** to create a layout that better matches how you work. Move the most important information closer to the beginning of the table, so it is easier to access the information.
Once you’re in the **Customize Table** menu, simply drag and drop columns to arrange them in the desired order.
Change column order
### Pin columns
Pin columns to keep them visible while scrolling horizontally across wide tables. Pinned columns remain fixed on the left side of the table, making it easier to keep key information in view while reviewing additional data.
{% hint style="info" %}
* Columns can be reordered separately within pinned and unpinned areas.
* Grouped columns cannot be pinned or reordered.
* You can pin up to 3 columns at a time. Once the limit is reached, an info tooltip will appear.
{% endhint %}
Pin the desired column(s) by clicking the pin button next to the column name in the **Customize Table** menu.
### Resize columns
Resize columns to better fit your screen and improve readability, especially when content exceeds column width. Hover over truncated text to view the full value in a tooltip.
The **Auto-Fit** toggle automatically adjusts column widths so the table fills the available space without horizontal scrolling.
{% hint style="info" %}
You can manually resize columns at any time.
* When **Auto-Fit is on**, resizing one column also adjusts the other columns automatically so the table continues to fill the available space.
* When **Auto-Fit is off**, only the column you resize changes.
If you turn **Auto-Fit** on and then switch it off again, your previously set column widths are preserved, so you can return to your preferred layout at any time.
{% endhint %}
Auto-Fit toggle
### Reset c**ustomization**
When using a **Reset** option, you can choose which settings to reset:
* Column visibility
* Column order
* Pinned columns
* Column widths
This gives you control over what changes are reverted without resetting the entire table layout.
### Keyboard navigation
You can navigate tables using your keyboard. Use the arrow keys to move through table rows and across wide tables:
* **Up / Down** to move between rows.
* **Left / Right** to move horizontally when the table has horizontal scrolling.
# Cluster settings
Review or edit the settings of the onboarded cluster
**Cluster Settings** contains a variety of configurable parameters, including:
* [General cluster settings](#general-settings)
* Various [customizations](#customizations)
* [Cluster labels](#cluster-labels)
* Status of the [Agent](#agent) connecting the cluster to PerfectScale
To navigate to the **`Custer Settings`** , go to the **`Overview`** tab and find the needed cluster from the list of onboarded clusters. Hover over the cluster and click on the **`Gear`** button.
Cluster settings
## General settings
General cluster settings
1. **Cluster Name** - (editable field) displays the name of the cluster.
2. **Optimization Policy** - specifies how resources should be allocated to achieve the desired level of resiliency. This allows setting policies to support the individual needs of your workloads. Keep the default or select one of the following values:
* MaxSavings - maximum cost savings, the best for non-production environments
* Balanced (default) - optimally balances cost and resiliency
* ExtraHeadroom - the best fit for latency-sensitive environments
* MaxHeadroom - keeps the environment above the highest spikes
Discover more about customizing the Optimization policy [here](/customize-workflow/optimization-policy-customization).
3. **Region** - (autodetected editable field) displays the region where the cluster is running.
4. **Cloud Provider** - (autodetected non-editable field) displays the cloud provider name.
5. **Initialization Date** - (non-editable field) displays the date when the cluster was added.
6. **Disconnect Cluster** - click on the button to disconnect the cluster from PerfectScale.
7. **Kubernetes version** - shows the Kubernetes version currently used by the cluster.
Click **`Save Changes`** button to save the changes or **`Cancel & Close`** to discard them.
## Customizations
PerfectScale offers a range of customizable features designed to empower you in building your own optimized K8s ecosystem. These enhancements boost observability throughout your environment while streamlining processes like alerting, ticketing, and pricing management.
Explore the available customizations [here](/customize-workflow/customization-overview).
## Cluster labels
PerfectScale allows you to leverage your cluster labels, enabling you to organize and manage your Kubernetes clusters in the usual way. This capability is particularly helpful when managing large-scale Kubernetes environments with numerous clusters. By defining and applying cluster labels, you can streamline navigation and filtering across PerfectScale dashboards, ensuring quick access to the data you need.
Cluster labels
### Adding labels via the UI
There are two seamless approaches to add cluster labels:
* [From the cluster settings](#add-cluster-labels-from-cluster-settings)
* [From the global settings](#add-cluster-labels-from-global-settings)
#### Add cluster labels from cluster settings
1. In the cluster list, click the **gear** icon next to the cluster name to open **Cluster Settings**.
2. Go to the **Labels** tab.
3. Click **+ New Label**.
4. Enter the **Label Key** and **Label Value**.
{% hint style="warning" %}
The label key must be unique per cluster. If you attempt to add a label using a key that already exists on that cluster, the label will not be applied.
{% endhint %}
{% hint style="info" %}
Both the Key and the Value are limited to a maximum of 225 characters. Only alphanumeric characters and the following special symbols are allowed: `. - _ /`.
{% endhint %}
5. Click **Save**.
Add cluster label - cluster settings
#### Add cluster labels from global settings
1. Navigate to **Global Settings** in the bottom-left corner of the screen.
2. Open **Cluster Labels**.
3. Under the desired cluster, click **+ New Label**.
4. Enter the **Label Key** and **Label Value**.
{% hint style="warning" %}
The label key must be unique per cluster. If you attempt to add a label using a key that already exists on that cluster, the label will not be applied.
{% endhint %}
{% hint style="info" %}
Both the Key and the Value are limited to a maximum of 225 characters. Only alphanumeric characters and the following special symbols are allowed: `. - _ /`.
{% endhint %}
5. Click **Save**.
Add cluster label - global settings
### How to edit/delete a label in the UI
You can seamlessly edit or delete labels. Simply hover over the label in either the cluster settings or the global settings, and select the desired action.
Label actions
### Adding labels via CR
{% hint style="warning" %}
Labels defined in the CR take precedence over labels added via the UI. If both define the same key, the value from the CR will be used.
**Example**:
Label added via the UI: `env: prod`
Label defined in the CR: `env: app`
**Result:** `env: app` is applied (CR has higher priority).
{% endhint %}
To add labels, you need to define and apply a Custom Resource (CR) that specifies these labels (`key: value`). This approach allows you to manage labels directly through Kubernetes manifests.
Here is an example of the CR configuration:
```yaml
apiVersion: perfectscale.io/v1
kind: ClusterSettings
metadata:
name: cluster-settings
namespace: perfectscale
spec:
# Cluster classification labels
clusterLabels:
env: production
team: platform
cost-center: engineering
```
{% hint style="info" %}
Both the Key and the Value are limited to a maximum of 225 characters. Only alphanumeric characters and the following special symbols are allowed: `. - _ /`.
{% endhint %}
Labels defined in the CR are displayed in the UI with a dedicated `CRD` flag. Such labels cannot be edited in the UI and can only be updated in the CR.
Cluster label via CR
## Automation
PerfectScale provides a quick, UI-based automation setup, so there’s no need to configure automation CRs. Learn more about Automation via UI [here](/enable-automation/configuring-automation-via-ui).
Automation via UI
## Agent
PerfectScale delivers its services through agents like the **PSC Exporter**, **PSC AutoScaler,** etc. To continuously analyze your environment, provide recommendations, and execute automation on your clusters, it's essential to have the relevant agent installed.
PerfectScale agents
In **Cluster Settings** (accessed by clicking the gear icon next to the cluster name), select **Agents**. Here you can seamlessly check the current version of the agents, explore recent release notes, and manage the agents. Clicking the **Release Notes** button takes you to [GitHub](https://github.com/perfectscale-io/perfectscale-io.github.io/releases), where you can explore comprehensive release details, including changes and assets.
### Updating agents
In order to ensure you receive superior service and benefit from the latest features, it’s important to keep the agents up to date.
You may encounter two types of update statuses:
* **Update Available** - a newer version is available, including new features and fixes, and can be installed at your convenience.
* **Update Required** - updating to a newer version is mandatory to continue receiving service (for example, if your current version is no longer supported).
PerfectScale provides you with several options on how to upgrade the Agents and keep operations smooth:
* [**Install PerfectScale Operator**](/administration/perfectscale-operator) (recommended)
* [Update the agents manually](/administration/cluster-settings#updating-agents-manually).
#### Updating agents with PerfectScale Operator
The PerfectScale Operator is a Kubernetes operator that handles the installation and upgrades of PerfectScale components, including the Exporter, Automation Agent, and others. It helps reduce maintenance, minimize manual work, and speed up feature rollouts across your clusters.
To learn more about installing and using the PerfectScale Operator, go [here](/administration/perfectscale-operator).
#### Updating agents manually
When updating agents manually, go to **Cluster Settings** (accessed by clicking the gear icon next to the cluster name), select **Agents**. Once there, click **Show Updates Instructions** next to the agent, where newer version is available.
Updating agent manually
Proceed with the two steps:
1. Update the Helm repository
```
helm repo update perfectscale
```
2. Update the PerfectScale version (example for the exporter):
```
helm upgrade --install -n perfectscale \
--reset-then-reuse-values \
perfectscale perfectscale/exporter
```
Alternatively, you can install the PerfectScale Operator to automate deployment and ensure your agents stay up to date with the latest features and product updates. [Learn how to migrate](/administration/perfectscale-operator#migration-of-existing-cluster-to-operator).
### Managing agents
PerfectScale provides a comprehensive **Agent Management** view, allowing you to easily access agent-related information and manage or update your agents. There are two ways to access the Agent Management view:
1. From the Overview screen.
2. From the Account settings.
Manage agent
From this view, you can:
* See information about available agent updates.
* Access detailed **Helm** or **CRD** instructions, depending on the selected update mode.
* Monitor the progress of ongoing updates or see a **Failed** status if an update encounters an issue.
By hovering over the displayed status or listed version you can view detailed information about the update or the reason for any failure.
Update failed flag
If the Agent Operator is installed to the cluster(s) and the mode is set to [webapp](/administration/perfectscale-operator#webapp), you can trigger updates for all agents with a single click for these cluster(s).
Upgrade all agents
#### Agent version history
When updating agents manually by explicitly setting the agent version in the CRD, you can review the CRD history to track previous versions, see changes made over time, and verify update actions for auditing or troubleshooting purposes. To access it, click the **CRD** label next to the agent version, then select **Show History**.
Agent version history
## Specifying an optimization policy or returning to the default
{% hint style="info" %}
If you set an optimization policy or custom time window using `helm upgrade`, you won’t be able to change these settings in the UI until you revert them to the defaults.
{% endhint %}
When upgrading the Agent, you can seamlessly specify an optimization policy and set a [custom time window](/customize-workflow/optimization-policy-customization).
To specify an optimization policy and/or a custom time window when installing the Agent, use the following command:
```yaml
helm upgrade --install -n some-release-name --create-namespace some-namespace \
--set secret.create=true \
--set secret.clientId=CLIENT_ID \
--set secret.clientSecret=CLIENT_SECRET \
--set settings.clusterName=CLUSTER_NAME \
--set settings.clusterOptimizationPolicy="MaxSavings" \ # specify a desired optimization policy
--set settings.optimizationPolicyTimeWindow="21d" \ # set a desired custom time window
perfectscale/exporter
```
To revert to the default optimization policy and time window, upgrade the Agent using the following command:
```yaml
helm upgrade --install -n some-release-name --create-namespace some-namespace \
--set secret.create=true \
--set secret.clientId=CLIENT_ID \
--set secret.clientSecret=CLIENT_SECRET \
--set settings.clusterName=CLUSTER_NAME \
perfectscale/exporter
```
{% hint style="info" %}
To revert to the default optimization policy and time window, do not specify `clusterOptimizationPolicy` and `optimizationPolicyTimeWindow`.
{% endhint %}
## Delete a cluster
{% hint style="info" %}
The endpoint removes the cluster instance from PerfectScale.
{% endhint %}
There may be situations where you need to remove a cluster from PerfectScale. You can seamlessly delete a cluster using the Public API. Learn more [here](/api/public-api#deleting-a-cluster).
# PerfectScale Operator
Explore the PerfectScale agent installation and upgrade management system.
The PerfectScale Operator is a Kubernetes operator that manages the installation and upgrade of PerfectScale components, including the exporter, automation agent, and other components. It is designed to reduce maintenance overhead, minimize manual work, and accelerate feature rollout across your clusters.
PerfectScale Operator
## Installing PerfectScale Operator on a new cluster
When onboarding a cluster, choose one deployment method: deploy with the PerfectScale Operator or deploy with Helm. Select **PerfectScale Operator** option to install the operator while onboarding a cluster.
Once you have named your cluster and selected the Optimization Policy, click **`Get Install Command`**, so PerfectScale will provide you with the needed installation components.
To upgrade, click **Show Update Instructions** to access step-by-step guidance. The upgrade process includes two simple steps:
{% stepper %}
{% step %}
Update the Helm repository
```
helm repo update perfectscale
```
{% endstep %}
{% step %}
Update the PerfectScale Operator version
```
helm upgrade --install psc-agent-operator perfectscale/psc-agent-operator \
-n perfectscale --reset-then-reuse-values
```
{% endstep %}
{% endstepper %}
Keeping the PerfectScale Operator updated ensures you benefit from the latest features, improvements, and overall service quality.
## Upgrade modes
PerfectScale supports three upgrade modes for agents: auto, webapp, and manual.
{% hint style="warning" %}
Both **Admin** and **PowerUser** roles can add and update clusters. Only **Admin** users can disconnect a cluster. Learn more about roles and permissions [here](/administration/rbac-or-roles-and-permissions).
{% endhint %}
{% hint style="info" %}
Learn more about upgrade modes prerequisites [here](/administration/perfectscale-operator/prerequisites-for-the-upgrade-modes).
{% endhint %}
### ⚡ **Auto (recommended)**
In this mode, PerfectScale Operator automatically upgrades to the latest version. The operator queries the PerfectScale settings API (`/psc-settings`) on every reconciliation to retrieve the latest version for the exporter and autoscaler. The version field is omitted, and the operator always upgrades to the newest release. To enable auto mode, you need to configure it in the PerfectScale Operator [CRD](#understanding-agents-operator-crd) for exporters and autoscaler.
Example:
```yaml
spec:
exporter:
installationMode: "on"
upgrade:
mode: auto
autoscaler:
installationMode: "on"
upgrade:
mode: auto
```
Once auto mode is enabled, your agents are always kept up to date automatically. This ensures you have access to the latest features and fixes without interrupting operations or requiring any manual effort from your team.
### **Webapp**
This mode offers semi-automatic upgrades, where you need to approve upgrades in the PerfectScale platform, and then the operator updates to the latest version. This approach gives your team a change-management gate while still automating the process.
To enable webapp mode, you need to configure it in the Agents Operator [CRD](#understanding-agents-operator-crd) for exporters and autoscaler.
Example:
```yaml
installationMode: "on"
upgrade:
mode: webapp
autoscaler:
installationMode: "on"
upgrade:
mode: webapp
```
In **Cluster Settings** (accessed by clicking the gear icon next to the cluster name), select **Agents**. Click the **Update** button next to a specific agent to upgrade it to the latest version.
Webbapp upgrade mode
Alternatively, you can upgrade all agents at once with a single click by selecting **Update All Agents** in the top-right corner, or from the global Agent section ([learn more here](/administration/cluster-settings#managing-agents)).
Upgrade all agents
{% hint style="info" %}
The cluster must be connected to the PerfectScale platform so the operator can receive approval actions from the web app.
{% endhint %}
### **Manual**
This mode suits the teams that do not want to upgrade agents automatically and prefer manual agent management. To enable manual mode, you need to configure it in the PerfectScale Operator [CRD](#understanding-agents-operator-crd) for exporters and autoscaler.
{% hint style="info" %}
When upgrading agents manually, users must explicitly set the desired version in the [CRD](#understanding-agents-operator-crd) to trigger an upgrade. The version field must specify an exact version (e.g., `v1.0.0`).
To check the latest available version, navigate to **Cluster Settings** (click the gear icon next to the cluster name) and select **Agents**. The available version is displayed in the **Update Available/Required** section.
{% endhint %}
Here’s an example of what a CRD may look like:
```yaml
apiVersion: perfectscale.io/v1alpha1
kind: Agents
metadata:
labels:
app.kubernetes.io/name: psc-agent-operator
name: agents
namespace: perfectscale
spec:
clustername: your_cluster
credentials:
name: perfectscale-credentials
exporter:
installationMode: "on"
upgrade:
mode: manual
version: "1.0.83" # The desired exporter version
autoscaler:
installationMode: "on"
upgrade:
mode: manual
version: "1.0.34" # The desired autoscaler version
promexporter:
installationMode: "on"
upgrade:
mode: manual
version: "0.2.0" # The desired prom exporter version
```
# Prerequisites for the upgrade modes
Learn about prerequisites for the upgrade modes.
Before enabling `auto` or `webapp` upgrade mode, ensure your cluster nodes can pull the required agent images. Choose the option that matches your network setup.
## Public ECR (no setup required)
All PerfectScale agent images are published to public ECR. If your cluster nodes have outbound internet access to `public.ecr.aws`, no extra configuration is needed.
**Operator image**
```
public.ecr.aws/perfectscale-io/psc-agent-operator
```
**Agent images** *(tags change with each release)*
```
public.ecr.aws/perfectscale-io/psc-exporter
public.ecr.aws/perfectscale-io/psc-autoscaler
public.ecr.aws/perfectscale-io/psc-prom-exporter
```
**Sub-agents and init images used by the Helm charts**
| Image | Purpose |
| --------------------------------------------------- | --------------------- |
| `public.ecr.aws/perfectscale-io/alpine` | Init container |
| `public.ecr.aws/perfectscale-io/coroot-node-agent` | Node-level metrics |
| `public.ecr.aws/perfectscale-io/kubectl` | In-cluster operations |
| `ghcr.io/prometheus-community/windows-exporter` | Windows nodes only |
| `public.ecr.aws/perfectscale-io/kube-state-metrics` | Kube-state-metrics |
***
## Private Registry (air-gapped or restricted clusters)
If your nodes cannot pull from `public.ecr.aws`, mirror the images to your own registry. Because image tags change with every release, you have two mirroring strategies:
**1. Pull-Through Cache *****(recommended for auto and webapp mode)***
Configure your registry (ECR, Harbor, Nexus, etc.) as a pull-through cache for `public.ecr.aws`. New image tags are fetched automatically whenever the operator upgrades agents, without manual intervention required between releases.
**2. Manual Mirror *****(suitable for manual and webapp mode)***
Pull and push each image for the specific version you intend to run. You must repeat this process for every upgrade.
Use the following Helm install command:
```bash
helm upgrade --install psc-agents-operator perfectscale/psc-agent-operator \
-n perfectscale --create-namespace \
--set secret.create=true \
--set secret.clientId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \
--set secret.clientSecret=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \
--set settings.clusterName={your-cluster-name} \
--set controllerManager.manager.image.repository=your-registry.example.com/perfectscale-io/psc-agent-operator
```
## Overriding the image registry in the AgentManager CR
Use `valuesOverride` per agent to point each component to your private registry.
The operator merges `valuesOverride` on top of its own defaults before every Helm install or upgrade. Your custom registry is preserved even when the operator auto-upgrades to a new version.
# Understanding PerfectScale Operator CRD
The Agents CR is the main resource used to install and manage PerfectScale components. Each component uses the same structure:
* `exporter`
* `autoscaler`
* `promexporter`
### Full annotated example
```yaml
apiVersion: perfectscale.io/v1alpha1
kind: Agents
metadata:
name: agents
namespace: perfectscale
spec:
exporter:
# Controls whether the operator installs and manages this agent.
# "on" - install the agent and keep it in sync
# "off" - do not install the agent. If the agent is already installed, it will be removed.
installationMode: "on"
upgrade:
# Controls how the operator chooses which version to run.
# auto: The operator automatically upgrades to the latest stable version after it is published by PerfectScale.
# webapp: The version is controlled from the PerfectScale webapp. The operator waits for approval before upgrading.
# manual: You pin a specific version using the version field. The operator will not upgrade the agent until you update that value.
mode: auto
# Required only when mode is set to manual. Forbiden when mode is auto or webapp.
# The version must use the vX.Y.Z format, for example: "v1.1.8".
# To upgrade, set a newer version and apply the manifest.
# To roll back, set an older version and apply the manifest.
# version: "v1.1.8"
# Optional. Use valuesOverride to override values supported by the exporter Helm chart.
# The operator manages credentials and cluster name automatically. You only need valuesOverride for registry overrides, resource tuning, or other advanced chart settings.
# Supported exporter values: https://perfectscale-io.github.io/charts/exporter
valuesOverride:
image:
repository: your-registry.example.com/perfectscale-io/psc-exporter
initContainer:
image:
repository: your-registry.example.com/perfectscale-io/alpine
coroot:
image:
repository: your-registry.example.com/perfectscale-io/coroot-node-agent
# kube-state-metrics uses separate registry and repository fields.
kube-state-metrics:
image:
registry: your-registry.example.com
repository: perfectscale-io/kube-state-metrics
crdUpgradeHook:
image:
repository: your-registry.example.com/perfectscale-io/kubectl
autoscaler:
installationMode: "on"
upgrade:
mode: auto
# Supported autoscaler values: https://perfectscale-io.github.io/charts/psc-autoscaler
# valuesOverride: {}
promexporter:
# Disabled by default. Set installationMode to "on" to enable Prometheus metrics export.
installationMode: "off"
upgrade:
mode: auto
# Supported promexporter values: https://perfectscale-io.github.io/charts/psc-prom-exporter
# valuesOverride: {}
```
### Upgrade modes
Each agent supports the following upgrade modes:
Mode
Description
auto
The operator automatically upgrades the agent to the latest stable version published by PerfectScale.
webapp
The version is controlled from the PerfectScale web app. The operator waits for approval before upgrading.
manual
You set a specific version in the version field. The operator will not upgrade the agent until you update that value.
When using `manual` mode, the `version` field is required and must use the `vX.Y.Z` format.
Example:
```yaml
upgrade:
mode: manual
version: "v1.1.8"
```
### valuesOverride
Use `valuesOverride` to override values supported by the underlying Helm chart.
The operator automatically manages credentials and cluster name, so you usually do not need to set those values manually.
Common use cases for `valuesOverride` include:
* Using a private image registry
* Overriding image repositories
* Adjusting resource requests or limits
* Configuring advanced Helm chart settings
### Status fields
The operator continuously updates `.status` to show the current state of each agent.
The status fields are read-only.
```yaml
status:
exporter:
currentVersion: "v1.1.8" # The version currently running in the cluster.
expectedVersion: "v1.1.8" # The version the operator is trying to apply.
errorMessage: "" # Shows the latest install or upgrade error, if one occurred.
autoscaler:
currentVersion: "v1.0.47"
expectedVersion: "v1.0.47"
promexporter:
currentVersion: "v0.2.4"
expectedVersion: "v0.2.4"
```
Check the current status at any time by running:
```bash
kubectl get agents agents -n perfectscale -o jsonpath='{.status}' | jq .
```
# User management
A brief guide on how to invite more users and collaborate with your team
## Invite more users to your team
{% hint style="info" %}
For safety reasons, users can only be added to an existing account through the `Invite User` feature in the app. Any other method will fail.
{% endhint %}
### Invite the user from the Overview
By clicking `Invite Users` on the Overview screen, you will access the user management window.
Invite users from overview
Once you are there, click the **`Invite User`** button and select the role from the drop-down list (Admin, Read Only, etc.). Ensure you have entered information in all the mandatory fields (Email, Full Name) and click **`Invite`**.
Invite user form
Once all the steps are completed, the user will receive an invitation link.
### Invite the user from the Organization Settings
Click on the **`Profile`** button in the bottom left corner and select **`Organization Settings`**. In the **`Users`** tab and click the **`Invite User`** button.
Invite users from org settings
Once you are there, select the role from the drop-down list (Admin, Read Only, etc.). Ensure you have entered information in all the mandatory fields (Email, Full Name) and click **`Invite`**.
Invite user form
Once all the steps are completed, the user will receive an invitation link.
# RBAC | Roles and permissions
Explore user roles and their associated permissions
PerfectScale provides detailed permission control to keep operations secure and organized. There are three user roles: Admin, Power User, and Read-Only, each with different access levels.
{% hint style="info" %}
The customers who have access to the DoiT Kubernetes Lens also have access to their PerfectScale tenant with an [**Admin role**](/administration/rbac-or-roles-and-permissions).
{% endhint %}
## Roles
### Admin
This role is assigned by default to the first user who creates the account. Admin has full access to all system features, including user management, cluster settings, policy configuration, and platform customization.
### Power User
This role is assigned by the account Admin. Power Users can configure workload policies and customization profiles. However, they cannot manage users or cluster settings.
### Read-Only
Users with the Read-Only role can only access and view the account and cannot make any configuration changes.
### **Weekly report receiver**
This role is automatically assigned alongside the Admin role and allows users to receive the PerfectScale weekly report.
## Permissions matrix
| Feature | Admin | Power User | Read-Only |
| ------------------------------------------ | ----- | ---------- | --------- |
| **Organization settings - Workspace** | ✅ | ❌ | ❌ |
| **Organization settings** - **My account** | ✅ | ✅ | ✅ |
| **Invite new user** | ✅ | ❌ | ❌ |
| **Read Access** | ✅ | ✅ | ✅ |
| **Disconnect cluster** | ✅ | ❌ | ❌ |
| **Add cluster** | ✅ | ✅ | ❌ |
| **Upgrade cluster** | ✅ | ✅ | ❌ |
| **Edit cluster** | ✅ | ✅ | ❌ |
| **Modify cluster policy** | ✅ | ✅ | ❌ |
| **Modify cluster policy window** | ✅ | ✅ | ❌ |
| **Modify workload policy** | ✅ | ✅ | ❌ |
| **Modify workload policy window** | ✅ | ✅ | ❌ |
| **Mute/Unmute workload** | ✅ | ✅ | ❌ |
| **Create/Delete Jira ticket** | ✅ | ✅ | ❌ |
| **View profile settings** | ✅ | ✅ | ❌ |
| **Edit profile settings** | ✅ | ✅ | ❌ |
| **Assign profile to a cluster** | ✅ | ✅ | ❌ |
# Subscription details
A detailed overview of your PerfectScale license, including the current Plan, start and expiration dates, and vCPU consumption
Subscription details enable easy management of your PerfectScale subscription, helping you track consumption metrics and maintain visibility of your license status.
Click the **Account button** in the bottom left corner, then go to the **Subscription Details** to review your license details.
Navigation
## Subscription Details window
Subscription details window
Field
Description
Plan
Current subscription plan (Trial, Community, Advanced or Expert)
Plan Start Date
The date of license activation
Contract Renewal Date
Current license expiration date
Active Since
Account creation date
Licensed Monthly vCPU
The available monthly vCPU under the license
Current Month vCPU Usage
The current monthly vCPU
Forecasted Monthly vCPU Usage
The forecasted monthly vCPU
For a comprehensive view of your vCPU consumption over time, click the **View over-time vCPU consumption report** button, and it will navigate you to the trend report, where you can slice and dice the data with various filters.
## Understanding billing
PerfectScale billing is based on the **average monthly vCPU usage**. This approach is designed for dynamic Kubernetes environments, where workloads often scale up and down over time, ensuring fair and accurate pricing.
{% hint style="success" icon="function" %}
**avg Monthly vCPU = total vCPU-hours used during the month / total hours in that month**
{% endhint %}
This gives the **average concurrent vCPU usage** across the full month.
# Security
Learn more about security measurements PerfectScale provides
PerfectScale cares about your secure usage of its application and prioritizes the protection of your data and infrastructure. Here, you’ll find detailed instructions on configuring Single Sign-On and Multi-Factor Authentication for enhanced access security. You’ll also learn what RBAC permissions PerfectScale needs in order to provide service, as well as how to configure proxy settings to ensure secure network communication.
# MFA
The step-by-step guide for adding MFA
PerfectScale provides multiple authentication steps to increase the security of your account.
{% hint style="info" %}
MFA is a premium feature available only with our EXPERT package for PerfectScale’s paying customers. Contact to learn more about how to gain access to this feature.
{% endhint %}
Setting up Multi-Factor Authentication (MFA) for your account is an easy process and can be done in a few simple steps.
1. Access **`Organization Settings`** by clicking on the **`Profile`** button and selecting it from the list.
2. Go to the **`Privacy & Security`** tab.
3. Click the **`Set up`** button located under the Multi-Factor Authentication section.
Set up authenticator app
4. Open your preferred authentication app, scan the displayed QR Code, and follow the app prompts.
5. Enter the passcode from the app to the provided cells and click the **`Confirm`** button.
6. To delete the MFA method, use the same account on the Authentication app.
Delete authenticator app
# SSO
The step-by-step guide for adding SSO
{% hint style="info" %}
SSO is a premium feature available only with our EXPERT package for PerfectScale’s paying customers. Contact to learn more about how to gain access to this feature.
{% endhint %}
## Setting up a new SSO connection
Setting up a new SSO connection is a simple process that can be completed in just a few steps.
1. Access **`Organization Settings`** by clicking on the **`Profile`** button and selecting it from the list.
2. Go to the **`SSO`** tab and click on the **`Setup SSO Connection`** button.
3. Select your provider and follow a step-by-step guided SSO process that is located in the app. The following IDPs are available:
* SAML:
1. Okta
2. Azure
3. Custom
* OpenID:
1. Okta
2. Custom
SSO
## Domain validation
PerfectScale recommends updating the domain in the DNS records. This allows verified end users to use their domain for SSO configuration without verifying domain ownership through DNS.
### How to access DNS settings
1. To access your DNS settings, log in to your domain registrar's account (e.g., GoDaddy, Namecheap, Google Domains, etc.).
2. Navigate to the DNS settings or DNS management section of your domain.
{% hint style="info" %}
If you encounter any issues accessing your DNS records, please reach out to us at or in our [Slack](https://join.slack.com/t/perfectscalecommunity/shared_invite/zt-1tu9teu9e-Z9tGt4LpNI8tUC3j8obcmQ) community.
{% endhint %}
### How to add a TXT record
1. After accessing the DNS settings, find the option to add a new record.
2. Choose TXT as the record type.
3. Depending on your registrar's requirements, you may leave the `Name` or `Host` field blank or enter `@`.
4. For the Value field, input the TXT record value provided by the Auth Provider. This value is unique to your configuration and can be found in your Organization Settings.
5. Make sure to save or apply the changes to add the TXT record to your DNS settings.
6. Once the process is complete, the authentication provider should detect the record and validate your domain.
{% hint style="info" %}
DNS TXT record changes usually apply within minutes to a few hours, but full global propagation can take up to 48 hours.
{% endhint %}
# Agent RBAC Permissions
Explore permissions PerfectScale Agent requires to provide the service
In order to provide the insights, and take the automated actions, to keep your K8s environment optimized, our agent requires the below permissions.
```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: perfectscale-exporter
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- nodes/proxy
- replicationcontrollers
- persistentvolumeclaims
- persistentvolumes
- services
- namespaces
- events
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- apiGroups:
- "apps"
resources:
- deployments
- replicasets
- daemonsets
- statefulsets
verbs:
- get
- list
- watch
- apiGroups:
- "storage.k8s.io"
resources:
- storageclasses
- csinodes
verbs:
- get
- list
- watch
- apiGroups:
- "batch"
resources:
- jobs
verbs:
- get
- list
- watch
- apiGroups:
- "autoscaling"
resources:
- horizontalpodautoscalers
verbs:
- get
- list
- watch
- apiGroups:
- "coordination.k8s.io"
resources:
- leases
verbs:
- create
- get
- list
- watch
- update
- apiGroups:
- "metrics.k8s.io"
resources:
- pods
verbs:
- get
- list
- nonResourceURLs:
- "/version"
verbs:
- "get"
```
{% hint style="info" %}
If you prefer not to expose node/proxy permissions, you can include the following configuration when [installing the exporter](/getting-started/how-to-onboard-a-cluster#onboarding-instructions):
```
--set settings.cAdvisorScrapingMode=direct
```
Learn more about scraping modes [here](/getting-started/how-to-onboard-a-cluster#setting-cadvisor-scraping-mode).
{% endhint %}
# psc-autoscaler RBAC Permissions
Explore permissions PerfectScale Automation Agent requires to provide the service
In order to take the automated actions, to keep your K8s environment optimized, our psc-autoscaler requires the below permissions.
{% @github-files/github-code-block url="" visible="true" %}
`settings.allowNamespaceCreation` Helm value is enabled by default. When enabled, this permission allows the autoscaler to create target namespaces required for cross-namespace ephemeral workload automation. To opt out of automatic namespace creation, set `settings.allowNamespaceCreation=false` when [installing PerfectScale automation agent](/enable-automation/automation-setup-instructions#step-1-install-perfectscale-automation-agent).
```bash
helm upgrade --install -n perfectscale psc-autoscaler \
--set secret.create=false \
--set settings.allowNamespaceCreation=false \
perfectscale/psc-autoscaler
```
{% hint style="warning" %}
If `settings.allowNamespaceCreation` is disabled, cross-namespace ephemeral workload grouping will not work as expected for newly created ephemeral namespaces. You will need to manually create or configure each required namespace so automation can be applied.
{% endhint %}
# ps-exporter via Proxy Configuration
Instructions on how to connect the PerfectScale agent via proxy
If your cluster does not have access to the Internet and you are utilizing a proxy server you can connect the exporter via proxy in order to use PerfectScale.
To configure ps-exporter via proxy you'll need to modify the helm chart while [connecting the cluster](/getting-started/how-to-onboard-a-cluster) (step: deploy PerfectScale) with additional settings (lines 6,7,8 and 9 in the following command):
{% code lineNumbers="true" %}
```yaml
helm upgrade --install -n perfectscale --create-namespace perfectscale \
--set secret.create=true \
--set secret.clientId="CHNAGE_TO YOUR" \
--set secret.clientSecret="CHNAGE_TO YOUR" \
--set settings.clusterName="CHNAGE_TO YOUR" \
--set settings.httpProxyEnabled=true \
--set settings.httpProxy="http://proxy.net:8080" \
--set settings.httpsProxy="http://proxy.net:8080" \
--set settings.noProxy=".corp\,.svc\,.cluster\,.local\,172.30.0.1\," \
perfectscale/exporter
```
{% endcode %}
:six:`--set settings.httpProxyEnabled=true` - enables proxy
:seven:`--set settings.httpProxy="http://proxy.net:8080" \` - your proxy URL with the `http` protocol
:eight:`--set settings.httpsProxy="http://proxy.net:8080" \` - your proxy URL with the `https` protocol
:nine:`--set settings.noProxy=".corp\,.svc\,.cluster\,.local\,172.30.0.1\," \` - the TLDs are allowed without proxy
{% hint style="info" %}
To use PerfectScale via a proxy, ensure that your proxy server allows `http` and `https` connections to the following endpoints:
* registry.k8s.io
* api.app.perfectscale.io/
* app.perfectscale.io
* perfectscale-io.github.io
* public.ecr.aws
* customer-data-prod-perfectscale-io.s3.eu-central-1.amazonaws.com
* customer-data-prod-perfectscale-io.s3.dualstack.eu-central-1.amazonaws.com
* s3.eu-central-1.amazonaws.com
* s3.dualstack.eu-central-1.amazonaws.com
* storage.googleapis.com
* cloudfront.net
{% endhint %}
# PerfectScale Prometheus Exporter
The PerfectScale Prometheus Exporter seamlessly integrates PerfectScale insights into your existing monitoring and alerting infrastructure
PerfectScale Prometheus Exporter is a powerful feature that converts PerfectScale's optimization recommendations, cost insights, and resource utilization metrics into Prometheus format. Exposing these insights as Prometheus metrics enables you to seamlessly incorporate PerfectScale's detailed analysis into your standard monitoring and alerting workflows, enhancing visibility into cost-efficiency, resource usage across your infrastructure, and overall system performance.
{% hint style="info" %}
To expose insights from PerfectScale as Prometheus metrics, a Kubernetes cluster with Helm v3 or higher and the PerfectScale agent installed and running is required.
{% endhint %}
## Installing the Chart
You can seamlessly install the chart in two simple steps:
1. Add the PerfectScale Helm repository
```
helm repo add perfectscale https://perfectscale-io.github.io --force-update
```
2. Install the chart
```
helm upgrade --install --namespace perfectscale psc-prom-exporter perfectscale/psc-prom-exporter \
--set settings.expandEphemeralWorkloads=false
```
{% hint style="info" %}
`expandEphemeralWorkloads` controls whether ephemeral workloads are shown individually or collapsed. It defaults to `false`, meaning workloads remain collapsed. Available in `psc-prom-exporter` v0.3.2 and later.
{% endhint %}
{% hint style="info" %}
The `Psc-Prom-Exporter` must be installed in the same namespace as the PerfectScale Agent to reuse the existing credentials secret.
{% endhint %}
This will install Prometheus exporter with basic configuration, without manually configured scrapers, dashboards, and alerts. To configure it according to your needs, please follow the documentation. [Quick install](#quick-install) or [Configuration](#configuration) and [Monitoring integrations](#monitoring-integrations) sections.
{% hint style="info" %}
The general behavior of the Prom-Exporter is defined in the `config` section of the values file. Monitoring tools integration is configured in the `scrapers`, `alerts`, and `dashboards` sections of the values file.
{% endhint %}
## Quick install:
### Prom-Exporter with Prometheus Operator with Alerts example and Grafana dashboard
```sh
helm upgrade --install --namespace perfectscale psc-prom-exporter \
--set scrapers.serviceMonitor.enabled=true \
--set alerts.prometheusRule.enabled=true \
--set dashbords.grafana.enabled=true \
perfectscale/psc-prom-exporter
```
### Prom-Exporter with DataDog and DataDog alerts
```
helm upgrade --install --namespace perfectscale psc-prom-exporter \
--set scrapers.datadog.enabled=true \
--set alerts.datadogMonitor.enabled=true \
perfectscale/psc-prom-exporter
```
## Configuration
```yaml
config:
metrics:
recommendations:
enabled: true # Enables resource recommendation metrics
costs:
enabled: true # Enables cost analysis metrics
indicators:
enabled: true # Enables resource utilization indicators
filters:
workloads:
includeMuted: true # Includes workloads marked as muted in PerfectScale
minRunningMinutes: 30 # Only includes workloads running longer than 30 minutes
types: # Workload types to include
- "*" # "*" includes all workload types (Deployments, StatefulSets, etc.)
namespaces:
exclude: # Namespaces to exclude from metrics
- kube-system
- default
indicators:
types: # Types of indicators to expose
- waste # Resource waste indicators
- risk # Risk indicators
labels:
includedLabels:
- "*" # Includes all Kubernetes labels
excludedLabels:
- "/.*perfectscale.*/" # Excludes labels matching this regex pattern
excludeClusterUID: false # Includes cluster UID in metrics
```
### Metrics configuration
```yaml
metrics:
recommendations:
enabled: true # Enable resource recommendation metrics
costs:
enabled: true # Enable cost analysis metrics
indicators:
enabled: true # Enable resource utilization indicators
```
#### recommendations
`recommendations` allows managing the exposure of resource recommendation metrics and, when enabled, exposes the following metrics:
* `ps_recommended_memory_request_bytes`
* `ps_recommended_cpu_request_cores`
* `ps_recommended_memory_limit_bytes`
* `ps_recommended_cpu_limit_cores`
* `ps_current_memory_request_bytes`
* `ps_current_cpu_request_cores`
* `ps_current_memory_limit_bytes`
* `ps_current_cpu_limit_cores`
#### costs
`costs` allows managing the exposure of cost analysis metrics and, when enabled, exposes the following metrics:
* `ps_cost_usd` - hourly workload cos in USD
* `ps_waste_usd` - estimated hourly waste in USD
#### indicators
`indicators` allows managing the exposure of resource utilization indicators and, when enabled, exposes the `ps_workload_indicators` metric with labels for different indicator types.
### Filters configuration
```yaml
filters:
workloads:
includeMuted: true # Include muted workloads
minRunningMinutes: 30 # Minimum runtime to include
types: # Workload types to include
- "*" # All types
# Or specify specific types:
# - "Deployment"
# - "StatefulSet"
# - "DaemonSet"
namespaces:
exclude: # Namespaces to exclude from metrics
- kube-system
- default
indicators:
types: # Types of indicators to expose
- waste # Resource waste indicators
- risk # Risk indicators
```
#### workloads
`workloads` allows managing which workload should be included in the metric based on its type, run-time, etc.
#### namespaces
`namespaces` allows managing which namespace should be excluded from the metric.
#### indicators
`indicators` allows managing indicators to expose
### Labels configuration
The labels section allows managing Kubernetes labels to be included in the metrics.
{% hint style="info" %}
The `excludedLabels` take precedence over `includedLabels`.
{% endhint %}
```yaml
labels:
includedLabels:
- "*" # Include all labels
# Or specify specific labels:
# - "label_app"
# - "label_environment"
excludedLabels:
- "/.*perfectscale.*/" # Exclude labels matching regex
# - "label_internal_id" # Exclude specific label
excludeClusterUID: false # Include cluster UID in metrics
```
{% hint style="info" %}
In the Prometheus format, all special label characters are replaced with an underscore `_`.\
\
**Example**:
Kubernetes: `app.kubernetes.io/instance`
Prometheus: `label_app_kubernetes_io_instance`
{% endhint %}
#### **Label pattern support**
| Pattern | Description |
| ------------------- | --------------------------------------------------- |
| `"label_full_name"` | Matches the exact label name |
| `"*"` | Matches all labels |
| `"/some-regexp/"` | Matches labels with the specific regular expression |
### Configuration examples
Here are a few configuration examples designed to help you effortlessly integrate PerfectScale's insights into your monitoring workflow.
#### Recommended configuration for production environments with short-living workloads excluded
```yaml
config:
metrics:
recommendations:
enabled: true
costs:
enabled: true
indicators:
enabled: true
filters:
workloads:
includeMuted: false
minRunningMinutes: 60
types:
- "Deployment"
- "StatefulSet"
- "DaemonSet"
namespaces:
exclude:
- kube-system
- default
- monitoring
indicators:
types:
- waste
- risk
labels:
includedLabels:
- "label_app"
- "label_environment"
- "label_team"
excludedLabels:
- "/.*internal.*/
- "/.*perfectscale.*/"
excludeClusterUID: false
```
## Monitoring integrations
The PerfectScale Prometheus Exporter is compatible with a range of out-of-the-box monitoring systems to enhance your monitoring process and make it more efficient. This support allows seamless integration, enabling you to quickly leverage PerfectScale's insights without requiring extensive configuration.
### Prometheus Operator integration
This integration will create a Service Monitor Custom Resource to scrape Prom-exporter with Prometheus Operator.
```yaml
scrapers:
serviceMonitor:
enabled: true
interval: 5m
path: /metrics
timeout: 30s
# Additional Label to the serviceMonitor
# labels:
# prometheus: kube-prometheus
```
### Prometheus (Standard Discovery)
This integration allows the use of annotations for Prometheus auto-discovery.
```yaml
scrapers:
prometheus:
enabled: true
path: "/metrics"
port: "http"
interval: "30s"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/metrics"
```
### Datadog Autodiscovery
This integration allows you to pull metrics directly into Datadog. It supports both v1 and v2 autodiscovery annotations.
scrapers:
datadog:
enabled: true
# Choose AD version: "v1" or "v2"
adVersion: "v2"
containerName: "psc-prom-exporter"
# Common configuration for both versions
config:
endpoint: "/metrics"
port: 8080
namespace: "perfectscale"
# Set maxReturnedMetrics, DataDog default value is 2000. Increase if needed.
maxReturnedMetrics: 10000
# List of metrics to collect with optional renaming
metrics:
# Wildcard pattern for ps_ metrics
- name: "ps_.*"
# Renaming exampple:
#- name: "ps_waste_usd"
#rename: "perfectscale.waste.usd"
# Additional configuration options
options: {}
## Dashboards
PerfectScale provides a pre-built Grafana dashboard that offers valuable insights. These dashboards allow you to visualize key metrics and enhance your monitoring experience with minimal configuration required.
**Single Workload Dashboard**
This dashboard offers a comprehensive view of your application's performance. It showcases detailed workload metrics and a clear overview of how resources are utilized, enabling you to monitor efficiency trends and identify optimization opportunities. It includes the following data:
* Detailed workload metrics
* Resource usage patterns
* Cost breakdown
* Optimization opportunities
### Grafana dashboard
Its available as part of helm chart:
```yaml
dashboards:
grafana:
enabled: true
namespace: monitoring
labels:
grafana_dashboard: "1"
team: perfectscale
annotations:
grafana.folder: "PerfectScale"
```
You can also find it in:
* Our monitoring Git repo:
* On Grafana Dashboards:
### DataDog dashboard
You can find it in our monitoring Git repo:
## Alert rules
You can effortlessly set up custom alert rules in the `prometheusRule` Helm values configuration section. This will help you stay ahead of key indicator changes, allowing you to address potential issues proactively and prevent them from impacting the system.
### Configuring alerts
**Cost optimization alert**
There is an example of a cost optimization alert designed to help you proactively manage expenses and enhance resource efficiency for Prometheus and DataDog.
```yaml
rules:
- alert: "PerfectScale Waste Cost Surge"
enabled: true
expr: |
(
ps_waste_usd > 0
and
(
ps_waste_usd
/
(ps_waste_usd offset 1h)
) > 1.5
)
for: 15m
labels:
severity: warning
team: cost-optimization
type: waste
annotations:
summary: "Waste cost increased by more than 50%"
description: "Workload {{ $labels.workload_name }} in namespace {{ $labels.namespace }} has increased its waste by more than 50% in the last hour. Current waste: {{ $value | humanizePercentage }} USD/hour"
- alert: "PerfectScale High Absolute Waste"
enabled: true
expr: |
ps_waste_usd > 100
for: 30m
labels:
severity: warning
team: cost-optimization
type: waste
annotations:
summary: "High waste cost detected"
description: "Workload {{ $labels.workload_name }} in namespace {{ $labels.namespace }} has waste cost exceeding 100 USD/hour. Current waste: {{ $value | humanize }} USD/hour"
```
### Enabling alerts
To activate the pre-configured alerts for Alert Manager (Prometheus Rule):
```yaml
alerts:
prometheusRule:
enabled: true
# Additional labels to the PrometheusRule
#labels:
#release: prometheus
```
To enable alerts in DataDog (Alert rules):
```yaml
alerts:
datadogMonitor:
enabled: true
namespace: "datadog"
labels:
team: cost-optimization
severity: warning
```
# Help Center
Explore Help Center - all the support and resources you need are available here!
Help Center is a centralized hub for documentation, support, and updates to enhance your experience.
To open the Help Center, click the question mark button on the left sidebar.
Help center
:book: Utilize the **Documentation** section to access comprehensive information regarding features, complete with detailed instructions on how to use them and derive maximum results.
:sos: If you encounter any issues with the platform or find a bug, our support team is always available to help. Open a [**Support Ticket**](https://console.doit.com/support) or ask us directly on [**Slack**](https://join.slack.com/t/perfectscalecommunity/shared_invite/zt-1tu9teu9e-Z9tGt4LpNI8tUC3j8obcmQ), and our support heroes will do their best to assist you.
:rocket: Do you have an idea for improving your experience with PerfectScale? Let us know! Use [**Feature Request**](/help-perfectscale-to-improve) to make a suggestion or review and vote for existing requests.
:loudspeaker: To be on top of our latest features and updates, visit the **Feature Announcements** section to review recent changes.
# Public API
Learn how to programmatically interact with PerfectScale, enabling automation, data retrieval, and integration with your existing workflows
The API is exclusively available for our paying customers, allowing them to interact with the platform programmatically. \
Check the following information to get acquainted with the PerfectScale [rate-limiting policy](#rate-limiting) and [authentication process](#authentication).
{% hint style="info" %}
The PerfectScale public API is a premium feature available only with our EXPERT package for PerfectScale’s paying customers. Contact to learn more about how to gain access to this feature.
{% endhint %}
PerfectScale API implements a rate-limiting policy to ensure fair usage and maintain the quality of service. The rate limit is set at **10 requests per minute** per client.
## Authentication
To interact with the PerfectScale API, a token is required. To get the token, follow the steps below:
1. **Retrieve Client Credentials**:
* Go to [https://app.perfectscale.io](https://app.perfectscale.io/).
* Click on your user avatar located at the bottom left corner of the page.
* Select **Organization Settings** from the menu.
* In the pop-up window, navigate to the **API Tokens** tab.
* Click on **Generate Token**.
* Assign a **Read Only** Role to the new token.
* Upon creation, you will be provided with a `client_id` (Client ID) and `client_secret` (Secret Key).
2. **Obtain Access Token**:
* Make a POST request to the [Authentication Endpoint](#authentication-endpoint) with the following payload:
```json
{
"client_id": "your_client_id",
"client_secret": "your_client_secret"
}
```
* The API will respond with a token payload, which will contain your access token.
3. **Access Other Endpoints**:
* With the obtained access token, you can make authorized requests to other endpoints of the PerfectScale API. Include the token in the Authorization header of your HTTP requests as follows:
```http
Authorization: Bearer your_access_token
```
### Authentication Endpoint
{% openapi src="/files/KMk1HD8syQcGTx0QMC8m" path="/auth/public\_auth" method="post" %}
[openapi-final.yaml](https://1573387604-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FABMqnYtsOO44JmQTVSnn%2Fuploads%2FrPCvY2lic8IXN4oAqagO%2Fopenapi-final.yaml?alt=media\&token=3383b5c3-1bec-4b1a-9278-55e652357cb5)
{% endopenapi %}
### Clusters Endpoint
## GET /clusters
>
```json
{"openapi":"3.0.3","info":{"title":"Perfectscale API - OpenAPI 3.0","version":"v1"},"servers":[{"url":"https://api.app.perfectscale.io/public/v1"}],"security":[{"auth":[]}],"components":{"securitySchemes":{"auth":{"type":"http","scheme":"bearer","bearerFormat":"JWT"}},"schemas":{"Cluster":{"type":"object","properties":{"uid":{"nullable":true,"type":"string"},"name":{"type":"string"},"cloud":{"nullable":true,"allOf":[{"$ref":"#/components/schemas/ClusterCloud"}]},"region":{"nullable":true,"type":"string"},"createdAt":{"type":"string","format":"date-time"},"lastTransmittedAt":{"type":"string","format":"date-time"}},"required":["uid","name","createdAt","lastTransmittedAt"]},"ClusterCloud":{"type":"string","enum":["aws","gcp","azure"]}}},"paths":{"/clusters":{"get":{"operationId":"GetClusters","responses":{"200":{"description":"OK","content":{"application/json":{"schema":{"type":"object","properties":{"data":{"type":"array","items":{"$ref":"#/components/schemas/Cluster"}}},"required":["data"]}}}}}}}}}
```
## GET /clusters/{cluster\_uid}
>
```json
{"openapi":"3.0.3","info":{"title":"Perfectscale API - OpenAPI 3.0","version":"v1"},"servers":[{"url":"https://api.app.perfectscale.io/public/v1"}],"security":[{"auth":[]}],"components":{"securitySchemes":{"auth":{"type":"http","scheme":"bearer","bearerFormat":"JWT"}},"schemas":{"ClusterDetail":{"type":"object","properties":{"uid":{"nullable":true,"type":"string"},"name":{"type":"string"},"cloud":{"nullable":true,"allOf":[{"$ref":"#/components/schemas/ClusterCloud"}]},"region":{"nullable":true,"type":"string"},"createdAt":{"type":"string","format":"date-time"},"lastTransmittedAt":{"type":"string","format":"date-time"},"emission":{"type":"object","additionalProperties":{"type":"number","format":"double"}}},"required":["uid","name","createdAt","lastTransmittedAt","emission"]},"ClusterCloud":{"type":"string","enum":["aws","gcp","azure"]}}},"paths":{"/clusters/{cluster_uid}":{"get":{"operationId":"GetCluster","parameters":[{"name":"cluster_uid","in":"path","required":true,"schema":{"type":"string"}},{"name":"period","in":"query","required":false,"schema":{"type":"string"},"description":"The period for which carbon emission is calculated"}],"responses":{"200":{"description":"OK","content":{"application/json":{"schema":{"type":"object","properties":{"data":{"$ref":"#/components/schemas/ClusterDetail"}},"required":["data"]}}}}}}}}}
```
### Workloads Endpoint
{% hint style="warning" %}
Ensure that the following request includes the **`cluster_uid`** parameter, as it is **mandatory**. Follow [the instructions](#how-to-get-cluster_uid) provided to obtain this value.
{% endhint %}
## GET /clusters/{cluster\_uid}/workloads
> List all workloads in a specified cluster
```json
{"openapi":"3.0.3","info":{"title":"Perfectscale API - OpenAPI 3.0","version":"v1"},"servers":[{"url":"https://api.app.perfectscale.io/public/v1"}],"security":[{"auth":[]}],"components":{"securitySchemes":{"auth":{"type":"http","scheme":"bearer","bearerFormat":"JWT"}},"schemas":{"TimeWindow":{"type":"string","description":"The time period for which the data is calculated (from 1h to 30d).","enum":["1h","4h","12h","1d","3d","7d","14d","30d"]},"Workload":{"type":"object","properties":{"id":{"type":"string","description":"A unique identifier for the workload, usually following a namespace-type-name convention."},"name":{"type":"string","description":"A human-readable name for the workload."},"type":{"type":"string","description":"The type of Kubernetes workload. Common types include Deployment, StatefulSet, and DaemonSet, but other string values can also be used to represent custom or extended workload types."},"namespace":{"type":"string","description":"The Kubernetes namespace in which the workload resides."},"firstSeen":{"type":"string","format":"date-time","description":"The timestamp when the workload was first observed."},"lastSeen":{"type":"string","format":"date-time","description":"The timestamp when the workload was last observed."},"runningMinutes":{"type":"integer","description":"The total number of minutes the workload has been running."},"replicasCounts":{"$ref":"#/components/schemas/ReplicasCounts"},"resilienceLevel":{"type":"string","description":"**Deprecated**: Use `optimizationPolicy` instead. The resilience level of the workload (configured on Perfectscale side).\n\nMapping to optimizationPolicy values:\n- `low` → `maxSavings`\n- `medium` → `balanced`\n- `high` → `extraHeadroom`\n- `highest` → `maxHeadroom`\n","deprecated":true,"enum":["low","medium","high","highest"]},"optimizationPolicy":{"allOf":[{"$ref":"#/components/schemas/OptimizationPolicy"}],"description":"The optimization policy applied to the workload."},"optimizationPolicyTimeWindow":{"allOf":[{"$ref":"#/components/schemas/OptimizationPolicyTimeWindow"}],"description":"The time window for optimization policy."},"cpuOptimizationPolicy":{"allOf":[{"$ref":"#/components/schemas/OptimizationPolicy"}],"description":"The CPU-specific optimization policy."},"memoryOptimizationPolicy":{"allOf":[{"$ref":"#/components/schemas/OptimizationPolicy"}],"description":"The memory-specific optimization policy."},"memoryRequestEqualsLimit":{"type":"boolean","description":"Whether memory request equals limit is enabled."},"muteStatus":{"$ref":"#/components/schemas/MuteStatus"},"costAnalysis":{"$ref":"#/components/schemas/CostAnalysis"},"workloadLabels":{"type":"object","description":"Key-value pairs representing kubernets labels associated with the workload.","additionalProperties":{"type":"string","description":"A label value associated with a particular key."}},"indicators":{"type":"array","description":"A list of indicators providing insights into the performance and health of the workload.","items":{"$ref":"#/components/schemas/Indicator"}},"containers":{"type":"array","description":"An object holding detailed information about each container within the workload.","items":{"$ref":"#/components/schemas/Container"}}},"required":["id","name","type","namespace","firstSeen","lastSeen","runningMinutes","replicasCounts","resilienceLevel","optimizationPolicy","optimizationPolicyTimeWindow","cpuOptimizationPolicy","memoryOptimizationPolicy","memoryRequestEqualsLimit","muteStatus","costAnalysis","workloadLabels","indicators","containers"]},"ReplicasCounts":{"type":"object","description":"An object representing the count statistics of replicas for a workload over a month period.","properties":{"maxCount":{"type":"integer","description":"The maximum count of replicas observed for the workload during the month period."},"avgCount":{"type":"integer","description":"The average count of replicas observed for the workload during the month period."}},"required":["maxCount","avgCount"]},"OptimizationPolicy":{"type":"string","description":"The optimization policy determining resource allocation strategy.","enum":["balanced","extraHeadroom","maxHeadroom","maxSavings"]},"OptimizationPolicyTimeWindow":{"type":"string","description":"The time window for optimization policy.","enum":["14d","21d","30d","45d","policyDefault"]},"MuteStatus":{"type":"object","properties":{"isMuted":{"type":"boolean","description":"Indicates whether the workload is currently muted."},"expires":{"type":"string","format":"date-time","nullable":true,"description":"The date and time when the workload will automatically become unmuted. If null, the mute status does not expire automatically."}},"required":["isMuted","expires"]},"CostAnalysis":{"type":"object","description":"Analysis of the costs associated with the workload's resource consumption for the requested period.","properties":{"past30Days":{"allOf":[{"$ref":"#/components/schemas/PastPeriodCostAnalysis"}],"description":"Cost analysis data for a 30-day window when an explicit period is requested (including period=30d). When no period is specified the default 28-day window is used for backward compatibility.\n"},"pastPeriod":{"$ref":"#/components/schemas/PastPeriodCostAnalysis"},"next30Days":{"allOf":[{"$ref":"#/components/schemas/NextPeriodCostAnalysis"}],"description":"Projected cost data for a 30-day window when an explicit period is requested (including period=30d). When no period is specified the default 28-day window is used for backward compatibility.\n"},"nextPeriod":{"$ref":"#/components/schemas/NextPeriodCostAnalysis"}},"required":["past30Days","next30Days","pastPeriod","nextPeriod"]},"PastPeriodCostAnalysis":{"type":"object","description":"Metrics detailing the total and wasted costs associated with the workload's resource consumption over the analysed period.","properties":{"totalCost":{"type":"number","format":"double","description":"The total cost incurred by the workload's resource consumption."},"wastedCost":{"type":"number","format":"double","description":"The cost attributed to over-provisioned or inefficiently utilized resources."},"costPerHour":{"type":"number","format":"double","description":"The cost of the workload per hour."}},"required":["totalCost","wastedCost","costPerHour"]},"NextPeriodCostAnalysis":{"type":"object","description":"Metrics detailing the cost increase and potential savings associated with the workload's upcoming potential resource consumption.","properties":{"costIncrease":{"type":"number","format":"double","description":"The workload total cost increase."},"potentialSavings":{"type":"number","format":"double","description":"The potential savings for the workload."}},"required":["costIncrease","potentialSavings"]},"Indicator":{"type":"object","description":"An object representing key performance indicators that provide insights into the performance and health of the workload or container.","properties":{"name":{"type":"string","description":"The name of the indicator representing a specific aspect of performance or health.","enum":["OOM","CpuThrottling","CpuRequestNotSet","MemRequestNotSet","MemLimitNotSet","UnderProvisionedMemRequest","UnderProvisionedMemLimit","UnderProvisionedCpuRequest","UnderProvisionedCpuLimit","OverProvisionedCpuRequest","OverProvisionedMemRequest","RestartsObserved"]},"type":{"type":"string","description":"The type of indicator, categorizing it as either a risk, waste, or no issue (none).","enum":["none","risk","waste"]},"severityLevel":{"type":"integer","description":"The severity level of the indicator, ranging from 0 (no issue) to 3 (critical issue).","enum":[0,1,2,3]}},"required":["name","type","severityLevel"]},"Container":{"type":"object","description":"Detailed information about a specific container within the workload, including its performance indicators and usage statistics.","properties":{"name":{"type":"string","description":"The name of the container."},"runningMinutes":{"type":"integer","description":"The total number of minutes the container has been running."},"indicators":{"type":"array","description":"A list of indicators providing insights into the performance and health of the container.","items":{"$ref":"#/components/schemas/Indicator"}},"resources":{"type":"object","description":"The resource allocation details for the workload, including the current and recommended by PerfectScale allocation.","properties":{"current":{"$ref":"#/components/schemas/Resources"},"recommended":{"$ref":"#/components/schemas/Resources"}},"required":["current","recommended"]},"usage":{"type":"object","description":"An object representing the resource usage statistics of a container, broken down into CPU and memory consumption.","properties":{"cpuCores":{"$ref":"#/components/schemas/Percentiles"},"memoryMiB":{"$ref":"#/components/schemas/Percentiles"}},"required":["cpuCores","memoryMiB"]}},"required":["name","runningMinutes","indicators","resources","usage"]},"Resources":{"type":"object","description":"Metrics detailing the resource allocation in terms of memory and CPU.","properties":{"memoryRequestMiB":{"type":"number","format":"double","description":"The amount of memory requested for the workload in MiB."},"memoryLimitMiB":{"type":"number","format":"double","description":"The maximum amount of memory that can be allocated for the workload in MiB."},"cpuRequestCores":{"type":"number","format":"double","description":"The amount of CPU resources requested for the workload in cores."},"cpuLimitCores":{"type":"number","format":"double","description":"The maximum amount of CPU resources that can be allocated for the workload in cores."}},"required":["memoryRequestMiB","memoryLimitMiB","cpuRequestCores","cpuLimitCores"]},"Percentiles":{"type":"object","description":"An object representing percentile statistics of resource usage, providing insights into the distribution of usage values over a certain period.","properties":{"p90":{"type":"number","format":"double","description":"The 90th percentile value of resource usage, indicating that 90% of the usage values are below this value and 10% are above."},"p95":{"type":"number","format":"double","description":"The 95th percentile value of resource usage, indicating that 95% of the usage values are below this value and 5% are above."},"p100":{"type":"number","format":"double","description":"The 100th percentile value (or maximum value) of resource usage observed during the specified period."}},"required":["p90","p95","p100"]}}},"paths":{"/clusters/{cluster_uid}/workloads":{"get":{"tags":["Workloads"],"summary":"List all workloads in a specified cluster","parameters":[{"name":"cluster_uid","in":"path","required":true,"schema":{"type":"string"},"description":"The unique identifier for the cluster"},{"name":"period","in":"query","required":false,"schema":{"allOf":[{"$ref":"#/components/schemas/TimeWindow"}]},"description":"The period for which data is calculated"},{"name":"expandEphemeralWorkloads","in":"query","required":false,"schema":{"type":"boolean","default":false},"description":"By default, short-lived ephemeral workloads in large clusters are collapsed into a single aggregate row named \"Collapsed Ephemeral Pods\" (namespace \"EphemeralPods\", type \"-\") carrying their summed cost and no recommendations. The aggregate row has empty-string values for resilienceLevel, optimizationPolicy, cpuOptimizationPolicy, memoryOptimizationPolicy and optimizationPolicyTimeWindow, a single placeholder container named \"-\", and firstSeen/lastSeen spanning the collapsed workloads. Collapse eligibility is evaluated within the requested period's data window, so the same cluster may collapse for one period and not another. Set to true to return the full, uncollapsed workload list.\n"}],"responses":{"200":{"description":"A list of workloads","content":{"application/json":{"schema":{"type":"object","properties":{"meta":{"type":"object","properties":{"count":{"type":"integer","description":"The total number of workloads in the specified cluster."}},"required":["count"]},"data":{"type":"array","items":{"$ref":"#/components/schemas/Workload"}}},"required":["meta","data"]}}}},"404":{"description":"Cluster not found"},"500":{"description":"Internal server error"}}}}}}
```
### How to get cluster\_uid
There are two options available to obtain the **cluster\_uid**.
1. Get **cluster\_uid** directly from PerfectScale UI:
* Go to the **`Overview`** tab
* Click the three-dot button located next to the needed cluster
Get cluster UID
* Click the **`Copy Cluster UID`** button -> now, your cluster\_uid is copied.
Cluster UID copied
2. Get cluster\_uid with the following command:
```
kubectl get namespace kube-system -o=jsonpath='{.metadata.uid}'
```
3. Get the full list of clusters via [API](#clusters-endpoint) and extract cluster\_uids from it.
## Automation Audit Log Endpoint
The Automation Audit Log provides visibility into all actions performed by PerfectScale automation. This endpoint allows teams to programmatically access, retrieve, and filter audit logs and seamlessly integrate them into their existing systems or tools.
{% hint style="info" %}
The Automation Audit Log shows data from the last **30 days**.
{% endhint %}
## Get Automation Audit Logs with Cursor Pagination
> Retrieves a list of automation audit log entries within a specified time range, using cursor-based pagination.
```json
{"openapi":"3.0.3","info":{"title":"Perfectscale API - OpenAPI 3.0","version":"v1"},"servers":[{"url":"https://api.app.perfectscale.io/public/v1"}],"security":[{"auth":[]}],"components":{"securitySchemes":{"auth":{"type":"http","scheme":"bearer","bearerFormat":"JWT"}},"schemas":{"AutomationLogEntry":{"type":"object","properties":{"started_at":{"type":"string","format":"date-time","description":"Timestamp when the action started."},"cluster_name":{"type":"string","description":"Name of the cluster."},"cluster_uid":{"type":"string","description":"ID of the cluster."},"workload_id":{"type":"string","description":"ID of the workload."},"workload_name":{"type":"string","description":"Name of the workload."},"workload_type":{"type":"string","description":"Type of deployment."},"namespace":{"type":"string","description":"Namespace of the workload."},"labels":{"type":"object","additionalProperties":{"type":"string"},"description":"Labels associated with the workload."},"container":{"$ref":"#/components/schemas/AutomatedLogsContainer"},"executed":{"$ref":"#/components/schemas/ExecutionType"},"workload_indicators":{"type":"array","description":"Workload-level indicators addressed by the automation change.","items":{"$ref":"#/components/schemas/Indicator"}}},"required":["started_at","cluster_name","cluster_uid","workload_id","workload_name","workload_type","namespace","labels","container","executed"]},"AutomatedLogsContainer":{"type":"object","description":"Properties of the container.","properties":{"name":{"type":"string","description":"Name of the container."},"cpu":{"$ref":"#/components/schemas/AutomatedLogsContainerCpu"},"memory":{"$ref":"#/components/schemas/AutomatedLogsContainerMemory"},"indicators":{"type":"array","description":"Container-level indicators addressed by the automation change.","items":{"$ref":"#/components/schemas/Indicator"}}},"required":["name","cpu","memory"]},"AutomatedLogsContainerCpu":{"type":"object","description":"CPU related properties.","properties":{"cpuCoresRequest":{"type":"integer","format":"int64","description":"Request of cpu before apply of automation."},"recommendCpuCoresRequest":{"type":"integer","format":"int64","description":"Request of cpu after apply of automation."},"cpuCoresLimits":{"type":"integer","format":"int64","description":"Limit of cpu before apply of automation."},"recommendCpuCoresLimits":{"type":"integer","format":"int64","description":"Limit of cpu after apply of automation."},"cpuRequestImpact":{"type":"integer","format":"int64","description":"Impact of automation on cpu request value."},"cpuLimitImpact":{"type":"integer","format":"int64","description":"Impact of automation on cpu limit value."},"cpuRequestChangePercent":{"type":"number","format":"double","description":"Single instance cpu request change in %."},"cpuLimitChangePercent":{"type":"number","format":"double","description":"Single instance cpu limit change in %."},"cpuRequestChangeAbsolute":{"type":"integer","format":"int64","description":"Single instance cpu request absolute value change."},"cpuLimitChangeAbsolute":{"type":"integer","format":"int64","description":"Single instance cpu limit absolute value change."}},"required":["cpuCoresRequest","recommendCpuCoresRequest","cpuCoresLimits","recommendCpuCoresLimits","cpuRequestImpact","cpuLimitImpact","cpuRequestChangePercent","cpuLimitChangePercent","cpuRequestChangeAbsolute","cpuLimitChangeAbsolute"]},"AutomatedLogsContainerMemory":{"type":"object","description":"Memory related properties.","properties":{"memMiBRequest":{"type":"integer","format":"int64","description":"Request of memory before apply of automation."},"recommendMemMiBRequest":{"type":"integer","format":"int64","description":"Request of memory after apply of automation."},"memMiBLimits":{"type":"integer","format":"int64","description":"Limit of memory before apply of automation."},"recommendMemMiBLimits":{"type":"integer","format":"int64","description":"Limit of memory after apply of automation."},"memMiBRequestImpact":{"type":"integer","format":"int64","description":"Impact of automation on memory request value."},"memMiBLimitImpact":{"type":"integer","format":"int64","description":"Impact of automation on memory limit value."},"memRequestChangePercent":{"type":"number","format":"double","description":"Single instance memory request change in %."},"memLimitChangePercent":{"type":"number","format":"double","description":"Single instance memory limit change in %."},"memMiBRequestChangeAbsolute":{"type":"integer","format":"int64","description":"Single instance memory request absolute value change."},"memMiBLimitChangeAbsolute":{"type":"integer","format":"int64","description":"Single instance memory limit absolute value change."}},"required":["memMiBRequest","recommendMemMiBRequest","memMiBLimits","recommendMemMiBLimits","memMiBRequestImpact","memMiBLimitImpact","memRequestChangePercent","memLimitChangePercent","memMiBRequestChangeAbsolute","memMiBLimitChangeAbsolute"]},"Indicator":{"type":"object","description":"A risk/waste signal addressed by an automation change.","properties":{"name":{"type":"string","description":"Indicator name."},"type":{"type":"string","description":"Indicator type (e.g. risk, waste, none)."},"severity":{"type":"integer","description":"Severity level (0 = none, higher = more severe)."}},"required":["name","type","severity"]},"ExecutionType":{"type":"string","enum":["regular-eviction","inplace-resize","cleanup"]},"Pagination":{"type":"object","properties":{"has_next":{"type":"boolean","deprecated":true,"description":"Deprecated: derive from 'next' (true when 'next' is non-null)."},"next":{"type":"string","nullable":true,"description":"An opaque token to be used in the 'page_token' parameter to fetch the next page."},"has_prev":{"type":"boolean","deprecated":true,"description":"Deprecated: derive from 'prev' (true when 'prev' is non-null)."},"prev":{"type":"string","nullable":true,"description":"An opaque token to be used in the 'page_token' parameter to fetch the previous page."},"page_size":{"type":"integer","description":"The maximum number of items to return. Defaults to 1000. Must be between 1 and 5000."}},"required":["has_next","next","has_prev","prev","page_size"]}}},"paths":{"/automation/audit_logs":{"post":{"operationId":"AutomationAuditLogs","summary":"Get Automation Audit Logs with Cursor Pagination","description":"Retrieves a list of automation audit log entries within a specified time range, using cursor-based pagination.\n","requestBody":{"content":{"application/json":{"schema":{"type":"object","properties":{"from":{"type":"string","format":"date-time","description":"The start of the time range in ISO 8601 format with UTC timezone (e.g., \"2025-09-17T00:00:00Z\").\nIf omitted, defaults to 00:00:00 UTC of the day that is 30 days ago from today.\n\nConstraints:\n- Must be in UTC timezone (ends with 'Z' or has '+00:00' offset)\n- Cannot be in the future\n- Cannot be before 00:00:00 UTC of 30 days ago from today\n- Must be before 'to' date when both are provided\n- Must be different from 'to' date\n\nExample: If today is 2025-10-17, the earliest allowed value is \"2025-09-17T00:00:00Z\"\n"},"to":{"type":"string","format":"date-time","description":"The end of the time range in ISO 8601 format with UTC timezone (e.g., \"2025-10-17T23:59:59Z\").\nIf omitted, defaults to the current time in UTC.\n\nConstraints:\n- Must be in UTC timezone (ends with 'Z' or has '+00:00' offset)\n- Cannot be in the future\n- Must be after 'from' date when both are provided\n- Must be different from 'from' date\n"},"page_size":{"type":"integer","default":1000,"minimum":1,"maximum":5000,"description":"The maximum number of items to return. Defaults to 1000. Must be between 1 and 5000."},"page_token":{"type":"string","description":"An opaque token from the 'next' or 'prev' field in the response's pagination object, used to fetch the next/previous page. The token encodes both the boundary position and the page direction. Preferred over the deprecated 'after'/'before' fields."},"after":{"type":"string","deprecated":true,"description":"Deprecated: use 'page_token' instead. An opaque token from the 'next' field in the response's pagination object, used to fetch the next page."},"before":{"type":"string","deprecated":true,"description":"Deprecated: use 'page_token' instead. An opaque token from the 'prev' field in the response's pagination object, used to fetch the previous page."},"cluster_uids":{"type":"array","items":{"type":"string"},"description":"Filter by cluster UIDs. Accepts multiple cluster UIDs to filter the audit logs."},"namespaces":{"type":"array","items":{"type":"string"},"description":"Filter by Kubernetes namespaces. Accepts multiple namespaces to filter the audit logs."}}}}}},"responses":{"200":{"description":"OK","content":{"application/json":{"schema":{"type":"object","properties":{"data":{"type":"array","items":{"$ref":"#/components/schemas/AutomationLogEntry"}},"meta":{"type":"object","properties":{"pagination":{"$ref":"#/components/schemas/Pagination"}}}},"required":["data","meta"]}}}}}}}}}
```
## Deleting an API Token
In certain situations, you may find it necessary to delete an API token.
Follow these easy steps:
1. Click on the **`Profile`** button in the bottom left corner and select **`Organization Settings`**.
2. In the pop-up window, navigate to the **`API Tokens`** tab.
3. Click the hamburger on the right-hand side, select **`Delete API Token`**, and click the **`Delete`** button.
Delete API token
## Deleting a cluster
{% hint style="info" %}
The endpoint removes the cluster from PerfectScale without deleting the cluster itself.
{% endhint %}
There may be situations where you need to remove a cluster from PerfectScale. You can seamlessly delete a cluster using Public API:
## DELETE /clusters/{cluster\_uid}
>
```json
{"openapi":"3.0.3","info":{"title":"Perfectscale API - OpenAPI 3.0","version":"v1"},"servers":[{"url":"https://api.app.perfectscale.io/public/v1"}],"security":[{"auth":[]}],"components":{"securitySchemes":{"auth":{"type":"http","scheme":"bearer","bearerFormat":"JWT"}}},"paths":{"/clusters/{cluster_uid}":{"delete":{"operationId":"DeleteCluster","parameters":[{"name":"cluster_uid","in":"path","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"OK"},"404":{"description":"NotFound"}}}}}}
```
### How to get cluster UID
To get a **cluster UID** with a few clicks, navigate to the **`Overview`** tab, hover over the needed cluster, and click the **`three dots`** button. Then click **`Copy Cluster UID`** so that it will be automatically copied to the clipboard.
Get cluster UID
### How to list all cluster UIDs
You can seamlessly list all the clusters' UIDs of the tenant by using the following API:
{% openapi src="/files/EqHf8v7t6x1T7bT9LFNq" path="/clusters" method="get" %}
[openapi-final-2024-07-02.yaml](https://1573387604-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FABMqnYtsOO44JmQTVSnn%2Fuploads%2F6f3eZ5nGXcrp9tOMI7Li%2Fopenapi-final-2024-07-02.yaml?alt=media\&token=f29d2cea-6dd7-44c5-8e02-a8eb794fe493)
{% endopenapi %}
# PerfectScale trial
Learn more about PerfectScale trial
PerfectScale offers a **free**, **30-day** trial, providing you with valuable insights and actionable recommendations for eliminating waste, saving costs, and improving your environment's stability and resilience.
{% hint style="info" %}
The Trial officially begins only **when the first cluster is onboarded** and not when the account is created, but you'll need to upload your first cluster within a week.
{% endhint %}
## Trial Limitations
:bulb:**NOTE:** PerfectScale Trial does NOT include the following features, which are only available on the **`Expert Plan`**:
1. [MFA](/administration/security/mfa)
2. [SSO](/administration/security/sso)
3. [Automated Remediation](/enable-automation/introduction-to-automation) (limited to 1 cluster)
4. [Public API](/api/public-api)
If you want to evaluate these features as well, contact us at [sales@perfectscale.io](mailto:sales@perfectfectscale.io)
{% hint style="info" %}
Find the information regarding the available subscriptions [here](https://www.perfectscale.io/pricing).
{% endhint %}
## End of Trial
When your Trial is almost over, one of our team members will assess your monthly v**C**PU **C**onsumption **F**orecast (**CCF**) and provide further instructions based on the available [Plans](https://www.perfectscale.io/pricing).
{% hint style="info" %}
[Learn more](/perfectscale-trial/perfectscale-trial/how-to-find-your-allocated-vcpu) about how to retrieve the vCPU consumption.
{% endhint %}
PerfectScale will notify users that the Trial is nearing its end.
#### CCF < 300 vCPU per month
Once the Trial is completed and assuming the monthly CCF is less than 300 vCPU, the account will be moved to the Free Community plan, and the service will continue uninterrupted.
#### CCF > 300 vCPU per month
If the monthly CCF exceeds 300 vCPU, one of our team members will remind the account owner that the Trial is nearing its end, and since your CCF is bigger than the free community package allowance (> 300 vCPU per month), your account will be restricted once the Trial ends unless you purchase a license.
{% hint style="warning" %}
The data collected during the Trial will be stored for 30 days. After that, the trial data will be deleted for the accounts that haven't migrated to the available [Plan](https://www.perfectscale.io/pricing). To reactivate and regain access to the account, contact .
{% endhint %}
# How to find your allocated vCPU?
Learn how to retrieve the number of CPU Units per Kubernetes node to find your perfect license
To provide a more accurate cost metric and improve pricing transparency, the PerfectScale solution is licensed by the monthly vCPU consumption of your clusters.
{% hint style="info" %}
If your usage remains under 300 vCPU per month during your trial, you will be automatically converted to the free community package. Learn more about the Pricing model [here](https://www.perfectscale.io/pricing).
{% endhint %}
## Determine vCPU consumption
There are a few ways to determine your vCPU consumption:
[In the PerfectScale platform](#in-app)
[In your observability tool](#observability-tools)
### **In-app**
PerfectScale by DoiT has built a valuable feature that provides both the team and customers seamless access to subscription details, including visibility into normalized monthly vCPU consumption right within the platform.
To access your license details, go to **Subscription Details**, located under the account button at the bottom left of the screen.
Subscription details
To view more details and track monthly vCPU consumption over time, simply click **View over-time vCPU consumption report** to access granular data in a trend report.
Monthly vCPU trend
### **Observability tools**
With your observability tool, like Grafana and DataDog you can either use some existing dashboard or create a new graph summarizing cluster capacity. \
\
For example, Prometheus can be used to create a visualization of cluster capacity, as shown below:
```promql
SUM(kube_node_status_capacity{resource="cpu", unit="core"})
```
vCPU/hours in the observability tool
In the above example, the allocated vCPU is around 90 cores. We can estimate monthly core hours as `90*24*30 = 64800`
Use the terminal following the guide below:
* Open a new terminal on the machine that has `kubectl` been installed. Make sure you are running in the correct context (=cluster)
* Type the following command on the prompt:
```yaml
kubectl get nodes -o custom-columns=NAME:'{.metadata.name}',CORES:'{.status.capacity.cpu}'
```
This will show you the currently running instances and their core capacity at a single point in time:
```
NAME CORES
ip-10-107-0-11.eu-central-1.compute.internal 2
ip-10-107-12-213.eu-central-1.compute.internal 2
ip-10-107-24-124.eu-central-1.compute.internal 8
ip-10-107-24-142.eu-central-1.compute.internal 2
ip-10-107-26-193.eu-central-1.compute.internal 2
ip-10-107-31-30.eu-central-1.compute.internal 8
ip-10-107-32-253.eu-central-1.compute.internal 4
ip-10-107-32-47.eu-central-1.compute.internal 4
ip-10-107-37-121.eu-central-1.compute.internal 2
ip-10-107-4-114.eu-central-1.compute.internal 2
ip-10-107-40-28.eu-central-1.compute.internal 4
ip-10-107-45-210.eu-central-1.compute.internal 2
ip-10-107-47-136.eu-central-1.compute.internal 2
ip-10-107-47-184.eu-central-1.compute.internal 4
ip-10-107-7-175.eu-central-1.compute.internal 4
ip-10-107-7-204.eu-central-1.compute.internal 2
```
* Now you can summarize the cores to have cluster cores ***`right now`***. You can multiply this number by 24\*30 to have a monthly estimate.
{% hint style="warning" %}
If your cluster has a dynamic nature, this approach may be less accurate than Observability tools or PerfectScale.
{% endhint %}
# Help PerfectScale to improve
Your hones feedback - our best driver!
Our customers are our main inspiration. Your feedback is helping us continuously improve our capabilities and deepen the value of our solution.
Do you have an idea on how to make your experience with PerfectScale better? [**Let us know**](https://perfectscale.atlassian.net/servicedesk/customer/portals)!
# Kubernetes Optimization
Learn how to effortlessly scale and right-size Kubernetes with granular control that ensures peak performance at the lowest possible cost
Kubernetes optimization tool - **PerfectScale by DoiT** empowers you to effortlessly fine-tune your environment, ensure optimal resource allocation, and achieve peak performance with comprehensive insights for the lowest possible cloud cost.
This documentation serves as your comprehensive guide to mastering the platform, ensuring you achieve maximum results efficiently.
{% hint style="success" %}
PerfectScale is compatible with and can be deployed in **any Kubernetes environment**, including **EKS/EKS Anywhere**, **GKE**, **AKS**, **KOPS**, **private cloud**, and other Kubernetes distributions.
{% endhint %}
## Getting started with the self-hosted K8s optimization tool
1. [Provision a self-hosted environment.](/2.0-self-hosted-or-perfectscale-documentation/getting-started/provisioner-installation)
2. [Onboard a K8s cluster](/2.0-self-hosted-or-perfectscale-documentation/getting-started/how-to-onboard-a-cluster).
3. [Get a thorough analysis of risk and cost](/2.0-self-hosted-or-perfectscale-documentation/visibility-and-optimization/podfit-or-vertical-pod-right-sizing), and start leveraging a full optimization suite that is living within your environment.
4. Facilitate a productive collaboration by [inviting your team](/2.0-self-hosted-or-perfectscale-documentation/administration/user-management).
## Next to achieve
### [Cut your K8s cloud spending by up to 50%](/2.0-self-hosted-or-perfectscale-documentation/visibility-and-optimization/podfit-or-vertical-pod-right-sizing)
Get a detailed multi-cloud, multi-cluster view of your K8s spending based on actual resource utilization, pinpoint problematic areas, prioritize the most impactful actions, and apply data-driven recommendations to reduce waste safely without affecting stability and performance.
### [Improve K8s performance](/2.0-self-hosted-or-perfectscale-documentation/visibility-and-optimization/podfit-or-vertical-pod-right-sizing/understanding-at-risk-indicators)
Identify up to 30 different types of resilience risks, prioritize them by their impact on your services, and apply actionable recommendations to resolve them in just a few clicks.
### [Automate K8s optimization tasks](/2.0-self-hosted-or-perfectscale-documentation/enable-automation/introduction-to-automation)
Effortlessly and proactively optimize your Kubernetes environment by adjusting workloads' CPU and memory to achieve optimal values and ensure your environment is always perfectly tuned without manual intervention.
### [Get accurate cost visibility across your entire K8s stack](/2.0-self-hosted-or-perfectscale-documentation/visibility-and-optimization/trends-monitoring)
Analyze your K8s data across clusters, namespaces, and node groups to evaluate effectiveness, pinpoint anomalies, improve forecasting, and enable data-driven decisions based on long-term trends.
## Maximizing your optimization results
Create a tailored optimization ecosystem with diverse integration options to keep up with all your Kubernetes activities, streamline your optimization processes, and enhance your team's productivity.
1. [**Configure Alerts**](/2.0-self-hosted-or-perfectscale-documentation/customize-workflow/alerting/resiliency-alerts)**:** Receive real-time notifications about resilience risks or cost anomalies before they affect the performance or effectiveness of your environment.
2. [**Set Up Communication Channels**](/2.0-self-hosted-or-perfectscale-documentation/customize-workflow/communication-and-messaging): Manage alerts via Slack, MS Teams, or Datadog.
3. [**Connect Issue Management**](/2.0-self-hosted-or-perfectscale-documentation/customize-workflow/ticketing-and-bug-tracking): Integrate with Jira to handle issues efficiently within clusters.
4. [**Customize Pricing**](/2.0-self-hosted-or-perfectscale-documentation/customize-workflow/pricing): Tailor pricing to your needs with various pricing customization options
5. [**Integrate Observability**](/2.0-self-hosted-or-perfectscale-documentation/customize-workflow/observability): Enhance K8s visibility with tools like Grafana, Datadog, and more.
## When a self-hosted solution makes the most sense
While PerfectScale's SaaS Kubernetes optimization platform works seamlessly for many teams, some organizations may require more control over their environment. In some situations, using the self-hosted solution makes a lot of sense to meet specific infrastructure requirements, whether driven by security and compliance needs or a desire for greater control and independence.
Here are some key considerations to help you decide when a self-hosted platform might be the best option for you.
#### Security and Compliance
A self-hosted PerfectScale platform provides you with full control over your data, making it easier to meet strict security, compliance, and residency requirements. It allows you to implement custom protection measures, avoid shared environments, and easily align with the security standards.
#### Operational independence and availability
A self-hosted PerfectScale platform ensures offline capability, allowing it to run in environments with limited internet access and restricted or air-gapped environments, giving you full control over where your data is stored and keeping it within required geographic boundaries.
#### Perception and enterprise readiness
A self-hosted PerfectScale platform is commonly chosen by large organizations because it meets their needs for self-hosted deployments of core services, as well as allowing for faster procurement approvals in highly regulated industries such as finance, defense, and healthcare.
## :handshake: Community
Join our [Сommunity](https://join.slack.com/t/perfectscalecommunity/shared_invite/zt-1tu9teu9e-Z9tGt4LpNI8tUC3j8obcmQ) to stay updated on what is trending in the atmosphere, our latest releases, upcoming events, technical discussions, and more.
## :woman\_superhero: Support
Feel free to ask any questions on [Slack](https://join.slack.com/t/perfectscalecommunity/shared_invite/zt-1tu9teu9e-Z9tGt4LpNI8tUC3j8obcmQ). You can also reach our support at .
---
[Next Page](/llms-full.txt/1)
