> For the complete documentation index, see [llms.txt](https://docs.perfectscale.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.perfectscale.io/enable-automation/hpa-automation.md).

# HPA automation

Keep HPA settings efficient with automated minReplicas and trigger optimization

{% hint style="info" %}
Autoscaler version **v1.0.71 or later** and PerfectScale exporter version **v1.1.28 or later** are required.
{% endhint %}

PerfectScale analyzes every HorizontalPodAutoscaler (HPA) in your cluster and recommends changes to `minReplicas` and to the CPU and memory triggers (`averageUtilization`). With HPA automation enabled, PerfectScale applies these recommendations for you.

{% hint style="success" %}
Recommendations are available for every HPA without any setup. Automation is opt-in and is configured with the Automation CR.
{% endhint %}

## HPA recommendations

HPA recommendations are shown in the **HPA** tab of PodFit. A recommended value appears as a badge next to the current one.

<figure><img src="https://1573387604-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FABMqnYtsOO44JmQTVSnn%2Fuploads%2F70L8sBB0fj59cJmjBuL8%2Fimage.png?alt=media&amp;token=d87fe597-3d55-4f44-aee9-01967f802e8e" alt=""><figcaption><p>HPA tab in PodFit</p></figcaption></figure>

Hover over the badge to see the reason for the recommendation.

<figure><img src="https://1573387604-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FABMqnYtsOO44JmQTVSnn%2Fuploads%2FQya3wfxbGMEIrLBOZR8J%2Fpodfit-mem-trigger-tooltip-with-reason-preview.png?alt=media&amp;token=107fc059-986e-42ce-8fc7-3e05d6caa95f" alt=""><figcaption><p>Recommendation reason</p></figcaption></figure>

To see the details for a workload, open it and click **Review Recommendations** in the Replicas widget.

<figure><img src="https://1573387604-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FABMqnYtsOO44JmQTVSnn%2Fuploads%2FCXmOr62UdRDYNVoUSwtv%2Fzoomin-panel-cpu-throttling.png?alt=media&amp;token=7bbbadba-38ee-4fd8-8a27-dbedb941f1a7" alt=""><figcaption><p>HPA Triggers panel</p></figcaption></figure>

Each recommended change is shown as YAML that you can copy, together with the reason for it.

<figure><img src="https://1573387604-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FABMqnYtsOO44JmQTVSnn%2Fuploads%2FypPOL82XVRI3jfDWbfYS%2Fimage%20(2).png?alt=media&amp;token=7bdcc371-374a-4ae4-94d8-2252e3beb3ef" alt=""><figcaption><p>HPA Recommendations dialog</p></figcaption></figure>

<table><thead><tr><th width="230">What PerfectScale detects</th><th>Recommendation</th></tr></thead><tbody><tr><td><strong><code>minReplicas</code> is too high</strong></td><td>Lower <code>minReplicas</code> to <strong>2</strong>, or to the value the PodDisruptionBudget (PDB) needs when it is higher. PerfectScale never raises <code>minReplicas</code>, so an existing value of 1 is kept.</td></tr><tr><td><strong><code>minReplicas</code> equals <code>maxReplicas</code></strong></td><td>The HPA cannot scale. If <code>maxReplicas</code> is above the recommended value, lower <code>minReplicas</code>. Otherwise, the finding is reported only: raise <code>maxReplicas</code> yourself.</td></tr><tr><td><strong>Trigger is too low</strong></td><td>A trigger below <strong>80%</strong> inflates resource requests. Raise it to 80%.</td></tr><tr><td><strong>Trigger is unreachable</strong></td><td>The trigger is above the highest usable trigger, so the container is throttled or OOM-killed before the HPA scales. Lower it to the highest usable trigger. Without a limit, a trigger above <strong>150%</strong> is reported only.</td></tr><tr><td><strong>Pod-level trigger on a multi-container pod</strong></td><td>A sidecar can hide the real load of the application. Move the trigger to the application container.</td></tr><tr><td><strong>Stays at <code>maxReplicas</code> with a slow scale-down</strong></td><td>Reported only. Review <code>maxReplicas</code> and the scale-down <code>behavior</code>.</td></tr></tbody></table>

{% hint style="info" %}
PerfectScale never changes `maxReplicas` or the scaling `behavior`. Only CPU and memory triggers of the `Utilization` type are analyzed.
{% endhint %}

### Highest usable trigger

A trigger is a percentage of the request, and usage cannot exceed the limit. Kubernetes scales only when usage is more than 10% above the trigger, so the highest trigger that can still fire is:

* **CPU:** `limit / request * 100 / 1.1`
* **Memory:** `0.9 * limit / request * 100 / 1.1` (the pod is OOM-killed at the limit, so only 90% of it is reachable)

For example, with a 1 core request and a 1.8 core limit, the highest usable CPU trigger is 163%.

{% hint style="info" %}
The calculation uses the requests and limits PerfectScale recommends for the workload, not only the current ones. Learn more about [combining vertical and horizontal optimization](https://docs.perfectscale.io/visibility-and-optimization/combining-vertical-podfit-and-horizontal-hpa-optimization.md).
{% endhint %}

### PodDisruptionBudget

The PDB floor is the `desiredHealthy` of the PDB plus one, so a disruption is still allowed. It is used only when exactly one PDB covers the workload and nothing else. When several PDBs match, or a PDB spans other workloads, the automation does not change `minReplicas`.

## Prerequisites

1. The [PerfectScale Automation Agent](https://docs.perfectscale.io/enable-automation/automation-setup-instructions.md) version **v1.0.71 or later** is installed in the cluster.
2. [Pod automation](https://docs.perfectscale.io/enable-automation/configuring-automation-with-crd.md) is enabled for the workload. Triggers depend on resource requests, so PerfectScale changes an HPA only while the workload runs on a revision created by PerfectScale Automation.

{% hint style="info" %}
To apply the changes, the agent patches `horizontalpodautoscalers` and KEDA `scaledobjects`, and registers an admission webhook for them.
{% endhint %}

## Enabling HPA automation

Add the `hpa` block to the Automation Configuration CR:

```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
  name: cluster-automation-config
spec:
  automation:
    hpa:
      operational:
        automationMode: Enabled # default Disabled
```

{% hint style="success" %}
HPA automation can be configured at the **cluster**, **namespace**, or **workload** level. The most specific level takes priority.
{% endhint %}

Once enabled, **HPA** is listed as **Active** in the **Automation** menu in the header.

<figure><img src="https://1573387604-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FABMqnYtsOO44JmQTVSnn%2Fuploads%2FkxHEQ3Y9oKCWa8LnAbXt%2F05-header-automation-menu-pod-and-hpa.png?alt=media&amp;token=a72ca1f8-fec2-4fbc-8a42-c4c23f161a13" alt=""><figcaption><p>Automation menu</p></figcaption></figure>

The same exclusions as for pod automation apply: the namespaces excluded in the Helm chart, and `kube-system` unless it has its own `NamespaceAutomationConfig`. To limit HPA automation to specific workloads, use `workloadLabelSelectors` in the `hpa` block of the cluster or namespace config:

```yaml
spec:
  automation:
    hpa:
      operational:
        automationMode: Enabled
        workloadLabelSelectors:
          - key: team
            value: payments
            allowAutomation: true
```

## Customizing HPA automation

Each group of values has its own switch and range:

```yaml
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig #, NamespaceAutomationConfig, or WorkloadAutomationConfig
metadata:
  name: example
spec:
  automation:
    hpa:
      operational:
        automationMode: Enabled
        restrictions:
          minReplicas:
            enabled: true # default true
            min: 2        # default 2, the lowest minReplicas to write
            max: 5        # default unset, the highest minReplicas to write
          cpuTrigger:
            enabled: true # default true
            min: 80       # default 80, the lowest trigger to write
            max: 150      # default unset, the highest trigger to write
          memoryTrigger:
            enabled: true # default true
            min: 80       # default 80
            max: 150      # default unset
```

<table><thead><tr><th width="200">Setting</th><th>Description</th></tr></thead><tbody><tr><td><strong><code>enabled</code></strong></td><td>Set to <code>false</code> to keep PerfectScale from changing this value.</td></tr><tr><td><strong><code>minReplicas.min</code></strong></td><td>The lowest <code>minReplicas</code> PerfectScale writes. <code>minReplicas</code> is lowered to 2, or to this value when it is higher.</td></tr><tr><td><strong><code>minReplicas.max</code></strong></td><td>The highest <code>minReplicas</code> PerfectScale writes. When the PDB needs more replicas than this value, <code>minReplicas</code> is not changed.</td></tr><tr><td><strong><code>cpuTrigger.min</code></strong><br><strong><code>memoryTrigger.min</code></strong></td><td>The lowest trigger PerfectScale writes. A low trigger is raised to 80%, or to this value when it is higher. When the highest usable trigger is below this value, the trigger is not changed.</td></tr><tr><td><strong><code>cpuTrigger.max</code></strong><br><strong><code>memoryTrigger.max</code></strong></td><td>The highest trigger PerfectScale writes.</td></tr></tbody></table>

{% hint style="info" %}
`min` and `max` must be at least 1, and `min` cannot be greater than `max`.
{% endhint %}

### Container-level triggers

In `WorkloadAutomationConfig`, the trigger ranges can be set per container. A field set for the container replaces the workload value, and a field left out is inherited. `minReplicas` is always workload-wide.

```yaml
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: workload-automation-config
  namespace: your-namespace
spec:
  targetRef:
    kind: Deployment
    name: your-deployment
  automation:
    hpa:
      operational:
        automationMode: Enabled
    containers:
      app:
        hpa:
          operational:
            restrictions:
              cpuTrigger:
                min: 85 # a low CPU trigger of this container is raised to 85%
      log-shipper: # a sidecar declared in the workload spec
        hpa:
          operational:
            restrictions:
              cpuTrigger:
                enabled: false
              memoryTrigger:
                enabled: false
```

## How changes are applied

The agent checks for new recommendations every 5 minutes and applies them within the [maintenance window](https://docs.perfectscale.io/enable-automation/configuring-automation-with-crd/automation-customization-with-crd.md#maintenance-window) of the workload.

The whole workload is skipped when:

* The workload does not run on a revision created by PerfectScale Automation.
* More than one HPA targets the workload.

A single change is skipped when:

* The workload spec has changed since the recommendation was calculated (trigger changes only).
* The value in the HPA has changed since the recommendation was calculated.
* The HPA has several triggers for the same resource.

Each change that is applied, limited by your CR, or skipped is recorded as a Kubernetes event on the HPA: `HpaAutomationApplied`, `HpaAutomationLimitedByRule`, or `HpaAutomationSkipped`. A workload that is skipped as a whole gets no event.

```bash
kubectl describe hpa <hpa-name> -n <namespace>
```

### Moving a pod-level trigger to a container

PerfectScale replaces a pod-level trigger with a container-level one when:

* The cluster runs Kubernetes **1.33 or later**.
* Exactly one container is a candidate. Containers with both triggers disabled in the CR are not candidates. If several remain, the `kubectl.kubernetes.io/default-container` annotation selects the container.
* Every running pod has this container with a resource request.

Otherwise, the pod-level trigger is kept.

{% hint style="info" %}
Init containers and sidecars injected by an admission webhook, such as `istio-proxy`, are ignored automatically. Only sidecars declared in the workload spec need their triggers disabled in the CR.
{% endhint %}

### KEDA

When the HPA is generated by KEDA, PerfectScale patches the `ScaledObject` instead: `minReplicaCount` and the `cpu` and `memory` triggers with `metricType: Utilization`. `minReplicaCount` is never set below `idleReplicaCount` + 1. The events are recorded on the `ScaledObject`.

### Keeping the applied values

The admission webhook keeps the values PerfectScale applied. When another tool or user changes them, for example on a GitOps sync, the applied value is restored in the same request and `kubectl` prints a warning.

{% hint style="warning" %}
With HPA automation enabled, the HPA in the cluster no longer matches the manifest in Git. To take a value back, set `enabled: false` for it in the CR or exclude the workload from HPA automation.
{% endhint %}

## Stopping HPA automation

To stop HPA automation for the whole cluster while pod automation continues, add the following to the `ClusterAutomationConfig`:

```yaml
spec:
  automation:
    hpa:
      operational:
        stopAllAutomation: true
```

{% hint style="info" %}
`spec.automation.operational.stopAllAutomation` stops both pod and HPA automation. In both cases, the values already applied to the HPA stay as they are, and the webhook stops restoring them.
{% endhint %}
