> For the complete documentation index, see [llms.txt](https://docs.perfectscale.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.perfectscale.io/enable-automation/karpenter-automation.md).

# Karpenter automation

Let PerfectScale keep Karpenter consolidation moving by automatically adjusting NodePool disruption budgets

Karpenter automation applies PerfectScale's Karpenter recommendations to your NodePools for you. You choose which NodePools are automated and which settings PerfectScale may change; the PerfectScale automation agent then patches the NodePool, keeps a record of your original values, and can restore them at any time.

The first automated recommendation unblocks consolidation when nodes stuck in deletion hold disruption budget slots.

## The problem: stuck nodes hold budget slots

Karpenter counts every node that is being deleted against the NodePool's disruption budgets until its instance terminates, whatever the reason for the deletion. A node whose drain is blocked, for example by a PodDisruptionBudget or a pod annotated with `karpenter.sh/do-not-disrupt`, can stay in that state indefinitely.

While it does, it occupies a budget slot. In a NodePool with the common `nodes: "1"` budget, a single stuck node uses the only slot, so Karpenter stops consolidating the whole pool and underutilized nodes keep running.

Stuck nodes usually come from forced deletions, such as node expiry, spot interruption, or a manual delete. Regular consolidation checks that a node can be drained before disrupting it and rarely produces them.

## How PerfectScale resolves it

PerfectScale continuously monitors automated NodePools for nodes that stay in deletion because Karpenter cannot drain them.

When such nodes hold the budget slots of a NodePool, PerfectScale temporarily raises its consolidation budgets, just enough for Karpenter to continue consolidating at the pace you configured. When the stuck nodes are gone, PerfectScale lowers the budgets back to your original values.

**Example.** A NodePool allows one node to be disrupted at a time, and one node is stuck in deletion:

```yaml
spec:
  disruption:
    budgets:
      - nodes: "1"
```

Consolidation is blocked for the whole pool. PerfectScale raises the budget so that Karpenter can consolidate again, and returns it to `nodes: "1"` once the stuck node terminates.

### Safeguards

* Only budgets that apply to consolidation are adjusted. Budgets limited to other disruption reasons, such as `Drifted`, are left as is.
* A zero budget is a deliberate block and is never raised.
* Only the `nodes` value changes. The `reasons`, `schedule` and `duration` of a budget are never modified.
* A budget is never lowered below your original value.
* If cluster data is missing or outdated, no change is made in either direction.

{% hint style="info" %}
The adjustment is not immediate. Detection relies on periodically collected cluster data, and every change respects the [maintenance window and cooldown](#time-constraints) you configure.
{% endhint %}

## Prerequisites

* The [PerfectScale automation agent](https://docs.perfectscale.io/enable-automation/automation-setup-instructions.md) (psc-autoscaler) is installed in the cluster.
* Karpenter NodePools use the `karpenter.sh/v1` API.

{% hint style="info" %}
Read-only NodePools managed by the cloud provider, such as the built-in `system` and `general-purpose` NodePools of EKS Auto Mode, are never automated.
{% endhint %}

## Enable Karpenter automation

Karpenter automation is configured with two cluster-scoped custom resources:

<table><thead><tr><th width="330">Resource</th><th>Purpose</th></tr></thead><tbody><tr><td><code>ClusterNodeGroupAutomationConfig</code></td><td>Settings for all NodePools in the cluster. Only one is allowed per cluster.</td></tr><tr><td><code>NodeGroupAutomationConfig</code></td><td>Settings for a single NodePool, overriding the cluster configuration. Only one is allowed per NodePool.</td></tr></tbody></table>

Automation is disabled until a configuration sets `automationMode: Enabled`, and budgets are changed only when the `disruptionBudgets` restriction is enabled.

### Enable for the whole cluster

```yaml
apiVersion: perfectscale.io/v1
kind: ClusterNodeGroupAutomationConfig
metadata:
  name: cluster-nodegroup-automation
spec:
  automation:
    karpenter:
      operational:
        automationMode: Enabled
        restrictions:
          disruptionBudgets:
            enabled: true   # allow PerfectScale to adjust spec.disruption.budgets
```

```bash
kubectl apply -f cluster-nodegroup-automation.yaml
```

### Enable for selected NodePools only

Keep automation disabled at the cluster level and enable it for individual NodePools:

```yaml
apiVersion: perfectscale.io/v1
kind: ClusterNodeGroupAutomationConfig
metadata:
  name: cluster-nodegroup-automation
spec:
  automation:
    karpenter:
      operational:
        automationMode: Disabled
---
apiVersion: perfectscale.io/v1
kind: NodeGroupAutomationConfig
metadata:
  name: general-purpose
spec:
  targetRef:
    kind: NodePool
    name: general-purpose   # NodePool name
  automation:
    karpenter:
      operational:
        automationMode: Enabled
        restrictions:
          disruptionBudgets:
            enabled: true
```

{% hint style="info" %}
A `ClusterNodeGroupAutomationConfig` must exist for any NodePool to be automated, even when automation is enabled per NodePool.
{% endhint %}

The same way, you can enable automation for the cluster and opt a NodePool out with a `NodeGroupAutomationConfig` that sets `automationMode: Disabled`.

### Select NodePools by label

Label selectors include or exclude NodePools by the labels on the NodePool object:

```yaml
apiVersion: perfectscale.io/v1
kind: ClusterNodeGroupAutomationConfig
metadata:
  name: cluster-nodegroup-automation
spec:
  automation:
    karpenter:
      operational:
        automationMode: Enabled
        labelSelectors:
          - key: nodepool-automation
            value: enabled
            allowAutomation: true    # only NodePools with this label are automated
          - key: environment
            value: production
            allowAutomation: false   # NodePools with this label are never automated
        restrictions:
          disruptionBudgets:
            enabled: true
```

* A NodePool matching a selector with `allowAutomation: false` is excluded.
* If at least one selector has `allowAutomation: true`, only NodePools matching such a selector are automated.
* Label selectors do not apply to a NodePool that has its own `NodeGroupAutomationConfig`.

## Customize automation

### Time constraints

```yaml
apiVersion: perfectscale.io/v1
kind: ClusterNodeGroupAutomationConfig   # or NodeGroupAutomationConfig
metadata:
  name: cluster-nodegroup-automation
spec:
  automation:
    karpenter:
      operational:
        automationMode: Enabled
        timeConstraints:
          consolidationMaxAutomationFrequency: "24h"   # default "30m"
          maintenanceWindow:   # UTC
            monday: ["02:00-05:00"]
            tuesday: ["02:00-05:00"]
        restrictions:
          disruptionBudgets:
            enabled: true
```

<table><thead><tr><th width="330">Setting</th><th>Description</th></tr></thead><tbody><tr><td><code>consolidationMaxAutomationFrequency</code></td><td>Minimum time between two changes of the same NodePool. Default: <code>30m</code>.</td></tr><tr><td><code>maintenanceWindow</code></td><td>Time frames, in UTC, when PerfectScale may change NodePools. Uses the same format as the <a href="https://docs.perfectscale.io/enable-automation/configuring-automation-with-crd/automation-customization-with-crd.md#maintenance-window">workload maintenance window</a>. Days that are not listed default to <code>00:00-23:59</code>.</td></tr></tbody></table>

{% hint style="warning" %}
Time constraints apply to lowering a budget as well as raising it. A narrow maintenance window or a long cooldown keeps a raised budget in place for longer after the stuck nodes are gone.
{% endhint %}

### Number of NodePools changed at once

By default, PerfectScale changes one NodePool at a time. To change this, set `maxConcurrentApplies` in the cluster configuration:

```yaml
spec:
  automation:
    operational:
      maxConcurrentApplies: 1   # default
```

### How settings are merged

Settings can be placed at four levels. Each level overrides the previous one:

1. `ClusterNodeGroupAutomationConfig`: `spec.automation.operational`
2. `ClusterNodeGroupAutomationConfig`: `spec.automation.karpenter.operational`
3. `NodeGroupAutomationConfig`: `spec.automation.operational`
4. `NodeGroupAutomationConfig`: `spec.automation.karpenter.operational`

`timeConstraints` and `labelSelectors` can be set at any level. The maintenance window is overridden per day. `automationMode` and `restrictions` are set at the `karpenter.operational` levels; `restrictions` set for a NodePool replace the cluster's.

`stopAllAutomation`, `cleanUpAutomation` and `maxConcurrentApplies` can be set in the `ClusterNodeGroupAutomationConfig` only.

## Automated fields are owned by PerfectScale

Once PerfectScale changes `spec.disruption.budgets` of a NodePool, it owns that field while automation is enabled for the pool, including after the budgets return to your original values. If a user or a GitOps controller writes a different value, the automation agent restores the automated value and the writer gets a warning:

```
Warning: this NodePool is automated by PerfectScale, disable automation to change it
```

All other NodePool fields remain yours to change.

To change the budgets of an automated NodePool yourself, do one of the following first:

* disable the restriction: set `restrictions.disruptionBudgets.enabled: false`, or
* disable automation for the NodePool: set `automationMode: Disabled` in its `NodeGroupAutomationConfig`.

{% hint style="info" %}
The webhook that restores automated values never blocks your writes. If the automation agent is unavailable, NodePool updates go through unchanged.
{% endhint %}

### GitOps

When a GitOps controller syncs an automated NodePool, the automation agent keeps the automated budgets in place, so the value in Git does not take effect. The controller still sees the adjusted budgets as drift: the application is reported as out of sync and, with self-healing enabled, is synced again and again.

To avoid this, exclude the field from the comparison. With ArgoCD, add the following to the Application that manages your NodePools:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  ignoreDifferences:
    - group: karpenter.sh
      kind: NodePool
      jsonPointers:
        - /spec/disruption/budgets
```

## Monitor Karpenter automation

### In PerfectScale

* **Automation status** shows whether Karpenter automation is enabled for the cluster.
* **NodeFit** shows the automation state of each node group in the [node group table](https://docs.perfectscale.io/visibility-and-optimization/nodefit-node-right-sizing/node-group-view.md).
* [**Automation audit**](https://docs.perfectscale.io/enable-automation/automation-audit.md) lists every change applied to a NodePool, with the values before and after.

### In the cluster

PerfectScale marks every NodePool it patched with an annotation holding the time of the last change:

```yaml
metadata:
  annotations:
    automation.perfectscale.io/patchedAt: "2026-09-11T02:01:05Z"
```

For each automated NodePool, the agent keeps a `KarpenterAutomationStatus` resource with the same name as the NodePool:

```bash
kubectl get kastatus
kubectl get kastatus general-purpose -o yaml
```

<table><thead><tr><th width="330">Field</th><th>Description</th></tr></thead><tbody><tr><td><code>status.original</code></td><td>Your value of each field before PerfectScale first changed it.</td></tr><tr><td><code>status.target</code></td><td>The change PerfectScale currently recommends, if any.</td></tr><tr><td><code>status.current</code></td><td>The last change applied, with the values before and after.</td></tr><tr><td><code>status.automationAppliedAt</code></td><td>Time of the last change. The cooldown counts from it.</td></tr></tbody></table>

{% hint style="warning" %}
Do not delete `KarpenterAutomationStatus` resources. They are the only record of your original values and are required to revert the changes.
{% endhint %}

To list the configuration resources:

```bash
kubectl get cngaconf   # ClusterNodeGroupAutomationConfig
kubectl get ngaconf    # NodeGroupAutomationConfig
```

## Stop or revert automation

Both switches are set in the `ClusterNodeGroupAutomationConfig`, either under `spec.automation.operational` or under `spec.automation.karpenter.operational`.

### Stop

`stopAllAutomation: true` pauses Karpenter automation in the cluster. No further changes are made and PerfectScale stops restoring automated fields. Values already applied stay as they are.

```yaml
spec:
  automation:
    karpenter:
      operational:
        stopAllAutomation: true
```

### Revert

`cleanUpAutomation: true` stops automation and restores every automated NodePool to your original values. Time constraints and `maxConcurrentApplies` do not apply to a revert. The `patchedAt` annotation and the `KarpenterAutomationStatus` resources are removed.

```yaml
spec:
  automation:
    karpenter:
      operational:
        cleanUpAutomation: true
```

To start automation again, set `cleanUpAutomation` back to `false`.

{% hint style="info" %}
Disabling automation for a single NodePool with `automationMode: Disabled` stops further changes but does not restore the original values. Use `cleanUpAutomation` to revert.
{% endhint %}

## Limitations

* Only the `karpenter.sh/v1` NodePool API is supported.
* While a budget is raised, Karpenter may disrupt more nodes at once than your original budget allows for a short time after the stuck nodes terminate, until PerfectScale lowers the budget back.
* The budgets of an automated NodePool cannot be edited manually while automation is enabled for it.
