> For the complete documentation index, see [llms.txt](https://docs.perfectscale.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.perfectscale.io/enable-automation/bin-packing-automation.md).

# Bin packing automation

Remove underutilized nodes that your autoscaler cannot remove, safely and without manual drains

{% hint style="info" %}
Autoscaler version **v1.0.77** or later is required.
{% endhint %}

Workload right-sizing frees capacity on your nodes, but the cloud bill only drops when a node is removed. Karpenter and Cluster Autoscaler leave a node alone when it runs a pod they must not move, such as a pod with a `do-not-disrupt` annotation, a pod with no owner, or a pod whose PodDisruptionBudget has no room. One such pod can keep a nearly empty node running for months.

PerfectScale finds these blocked nodes, checks that every pod fits on other nodes, moves the pods, and hands the empty node back to your autoscaler.

{% hint style="info" %}
To see which pods block node removal in your cluster, open the [Bin packing view](https://docs.perfectscale.io/visibility-and-optimization/bin-packing-optimization.md).
{% endhint %}

## Bin packing automation concept

With bin packing automation enabled, you can:

1. **Cut node costs** by removing nodes that your autoscaler cannot remove.
2. **Keep workloads available**: every pod is placed on another node first, pods are evicted through the Kubernetes Eviction API so PodDisruptionBudgets hold, and the node is removed only after the workloads are ready again.
3. **Stay in control** by choosing which node groups are drained and which nodes or pods are never touched.

{% hint style="success" %}
Bin packing automation never edits pod specs, PodDisruptionBudgets, pod annotations or node pools.
{% endhint %}

## How each autoscaler is handled

| Autoscaler                                   | What PerfectScale does                                                           |
| -------------------------------------------- | -------------------------------------------------------------------------------- |
| Karpenter                                    | Drains the node and deletes its `NodeClaim`. Karpenter terminates the instance.  |
| Cluster Autoscaler (EKS, kops, self-managed) | Drains the node and leaves it empty and cordoned. Cluster Autoscaler removes it. |
| Managed Cluster Autoscaler (GKE, AKS)        | Drains the node and leaves it empty and cordoned. The autoscaler removes it.     |
| Any other setup                              | Detection only. PerfectScale shows the blocked node but does not drain it.       |

## Prerequisites

1. The PerfectScale automation agent (`psc-autoscaler`) **v1.0.77** or later is installed in the cluster. Learn how to install it [here](https://docs.perfectscale.io/enable-automation/automation-setup-instructions.md).
2. A `ClusterNodeGroupAutomationConfig` for the cluster. Without it, no node is drained.

   ```yaml
   apiVersion: perfectscale.io/v1
   kind: ClusterNodeGroupAutomationConfig
   metadata:
     name: your-cluster-config
   spec:
     automation:
       binpacking:
         operational:
           automationMode: Enabled
   ```
3. A `NodeGroupAutomationConfig` for every node group you want drained, with `automationMode` set to `Enabled`. A node group without its own resource is never drained.

{% hint style="success" %}
**Start with `DryRun`.** PerfectScale runs every check and records the plan for each node but changes nothing. Switch to `Enabled` when the plans look right.
{% endhint %}

## Enabling bin packing automation with CRD

**Create** a YAML file with the node group-level automation configuration using the following template:

```yaml
apiVersion: perfectscale.io/v1
kind: NodeGroupAutomationConfig
metadata:
  name: your-node-group-config
spec:
  targetRef:
    kind: NodeGroup # NodePool for Karpenter
    name: your-node-group-name # the name from the Node Group column in the Node group view
  automation:
    binpacking:
      operational:
        automationMode: Enabled # Enabled | DryRun | Disabled
        evictBarePods: true # default
      karpenter:
        doNotDisruptFor: "true" # Karpenter only, any non-empty value turns it on
      safety:
        allowHostPath: false # default
        allowRunningJobs: false # default
        allowAtMinSize: false # default, Cluster Autoscaler only
        block:
          kubeSystemPods: false # default
          gpuNodes: false # default
          diskEmptyDirPods: false # default
        blockOnPodAnnotations:
          example.com/never-drain: "true" # annotation: blocking value, empty = any value
        pdbOverride:
          enabled: false # default
          respectGracePeriod: true # default
        protectedPodSelectors:
          - matchLabels:
              app: payments
        excludedNodeSelectors:
          - matchLabels:
              workload-tier: critical
```

**Apply** the created configuration by running the following command (where `<file-name>.yaml` is the name of the file you created in the previous step):

```bash
kubectl apply -f <file-name>.yaml
```

{% hint style="info" %}
Prefer GitOps? Commit the file to Git and let a GitOps tool, such as ArgoCD or Flux, apply it. Learn more [here](https://docs.perfectscale.io/enable-automation/automation-with-argocd.md).
{% endhint %}

### CR parameters

| Parameter                                         | Description                                                                                                                                                                                                                                                                                                         |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `automationMode`                                  | `Enabled` drains nodes. `DryRun` only records the plan. `Disabled` does nothing.                                                                                                                                                                                                                                    |
| `evictBarePods` (default `true`)                  | Pods with no owner are evicted and deleted permanently, because nothing recreates them. Set `false` to skip nodes that run such pods.                                                                                                                                                                               |
| `doNotDisruptFor`                                 | On Karpenter nodes, marks the node as do-not-disrupt while it is being drained. Any non-empty value turns it on.                                                                                                                                                                                                    |
| `allowHostPath` (default `false`)                 | Nodes with pods that use a `hostPath` volume are skipped, because the data is lost with the node. Set `true` to allow the drain.                                                                                                                                                                                    |
| `allowRunningJobs` (default `false`)              | Nodes with running Job pods are skipped. Set `true` to allow the drain.                                                                                                                                                                                                                                             |
| `allowAtMinSize` (default `false`)                | Cluster Autoscaler only. Nodes in a node group that is already at its minimum size are skipped, because the autoscaler cannot remove them. Set `true` to allow the drain. The size is read from the `cluster-autoscaler-status` ConfigMap in `kube-system`. If it cannot be read, the check is skipped.             |
| `block.kubeSystemPods` (default `false`)          | Set `true` to skip nodes that run a pod from the `kube-system` namespace. DaemonSet and static pods are ignored.                                                                                                                                                                                                    |
| `block.gpuNodes` (default `false`)                | Set `true` to skip nodes that have a GPU.                                                                                                                                                                                                                                                                           |
| `block.diskEmptyDirPods` (default `false`)        | Set `true` to skip nodes that run a pod with a disk-backed `emptyDir` volume, because its data is lost when the pod moves. Memory-backed volumes (`Memory`, `HugePages`) are ignored.                                                                                                                               |
| `pdbOverride.enabled` (default `false`)           | When a PodDisruptionBudget keeps refusing an eviction, deletes the pod anyway after a 2 minute wait.                                                                                                                                                                                                                |
| `pdbOverride.respectGracePeriod` (default `true`) | A deleted pod keeps its termination grace period. Set `false` to delete it immediately.                                                                                                                                                                                                                             |
| `protectedPodSelectors`                           | Nodes that run a pod matching any selector are never drained.                                                                                                                                                                                                                                                       |
| `blockOnPodAnnotations`                           | Nodes that run a pod with one of these annotations are skipped. Each key is an annotation and its value is the value that blocks. An empty value blocks on any value of that key. For example `karpenter.sh/do-not-disrupt: "true"` or `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"`. Empty by default. |
| `excludedNodeSelectors`                           | Nodes with labels matching any selector are never drained.                                                                                                                                                                                                                                                          |

{% hint style="warning" %}
**Enable `pdbOverride` only when you accept the risk.** The budget is not respected for the deleted pod, so the workload can lose availability or data.
{% endhint %}

### Cluster-level configuration

The `binpacking` block in a `ClusterNodeGroupAutomationConfig` sets defaults for every node group that has its own `NodeGroupAutomationConfig`. The cluster resource alone never starts a drain. The node group value takes precedence, except for settings that only add protection: the two selector lists and `blockOnPodAnnotations` are combined (for the same annotation, the cluster value wins), and a `block` option turned on at the cluster level cannot be turned off for a node group.

**CR example** for the cluster level:

```yaml
apiVersion: perfectscale.io/v1
kind: ClusterNodeGroupAutomationConfig
metadata:
  name: your-cluster-config
spec:
  automation:
    operational:
      stopAllAutomation: false # cluster level only, true stops all node automation
    binpacking:
      operational:
        automationMode: DryRun # Enabled | DryRun | Disabled
        evictBarePods: true # default
      karpenter:
        doNotDisruptFor: "true" # Karpenter only, any non-empty value turns it on
      safety:
        allowHostPath: false # default
        allowRunningJobs: false # default
        allowAtMinSize: false # default, Cluster Autoscaler only
        block:
          kubeSystemPods: false # default
          gpuNodes: false # default
          diskEmptyDirPods: false # default
        blockOnPodAnnotations:
          example.com/never-drain: "true" # annotation: blocking value, empty = any value
        pdbOverride:
          enabled: false # default
          respectGracePeriod: true # default
        protectedPodSelectors:
          - matchLabels:
              app: payments
        excludedNodeSelectors:
          - matchExpressions:
              - key: node-role.kubernetes.io/control-plane
                operator: Exists
```

## Bin packing automation limitations

| Limitation                        | Description                                                                                                        |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Static pods and custom schedulers | Nodes with static pods or pods that use a custom scheduler are not drained.                                        |
| Pods without a place              | If one pod does not fit on any other node, the node is not drained.                                                |
| Nodes already being removed       | Nodes that are cordoned or already being removed by your autoscaler are skipped.                                   |
| Local volumes                     | Nodes with pods that use a PersistentVolume tied to that node are not drained, because the data stays on the node. |

## Reviewing drains

Each drain is one row in the **Bin Packing Automation** tab of the [Automation audit](https://docs.perfectscale.io/enable-automation/automation-audit.md#bin-packing-automation), at the organization and cluster levels. The row shows the status (**Success** for a removed node), **Realized Savings**, how much of the node was used at drain time, and the node group, node name and node age.

<figure><img src="https://1573387604-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FABMqnYtsOO44JmQTVSnn%2Fuploads%2FPYYyrFNzi3tTxSwikxu5%2Fbinpacking-automation-audit.png?alt=media&amp;token=10caa5fe-88c3-42cc-b36f-9bd4ac1be876" alt=""><figcaption><p>Bin packing automation audit</p></figcaption></figure>

Click a row to open the **node snapshot**: the node as it was before the drain, kept after it is removed, with its instance type, price, pods by size and **Node Details**. Select a pod for its **Pod Details**, including the reasons it was unevictable. A **Removed** badge marks a deleted node, and a failed drain shows its reason.

<figure><img src="https://1573387604-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FABMqnYtsOO44JmQTVSnn%2Fuploads%2Fx6Gekt49gzy6tYEkkA29%2Fbinpacking-node-snapshot.png?alt=media&amp;token=ee69da09-2ba9-4bdb-8889-a4afac44fbc9" alt=""><figcaption><p>Removed node snapshot</p></figcaption></figure>

In the [Node group view](https://docs.perfectscale.io/visibility-and-optimization/nodefit-node-right-sizing/node-group-view.md), the **Bin Packing Automation** column shows **Active**, **Dry Run**, **Disabled by User** or **Not Enabled** for each node group.

<figure><img src="https://1573387604-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FABMqnYtsOO44JmQTVSnn%2Fuploads%2FiEGbuGfkEWvdHj9XoFY0%2Fbinpacking-node-group-column.png?alt=media&amp;token=19e8cebd-52a7-4f53-b447-4ad82c44b060" alt=""><figcaption><p>Bin Packing Automation column in the Node group view</p></figcaption></figure>

## Stopping bin packing automation

To stop one node group, set `automationMode` to `Disabled`:

```yaml
spec:
  automation:
    binpacking:
      operational:
        automationMode: Disabled # stops draining this node group
```

To stop all node automation in the cluster, set `stopAllAutomation: true` in the `ClusterNodeGroupAutomationConfig`.

{% hint style="warning" %}
If a drain fails, the node can stay cordoned. Run `kubectl uncordon <node-name>` to release it.
{% endhint %}
