Configuring Automation for a workload

Learn how to effortlessly automate the right-sizing of the specific workloads

Including a workload in the Automation

Workload-level automation allows fine-tuning resource management of individual workloads. Automation configured on the workload level overrides cluster and namespace-level configuration.

Workload automation config should specify a namespace, and it should match the namespace of the associated workload.

Create a YAML file with the workload-level automation configuration using the following template:

apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: workload-automation-config
  namespace: your-namespace # The namespace where the configuration should be applied
spec:
  targetRef:
    kind: Deployment
    name: workload
  automation:
    operational:
      automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
      timeConstraints:
        wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
        fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
        maintenanceWindow:
          monday:
            - "00:00-23:59"
          tuesday:
            - "00:00-23:59"
          wednesday:
            - "00:00-23:59"
          thursday:
            - "00:00-23:59"
          friday:
            - "00:00-23:59"
          saturday:
            - "00:00-23:59"
          sunday:
            - "00:00-23:59"
        maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
        maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
      restrictions:
        workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
        cpuManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
            minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead            maximumCores: -1
          limit:
            keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
        memoryManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
          limit:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
            decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead            maximumGiB: -1
            memoryLeakDetection:
              maxMemoryIncreaseIterations:
                daily: 3 # Allows automation to increase memory limit up to 3 times per day
                weekly: 6 # Allows automation to increase memory limit up to 6 times per week
    containers:
      container1:
        operational:
          restrictions:
            cpuManagement:
              request:
                increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
                decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
                minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
                maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
              limit:
                keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
            memoryManagement:
              request:
                increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
                decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
                minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
                maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
              limit:
                increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
                decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
                minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
                maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
                memoryLeakDetection:
                  maxMemoryIncreaseIterations:
                    daily: 3 # Allows automation to increase memory limit up to 3 times per day
                    weekly: 6 # Allows automation to increase memory limit up to 6 times per week

Apply the created configuration by running the following command (where workload-automation-config.yaml - the name of your file created in the previous step)

kubectl apply -f workload-automation-config.yaml

Automating workloads with a specific type

DaemonSet

DaemonSet workloads are automated in decrease-only mode to prevent over-provisioning. This avoids excessive resource recommendations caused by spikes on one node, reducing waste and ensuring pods can still be scheduled on smaller nodes.

Create a YAML file with the workload-level automation configuration using the following template:

apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: your-workload-automation-config
  namespace: your-namespace # The namespace where the configuration should be applied
spec:
  targetRef:
    kind: DaemonSet #Case sensitive, must be DaemonSet, not Daemonset
    name: workload
  automation:
    operational:
      automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
      timeConstraints:
        wasteMaxAutomationFrequency: "4h" # Applies recommendations to decrease resources every 4 hours
        fixResiliencyMaxAutomationFrequency: "4h" # Applies recommendations to increase resources every 4 hours
        maintenanceWindow:
          monday:
            - "00:00-23:59"
          tuesday:
            - "00:00-23:59"
          wednesday:
            - "00:00-23:59"
          thursday:
            - "00:00-23:59"
          friday:
            - "00:00-23:59"
          saturday:
            - "00:00-23:59"
          sunday:
            - "00:00-23:59"
        maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
        maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
      restrictions:
        workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
        cpuManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
            minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
          limit:
            keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
        memoryManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
          limit:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
            decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
            memoryLeakDetection:
              maxMemoryIncreaseIterations:
                daily: 3 # Allows automation to increase memory limit up to 3 times per day
                weekly: 6 # Allows automation to increase memory limit up to 6 times per week                weekly: 6

Apply the created configuration by running the following command (where your-workload-automation-config.yaml - the name of your file created in the previous step)

kubectl apply -f your-workload-automation-config.yaml

StatefulSet

Automating StatefulSets requires careful consideration:

1. You may want someone to manage production database resource changes, including any restart. Pods with persistent volumes are more sensitive.

2. In many cases, a StatefulSet runs on dedicated nodes. Consequently, reducing resources will not lead to a reduction in the actual cost.

Create a YAML file with the workload-level automation configuration using the following template:

apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: your-workload-automation-config
  namespace: your-namespace # The namespace where the configuration should be applied
spec:
  targetRef:
    kind: StatefulSet #Case sensitive, must be StatefulSet, not Statefulset
    name: workload
  automation:
    operational:
      automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
      timeConstraints:
        wasteMaxAutomationFrequency: "24h" # Applies recommendations to decrease resources every 4 hours
        fixResiliencyMaxAutomationFrequency: "24h" # Applies recommendations to increase resources every 4 hours
        maintenanceWindow:
          monday:
            - "00:00-23:59"
          tuesday:
            - "00:00-23:59"
          wednesday:
            - "00:00-23:59"
          thursday:
            - "00:00-23:59"
          friday:
            - "00:00-23:59"
          saturday:
            - "00:00-23:59"
          sunday:
            - "00:00-23:59"
        maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
        maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
      restrictions:
        workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
        cpuManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
            minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
          limit:
            keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
        memoryManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
          limit:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
            decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
            memoryLeakDetection:
              maxMemoryIncreaseIterations:
                daily: 3 # Allows automation to increase memory limit up to 3 times per day
                weekly: 6 # Allows automation to increase memory limit up to 6 times per week                       

Apply the created configuration by running the following command (where your-workload-automation-config.yaml - the name of your file created in the previous step)

kubectl apply -f your-workload-automation-config.yaml

CronJob and Job

PerfectScale doesn't restart workloads with the CronJob or Job type and automates only new workloads.

Create a YAML file with the workload-level automation configuration using one of the following templates:

CronJob configuration

apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: your-workload-automation-config
  namespace: your-namespace # The namespace where the configuration should be applied
spec:
  targetRef:
    kind: CronJob #Case sensitive, must be CronJob, not Cronjob
    name: workload
  automation:
    operational:
      automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
      timeConstraints:
        wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 4 hours
        fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 4 hours
        maintenanceWindow:
          monday:
            - "00:00-23:59"
          tuesday:
            - "00:00-23:59"
          wednesday:
            - "00:00-23:59"
          thursday:
            - "00:00-23:59"
          friday:
            - "00:00-23:59"
          saturday:
            - "00:00-23:59"
          sunday:
            - "00:00-23:59"
        maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
        maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
      restrictions:
        workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
        cpuManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
            minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
          limit:
            keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
        memoryManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
          limit:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
            decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
            memoryLeakDetection:
              maxMemoryIncreaseIterations:
                daily: 3 # Allows automation to increase memory limit up to 3 times per day
                weekly: 6 # Allows automation to increase memory limit up to 6 times per week                             

Job configuration

apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: your-workload-automation-config
  namespace: your-namespace # The namespace where the configuration should be applied
spec:
  targetRef:
    kind: Job
    name: workload
  automation:
    operational:
      automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
      timeConstraints:
        wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 4 hours
        fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 4 hours
        maintenanceWindow:
          monday:
            - "00:00-23:59"
          tuesday:
            - "00:00-23:59"
          wednesday:
            - "00:00-23:59"
          thursday:
            - "00:00-23:59"
          friday:
            - "00:00-23:59"
          saturday:
            - "00:00-23:59"
          sunday:
            - "00:00-23:59"
        maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
        maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
      restrictions:
        workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
        cpuManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
            minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
          limit:
            keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
        memoryManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
          limit:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
            decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
            memoryLeakDetection:
              maxMemoryIncreaseIterations:
                daily: 3 # Allows automation to increase memory limit up to 3 times per day
                weekly: 6 # Allows automation to increase memory limit up to 6 times per week                            

Apply the created configuration by running the following command (where your-workload-automation-config.yaml - the name of your file created in the previous step)

kubectl apply -f your-workload-automation-config.yaml

Automating workloads by label

You can easily configure automation for specific workloads by using labels. To do so, add the workloadLabelSelectors section to your cluster-level or namespace-level configuration. Specify the label key and value, then choose whether to enable or disable automation for workloads with the given label.

If workloadLabelSelectors are specified in both ClusterAutomationConfig and NamespaceAutomationConfigconfigurations, PerfectScale will merge them.

In cases of a conflict on the key and value between the namespace and cluster configurations, the allowAutomation value from the namespace configuration will take precedence.

apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
  name: your-cluster-automation-config
spec:
  automation:
    operational:
      workloadLabelSelectors:
        - key: "environment"
          value: "production"
          allowAutomation: false    # Disables Automation for workload with the specified label
        - key: "tier"
          value: "frontend"
          allowAutomation: true     # Enables Automation for workload with the specified label

Expected behavior:

1. If workloadLabelSelectors is empty or not set, automation configurations will be applied according to the other configurations in place.

2. If workloadLabelSelectors only contains labels with allowAutomation: true, automation configurations will be applied only to the workloads that match at least one label specified in this configuration.

3. If workloadLabelSelectors only contains labels with allowAutomation: false, automation configurations will be applied solely to workloads that do not match any specified labels in this configuration.

4. If workloadLabelSelectors contains a mix of labels with allowAutomation: false and allowAutomation: true, automation configurations will be applied to workloads that match the labels with allowAutomation: true and do not match the labels with allowAutomation: false in this configuration.

If a workload has a label selector allowAutomation: true, indicating it should be automated, but workloadType.X.operational.automationMode is set to Disabled, the automationMode setting takes precedence.

If you enable automation for a custom workload type, the WorkloadLabelsSelector in a cluster or namespace configuration will not be applied. All workloads of that custom type will be automated despite the label's configuration.

Label-based automation execution logic

When autonomous optimization is configured using labels, PerfectScale applies the following logic:

1. Check the labels section in the Automation CR.

2. Check the configurations applied to workloads that have these labels, including the automationMode value.

To automate workloads using labels, ensure an exact match between the labels defined in the Automation CR and the labels applied to the target workloads. If the labels do not match exactly, the workload will not be automated.

Expected results example

apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
  name: your-cluster-automation-config
spec:
  automation:
    operational:
      automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
      workloadLabelSelectors:
        - key: "environment"
          value: "production"
          allowAutomation: false    # Disables Automation for workload with the specified label
        - key: "env"
          value: "dev"
          allowAutomation: true    # Disables Automation for workload with the specified label
        - key: "tier"
          value: "frontend"
          allowAutomation: true     # Enables Automation for workload with the specified label

✅ Automated workloads

As a result of the Automation CR configuration above, PerfectScale verifies that automationMode is set to Enabled and checks the workloadLabelSelectors. Automation will be applied to all workloads in the specified namespace that match at least one configured label with allowAutomation: true, and do not match label with allowAutomation:false.

In this case, the automation mode is applied according to the default merge logic based on the cluster- or namespace-level configuration.

🔴 Non-automated workloads

As a result of the Automation CR configuration above, PerfectScale verifies that automationMode is set to Enabled and checks the workloadLabelSelectors. Workloads with allowAutomation: false, as well as workloads in the specified namespace that do not have at least one label with allowAutomation: true, will not be automated.

Automating ephemeral workloads

Ephemeral workload automation is supported, starting with autoscaler version 1.0.17.

PerfectScale's advanced Automation empowers you to effortlessly optimize K8s workloads, even in highly dynamic and complex environments. By optimizing workloads with a transient nature, you can ensure that you’re not over-provisioning or paying for unused capacity, significantly reduce cloud costs, and meet performance demands without latency or bottlenecks.

By grouping ephemeral workloads with specific workload labels, PerfectScale automates the optimization of ephemeral workloads such as:

• Airflow

• Knative

• Spark jobs

• Temporal

• Github actions

• and more

Step 1: Group ephemeral workloads by labels

To automate the optimization of ephemeral workloads, you first need to group them by applying specific labels. This is done by adding predefined labels to each workload. For detailed instructions on how to group ephemeral workloads, refer to this page.

Step 2: Apply CR that contains these labels

For workloads with a custom type, automation is enabled by setting up a Custom Resource (CR), where you can specify all the necessary configurations.

Create a YAML file with the automation configuration and indicate the custom workload type:

apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: workload-automation-config
  namespace: your-namespace # The namespace where the configuration should be applied
spec:
  targetRef:
    kind: CustomSparkJob # Custom workload type
    name: customWorkload
  automation:
    operational:
      automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace

Apply the created configuration by running the following command (where workload-automation-config.yaml - the name of your file created in the previous step)

kubectl apply -f workload-automation-config.yaml