Configuring Automation for a workload

Learn how to effortlessly automate the right-sizing of the specific workloads

Including a workload to the Automation

Workload-level automation allows fine-tuning resource management of individual workloads. Automation configured on the workload level overrides cluster and namespace-level configuration.

Create a YAML file with the workload-level automation configuration using the following template:

apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: workload-automation-config
  namespace: your-namespace # The namespace where the configuration should be applied
spec:
  targetRef:
    kind: Deployment
    name: workload
  automation:
    operational:
      automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
      timeConstraints:
        wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
        fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
        maintenanceWindow:
          monday:
            - "00:00-23:59"
          tuesday:
            - "00:00-23:59"
          wednesday:
            - "00:00-23:59"
          thursday:
            - "00:00-23:59"
          friday:
            - "00:00-23:59"
          saturday:
            - "00:00-23:59"
          sunday:
            - "00:00-23:59"
        maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
        maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
      restrictions:
        workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
        cpuManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
            minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead            maximumCores: -1
          limit:
            keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
        memoryManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
          limit:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
            decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead            maximumGiB: -1
            memoryLeakDetection:
              maxMemoryIncreaseIterations:
                daily: 3 # Allows automation to increase memory limit up to 3 times per day
                weekly: 6 # Allows automation to increase memory limit up to 6 times per week
    containers:
      container1:
        operational:
          restrictions:
            cpuManagement:
              request:
                increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
                decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
                minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
                maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
              limit:
                keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
            memoryManagement:
              request:
                increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
                decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
                minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
                maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
              limit:
                increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
                decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
                minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
                maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
                memoryLeakDetection:
                  maxMemoryIncreaseIterations:
                    daily: 3 # Allows automation to increase memory limit up to 3 times per day
                    weekly: 6 # Allows automation to increase memory limit up to 6 times per week

Apply the created configuration by running the following command (where workload-automation-config.yaml - the name of your file created in the previous step)

kubectl apply -f workload-automation-config.yaml

Automating workloads with the specific type

DaemonSet

DaemonSet workloads are automated in decrease-only mode to prevent over-provisioning. This avoids excessive resource recommendations caused by spikes on one node, reducing waste and ensuring pods can still be scheduled on smaller nodes.

Create a YAML file with the workload-level automation configuration using the following template:

apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: your-workload-automation-config
  namespace: your-namespace # The namespace where the configuration should be applied
spec:
  targetRef:
    kind: DaemonSet #Case sensitive, must be DaemonSet, not Daemonset
    name: workload
  automation:
    operational:
      automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
      timeConstraints:
        wasteMaxAutomationFrequency: "4h" # Applies recommendations to decrease resources every 4 hours
        fixResiliencyMaxAutomationFrequency: "4h" # Applies recommendations to increase resources every 4 hours
        maintenanceWindow:
          monday:
            - "00:00-23:59"
          tuesday:
            - "00:00-23:59"
          wednesday:
            - "00:00-23:59"
          thursday:
            - "00:00-23:59"
          friday:
            - "00:00-23:59"
          saturday:
            - "00:00-23:59"
          sunday:
            - "00:00-23:59"
        maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
        maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue
      restrictions:
        workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
        cpuManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
            minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
          limit:
            keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
        memoryManagement:
          request:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
            decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
          limit:
            increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
            decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
            minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
            maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
            memoryLeakDetection:
              maxMemoryIncreaseIterations:
                daily: 3 # Allows automation to increase memory limit up to 3 times per day
                weekly: 6 # Allows automation to increase memory limit up to 6 times per week                weekly: 6

Apply the created configuration by running the following command (where your-workload-automation-config.yaml - the name of your file created in the previous step)

kubectl apply -f your-workload-automation-config.yaml

StatefulSet

Automating StatefulSets requires careful consideration:

1. You may want someone to manage production database resource changes, including any restart. Pods with persistent volumes are more sensitive.

2. In many cases, a StatefulSet runs on dedicated nodes. Consequently, reducing resources will not lead to a reduction in the actual cost.

Create a YAML file with the workload-level automation configuration using the following template:

apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: your-workload-automation-config
  namespace: your-namespace # The namespace where the configuration should be applied
spec:
  automation:
    operational:
      stopAllAutomation: false
    workloadTypes:
      StatefulSet:
        operational:
          automationMode: "Enabled" #Enables automation for the workloads with a specific type in the cluster/namespace
          timeConstraints:
            wasteMaxAutomationFrequency: "24h" # Applies recommendations to decrease resources every 24 hours
            fixResiliencyMaxAutomationFrequency: "24h" # Applies recommendations to increase resources every 24 hours
          restrictions:
            workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
            cpuManagement:
              request:
                increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
                decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
              limit:
                keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
            memoryManagement:
              request:
                increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
                decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
              limit:
                increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
                decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit               

Apply the created configuration by running the following command (where your-workload-automation-config.yaml - the name of your file created in the previous step)

kubectl apply -f your-workload-automation-config.yaml

CronJob and Job

PerfectScale doesn't restart workloads with the CronJob or Job type and automates only new workloads.

Create a YAML file with the workload-level automation configuration using one of the following templates:

CronJob configuration

apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: your-workload-automation-config
  namespace: your-namespace # The namespace where the configuration should be applied
spec:
  automation:
    operational:
      stopAllAutomation: false
    workloadTypes:
      CronJob:
        operational:
          automationMode: "Enabled"
          timeConstraints:
            wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
            fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
          restrictions:
            cpuManagement:
              request:
                increaseEnabled: false
                decreaseEnabled: true
              limit:
                keepLimit: true
            memoryManagement:
              request:
                increaseEnabled: false
                decreaseEnabled: true
              limit:
                increaseEnabled: false
                decreaseEnabled: false               

Job configuration

apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: your-workload-automation-config
  namespace: your-namespace # The namespace where the configuration should be applied
spec:
  automation:
    operational:
      stopAllAutomation: false
    workloadTypes:
      Job:
        operational:
          automationMode: "Enabled"
          timeConstraints:
            wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
            fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
          restrictions:
            cpuManagement:
              request:
                increaseEnabled: false
                decreaseEnabled: true
              limit:
                keepLimit: true
            memoryManagement:
              request:
                increaseEnabled: false
                decreaseEnabled: true
              limit:
                increaseEnabled: false
                decreaseEnabled: false               

Apply the created configuration by running the following command (where your-workload-automation-config.yaml - the name of your file created in the previous step)

kubectl apply -f your-workload-automation-config.yaml

Automating workloads by label

You can easily configure automation for specific workloads by using labels. To do so, add the workloadLabelSelectors section to your cluster-level or namespace-level configuration. Specify the label key and value, then choose whether to enable or disable automation for workloads with the given label.

If workloadLabelSelectors are specified in both ClusterAutomationConfig and NamespaceAutomationConfigconfigurations, PerfectScale will merge them.

In cases of a conflict on the key and value between the namespace and cluster configurations, the allowAutomation value from the namespace configuration will take precedence.

apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
  name: your-cluster-automation-config
spec:
  automation:
    operational:
      workloadLabelSelectors:
        - key: "environment"
          value: "production"
          allowAutomation: false    # Disables Automation for workload with the specified label
        - key: "tier"
          value: "frontend"
          allowAutomation: true     # Enables Automation for workload with the specified label

Expected results:

1. If workloadLabelSelectors is empty or not set, automation will be applied according to the other configurations in place.

2. If workloadLabelSelectors only contains labels with allowAutomation: true, automation will apply only to the workloads that match at least one label specified in this configuration.

3. If workloadLabelSelectors only contains labels with allowAutomation: false, automation will apply solely to workloads that do not match any specified labels in this configuration.

4. If workloadLabelSelectors contains a mix of labels with allowAutomation: false and allowAutomation: true, automation will only apply to workloads that match the labels with allowAutomation: true and do not match the labels with allowAutomation: false in this configuration.

If you enable automation for a custom workload type, the WorkloadLabelsSelector in a cluster or namespace configuration will not be applied. All workloads of that custom type will be automated despite the label's configuration.

Automating ephemeral workloads

Ephemeral workload automation is supported, starting with autoscaler version 1.0.17.

PerfectScale's advanced Automation empowers you to effortlessly optimize K8s workloads, even in highly dynamic and complex environments. By optimizing workloads with a transient nature, you can ensure that you’re not over-provisioning or paying for unused capacity, significantly reduce cloud costs, and meet performance demands without latency or bottlenecks.

By grouping ephemeral workloads with specific workload labels, PerfectScale automates the optimization of ephemeral workloads such as:

• Airflow

• Knative

• Spark jobs

• Temporal

• Github actions

• and more

Step 1: Add grouping labels to the workload

To automate the optimization for the ephemeral workloads, two predefined labels should be added to the workload.

These labels are required to configure automation for ephemeral workloads.

Key	Value	Description
perfectscale.io/workload-grouping-workload-name	custom-workload-name	Specifies a target workload name
perfectscale.io/workload-grouping-workload-type	custom-workload-type	Specifies a target workload type

After applying the labels, a new workload will appear in PerfectScale. However, automation and recommendations can only reduce resources after sufficient data has been collected.

You can specify additional labels to ensure PerfectScale considers all the revisions, including those that were executed not by Automation, to drive better results. All the revisions will be displayed in the Revision history log.

Key	Value	Description
perfectscale.io/workload-grouping-honor-spec	true	Allows PerfectScale to consider the resource changes in the original spec and changes of current resources.
perfectscale.io/workload-grouping-honor-image	true	Allows PerfectScale to consider the image name in the calculated hash.

If you enable automation for a custom workload type, the WorkloadLabelsSelector in a cluster or namespace configuration will not be applied. All workloads of that custom type will be automated despite the label's configuration.

Step 2: Apply CR that contains these labels

For workloads with a custom type, automation is enabled by setting up a Custom Resource (CR), where you can specify all the necessary configurations.

Create a YAML file with the automation configuration and indicate the custom workload type:

apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
  name: workload-automation-config
  namespace: your-namespace # The namespace where the configuration should be applied
spec:
  targetRef:
    kind: CustomSparkJob # Custom workload type
    name: customWorkload
  automation:
    operational:
      automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace

Apply the created configuration by running the following command (where workload-automation-config.yaml - the name of your file created in the previous step)

kubectl apply -f workload-automation-config.yaml

If a workload is already grouped in PerfectScale using a grouping profile, ensure that the automation configuration uses the same name and type as the existing grouped workload.

Example:

In the aws-us cluster, there’s a workload named myjob with the type GroupedJob, previously grouped via a grouping profile. To maintain consistency, the automation config and labels should match this name and type exactly.