Workload-level automation allows fine-tuning resource management of individual workloads. Automation configured on the workload level overrides cluster and namespace-level configuration.
Create a YAML file with the workload-level automation configuration using the following template:
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
targetRef:
kind: Deployment
name: workload
automation:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
timeConstraints:
wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
maintenanceWindow:
monday:
- "00:00-23:59"
tuesday:
- "00:00-23:59"
wednesday:
- "00:00-23:59"
thursday:
- "00:00-23:59"
friday:
- "00:00-23:59"
saturday:
- "00:00-23:59"
sunday:
- "00:00-23:59"
maintenanceWindowIgnoredForResiliency: false # Indicates if the maintenance window constraints are ignored for resiliency
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead maximumCores: -1
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead maximumGiB: -1
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week
containers:
container1:
operational:
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week
Apply the created configuration by running the following command (where workload-automation-config.yaml - the name of your file created in the previous step)
kubectl apply -f workload-automation-config.yaml
Automating workloads with the specific type
DaemonSet
DaemonSet workloads are automated in decrease-only mode to prevent over-provisioning. This avoids excessive resource recommendations caused by spikes on one node, reducing waste and ensuring pods can still be scheduled on smaller nodes.
Create a YAML file with the workload-level automation configuration using the following template:
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: your-workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
targetRef:
kind: DaemonSet #Case sensitive, must be DaemonSet, not Daemonset
name: workload
automation:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
timeConstraints:
wasteMaxAutomationFrequency: "4h" # Applies recommendations to decrease resources every 4 hours
fixResiliencyMaxAutomationFrequency: "4h" # Applies recommendations to increase resources every 4 hours
maintenanceWindow:
monday:
- "00:00-23:59"
tuesday:
- "00:00-23:59"
wednesday:
- "00:00-23:59"
thursday:
- "00:00-23:59"
friday:
- "00:00-23:59"
saturday:
- "00:00-23:59"
sunday:
- "00:00-23:59"
maintenanceWindowIgnoredForResiliency: false # Indicates if the maintenance window constraints are ignored for resiliency
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead
maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead
memoryLeakDetection:
maxMemoryIncreaseIterations:
daily: 3 # Allows automation to increase memory limit up to 3 times per day
weekly: 6 # Allows automation to increase memory limit up to 6 times per week weekly: 6
Apply the created configuration by running the following command (where your-workload-automation-config.yaml - the name of your file created in the previous step)
You may want someone to manage production database resource changes, including any restart. Pods with persistent volumes are more sensitive.
In many cases, a StatefulSet runs on dedicated nodes. Consequently, reducing resources will not lead to a reduction in the actual cost.
Create a YAML file with the workload-level automation configuration using the following template:
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: your-workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
automation:
operational:
stopAllAutomation: false
workloadTypes:
StatefulSet:
operational:
automationMode: "Enabled" #Enables automation for the workloads with a specific type in the cluster/namespace
timeConstraints:
wasteMaxAutomationFrequency: "24h" # Applies recommendations to decrease resources every 24 hours
fixResiliencyMaxAutomationFrequency: "24h" # Applies recommendations to increase resources every 24 hours
restrictions:
workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
cpuManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste
limit:
keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero
memoryManagement:
request:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed
decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste
limit:
increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed
decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit
Apply the created configuration by running the following command (where your-workload-automation-config.yaml - the name of your file created in the previous step)
PerfectScale doesn't restart workloads with the CronJob or Job type and automates only new workloads.
Create a YAML file with the workload-level automation configuration using one of the following templates:
CronJob configuration
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: your-workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
automation:
operational:
stopAllAutomation: false
workloadTypes:
CronJob:
operational:
automationMode: "Enabled"
timeConstraints:
wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
restrictions:
cpuManagement:
request:
increaseEnabled: false
decreaseEnabled: true
limit:
keepLimit: true
memoryManagement:
request:
increaseEnabled: false
decreaseEnabled: true
limit:
increaseEnabled: false
decreaseEnabled: false
Job configuration
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: your-workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
automation:
operational:
stopAllAutomation: false
workloadTypes:
Job:
operational:
automationMode: "Enabled"
timeConstraints:
wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
restrictions:
cpuManagement:
request:
increaseEnabled: false
decreaseEnabled: true
limit:
keepLimit: true
memoryManagement:
request:
increaseEnabled: false
decreaseEnabled: true
limit:
increaseEnabled: false
decreaseEnabled: false
Apply the created configuration by running the following command (where your-workload-automation-config.yaml - the name of your file created in the previous step)
You can easily configure automation for specific workloads by using labels. To do so, add the workloadLabelSelectors section to your cluster-level or namespace-level configuration. Specify the label key and value, then choose whether to enable or disable automation for workloads with the given label.
If workloadLabelSelectors are specified in both ClusterAutomationConfig and NamespaceAutomationConfigconfigurations, PerfectScale will merge them.
In cases of a conflict on the key and value between the namespace and cluster configurations, the allowAutomation value from the namespace configuration will take precedence.
apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
name: your-cluster-automation-config
spec:
automation:
operational:
workloadLabelSelectors:
- key: "environment"
value: "production"
allowAutomation: false # Disables Automation for workload with the specified label
- key: "tier"
value: "frontend"
allowAutomation: true # Enables Automation for workload with the specified label
Expected results:
If workloadLabelSelectors is empty or not set, automation will be applied according to the other configurations in place.
If workloadLabelSelectors only contains labels with allowAutomation: true, automation will apply only to the workloads that match at least one label specified in this configuration.
If workloadLabelSelectors only contains labels with allowAutomation: false, automation will apply solely to workloads that do not match any specified labels in this configuration.
If workloadLabelSelectors contains a mix of labels with allowAutomation: false and allowAutomation: true, automation will only apply to workloads that match the labels with allowAutomation: true and do not match the labels with allowAutomation: false in this configuration.
If you enable automation for a custom workload type, the WorkloadLabelsSelector in a cluster or namespace configuration will not be applied. All workloads of that custom type will be automated despite the label's configuration.
Automating ephemeral workloads
Ephemeral workload automation is supported, starting with autoscaler version 1.0.17.
PerfectScale's advanced Automation empowers you to effortlessly optimize K8s workloads, even in highly dynamic and complex environments. By optimizing workloads with a transient nature, you can ensure that you’re not over-provisioning or paying for unused capacity, significantly reduce cloud costs, and meet performance demands without latency or bottlenecks.
By grouping ephemeral workloads with specific workload labels, PerfectScale automates the optimization of ephemeral workloads such as:
Airflow
Knative
Spark jobs
Temporal
Github actions
and more
Step 1: Add grouping labels to the workload
To automate the optimization for the ephemeral workloads, two predefined labels should be added to the workload.
These labels are required to configure automation for ephemeral workloads.
Key
Value
Description
perfectscale.io/workload-grouping-workload-name
custom-workload-name
Specifies a target workload name
perfectscale.io/workload-grouping-workload-type
custom-workload-type
Specifies a target workload type
After applying the labels, a new workload will appear in PerfectScale. However, automation and recommendations can only reduce resources after sufficient data has been collected.
Key
Value
Description
perfectscale.io/workload-grouping-honor-spec
true
Allows PerfectScale to consider the resource changes in the original spec and changes of current resources.
perfectscale.io/workload-grouping-honor-image
true
Allows PerfectScale to consider the image name in the calculated hash.
Step 2: Configure Custom Resource (CR)
For workloads with a custom type, automation is enabled by setting up a Custom Resource (CR), where you can specify all the necessary configurations.
Create a YAML file with the automation configuration and indicate the custom workload type:
apiVersion: perfectscale.io/v1
kind: WorkloadAutomationConfig
metadata:
name: workload-automation-config
namespace: your-namespace # The namespace where the configuration should be applied
spec:
targetRef:
kind: CustomSparkJob # Custom workload type
name: customWorkload
automation:
operational:
automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace
Apply the created configuration by running the following command (where workload-automation-config.yaml - the name of your file created in the previous step)
You can specify additional labels to ensure PerfectScale considers all the revisions, including those that were executed not by Automation, to drive better results. All the revisions will be displayed in the .
If you enable automation for a custom workload type, the in a cluster or namespace configuration will not be applied. All workloads of that custom type will be automated despite the label's configuration.
