# Configuring Automation for a workload ## Including a workload in the Automation Workload-level automation allows fine-tuning resource management of individual workloads. Automation configured on the workload level overrides cluster and namespace-level configuration. {% hint style="info" %} Workload automation config should specify a namespace, and it should match the namespace of the associated workload. {% endhint %} **Create** a YAML file with the workload-level automation configuration using the following template: ```yaml apiVersion: perfectscale.io/v1 kind: WorkloadAutomationConfig metadata: name: workload-automation-config namespace: your-namespace # The namespace where the configuration should be applied spec: targetRef: kind: Deployment name: workload automation: operational: automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace timeConstraints: wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes maintenanceWindow: monday: - "00:00-23:59" tuesday: - "00:00-23:59" wednesday: - "00:00-23:59" thursday: - "00:00-23:59" friday: - "00:00-23:59" saturday: - "00:00-23:59" sunday: - "00:00-23:59" maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue restrictions: workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5) cpuManagement: request: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead maximumCores: -1 limit: keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero memoryManagement: request: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead limit: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead maximumGiB: -1 memoryLeakDetection: maxMemoryIncreaseIterations: daily: 3 # Allows automation to increase memory limit up to 3 times per day weekly: 6 # Allows automation to increase memory limit up to 6 times per week containers: container1: operational: restrictions: cpuManagement: request: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead limit: keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero memoryManagement: request: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead limit: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead memoryLeakDetection: maxMemoryIncreaseIterations: daily: 3 # Allows automation to increase memory limit up to 3 times per day weekly: 6 # Allows automation to increase memory limit up to 6 times per week ``` **Apply** the created configuration by running the following command (where `workload-automation-config.yaml` - the name of your file created in the previous step) ``` kubectl apply -f workload-automation-config.yaml ``` ## Automating workloads with a specific type ### DaemonSet {% hint style="info" %} DaemonSet workloads are automated in **decrease-only mode** to prevent over-provisioning. This avoids excessive resource recommendations caused by spikes on one node, reducing waste and ensuring pods can still be scheduled on smaller nodes. {% endhint %} **Create** a YAML file with the workload-level automation configuration using the following template: ```yaml apiVersion: perfectscale.io/v1 kind: WorkloadAutomationConfig metadata: name: your-workload-automation-config namespace: your-namespace # The namespace where the configuration should be applied spec: targetRef: kind: DaemonSet #Case sensitive, must be DaemonSet, not Daemonset name: workload automation: operational: automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace timeConstraints: wasteMaxAutomationFrequency: "4h" # Applies recommendations to decrease resources every 4 hours fixResiliencyMaxAutomationFrequency: "4h" # Applies recommendations to increase resources every 4 hours maintenanceWindow: monday: - "00:00-23:59" tuesday: - "00:00-23:59" wednesday: - "00:00-23:59" thursday: - "00:00-23:59" friday: - "00:00-23:59" saturday: - "00:00-23:59" sunday: - "00:00-23:59" maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue restrictions: workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5) cpuManagement: request: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead limit: keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero memoryManagement: request: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead limit: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead memoryLeakDetection: maxMemoryIncreaseIterations: daily: 3 # Allows automation to increase memory limit up to 3 times per day weekly: 6 # Allows automation to increase memory limit up to 6 times per week weekly: 6 ``` **Apply** the created configuration by running the following command (where `your-workload-automation-config.yaml` - the name of your file created in the previous step) ``` kubectl apply -f your-workload-automation-config.yaml ``` ### StatefulSet {% hint style="info" %} Automating StatefulSets requires careful consideration: 1. You may want someone to manage production database resource changes, including any restart. Pods with persistent volumes are more sensitive. 2. In many cases, a StatefulSet runs on dedicated nodes. Consequently, reducing resources will not lead to a reduction in the actual cost. {% endhint %} **Create** a YAML file with the workload-level automation configuration using the following template: ```yaml apiVersion: perfectscale.io/v1 kind: WorkloadAutomationConfig metadata: name: your-workload-automation-config namespace: your-namespace # The namespace where the configuration should be applied spec: targetRef: kind: StatefulSet #Case sensitive, must be StatefulSet, not Statefulset name: workload automation: operational: automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace timeConstraints: wasteMaxAutomationFrequency: "24h" # Applies recommendations to decrease resources every 4 hours fixResiliencyMaxAutomationFrequency: "24h" # Applies recommendations to increase resources every 4 hours maintenanceWindow: monday: - "00:00-23:59" tuesday: - "00:00-23:59" wednesday: - "00:00-23:59" thursday: - "00:00-23:59" friday: - "00:00-23:59" saturday: - "00:00-23:59" sunday: - "00:00-23:59" maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue restrictions: workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5) cpuManagement: request: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead limit: keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero memoryManagement: request: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead limit: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead memoryLeakDetection: maxMemoryIncreaseIterations: daily: 3 # Allows automation to increase memory limit up to 3 times per day weekly: 6 # Allows automation to increase memory limit up to 6 times per week ``` **Apply** the created configuration by running the following command (where `your-workload-automation-config.yaml` - the name of your file created in the previous step) ``` kubectl apply -f your-workload-automation-config.yaml ``` ### CronJob and Job {% hint style="info" %} PerfectScale doesn't restart workloads with the CronJob or Job type and automates only new workloads. {% endhint %} **Create** a YAML file with the workload-level automation configuration using one of the following templates: #### **CronJob configuration** ``` apiVersion: perfectscale.io/v1 kind: WorkloadAutomationConfig metadata: name: your-workload-automation-config namespace: your-namespace # The namespace where the configuration should be applied spec: targetRef: kind: CronJob #Case sensitive, must be CronJob, not Cronjob name: workload automation: operational: automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace timeConstraints: wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 4 hours fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 4 hours maintenanceWindow: monday: - "00:00-23:59" tuesday: - "00:00-23:59" wednesday: - "00:00-23:59" thursday: - "00:00-23:59" friday: - "00:00-23:59" saturday: - "00:00-23:59" sunday: - "00:00-23:59" maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue restrictions: workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5) cpuManagement: request: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead limit: keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero memoryManagement: request: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead limit: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead memoryLeakDetection: maxMemoryIncreaseIterations: daily: 3 # Allows automation to increase memory limit up to 3 times per day weekly: 6 # Allows automation to increase memory limit up to 6 times per week ``` ### **Job configuration** ```yaml apiVersion: perfectscale.io/v1 kind: WorkloadAutomationConfig metadata: name: your-workload-automation-config namespace: your-namespace # The namespace where the configuration should be applied spec: targetRef: kind: Job name: workload automation: operational: automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace timeConstraints: wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 4 hours fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 4 hours maintenanceWindow: monday: - "00:00-23:59" tuesday: - "00:00-23:59" wednesday: - "00:00-23:59" thursday: - "00:00-23:59" friday: - "00:00-23:59" saturday: - "00:00-23:59" sunday: - "00:00-23:59" maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue restrictions: workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5) cpuManagement: request: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase CPU requests if the performance risks are observed decreaseEnabled: true # Allows PerfectScale Automation to decrease CPU requests to minimize waste minimumCores: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumCores: -1 # If a recommendation falls above the value, the automation will implement the maximum instead limit: keepLimit: true # Automation will maintain a non-zero CPU limit and will not set the value to zero memoryManagement: request: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory requests if the performance risks are observed decreaseEnabled: true # Allows PerfectScale Automation to decrease Memory requests to minimize waste minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead limit: increaseEnabled: false # Doesn't allow PerfectScale Automation to increase Memory limit if the performance risks are observed decreaseEnabled: false # Doesn't allow PerfectScale Automation to decrease Memory limit minimumGiB: -1 # If a recommendation falls below the value, the automation will implement the minimum instead maximumGiB: -1 # If a recommendation falls above the value, the automation will implement the maximum instead memoryLeakDetection: maxMemoryIncreaseIterations: daily: 3 # Allows automation to increase memory limit up to 3 times per day weekly: 6 # Allows automation to increase memory limit up to 6 times per week ``` **Apply** the created configuration by running the following command (where `your-workload-automation-config.yaml` - the name of your file created in the previous step) ``` kubectl apply -f your-workload-automation-config.yaml ``` ## **Automating workloads by label** You can easily configure automation for specific workloads by using labels. To do so, add the workloadLabelSelectors section to your cluster-level or namespace-level configuration. Specify the label key and value, then choose whether to enable or disable automation for workloads with the given label. {% hint style="info" %} If `workloadLabelSelectors` are specified in both `ClusterAutomationConfig` and `NamespaceAutomationConfig`configurations, PerfectScale will merge them. In cases of a conflict on the key and value between the namespace and cluster configurations, the `allowAutomation` value from the namespace configuration will take precedence. {% endhint %} ```yaml apiVersion: perfectscale.io/v1 kind: ClusterAutomationConfig metadata: name: your-cluster-automation-config spec: automation: operational: workloadLabelSelectors: - key: "environment" value: "production" allowAutomation: false # Disables Automation for workload with the specified label - key: "tier" value: "frontend" allowAutomation: true # Enables Automation for workload with the specified label ``` **Expected behavior:** 1. If `workloadLabelSelectors` is empty or not set, automation configurations will be applied according to the other configurations in place. 2. If `workloadLabelSelectors` only contains labels with `allowAutomation: true`, automation configurations will be applied only to the workloads that match at least one label specified in this configuration. 3. If `workloadLabelSelectors` only contains labels with `allowAutomation: false`, automation configurations will be applied solely to workloads that do not match any specified labels in this configuration. 4. If `workloadLabelSelectors` contains a mix of labels with `allowAutomation: false` and `allowAutomation: true`, automation configurations will be applied to workloads that match the labels with `allowAutomation: true` and do not match the labels with `allowAutomation: false` in this configuration. {% hint style="info" %} If a workload has a label selector `allowAutomation: true`, indicating it should be automated, but `workloadType.X.operational.automationMode` is set to **Disabled**, the `automationMode` setting takes precedence. {% endhint %} {% hint style="info" %} If you enable automation for a custom workload type, the `WorkloadLabelsSelector` in a cluster or namespace configuration will not be applied. All workloads of that custom type will be automated despite the label's configuration. {% endhint %} ### Label-based automation execution logic When autonomous optimization is configured using labels, PerfectScale applies the following logic: 1. Check the `labels` section in the **Automation CR**. 2. Check the configurations applied to workloads that have these labels, including the `automationMode` value. {% hint style="warning" %} To automate workloads using labels, ensure an exact match between the labels defined in the Automation CR and the labels applied to the target workloads. If the labels do not match exactly, the workload will not be automated. {% endhint %} #### Expected results example ```yaml apiVersion: perfectscale.io/v1 kind: ClusterAutomationConfig metadata: name: your-cluster-automation-config spec: automation: operational: automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace workloadLabelSelectors: - key: "environment" value: "production" allowAutomation: false # Disables Automation for workload with the specified label - key: "env" value: "dev" allowAutomation: true # Disables Automation for workload with the specified label - key: "tier" value: "frontend" allowAutomation: true # Enables Automation for workload with the specified label ``` :white\_check\_mark: **Automated workloads** As a result of the Automation CR configuration above, PerfectScale verifies that `automationMode` is set to **Enabled** and checks the `workloadLabelSelectors`. Automation will be applied to all workloads in the specified namespace that match at least one configured label with `allowAutomation: true`, and do not match label with `allowAutomation:false`. In this case, the automation mode is applied according to the default merge logic based on the cluster- or namespace-level configuration. :red\_circle: **Non-automated workloads** As a result of the Automation CR configuration above, PerfectScale verifies that `automationMode` is set to **Enabled** and checks the `workloadLabelSelectors`. Workloads with `allowAutomation: false`, as well as workloads in the specified namespace that do not have at least one label with `allowAutomation: true`, will **not** be automated. ## Automating ephemeral workloads {% hint style="info" %} Ephemeral workload automation is supported, starting with autoscaler version 1.0.17. {% endhint %} PerfectScale's advanced Automation empowers you to effortlessly optimize K8s workloads, even in highly dynamic and complex environments. By optimizing workloads with a transient nature, you can ensure that you’re not over-provisioning or paying for unused capacity, significantly reduce cloud costs, and meet performance demands without latency or bottlenecks. By grouping ephemeral workloads with specific workload labels, PerfectScale automates the optimization of ephemeral workloads such as: * Airflow * Knative * Spark jobs * Temporal * Github actions * and more #### Step 1: Group ephemeral workloads by labels To automate the optimization of ephemeral workloads, you first need to group them by applying specific labels. This is done by adding predefined labels to each workload. For detailed instructions on how to group ephemeral workloads, refer to [this page](/2.0-self-hosted-or-perfectscale-documentation/customize-workflow/ephemeral-pods-grouping.md#grouping-by-labels). #### Step 2: Apply CR that contains these labels For workloads with a custom type, automation is enabled by setting up a Custom Resource (CR), where you can specify all the necessary configurations. **Create** a YAML file with the automation configuration and indicate the custom workload type: ```yaml apiVersion: perfectscale.io/v1 kind: WorkloadAutomationConfig metadata: name: workload-automation-config namespace: your-namespace # The namespace where the configuration should be applied spec: targetRef: kind: CustomSparkJob # Custom workload type name: customWorkload automation: operational: automationMode: "Enabled" # Enables automation for the workloads with a specific type in the cluster/namespace ``` **Apply** the created configuration by running the following command (where `workload-automation-config.yaml` - the name of your file created in the previous step) ``` kubectl apply -f workload-automation-config.yaml ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.perfectscale.io/2.0-self-hosted-or-perfectscale-documentation/enable-automation/including-a-cluster-namespace-or-workload-to-the-automation/configuring-automation-for-a-workload.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.