Automation customization with CRD

Learn how to customize your Automation setup

• All example YAML HashMaps within timeConstraints and restrictions, are applicable to 3 different kinds of yaml objects:

  • kind: ClusterAutomationConfig

  • kind: NamespaceAutomationConfig

  • kind: WorkloadAutomationConfig

• All example YAML HashMaps within timeConstraints , are applicable to multiple YAML locations:

  • kind: ClusterAutomationConfig and kind: NamespaceAutomationConfig

    • spec.automation.operational.timeConstraints:

    • spec.automation.workloadTypes.(StatefulSet|Deployment|DaemonSet|CronJob|Job).operational.timeConstraints:

  • kind: WorkloadAutomationConfig

    • spec.automation.operational.timeConstraints:

    • spec.automation.operational.containers.(a-specific-container-name).operational.timeConstraints:

• All example YAML HashMaps within restrictions , are applicable to multiple YAML locations:

  • kind: ClusterAutomationConfig and kind: NamespaceAutomationConfig

    • spec.automation.operational.restrictions:

    • spec.automation.workloadTypes.(StatefulSet|Deployment|DaemonSet|CronJob|Job).operational.restrictions:

  • kind: WorkloadAutomationConfig

    • spec.automation.operational.restrictions:

    • spec.automation.operational.containers.(a-specific-container-name).operational.restrictions:

• kind: NamespaceAutomationConfig and kind: WorkloadAutomationConfig

  • Can have an optional value: metadata.namespace: example-namespace. Most examples omit this for brevity.

• All restrictions are compared to the original spec values of the workload.

  • kind: ClusterAutomationConfig is used to set the original spec value.

    • If a minimal config is used several implicit default values will be used.

    • To determine the implicit default values

      1. Login to https://app.perfectscale.io/

      2. Navigate to the Overview tab

      3. Select a cluster that has automation enabled

      4. Select any running workload by name

      5. Click in the top left of the pop-up window to see Automation Config CR mergedConfig YAML manifest.

      6. The mergedConfig view makes explicit the end-result of merged values and implicit default values.

  • kind: NamespaceAutomationConfig can be used to override ClusterAutomationConfig.

  • kind: WorkloadAutomationConfig can be used to both ClusterAutomationConfig and NamespaceAutomationConfig.

Customize your automation with the following settings:

apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig #, NamespaceAutomationConfig, or WorkloadAutomationConfig
metadata:
  name: example
spec:
  automation:
    operational:
      timeConstraints:
        wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
        fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
        maintenanceWindow: #UTC
          monday: ["00:00-23:59"]
          tuesday: ["00:00-23:59"]
          wednesday: ["00:00-23:59"]
          thursday: ["00:00-23:59"]
          friday: ["00:00-23:59"]
          saturday: ["00:00-23:59"]
          sunday: ["00:00-23:59"]
        maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
        maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue 
      restrictions:
        workloadMinWasteUSDPerMonth: 5 # Activates automation only if monthly waste exceeds a set threshold (for example, $5)
        cpuManagement:
          request:
            increaseEnabled: false # default false
            decreaseEnabled: true  # default true
            minimumCores: 0.1      # default unset, to override to unset, use -1
            maximumCores: 7.5      # default unset, to override to unset, use -1
          limit:
            keepLimit: true        # default true
        memoryManagement:
          request:
            increaseEnabled: true  # default false
            decreaseEnabled: true  # default true
            minimumGib: 0.1        # default unset, to override to unset, use -1
            maximumGib: 15.1       # default unset, to override to unset, use -1
          limit:
            increaseEnabled: true  # default false
            decreaseEnabled: false # default false
            minimumGib: 2          # default unset, to override to unset, use -1
            maximumGib: 31.1       # default unset, to override to unset, use -1
            memoryLeakDetection:
              maxMemoryIncreaseIterations:
                daily: 4           # default 3
                weekly: 7          # default 7

Time constraints

apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
  name: example
spec:
  automation:
    operational:
      timeConstraints:
        wasteMaxAutomationFrequency: "30m" # Applies recommendations to decrease resources every 30 minutes
        fixResiliencyMaxAutomationFrequency: "30m" # Applies recommendations to increase resources every 30 minutes
        maintenanceWindow:
          sunday: [00:00-23:59]
        maintenanceWindowIgnoredForResiliency: false # deprecated, use "maintenanceWindowIgnoredMinResiliencyLevel" instead
        maintenanceWindowIgnoredMinResiliencyLevel: None # Indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue

🗑️ Maximum automation frequency for waste

maxAutomationFrequency field is deprecated as of autoscaler version v1.0.16. If maxAutomationFrequency is present and wasteMaxAutomationFrequency is not specified, maxAutomationFrequency will override wasteMaxAutomationFrequency.

This optional field defines the maximum frequency at which automated actions can be executed to apply recommendations when waste is detected, and resource reduction is needed.

If no wasteMaxAutomationFrequency is specified, PerfectScale defaults to values based on the type of workload:

Deployment: "30m" Rollout: "30m" DaemonSet: "4h" StatefulSet: "24h" CronJob: "30m" Job: "30m"

🚨 Maximum automation frequency for waste resiliency

maxAutomationFrequency field is deprecated as of autoscaler version v1.0.16.

This optional field defines the maximum frequency at which automated actions can be executed to apply recommendations when resiliency risks are detected, and resource increase is needed.

If no fixResiliencyMaxAutomationFrequency is specified, PerfectScale defaults to values based on the type of workload:

Deployment: "30m" Rollout: "30m" DaemonSet: "4h" StatefulSet: "24h" CronJob: "30m" Job: "30m"

🕙 Maintenance window

The maintenance window defines the timeframe for PerfectScale to implement the recommendations.

The maintenance window is particularly helpful in minimizing the impact of changes on business operations and provides a structured approach to system updates and changes.

Time is indicated in UTC.

Example 1

The automation actions are scheduled daily between 00:00 and 23:59 UTC:

apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
  name: example1
spec:
  automation:
    operational:
      timeConstraints:
        maintenanceWindow: #UTC
          monday: ["00:00-23:59"]    # <-- Implicit Default Value 
          tuesday: ["00:00-23:59"]
          wednesday: ["00:00-23:59"]
          thursday: ["00:00-23:59"]
          friday: ["00:00-23:59"]
          saturday: ["00:00-23:59"]
          sunday: ["00:00-23:59"]

Example 2

Excluding specific days from the automation maintenanceWindow. To exclude a specific day(s) from an automation maintenanceWindow, set the timeframe to 00:00-00:00 (e.g., Monday - Friday in the example above).

apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
  name: example2
spec:
  automation:
    operational:
      timeConstraints:
        maintenanceWindow: #UTC
          monday: ["00:00-00:00"]
          tuesday: ["00:00-00:00"]
          wednesday: ["00:00-00:00"]
          thursday: ["00:00-00:00"]
          friday: ["00:00-00:00"]

In the example above, the maintenanceWindow does not specify a timeframe for Saturday and Sunday. As a result, these days default to the implicit default value of 00:00-23:59 .

Example 3

The automation actions are scheduled for the end of one day and the beginning of the next.

apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
  name: example3
spec:
  automation:
    operational:
      timeConstraints:
        maintenanceWindow:
          monday:
          - "00:00-02:30"
          - "23:30-23:59"
          tuesday:
          - "00:00-02:30"
          - "23:30-23:59"
          wednesday:
          - "00:00-02:30"
          - "23:30-23:59"
          thursday:
          - "00:00-02:30"
          - "23:30-23:59"
          friday:
          - "00:00-02:30"
          - "23:30-23:59"
          saturday:
          - "00:00-02:30"
          - "23:30-23:59"
          sunday:
          - "00:00-02:30"
          - "23:30-23:59"

Example of a Potential Syntax Error: The following configuration is invalid because time intervals must be contained within a single day. The example below ranges from late Monday into early Tuesday:

spec:
  automation:
    operational:
      timeConstraints:
        maintenanceWindow:
          monday:
          - "23:30-02:30"

🙅 Minimum resiliency level to override maintenance window

maintenanceWindowIgnoredForResiliency is deprecated.

In some cases, you may want Automation to override the maintenance window to ensure that critical resiliency issues, such as CPU throttling or OOM occurring outside the maintenance window, are properly addressed by an immediate resource increase.

apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
  name: example
spec:
  automation:
    operational:
      timeConstraints:
        maintenanceWindowIgnoredMinResiliencyLevel: None

This field indicates the minimum resiliency risk level required to bypass maintenance window constraints and resolve the issue. When set to None (default), the maintenance window constraints are enforced. However, when one of the resiliency levels is selected (Low, Medium, High), PerfectScale Automation will ignore maintenance window constraints for risks with the indicated resiliency level and above, applying changes to resolve the issue.

Example 1

The maintenance window will be ignored for risks at any level.

apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
  name: example1
spec:
  automation:
    operational:
      timeConstraints:
        maintenanceWindowIgnoredMinResiliencyLevel: Low

Example 2

The maintenance window will be ignored for risks with Medium and High risk levels

apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
  name: example2
spec:
  automation:
    operational:
      timeConstraints:
        maintenanceWindowIgnoredMinResiliencyLevel: Medium

Example 3

The maintenance window will be ignored for High risks only.

apiVersion: perfectscale.io/v1
kind: ClusterAutomationConfig
metadata:
  name: example3
spec:
  automation:
    operational:
      timeConstraints:
        maintenanceWindowIgnoredMinResiliencyLevel: High

💡Keep CPU limit

In general, we recommend removing CPU limits when possible. Doing so can improve the performance and efficiency of your workloads by allowing them to use available CPU resources more freely.

However, be aware that removing misconfigured CPU limits can increase memory consumption in many cases. This requires adjusting both memory requests and limits to ensure optimal performance and efficient use of resources.

We highly recommend that removing CPU limits always be combined with enabling increase allowed memory settings. This allows the system to automatically adjust memory resources to meet the increased demands, maintaining the stability and performance of your workloads.

🔄 Memory increase iteration limit

maxMemoryIncreaseIterations sets the maximum allowed number of daily and weekly memory limit increase iterations.

max_memory_increase_iterations is a mandatory field. Make sure it exists and doesn't equal to 0.

Once the number of daily or weekly memory increase iterations reaches the maximum value, to prevent potential memory leaks, PerfectScale Automation will stop recursive memory increase.

In order to notify the customer, PerfectScale will raise a relevant indicator and create an Alert.

⚙️ Optimization policy

PerfectScale offers flexible policy customization, giving you greater control over how optimization is applied.

With this feature, you can:

• Configure independent optimization policies for CPU and Memory.

• Apply a Memory Request = Limit policy for memory-sensitive workloads.

• Define custom time windows when the default setup is not sufficient due to known and predictable resource spikes.

These options enable you to tailor optimization behavior to your workload requirements, reliability constraints, and operational preferences while maintaining efficient and predictable optimization.

Learn more about optimization policy customization here. Additional YAML Examples can be found here.