Resiliency alerts

Resolve resilience risks proactively before they affect your users with impact-driven prioritization

What are resiliency alerts

Alerts are designed to quickly identify and notify of relevant indicators changes, ensuring they can be eliminated before impacting the system.

For faster updates, utilize Slack, MS Teams, or DataDog Alerts Integrations to receive notifications when an Alert is generated.

By default, PerfectScale generates alerts for every cluster where a resilience indicator with a min_risk_level: high was identified.

In order to customize your alerts, an Alerts Profile must be set up and applied to the cluster. It will override the default alert generation process, ensuring identification and notification about relevant indicator changes.

Resiliency alerts profile

You can apply only one Resiliency Alert Profile per cluster.

A cluster that transmits data is required to start receiving alerts configured with an Alert Profile.

🛠️ How to create Alerts Profile

There are two options for creating a Profile: from the Settings tab or directly from the Overview.

From the Settings tab.

Go to the Settings tab on the left panel -> select Alerts -> click the +Add Profile button -> name the profile in the corresponding row -> specify the min_risk_level (low, medium, high) and ignored entities (if needed) -> click Save button.

Resiliency alerts profile from settings

If the min_risk_level is set to medium, only indicators with high or medium severity will trigger alerts.

From the Overview tab

Go to the Overview tab on the left panel -> find the cluster to which you want to apply the Alert Profile and click gear button -> go to Customizations -> click on Add New Profile in the Alerts drop-down list -> name the profile in the corresponding row -> specify the min_risk_level (low, medium, high) and ignored entities (if needed) -> click the Save And Apply button.

Resiliency alerts profile from overview

Alerts Profile Configuration

Resiliency alerts profile configuration

1. Name the profile.

2. Configure the profile:

   • min_risk_level - the minimum risk level of the issue to trigger the alert. 💡NOTE: if the min_risk_level is set to medium, only indicators with high or medium severity will trigger alerts.

   • ignored_entities - allows removing specific workloads, namespaces, and containers from triggering alerts.

   • ignored_indicator - allows ignoring certain types of issues from triggering alerts.

   • active_notification_resend - enables the configuration of the active alert resend feature at a specified interval to ensure critical alerts remain visible until addressed. 💡 NOTE: active_notification_resend is set to off by default. 👉 Examples: active_notification_resend: 5h - resend notifications for active alerts every 5 hours. active_notification_resend: 1d - resend notifications for active alerts every day.

🛠️ How to apply Alerts Profile

Apply to a single cluster

To apply Alert Profile to the cluster, go to the Overview tab on the left panel -> find the cluster to which you want to apply the Alert Profile and click gear button -> go to Customizations -> select the needed profile in the Alerts drop-down list.

Applying resiliency alerts profile to a single cluster

Apply to multiple clusters

To apply the profile to multiple clusters from a single view, use the Manage Assignments feature.

Go to the Settings tab on the left panel -> select the Alerts -> click the Manage Assignments button -> apply the profiles for the needed clusters -> click the Save Changes button.

💡NOTE: You can easily manage your profiles (create, delete), but deleting such profiles is impossible if the profile is already connected to the cluster. Change the profile to default or any other, and only after that remove the current. If you change the Alert Profile to 'None', it will use the default action for alert generation (detected resilience indicator with a min_risk_level: high).

If you want to stop generating alerts to your communication channels, disconnect the messaging profile from the cluster (Slack, MS Teams).