Skip to content
Nubent

/Stable production

AWS high availability and incident response

When production goes down, you need someone who understands the platform and knows where to start. We help you stabilize the service, find the root cause and fix it so it does not happen again. Then we harden the architecture with Multi-AZ, tested backups and a disaster recovery plan with recovery targets agreed with the business.

/Diagnosis

Signs it is time

Outages that keep coming back

The same kind of incident returns every few weeks, and the team only has time to restart services until the next one.

Nobody knows what happened

After an outage there are not enough logs, metrics or traces to explain the cause with confidence.

A single point of failure

One database, one instance or one Availability Zone carries the entire service with no redundancy.

Backups nobody has restored

There are backups, but nobody has tested how long a restore takes or whether the data is complete.

/What we deliver

Concrete deliverables, in your repository

  • Service stabilization

    Hands-on work to bring production back, with temporary mitigations documented and a plan to replace them.

  • Root cause analysis

    A blameless postmortem with a timeline, the technical cause, contributing factors and prioritized corrective actions.

  • High availability architecture

    Multi-AZ redundancy across compute, data and load balancing, with health checks and automatic failover.

  • Disaster recovery plan

    A recovery strategy based on RTO and RPO, backups with AWS Backup and documented restore drills.

  • Runbooks and alerts

    Procedures for the most likely incidents, and alarms based on the symptoms users actually feel.

/How we work

Our process

  1. Stabilize

    First we bring the service back and contain the impact, even if the initial fix is temporary.

  2. Understand

    We rebuild the timeline from logs, metrics and recent changes until we reach the root cause.

  3. Fix

    We implement the permanent fix and the architecture improvements through pull requests, prioritized by risk.

  4. Prevent

    We test recovery with drills, tune alerts and leave runbooks for the on-call team.

What to do when production goes down on AWS

During an incident, the priority is getting the service back; the full explanation comes later. We start with what changed recently: deployments, configuration changes, AWS Health events and shifts in traffic. Many outages are resolved by reverting a change or freeing up a saturated resource. The mitigation is recorded so nobody mistakes it for the permanent fix.

Once the service is stable, we rebuild the timeline from CloudWatch logs, traces, CloudTrail events and database metrics. We look for the technical cause, but also for the factors that made it possible: an alarm that did not exist, an unknown service quota or a deployment with no automatic rollback. Those are the factors we fix so the next failure does not escalate.

Blameless postmortems that improve the platform

A good postmortem describes what happened, why the system allowed it and what is going to change, without looking for someone to blame. We document the timeline, the impact on users, the root cause and the corrective actions, each with an owner and a priority. Those actions become concrete tasks, such as infrastructure pull requests, new alarms or automated tests.

Over time, postmortems reveal patterns: components that fail often, fragile external dependencies or risky deployment processes. That information guides reliability investments better than any generic list of best practices, because it reflects how your platform actually fails, not how an average platform fails.

High availability architecture on AWS

High availability starts with removing single points of failure. We spread compute across multiple Availability Zones behind a load balancer with health checks, set up Multi-AZ databases or Aurora clusters with replicas and automatic failover, and review the dependencies people tend to forget, such as a NAT Gateway in a single zone or a self-managed queue server.

More redundancy is not always better. Every level of availability has a cost in infrastructure and in operational complexity, and not every service needs it. We agree with the business on which components are critical and what service level justifies each investment, and design accordingly, from Multi-AZ in one region to multi-region architectures when the case calls for it.

Disaster recovery on AWS with defined RTO and RPO

A disaster recovery plan answers two questions: how long the service can be down, the RTO, and how much data you can afford to lose, the RPO. With those answers we choose the right strategy, from backup and restore to pilot light, warm standby or active-active across regions, each with a different cost and complexity.

A plan that has never been tested is not a plan. We schedule restore drills with AWS Backup and, when it adds value, controlled experiments with AWS Fault Injection Service to verify that failover works as expected. The results are documented, and measured times replace assumptions in the business continuity plan.

  • Backup and restore: the lowest cost, with the longest recovery time.
  • Pilot light: replicated data and minimal services ready to switch on.
  • Warm standby: a scaled-down copy of the environment running in another region.
  • Active-active: multiple regions serving traffic at the same time.

/FAQ

Incidents and availability: frequently asked questions

Another question? Write to hello@nubent.com and an engineer will answer.

Can you help us during an active incident?

Yes. If we already have access to your environment, we join to help stabilize the service, review recent changes and apply mitigations. If it is our first time working together, the first step is agreeing on temporary, secure access with your team. After the incident we run the root cause analysis and propose the permanent fixes.

What is a root cause analysis?

It is the investigation that explains why an incident happened, beyond the visible symptom. It rebuilds the timeline from logs and metrics, identifies the technical failure and the factors that allowed it, and ends in concrete corrective actions. It is done without blame, with the goal of keeping the same problem from happening again.

What is the difference between high availability and disaster recovery?

High availability keeps the service running through common failures, such as losing an instance or an Availability Zone, using redundancy and automatic failover. Disaster recovery covers more serious events, such as losing a region or data corruption, and defines how and how fast the service is restored. The two complement each other.

How are RTO and RPO defined?

Together with the business and the technical team, at the same table. We analyze the impact of each service being down for different periods and how much data can be rebuilt or lost without serious consequences. That sets targets per service, because an internal dashboard and the payments flow do not need the same level of protection.

How often should we test our backups?

Ideally, restore regularly and automatically, before you ever need to. A scheduled restore verifies the backups are complete, measures how long the process takes and catches missing permissions or dependencies. AWS Backup can automate restore testing, and we document the result of each one.

Next step

Let's talk about your Incidents and availability project

30 minutes with an engineer. Concrete next steps, whether or not you work with us.