Your first production incident: a runbook for a small team
A one-page runbook for the first-time CTO of a small startup: confirm, ask what changed, roll back first, check the server, tell customers, fix forward, write it down.
Your app is down, a customer has noticed, and you are the CTO of a five-person startup with no ops engineer. This post is the one-page runbook for that hour: what to do, in what order, and what to set up so the next one is shorter.
Disclosure: we build DollarDeploy, a control plane that deploys and operates apps on servers you own. The runbook works with any setup. Where DollarDeploy covers a step, we say so, and where it does not, we say that too.
Why it matters: in a small team, the length of an outage depends less on how clever the fix is than on two things decided in advance: can you see what changed, and can you go back.
The short version
If you only read the bold lines, this is the runbook:
- Confirm it and say so, in 2 minutes.
- Ask "what changed?" before you debug.
- Roll back first, debug later.
- Check the server, not only the app.
- Tell customers early, in plain words.
- Fix forward when the pressure is off.
- Write it down within 48 hours.
1. Confirm it and say so
The rule: before anything else, check the app yourself from outside your network, then post one line in the team channel: "Investigating: checkout is returning errors. I am on it."
Why: one person owns the incident from that moment. Nobody else restarts the server at the same time, and nobody wonders whether someone is on it.
How: open the app in a private window on your phone (not on office Wi-Fi), and note the time you started. You will need it for the write-up.
2. Ask "what changed?"
The rule: list everything that changed in the last 24 hours before you open a single log file.
Why: most incidents in a small team follow a change. Typical ones: a deploy, an env var edit, a database migration, a package update on the server, or a scheduled job that ran for the first time.
How: check these in order:
- Deploys: the last build, who started it, and the commit or pull request behind it.
- Config: env vars and secrets edited since yesterday.
- Server: security updates, a reboot, a full disk.
- Outside: your database host, payment provider or email service status pages.
In DollarDeploy, the app's Builds table shows the date, who started each build and the pull request it came from, with a Deployed badge on the build that is live.
3. Roll back first, debug later
The rule: if the incident started after a deploy, put the previous build back now, and find the bug afterwards.
Why: a rollback takes a minute. Debugging under pressure, with customers waiting, takes an hour and produces worse fixes.
How: redeploy the last build that worked, without rebuilding it. In DollarDeploy, press Deploy on any earlier build in the Builds table; it ships that build again as it was.
Yes, but: a rollback does not undo a database migration. If the new code changed the schema in a way the old code cannot read, rolling back the app makes things worse. The fix is a habit, not a tool: make every migration work with both the old and the new code (add a column first, remove the old one a release later).
4. Check the server, not only the app
The rule: if nothing changed, or the rollback did not help, look at the server the app runs on.
Why: a per-app health check does not see the problems that take down everything at once.
How: check, in this order:
- Disk: a full disk stops databases from writing and apps from logging.
- Memory: out of memory kills and crash loops in the system journal.
- Database: connection count, long-running queries, and whether Redis can still save to disk.
- HTTPS: a certificate that expired or failed to renew.
- Access: SSH logins you do not recognise.
In DollarDeploy, Inspect in the host menu runs these checks and more (13 areas in total) and shows an ok, warning or error status for each on the Health tab. It only reads: it changes nothing on the server and masks secrets in the output. App logs and monitoring are kept for 30 days, so you can compare with yesterday.
5. Tell customers early
The rule: if customers are affected for more than 15 minutes, tell them, even before you know the cause.
Why: customers forgive outages more easily than silence. The first message only needs three parts: what is broken, that you are working on it, and when the next update comes.
How: one short post where your customers already look: an in-app banner, an email to affected accounts, or a pinned message in your community. Then update them every 30-60 minutes until it is fixed.
6. Fix forward when the pressure is off
The rule: once the app is back up, fix the real bug in a normal pull request, with review, during working hours.
Why: a fix written during an outage skips the checks that would have caught the original bug.
How: reproduce it locally or on a staging copy, add a test that fails without the fix, and ship it through the same pipeline as any other change.
7. Write it down within 48 hours
The rule: every incident gets a half-page write-up, without blame.
Why: the write-up turns one bad evening into a fix the whole team keeps. Without it, the same incident comes back in three months.
How: five lines are enough:
- What happened: the symptom customers saw.
- Timeline: when it started, when someone noticed, when it was fixed.
- Cause: the change or failure behind it.
- What helped and what slowed us down.
- One follow-up: a single change that makes the next one shorter, with an owner and a date.
Before the next one
Most of the minutes in an incident are lost before it starts. Five things shorten the next one:
- An external uptime check that pings the app every minute and alerts a person. Uptime Kuma is one option you can self-host.
- One alerts channel the whole team reads, for deploys, failed jobs and backups. DollarDeploy sends these notifications to Slack, Telegram, Discord, email or a webhook, per project or app.
- Build history with one-click redeploy, tested once on a quiet afternoon.
- Logs kept for weeks, not hours.
- A backup you have restored, not only one you have taken.
Our honest limit: DollarDeploy does not page anyone when your app is down; pair it with an uptime monitor. We are also a small team without a public SLA or status page yet. If you need contractual uptime guarantees, factor that in.
The bottom line
The first production incident is stressful because every decision is made for the first time. Decide them now: who owns it, what changed, roll back first, check the server, tell customers, write it down.
Go deeper: Running production without a DevOps hire: a checklist for the first-time CTO covers the ten things to set up before the first incident.
Incidents are easier when the ops are not on you alone. DollarDeploy deploys your apps from GitHub to servers you own and keeps the day-to-day operations running: build history with one-click rollback, 30 days of logs and monitoring, host inspection, backups and notifications to the channel your team reads. Nothing for you to maintain. Start free on DollarDeploy, no card needed.