How We Saved 250 Hours a Month with Ticket Triage Automation (Full Breakdown)

I want to give you the full story of the ticket triage automation I built at my last company — not the LinkedIn summary version, but the actual breakdown: what the problem was, what I built, how it worked, and what the numbers actually showed at the end.

This is the case study that became the backbone of chapter 2 in Live Life Automated.

The Problem

We had a 5-person DevOps team supporting about 1,000 employees across 4 offices. Our ticketing system was receiving somewhere between 180 and 240 tickets per day, depending on the week. About 40% of those were infrastructure-related — the rest were software and user support.

The problem wasn't ticket volume. The problem was triage time.

We had specific roles for sorting, labeling, prioritizing, and routing tickets. This included:

• Reading ticket descriptions and categorizing them (network, storage, auth, deployment, etc.)
• Checking whether the ticket had enough information to act on, and if not, sending a templated follow-up request
• Routing to the right sub-team
• Setting priority based on a combination of the submitter's reported severity and our own criteria
• Checking for duplicate or related tickets and linking them

Manual, repetitive, boring — and wildly inconsistent. Different engineers triaged differently. Priority levels were applied inconsistently. Routing errors were common.

We were also doing this 7 days a week because tickets didn't stop on weekends.

What I Built

The solution was a Python-based triage service that ran as a systemd daemon and processed new tickets every 15 minutes. Here's the architecture:

Step 1: Ingestion

New tickets came in via our ticketing API. The service polled for tickets in "New" status and pulled them into a local processing queue.

Step 2: Classification

I trained a simple text classifier (sklearn, TF-IDF features, logistic regression) on 6 months of historical tickets with their final categories. Training accuracy was around 87%. For anything below a 70% confidence threshold, the ticket got flagged for human review rather than auto-classified.

The classifier covered 14 categories: network, storage, auth, compute, deployment, monitoring, database, security, user-access, hardware, vendor, billing, documentation, and "other."

Step 3: Information completeness check

Each category had a set of required fields. A "network" ticket needed: affected system(s), symptom description, when it started, and whether it was affecting multiple users. If any required fields were missing, the service automatically posted a comment requesting the missing info and moved the ticket to "Pending Customer" status.

This alone eliminated about 35% of the manual back-and-forth.

Step 4: Priority scoring

Priority was calculated from a weighted formula:

• Submitter's reported severity (weight: 0.3)
• Number of affected users (weight: 0.4)
• Category base priority (some categories are inherently higher priority — security always gets P1 floor) (weight: 0.3)

Step 5: Routing

Each category mapped to a sub-team queue. The service pushed the ticket to the right queue and assigned it to the on-call engineer for that sub-team.

Step 6: Duplicate detection

Before finalizing, the service checked for open tickets with similar summaries (cosine similarity > 0.8) and linked them as related. Engineers reviewing duplicate-linked tickets could quickly determine if they were the same issue.

The Code

Here's the core of the classification and routing step, simplified:

import requests
import json
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
import pickle
import logging

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s %(levelname)s %(message)s',
    handlers=[
        logging.FileHandler('/var/log/triage-service/triage.log'),
        logging.StreamHandler()
    ]
)
logger = logging.getLogger(__name__)

CATEGORY_ROUTING = {
    "network": "team-network",
    "storage": "team-storage",
    "auth": "team-security",
    "compute": "team-compute",
    "deployment": "team-devops",
    "monitoring": "team-devops",
    "database": "team-dba",
    "security": "team-security",
    "user-access": "team-helpdesk",
    "hardware": "team-hardware",
    "vendor": "team-procurement",
    "billing": "team-finance",
    "documentation": "team-devops",
    "other": "team-triage-queue"
}

CONFIDENCE_THRESHOLD = 0.70

def load_model(model_path: str):
    with open(model_path, 'rb') as f:
        return pickle.load(f)

def classify_ticket(model, ticket_text: str) -> tuple[str, float]:
    proba = model.predict_proba([ticket_text])[0]
    max_idx = proba.argmax()
    category = model.classes_[max_idx]
    confidence = proba[max_idx]
    return category, confidence

def process_ticket(ticket: dict, model) -> dict:
    text = f"{ticket['subject']} {ticket['description']}"
    category, confidence = classify_ticket(model, text)

    if confidence < CONFIDENCE_THRESHOLD:
        logger.warning(
            f"Low confidence ({confidence:.2f}) for ticket {ticket['id']} "
            f"— routing to human review queue"
        )
        return {
            "ticket_id": ticket['id'],
            "status": "needs_human_review",
            "confidence": confidence,
            "suggested_category": category
        }

    destination_queue = CATEGORY_ROUTING.get(category, "team-triage-queue")
    logger.info(
        f"Ticket {ticket['id']} classified as '{category}' "
        f"(confidence: {confidence:.2f}) → {destination_queue}"
    )

    return {
        "ticket_id": ticket['id'],
        "status": "classified",
        "category": category,
        "confidence": confidence,
        "destination_queue": destination_queue
    }

The systemd service unit:

[Unit]
Description=Ticket Triage Automation Service
After=network.target
Wants=network.target

[Service]
Type=simple
User=triage-service
ExecStart=/usr/bin/python3 /opt/triage-service/main.py
Restart=always
RestartSec=30
StandardOutput=journal
StandardError=journal
Environment=TRIAGE_CONFIG=/etc/triage-service/config.yaml

[Install]
WantedBy=multi-user.target

The Results

We ran the service in shadow mode for two weeks first — it processed tickets and logged what it would have done, but didn't actually change anything. We compared its routing decisions to the human triage decisions: 89% agreement.

We went live in week 3.

At the 90-day mark:

• Average triage time per ticket: 23 minutes → 8 minutes (for the ~15% that still required human review)
• Engineer morning triage rotation: eliminated entirely for routine tickets
• Routing accuracy: 91% (up from ~78% with manual triage)
• Information completeness requests: automated for 94% of cases
• Team time reclaimed: 250 hours/month across the 12-person team
• Throughput improvement on infrastructure tasks: 43% (measured by tickets resolved per sprint)

The 250 hours/month is the one I get asked about most. That's roughly 20 hours per engineer per month — half a week of capacity. In a team where every hour matters, that's significant.

We were able to cut down the number of engineers in that role significantly and repurpose those folks for higher level thought roles where they continue to flourish today!

What I'd Do Differently

A few things I'd change with the benefit of hindsight:

Start with rules, not ML. The first two weeks of the ML model were worse than a simple keyword-based classifier would have been. I should have shipped a rule-based version first and layered ML on top after I had more feedback data.

Build the feedback loop earlier. Engineers could override the service's decisions, but we didn't start logging those overrides and feeding them back into model retraining until month 4. That data was valuable and I left it on the table too long.

Alert on drift sooner. Category distribution shifted noticeably after a major product launch. The model didn't degrade catastrophically, but its accuracy dropped 6 points over two weeks before anyone noticed. A simple drift detector would have caught it faster.

The Takeaway

This wasn't a sophisticated ML project. It was a text classifier, a priority formula, and a routing table. The sophistication was in the design — understanding exactly what manual triage involved, which parts were truly automatable, and building appropriate fallbacks for the parts that weren't.

That's the pattern. Not clever algorithms. Careful observation, honest scoping, and then building the simplest thing that works.

The full breakdown of this project is in chapter 2 of Live Life Automated. Link on the site

Matt Fitzgerald is the founder of Fitzgerald Tech Solutions, author of Live Life Automated and Geeking Out, and a former DevOps lead. The Operator's Edge newsletter goes out every Monday.