ATPL

AI · Start right

Keep AI accurate after it goes live.

For companies running AI features built by anyone: a service that watches accuracy, cost and failures every day, catches drift and model changes, and keeps the feature as good as the day it launched.

Describe the problemAll AI capabilities

Live but unwatched

Shipped AI that is live but unwatched

Monitoring, evaluation and cost control for AI already in production.

Describe the problem

  1. Drift

    The chatbot was great at launch; six months on, nobody knows whether it still is.

  2. Cost

    The provider updated the model and the answers changed overnight.

  3. Silent

    The monthly API bill doubled and no one can say which feature did it.

How it ships

Watch, alert, improve

  1. ops/01

    Understand

    We read the existing feature’s code, prompts and logs, whoever built it, and find out what is measured today; usually nothing.

  2. ops/02

    Design

    We agree the measures, the alert thresholds, the spend caps and the review cadence, and build the evaluation set from real past cases.

  3. ops/03

    Build

    Logging, the nightly evaluation run and the dashboard are added around the feature without changing its behaviour.

  4. ops/04

    Stay

    Monthly review with your team, incident response when a provider changes or an alert fires, and a quarterly cost and quality report.

Your data

Logs, evaluation sets and dashboards live in your infrastructure, under your accounts. ATPL staff access them under a named login that you can revoke, and see only the features in scope. Nothing in the monitoring pipeline sends your data to a model provider beyond the calls the feature itself already makes.

Ops or rebuild

Where it fits, where it doesn’t

  • Fits when
  • Any AI feature in production, whether ATPL built it or someone else did.
  • Teams that want a number for accuracy and cost each month instead of a feeling.
  • Features exposed to customers, where a silent regression costs trust.
  • Doesn’t fit when
  • Features still in prototype; finish the proof of concept first, then operate it.
  • AI inside a vendor’s closed product where you cannot see the calls; take it up with the vendor.
  • A one-time audit with no follow-through; drift returns, so monitoring has to stay.

Stay after launch

Monitoring, evals and a person

  1. 01

    Monitoring in place

    Every model call logged with its cost, latency, input size and outcome, in a dashboard your team reads, with alerts on the limits you set.

  2. 02

    Evaluation set and nightly runs

    A fixed set of real cases with known good answers, run automatically against the live configuration, so a regression is found before a customer finds it.

  3. 03

    Drift and feedback review

    Sampled outputs reviewed monthly with your team, user corrections and thumbs-down collected, and failing cases added to the evaluation set.

  4. 04

    Cost control

    Caching, routing of easy cases to cheaper models, spend caps per feature, and a monthly report of cost per feature and per call.

  5. 05

    Version pinning and change process

    Model versions, prompts and retrieval indexes are versioned; nothing changes in production without passing the evaluation set.

How we work

ATPL has supported Ethos in production for restaurants since 2013 and brings the same habit of watching, measuring and fixing to AI features. ATPL will confirm which AI systems it operates for clients.

    Chosen for the job

    Node.js · Python · Next.js

    Decide the shape

    Before you start

    Plain answers on scope, data and what a first proof usually takes.

    AI operations, sometimes called LLMOps or MLOps, is the work of keeping an AI feature accurate, affordable and available after launch: logging every call, running an evaluation set against the live system, watching for drift and provider changes, and controlling cost. ATPL offers it as a monthly service for AI built by anyone.

    Usually because the world changes while the model does not: new products, new phrasing from customers, a provider updating or retiring a model version, or a prompt edited without a test. ATPL catches this with a fixed evaluation set run nightly and version pinning, so a change is seen the day it happens.

    Log the cost of every call per feature, cache repeated context, route easy cases to a cheaper model, cap daily spend so a loop or an attack cannot run up the bill, and review the monthly report. ATPL sets all of this up as the first step of AI operations, usually within the first week.

    Yes. ATPL takes over AI features built by freelancers, agencies or in-house teams, starting with a read of the code, prompts and logs, then adding logging, an evaluation set and a spend cap before changing anything about how the feature behaves.

    Continue to