From Signal to Shipped Fix,
Without Guessing

Before the architecture: a limit worth stating plainly, because designing around it is the difference between a system leadership trusts and a system that gets switched off after its first confident wrong answer.

The architectural consequence: server-side correlation is load-bearing, not supporting. The Dynatrace trace and the Splunk log are what localize the defect; Fullstory is what proves the defect mattered and sizes the money. Neither source alone authorizes a code change. This is why the recommended topology separates observation from inference, why the orchestrator requires two independent sources to agree before a case advances, and why autonomous PR generation begins on front-end presentation defects — the stay-date display class of bug, where the client-side evidence is the localization — and expands outward only as measured precision earns it.

Recommendation: a central Triage Orchestrator routing to four tool-scoped specialist agents, strict hub-and-spoke, no peer-to-peer calls. Not a monolith, and not a swarm.

Why not one monolithic agent

Four of these reasons are standard enterprise architecture. The first is specific to this toolchain and is, on its own, decisive.

• The session lease is a single-holder resource. Fullstory's visual inspection tools are stateful: a session is opened, inspected, and must be closed before another is opened in the same run. A monolithic agent that fans out across several sessions in parallel will leak or deadlock leases. Confining the lease to one single-threaded specialist with a TTL and a reaper makes close-safety an architectural property rather than a prompt instruction the model may forget under pressure.
• Tool-selection accuracy degrades with surface area. Fullstory MCP alone exposes 35 tools. Add Dynatrace, Splunk, Qualtrics, GitHub, Jira, Snowflake and Teams and a single agent is choosing among well over a hundred — in an incident context, under time pressure, where the wrong tool choice is a wrong answer. Each specialist here sees between three and eight.
• Separation of duties limits blast radius. The agent that can write to the repository holds no Fullstory, Dynatrace, or Splunk credential. The agent that gathers evidence cannot see code. No single compromised or drifting context can both fabricate evidence and act on it.
• Narrow contracts are testable. "Did the evidence agent correctly report the console error present in this session?" is a regression test. "Did the agent handle the incident well?" is not. Specialists can be evaluated and improved independently, which is what makes staged autonomy expansion possible.
• Model routing and cost. Multimodal session inspection is the expensive step; routing and classification are nearly free. Per-agent model selection lets Marriott spend reasoning budget where judgment actually lives, and swap models without rewriting the system.

Why not a peer-to-peer multi-agent swarm

• No agent calls another agent. All control flows through the orchestrator. This keeps the audit trail linear — one ordered case file per incident — and makes emergent loops structurally impossible rather than merely discouraged.
• Intake and routing are deterministic code, not inference. Webhook parsing, signature verification, dedupe, and threshold checks run as ordinary services. The model is invoked only where judgment is genuinely required, which shrinks both the hallucination surface and the bill.
• Specialist count stays fixed. Capability grows by widening a specialist's allowlist, never by spawning new agents at runtime.

Two trigger loops and one always-on monitor. Gold rows are mandatory human checkpoints. Red rows are stop conditions where the system is designed to decline rather than proceed.

Loop A  ·  Infrastructure / Ops (Dynatrace-triggered)

Target: ops signal to approved evidence package in minutes; signal to merged fix within the same business day. The X-FullStory-URL request attribute on the failing PurePath is the bridge that makes the whole loop possible — and its absence is the loop's first stop condition.

Loop B  ·  Voice of Customer (Qualtrics-triggered)

This loop differs from Loop A in one structural way, and the difference matters more than anything else in it: a single negative survey is an anecdote. Loop B will not run root cause analysis on a sample of one.

Loop C  ·  Always-on funnel vigilance

Loops A and B are both reactive — they wait for infrastructure to notice or a guest to complain. The highest-value failures are the silent ones that do neither.

A scheduled job recomputes the checkout funnels hourly against statistical control limits rather than fixed thresholds, so normal daily and weekly seasonality does not generate noise. A step-level drop outside the control band opens a case at step A1 of Loop A, bypassing the Dynatrace trigger entirely. This is the pattern that catches the failure class that never throws a server error and never produces a survey response: the rate card that renders empty, the date that silently reverts, the points balance that displays as unavailable. Early cases from this loop should run evidence-only, because without a server-side signature there is nothing to localize — which means Loop C feeds human investigation first, and earns code-generation rights last.

Ten controls. The first three are the ones that matter most, and none of them is a prompt instruction — they are enforced by deterministic code between the agents.

1  ·  The citation contract

Every assertion written into a case file must carry a provenance token: a session ID plus timestamp, a Dynatrace problem ID, a saved-search hash, or a metric ID plus the computed value. A deterministic validator sits between every agent hand-off and strips any claim without one. The stripped claim is logged — a rising strip rate is an early warning that an agent is drifting, and it is one of the metrics the review board watches.

2  ·  How Fullstory MCP grounds the reasoning

"Use MCP as ground truth" is a slogan until it is four specific mechanisms. These are the four.

• Confirm the definition before computing it. The quantification agent must read a funnel, segment, or metric definition and state in the case file what it is about to measure, then compute. This single discipline eliminates the most common analytical error: reporting "checkout drop" from an object that measures something adjacent to checkout.
• Named-object allowlist. Agents may reference only objects that exist in the org — the checkout conversion funnels, the confirmation and rate-list pages, the sign-in failure segment, the watched elements already active on critical front-end error messages. If the object required to answer the question does not exist, the agent must say so and stop. Approximating a missing object is the mechanism by which an agent invents confidence.
• Visual and DOM verification. The accessibility tree and screenshot give falsifiable client-side truth. A claimed error message must be present in the captured DOM or visible in the frame. This turns "the agent believes guests saw an error" into a check that either passes or fails.
• Methodology persistence against MCP non-determinism. The analytics MCP routes through a reasoning intermediary, so the same question asked twice can resolve to different query strategies. Left alone, two agent runs on one incident produce two different dollar figures — and nothing destroys executive trust faster. Each case therefore persists the object IDs, the original query strings, the time window, and the exclusions applied. Re-runs use the saved IDs with a drift check; if a saved ID returns empty, the agent reports drift rather than silently rebuilding.

3  ·  Instrumentation fragility is a first-class outcome

The control is a pre-flight instrumentation health check on every case: are the elements and pages this case depends on being captured stably across the window under examination? If not, the agent returns a tracking cause, not a behavioral one, and routes to the capture-governance queue. This must be a rewarded outcome in the agent's evaluation, not a failed run. A system that can only ever find product defects will find product defects in instrumentation gaps.

4  ·  Confidence floors and abstention

• Cohort floor. No dollar figure below 100 sessions. Below the floor, the finding ships as directional with the sample size stated and no monetary estimate attached.
• Corroboration floor. Two independent sources minimum, as above. For Loop B, two of three.
• Reproduction floor. No PR without a regression test that fails before the change and passes after. This is checked by CI, not asserted by the agent.
• Abstention is success. "Insufficient evidence, escalating" is a valid terminal state and is measured as such. An agent whose abstention rate is near zero is not accurate — it is hallucinating, and the metric will show it.

5  ·  Destructive-action prevention

Capability-scoped at the platform layer. A prompt that says "do not merge" is a suggestion; a GitHub App without merge permission is a guarantee.

6  ·  Prompt injection through guest-supplied text

• No agent holds a credential. An agent holding a Jira or GitHub token is one successful injection away from an attacker holding that token. Agents hold capability references and a broker holds the secrets, so a compromised context can do exactly what it was granted and nothing else. This is the strongest available mitigation and it is architectural, not textual — see 4.3b.
• All survey verbatims and all captured DOM content are wrapped and labelled as untrusted data, never as instruction, at every hand-off.
• The classifier emits a label from a closed taxonomy — it does not generate free text downstream.
• The Remediation Engineering Agent never receives raw verbatim or raw DOM text. It receives only the structured, human-approved finding. The agent with write capability is the agent furthest from untrusted input.
• Outbound tool calls are allowlisted by host. An injected instruction has nowhere to send anything.

7  ·  Human-in-the-loop checkpoints, stated plainly

8  ·  Observability of the agents themselves

• Every prompt, tool call, response, and decision streams to Splunk as structured events under the existing retention policy, threaded by one correlation ID from Dynatrace problem through case, Jira key, PR number, and release.
• Case files land in Snowflake for longitudinal evaluation — this is the dataset that tells the board whether precision is improving.
• Dynatrace monitors the agent services as first-class services. The system that watches the booking path is itself watched.

9  ·  The evaluation harness Marriott already owns

Twelve quantified frictions have been identified this year and eight have been found and fixed, each with a human-ratified root cause and a human-ratified dollar figure. That is a golden regression set. Before any agent is trusted with a live case, it replays those twelve incidents from their original signals and is scored on two things: did it reach the same root cause, and did it land inside a tolerance band of the ratified number. This converts "do we trust the agent" from a matter of opinion into a measurement, and it is the single highest-leverage thing to build in the first sixty days.

10  ·  Graceful degradation

• Fullstory unavailable: Loop A continues to ops-only triage with a flagged evidence gap. It does not proceed to quantification or code.
• Model endpoint degraded or rate-limited: cases queue durably. Nothing is dropped, nothing is retried into a duplicate.
• Reconciliation repeatedly failing across cases: the orchestrator trips a global breaker and reverts the whole system to human triage. The system is designed to fail into the current process, which already works.

No agentic runtime has been chosen yet. That is an advantage, not a gap — these two loops are a sharp enough specification to drive the decision, and they are a far better selection test than a generic pilot would be.

4.1  ·  Six requirements that should drive the decision

Most agent-platform evaluations start from vendor feature matrices and end in a tie. Starting from what these loops actually demand eliminates most of the field on the first pass.

4.1b  ·  Three compute tiers — why each exists

These loops require three distinct layers of compute. Conflating them is the most common platform selection error.

• Durable execution engine (Temporal, Step Functions, Cloud Workflows, or Durable Functions): holds the case state machine across multi-week pauses, human-approval gates, redeployments, and model failures. This is the control plane — it does not run the agents, it orchestrates their invocation and records every state transition for audit. Without this layer, a case that pauses at HITL review loses its context the moment the container restarts.
• Container jobs (scale-to-zero): the agent compute tier. Session inspection, cohort quantification, and remediation all run as minutes-long jobs with real disk — the remediation agent clones a repository and runs a test suite; the session evidence agent walks up to five sessions in sequence. A functions-only substrate cannot host these. Container jobs that scale to zero when idle keep cost proportional to case volume.
• Lightweight functions: webhook receipt, HMAC signature verification, alert deduplication, and theme classification. These are deterministic, milliseconds-long, high-volume operations that require no model call. Running them in the durable engine or in a container job wastes capacity and adds latency at the ingest boundary.

The table below maps each layer to the cloud-specific implementation options. Marriott's cloud decision will be driven by commercial terms and existing operational muscle — the capability requirements above are cloud-neutral and should anchor the evaluation regardless of which path is chosen.

4.2  ·  Candidate stacks

The cloud decision will most likely be made on grounds that have nothing to do with agents — existing gravity, commercial terms, where enterprise architecture already has operational muscle. Marriott's current estate gives weak and conflicting signals: Snowflake is cloud-neutral, and the Gemini reference on the agentic platform slide hints Google without establishing it. So rather than pick a cloud, here is what each path looks like and where each one is weak for this workload.

4.3  ·  The two decisions that actually require deliberation

Most of the table above resolves itself once a cloud is picked. Two choices do not, they are genuinely portable, and they carry the most consequence. These are where we would spend the architecture group's time.

4.3b  ·  The agent-to-system control plane

Requirement three in 4.1 — per-agent credential scoping — is the one no cloud platform answers completely, and it is the requirement the entire separation-of-duties argument rests on. We recommend solving it with a dedicated control plane rather than waiting for a cloud primitive to mature.

The obvious benefit is arithmetic: five agents needing six systems is thirty integrations, each with its own credential, audit gap, rate limit and error handling. A broker makes that N + M instead of N × M. That collapse is worth having but it is not the reason to adopt it. The reason is that it turns three controls this proposal describes as design intent into properties the infrastructure actually enforces.

What it changes in the loops

Sections 2 and 3 specify that intake, deduplication and theme classification must be deterministic code rather than model inference, precisely so those steps present no injection surface. That primitive has a name here: a reflex — a deterministic event-to-action rule that fires with no model in the loop, with its condition checked against the schema at boot, its own firing budget, and shadow mode as the default so the dangerous option is never the easy one. A reflex is a principal and takes the identical enforcement path an agent does, so it is not a bypass.

• A0, B0 and B1 — webhook verification, signature-hash deduplication, and closed-taxonomy theme classification — are reflexes rather than bespoke services.
• Loop C, the hourly funnel check against control limits, becomes a scheduled poll feeding a reflex. Durable cursors matter here: a poll cursor survives kill -9, re-delivers at most one event, and names a window it could not re-read rather than silently skipping it. A monitor that loses an hour without saying so is worse than no monitor.
• Shadow mode as default maps directly onto the phased rollout. Phase 4a can run the full reflex set in shadow — recording what each rule would have fired — which is a materially better Phase 4a than one that simply has the capability switched off.

What Marriott should probe before committing

• Connector coverage is the practical gating question. These loops need Dynatrace, Qualtrics, Splunk, GitHub and Snowflake alongside Fullstory. Confirm which connectors exist today versus which are scoped work, because a missing connector is either an integration to build or a system that sits outside the governed path — and a system outside the path defeats the purpose.
• Maturity. It reached v1 recently. Ask for production references, the results of any external hardening review, and what the upgrade cadence looks like.
• Who operates it. MIT licensing means Marriott can run it; it also means Marriott must run it. Decide whether that sits with platform engineering or security, and what the support path is when it refuses to start — which it will do by design rather than hold a credential it cannot rotate or audit.
• Secret-manager integration is a prerequisite, not a follow-on. Credentials arrive from a mounted file under a declared root, from platform workload identity, or from OAuth. That integration has to be settled before the Phase 4a pilot, not during it.

4.4  ·  What we would deliberately not buy yet

• A managed agent runtime as the foundation. The hosted agent services from all three clouds are credible and worth a look — several are framework-agnostic, so they are closer to runtimes than to frameworks. But they are optimized for conversational and single-agent shapes, their built-in session and memory models overlap awkwardly with durable execution plus the citation contract, and agent-native identity — the requirement that matters most here — is the least mature thing any of them ships. Pilot one. Do not found the program on one.
• An eval platform on the critical path. The golden-set scoring is plain code against Snowflake, where case files already land. Tracing UX is worth buying once the team feels the pain of not having it — not in the first sixty days, and not as a dependency of the pilot.
• A dedicated vector store. Nothing in either loop is a retrieval problem. Evidence arrives by ID from Dynatrace and Fullstory; quantification runs on governed SQL. If a vector database appears in a proposed architecture for this workload, ask what question it answers.
• GPU capacity. Managed in-region inference with customer-managed keys and no-retention terms covers it. Capacity planning for bursty incident traffic is a poor use of capital, and the self-hosted tier we do recommend — theme classification and signature dedupe — is small enough to run on CPU.

4.5  ·  Security and privacy posture — cloud-independent

None of this changes with the platform decision, so it can be ratified with security and architecture in parallel with the selection rather than after it.

4.6  ·  Why this should be the first agent Marriott builds

If the platform decision is open, the choice of first workload is a decision about what the platform has to prove. We would argue for this one, on four grounds — and then give the honest counterargument.

• It has a hard, pre-existing success criterion. Most first agents fail quietly because nobody can say whether they worked. This one is measured against 48 hours to containment and twelve already-ratified frictions. The verdict is arithmetic, not opinion.
• Phase 4a is read-only. No write capability for the first sixty days, so the blast radius during platform shakedown is close to zero. You are load-testing a runtime with an agent that cannot break anything.
• It exercises every hard capability the platform will ever need. Multi-week durable pauses, human gates, stateful tool leases, per-agent credential scoping, multimodal reasoning, write-capable tooling under blast-radius control, and an end-to-end audit trail. A conversational pilot exercises roughly one of those. Anything Marriott builds after this one inherits a runtime that has already been proven against the difficult case.
• It pays for the platform. Recovered revenue is the funding argument for everything downstream, and this is the only candidate first workload that produces a dollar figure as its native output.

Autonomy expands as a function of measured precision, never by calendar date. Each phase must clear its metrics on the golden regression set before the next unlocks.

The metrics that govern the program

Governance

We recommend a standing Agent Review Board with representation from Global Digital Technology, Digital Innovation, Experimentation & Analytics, capture and instrumentation governance, and enterprise security and architecture. It owns four decisions and only four: scope expansion, methodology change, phase advancement, and incident review on any agent-attributable defect. It meets on the existing cadence rather than creating a new one, and it is the natural first charter for the Center of Excellence the account audit already recommends.

Open risks, stated plainly

• Behavioral evidence does not localize backend defects. The Ops Correlation Agent is load-bearing for anything server-side. Where the bridge attribute is missing, the system has no localization path and will decline the case. Loop A's code generation therefore starts on front-end defects by design, not by caution.
• MCP non-determinism. Mitigated by methodology persistence and drift checks, not eliminated. Published figures must be treated as reproducible only when accompanied by their persisted methodology.
• Instrumentation drift. Until data-attribute and selector standards are consistent, a share of cases will correctly resolve as tracking causes. This is the system working, and it needs to be framed that way to leadership before the first such case lands.
• Lower-environment instrumentation does not exist today. Loop B's closed validation step is contingent on it. Until then, Loop B ends at a human-approved Jira issue.
• Guest-supplied text is an attack surface. Mitigated by data-labelling, closed-taxonomy classification, egress allowlisting, and keeping the write-capable agent away from raw input. Worth an explicit security review rather than a footnote.