architecture-and-performance · diff
git:20260828.5f76990 to git:20260906.e6ad4bd
60 added, 144 removed. Audit A to A.
---
name: architecture-and-performance
description: >
- Reasoning about performance across the whole request path rather than pattern by pattern:
- budgeting latency from controller to database, counting round trips as the primary
- currency, and knowing which architectural changes actually move a number. Use when a page
- is slow and each layer looks fine in isolation, when a pattern is chosen or rejected on
- performance grounds without a measurement, when adding a layer or a service is proposed as
- a fix, when query counts scale with result size, when a remote interface is chatty, when a
- transaction holds a connection across a network call, or when a load test passes and
- production does not. Does not cover the investigation process (performance-methodology),
- profiling tools (jfr-and-async-profiler), JVM behaviour (java-performance), queue
- arithmetic (littles-law-and-queueing), collector and heap tuning (jvm-gc-tuning), or
- designing and running the load test itself (load-testing).
+ Attribute endpoint latency and throughput limits to architectural choices when query counts
+ grow with result size, remote calls are chatty, connections are held across other work,
+ or a cache, layer removal or service extraction is proposed as a performance fix.
+ Compare fetching, call topology, resource occupancy and data movement across the whole
+ request path. Does not replace investigation methodology (performance-methodology),
+ profiling (jfr-and-async-profiler), individual SQL tuning (sql-query-performance),
+ pool configuration (connection-pool-sizing) or load-test construction (load-testing).
---
# Architecture and Performance
- ## Purpose
-
- Attribute latency to architecture. Most enterprise performance problems are not slow code:
- they are a request doing more work than it needs to, across more boundaries than it needs
- to cross, in more round trips than anyone counted. Optimising a pattern in isolation
- answers the wrong question; the useful unit is the whole path from request to response and
- back.
-
- The failure this prevents is the local optimisation that changes nothing — a faster mapper
- on a path whose 400 ms is 180 database round trips, or a cache added in front of a query
- that was never the bottleneck.
-
- ## The request path is the unit
-
- ```text
- Client
- └─ HTTP framework parse, bind, validate, filter chain, security
- └─ Controller mapping in
- └─ Application service transaction begins ← connection acquired here
- └─ Domain rules; how much state must be loaded?
- └─ Repository how many queries, and are they in a loop?
- └─ ORM flush, dirty check, lazy loads, identity map
- └─ Pool wait time when saturated
- └─ DB execution, locks, index use
- ← mapping out DTO construction, serialisation, response size
- ```
-
- Two numbers explain most enterprise latency: **round trips** (database and remote) and
- **transaction duration**. Everything else — mapping cost, object allocation, serialisation
- — matters after those two are right, and rarely before.
+ The unit is the operation from arrival to its promised completion, including work deferred
+ to other services when that completion requires it. Attribute cost to concrete choices:
+ how much data is fetched, how often a boundary is crossed, which calls must wait for others,
+ and how long scarce resources remain occupied. A faster mapper cannot remove database waits;
+ fewer queries cannot remove measured serialization CPU.
## Workflow
- 1. **Budget the path before measuring.** Write the expected count of database queries and
- remote calls for the operation. A number written in advance turns "it feels slow" into a
- falsifiable claim.
- 2. **Count what actually happens.** Query counts per request from the ORM statistics or a
- proxy; remote calls from client metrics; both attributed per endpoint, not aggregated.
- 3. **Find the multiplier.** If the count scales with result size, you have an N+1 — the
- dominant architectural performance defect in enterprise applications, in both the
- database and the remote-call forms.
- 4. **Measure transaction duration**, not just query duration. A 5 ms query inside a 900 ms
- transaction is a connection held for 900 ms, and the pool is the real constraint.
- 5. **Attribute the remainder by layer** before optimising anything: framework, mapping,
- domain, persistence, database. Optimising the wrong layer is the norm, not the
- exception.
- 6. **Change one architectural thing and re-measure the same budget.** Verify by mechanism —
- the query count should drop to the predicted number, not merely "get better"
- (`performance-methodology`).
-
- ## Decision rules
-
- ```text
- Query count scales with the number of rows displayed
- → N+1. Fix by fetching what is needed in one query or a bounded
- number (jpa-fetching-and-n-plus-one style: join fetch, entity
- graph, or a projection). Do not fix it with a cache.
-
- Query count is constant but large (30+ for one page)
- → the read path is going through the write model. Use a
- projection or a query object, not the aggregate
- (query-objects-and-specifications).
-
- Latency is dominated by remote round trips
- → coarsen the interface (remote-facade-and-dto), parallelise
- independent calls, or replicate the data locally
- (distribution-boundaries). A faster serialiser will not help.
-
- Connection pool wait time is non-zero
- → transactions are too long, or the pool is smaller than
- concurrency demands. Shorten the transaction first; sizing
- second (connection-pool-sizing).
-
- Throughput is flat while CPU is low and the database is idle
- → something is serialised: a lock, a pool, a synchronised block,
- a single-threaded executor. Find the queue
- (littles-law-and-queueing).
-
- The proposal is "add a cache"
- → establish h × T_source first, and establish that the source is
- the bottleneck. A cache over a fast query on a cold path is
- pure risk (caching-strategies).
-
- The proposal is "extract a service to scale it"
- → the bottleneck is usually the shared database, which
- extraction does not move. Measure which resource is saturated
- before splitting a process.
+ 1. **Define the operation and workload.** Obtain the latency/throughput objective, measurement
+ boundary, request mix and rate, page/payload sizes, data distribution, concurrency and cache
+ state. Identify the relevant code, SQL/call graph and deployment/framework versions.
+ Without measurements, produce a conditional cost model and ask for the smallest evidence
+ needed to distinguish options; do not announce a bottleneck or predict an empirical speedup.
+ 2. **Map work and waits.** Record query/remote-call counts as functions of input size, rows
+ and bytes returned, serial versus parallel dependencies, retries, queue waits and resource
+ hold intervals. Compare expected work with observed work. Read
+ [request-path-budget.md](references/request-path-budget.md) when attributing a slow path,
+ estimating occupancy or checking whether measurements support a proposal.
+ 3. **Locate the multiplier or limiting resource.** Correlate representative normal and slow
+ requests with SQL, pool, downstream and CPU evidence. A growing call count suggests a
+ repeated-access problem; low CPU or any nonzero pool wait alone does not identify its cause.
+ Keep observation, hypothesis and recommendation distinct. State what would refute the
+ proposed architectural explanation.
+ 4. **Compare the smallest adequate change.** For fetching, persistence boundaries, mapping,
+ locking or remote-call alternatives, read
+ [persistence-cost-model.md](references/persistence-cost-model.md). Predict the mechanism:
+ calls removed, rows avoided, occupancy reduced or critical-path work overlapped. Compare
+ against keeping the design and fixing a local query/implementation issue.
+ 5. **Validate the mechanism and outcome.** Preserve authorization, invariants, consistency,
+ pagination and failure behavior. Re-run comparable work and check both the predicted
+ mechanism and end-to-end latency, achieved throughput and errors. Delegate experimental
+ design to `performance-methodology`; code/query-budget tests do not establish a production
+ latency improvement.
- Load test passes, production does not
- → the test's data volume, cardinality, cache state or
- concurrency shape differs. Compare query plans on production
- -shaped data, not on a seeded 100-row table.
- ```
+ ## Decision constraints
- ## Rules
+ - Count round trips, but also work per trip. One huge join can cost more than several bounded
+ queries. There is no universal query-count threshold proving an architectural defect.
+ - Separate time on the critical path from total resource consumption. Parallel calls can
+ shorten elapsed time while increasing downstream concurrency, queues and failure exposure.
+ - Distinguish method duration, transaction lifetime, connection hold time and lock hold time.
+ Changing one does not guarantee that the others change.
+ - Before adding a cache, quantify source demand and representative hit/miss behavior, lookup
+ cost, freshness requirements and miss amplification. A cache can reduce saturated-source
+ load, but cannot repair incorrect fetching or make stale reads acceptable by itself.
+ - Before extracting a service, identify the resource or placement advantage that extraction
+ creates and the new network/serialization/coordination cost. Neither “distribution scales”
+ nor “distribution can never reduce latency” is a supported conclusion without this model.
+ - Do not remove a useful layer or weaken a consistency boundary based on a pattern's reputation.
+ State the observable cost and prove required behavior survives the adjustment.
- - **Round trips are the currency.** A round trip costs a fixed overhead — network,
- parse, pool, ORM bookkeeping — that dwarfs the work in most enterprise queries. Reducing
- 180 queries to 3 beats making each of the 180 twice as fast, by an order of magnitude.
- - Architectural patterns have predictable query costs, and those costs are knowable before
- writing the code. Class Table Inheritance costs a join per level; Lazy Load costs a query
- per traversal; an aggregate costs whatever its boundary spans
- (`persistence-cost-model.md`).
- - **Transaction duration, not query duration, sizes the connection pool.** The pool is
- sized by concurrency × transaction duration; the single most effective pool fix is
- usually removing work from inside the transaction.
- - A layer costs mapping and allocation, not architecture-scale latency. Do not remove a
- layer for performance without measuring — but do count the mapping when a single request
- maps the same data four times across six layers, because that is a real number at
- scale.
- - Distribution adds latency it never removes. Extracting a service moves work across a
- network; it improves latency only when the extracted work was contending for a resource
- the caller needed, and it usually is not.
- - Tail latency, not average, determines the user's experience and the system's stability
- under load. An average of 80 ms with a p99 of 3 s is a system that fails under
- concurrency, and averaging hides it (`latency-statistics`, `tail-latency-analysis`).
- - Measure with production-shaped data. Query plans change with cardinality; a nested loop
- that is optimal at 100 rows is catastrophic at 10 million, and no amount of testing on a
- seeded database finds it.
- - Do not benchmark a pattern. Benchmark the endpoint, with the real schema, the real
- volume and the real concurrency; the patterns' differences are almost always smaller than
- the differences between those conditions.
- - Serialisation and payload size matter at the network edge, where they are frequently the
- largest single component of a slow API response — an endpoint returning 4 MB of JSON is
- not fixed by a database index (`serialization-performance`).
- - Write the budget into a test. A test that fails when an endpoint exceeds its query budget
- catches the reintroduced N+1 on the day it is written, not in the incident review
- (`architecture-testing`).
+ ## Minimum deliverable
- ## References
+ For a small review, report evidence or gap, architectural hypothesis, consequence, proposed
+ adjustment and a confirming/refuting check. For a design, add the workload assumptions,
+ cost/budget comparison and correctness constraints. Separate measured results from estimates
+ and tests still to run; leave unsupported conclusions open.
- - [Budgeting the request path](references/request-path-budget.md) — building a latency and
- round-trip budget per endpoint, attributing time by layer, the measurements that
- discriminate between layers, transaction duration and pool arithmetic, and the load-test
- conditions that make a result transferable to production. Read when a path is slow and
- the cause is not yet located.
- - [What each pattern costs](references/persistence-cost-model.md) — the query and call cost
- of the persistence and distribution patterns: lazy versus eager, identity map hits,
- inheritance strategies, aggregate size, DTO mapping, chatty remote interfaces, with the
- arithmetic and the fix for each. Read when choosing a pattern, or when explaining why an
- existing one is expensive.
+ When reviewing this skill's behavior, use
+ [validation-cases.md](references/validation-cases.md), which defines reproducible prompts and
+ failure conditions rather than tests of application code.