Phrases such as “the payment must not be processed twice,” “the API needs to remain available,” or “order confirmation should be fast” are important requirements. But on their own, they still do not tell us which architecture to use, which infrastructure to declare, or which behavior to observe in production.
Before choosing a queue, Kubernetes, a cache, a database, or any other tool, we need to understand the ground we are standing on. Does this application serve a few people or thousands? Is it new, or does it already have a bottleneck? Does it depend on an external service? How long can it remain unavailable? In the end, context matters.
The point of this article is to advocate for a mental model: turning business requirements into technical intentions for distributed systems. In other words, describe which condition the system needs to sustain, under which constraints, how it appears in the environment, and how we will notice when it is no longer true.
The best solution is not necessarily the most complex one. It is the one that solves the problem simply, keeps the system reliable, and does not become yet another thing that is difficult to maintain, fix, and explain later.
TL;DR
- A business requirement is not the same as a technical decision.
- “Do not charge twice” can become a technical intention such as: “the same payment request must produce at most one confirmed charge.”
- A policy defines limits and conditions: scope, deadlines, authorization, failure tolerance, and acceptable degradation.
- Desired state is a declared way of representing a condition that the environment must maintain.
- Observability, testing, and reconciliation close the loop: without validation, an intention is just a nice sentence.
- Terraform and Docker Compose help declare expected environments; Kubernetes and Intent-Based Networking make the cycle of observing current state and acting on deviations more explicit.
- The main question is not “which technology do we use?” but “which system condition do we need to sustain—and how will we prove it?”
Technical intentions for distributed systems: why start with expected behavior
A common mistake in architecture discussions is jumping straight from a requirement to a tool.
Product says:
“The payment must not be processed twice.”
And the conversation immediately becomes:
“So, do we need a queue?”
“Would Kubernetes solve it?”
“Add a distributed lock?”
“What if we use a different database?”
Maybe one of these options makes sense. Maybe none of them does. The problem is that the conversation started in the wrong place.
Before deciding on the mechanism, we need to understand exactly what must be true in the system. There is an important difference between:
- business outcome: prevent the customer from being charged twice;
- technical intention: ensure idempotent processing within a defined scope;
- policies and constraints: define deadlines, limits, expected failures, and acceptable degradation;
- desired state: represent the condition that the environment must sustain;
- validation: observe whether the expected behavior continues to occur.
Tools come after that. A queue can help with asynchronous processing, but it does not define what to do with duplicate messages by itself. Kubernetes can recreate a pod, but it does not know whether a charge was made twice. A cache can reduce latency, but it can also expose stale data if the real problem is consistency.
The order matters because a tool without a clear intention tends to create complexity that is difficult to justify. Sometimes the system needs a more robust platform. In other cases, it needs a better-modeled domain rule, a unique index, a clearer log, or a reconciliation routine.
Payments: idempotency, retries, and reconciliation
Let us use a fictional payment example. It does not represent a real company, but it helps make common problems in distributed systems visible: idempotency, partial failures, redelivery, timeouts, retries, intermediate states, and reconciliation.
An initial requirement could be:
“The payment must not be processed twice.”
This is understandable from a product perspective, but it is still too broad to guide a technical decision. We can translate it as follows:
| Layer | Example |
|---|---|
| Business requirement | The customer should not be charged twice for the same order. |
| Technical intention | The same payment request must produce at most one confirmed charge. |
| Technical constraint | Retries, timeouts, and communication failures must not generate a new charge for the same operation. |
| Desired state | Equivalent operations are recognized and handled consistently. |
| Possible mechanisms | Idempotency key, attempt record, unique index, transaction state control, reconciliation with the payment provider. |
| Validation | Monitor repeated attempts, conflicts, orders without charges, charges without a corresponding order, and discrepancies between systems. |
Notice that we still have not chosen a stack. We did not say “it has to be Kafka,” “it has to be Kubernetes,” or “it has to be database X.” First, we defined the behavior we want to protect.
It is also worth being careful with the expression “exactly once.” In distributed systems, especially when external integrations are involved, this promise can be difficult, expensive, or inappropriate depending on the system boundaries.
In practice, many solutions combine:
- idempotency: repeating the same operation should not cause a duplicate effect;
- deduplication: recognize repeated messages or requests;
- explicit intermediate states: payment started, pending, confirmed, expired, or failed;
- reconciliation: compare states between systems and correct discrepancies.
The trade-off is that more protection against duplication usually requires more storage, expiration rules, handling for incomplete cases, and decisions about what to do when a request gets stuck “halfway through.” None of this comes for free.
Availability, performance, security, and reliability
Some requirements frequently appear in digital products: availability, response time, security, and reliable processing. The challenge is not to reduce architecture to isolated phrases such as “it has to be fast” or “it needs high availability.”
These phrases need to become observable behavior.
Availability
Product requirement:
“The API must remain available if one instance fails.”
A more useful technical intention would be:
“The service must not depend on a single instance to serve critical routes.”
The desired state could be:
“Even after losing one execution unit, there is still enough capacity to serve the main operations.”
Validation may involve error rate, latency, remaining capacity, simulated failures, and the impact perceived by users. The Google Cloud Architecture Framework reliability material is a good reference for thinking about systems that continue operating in the face of failures.
But there is a trap: declaring replicas does not, by itself, prove that the API is available to customers. They may all depend on the same unavailable database. They may all be in the same zone. The load balancer may not be routing correctly. The health check may say “OK” for an application that cannot complete a real operation.
Configuration helps, but it is not a guarantee by itself.
Response time
Product requirement:
“Order confirmation must be fast for the user.”
“Fast” is a dangerous word. Fast for whom? For which operation? Under what load? At which percentile?
A more useful technical intention would be:
“The order confirmation operation must respond within an acceptable limit for most users, considering the expected request volume.”
This brings in decisions such as:
- which operation we are measuring;
- which percentile matters;
- under which load scenarios;
- which limit makes the experience acceptable;
- what can be done asynchronously afterward.
Not every operation needs to be optimized to the same level. Perhaps the user interface needs a fast confirmation, while email delivery, inventory deduction, or report generation can happen later. The point is to separate what affects the immediate experience from what can be processed safely at another time.
Security
Product requirement:
“Only authorized users may access financial documents.”
A technical intention could be:
“Every access to financial documents must be authenticated, authorized, and traceable according to the domain rules.”
This changes the conversation. Instead of starting with “let us use tool X,” the team needs to answer:
- who can access which documents;
- under which conditions;
- how permissions are granted and revoked;
- which accesses need to be auditable;
- how to identify improper attempts;
- which data requires additional protection.
Choosing an authentication or authorization tool may be part of the solution, but it does not replace the policy. “Use product X” does not, by itself, define who can access what.
Reliable processing
Product requirement:
“The user should receive order confirmation even when a dependency is temporarily unavailable.”
This introduces partial failures, redelivery, timeouts, retries, temporary dependency unavailability, and eventual consistency.
A technical intention could be:
“The system must durably record the purchase intent, inform the user about the order state, and complete or reconcile dependent steps when external services become available again.”
This does not necessarily mean “add a queue.” It could mean a queue, database, periodic job, outbox, workflow, asynchronous processing, or a simpler solution, depending on the context. This article about asynchronous processing with Node.js and Google Cloud Platform aligns well with decisions involving slow or temporarily unavailable dependencies.
The main trade-off is straightforward: requiring synchronous confirmation at every step may simplify how the flow reads, but it tends to increase latency and reduce fault tolerance. On the other hand, deferring steps requires more care with intermediate states, user communication, reconciliation, and operational support.
From desired state to validation
To make this reasoning more concrete, it is useful to distinguish a few terms.
Intention is the intended technical outcome.
Example: “the service continues serving requests after one instance fails.”
Policy is the set of rules, limits, and conditions that constrain how this outcome may be achieved.
Example: “maintain minimum capacity, avoid concentration in a single point of failure, and do not accept traffic on unhealthy instances.”
Desired state is a declared representation of the condition that the environment must maintain.
Example: “there are enough replicas, distributed and ready to receive traffic.”
Current state is the condition observed in the system at a given moment.
Example: “two replicas are ready, one is restarting, and the error rate has increased.”
Reconciliation is the process of comparing the current state with the desired state and acting on differences.
Example: “create a new replica, remove an instance from load balancing, or trigger an alert.”
This model is especially useful in distributed systems because failures, restarts, network changes, and partial outages are not exceptions. They are part of the environment.
The trap is confusing configuration with a guarantee.
Declaring replicas, queues, retry policies, or CPU limits may be part of the desired state. But it does not prove that:
- the database can handle the load;
- the load balancer is working;
- the application is healthy;
- external dependencies are available;
- the user can complete the journey;
- the business rule is protected.
That is why the question needs to go beyond “did we declare the resource?” We also need to ask: “is the behavior that motivated this configuration actually being delivered?”
Declarative infrastructure, Kubernetes, and Intent-Based Networking
This mental model appears in several technologies, but with different responsibilities. It is important not to put everything in the same bucket.
Terraform and Docker Compose
Declarative tools help describe expected resources and properties. Instead of thinking only in terms of manual commands, we start declaring something like:
“This is the set of resources and configurations that should exist.”
In Terraform, the configuration describes desired resources, which helps version, review, and reproduce parts of the infrastructure. But a successful apply does not automatically mean that a reliability, security, or performance goal has been met.
Docker Compose also allows services, ports, environment variables, volumes, networks, and dependencies to be expressed. This is useful for local development, testing, and simple environments. However, describing local services does not replace operational policies, monitoring, fault tolerance, or production recovery strategies.
In short: Terraform and Docker Compose help declare the expected environment. They do not, by themselves, prove that the technical intention has been met.
Kubernetes and control loops
Kubernetes makes the reconciliation cycle more explicit through controllers.
Conceptually, a controller observes the cluster's current state and works to bring it closer to the desired state. The Kubernetes documentation explains this pattern on the page about controller architecture.
A common example is a Deployment. By declaring a Deployment, you express a desired condition, such as a number of replicas of an application that should be running. The Kubernetes Deployments documentation presents this resource as a way to manage applications and updates declaratively.
Even so, Kubernetes does not replace domain understanding. It can recreate a pod that has crashed, but it does not know whether a charge was duplicated, whether an authorization was applied correctly, or whether the user experience remains acceptable.
Controllers reduce the manual work of recovery, but the team still needs to define limits, probes, success criteria, critical dependencies, alerts, update policies, and rollback strategies.
Intent-Based Networking
Intent-Based Networking, or IBN, takes a similar idea to networks: instead of relying only on isolated configurations on devices, the proposal is to express the expected network outcome and verify that it continues to be met.
RFC 9315 presents a perspective on Intent-Based Networking as an approach for working with intents, policies, and validation in the context of networks.
Bringing it into this article's model:
| Layer | Network example |
|---|---|
| Intention | Specific applications can communicate with one another with the expected properties. |
| Policy | Segmentation, security rules, permitted connectivity, and performance constraints. |
| Desired state | Expected network behavior for those flows. |
| Validation | Continuous compliance checking and deviation detection. |
The point is not to say that every team needs to adopt an IBN platform. Nor is it to sell “autonomous” networking as a universal answer. The important connection is the mental model:
declare the desired outcome, observe the real state, and act when there is a deviation.
Automation without a clear intention only makes the wrong thing happen faster.
A practical guide for the next requirement
The next time a requirement appears such as “it has to be fast,” “it must not duplicate,” or “it needs to remain available,” try following this guide before choosing the technology.
-
Write the business need in product language.
Example: “the customer needs to receive order confirmation even if a dependency is temporarily unavailable.” -
Define the observable technical behavior that protects this need.
Example: “the system durably records the order, provides the user with an understandable status, and processes pending steps when the dependency returns.” -
Choose relevant metrics, limits, and failure conditions.
Example: maximum response time for the user, pending-order rate, orders without confirmation, integration failures, and reconciliation time. -
Record policies and constraints before selecting tools.
Example: which steps need to be synchronous, which can be asynchronous, which data requires stronger consistency, and which states need to be auditable. -
Declare the desired state where it makes sense.
Example: required services, workers, queues, scheduled tasks, replicas, retry policies, operational limits, and health signals. -
Define how to compare the expected outcome, current state, and real user experience.
Example: dashboards, alerts, structured logs, audit trails, failure tests, and reconciliation between the order, payment, and confirmation.
Applying this to the requirement:
“The user should receive order confirmation even when a dependency is temporarily unavailable.”
A more useful translation could be:
| Layer | Example |
|---|---|
| Business requirement | The user should not be left without a response when completing an order because a dependency is temporarily unavailable. |
| Technical intention | The system durably records the order intent and provides the user with a reliable status. |
| Policy | The visual confirmation may be immediate, but external steps may remain pending and be reconciled later. |
| Desired state | Pending orders are stored, processed later, and not lost during restarts. |
| Possible mechanisms | Transactional database, queue, reconciliation job, asynchronous processing, controlled retries. |
| Validation | Monitor pending orders, completion time, integration failures, and discrepancies between systems. |
Only after this does it make sense to discuss whether the solution involves a queue, database, replicas, Kubernetes, a managed service, or something simpler. Technology exists to support an intention, not to replace thinking.
Next Steps
If you want to apply this model with your team, a simple practice is to take a real requirement from the next planning session and write four sentences before discussing tools:
- The business needs...
- Technically, this means the system must...
- This behavior applies when...
- We will know it has failed if we observe...
After that, the conversation about infrastructure, architecture, and operations tends to become less abstract.
Good architecture decisions begin less with the question “which technology do we use?” and more with:
Which system condition do we need to sustain—and how will we prove it?
