Idempotency

Idempotency is the property of a system that ensures an action is applied at most once . This may sound strange if you haven’t heard of it before, but the idea behind idempotency is this: in an idempotent system if you accidentally send the same instruction 1,000 times, it will only be executed once. This has two very powerful implications in financial systems:

• It protects from duplicate charges and payouts. Duplicate charges make for unhappy customers and duplicate payouts make for unhappy operating accounts. Both will cost you money, time, and reputation when they happen.
• It makes retry policies extremely straight forward. In an idempotency-protected system it is always ok to blindly retry an action you aren’t sure finished. Whether it finished or not, you can be confident that it will never be executed a second time.

Idempotency Keys

This behavior is accomplished through the use of idempotency keys . These are unique identifiers that are used a single time for an unit of financial work. Idempotency keys should be unique based on the business action you are taking. For example, if you are paying your employees for payroll you might use an idempotency key of payroll_07_15_2023:employeeId . This means even if someone runs payroll a second, third, or fourth time for the same period, the system will pay each person a single time.

Idempotency Key Structure

String Theory uses idempotency keys made up of two fields: type and token . The type is the category of key. It should be specific enough to be unique to the action type, but not unique to the action. The token should be a unique identifier for this specific key. In this section we’ll show them in the format of type:token . Some good and bad examples of idempotency keys are provided below.

• order:1234
• payout:a3nb90018c
• payroll_07_15_2023:employee_819234
• manual_deposit:429ff7e16837b203
• manual:cs
• employee_819234:pay

The core to a good idempotency key is knowing the business context for the financial change and building the key based on the business rules. Some example:

• Do you charge a guest only once per order? order:orderId might be a good idempotency candidate
• Do you payout out to sellers at most once per day? payout_02_15_2023:sellerId might be a good idempotency candidate
• Do you only allow coupons to be redeemed once? coupon:couponId might be a good idempotency key

These are just examples, but good idempotency keys are closely tied to how your business functions . You need to think critically about when actions should be repeatable, and when they should be unique.

Important Warnings

Don’t generate an idempotency key just to meet the requirement of having one. It is important to think about your use case and if the operation should be allowed to be repeated. Passing random keys for every operation will create the potential for duplicate financial actions in your system that are difficult and costly to fix. In particular these issues tend to be very impactful to reputation as overcharging customers or over paying employees degrades trust rapidly.

Idempotency Contract

String Theory requires idempotency keys on every interaction with the API when data may change. The side effects (actual impacts) of each API call are atomic, and idempotent. If the idempotency key has ever been used before the system will return the result of the old operation, instead of applying the effect again. The contract with String Theory for idempotency keys is as follows:

String Theory will only execute operations once per idempotency key.

Common Use Cases

We believe that having strong idempotency keys is imperative for all mutating operations. That said, the ways in which strong idempotency keys protect you may differ depending on context. Here, we discuss a few key operations to highlight the benefits of idempotency.

Deposits

Without a strong idempotency key, it’s possible to deposit money into the system multiple times. Multiple fast clicks of a button (UI) or automated retries (API) may result in duplicate deposits. While these are correctable mistakes, if another system then “spends” the money before you can make the correction, clean up gets much more complicated (see Fixing Data). Prevent this problem in the first place with deterministic idempotency keys.

Withdrawals from Groups/Threads

Groups and threads act similar to accounts for withdrawals. As long as they have enough money to satisfy the amount to be withdrawn, they’ll allow it. This means if we have a $4,000 in an account and accidentally request $2,000 twice, both withdrawals will go through. Strong idempotency keys will protect you from this problem.

Transfers from Groups/Threads

Similar to withdrawals, transfers from a thread or a group have the potential to be duplicated if there is still enough money left. However, correcting a mistake is even more complicated. Accidental withdrawals can be returned. Accidental transfers cross an owner boundary, which complicates cleanup. While there is always a path to reconciliation (again, see Fixing Data), the process will cost you time and money. Be careful to specify a strong idempotency key when working with these operations.

Exchanges

Exchanges are arguably the most important operation to consider when thinking about idempotency. While an accidental withdrawal can be returned, an accidental exchange can’t be naively reversed. The reason is that exchange rates are often volatile, and are not necessarily symmetrical. If an accidental exchange is just recorded and not actually made, you may be able to clean up the ledger by withdrawing the exchanged money and manually depositing the original asset at the right amount. However, if the exchange was actually made, there are no perfect resolutions. Avoid this outcome; use strong idempotency keys.

Specifying Knots

Working with specific knots by explicit knot ids is a special case. While there is no sense in “softening” your strong idempotency keys in this case, you do have a bit more protection against duplicate outcomes. Knots are immutable, so once they’ve been changed once they can never be changed again. This makes operations targeting specific knots effectively idempotent.

FAQ (Technical)

How does idempotency work in String Theory?

String Theory idempotency is based off of 3 distinct dimensions:

1. The idempotency key (which consists of a type and a token)
2. The endpoint operation you are calling (deposit, withdraw, etc.)
3. The version of the endpoint you are calling

If all of those dimensions are the same, no matter how many times you call the endpoint, you will only actually apply the operation once. Successive calls will always succeed, and return the exact same response as the first successful call.

String Theory skips most validation checks for requests that have previously successful idempotent responses. It also ignores the content of the request body when the idempotency key has been previously successful.

If the operation wasn’t applied, due to this being a repeated call with the same idempotency key, the idempotentResponse field will be set to true in the response.

Why does it return the exact same response?

There are three camps when it comes to how to handle successive calls on an idempotent endpoint:

• Exact Response: Store the original response, and return it on successive calls
• Regenerate Response: Don’t apply the side effects of the operation, but regenerate the response from current data
• Throw Error: Throw an error if the operation is called again with a specific status code (like 409 Conflict)

String Theory uses the Exact Response approach. We’ll go over pros and cons of each approach below and why we chose this one.

String Theory’s overall goal for idempotency is to allow consumers to blindly retry failed idempotent calls without needing branching logic to handle idempotent responses separately. That fundamental goal is was the most important factor in our decision to use the Exact Response approach.

Exact Response

The Exact Response approach is the simplest to implement. You store the original response in the database by an o(1) lookup key, then return it on successive calls. It consumes more storage then some of the other approaches, but it is incredibly fast, and is extremely reliable.

The biggest benefit for this approach is that even in complex circumstances it allows the consumer to blindly retry failed idempotent calls. Since each idempotent call is guaranteed to receive its original response, even if you are chaining together multiple operations, you can retry the entire chain without worrying about the side effects of the operations.

The largest downside for this approach is that it requires a lot of storage. This can be mitigated by switching to a compressed or binary format for the response in the db, and/or by implementing a cleanup process to remove very old responses, both of which can be done with no impact to our consumers.

Regenerate Response

The Regenerate Response approach is more complex to implement. It requires significant code branching inside endpoints to handle idempotency. You need to store the keys of the objects the original operation was applied to, then fetch the current state of each of those to present the new data. This is further complicated by the fact that querying for the updated data is a completely different query than for the input data, requiring a separate endpoint implementation for when the endpoint is returning an idempotent response.

The biggest benefit of this approach is that the consumer gets the most up to date data. However, since there is only one mutable field in String Theory’s entire data model, this is not a particularly compelling benefit.

Additionally, it comes with a significant downside to consumers. It requires the consumer to make extremely complex and nuanced decisions about how to handle idempotent responses.

Take this operation chain as an example:

1. Allocate 100 USD into 70 USD
2. Withdraw the 70 USD

Somewhere in the chain, the job fails (either from a ST error, or consumer code) so we retry it. However, the allocate call now returns withdrawn knots. The easy assumption here is that the withdraw call succeeded, but actually there is no guarantee that this withdraw call happened. The money could have easily been withdrawn by another call, for a completely different business use case. In order to check, the consumer needs to make the withdraw call anyhow even though the data from the allocate call is telling them that it will fail (since withdrawn knots cannot be withdrawn again).

There are many of these types of edge cases, and the get worse with more complex operation chains.

In summary, slower speed, more complex code on the String Theory side, and more complex code on the consumer side are the reasons that this wasn’t an appealing option to us.

Throw Error

The error approach is simple to implement, but comes with significant cost to consumers via code complexity. This pattern is most often used in systems originally designed without idempotency in mind, that add it on later.

The big benefit here is on the service (String Theory) side, as it is the most trivial for the service to implement. It takes very little space, since all you need to do is store the keys that have been used, and throw an error if the same key is used again.

Much like the regenerate response approach, this pattern’s biggest downside is that it makes handling idempotent responses far more difficult on the consumer side. The reason is that the consumer now needs branching logic to convert idempotency errors into “effective” successes for operation chains. Additionally, because the thrown error doesn’t contain the original response data, the consumer can’t make any business decisions based on the responses of any idempotent calls. For String Theory this is particularly problematic, because using knot ids in operational chains is the safest way to implement them. Without being able to inspect the knot ids from previous calls the consumer will have to resort to fetching knots in other ways that aren’t as safe.

Why We Chose Exact Response

Even though the Exact Response approach is more costly to String Theory, we chose it because we believe that it has the best benefits for our consumers. It also matches perfectly with our design goal for idempotency, which is to allow consumers to blindly retry failed idempotent calls even in complex operation chains.

If my idempotent call gets an error, will it always return an error?

No. String Theory only saves responses and idempotency keys when the call is successful. If the call fails neither the idempotency key nor the response are saved. There are two major reasons for this:

Retryable Errors

Under some circumstances, String Theory will throw errors that are caused by specific timing issues. The requests that resulted in these errors are likely to succeed if they are retried. If we saved these errors and considered them to be “the result” of the idempotent call, it would require consumers to change their idempotency keys for successive retries which would be a significant usability regression. Additionally, idempotent patterns work best when idempotency keys are deterministic, so introducing a pattern that requires changing them is dangerous.

String Theory Errors

In a situation where the String Theory code base itself has a bug that is resulting in errors, if we saved those errors we would either have to clean them up from our database after the fix for consumers to be able to replay their requests, or we’d have to ask the consumer to replay the requests with new idempotency keys, which partially defeats the purpose of idempotency.

However, by not saving the error, then as soon as the code fix is deployed the consumer can blindly retry their calls (as usual) in order to complete the operation.

What happens if two requests with the same idempotency key are sent at the same time?

Idempotency results are saved transactionally in the database alongside the side effect of the operation. If two or more requests come in at the same time, only one will succeed and the others will fail. If the timing of the requests is very tight they’ll get a retryable TemporaryConflictError, but if one completes fast enough the others are likely to end up with a StaleKnotError instead.

What operations are idempotent?

All mutating operations in String Theory are idempotent. This means, if a knot will be created for any reason, the entire operation must be idempotent.

Read only operations are not idempotent, and always return the most up to date data.

See the API Reference for details on individual endpoints.

Why doesn't String Theory save the request?

It’s common in some idempotency frameworks to save the body of the original request, and validate on subsequent calls with the idempotency key that the new request matches the original.

String Theory does not do this. There are two main reasons:

We trust your idempotency keys

If your idempotency keys are stable and deterministic then validating the request isn’t necessary or useful. Where this is implemented it is largely due to the framework not trusting the consumers to create good idempotency keys. At String Theory we believe that creating good idempotency keys is a critical responsibility of the consumer that you should take seriously.

There is simply no way to implement effective idempotency without good keys. They require careful consideration and business context which our consumers are best positioned to provide. We trust you to create them.

Dynamic information breaks the pattern

The biggest drawback of validating that requests are the same is the presence of dynamic or temporal information. This generally takes the form of either: generated ids, or timestamps.

Imagine you have a call to String Theory which adds a tag with a type of “business_timestamp” and a token of the current timestamp. That request will never pass the request body validation, since the current timestamp will always be different. The only way around this is to save the request body on the consumer side, which effectively forces the consumer to build their own idempotency implementation on top of String Theory’s.

We’ve carefully designed String Theory’s idempotency implementation so that our consumers won’t have to build their own. The existence of dynamic information makes validating requests incompatible with those design goals.

Other questions?

Idempotency can be a difficult concept. It requires both engineering understanding as well as business context to get right. If you need help determining good idempotency keys for your system the String Theory team is always available to help.