cap-0066.md

CAP: 0066
Title: Soroban In-memory Read Resource
Working Group:
    Owner: Garand Tyson <@SirTyson>
    Authors: Garand Tyson <@SirTyson>
    Consulted: Dmytro Kozhevin <@dmkozh>, Nicolas Barry <@MonsieurNicolas>
Status: Final
Created: 2024-12-09
Discussion: https://github.com/stellar/stellar-protocol/discussions/1585
Protocol version: 23

Simple Summary

This proposal introduces a new resource type for Soroban reads, distinguishing between in-memory and disk reads. This also proposes automatic restoration for archived entries via InvokeHostFunctionOp.

Working Group

As specified in the Preamble.

Motivation

By distinguishing disk and in-memory reads, this proposal allows for significant increase in Soroban read limits. This distinction also allows for safe automatic entry restoration, significantly improving UX.

Goals Alignment

This change is aligned with the goal of lowering the cost and increasing the scale of the network.

Abstract

CAP-0062 introduces partial State Archival, where evicted and live Soroban state is stored in separate databases. This separation allows live Soroban state to be efficiently cached entirely in memory, removing disc reads entirely for all live Soroban state. Todays eviction scan can then delete entries from both the live BucketList disk and in-memory cache efficiently. However, classic state and evicted entries are still subject to disk reads.

This CAP introduces a new resource type for reads, distinguishing between disk reads and in-memory reads. By making this distinction at the protocol level, read limits for Soroban data can greatly increase. This also opens the door for other optimizations, such as a complete module cache for all live contracts.

Additionally, this CAP introduces automatic restoration via InvokeHostFunctionOp, where any archived key present in the footprint is automatically restored. Initially in protocol 20, the state archival design was not solidified enough to enable automatic restoration, so an explicit restore operation was required. While this was not technically required in protocol 20, it was important for contract interfaces to reason properly about restoration such that when full state archival was introduce, it would not break preexisting deployments. Given that the full State Archival design has been mostly finalized in CAP-0057, it is now appropriate to introduce automatic restoration.

Specification

XDR changes

--- a/Stellar-contract-config-setting.x
+++ b/Stellar-contract-config-setting.x
@@ -26,38 +26,48 @@ struct ConfigSettingContractComputeV0
 // Ledger access settings for contracts.
 struct ConfigSettingContractLedgerCostV0
 {
-    // Maximum number of ledger entry read operations per ledger
-    uint32 ledgerMaxReadLedgerEntries;
-    // Maximum number of bytes that can be read per ledger
-    uint32 ledgerMaxReadBytes;
+    // Maximum number of disk entry read operations per ledger
+    uint32 ledgerMaxDiskReadEntries;
+    // Maximum number of bytes of disk reads that can be performed per ledger
+    uint32 ledgerMaxDiskReadBytes;
     // Maximum number of ledger entry write operations per ledger
     uint32 ledgerMaxWriteLedgerEntries;
     // Maximum number of bytes that can be written per ledger
     uint32 ledgerMaxWriteBytes;
 
-    // Maximum number of ledger entry read operations per transaction
-    uint32 txMaxReadLedgerEntries;
-    // Maximum number of bytes that can be read per transaction
-    uint32 txMaxReadBytes;
+    // Maximum number of disk entry read operations per transaction
+    uint32 txMaxDiskReadEntries;
+    // Maximum number of bytes of disk reads that can be performed per transaction
+    uint32 txMaxDiskReadBytes;
     // Maximum number of ledger entry write operations per transaction
     uint32 txMaxWriteLedgerEntries;
     // Maximum number of bytes that can be written per transaction
     uint32 txMaxWriteBytes;
 
-    int64 feeReadLedgerEntry;  // Fee per ledger entry read
-    int64 feeWriteLedgerEntry; // Fee per ledger entry write
+    int64 feeDiskReadLedgerEntry;  // Fee per disk ledger entry read
+    int64 feeDiskRead1KB;          // Fee for reading 1KB disk
+    int64 feeWriteLedgerEntry;     // Fee per ledger entry write
 
-    int64 feeRead1KB;  // Fee for reading 1KB
 
     // The following parameters determine the write fee per 1KB.
-    // Write fee grows linearly until bucket list reaches this size
-    int64 bucketListTargetSizeBytes;
-    // Fee per 1KB write when the bucket list is empty
-    int64 writeFee1KBBucketListLow;
-    // Fee per 1KB write when the bucket list has reached `bucketListTargetSizeBytes` 
-    int64 writeFee1KBBucketListHigh;
-    // Write fee multiplier for any additional data past the first `bucketListTargetSizeBytes`
-    uint32 bucketListWriteFeeGrowthFactor;
+    // Rent fee grows linearly until soroban state reaches this size
+    int64 sorobanStateTargetSizeBytes;
+    // Fee per 1KB rent when the soroban state is empty
+    int64 rentFee1KBSorobanStateSizeLow;
+    // Fee per 1KB rent when the soroban state has reached `sorobanStateTargetSizeBytes`
+    int64 rentFee1KBSorobanStateSizeHigh;
+    // Rent fee multiplier for any additional data past the first `sorobanStateTargetSizeBytes`
+    uint32 sorobanStateRentFeeGrowthFactor;
+};
+
+// Ledger access settings for contracts.
+struct ConfigSettingContractLedgerCostExtV0
+{
+    // Maximum number of RO+RW entries in the transaction footprint
+    uint32 txMaxFootprintEntries;
+    // Fee per 1 KB write
+    uint32 feeWrite1KB;
 };
 
 // Historical data (pushed to core archives) settings for contracts.
@@ -302,7 +312,8 @@ enum ConfigSettingID
     CONFIG_SETTING_STATE_ARCHIVAL = 10,
     CONFIG_SETTING_CONTRACT_EXECUTION_LANES = 11,
     CONFIG_SETTING_BUCKETLIST_SIZE_WINDOW = 12,
-    CONFIG_SETTING_EVICTION_ITERATOR = 13
+    CONFIG_SETTING_EVICTION_ITERATOR = 13,
+    CONFIG_SETTING_CONTRACT_LEDGER_COST_EXT_V0 = 14
 };
 
 union ConfigSettingEntry switch (ConfigSettingID configSettingID)
@@ -335,5 +346,7 @@ case CONFIG_SETTING_BUCKETLIST_SIZE_WINDOW:
     uint64 bucketListSizeWindow<>;
 case CONFIG_SETTING_EVICTION_ITERATOR:
     EvictionIterator evictionIterator;
+case CONFIG_SETTING_CONTRACT_LEDGER_COST_EXT_V0:
+    ConfigSettingContractLedgerCostExtV0 contractLedgerCostExt;
 };
 }
diff --git a/Stellar-ledger.x b/Stellar-ledger.x
--- a/Stellar-ledger.x
+++ b/Stellar-ledger.x
@@ -528,12 +528,11 @@ struct LedgerCloseMetaV1
     // systems calculating storage fees correctly.
     uint64 totalByteSizeOfBucketList;
 
-    // Temp keys that are being evicted at this ledger.
-    LedgerKey evictedTemporaryLedgerKeys<>;
+    // TTL and data/code keys that have been evicted at this ledger.
+    LedgerKey evictedKeys<>;
 
-    // Archived restorable ledger entries that are being
-    // evicted at this ledger.
-    LedgerEntry evictedPersistentLedgerEntries<>;
+    // Maintained for backwards compatibility, should never be populated.
+    LedgerEntry unused<>;
 };
 
 union LedgerCloseMeta switch (int v)
diff --git a/Stellar-transaction.x b/Stellar-transaction.x
index 7d32481..f966640 100644
--- a/Stellar-transaction.x
+++ b/Stellar-transaction.x
@@ -882,16 +882,30 @@ struct SorobanResources
     // The maximum number of instructions this transaction can use
     uint32 instructions; 
 
-    // The maximum number of bytes this transaction can read from ledger
-    uint32 readBytes;
+    // The maximum number of bytes this transaction can read from disk backed entries
+    uint32 diskReadBytes;
     // The maximum number of bytes this transaction can write to ledger
     uint32 writeBytes;
 };
 
+struct SorobanResourcesExtV0
+{
+    // Vector of indices representing what Soroban
+    // entries in the footprint are archived, based on the
+    // order of keys provided in the readWrite footprint.
+    uint32 archivedSorobanEntries<>;
+};
+
 // The transaction extension for Soroban.
 struct SorobanTransactionData
 {
-    ExtensionPoint ext;
+    union switch (int v)
+    {
+    case 0:
+        void;
+    case 1:
+        SorobanResourcesExtV0 resourceExt;
+    } ext;
     SorobanResources resources;
     // Amount of the transaction `fee` allocated to the Soroban resource fees.
     // The fraction of `resourceFee` corresponding to `resources` specified 

Semantics

In-memory vs Disk Read Resources

Read resources are now separated between memory backed and disk backed state as follows:

• in-memory entries:
  • Live Soroban entries
• disk entries:
  • Archived Soroban entries
  • Classic entries

In-memory entries are subject to the corresponding in-memory limits, but charge no entry or byte based read fees. On disk entries are subject to the corresponding limits and also charge entry and byte based read fees.

sorobanLiveStateTargetSizeBytes

bucketListTargetSizeBytes has been renamed to sorobanLiveStateTargetSizeBytes. Instead of tracking the current amount of disk being used by the network, this value has changed semantically to track the amount of in-memory (i.e. live soroban) state.

The value of sorobanLiveStateTargetSizeBytes is the total size of live Soroban state in the ledger, in bytes. Previously, bucketListTargetSizeBytes was based on the total size of the BucketList. In addition to no longer counting classic state, sorobanLiveStateTargetSizeBytes is now based on the "live soroban state" instead of on-disk size of entries. The "live soroban state" corresponds to live CONTRACT_DATA entries, TTL entries for both CONTRACT_DATA and CONTRACT_CODE, and instantiated contract code. Archived and deleted state is not included, even if a BucketEntry for the archived or deleted state exists in the Live BucketList.

sorobanLiveStateTargetSizeBytes is the sum of the size of each live CONTRACT_DATA entry, CONTRACT_CODE entry, and corresponding TTL entry, where the size of each entry is calculated as follows:

• CONTRACT_DATA: XDR serialized size of CONTRACT_DATA LedgerEntry.
• TTL: XDR serialized size of TTL LedgerEntry.
• CONTRACT_CODE: The memory cost of the given code based on the module's ContractCodeCostInputs, as calculated by the cost model. Note that this is usually much larger than the CONTRACT_CODE LedgerEntry XDR size.

Write fee computation changes

Previously, a dynamic write fee was used in order to limit the amount of disk space being used by the network. With State Archival, the size of disk usage is no longer as significant of a concern. However, the size of in-memory state must be bounded to prevent OOM DOS attacks. To accomplish this, we maintain a dynamic rent fee to limit the amount of in-memory state on the ledger at a given time.

Since sorobanLiveStateTargetSizeBytes tracks in-memory state instead of on-disk state, it no longer makes sense to charge dynamic write fees based off of sorobanLiveStateTargetSizeBytes, but we still need to charge a fee for the cost of the write operation itself. To accomplish this, we introduce a constant per-KB write fee.

Soroban and classic state are both charged the flat per-KB write fee. The transaction write fee is computed as follows:

write_fee = ceil(write_bytes * feeWrite1KB / 1024)

Rent fee computation changes

Intuitively, the rent fee now represents the cost of reserving memory in the ledger state, so it should be dynamic wrt sorobanLiveStateTargetSizeBytes. The Rent_fee and rent_fee_per_entry_change calculations from CAP-46-07 remain unchanged, with rent_fee_for_size_and_ledgers now being computed as follows:

Renting S bytes of ledger space for the period of L ledgers:

rent_fee_for_size_and_ledgers(is_persistent, S, L) = round_up(
    S * L * rent_fee_per_1kb(BucketListSize) /
        if (is_persistent, persistentRentRateDenominator, tempRentRateDenominator)
)

The rent_fee_per_1kb function is defined as follows:

MINIMUM_RENT_FEE_PER_1KB = 1000

// this is the fee rate slope
// feeRate1KB = (rentFee1KBSorobanStateSizeHigh - rentFee1KBSorobanStateSizeLow)/sorobanStateTargetSizeBytes

// in all cases, rate is clamped as to not fall under MINIMUM_RENT_FEE_PER_1KB in case
// rentFee1KBSorobanStateSizeLow or rentFee1KBSorobanStateSizeHigh are too low

// if s < sorobanStateTargetSizeBytes,
//   grow by feeRate1KB until we reach rentFee1KBSorobanStateSizeHigh
rent_fee_per_1kb(s) = max(MINIMUM_RENT_FEE_PER_1KB,
    (rentFee1KBSorobanStateSizeHigh - rentFee1KBSorobanStateSizeLow)*s/sorobanStateTargetSizeBytes)

// else (s >= sorobanStateTargetSizeBytes),
//   grow by sorobanStateRentFeeGrowthFactor*feeRate1KB from rentFee1KBSorobanStateSizeHigh
rent_fee_per_1kb(s) = max(MINIMUM_RENT_FEE_PER_1KB,
    rentFee1KBSorobanStateSizeHigh +
    sorobanStateRentFeeGrowthFactor*(rentFee1KBSorobanStateSizeHigh - rentFee1KBSorobanStateSizeLow)*
        (s-sorobanStateTargetSizeBytes)/sorobanStateTargetSizeBytes)

Note this is the same from the write_fee_per_1kb calculation in CAP-46-07. The only difference is that rent_fee_for_size_and_ledgers does not apply a discount to rent_fee_per_1kb (other than the contract data type discount), which was the case for write_fee_per_1kb.

CONTRACT_CODE size calculation for limits and fees

CONTRACT_CODE entries have two "sizes" that are relevant to fees and limits:

• "Disk Size": The size of the CONTRACT_CODE LedgerEntry XDR.
• "In-memory Size": The total memory footprint for a contract code entry. We store every CONTRACT_CODE in memory in two different places: once in the Soroban ledger entry cache and once in the global Soroban module cache. Thus in-memory entry size is a sum of two components:
  • The memory size of the Wasm module corresponding to the Wasm stored in the CONTRACT_CODE entry, as calculated by the cost model. More specifically, the in-memory size of a given CONTRACT_CODE is defined as the metered memory consumption of parsing the Wasm code into a Wasm VM module.
  • The size of the CONTRACT_CODE LedgerEntry XDR.

Disk size is consistent between protocol upgrades, as the CONTRACT_CODE bytes themselves are immutable. However, the in-memory size is dependent on the cost model. For this reason, limits can not be based on in-memory size. If in-memory size was used for limits, a contract could be "bricked" by any upgrade that changes the cost model such that the in-memory size increases.

For this reason, contractMaxSizeBytes limit enforcement will continue to be based on disk size. But in-memory size will be used for accounting CONTRACT_CODE entries towards the Soroban state size, as well as the rent computations.

Since the in-memory size may be significantly larger than the disk size (up to 40x as of protocol 23) the rent fees will be adjusted with the protocol upgrade in order to keep the effective rent fees for contract code close to those in protocol 22 (since these fees seemed to be reasonable enough for DOS protection). Moreover, in order to reduce the gap in rent fees between code and data entries a discount factor of 3 will be applied to the final rent fees. This is the highest possible discount the protocol can implement without creating an incentive to store the contract data in code entries - the minimum ratio between the disk and in-memory size for any Wasm is greater than 3 as of protocol 23 (the Wasm module size is at least 2x of the actual Wasm size, and another 1x Wasm size is charged for the entry cache).

To summarize, here is the breakdown of the sizes used for CONTRACT_CODE in different contexts:

• Maximum contract code size enforcement - disk size
• Disk read bytes limit and fee - disk size
• Disk write bytes limit and fee - disk size
• Soroban state size accounting - in-memory size
• Rent fee - in-memory size, with an additional 3 times discount

While instantiated contract code is significantly larger than it's on disk size, this is not true for cached contract data. For this reason, rent fees for CONTRACT_DATA will continue to be based on the serialized XDR disk size.

Initial Network Config Settings

Initially, all disk read limits should match exactly the limits of reads today. Limits must not decrease from current values in order to prevent bricking existing contracts. In the worst case, a contract can read up to txMaxReadLedgerEntries - 1 classic entries today (all classic entries + SAC instance). Current limits have been carefully measured assuming disk access only, so the addition of in-memory state should not negatively affect the network in any way should we maintain these values via disk read limits.

The new config setting txMaxFootprintEntries sets the limit on the total number of entries in the transaction footprint, i.e. the number of entries in both read-only and read-write footprints. The footprint size limits are not required wrt transaction execution time, as in-memory reads are very inexpensive and disk reads and writes are limited by the separate settings. For this reason, no respective ledger wide read limit is necessary, and no byte based read limit is necessary for in-memory reads. However, the size of transaction footprints does impact the cost and complexity of assembling transactions sets and potentially maintaining the mempool, as with implementation CAP-63 Core will need to verify the presence of the conflicts in transaction footprints. Thus a tx footprint entry limit is introduced to ensure efficient transaction set construction.

Note, that there is also a 'soft' limit on the maximum size of footprint per transaction that is caused by the transaction size limit. But coupling the limits in this fashion is risky - for example, network might need to increase the minimal transaction size in order to accommodate larger contracts, but the simultaneous increase in footprint entry count might not be desired.

When the protocol is upgraded to version 23 the initial values of the new configuration settings will be set to the following values:

• txMaxFootprintEntries will be set to the current value of txMaxDiskReadEntries (renamed from txMaxReadLedgerEntries) in order to prevent breakages of any existing contracts
• feeWrite1KB will be set to 3'500 stroops, which is roughly 2x of the current Soroban read fee per 1 KB. Note, that this fee is decoupled from the rent write fee now.

Rent-related network settings update

Since Soroban live state will be used for the rent write fee computation, the fee-related configuration will be updated. The goal of the update is to minimize the immediate effective rent cost increase for the contract code entries due to using their in-memory size for the rent fee computation.

Specifically, the fees are tuned using the following input constraints:

• The current in-memory state size during the protocol upgrade is projected to be 1 GB
• If a contract code entry size is increased 40x, the effective rent cost at the current size should increase just 2x. Note, that given a 2x rent fee discount for the code entries, this means that the effective rent costs for the regular contract data entries will go down 10x compared to protocol 22.
• The 'target' size of the in-memory state after which the fees grow steeply is set to 3 GB (3x of the current size)
• The rent cost will stay constant until the state growth 2 times (to 2 GB)
• During the state size growth from 2 to 3 GB the rent fees will grow up to 10x
• After 3 GB the rent fee growth factor of 5000 will be applied (no change from protocol 22)

Given these constraints, the following rent fee settings will be updated to the following values during the protocol 23 upgrade:

• sorobanStateTargetSizeBytes set to 3'000'000'000 (3 GB)
• rentFee1KBSorobanStateSizeLow set to -17'000
• rentFee1KBSorobanStateSizeHigh set to 10'000
• persistentRentRateDenominator set to 1'215
• tempRentRateDenominator set to 2'430

Allow resource fees to exceed uint32

Currently the maximum possible Stellar transaction fee is limited by the transaction fee field's is limited by its uint32 type, i.e. it is 2^32 - 1 stroops or ~429 XLM. The combination of using the memory size for the contract code entries, as well as the rent changes above, makes it possible for a certain transaction, such as a Wasm upload or TTL extension to exceed that fee. Specifically, when the state size is close to its limit the fees will become high enough to make it impossible to upload a large Wasm (currently the protocol requires a payment of at least 120 days of rent for any entry creation).

While this is an edge case scenario that should not happen under normal network utilization, its important to ensure that the network remains usable (albeit expensive) in case if state size grows too big too quickly.

Instead of modifying the transaction data structure, protocol 23 just allows a fee bump transaction to have a negative inclusion fee for its inner transaction. Since fee bump transactions have int64 field for the fee, they can cover any resource fee, and there is no real need for the inner transaction to be fully valid on its own. Thus, in order to submit a transaction with resource fee that exceeds uint32 type limit, a user will need to create a fee bump transaction with an arbitrary inner transaction's fee and a fee bump transaction fee that is sufficient to cover the declared resource fee.

archivedEntries Vector

For the purposes of block creation, the LedgerKey alone is sufficient to distinguish between classic and Soroban data. However, a disk read would be required prior to tx application in order to determine if a Soroban entry was live or archived. This is necessary for determining what resources limits a Soroban key counts towards, but exposes a significant DOS angle. To prevent this attack, the footprint must statically declare which entries are live vs. archived.

This is accomplished via the archivedEntries vector. If any archived Soroban entry is present in a TX's footprint, it must provide the archivedEntries vector containing the index of each archived key based on the ordering of the readWrite footprint. Because restores are a write operation, no archived key should be in the readOnly footprint. This vector should contain no classic key indexes, as classic keys are always considered disk reads.

Changes to InvokeHostFunctionOp

Automatic Entry Restoration

Whenever InvokeHostFunctionOp is applied, any archived state is automatically restored prior to host function invocation. This restored state is then accessible to the host function.

Restored state is still subject to the same minimum rent and write fees that exist currently based on the final result of the invocation. Even if an entry is archived but not yet evicted such that it technically still exists in memory, it is still subject to the same limits and fees as disk based entries in order ot provide a simpler unified interface for downstream systems.

All archived keys must be declared in the archivedEntries vector via SorobanResourcesExtV0. If no archived entries are present in the footprint, SorobanResourcesExtV0 may be omitted.

Fees

Given that restoration and invocation are happening atomically, rent fees will be charged based on the final state following invocation, not the initial restore state. For example, if an entry is autorestored, then deleted as part of the invocation, it will only pay disk read fees, not write fees or rent fees.

Suppose we are auto restoring some entry e:

min_ttl = minimum required ttl following restore
e = initial value of ledger entry that is restored
e' = value of e following transaction execution
e'_ttl = ttl value of e following transaction execution

Metering and Fees for InvokeHostFunctionOp with autorestore of e:

diskReadBytes = sizeof(e)
writeBytes = sizeof(e')

if sizeof(e') == 0:
     rent_fee = 0
else:
     rent_fee = rent_fee_for_size_and_ledgers(/*isPersistent*/ true, sizeof(e'), max(min_ttl, e'_ttl))

Incorrect Entries in archivedEntries Vector

If a key is not marked as archived, but the entry is actually archived, then the TX fails. If a Soroban key is marked as archived when it is not, the transaction does not fail. The entry is treated as if it is archived from a resource and fee perspective. This is appropriate, as on disk reads are more expensive than in-memory. So long as the TX pays the disk based fees, there is no issue with actually loading an in-memory entry.

archivedSorobanEntries must only point to persistent Soroban entries. If archivedSorobanEntries contains an invalid index (an out of bounds index, classic entry, or temporary entry) the transaction is malformed. Indices in archivedSorobanEntries must be sorted. If the indices are not sorted, or include a duplicate value, the transaction is malformed.

Failure Codes

If an archived entry is included in the TX's footprint but is not specified via the archivedEntries vector, the tx will fail with the INVOKE_HOST_FUNCTION_ENTRY_ARCHIVED code. If there are insufficient resources for entry restoration, the tx will fail with the INVOKE_HOST_FUNCTION_RESOURCE_LIMIT_EXCEEDED code.

Meta

Whenever an entry is restored, the data/code entry will be emitted as a LedgerEntryChangeType of type LEDGER_ENTRY_RESTORE. This will include the complete LedgerEntry of the restored entry. It does not matter if the entry has been evicted or not, the meta produced will be the same. Additionally, the corresponding TTL entry will also be emitted as a LedgerEntryChangeType of type LEDGER_ENTRY_RESTORE.

If a restored entry is modified during host function invocation, another LedgerEntryChangeType will be emitted corresponding to the entry change in addition to the LEDGER_ENTRY_RESTORE restore event. This is similar to the current meta schema, where the starting value of a LedgerEntry is emitted as LEDGER_ENTRY_STATE followed by the change. This is similar, except that LEDGER_ENTRY_STATE is replaced by LEDGER_ENTRY_RESTORE. This applies to both data/code and TTL entries. For example, suppose a data/code entry is restored via an InvokeHostFunctionOp, then during invocation its TTL value is bumped. Meta for the TX will be as follows:

data/code: LEDGER_ENTRY_RESTORE(restoredValue)
TTL: LEDGER_ENTRY_RESTORE(minimumTTLfromRestore), LEDGER_ENTRY_UPDATED(bumpedTTLValue)

Similarly, suppose a CONTRACT_DATA entry is restored, then deleted during invocation. Meta for the TX will be:

CONTRACT_DATA: LEDGER_ENTRY_RESTORE(restoredValue), LEDGER_ENTRY_REMOVED(dataKey)
TTL: LEDGER_ENTRY_RESTORE(minimumTTLfromRestore), LEDGER_ENTRY_REMOVED(ttlKey)

If an entry is restored and not modified (either from RestoreFootprintOp or InvokeHostFunctionOp), the meta will be:

CONTRACT_DATA: LEDGER_ENTRY_RESTORE(restoredValue)
TTL: LEDGER_ENTRY_RESTORE(minimumTTLfromRestore)

getledgerentry Endpoint

In order to facilitate state queries for RPC preflight, captive-core will expose the following HTTP endpoint:

getledgerentry

Used to query both live and archived LedgerEntries. While getledgerentryraw does simple key-value lookup on the live ledger, getledgerentry will query a given key in both the live BucketList and Hot Archive BucketList. It will also report whether an entry is archived or live, and return the entry's current TTL value.

A POST request with the following body:

ledgerSeq=NUM&key=Base64&key=Base64...

• ledgerSeq: An optional parameter, specifying the ledger snapshot to base the query on. If the specified ledger in not available, a 404 error will be returned. If this parameter is not set, the current ledger is used.
• key: A series of Base64 encoded XDR strings specifying the LedgerKey to query. TTL keys must not be queried and will return 400 if done so.

A JSON payload is returned as follows:

{
"entries": [
     {"e": "Base64-LedgerEntry", "state": "live", /*optional*/ "ttl": uint32},
     {"e": "Base64-LedgerKey", "state": "new"},
     {"e": "Base64-LedgerEntry", "state": "archived"},
],
"ledger": ledgerSeq
}

• entries: A list of entries for each queried LedgerKey. Every key queried is guaranteed to have a corresponding entry returned.
• e: Either the LedgerEntry or LedgerKey for a given key encoded as a Base64 string. If a key is live or archived, e contains the corresponding LedgerEntry. If a key does not exist (including expired temporary entries) e contains the corresponding LedgerKey.
• state: One of the following values:
  • live: Entry is live.
  • new: Entry does not exist. Either the entry has never existed or is an expired temp entry.
  • archived: Entry is archived, counts towards on-disk resources.
• ttl: An optional value, only returned for live Soroban entries. Contains a uint32 value for the entry's liveUntilLedgerSeq.
• ledgerSeq: The ledger number on which the query was performed.

Classic entries will always return a state of live or new. If a classic entry does not exist, it will have a state of new.

Similarly, temporary Soroban entries will always return a state of live or new. If a temporary entry does not exist or has expired, it will have a state of new.

This endpoint will always give correct information for archived entries. Even if an entry has been archived and evicted to the Hot Archive, this endpoint will still the archived entry's full LedgerEntry as well as the proper state.

Design Rationale

Reasoning behind in-memory limits and no read fees

While there is a resource limit for the maximum number of in-memory entry reads, there is no limit on the number of in-memory read bytes. Additionally, there are no read-specific fees for in-memory state.

There does not seem to be a need for limits on in-memory bytes read. The cost associated with loading an in-memory entry is searching an in-memory map and making a copy of the LedgerEntry. These copies will be metered against the txMemoryLimit limit. To prevent OOM based DOS, operation apply will exit early if loading entries exceeds this limit.

There is very little execution time associated with copying an in-memory entry. Given that the size of the copies is bounded by the txMemoryLimit and no expensive disk IO is required, it is not necessary to charge a fee for in-memory reads. An attacker could potentially exploit this by creating TXs with large readOnly footprints that don't actually interact with the loaded state. The TX would consume little resources and would be cheap, but would load the maximum amount of in-memory state and. Due to the low computational cost of these in-memory copies, it seems unlikely this would negatively affect the network. Even the TX size costs associated with the large footprint may make the attack economically nonviable relative to the amount of stress on the network. Additionally, memory allocations are short lived and freed immediately following TX application. Only a small number of TXs are executed at any given time such that memory is quickly freed such that OOM based attacks are not possible.

Changes required for CONTRACT_CODE

This CAP combined with CAP-0065 cache all live Soroban state in memory, including CONTRACT_DATA, TTL, and instantiated contract code. These caches open potential DOS vectors with the way CONTRACT_CODE size is metered currently.

For CONTRACT_DATA and TTL, we cache all live LedgerEntries. This does not present a DOS angle. We meter these entry types based on serialized LedgerEntry XDR size such that we have a 1 to 1 ratio between the bytes that are metered and the bytes that are cached in memory.

For CONTRACT_CODE, this is no longer the case. CAP-0065 caches instantiated modules in memory, which in the worst case could be up to 40x the size of the CONTRACT_CODE LedgerEntry size. This is a significant OOM risk, as sorobanLiveStateTargetSizeBytes must bound the amount of state validators are required to cache in memory. If CONTRACT_CODE XDR size is used, an attacker could upload code that once instantiated could bloat the cache size to sorobanLiveStateTargetSizeBytes * 40, causing an OOM based DOS of the network. To prevent this, we use the instantiated module memory cost model size instead of the CONTRACT_CODE LedgerEntry size for rent fees. While fees become slightly higher, this is reasonable given that the contract modules consume more network resources (memory).

Requirement for Module Cache and Cost Model Consistency

Using the instantiated module size for rent fees requires that the Module Cache be up to date with the current cost model. This means that each upgrade to the cost model must trigger a rebuild of the Module Cache.

UX Expectations

With the addition of the new read resource type, contract developers will need to reason about what types of data they are accessing wrt limits. However, the system has been designed such that this should be straight forward. From a correctness standpoint, all smart contracts can be guaranteed to only interact with live state. A contract developer can assume that all Soroban state consumes only in-memory resources, since the RestoreFootprintOp (or equivalent in the case of deprecation) can restore all required Soroban entries before host function invocation in the worst case. This means that from a limits perspective, only Soroban state vs. classic state must be considered for contract correctness. However, there may be issues with usability or efficiency, discussed in the section below.

While contract developers only need to distinguish between Soroban and classic state for contract correctness, the tx itself will still need to properly populate limits and the archivedEntries vector based on whether or not entries are archived. Similar to today, these resources and footprints will be generated automatically via RPC. Specifically, RPC will correctly generate the footprint, archivedEntries vector, in-memory read resources, and disk read resources. Thus most of the complexity introduced by these changes should be abstracted away.

Note that the RPC itself is not required to implement this behavior, but will call captive-core endpoints.

Expectations for Downstream Systems

RPC must make the following changes:

1. Ingest the new LEDGER_ENTRY_RESTORE restore changes. This includes interpreting the new LedgerEntryChangeType correctly, but also correctly handling the case where an entry is restored and modified in the same ledger.
2. Allow user queries for both live and evicted state via the getledgerentry endpoint.
3. Handle transactions that cannot use automatic restore due to resource limits.

Change 1 should be straight forward and is defined in the InvokeHostFunctionOp Meta section.

Change 2 allows wallets and dapps to effectively reason about ledger state. RPC integration is minimal, as internally this endpoint can be implemented by calling the captive-core getledgerentry HTTP endpoint.

Change 3 is more involved and discussed in the section below.

Handling Invocations that restore too much state for automatic restore

Suppose an expensive contract is written, such that it uses the maximum in-memory resources and the maximum disk resources when no entries are archived. Should any Soroban entry be archived automatic restoration would not be possible, as the restoration would exceed resource limits. In this case, it would be necessary to manually issue one or moreRestoreFootprintOp prior to the host function invocation.

To facilitate this, RPC must change the structure of restorePreamble to allow for multiple RestoreFootprintOps. Instead of returning a single set of keys and resource object in restorePreamble, it should return a vector of {keySet, resourceObject} pairs, where each pair represents a RestoreFootprintOp. This vector should be constructed as follows:

restorePreamble = []
archivedKeys = keysReturnedFromSimulationLibrary
invokeHostFunctionOp = preflightedOp

# First, append as many archived keys as possible to be auto restored via invokeHostFunctionOp
while invokeHostFunctionOp.hasSpaceInDiskResources() and not archivedKeys.empty():
     keyToAutoRestore = archivedKeys.pop()

     # Function will mark the key as archived in the footprint and add
     # the required diskRead resources to invokeHostFunctionOp
     markAsArchivedEntry(invokeHostFunctionOp, keyToAutoRestore)

# If we get here, there are more archived keys than can be restored via invokeHostFunctionOp auto restore
# We need to construct RestoreOps to restore the remaining keys
if not archivedKeys.empty():
     restorePreamble.append(RestoreOp())
     while not archivedKeys.empty():
          while restorePreamble[-1].hasSpaceInDiskResources() and not archivedKeys.empty():
               keyToRestore = archivedKeys.pop()
               restorePreamble[-1].addKey(keyToRestore)

          # If we filled up out first restore op, create a new one
          if not archivedKeys.empty():
               restorePreamble.append(RestoreOp())

Ideally, developers should be avoid creating contracts that require these expensive restores if possible. The challenge will be communicating this to developers, but may be accomplished via documentation.

Concerns regarding Automatic Restoration with Full State Archival

In protocol 20, there was no technical reason against enabling automatic restoration. However, the proof system for full state archival was not finalized, and the explicit RestoreFootprintOp provided more future flexibility. Additionally, it was important for developers to build preflighting transactions into their dApp flows.

If Soroban had launched with automatic restoration, it is possible many developers would attempt to avoid preflight by creating transaction footprint and resource declarations manually with reasonable fee and resource assumptions. In a system where explicit entry restoration is not required, the burden of resource discoverability is low, so developers are not incentivized or required to build preflight into their tx submission system. Long term this is a significant issue, as the addition of full state archival with proofs makes manual resource and footprint declaration largely impossible. Any system that did not use RPC preflight would be broken by full state archival.

With this proposal, it is vital to ensure that automatic restore does not result in a situation where large numbers of developers circumvent preflight. Two specific features of this proposal help ensure developers are incentivized to build preflight into their flows now such that they are not broken later by archival proofs. Specifically, no in-memory read fees incentivize developers to minimize fees by using preflight to detect the maximum number of entries currently live. Additionally, the requirement of archivedEntries vectors makes manually constructing footprints challenging, as the developer would need to be able to distinguish live vs evicted state to minimize fees. Given that preflighting will result in lower on-chain fees, and given that resource footprints are somewhat challenging to create without captive-core seems to indicate that developers will preflight transactions prior to submission as intended.

Security Concerns

Memory Based DOS Attack

By distinguishing between disk backed and memory backed reads at the protocol level, it is now required that validators maintain all live Soroban state in memory. A significant increase in live Soroban state could lead to an OOM based attack on the network.

However, this is not a significant concern. The sorobanLiveStateTargetSizeBytes (see CAP-0062) provides a soft cap on the amount of live Soroban state that can exist at any given time. This prevents any sort of runaway memory issue. Additionally, because of the eviction process, there is a natural back pressure applied to in-memory state, the rate at which can be controlled via network config settings. Finally, the rate at which new state is created is also controlled via network configs, so there does not appear to be any memory based DOS attack possible.

Test Cases

TBD

Implementation

TBD