Research is a big part of building, recently the core team has been diving deep into Open Transactions. In this thread, we'll explain what has been narrowed down so far about Open Transactions in (hopefully) simple terms!
https://preview.redd.it/jqqob3dwj0pb1.png?width=1200&format=png&auto=webp&s=c149f30a873ed68d0933b6f7bfdb21a314776a83
Open Transactions are a more efficient way for multiple parties to collaborate on creating a single transaction. Imagine working on a group project where everyone contributes their piece at different times. That's what Open Transactions can do for CKB users.
https://preview.redd.it/c57z2d4zj0pb1.png?width=1200&format=png&auto=webp&s=1029be8af586a69ba827b9729657c2855abd8c10
Open Transactions help users and services communicate and share information more easily. They're especially useful when multiple people or services need to work together on a single transaction.
https://preview.redd.it/19b9hb52k0pb1.png?width=1200&format=png&auto=webp&s=fc6a13a536cebcbe21f494eb932bbd8657a644b1
This innovative approach helps in two main ways: (1) allowing participants to add their part one after the other, and (2) letting third parties combine different transactions into one. It's like everyone working together on a puzzle!
https://preview.redd.it/4386b3b8k0pb1.png?width=1200&format=png&auto=webp&s=e3f17fe376e7814abb31321fd213a27dc956fd88
But what does it mean for decentralized applications (dApps) on CKB? Open Transactions can have a significant impact on #dApps in various ways, leading to better user experiences and more powerful applications.
First, they enhance composability, making it easier for dApps to interact and compose with other dApps, services, or users. This enables the creation of more complex and feature-rich applications that can leverage multiple services and components.
https://preview.redd.it/7o19fhoak0pb1.png?width=1200&format=png&auto=webp&s=4c9d4518d8b595e60770f9d09568a3f6ae246bb5
Improved transaction efficiency is another benefit. Incremental transaction construction and aggregation facilitated by Open Transactions can help optimize network usage, reducing congestion and improving the overall performance of dApps.
https://preview.redd.it/652man3dk0pb1.png?width=1200&format=png&auto=webp&s=ffe70129e7b063db951b5b8afbc18296df00b646
Greater flexibility and choice for users are possible with Open Transactions. Users can choose different services at each step of the transaction process, giving them more control and options when using dApps and interacting with the CKB ecosystem.
Enhanced privacy is another advantage. Open Transactions can enable privacy-preserving techniques, like CoinJoin, to help anonymize transactions and protect user privacy. This can make dApps on CKB more appealing to users who value privacy and security.
https://preview.redd.it/wv2lqvggk0pb1.png?width=1200&format=png&auto=webp&s=32bab4da68005fe042ae98bd867d37f7c264faba
Moreover, Open Transactions help mitigate the hot cell problem, where multiple transactions compete for the same input cell. This reduces the need for users to repeatedly create and sign transactions, resulting in a smoother experience when using dApps.
In conclusion, Open Transactions on CKB simplify collaboration and communication between different parties involved in creating a transaction. They provide a more efficient and secure way to complete transactions on the network.
https://preview.redd.it/eznwbrklk0pb1.png?width=1200&format=png&auto=webp&s=265f6d78cb956b7ea9563eb45cb1af123e218bb8
With their potential to enhance the functionality, efficiency & user experience of dApps on CKB, Open Transactions can contribute to the growth & success of the CKB ecosystem. If you want to dive into the deep end, check out this post on Nervos Talk:
https://talk.nervos.org/t/exploring-the-ckb-otx-paradigm-accomplishments-and-insights-from-building-a-transaction-streaming-prototype/7346
1 Synopsis
The CKB Open Transaction (OTX) allows users to create transactions that are open to change. Users send these open transactions to a network, where many autonomous agents provide services for assembling CKB transactions from open transactions.
This report will recap the accomplishments and insights we gained in designing a OTX proposal and building a prototype. The purpose of this prototype is to test a possible design, understand its pros and cons, and find unexpected obstacles and potential improvements to the core protocol. Most of the OTX prototyping work is done by Ethan Yuan and myself. Credits go to Jiandong for helping us design and implement the first version of OTX Lock based on the Instructions List.
The OTX Prototype project can be divided into four areas, each of which will be discussed in a separate chapter later in the text. Please note that all the work here is for prototype purposes, the OTX protocol could be very different from this prototype in future iterations. Here is a summary of these areas:
- OTX Format: An extensible transaction format used to describe a CKB transaction along with its attached metadata.
- OTX Streaming Pattern: A pattern that involves placing open transactions in a stream and dividing the process of constructing transactions into smaller, reusable components.
- OTX Lock: A partial signing lock script that excludes certain transaction properties from the signature, allowing the signature to remain valid even if those properties are altered.
- OTX Framework: A framework for designing and developing dApps utilizing OTX Lock, OTX Streaming Pattern, and OTX format.
2 OTX Format
Constructing a CKB transaction can be complex. Even a basic CKB transfer transaction requires gathering live cells, forming an output for the recipient, allocating sufficient fees, and creating the change output for the remaining CKB.
To address the issue of transaction construction, several teams have formulated step-by-step approaches and developed proprietary formats to store intermediate results. For instance, ckb-cli incorporates additional fields for signing informations 2 and stores them into a JSON file within the tx
sub-command. Similarly, Lumos offers the TransactionSkeleton interface to construct transactions. However, the use of proprietary formats prevents collaboration across different tools. For example, it’s impossible to construct a transaction in Lumos first, then sign it using ckb-cli.
To address this, the OTX Format has been introduced as a transaction format specification standard that promotes collaboration and component reuse, enabling seamless movement of transactions between components, processes, and machines across the network. The proposed specification, submitted as PR#406 4, is available for review in the Nervos Network/rfcs repository.
The OTX Format stores metadata as key-value pairs, with keys as 32-bit integers and values as opaque data types. The key determines the encoding of the corresponding value. Applications serialize and deserialize values of interest while treating others as raw byte arrays. This modular design decouples dependencies and enables the creation of simple, reusable, and composable components.
To avoid key conflicts, the community should collectively determine the allocation of keys via the RFC process, where the implication and usage of the keys are defined, specified, and reserved in the RFC repository. The RFC repository serves as a registry for sharing information among transaction construction applications. Proprietary keys can be used by applications without community consensus if they are not transmitted over the public network. A rich library of shared keys is essential for facilitating successful collaboration among applications using the OTX Format.
A typical pattern to use the keys is to store commands that instruct a service to construct transactions. For instance, a service may have a key named TRANSFER_CKB
. The code in Listing 1 below shows an example of the value. From the code snippet, it is easy to derive that the user “alice@ckb” intends to transfer 1,000 CKB to “bob@ckb”, where the fee rate must be within the range of 0.0001 to 0.00011. As depicted in the example, the sender and recipient addresses use an email-like format, and the service can lookup the corresponding CKB lock script in the name registries. Clients who wish to transfer CKB can initiate an open transaction by setting TRANSFER_CKB
without needing to know the underlying details. The service will use the TRANSFER_CKB
key to generate a valid CKB transfer transaction, much like calling a method transferCKB
. This pattern has evolved into the OTX Streaming Pattern, which is discussed in Chapter 3.
Listing 1: Example value of the key TRANSFER_CKB
{ "from": "alice@ckb", "to": "bob@ckb", "amount": "1000", "minFeeRate": "0.0001", "maxFeeRate": "0.00011", }
3 OTX Streaming Pattern
The Streaming Pattern is an architecture that enables software to react and operate as events occur. This pattern allows for software components to work together in a real-time, decoupled, and scalable fashion. It is well suited for the development of modern real-time distributed systems, such as dApps. Confluent offers an excellent introduction to the Streaming Pattern; 1 and ReactiveX an example of the Streaming Pattern framework that supports numerous languages.
The OTX Streaming Pattern uses the OTX Format as the event payload. Autonomous OTX Agents subscribe to the OTX Stream to receive notifications of new open transactions and then process them selectively based on their criteria. The agents also emit modified or new transactions, which are merged into the OTX Stream.
Figure 1: OTX Streaming Pattern
📷
As demonstrated in Figure 1, one of the components is optional. For instance, an agent that receives transactions from RPC will only require an OTX Source, while an agent that sends finished open transactions to a CKB node only needs an OTX Processor.
To further illustrate the pattern, here are some examples of agents.
- RPC Emitter receives transactions from an RPC endpoint and forwards them to the OTX Stream.
- CKB Sender sends completed transactions to the CKB RPC endpoint.
- Signer identifies signing requests in the stream via a registered OTX Format key and stores them locally. A wallet application retrieves the pending signing requests from the Signer agent for users. These requests are presented to users as a list in the application’s UI. After reviewing the requests, users authorize the wallet application to sign the transactions using their private keys. The application then submits the signed transactions to the RPC Emitter agent. Once the Signer detects the presence of the signed transactions in the stream, it removes the corresponding local unsigned versions.
- Atomic Swap Matcher functions as an order book for the atomic swap proposals. It indexes the swap proposals locally and offers an RPC to search for these proposals. Clients have the option to search their own proposals to check their status or search proposals made by others to take the orders. The agent also tries to merge matched proposals and emit the merged open transaction to the OTX Stream. Therefore, the Atomic Swap Matcher acts the roles of both OTX Processor and OTX Source.
4 OTX Lock
CKB lock scripts typically require users to sign the entire transaction, which involves two steps to complete an open transaction. First, users need to send the initial open transaction. Then, they must wait for its completion before signing it. In case of failure, users have to wait for a new completed transaction to sign again.
The OTX Lock offers partial signing mechanisms that protect specific properties of a transaction while leaving the other parts free to change. The signature remains valid as long as the signed properties remain unchanged. This enables users to pre-sign open transactions using OTX Lock, facilitating a fire-and-forget style similar to typical CKB transaction processes.
In the following sections, we will discuss two different ways to design the Lock: the Instruction List Lock and the Sighash Lock. Furthermore, Chapter 6 will present several new directions for further exploration.
4.1 Instructions List Lock
The Instructions List Lock enables users to specify the content for inclusion in the digest message for signing through an instructions list. Each instruction adds transaction properties to the digest, such as a full input cell or the number of output cells. We previously presented a design proposal in the RFC: Composable Open Transaction Lock Script 2.
We have tried to use the lock in different scenarios; two of the most studied ones are Atomic Swap and Unilateral Payment.
- In Atomic Swap, users send swap proposals as open transactions. These proposals describe the assets and quantities users wish to obtain, as well as the assets and quantities they want to pay. Each proposal is unbalanced, and it is the responsibility of the Atomic Swap OTX Agent to merge the matched proposals into a balanced CKB transaction.
- In Unilateral Payment, we want to design a mechanism based on open transactions to pay assets without interactions from payees. The requirement is from the scenario that users pay small amounts of CKB or user-defined tokens (UDT) but don’t want to incur the storage costs of creating a cell for the payee.
The design of the transaction structures and the instruction lists can be found in an older revision of the repository EthanYuan/open-transaction-pool.
During the experiments to design those scenarios, we experienced significant drawbacks of the proposal. It’s challenging to create a secure instructions list, and the field-by-field inclusion algorithm is not powerful enough for most complex dApps.
The security issues originate from the uncertainty of uncovered transaction fields, making it difficult to predict how open transactions will be consumed. In subsequent paragraphs, we will recapitulate two categories of security issues.
The first category is the Replay Attack, which occurs when an attacker reuses the signature created from a specific instruction list to gain access to other assets owned by the same user. This can occur when the instructions list does not include a field unique to the current transaction. To prevent this, users must avoid signing any unknown digest messages. Additionally, instructions lists must include at least one input cell outpoint to ensure uniqueness.
The second category of security issues arises from the absence of support for Cell Grouping. One of the key features of OTX is the ability to merge open transactions. However, when multiple transactions are merged, it becomes impossible to reference cells based on their absolute locations. To address this, the proposal introduces the method of referencing a cell using the relative index, which is a number relative to a base value. While anyone is allowed to modify the base values, it is typically the responsibility of the Agent who merges open transactions to set the appropriate values without compromising the existing signatures. See how Listing 3 sets the base values for the Open Transaction 2 after merging two atomic swap proposals.
Listing 3: Atomic Swap Merging Example
Open Transaction 1: # Alice wants to get 20 SUDT X by paying 200 CKB inputs: 0: CKB 200 owned by Alice outputs: 0: SUDT X 20 owned by Alice inputWitnesses: 0: # Initially, both base values are set to 0 inputBase: 0 outputBase: 0 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 2: # Bob wants to get 200 CKB by paying 20 SUDT X inputs: 0: SUDT X 20 owned by Bob outputs: 0: CKB 200 owned by Bob inputWitnesses: 0: # Initially, both base values are set to 0 inputBase: 0 outputBase: 0 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 1 + 2: # Make proposals from Alice and Bob inputs: 0: CKB 200 owned by Alice 1: SUDT X 20 owned by Bob outputs: 0: SUDT X 20 owned by Alice 1: CKB 200 owned by Bob inputWitnesses: 0: # Transaction 1 comes first, so the base values are zeros. inputBase: 0 outputBase: 0 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." 1: # Shift the base values by one to reference the correct cells. inputBase: 1 outputBase: 1 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..."
Allowing anyone to set the base values is dangerous, as adversaries can exploit this by reusing a cell in different open transactions. This is possible because output cells in the CKB transaction lack unique identifications. Let’s revise the swap example by splitting Bob’s proposal into two identical proposals as in Listing 4. By reusing the last output, Bob only receives 100 CKB instead of the intended 200 CKB.
Listing 4: Cell Output Reusing
Open Transaction 1: # Alice wants to get 20 SUDT X by paying 200 CKB inputs: 0: CKB 200 owned by Alice outputs: 0: SUDT X 20 owned by Alice inputWitnesses: 0: inputBase: 0 outputBase: 0 instructions: - "inputs[inputBase + 0]" - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 2: # Bob wants to get 100 CKB by paying 10 SUDT X inputs: 0: SUDT X 10 owned by Bob outputs: 0: CKB 100 owned by Bob inputWitnesses: 0: inputBase: 0 outputBase: 0 instructions: - "inputs[inputBase + 0]" - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 3: inputs: 0: SUDT X 10 owned by Bob outputs: # This output is identical to the one in Transaction 2 0: CKB 100 owned by Bob inputWitnesses: 0: inputBase: 0 outputBase: 0 instructions: - "inputs[inputBase + 0]" - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 1 + 2 + 3: # Make proposals from Alice and Bob inputs: 0: CKB 200 owned by Alice 1: SUDT X 10 owned by Bob 2: SUDT X 10 owned by Bob outputs: 0: SUDT X 20 owned by Alice 1: CKB 100 owned by Bob # The adversary removes the output from Open Transaction 3 inputWitnesses: 0: inputBase: 0 outputBase: 0 instructions: - "inputs[inputBase + 0]" - "outputs[outputBase + 0]" signature: "0x..." 1: inputBase: 1 outputBase: 1 instructions: - "inputs[inputBase + 0]" - "outputs[outputBase + 0]" signature: "0x..." 2: inputBase: 2 # The adversary sets this to 1 to reuse the output in Transaction 2 outputBase: 1 instructions: - "inputs[inputBase + 0]" - "outputs[outputBase + 0]" signature: "0x..."
In conclusion, a cells-grouping mechanism that prevents tampering by adversaries is crucial for merging open transactions. Implementing the mechanism in CKB is much simpler compared to using contracts, but it requires careful design to minimize overhead and ensure compatibility.
There is a concept called script group, which is similar to cell grouping. It involves grouping cells based on scripts and arguments and then running the script once for each group. However cells in an open transaction often do not align with a script group. What’s even worse is that there are often overlaps between open transactions and script groups. If a lock script fails to iterate through all the cells in the group or limit the number of cells, it becomes vulnerable to an attack where adversaries can append input cells that belong to the same script group. These additional inputs get unlocked along with existing ones without requiring a new signature from the user. This clearly violates the intentions of the user. Listing 5 depicts an OTX Lock implementation that is vulnerable, as it only verifies the first input witness in the group. Listing 6 demonstrates an attack that exploits this vulnerability. Pay attention to input 2 of the merged transaction. Specifying the number of input cells when signing can solve this issue, but would prevent merging open transactions from the same user into a single CKB transaction. This is a common scenario in merchant apps where a buyer purchases multiple items in a single transaction. Restricting the number of input cells would result in a complex design where goods belonging to the same seller cannot be combined in the same transaction.
Listing 5: A vulnerable OTX Lock which verifies only the first input witness in the group
function main() { const { instructions, signature } = getInputWitnessInGroup(0); const digest = generateDigest(instructions); verifySignature(digest, getPubkeyFromArgs(), signature); }
Listing 6: An attack that appends cells in the same script group
Open Transaction 1: # Alice wants to get 20 SUDT X by paying 200 CKB inputs: 0: CKB 200 owned by Alice outputs: 0: SUDT X 20 owned by Alice inputWitnesses: 0: # Initially, both base values are set to 0 inputBase: 0 outputBase: 0 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 2: # Bob wants to get 200 CKB by paying 20 SUDT X inputs: 0: SUDT X 20 owned by Bob outputs: 0: CKB 200 owned by Bob inputWitnesses: 0: # Initially, both base values are set to 0 inputBase: 0 outputBase: 0 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 1 + 2: # Make proposals from Alice and Bob inputs: 0: CKB 200 owned by Alice 1: SUDT X 20 owned by Bob # Adversaries append the cell which is in the same script group as input 0 2: CKB 1000 owned by Alice outputs: 0: SUDT X 20 owned by Alice 1: CKB 200 owned by Bob inputWitnesses: 0: # Transaction 1 comes first, so the base values are zeros. inputBase: 0 outputBase: 0 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." 1: # Shift the base values by one to reference the correct cells. inputBase: 1 outputBase: 1 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..."
Another drawback that was mentioned earlier is that the Instruction List Lock is not powerful enough. For instance, it is not possible to replicate the logic of Anyone-Can-Pay (ACP) Lock using an instruction list, because there are no commands available for performing arithmetic calculations and comparisons. We have been cautious about adding new instructions, due to the possibility of requiring endless additional instructions. The sustainable approach would be through script composition, where dApps expose verified assertions for signing. There are two threads 4 in the CKB GitHub repository relevant to this topic.
4.2 Sighash Lock
To simplify the design of the instructions list, why not limit users to a set of established patterns for signing transactions? This is where the concept of OTX Sighash Lock comes into play.
The OTX Sighash Lock design is modeled on the Bitcoin Sighash pattern. If you want to learn more about the Bitcoin Sighash, check out the tutorial from saylor.org: CS120: Bitcoin for Developers I, Elliptic Curve Signatures. For those interested in implementing OTX Sighash Lock, refer to the following Github repository: EthanYuan/otx-sighash-lock.
Although the Sighash pattern provides only six options, it does not simplify the issue. Rather, the complexity is shifted elsewhere. Sighash pattern requires an elaborate transaction layout design, as seen in the Atomic Swap demo, where users need to prepare a dedicated cell for the open transaction instead of using existing ones. For a reference, see the atomic swap documentation 2. By contrast, the older Instructions List Lock design gave users more freedom to choose from existing cells.
5 OTX Framework
OTX Framework combines OTX Format, OTX Streaming Pattern, and OTX Lock together to provide an easy-to-use instance for developers.
The central component of the framework is the implementation of the OTX Streaming Pattern, referred to as the open transaction pool. A Rust Proof of Concept (PoC) is available on GitHub at EthanYuan/open-transaction-pool 1.
Using the open transaction pool as a foundation, we can develop various agents, such as Atomic Swap and Signer. In our vision, there will be a public marketplace where agents can be shared. Developers can easily incorporate agents from this marketplace and construct complex transactions through agent composition. Certain agents, such as the one designed to collect live cells based on a specific criterion, can significantly facilitate the transaction construction process.
The SDK acts as the interface of the framework for developers. We already have code snippets to work with Open Transactions in Rust and Javascript 2, but these are far from an SDK for the framework.
6 Future Works
6.1 New Directions of OTX Lock
Both the Instruction List Lock and Sighash Lock require a redesign of the Cell Grouping mechanism. In the short term, it is necessary to integrate a solution into the OTX Lock. In the long term, further research is needed to determine how to support cell grouping in the CKB transaction structure.
There are also other mechanisms of partial signing, such as signing the user intent rather than specific fields. An intent is a message indicating the operation that the user wants to perform, such as the example in Listing 7.
Listing 7: An intent to swap 10 CKB with 50 SUDT
{ "app": "0x...", "nonce": 1, "command": "swap", "from": { "assets": "ckb", "balance": "10", }, "to": { "assets": "sudt", "id": "0x...", "balance": "50", } }
The dApp checks the intent has been successfully carried out. The transaction properties that do not affect the intent execution are free to change. For example, the intent above does not care which inputs have been collected to provide the 10 CKB balance; it only needs to check whether the user’s CKB balance has decreased by exactly 10 CKB.
The dApp verifies the successful execution of intent. The transaction properties that do not affect execution are free to alter. For instance, the specific inputs used to attain the 10 CKB balance do not matter - only the user’s CKB balance being reduced by precisely 10 CKB matters.
To prevent repay attacks, it’s crucial to implement a mechanism to make the intent unique, such as the nonce
field in the example.
Intent functions as the instruction for constructing a transaction as well. This means that dApps can use the same logic for both constructing and verifying transactions. The verification code rebuilds the transaction using the intent and checks that it matches the target transaction being verified.
6.2 Others
- A public P2P network to exchange open transactions is crucial for the adoption of dApps. We could incorporate an opt-in protocol in CKB to relay open transactions.
- Create a uniform interface for transaction construction across SDKs, for both CKB and open transactions. Developers can experience a seamless transition with a turn to the open transactions solution.
- Improve tools for better developer experience.
- We already have code snippets to work with Open Transactions in Rust and Javascript 2. We could improve the JavaScript SDK further. Besides working with the OTX Format, the SDK must hide the complexity of the OTX Lock and support a mechanism to seamlessly integrate the features provided by agents.
- We have a Rust PoC for the Streaming Pattern available on GitHub at EthanYuan/open-transaction-pool 1. We could implement the Streaming Pattern and all agents in JavaScript.
We're excited to see the potential of Open Transactions unfold and how they can revolutionize the way people interact with the CKB network and its dApps. Keep an eye out for more updates and developments from Nervos!
https://preview.redd.it/haovtlocl0pb1.png?width=1200&format=png&auto=webp&s=65645b3b5480228c4205a4650d3b20e14707fa37
