On August 8th 2019, the Elastos Main Net experienced a “pause” at block height 439,668. At the time of this incident, a brief summary was published to explain the cause of the issue. This article serves as a comprehensive attempt at analyzing the root causes behind the blockchain pause, how a quick temporary fix was put in place to resolve this issue and what the long term implementation strategy is regarding the blockchain code changes. In summary, the code will undergo numerous upgrades that will further strengthen the way data is processed and stored on the blockchain, thereby preventing this issue from occurring in the future.
The corresponding pause has been attributed to the following two causes:
- Root Cause: Competing conditions were created when an inactive transaction and a confirmed block without such a transaction existed at the same block height. Once one of the competing conditions was communicated to any node, the node would then refuse to accept the second condition, thus preventing participant DPoS nodes from reaching consensus due to the chronological discrepancies in ordering.
- Causal Factor: An inactive switchover occurred at this particular block height because a high volume of input was taking place in the block which involved the corresponding query operations precursive to transactions. Unfortunately, there was a bottleneck in the performance of this query type.
In response to the identified causes, a number of adjustments have already been completed, and further work is currently in progress. All is outlined below:
Competing Conditions in Consensus
Inactive switchovers have already taken place many times prior to the blockchain pause – not because 1/3 of DPoS nodes were unable to operate normally, but because a large volume of pre-query transactions (i.e. a high volume of transaction input) caused a longer validation time, thereby affecting the consensus initiation time of each individual DPoS node. In this particular case, when the inactive switchover occurred, the entire network was unable to promptly recover to the present block height mainly because a certain amount of time is required for an incoming arbitrator to establish direct connections with other arbitrators. If this process is excessively lengthy and is affected by the delay stemming from the large volume of transaction pre-queries, then there is potential that an inactive switchover can be triggered again. Seeing as the present dilemma may be reproduced, and seeing as it does not serve the original intent of designing an inactive transaction used for accelerating consensus error recovery, all conditions in which inactive transactions may potentially trigger blockchain pauses have been removed from v0.3.6.
Because versions later than v0.3.6 will no longer have inactive transactions, errors relating to competing conditions in inactive transactions will no longer arise, thereby permanently resolving the root cause.
At this point, it has been established that the inactive switchover is temporarily prevented, due to a bottleneck for blockchain block processing performance (for more detail, see Performance Optimization further below). However, once performance optimization is improved, the inactive switchover will be triggered once more to resolve the problem of 1/3 of arbitrators being unable to function normally. In addition, we are also considering a possibility where, in the extreme case where a majority of current arbitrators are unable to function normally, consensus block production can revert to PoW. When arbitrators are ready, DPoS+PoW consensus block production can then be resumed. Of course, in this circumstance, only a certain portion of transactions would be allowed during PoW block production (for example, cross-chain transfer transactions during PoW block production will not be prohibited), in order to maximize security on the main chain and side chains.
The consensus mechanism improvements will require ample proof and testing. At the present time, the majority of development efforts are being dedicated to CR-related functions and optimization. For this reason, the official initiation of long-term solutions are slated to follow the release of Cyber Republic Stage Two.
Other Competing Factors
In addition to the aforementioned competing conditions related to inactive transactions, there is another competing condition that remains on the main chain: two blocks with different view offsets can coexist at the same height. While these competing conditions can be fixed automatically by the blockchain’s fork reorganization mechanism in order to avoid a full blockchain pause, the fork reorg still involves changes to the status of all internal memory. Internal memory statuses and related transactions – including the amount values in the transaction outputs – are all connected, and if the network frequently experiences instances of competing conditions, it will create status abnormalities. To address these issues, solutions have been implemented in v0.3.8 and v.0.3.9.
After Cyber Republic Stage Two is released, the Elastos Foundation plans to more comprehensively resolve the frequent occurrence of competing conditions at the DPoS consensus level.
As established above, there is currently a bottleneck in block processing which will expand the current time difference between arbitrator sequential consensus initiations and thus further extend consensus completion time and increasingly create competing conditions. Consequently, performance optimization has been the main focus of our recent work and it is the target of our future development plans concerning key improvements. The following is an outline of the main performance optimization work for Cyber Republic Stage 1 and Cyber Republic Stage 2:
Cyber Republic Stage 1
The UTXO cache pool and database are the two main aspects which have been optimized during this stage:
UTXO Cache Pool
Because Elastos uses a UTXO model, each block contains many transactions, and each transaction is composed of numerous UTXOs. In order to verify each transaction, UTXO queries must be conducted on the part of the DPoS nodes that provide finality for each blocked solved by PoW miners. Additionally, DPoS nodes have only five seconds to validate each block with triple efficacy. In the case of a block containing many high-UTXO transactions, DPoS nodes are unable to validate the entire block’s contents within the allotted time, as each UTXO query accesses the database, thus extending the processing time. To resolve this issue, a cache (temporary memory) pool has been developed in order to dramatically accelerate each UTXO query, thereby reducing the total verification/consensus time.
Block Data Scraped from the Database
The entirety of the block data in the current version is saved in the leveldb database, which is not advantageous for a database that is used for query optimization cache – in other words, the greater the data, the lower the buffer hit rate. Consequently, we will save blockchain data in document format, and will only save block-related index data in the database, thereby increasing database query speeds.
After these changes are made, the part of the database saving blocks will only save metadata information, but the database will still contain some data which is not metadata, but rather is full-block data. This part of the remodeling work will be completed during Cyber Republic Stage Two.
Cyber Republic Stage Two
The main optimization work that will take place during Stage Two will include:
Database Transformation into Metadata
To continue the database remodeling efforts which were started during Cyber Republic Stage One, we will completely transform the database into an index database for saving metadata. The transformed index database will mainly serve two aspects of data migration:
- UTXO Data: Includes UTXO data indexed by addresses and mainly serving wallet services, as well as UTXO data that is indexed by transaction hash and transaction outputs that are predominantly used for double-spend inspections.
- Confirm Data: Includes Block confirmation information.
In addition to improving performance, database transformation into metadata also supports the recovery of index databases for saved metadata through block document data, which can optimize data copy deployment and disaster recovery processes.
UTXO Subdivision according to Use Scenario
Further subdivision and responsive optimization performance according to UTXO use scenarios will be performed on the basis of Stage One. These processes will primarily involve the following components:
- UTXO Data Storage Indexed by Address: After breaking into parts, this part of the cache is mainly used for wallet services. Because this is where the block processing performance bottleneck occurs, we can further support configurable registration addresses and cut this part of storage according to address, thus enabling major accelerations in block processing performance.
- UTXO Cache Indexed by Transaction Hash: Generally used for UTXO access operations. Using the LRU strategy and proposing a configurable size enables DPoS nodes with good performance to optimize according to changing conditions.
- UTXO Data Cache used for Cross Chain Transactions: Cross chain transactions are comprised of a series of special addresses, mainly used for cross-chain consensus.
- UTXO Cache used for DPoS Voting: DPoS voting is hot data, and must be cached to optimize transaction processing speeds for processing DPoS votes.
- UTXO Cache used for CRC: A cache for CRC that mainly involves transactions related to both voting and addresses for throughput statistics.
In fully analyzing the blockchain pause that took place on August 8th, implementing strategic technical adjustments, and preparing a comprehensive future development plan (see: Cyber Republic Phase 2), the Elastos Foundation is successfully strengthening the performance of its blockchain processes and enhancing its overall security.