- WOO X experienced around 2 hours of being offline, with full service restored in 11 hours
- 227 users who experienced liquidations as a result of abnormal liquidity conditions are being compensated
- 100% of withdrawals were processed (~$13m in net outflows over a 48-hour period)
- 3 additional market-making teams onboarded to reduce reliance on any single liquidity provider
- WOO X has fast-tracked the introduction of new competitive and sustainable market maker incentives
On Saturday, November 18, WOO X received notice from its largest liquidity provider, Kronos Research, that they would need to shut down their trading on WOO X for an indefinite period of time while they investigated the source of a security incident. According to the information received, several of their exchange API keys had been compromised, which resulted in $26 million USDT worth of losses for Kronos at the time of writing.
Three transactions had been withdrawn from the Kronos Research account on WOO X by the hacker, which totaled just under 6 million WOO tokens (approximately $1.25m USD value). A fourth transaction, for $318k USDT, was manually rejected and restrictions were placed on the account, blocking further withdrawals by the attacker. This rejection began the series of events that led to WOO X temporarily pausing operations. After several hours and collaboration with designated market makers (DMMs), WOO X resumed trading for Perpetual Futures markets, followed by withdrawals and spot trading. According to our records, we will outline a timeline of major events and planned steps to be taken to ensure a similar outage on WOO X will not occur again. Based on our analysis, 227 users experienced partial or full liquidations as a result of limited liquidity during the restart process, of which all were compensated, regardless of whether they contacted our support team.
Regarding WOO and Kronos
Although the two companies function as entirely separate entities, Kronos Research did incubate WOO Network in 2019, and presently serves as the backstop liquidity provider for WOO X. Originally known as Wootrade, the network was initially envisioned as a solution that exchanges and other platforms could connect via API to access Kronos’ liquidity for their end users. It evolved over time to have a user interface and suite of products, like Perpetual Futures and earn products, that appeal to retail investors and professional traders alongside the initially targeted institutional clients.
During this evolution, the crypto asset market has gone through many events, most notably FTX. Since the FTX collapse, WOO Network has made concerted efforts to reduce reliance on Kronos Research for liquidity through changing its fee structure in July and onboarding more market makers through a Designated Market Maker (DMM) program in August.
Over the past few months, these DMMs have made up almost half of all Perpetual Futures volumes on the platform. We have always tried to balance maintaining operations and sustaining our revenue with improving our operational model and integrating more market makers, all while offering world-class liquidity and user experience. With the events of this past weekend, our plans to improve platform resiliency through diversifying liquidity providers have been accelerated significantly.
Regarding our incident response
Upon confirming there was an issue with Kronos, the WOO team quickly activated over 50 members of the 180-person team to manage the incident and resume operations. Many of these team members were based in Asia, meaning the incident response began between 3 and 5 AM locally.
To protect users from abnormal liquidity caused by Kronos halting their market-making activities, the team reached a consensus to pause all operations, and then separate the restart processes into three areas - Perpetual Futures, Withdrawals, and Spot. Restarting Perpetual Futures markets was the immediate concern because users needed to be able to manage leveraged positions. Priority was given to ETH and BTC markets, where a majority of the open interest was concentrated.
Withdrawals were not reopened until internal teams could assess erroneous balances for users whose limit orders were filled by the 227 incorrect liquidations, a result of the abnormal liquidity conditions after Futures trading had been reopened.
Many actions then began in parallel:
- Our business development team began the process of contacting DMMs, to communicate the upcoming relaunch schedule and help size up their liquidity to step in for Kronos Research
- Our PR team began a communication plan to announce every step of the process
- Our operations and product teams began to relaunch each trading market in coordination with the DMMs
- Our risk team needed to ensure that users could manage their positions and avoid unnecessary liquidation given poor market liquidity
- Customer support team worked to assist users who had questions or concerns
Early in the morning of Sunday, a malicious security firm began targeting WOO X’s frontend with a DDoS attack. This is believed to be unrelated to the hack, as the entity has repeatedly solicited payment in the past. Our security team adjusted various rules to reduce the effect of concentrated traffic on WOO X’s performance.
Timeline of key events:
Note: All times are UTC and approximated
Saturday, Nov 18th
19:00: WOO X received a few unusual withdrawal requests from the Kronos account
19:30: Confirmed with Kronos that activity was part of an ongoing hack, and halted their withdrawals. Began investigation and organized teams to take next steps.
20:30 Received notice from Kronos that they would pause providing liquidity on WOO X while investigating the incident
21:00 Made decision to halt operations to protect user positions from incorrect liquidations in the absence of regular market liquidity
21:30: Announced the temporary halting of all operations, including trading on WOO X via support center and Telegram
22:00 Began communicating with market makers and executing plan to restart trading operations
23:15 Began our status update Tweet thread to keep the users and general public informed
23:30 Restarted trading on ETH-Perp and BTC-Perp markets
Sunday, Nov 19th
00:30 Restarted remaining 134 perp markets
04:30 Reopened withdrawals
05:30 Experienced first wave of DDoS attacks targeting frontend
08:00 Restarted spot trading
By early morning UTC on Sunday, all of the core exchange functions had been restored and were operating normally. We provided 8 status updates via Twitter, keeping the community and users informed throughout the incident and response.
Regarding WOOFi
WOOFi Pro, the orderbook DEX, functioned as normal throughout, as liquidity has been provided by multiple market makers since its inception. WOOFi Swap, which simulates CeFi liquidity on-chain by active quoting from Kronos Research, routed all transactions through 1inch for a period of around 12 hours while quoting was restarted. Assets in on-chain earn vaults were unaffected, with the amount of staked WOO tokens on WOOFi actually increasing throughout the weekend.
How do we prevent this from happening again?
Each stress test that WOO X undergoes brings essential insights into ways we can continue to improve our product and safeguard our users’ best interests. Over the last 3 months, we have been continuing to reduce Kronos’ role in providing liquid markets on WOO X - with the DMM program being launched in August, resulting in a reduction in Kronos’ share of maker volume on Futures by almost 50% in this time.
The events that unfolded during the weekend have further proven that these efforts to diversify liquidity on WOO X are of paramount importance and urgency. While Kronos is a valuable partner of WOO, it is in the interest of minimizing the platform’s counterparty risk to further reduce reliance on Kronos for liquidity. As a result, we will now be allocating maximum resources in the pursuit of accelerating these ongoing initiatives:
- Onboarding more industry-leading market makers and introducing sustainable and competitive market maker incentives, eliminating dependence on any single market maker
- Speeding up improvements on platform performance that will enable our DMMs to provide liquidity more effectively
- Making liquidation, mark price, and funding rate logic more resilient to extreme liquidity situations, creating a more robust trading environment and reducing the need to halt trading in the future
- Continue plans for launching a spot DMM program, improved credit margining system, and a combined spot margin & perps mode
- Communicating these steps clearly to our users, allowing them to understand our approach and uphold a high level of trust
Special thanks to a few key stakeholders:
- Market makers, including Selini Capital and Black Code Group, who responded quickly to spin up liquidity and supported throughout the night
- Ecosystem partners, who reached out and offered well-wishes and support
- Users, who showed us patience and understanding
- Community, who showed up in the face of strong skepticism on social media to voice their encouragement and support
- Our team, who worked tirelessly to resume operations and support our users
- Our critics, who always push us to be even more transparent even when they shout inaccuracies and whisper our accomplishments
The content above is neither a recommendation for investment and trading strategies nor does it constitute an investment offer, solicitation, or recommendation of any product or service. The content is for informational sharing purposes only. Anyone who makes or changes the investment decision based on the content shall undertake the result or loss by himself/herself.
The content of this document has been translated into different languages and shared throughout different platforms. In case of any discrepancy or inconsistency between different posts caused by mistranslations, the English version on our official website shall prevail.
