Market shut down

Short version

If there is currently an emergency and you need to shut down a market immediately, you can get the appropriate CW3 multisig information from running the following in the perps repo:

cargo run --bin perps-deploy mainnet wind-down --factory osmomainnet1 --impacts new_trades --kill-switch

Or to shut down specific markets:

cargo run --bin perps-deploy mainnet wind-down --factory osmomainnet1 --impacts new_trades --kill-switch --market ATOM_USD --market ryETH_USD

Then use the smart contract GUI or Apollo Safe to make the recommended proposal, have the multisig participants vote, and then execute.

With the short version done, let's get into details.

Factory shutdown messages

Levana Perps is made up of multiple contracts, but the two most important for us are:

  • Factory: this is the central hub for managing all the different markets. Each chain has a single factory. We name them things like osmomainnet1 and seimainnet1.
  • Market: each individual market is its own contract deployment, and is always associated with a single factory.

The factory contract supports a shutdown message which can be used to disable (and then re-enable) different pieces of individual markets. For example, you can disable the ability to open new trades and deposit liquidity, but allow people to close their existing positions. The details of what can be shut down are available in the docs above.

The shutdown message is permissioned, allowing only authorized wallets to perform these actions. And we have two different wallets authorized for performing most shutdowns. Which brings us to...

Kill switch vs wind down

The kill switch is intended to cover emergency situations, such as "an exploit is happening right now." By contrast, the wind down wallet is intended for longer term operations, like "we're phasing out this market in favor of another one." Permissions-wise, the wind down wallet has the ability to force-close all positions in a market, which the kill switch wallet does not have.

Currently, with the way we've set up our multisigs (more on that below), the same people have access to both the kill switch and wind down mechanisms. But in the future, the plan is that different groups of people would have access to each.

Generally, it's a good idea to use the right wallet for the job. If you're trying to shut down markets because of an emergency, use kill switch. If a market needs to be slowly shut down, use wind down. But in practice, it doesn't matter which one you use, as long as it has the right permissions.

Multisigs

If the kill switch and wind down were simple hot wallets, or even hardware wallets, it would mean that one person has direct control over shutting down markets. That's a situation we want to avoid. Instead, all our mainnet factories are configured to use multisig contracts for their kill switch and wind down mechanism. Specifically, we use a cw3 flex multisig with a cw4 group. At time of writing, each multisig is a 3 of 5 wallet, meaning there are five parties that can vote, and any proposal requires at least 3 yes votes to pass.

Information on wallets and multisigs is available on Google Drive.

The factory contract will only accept shutdown messages from one of these multisigs. To trigger a shutdown, you need to follow these steps:

  1. Create a multisig message. This is a JSON value indicating what contract you want to interact with, the message to send to that contract, and other details that aren't usually necessary. We'll discuss this step in the next section.
  2. One of the multisig holders needs to create a proposal. When you create a proposal, you need to provide the multisig contract address, a title, a description, and the JSON value from the previous step. Remember that title and description are publicly viewable on-chain, and should be chosen with that in mind. Proposals can be created with the smart contract GUI or Apollo Safe. (Note: Apollo Safe does not support Sei.)
  3. Two other multisig holders must then vote yes on the proposal. This can be done from either smart contract GUI or Apollo Safe.
  4. Once voted on, anyone--even people outside of the multisig group--can execute the proposal.

Here's a sample of a multisig proposal you might make:

Multisig contract: osmo1g03ue88t8ufu5f6pste3x52dfp72njmedp5hnf8q6z0fz0jgw6nq27mrl0
Title: Testing out shutdowns
Description: This is a stress test trying to shut down the ATOM_USD and ryETH_USD markets.
Message:

[{"wasm":{"execute":{"contract_addr":"osmo1ssw6x553kzqher0earlkwlxasfm2stnl3ms3ma2zz4tnajxyyaaqlucd45","msg":"eyJzaHV0ZG93biI6eyJtYXJrZXRzIjpbIkFUT01fVVNEIiwicnlFVEhfVVNEIl0sImltcGFjdHMiOlsibmV3X3RyYWRlcyJdLCJlZmZlY3QiOiJkaXNhYmxlIn19","funds":[]}}}]

The long bit of weird text starting with eyJz is a base64-encoded contract message, which decodes to:

{"shutdown":{"markets":["ATOM_USD","ryETH_USD"],"impacts":["new_trades"],"effect":"disable"}}

This lines up with the shutdown message in the API docs listed above. But constructing these by hand is tedious. So instead, we have...

perps-deploy

The perps-deploy tool is part of the levana-perps repo. It provides lots of different helper functionality as subcommands, including the ability to automatically create these shutdown proposals. The proposal above was created by using the following command:

cargo run --bin perps-deploy mainnet wind-down --factory osmomainnet1 --impacts new_trades --kill-switch --market ATOM_USD --market ryETH_USD

Which generates the following output:

[2024-03-21T11:33:41Z INFO  perps_deploy::app] Connecting to https://grpc.osmosis.zone
[2024-03-21T11:33:41Z INFO  perps_deploy::mainnet::wind_down] CW3 contract: osmo1g03ue88t8ufu5f6pste3x52dfp72njmedp5hnf8q6z0fz0jgw6nq27mrl0
[2024-03-21T11:33:41Z INFO  perps_deploy::mainnet::wind_down] Message: {"wasm":{"execute":{"contract_addr":"osmo1ssw6x553kzqher0earlkwlxasfm2stnl3ms3ma2zz4tnajxyyaaqlucd45","msg":"eyJzaHV0ZG93biI6eyJtYXJrZXRzIjpbIkFUT01fVVNEIiwicnlFVEhfVVNEIl0sImltcGFjdHMiOlsibmV3X3RyYWRlcyJdLCJlZmZlY3QiOiJkaXNhYmxlIn19","funds":[]}}}
[2024-03-21T11:33:42Z INFO  perps_deploy::mainnet::wind_down] Successfully simulated messages

Notable options:

  • You can use --market as many times as you want, but if you leave it out entirely, the message will target all markets in the factory.
  • --factory can take strings like osmomainnet1, but can also take contract addresses if necessary.
  • The --impacts can be specified as many times as you want, and indicates what you want to impact. See the API docs above for examples.
  • By using the --kill-switch flag, the message will be prepared against the kill switch multisig, not the wind down multisig.
  • If you include the --enable flag, instead of disabling functionality in the contract, it will turn it back on.

Wind down steps

The above gives the technical explanation for how to perform shutdowns. If there's a market that needs to be wound down, you should proceed with the following steps:

  1. Disable NewTrades, DepositLiquidity, and Staking.
  2. Wait at least 7 days.
  3. Execute CloseAllPositions on the market.
  4. At this point, the bots should ignore this market entirely.