Investigating user questions

"I didn't close my position"

This is a sample session of trying to track down a user bug report. This particular case ended up with no bugs at all (which is fairly common), but the important bit is the tooling and approaches we use for solving it. Let's start with the bug report:

wallet osmo1nvqn77ygcfm2w4rvqrwj7m4ugul73vt9ednt7j

position 3462

I was jumping between pages (trade and history) and the position closed without my intervention - I'm using Kepler wallet and was not asked for approval to close the position. I closed the position at a -12.65USD + fee loss as a result.

There are a few things to note about this right off the bat:

  1. It doesn't specify which chain the user was trading on. Fortunately, that's easy enough to figure out from the wallet address: Osmosis.
  2. It doesn't specify which market the uesr was trading in. Generally it's reasonable to respond to users directly and ask for clarification on this point. However, in this case, it was easy enough to figure it out without the user's input.

The first thing I did was viewed the site as the user. In the near future, this will be an official part of the public site. For now, it's only available on internal deploys of the site, such as the develop branch. From this page:

  • Make sure to switch networks to Osmosis mainnet
  • Click on connect wallet and copy-paste the wallet address above
  • Go to the history tab

From here, you can look for the relevant position from the "trade history" section. When I checked, the user had 11 positions in the history, and only one with an ID of 3462. That position is in the TIA/USD market, and shows a -12.65 USD for PnL, so it lines up with the user report. We'll go with that.

The next step is to determine the market contract address for the TIA/USD market. The easiest way to get that is to check out the frontend config endpoint from the bots. From this, I see a contract address for TIA_USD of osmo1kqzkupfec3zemmaj3kuhcf0h2wke02wa7sgp2a9vq5mugtgs5pzs8avjzt.

Next, we want to look at the on-chain history for this position. To do that, we need to run the following query against the contract:

{"position_action_history":{"id":"3462"}}

You can do this on the command line with the cosmos CLI tool:

➜ COSMOS_NETWORK=osmosis-mainnet cosmos query-contract osmo1kqzkupfec3zemmaj3kuhcf0h2wke02wa7sgp2a9vq5mugtgs5pzs8avjzt '{"position_action_history":{"id":"3462"}}'
{"actions":[{"id":"3462","kind":"open","timestamp":"1701710617851441037","collateral":"10.857540857765601958","transfer_collateral":"12","leverage":"10.947003784264420053","max_gains":"1.22606442383761503","trade_fee":"2.078089105405467622","delta_neutrality_fee":"7.907517698694531141","old_owner":null,"new_owner":null,"take_profit_override":"9.3","stop_loss_override":"8.15"},{"id":"3462","kind":"close","timestamp":"1701712745309314227","collateral":"10.639037850826950797","transfer_collateral":"10.639037850826950797","leverage":null,"max_gains":null,"trade_fee":null,"delta_neutrality_fee":"-5.798714565570383823","old_owner":null,"new_owner":null,"take_profit_override":null,"stop_loss_override":null}],"next_start_after":null}

Or in the smart contract GUI. Either way, we see that the position was closed at the timestamp 1701712745309314227.

If you go to epoch converter and enter that value, you get a timestamp of Monday, December 4, 2023 5:59:05.309 PM (in GMT). The next step is to figure out the block height for that block. My preferred way of doing that is to use the cosmos CLI tool:

➜ COSMOS_NETWORK=osmosis-mainnet cosmos chain first-block-after --timestamp 2023-12-04T17:59:05Z
12631606

With that in place, you can now go to Mintscan and find the block in question. It turns out that this block has 30 transactions, and in theory you'll need to look through each one to find either a crank message or something else related to the protocol. Basically: look for anything that's using the TIA_USD contract address mentioned above. Generally, you can look in the "messages" column and look for either the word "crank" or "contract-update price", which is what most Perps transactions look like.

Going one by one through the transactions, you'll eventually come to the relevant transaction. This transaction--like most transactions users send--has two messages. The first updates the Pyth oracle price. You can ignore that one. The second one is what the user actually did:

{
  "msg": {
    "close_position": {
      "id": "3462",
      "slippage_assert": {
        "price": "8.662108",
        "tolerance": "0.005"
      }
    }
  },
  "funds": []
}

Even if you aren't familiar with the internals of perps, it's pretty obvious that this is trying to close position #3462. Mintscan lists the sender as osmo1nvqn77ygcfm2w4rvqrwj7m4ugul73vt9ednt7j, which is the wallet address mentioned above.

At this point:

  • We've identified why the user's position was closed: the user (or someone controlling the user's wallet) closed it.
  • We have a Mintscan link demonstrating that.
  • We have no idea how this actually happened. Did the user make a mistake? Is the user's wallet compromised? Is the user pulling a prank on us? Is the entire blockchain broken and someone is randomly closing positions for fun?

In any event, there's no more work to be done on research here. Provide the link to the transaction back to the user, and call it a day.

"The price in the history never happened"

Relevant Jira bug report: UFI-78

Hi,

I just checked the WBTC/USD oracle feed for the day my position got liquidated and the lowest of the day was $40,662.70 and not 40,218.87 USD that it is written in history.

It clearly is a bug on your side and my position would be in quite good profit right now.

I expect you to please take a look at my issue asap and let me know whether you are going to reopen my position.

Thank you

position id is 6744
osmo13tqhtr7m67zw0hj7axrw9hap7dfds97rmkw8k7

Please see Zendesk Support tab for further comments and attachments.

Generally speaking, the correct response to this issue would be to point the user to the before reporting a bug page and ask them to provide comprehensive information to back up their claims. That didn't happen in this case, and we're going to use it as a learning exercise in how to investigate.

The first and easiest thing to do is to use the "view as" feature on the site to see that this position did, in fact, close with the price point provided in the bug report (40,218.87 USD). The next question is: was this a valid price point? One answer is "the smart contracts treated it as a price point so it must be valid," which is true. However, we can do better than this.

First, let's get the contract address for the BTC/USD market. My preference is to check the frontend-config file or the bot status page. Either way, you'll end up with the contract address osmo1nzddhaf086r0rv0gmrepn3ryxsu9qqrh7zmvcexqtfmxqgj0hhps4hruzu. Next, let's get the information about when this position closed. We can do that by performing the following query against the market contract:

{"position_action_history":{"id":"6744"}}

You can do this from the smart contract GUI and get the following response:

{
  "actions": [
    {
      "id": "6744",
      "kind": "open",
      "timestamp": "1702045382294420059",
      "collateral": "0.17710915390234318",
      "transfer_collateral": "0.179146",
      "leverage": "11.620755384767910956",
      "max_gains": "0.816302488254187237",
      "trade_fee": "87.510718328292878517",
      "delta_neutrality_fee": "1.412402852632010417",
      "old_owner": null,
      "new_owner": null,
      "take_profit_override": null,
      "stop_loss_override": null
    },
    {
      "id": "6744",
      "kind": "close",
      "timestamp": "1702323672974158242",
      "collateral": "0.010798702590289875",
      "transfer_collateral": "0.010798702590289875",
      "leverage": null,
      "max_gains": null,
      "trade_fee": null,
      "delta_neutrality_fee": "0.558535157234580985",
      "old_owner": null,
      "new_owner": null,
      "take_profit_override": null,
      "stop_loss_override": null
    }
  ],
  "next_start_after": null
}

We can see that the position closed at timestamp 1702323672974158242. It's not important what that converts to, but using Epoch Converter, we can see that it converts to Monday, December 11, 2023 7:41:12.974 PM. The next question is to find out the spot price in affect in the market at that time. We can do that using another market query:

{"spot_price":{"timestamp":"1702323672974158242"}}

Using smart contract GUI we get the response:

{
  "price_notional": "0.0000248639504275",
  "price_usd": "40218.870405",
  "price_base": "40218.870405",
  "timestamp": "1702323672974158242",
  "is_notional_usd": true,
  "market_type": "collateral_is_base",
  "publish_time": "1702323659000000000",
  "publish_time_usd": "1702323659000000000"
}

Looking at the publish_time field (which is identical to publish_time_usd in this case, since both base and collateral prices are the same BTC/USD feed), we see that the price point was published by Pyth at 1702323659000000000. This value is given in nanoseconds since the epoch. You can convert to seconds since the epoch (which we'll need shortly) by dividing by 1,000,000,000, or equivalently by removing the 9 trailing zeros. This gives us 1702323659, or Monday, December 11, 2023 7:40:59 PM.

Next, we need to get the Pyth feed ID for the BTC/USD market. You could look at the spot price config by sending a {"status":{}} query to the market, but in this case it may be easier to search for BTC/USD on the Pyth price feeds ID page. This gives us e62df6c8b4a85fe1a67db44dc12de5db330f7ac66b72dc658afedf0f4a415b43 (which does in fact match up with the spot price config).

To look up historical Pyth prices, we'll need to use the Benchmarks API. The benchmarks docs page provides /v1/updates/prices, described as "Historical Price Updates Proof," which is what we want here. You can construct the appropriate query directly in that web interface, or put it together by hand. Either way, you'll end up with a URL that looks like this:

https://benchmarks.pyth.network/v1/updates/price/1702323659?ids=e62df6c8b4a85fe1a67db44dc12de5db330f7ac66b72dc658afedf0f4a415b43

There's a bunch of data in there, but the end of it is what's important:

{"id":"e62df6c8b4a85fe1a67db44dc12de5db330f7ac66b72dc658afedf0f4a415b43","price":{"conf":"3649130373","expo":-8,"price":"4021887040500","publish_time":1702323659},"ema_price":{"conf":"3182778300","expo":-8,"price":"4099412400000","publish_time":1702323659}}

And as you can see, at that publish time, the price of BTC/USD did in fact reach 40218.87 USD.