Most common errors

This page is meant to be an easier-to-digest list of the most common actions an on-call engineer should take. If nothing in this page helps, see alert handling steps and on-call incident guide.

The two most common things to occur are node issues and Pyth issues.

Node issues

Node issues usually pop up as something like the following:

  • Primary node shows as unhealthy and we get an error from a fallback node
  • Messages like "block lag detected"
  • Query timeouts

In these cases, you should first ping the node provider. For Osmosis and Injective, this is Kingnodes. For Sei, you should reach out to the Sei team directly.

See Slack contact list for contacts.

Pyth issues

The most common Pyth issues will display as:

  • Errors from users not being able to see the tradingview chart. This is the lowest priority issue related to Pyth. You should reach out to the Pyth team about the benchmarks API, but there is no risk to the protocol itself.
  • The Hermes endpoint going down. We pay for a managed Hermes service from Triton and have a shared chat with them on Telegram. A short-term workaround can be switching our querier and bots services to use an alternate Hermes endpoint. Ask for help in #engineering to do that.

Price too old errors

This is getting its own section because it can happen for two completely unrelated reasons. First, let's explain what it is.

Every price update from Pyth includes a publish time. Whenever we need to perform an operation that requires an up-to-date price, such as an open position, the workflow is basically:

  1. User submits a transaction to the chain to open the position. This creates a deferred execution item.
  2. The bots notice that a deferred execution item is waiting, and performs a price-update-and-crank.
  3. Price update occurs by querying the Pyth Hermes endpoint (over HTTPS, it occurs off-chain), getting a price attestation (cryptographically signed proof of a price), and creates an on-chain message to update the Pyth oracle.
  4. The crank is an on-chain message that causes the contracts to query the Pyth oracle and get the most recent price point that has been submitted to it.

To avoid various price manipulation attack vectors, the market contract will only accept relatively recent price updates (42 seconds on Osmosis, 21 seconds on Injective and Sei, at least at time of writing). price_too_old occurs when the contract sees a price point in Pyth which is too old. This can happen for two different reasons:

  1. Pyth is broken in some way. This could be missing data for a specific feed, a problem with the entire Pyth network, or a problem with the specific Pyth Hermes endpoint we're using. This will look something like this:

    Price: CNH_USDC: price is too old. Check the price feed and try manual cranking in the frontend. Feed info: Pyth feed 0xeef52e09c878ad41f6a81803e3640fe04dceea727de894edd4ea117e2e332e66. Publish time: 2024-03-08 22:00:02 UTC. Checked at: 2024-03-08 22:01:24.347272368 UTC. Age: 82s. Tolerance: 42s.

    Note that the "price is too old" message is generated by the bots itself.

  2. There's a blockchain node issue causing sync issues, and when we ask the node to simulate a crank, it says "wait, I can't do that, the last price update happened too long ago." This looks something like the following:

    Unable to turn crank for market INJ_USD (inj1rw503v6kj56wxdgt83mvhmlmcqtrthkyelhjar)

Caused by:
    On connection to https://inj-priv-grpc.kingnodes.com, while performing:
    simulating transaction: Memo: Price bot discovered crank work available
    Message 0: inj14r7jtujrwmyd6nxqd8mcn4rmcx2n0t2vgf0zwm executing contract inj1rw503v6kj56wxdgt83mvhmlmcqtrthkyelhjar with message: {"crank":{"execs":null,"rewards":"inj1pgahlsqxvwncnqun46ajh27hlwz9l6gsva0qvr"}}
    Status { code: Unknown, message: "failed to execute message; message index: 0: {\n  \"id\": \"price_too_old\",\n  \"domain\": \"pyth\",\n  \"description\": \"Current price is not available. Price id: 0x7a5bc1d2b56ad029048cd63964b3ad2776eadf812edc1a43a31406cb54bff592, Current block time: 1709729249, price publish time: 1709729220, diff: 29, age_tolerance: 21\",\n  \"data\": null\n}: execute wasm contract failed [!injective!labs/wasmd@v0.45.0-inj/x/wasm/keeper/keeper.go:401] With gas wanted: '50000000' and gas used: '195226' ", details: b"\x08\x02\x12\xdc\x03failed to execute message; message index: 0: {\n  \"id\": \"price_too_old\",\n  \"domain\": \"pyth\",\n  \"description\": \"Current price is not available. Price id: 0x7a5bc1d2b56ad029048cd63964b3ad2776eadf812edc1a43a31406cb54bff592, Current block time: 1709729249, price publish time: 1709729220, diff: 29, age_tolerance: 21\",\n  \"data\": null\n}: execute wasm contract failed [!injective!labs/wasmd@v0.45.0-inj/x/wasm/keeper/keeper.go:401] With gas wanted: '50000000' and gas used: '195226' \x1al\n(type.googleapis.com/google.rpc.ErrorInfo\x12@\n\x1cexecute wasm contract failed\x1a\r\n\x08ABCICode\x12\x015\x1a\x11\n\tCodespace\x12\x04wasm", metadata: MetadataMap { headers: {"date": "Wed, 06 Mar 2024 12:47:31 GMT", "content-type": "application/grpc", "x-cosmos-block-height": "62500811", "cf-cache-status": "DYNAMIC", "report-to": "{\"endpoints\":[{\"url\":\"https:\/\/a.nel.cloudflare.com\/report\/v3?s=kG38%2FNgsy9pd7uj0ZTmxEas6kVOzGkoT6Rv6BvvJv5O282FWQkuM9erA%2B95OfpbfIo1ie8qN53JZYUGfJYXkZSZcYgesQD7KnHmf0meSBrMF2dFPTC82YW%2F1Pd2MkMbicUznGqd4noc7p7NZ3w%3D%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}", "nel": "{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}", "server": "cloudflare", "cf-ray": "860274705ef12a34-CDG", "alt-svc": "h3=\":443\"; ma=86400"} }, source: None }
    Height set to: None
    Health report for https://inj-priv-grpc.kingnodes.com. Fallback: false. Healthy: true. Last error: 2024-03-06 12:06:58.232977879 UTC (2433.715511315s): Transport error with gRPC endpoint. status: Unknown, message: "transport error", details: [], metadata: MetadataMap { headers: {} } during action smart query contract inj1rw503v6kj56wxdgt83mvhmlmcqtrthkyelhjar with message: {"oracle_price":{"validate_age":false}}
    Health report for http://injective-grpc.polkachu.com:14390. Fallback: true. Healthy: true. No errors

    Note that the error occurred after broadcasting the transaction, due to a delay between constructing the transaction and it being included in a block.

The first case happens most frequently with the non-24/7 markets (things like gold, silver, and forex). This happens because price updates sometimes stop coming in before the official market close hours.

Generally speaking, if the error occurs as part of the "crank run" actions, it's a node sync issue, and you should contact the node provider. If it happens as part of the price bot, it's likely a Pyth issue. If it is a Pyth issue, it's basically the most serious protocol risk possible on this page since the entire protocol will be stalled, and it could be part of a price manipulation attack. You should escalate to the Pyth team on Slack ASAP.