Introduction

This page documents on how to quickly check/get logs of a particular deployment.

This is usually needed to analyze a previous outage/alert or a current ongoing outage/deployment issue.

Pre-requisite softwares

  • AWS CLI (Tested with 2.13.33)
  • dateutils (Optional, useful for getting ArgoCD logs quickly. Provides datediff utility)

ECS infrastructure

Get all services

❯ aws ecs list-services --cluster bots
{
    "serviceArns": [
        "arn:aws:ecs:eu-west-3:917490793656:service/bots/bots-injmainnet1",
        "arn:aws:ecs:eu-west-3:917490793656:service/bots/indexer-injective-raw-ingester",
        "arn:aws:ecs:eu-west-3:917490793656:service/bots/bots",
        "arn:aws:ecs:eu-west-3:917490793656:service/bots/querier-web-server",
        "arn:aws:ecs:eu-west-3:917490793656:service/bots/companion",
        "arn:aws:ecs:eu-west-3:917490793656:service/bots/indexer-rest-api",
        "arn:aws:ecs:eu-west-3:917490793656:service/bots/indexer-cache-populator",
        "arn:aws:ecs:eu-west-3:917490793656:service/bots/indexer-osmosis-raw-ingester",
        "arn:aws:ecs:eu-west-3:917490793656:service/bots/indexer-osmosis-processor",
        "arn:aws:ecs:eu-west-3:917490793656:service/bots/indexer-injective-processor"
    ]
}

Cloudwatch group name of services

These are the different log groups:

Service nameCloudwatch Group name
bots-osmosis/aws/ecs/bots-osmosis/bots-osmosis
bots-injmainnet1/aws/ecs/bots-injmainnet1/bots-injmainnet1_app
indexer-injective-processor/aws/ecs/indexer-injective-processor/indexer-injective-processor_app
indexer-injective-raw-ingester/aws/ecs/indexer-injective-raw-ingester/indexer-injective-raw-ingester_app
indexer-osmosis-processor/aws/ecs/indexer-osmosis-processor/indexer-osmosis-processor_app
indexer-osmosis-raw-ingester/aws/ecs/indexer-osmosis-raw-ingester/indexer-osmosis-raw-ingester_app
indexer-cache-populator/aws/ecs/indexer-cache-populator/indexer-cache-populator_app
indexer-rest-api/aws/ecs/indexer-rest-api/indexer-rest-api_app
companion/aws/ecs/companion/companion_app
querier-web-server/aws/ecs/querier/querier

You can view current logs like this:

❯ aws logs tail /aws/ecs/bots/bots --follow

You can also check logs from a specific time:

❯ export ECS_SERVICE="/aws/ecs/bots/bots"
❯ aws logs tail $ECS_SERVICE --follow --since "2023-11-01T18:44:00Z"

This is how you can get the proper UTC time from your local timings:

❯ date -u +"%Y-%m-%dT%H:%M:%SZ" -d "2023-11-02 12:14 AM IST"
2023-10-24T02:22:00Z

In general it will be helpful if you go back a couple of minutes from the instant you got alert to find the root cause. The above command can be executed in a single command like this:

aws logs tail $ECS_SERVICE --follow --since $(date -u +"%Y-%m-%dT%H:%M:%SZ" -d "2024-04-05 02:33 AM IST")