GCP Logs Query Skill

When to Use

• Investigating relayer, validator, or scraper behavior

• Debugging message processing issues

• Analyzing operational incidents

• Looking for specific errors or patterns in agent logs

Prerequisites

• gcloud CLI installed and authenticated

• Access to abacus-labs-dev GCP project

Base Query Templates

Relayer (Omniscient)

gcloud logging read 'resource.type="k8s_container" AND resource.labels.project_id="abacus-labs-dev" AND resource.labels.location="us-east1-c" AND resource.labels.cluster_name="hyperlane-mainnet" AND resource.labels.namespace_name="mainnet3" AND labels.k8s-pod/app_kubernetes_io/component="relayer" AND labels.k8s-pod/app_kubernetes_io/instance="omniscient-relayer" AND labels.k8s-pod/app_kubernetes_io/name="hyperlane-agent"' --project=abacus-labs-dev --limit=50 --format=json --freshness=1d

Validator

gcloud logging read 'resource.type="k8s_container" AND resource.labels.project_id="abacus-labs-dev" AND resource.labels.location="us-east1-c" AND resource.labels.cluster_name="hyperlane-mainnet" AND resource.labels.namespace_name="mainnet3" AND labels.k8s-pod/app_kubernetes_io/component="validator" AND labels.k8s-pod/app_kubernetes_io/name="hyperlane-agent"' --project=abacus-labs-dev --limit=50 --format=json --freshness=1d

Scraper

gcloud logging read 'resource.type="k8s_container" AND resource.labels.project_id="abacus-labs-dev" AND resource.labels.location="us-east1-c" AND resource.labels.cluster_name="hyperlane-mainnet" AND resource.labels.namespace_name="mainnet3" AND labels.k8s-pod/app_kubernetes_io/component="scraper3" AND labels.k8s-pod/app_kubernetes_io/instance="omniscient-scraper" AND labels.k8s-pod/app_kubernetes_io/name="hyperlane-agent"' --project=abacus-labs-dev --limit=50 --format=json --freshness=1d

Noise Filtering

Add these filters to reduce noisy log lines that consume context without providing value:

-jsonPayload.fields.message="Found log(s) in index range" -jsonPayload.fields.message="Dispatching get_public_key" NOT "Instantiated AWS signer" -jsonPayload.fields.message="Ingesting leaf" -jsonPayload.fields.message="Message already marked as processed in DB" -jsonPayload.fields.message="Message destined for self, skipping" -jsonPayload.fields.message="Message has already been delivered, marking as submitted." -jsonPayload.fields.message="Popped OpQueue operations" -jsonPayload.fields.message="Validator returned latest index" -jsonPayload.fields.message="Found signed checkpoint" -jsonPayload.fields.return="Ok(None)" -jsonPayload.fields.message="Fast forwarded current sequence" -jsonPayload.fields.message="Cursor can't make progress, sleeping" -jsonPayload.fields.message="fallback_request" -jsonPayload.fields.message="No message found in DB for leaf index" -jsonPayload.fields.message="Processor working on message" -jsonPayload.fields.message="Message destined for unknown domain, skipping"

Progressive Query Strategy (Token Efficiency)

Step 1: Fetch Message Field Only First

To minimize context consumption, first fetch only the message field:

gcloud logging read '[BASE_QUERY] AND "[search_term]"' --project=abacus-labs-dev --limit=30 --format='json(jsonPayload.fields.message,timestamp)' --freshness=1d

This gives you a quick overview without the full log payload.

Step 2: Get Full Context for Specific Entries

Once you identify interesting log entries, fetch full details:

gcloud logging read '[BASE_QUERY] AND "[specific_identifier]"' --project=abacus-labs-dev --limit=20 --format=json --freshness=1d

Step 3: Extract Specific Fields

When you need specific details, use jq or grep to extract:

gcloud logging read '[QUERY]' --format=json | jq '.[].jsonPayload.fields.error'

Common Query Patterns

Search by Message ID

gcloud logging read '[BASE_QUERY] AND "0x[MESSAGE_ID]"' --project=abacus-labs-dev --limit=50 --format=json --freshness=1d

Search for Errors/Warnings

gcloud logging read '[BASE_QUERY] AND severity>="WARNING"' --project=abacus-labs-dev --limit=50 --format=json --freshness=1d

Search by Chain/Domain

gcloud logging read '[BASE_QUERY] AND jsonPayload.spans.domain:"[chain_name]"' --project=abacus-labs-dev --limit=50 --format=json --freshness=1d

Search for Stuck Messages (High Retry Count)

gcloud logging read '[BASE_QUERY] AND jsonPayload.fields.num_retries>=5' --project=abacus-labs-dev --limit=30 --format=json --freshness=1d

Search for Gas Estimation Errors

gcloud logging read '[BASE_QUERY] AND "eth_estimateGas"' --project=abacus-labs-dev --limit=30 --format=json --freshness=1d

Search by App Context

gcloud logging read '[BASE_QUERY] AND jsonPayload.fields.app_context:"[APP_CONTEXT]"' --project=abacus-labs-dev --limit=30 --format=json --freshness=1d

Time Range Options

• --freshness=1h

• Last hour

• --freshness=1d

• Last day

• --freshness=7d

• Last week

• Or use explicit timestamps in filter: timestamp>="2026-01-27T00:00:00Z"

Output Format Options

• --format=json

• Full JSON (verbose, high context)

• --format='json(jsonPayload.fields.message,timestamp)'

• Specific fields only (efficient)

• --format='value(jsonPayload.fields.message)'

• Just values, no structure

Key Log Fields to Focus On

Field Description

jsonPayload.fields.message

Main log message

jsonPayload.fields.error

Error details

jsonPayload.spans[].domain

Chain involved

jsonPayload.fields.num_retries

Retry count

jsonPayload.fields.operations

Pending message details

jsonPayload.span.id

Message ID in span context

Environment Variations

Environment Namespace Cluster

mainnet3 mainnet3

hyperlane-mainnet

testnet4 testnet4

hyperlane-mainnet

Tips

• Always start specific - Search for exact message IDs or error patterns first

• Use noise filters - The base logs are very noisy; always filter

• Limit results - Use --limit to avoid overwhelming context

• Progressive detail - Start with message field only, expand as needed

• Time bound queries - Use --freshness or timestamp filters

• Pipe to grep/jq - Post-process large results locally