Skip to main content

AWS API Gateway

Logs-based Instrumentation​

Tailing Logs with CloudWatch​

You may use CloudWatch Logs to instrument your AWS API Gateway endpoints.

The following script has been provided as an example to help you configure logging for your API Gateway endpoints. It validates your API ID and region, creates the log group with a retention policy, enables access logging on the stage, verifies the configuration, and then tails the log group.

Levo's Log Parser can be configured to parse the logs and send them to Levo.

Click to download: setup-apigw-logging.sh

Or fetch it directly on the host:

wget https://docs.levo.ai/artifacts/log-parser/setup-apigw-logging.sh
chmod +x ./setup-apigw-logging.sh

Edit the API_ID, LOG_GROUP, and REGION variables at the top of the script (or export them as environment variables), then run it:

export API_ID=<your-apigateway-api-id>
export LOG_GROUP=levo/api-gateway-logs
export REGION=<your-region>
./setup-apigw-logging.sh

Syncing Logs to the Log Parser​

Levo's Log Parser (Approach 1, Azure API Gateway format) tails newline-delimited JSON files under logs/azure/ on the host. To get the CloudWatch access logs created above into that format continuously, poll the log group with filter-log-events and append each event, reshaped to the fields the Log Parser expects, to a file such as logs/azure/api-gateway.json.

Do not simply re-run aws logs filter-log-events over a fixed rolling window (e.g. "the last 120 seconds") on a timer — every poll will re-fetch and re-append events already written, and if the poller is ever started twice at once, the two processes' writes to the same file can interleave mid-line and produce garbled JSON the Log Parser can't parse. sync-cloudwatch-logs.sh avoids both by persisting a watermark between polls and using flock to guarantee a single writer.

1. Download the Script​

Click to download: sync-cloudwatch-logs.sh

Or fetch it directly on the host:

wget https://docs.levo.ai/artifacts/log-parser/sync-cloudwatch-logs.sh
chmod +x ./sync-cloudwatch-logs.sh

2. Run the Script​

export LOG_GROUP=levo/api-gateway-logs   # matches the log group created above
export REGION=<your-region>
./sync-cloudwatch-logs.sh

Run it as a long-lived process (e.g. under systemd, or nohup) writing into the directory you mount as /mnt/levo/logs/azure for the Log Parser container — set OUT_DIR to that path. Requires aws, jq, and flock (part of util-linux) on the host.

info

POLL_INTERVAL (default 10 seconds) is also configurable via environment variable. To force the script to re-backfill from scratch, delete its state file ($OUT_DIR/.sync-cloudwatch-logs.state).

Streaming Logs with CloudWatch and Amazon Data Firehose​

You may also use Amazon Data Firehose to stream live access logs to Levo's satellite.

  1. Configure a CloudWatch log group for APIs in API Gateway (using the above example script)
  2. Create a Firehose stream to send incoming events to a publicly accessible satellite endpoint by following these steps:
    • Go to the AWS Data Firehose Service in the AWS Console.
    • Click on Create Firehose stream.
    • In Source, choose Direct PUT from the dropdown, and in the Destination, choose HTTP Endpoint.
    • In HTTP endpoint URL under Destination, put the Satellite's publicly accessible endpoint and click on Create Firehose Stream to complete the process.
  3. Connect the CloudWatch log group to the Firehose stream
    • Open the earlier created log group in AWS CloudWatch.
    • Go to Subscription Filters and click on Create.
    • Choose Create Amazon Data Firehose subscription filter and select the particular Amazon Data Firehose stream.
    • Click on Start Streaming and you can now see your traffic come up in our Application.

Please contact support@levo.ai if you are interested in this setup.

CloudFront Lambda@Edge Instrumentation​

You may configure AWS CloudFront with your API Gateway endpoints as the origin, and use Lambda@Edge functions to intercept and capture traffic.

Please visit the following links for more information:

Was this page helpful?