Skip to main content

Install Log Parsing Sensors

Access Logs Based Instrumentation​

Prerequisites​

  • The Satellite has been successfully installed.
  • You have noted the Satellite's hostname:port or ip-address:port.
  • The Satellite is reachable (via HTTP/HTTPS) from the machine where you will install the Log Parser.

Installation Approaches​

  • Approach 1: For the following log formats:

    • Nginx
    • Apache
    • Azure API Gateway

  • Approach 2: For the following log formats:

    • CEF

Note: For instrumenting Imperva WAF - Incapsula logs stored on an Amazon S3 Bucket, please proceed with Approach 2.

Approach 1​

Installation Instructions​

Install on Linux Host via Docker​

Prerequisites​

  • Docker Engine version 18.03.0 and above.
  • Application Name: Auto-discovered API endpoints and their OpenAPI specifications are displayed in the API Catalog, grouped under an Application Name. This helps segregate and group API endpoints from different environments.
    Choose a descriptive name (e.g., my-test-app) for the next step.

Installation​

# Replace '<SATELLITE_URL>' with the values you noted down from the Satellite install
#
# Specify below the 'APP_NAME'. Do not quote the 'APP_NAME'.
# Environment Name is optional. If not specified, it defaults to 'staging'
# 
docker run --rm -d --name=log-parser \
  -v ./logs:/mnt/levo/logs \
  -e LEVO_SATELLITE_URL=<LEVO_SATELLITE_URL> \
  -e LEVOAI_ORG_ID=<LEVOAI_ORG_ID> \
  -e APP_NAME=<APP_NAME> \
  -e ENV_NAME=<ENV_NAME> \
  levoai/log-parser

Notes:

  • The default Satellite address for Docker-based Log Parser installations is https://satellite.levo.ai.
  • For the Levo-hosted Satellite, you must specify an Organization ID (LEVOAI_ORG_ID).
  • If you are self-hosting the Satellite, specify its address to direct the Log Parser's data there.

2. Verify Connectivity with Satellite​

Execute the following command to check for connectivity health:

# Please specify the actual container name for log-parser below
docker logs log-parser | grep "starting fluentd"

If connectivity is healthy, you should see output similar to the following:

2024-02-22 01:27:06 +0000 [info]: starting fluentd-1.16.3 pid=7 ruby="3.2.2"
2024-02-22 01:27:06 +0000 [info]: #0 starting fluentd worker pid=16 ppid=7 worker=0
2024-02-22 01:27:06.831947051 +0000 fluent.info: {"pid":16,"ppid":7,"worker":0,"message":"starting fluentd worker pid=16 ppid=7 worker=0"}

Proceed to the next step if there are no errors.

Approach 2​

Install on Linux host via Docker​

1. Prerequisites​

  • Docker Engine version 18.03.0 and above.

2. Installation steps​

  • Loading...
    the Docker Compose and configuration files.
  • Run the following command to install the Log Parser containers (Filebeat and Logstash):
docker compose up -d

3. Verify the installation​

  • Verify the installation by checking the container logs:
docker ps -f name=levoai
  • View the logs of the containers by running the following commands:
docker logs -f levoai-logstash
docker logs -f levoai-filebeat

4. Uninstallation steps​

  • Clean up the containers by running the following command:
docker compose down
docker volume rm log-parser_filebeat-data log-parser_logstash-queue && rm -rf filebeat/data/ && rm -rf logstash/data/

Install on Kubernetes via Helm​

1. Prerequisites​

  • Kubernetes version >= v1.18.0
  • Helm v3 installed and working.
  • The Kubernetes cluster API endpoint is reachable from the machine running Helm.
  • kubectl access to the cluster with cluster-admin permissions.
  • At least 4 CPUs
  • At least 8 GB RAM

2. Add Levo Helm Repository​

helm repo add levoai https://charts.levo.ai && helm repo update levoai

3. Installation steps​

  • Create a secret for aws-credentials with your AWS credentials that have access to the S3 bucket:
kubectl apply -f aws-credentials.yaml
 # aws-credentials.yaml
 apiVersion: v1
 kind: Secret
 metadata:
   name: aws-credentials
   labels:
     app: levoai-filebeat
     app.kubernetes.io/name: aws-credentials-secret
 type: Opaque
 data:
   aws-bucket-arn: <Your AWS Bucket ARN like "arn:aws:s3:::<bucket-name>">
   aws-access-key-id: <Your Access Key ID>
   aws-secret-access-key: <Your Secret Access Key>
   aws-default-region: <Your Region like "us-west-2">
  • Run the following command to install the Log Parser containers in the levoai namespace:
helm upgrade --install -n levoai --create-namespace levoai-log-parser levoai/levoai-log-parser --set global.levoaiOrgId=<your-org-id>
info

By default, this will send all logs to the Satellite running in the same namespace. If you want to send logs to a different Satellite, you can specify the Satellite URL using --set global.levoaiSatelliteUrl=<your-satellite-url>.
For the Levo-hosted Satellite, use https://collector.levo.ai.

4. Verify the installation​

Check the status of the pods by running the following command:

kubectl get pods -n levoai

If the levoai-log-parser Helm chart is installed correctly, you should see output similar to the following:

NAME                               READY   STATUS    RESTARTS   AGE
levoai-filebeat-6bbf84d957-c44kh   1/1     Running   0          104s
levoai-logstash-77775d78c-q7cgm    1/1     Running   0          104s

Check the status of the services by running the following command:

kubectl get services -n levoai

If the levoai-logstash service is healthy, you should see output similar to the following:

NAME                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                          AGE
levoai-logstash            ClusterIP   10.0.81.244    <none>        5044/TCP                                         120s

5. Uninstallation steps​

  • Run the following command to uninstall the services:
helm uninstall levoai-log-parser
kubectl delete secret aws-credentials -n levoai

Need Help?​

For further assistance, please reach out to support@levo.ai.