Skip to main content

Install Log Parsing Sensors

Access Logs Based Instrumentation​

i. Pre-requisites​

  • Satellite has been successfully installed.
  • You have noted down the Satellite's hostname:port or ip-address:port information.
  • The Satellite is reachable (via HTTP/s) from the location where you are going to install the log-parser.

Approaches​

  • For below log formats proceed with Approach 1.

    • Nginx
    • Apache
    • Azure API Gateway

  • For below log formats proceed with Approach 2.

    • CEF

Note: For instrumenting Imperva WAF - Incapsula logs that are stored on Amazon S3 Bucket, please proceed with Approach 2.

Approach 1​

Follow instructions for your platform​

Install on Linux host via Docker​

Prerequisites​

  • Docker Engine version 18.03.0 and above
  • Pick an Application Name Auto discovered API endpoints and their OpenAPI specifications are shown in the API Catalog, grouped under an application name. The application name helps segregate and group API endpoints from different environments, similar to how file folders work in an operating system.
    Pick a descriptive name which will be used in the subsequent step below. For example: `my-test-app`.

Installation​

# Replace '<SATELLITE_URL>' with the values you noted down from the Satellite install
#
# Specify below the 'APP_NAME'. Do not quote the 'APP_NAME'.
# Environment Name is optional. If not specified, it defaults to 'staging'
# 
docker run --rm -d --name=log-parser \
  -v ./logs:/mnt/levo/logs \
  -e LEVO_SATELLITE_URL=<LEVO_SATELLITE_URL> \
  -e LEVOAI_ORG_ID=<LEVOAI_ORG_ID> \
  -e APP_NAME=<APP_NAME> \
  -e ENV_NAME=<ENV_NAME> \
  levoai/log-parser

NOTE:

  • The default address for the satellite in Docker-based Log Parser installations is https://satellite.levo.ai.
  • In case of levo hosted satellite, it is necessary that you must also specify an organization ID (LEVOAI_ORG_ID).
  • If you wish, you may also host the Satellite yourself and specify the address of the satellite to direct the Log Parser's data to it.

2. Verify connectivity with Satellite​

Execute the following command to check for connectivity health:

# Please specify the actual container name for log-parser below
docker logs log-parser | grep "starting fluentd"

If connectivity is healthy, you should see output similar to below.

2024-02-22 01:27:06 +0000 [info]: starting fluentd-1.16.3 pid=7 ruby="3.2.2"
2024-02-22 01:27:06 +0000 [info]: #0 starting fluentd worker pid=16 ppid=7 worker=0
2024-02-22 01:27:06.831947051 +0000 fluent.info: {"pid":16,"ppid":7,"worker":0,"message":"starting fluentd worker pid=16 ppid=7 worker=0"}

Please proceed to the next step, if there are no errors.

Approach 2​

Install on Linux host via Docker​

1. Prerequisites​

  • Docker Engine version 18.03.0 and above.

2. Installation steps​

  • Loading...
    the docker-compose and configuration files.
  • Run below command to install log-parser containers filebeat and logstash.
docker compose up -d

3. Verify the installation​

  • Verify the installation by checking the logs of the containers.
docker ps -f name=levoai
  • View the logs of the container by running below command.
docker logs -f levoai-logstash
docker logs -f levoai-filebeat

4. Uninstallation steps​

  • Clean up the containers by running below command.
docker compose down
docker volume rm log-parser_filebeat-data log-parser_logstash-queue && rm -rf filebeat/data/ && rm -rf logstash/data/

Install on Kubernetes via Helm​

1. Prerequisites​

  • Kubernetes version >= v1.18.0
  • Helm v3 installed and working.
  • The Kubernetes cluster API endpoint should be reachable from the machine you are running Helm.
  • kubectl access to the cluster, with cluster-admin permissions.
  • At least 4 CPUs
  • At least 8 GB RAM

2. Add levoai Helm repo​

helm repo add levoai https://charts.levo.ai && helm repo update levoai

3. Installation steps​

  • Create secret for aws-credentials with your AWS credentials which has access to the S3 bucket.
kubectl apply -f aws-credentials.yaml
 # aws-credentials.yaml
 apiVersion: v1
 kind: Secret
 metadata:
   name: aws-credentials
   labels:
     app: levoai-filebeat
     app.kubernetes.io/name: aws-credentials-secret
 type: Opaque
 data:
   aws-bucket-arn: <Your AWS Bucket ARN like "arn:aws:s3:::<bucket-name>">
   aws-access-key-id: <Your Access Key ID>
   aws-secret-access-key: <Your Secret Access Key>
   aws-default-region: <Your Region like "us-west-2">
  • Run the below command to install the log-parser containers in levoai namespace.
helm upgrade --install -n levoai --create-namespace levoai-log-parser levoai/levoai-log-parser --set global.levoaiOrgId=<your-org-id>
info

By default this will send all the logs to satellite running in the same namespace. If you want to send logs to a different satellite, you can specify the satellite URL using --set global.levoaiSatelliteUrl=<your-satellite-url>.
In case of levo hosted satellite, use satellite url as https://collector.levo.ai.

4. Verify the installation​

Check the status of the pods by running below command.

kubectl get pods -n levoai

If the levoai-log-parser helm is healthy, you should see output similar to below.

NAME                               READY   STATUS    RESTARTS   AGE
levoai-filebeat-6bbf84d957-c44kh   1/1     Running   0          104s
levoai-logstash-77775d78c-q7cgm    1/1     Running   0          104s

Check the status of the services by running below command.

kubectl get services -n levoai

If the levoai-logstash service is healthy, you should see output similar to below.

NAME                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                          AGE
levoai-logstash            ClusterIP   10.0.81.244    <none>        5044/TCP                                         120s

5. Uninstallation steps​

  • Run the below command to uninstall the services.
helm uninstall levoai-log-parser
kubectl delete secret aws-credentials -n levoai

Need Help?​

For further assistance, please reach out to Levo.ai Support.