Getting started

August 4, 2020 · View on GitHub

← README

Getting started

Requirements
Running ee-outliers
Additionnal Content

Requirements

Docker and Docker Compose to build and run ee-outliers
Access to an Elasticsearch cluster

Running ee-outliers

Using ee-outliers is basically a four-step process:

Define the ee-outliers configuration file.
Define your outlier detection use cases.
Define the docker container and the ee-outliers parameters inside a Compose file docker-compose.yml.
Build an image and run ee-outliers with docker-compose up.

Step 1: Configuring ee-outliers

ee-outliers makes use of a single configuration file containing all required parameters such as connectivity with your Elasticsearch cluster, logging, etc.

A default configuration file with all required configuration sections and parameters, along with an explanation, can be found in defaults/outliers.conf. We recommend starting from this file when running ee-outliers yourself.

A full description of all configuration parameters can be found here.

Step 2: Define the outlier detection use cases

Each detection use case can be defined in a shared or unique .conf file.

We provided 4 examples of use cases available in the use_cases/examples repository. A detailed description of these use case examples, along with information on how you can create your owns can be found here.

Step 3: Define docker container & ee-outliers parameters in the Compose file

The Compose file is located at docker-compose.yml and should look like this:

version: '3'
services:
  outliers:
    build: .
    container_name: your_outliers_container_name
    command: "python3 outliers.py RUN_MODE --config /mappedvolumes/config/outliers.conf --use-cases /use_cases/*.conf"
    environment:
      - es_username:elastic
      - es_password:password
    volumes:
      - ./defaults/outliers.conf:/mappedvolumes/config/outliers.conf
      - ./use_cases/examples:/use_cases
      - /certs/ca.crt:/certs/ca.crt
    network_mode: network_name

It allows you to define the docker container and the ee-outliers parameters for then build and run the ee-outliers image in one single command line. For more information about the Compose file, see the Compose file reference.

The main parameters of the Compose file are as follow:

container_name: Your custom container name.
command: The command line that will execute outliers.py.

The --config and --use-cases argument require respectively the location of the configuration and the use cases file. Note that the --use-cases argument can also contain wildcards, such as "/my/usecase/folder/*.conf".

The RUN_MODE argument should be replaced by one of the 3 running modes:
- interactive: In interactive mode, ee-outliers will run once and finish. This is the ideal run mode to use when testing ee-outliers straight from the command line. If you are testing ee-outliers for the first time, we are recommending to use it.
- daemon: In daemon mode, ee-outliers will continuously run based on a cron schedule defined in the outliers configuration file. The following example from the default configuration file will run ee-outliers at 00:10 each night (standard cron format).
```
[daemon]
schedule=10 0 * * *
```
- tests: In test mode, ee-outliers will run all unit tests and finish, providing feedback on the test results. This mode, which is developer-oriented, is useful for developing and debugging purposes.
environment The environment variables used by outliers to connect to Elasticsearch. If you haven't setup security in your elasticsearch cluster, you don't need to specify these environment variables.
- es_username: username to connect.
- es_password: password to connect.
- verify_certs: whether the Elasticsearch certificate must be validated or not.
- ca_certs: a path to a valid CA to check to server's certificate.
volumes: The mapped volumes so that your configuration and use case files can be found. In this example, the default configuration file in /defaults is mapped to /mappedvolumes/config and the /use_cases/examples is mapped to /use_cases. Moreover, we also map a valid CA certificate /certs/ca.crt used to trust the TLS connection with Elasticsearch.
network_mode: The name of the docker network through which the Elasticsearch cluster is reachable.

Step 4: Build & run ee-outliers with Docker Compose

Thanks to Docker Compose, we can build and run an image of ee-outliers with one single command line:

docker-compose up

To stop and remove the container use:

docker-compose down

Step 4 bis: Build & run ee-outliers with Docker

For convenience, we recommend using Docker Compose but the user can also use Docker and specify the ee-outliers parameters straight from the command line.

To use Docker, after following Step 1 and Step 2, you can enter the following commands:

# Build the image
docker build -t "outliers-dev" .

# Run the image
docker run \
--network=network_name \
-v "$PWD/defaults:/mappedvolumes/config" \
-i  outliers-dev:latest python3 outliers.py interactive \
--config /mappedvolumes/config/outliers.conf \
--use-cases "/my/usecase/folder/*.conf"

Additional content

Building detection use cases →