Install Tekton Pipelines / ECK / Elasticsearch / Kibana / Beats

December 16, 2020 ยท View on GitHub

= Monitoring Tekton Tasks and Pipelines with Elastic Observability :imagesdir: ./assets

In a few minutes, you will have the following components up and running in your local Kubernetes cluster:

image::elastic-o11-tekton-6.png[Tekton Dashboard]

image::elastic-o11-tekton-5.png[Logs UI]

== Prerequisites

  • Kubernetes environment (Docker for Desktop, Minikube, GKE...)

== Helpers

If you don't want to follow all the tutorial step by step, you can use the helpers scripts, make sure to be connected to your dev environment [source,shell]

Install Tekton Pipelines / ECK / Elasticsearch / Kibana / Beats

$ ./helpers/set-up-env.sh

To add some data by executing some Tasks and Pipelines, run the following: [source,shell]

Install some Tekton Tasks and Pipelines official examples

$ ./helpers/add-data.sh 0.18.1 default

The dashboard used in the demo is available at ./helpers/tekton-dashboard.ndjson

Use the script below to clean up your environment [source,shell]

Install some Tekton Tasks and Pipelines official examples

$ ./helpers/clean-all.sh

== Set up Tekton Pipelines and CLI

.Install Tekton Pipelines CRD [source,shell]

make sure you are using the right k8s context

$ kubectl config current-context docker-for-desktop

install Tekton Pipelines

$ kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.18.1/release.yaml

Wait until all Tekton pods are ready

$ kubectl get pods -w -n tekton-pipelines NAME READY STATUS RESTARTS AGE tekton-pipelines-controller-77f74f5bcf-rbj8s 1/1 Running 0 1m tekton-pipelines-webhook-f76c97965-5xkxq 1/1 Running 0 1m

.Set up Tekton CLI

Find the best way to install it based on your local system by loogin at the official documentation:

.Homebrew example [source,shell]

brew install tektoncd-cli

.Check the versions [source,shell]

$ tkn version Client version: 0.14.0 Pipeline version: v0.18.1

== Get your first Tekton Task running!

.Install Tekton Task from the catalog and execute it

# Install the git-clone Task from Tekton Catalog
$ tkn hub get task git-clone --version 0.2 |  kubectl apply -f -
task.tekton.dev/git-clone created
 
# Create a PVC to share data between tasks
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: go-source
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
EOF
persistentvolumeclaim/go-source created

# Execute the Task to clone the repo
$ cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: clone-my-code
spec:
  taskRef:
    name: git-clone
  workspaces:
  - name: output
    persistentVolumeClaim:
      claimName: go-source
  params:
  - name: url
    value: https://github.com/elastic/go-licenser.git
  - name: revision
    value: master
EOF
taskrun.tekton.dev/clone-my-code created

Everything is in place to execute a Task through the deployment of a first TaskRun that is going to invoke the Task:

# Execute a Task to clone the project
$ cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: clone-my-code
spec:
  taskRef:
    name: git-clone
  workspaces:
  - name: output
    persistentVolumeClaim:
      claimName: go-source
  params:
  - name: url
    value: https://github.com/elastic/go-licenser.git
  - name: revision
    value: master
EOF
taskrun.tekton.dev/clone-my-code created

.Check the result and logs

$ tkn taskruns list
NAME                                                  STARTED        DURATION     STATUS
clone-my-code                                         1 minute ago   9 seconds    Succeeded

$ tkn taskruns logs clone-my-code
...
[clone] + /ko-app/git-init -url https://github.com/elastic/go-licenser.git -revision v0.3.1 -refspec  -path /workspace/output/ '-sslVerify=true' '-submodules=true' -depth 1
[clone] {"level":"info","ts":1607989263.4070501,"caller":"git/git.go:165","msg":"Successfully cloned https://github.com/elastic/go-licenser.git @ 857b4969bc2f753ffb9eb3a885d01a59a9f22cdb (grafted, HEAD) in path /workspace/output/"}
[clone] ...

Let's see how to use Elastic Observability to monitor any Tekton Tasks and Pipelines.

== Set up Elastic Observability with Elastic Cloud on Kubernetes (ECK)

.Install ECK [source,shell]

$ kubectl apply -f https://download.elastic.co/downloads/eck/1.3.1/all-in-one.yaml

.Install Elastic Observability [source,shell]

Deploy Elasticsearch and Kibana

$ kubectl apply -n elastic-system -f https://raw.githubusercontent.com/mgreau/tekton-pipelines-elastic-tutorials/master/config/eck/monitoring-es-kb.yaml

Deploy Metricbeat and Filebeat

$ kubectl apply -n elastic-system -f https://raw.githubusercontent.com/mgreau/tekton-pipelines-elastic-tutorials/master/config/eck/monitoring-filebeat-metricbeat.yaml

When the set-up is done, you should have a all the Elastic components up and running in the elasic-system namespace:

[source,shell]

Check Elastic pods

$ kubectl get pods -n elastic-system NAME READY STATUS RESTARTS AGE elastic-operator-0 1/1 Running 1 5m elasticsearch-monitoring-es-default-0 1/1 Running 1 5m filebeat-beat-filebeat-b6vlt 1/1 Running 7 5m kibana-monitoring-kb-599698987-7cjqq 1/1 Running 1 5m metricbeat-beat-metricbeat-fsz5p 1/1 Running 7 5m

.Get the elastic user password needed to access the UI [source,shell]

echoecho(kubectl get secret -n elastic-system elasticsearch-monitoring-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode) XXXXXXXXXXXXXX

.Make Elastic Kibana available [source,shell]

$ kubectl port-forward -n elastic-system svc/kibana-monitoring-kb-http 5601

Then you can access the following URL and use the credentials from above:

Note: the intent of this tutorial is to provide a development environment. To install a valide certificate, please refer to the official Elastic documentation.

image::elastic-o11-tekton-1.png[Tekton Dashboard]

== Monitor Tasks and Pipelines with Elastic Observability

Run the tests Task and check the logs via the Logs UI

.Install Tekton Tasks from the Catalog to build and test a golang project [source,shell]

Install the generic golang-test Task from the catalog

$ tkn hub get task golang-test --version 0.1 | kubectl apply -f - task.tekton.dev/golang-test created

Execute a Task to run the tests

$ cat <<EOF | kubectl apply -f - apiVersion: tekton.dev/v1beta1 kind: TaskRun metadata: name: test-my-code spec: taskRef: name: golang-test workspaces:

  • name: source persistentVolumeClaim: claimName: go-source params:
  • name: package value: github.com/elastic/go-licenser
  • name: packages value: ./...
  • name: flags value: -timeout 10s -p 4 -race -cover EOF taskrun.tekton.dev/test-my-code created --

Go back to the Metrics UI, there is now a Pod showing up named test-my-code in the default namespace. By clicking on this Pod, you can access the logs, you should see the following output:

image::elastic-o11-tekton-2.png[Tekton Dashboard]

.Build the project [source,shell]

build

$ cat <<EOF | kubectl apply -f - apiVersion: tekton.dev/v1beta1 kind: TaskRun metadata: name: build-my-code spec: taskRef: name: golang-build workspaces:

  • name: source persistentVolumeClaim: claimName: go-source params:
  • name: package value: github.com/elastic/go-licenser
  • name: packages value: .
  • name: flags value: -o bin/go-licenser -ldflags="-X main.version=master-dev" EOF taskrun.tekton.dev/clone-my-code created --

image::elastic-o11-tekton-3.png[Tekton Dashboard] image::elastic-o11-tekton-4.png[Tekton Dashboard]

You can then filter the logs by the Task name, TaskRun name, and so on

image::elastic-o11-tekton-5.png[Tekton Dashboard]

== Execute several Tasks and Pipelines and Monitor through a Tekton dashboard

Tekton Pipelines expose some metrics by default in Prometheus format. The metricbeat configuration provided earlier in this post has been initialized with the Prometheus module to automatically gather all these data:

.Metricbeat Prometheus module [source,yaml]

 - module: prometheus
    period: 10s
    hosts:
    - http://tekton-pipelines-controller.tekton-pipelines:9090
    metrics_path: /metrics

--

Therefore, your cluster is ready for having dashboards use these data. I have initialized one Tekton Dashboard that you can import using the Kibana Import feature. This dashboard is available at ./helpers/tekton-dashboard.ndjson

.Execute multiple Tasks and Pipelines [source,shell]

Run Tekton examples from the 0.18.1 git tag in the default namespace

$ ./helpers/add-data.sh 0.18.1 default

Now, open the Tekton Dashboard and watch the number of tasks running grow. You can also see which Tasks take the most time to execute.

image::elastic-o11-tekton-6.png[Tekton Dashboard] image::elastic-o11-tekton-6bis.png[Tekton Dashboard]