Distributed Tracing in Async Manner

Prerequisite :- Rabbit MQ, Zipkin-Sleuth Configuration, ELK installed.

Recently done a POC on distributed tracing in asynchronous manner where traces are published on rabbit MQ. Logstash is configured in a way so that it can listen on queue with a heartbeart of 30 second.Whenever any client request comes through the microservice then zipkin-sleuth dependency push all traces in configured rabbit mq or any other JMS in async manner. There are different ways to push traces i.e — HTTP ( Synchronous Manner) and Asynchronous manner(rabbit/JMS), this is a configuration avaialble through spring.zipkin.sender.type=rabbit. This configuration will get available in microservices when zipkin/sleuth dependency found. Since there are a lot of tool avaialble to get traces and visualize in a proper format to get better result from the available traces. Here ELK is used as a tool to get traces from queue. To achieve this logstash is configured so that it can pull traces from queue within a heart beat interval. Once logstash get the traces the ElasticSearch start accepting traces since internally all traces persisted in NoSql manner in elastic. Now Kibana comes in picture to create index pattern based on the available data from the elastic. Based on the requirement we can take fields and create index to get better result. This is a one part where traces comes from microservice then it goes to MQ. Logstash will pull then elastic will start consuming so that kibana dashboard can prepare. Below is the logstash configuration required to start listening on defined MQ.

input { 
 rabbitmq {
 host => “localhost”
 queue => “test”
 heartbeat => 30
 durable => true
 password => “guest”
 user => “guest”
 exchange_type => “direct”
 }
 }
output {
 elasticsearch {
 hosts => “localhost:9200”
 index => “rabbitmq-%{+yyyy-MM-dd}”
 }
 stdout {}
}

In the above configuration queue is mandatory, so start rabbit mq and create a exchange and queue with direct type.

Now Zipkin comes in picture where we can check traces of different microservices and other metadata of individual microservice. To acheive this we need to configure on individual microservices so that traceid/span id can be replicated in the entire call of microservices. At the time of zipkin startup we need to set enviroment variable in a below manner so that zipkin start listening on rabbit MQ and start pulling of traces of individual microservice.

SET zipkin.collector.rabbitmq.uri=amqp://guest:guest@localhost:5672

Below is the individual microservice configuration for zipkin-sleuth traceing.

spring.zipkin.service.name=account-dept
spring.zipkin.enabled=true
spring.sleuth.sampler.probability=1.0
#spring.zipkin.sender.type=web
spring.zipkin.base-url=http://localhost:9411
#spring.zipkin.discovery-client-enabled=true
#spring.sleuth.trace-id128=true
spring.sleuth.enabled=true
spring.sleuth.http.enabled=true
spring.sleuth.web.enabled=true
#Rabbit MQ
spring.rabbitmq.host=localhost
spring.rabbitmq.port=5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
spring.zipkin.sender.type=rabbit
spring.zipkin.rabbitmq.queue=test

Below are the URL of the related projects:-

GitHub - kablu/account-dept: This is a root project from where call get start.

kablu/credit-card

https://github.com/kablu/card-dept.git

========== () -> Happy Learning ============