Java Asynchronous Programming

Introduction

When we start Java or any other programming the focus is mainly towards implementing complex logics. We tend to find efficient but sequential ways of writing code. Sequential programming is easy to present your logical way of handling problems. In early days hardware support was lagging, so sequential way of handling problems was quite okay.

The things have changed rapidly and people thought of doing things in concurrent manner. Hardware also improved with multi-core with multi-threaded processors. Then programmers were asked to compose more efficient code. The main circumstance was the support from the programming language. When we consider Java, the support for concurrent programming came to picture step-by-step.

lets_build_our_future_together_807295

Threads

Threading is the most common approach that is in use for concurrent implementations. You can use an implement of Runnable interface or extension of Thread class to implement your scenario which need to perform independently.

Later Java version introduced Thread Executors to handle pool of threads. With thread pools you can handover the task to Executor, and Executor manage how many threads it needs according to the load.

Sample code:


import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class SampleThreading {

public static void main(String[] args) {

ExecutorService executorService = Executors.newFixedThreadPool(2);

Runnable runnable = new Runnable() {
@Override
public void run() {
System.out.println("Hello World from another thread!");
}
};

executorService.submit(runnable);

// Do something else at main thread

executorService.shutdown();
}
}

Callable & Future

A major drawback of runnable implementation is it can’t not return a value. That requirement can be addressed through using Callable implementations and Futures. The implementation of callable interface allows you to specify a return data type at call() method. While submitting a callback to an executor, it returns a Future. The returned Future can be used to examine the status of callable and get the result of callable.

Sample code:


import java.util.concurrent.*;

public class SampleCallable {

public static void main(String[] args) throws ExecutionException, InterruptedException {

ExecutorService executorService = Executors.newFixedThreadPool(2);

Callable callable = new Callable() {

@Override
public String call() throws Exception {
String result = "Hello " + "World " + "from " + "callable.";

// Do some work

return result;
}
};

Future stringFuture = executorService.submit(callable);

// Do something else at main thread

String resultingString = stringFuture.get();

System.out.println(resultingString);

executorService.shutdown();

}
}

CompletableFuture

CompletableFuture was introduced with recent Java release (Java 1.8). This new type is to fulfil missing aspects of callable and future implementation. Let’s see what we can’t do using callable and futures.

  1. Stopping future manually
  2. Performing another task on the result of Future without blocking
  3. Chaining multiple futures
  4. Handling exceptions

ComplatableFuture has runAsync() which returns CompletableFuture<Void> and supplyAsync() which returns CompletableFuture<Object> which can be used to retrieve the result.

Sample code:


import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.function.Supplier;

public class SampleCompletableFutures {

public static void main(String[] args) throws ExecutionException, InterruptedException {

CompletableFuture future = CompletableFuture.supplyAsync(new Supplier() {
@Override
public String get() {
return "Hello World from Future!!!";
}
});

// Do something else at main thread

String result = future.get();

System.out.println(result);
}
}

Moreover you can use thenApply() to chain another task which returns a result (CompletableFuture<Object>). If no result required, you may use thenAccept() or thenRun() methods.

Sample code:


import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.function.Function;
import java.util.function.Supplier;

public class SampleCompletableFutures2 {

public static void main(String[] args) throws ExecutionException, InterruptedException {

CompletableFuture future = CompletableFuture.supplyAsync(new Supplier() {
@Override
public String get() {
return "Hello World from Future!!!";
}
});

// Do something else at main thread

CompletableFuture future2 = future.thenApply(new Function() {
@Override
public String apply(String s) {
return s + " And more greetings from new Future!!!";
}
});

// Do something more at main thread

String result = future2.get();

System.out.println(result);
}
}

You can also combine multiple CompletableFutures and proceed using allOf() or anyOf() methods.

Conclusion

In this post I wanted to give you a simple introduction on how to use Java in asynchronous programming. Hope this will be useful for your coding.

References

  1. Executor Services Java docs: https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorService.html
  2. Java Callable and Futures: https://www.callicoder.com/java-callable-and-future-tutorial/
  3. CompletableFuture Java docs: https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html
  4. CompletableFuture tutorial: https://www.callicoder.com/java-8-completablefuture-tutorial/
Advertisements

What is SMPP Protocol

Introduction

What is SMPP protocol? The simplest answer that you can give is “Short Message Peer to Peer” protocol. This protocol is use to define how you can communicate with Message Centres such as Short Message Service Centre (SMSC), GSM Unstructured Supplementary Services Data (USSD) Server etc.

The client which communicate with a Message Centre is known as External Short Message Entity (ESME). The ESME needs to comply the same protocol version as SMSC does to communicate. The current established SMPP protocol version is version 3.4 [1].

smpp-15-728

Binding Types

To start communicating with a Messaging Centre, the ESME needs to initiate a session by sending a bind request. In SMPP protocol, there are 3 binding types defined.

  1. Transmitter: Messages sent from the ESME to the SMSC
  2. Receiver: Messages sent from the SMSC to the ESME
  3. Transceiver: Messages sent from the SMSC to the ESME and vise-versa

If ESME wants to transfer messages both sides either it has to use Transmitter + Receiver binds or a Transceiver bind.

Exchanging Data

The elements on SMPP protocol are request and response Protocol Data Units (PDUs). Data exchange in SMPP is defined using types of PDUs over an underlying TCP/IP or X.25 network connection. Here I will discuss on some important PDUs.

bind_

Bind PDU is use to register an ESME to a Message Centre. Following binding types; bind_transmitter, bind_receiver and bind_transceiver PDUs are use to initiate Transmitter, Receiver and Transceiver bindings respectively. As the response, Messaging Centre sends bind_***_resp PDU indicating the status of binding.

unbind

Unbind PDU is use by ESME to terminate the session and unregister from Message Centre.

submit_sm

This PDU is use to submit a short message to SMSC for onward transmission.

submit_multi

This PDU is use to submit a SMPP message for delivery to multiple recipients or one.

deliver_sm

This PDU is issued by SMSC to send a message to an ESME. Two main purposes of using this PDU by SMSC are;

  1. SMSC may route a short message to the ESME for delivery.
  2. SMSC Delivery Receipt: The message from SMSC indicating the delivery status of the submitted message. If ESME wishes to receive this, it has to change registered_delivery accordingly during submit_sm.

query_sm

This PDU is issued by ESME to query about the delivery state of a submitted short message. ESME must indicate the message_id generated from SMSC, received with submit_sm_resp.

enquire_link

This PDU is issued either by SMSC or ESME to provide a confidence-check of the communication path between an ESME and an SMSC.

Implementing an ESME

Implementation of an ESME can be in any programming language. You could either implementing SMPP specification from scratch or use an existing library. I found jsmpp library [2] for implementing an ESME using Java. You may refer Java SMPP library comparison discussion at [3]. You will also find SMPP libraries for wide range of languages including C#, NodeJS, PHP etc.

Conclusion

In this discussion I wanted to discuss the basic of SMPP protocol and its use. I also discussed some important PDUs you should know. The implementation of ESME is now become simpler as you will find convenient methods are exposed by different libraries.

References

[1] SMPP specification version 3.4: http://opensmpp.org/specs/SMPP_v3_4_Issue1_2.pdf

[2] JSMPP library: https://jsmpp.org/

[3] Java SMPP library comparison: https://stackoverflow.com/questions/14368611/java-smpp-library-comparison

 

Tracing and Circuit-Breaker with Microservices

Introduction

Up to now I have discussed about using Spring Boot for developing microservices [1]. Netflix stack has provided utility solutions for microservices deployments. And I have shown some of those solutions can be used with Spring Boot microservices [2]. So this article is going to be a continuation from the last article on microservices. In here I am going to discuss on using Zipkin tracing, Hystrix Circuit-Breaker and Hystrix Dashboard with Turbine.

The use-case so far

The use-case we have taken is 2 microservices; payment-service and customerinfo-service communicating each other and fronted by a gateway service (based on Zuul). Eureka-service sits as a registry service and helps to figure out the correct service. In this article we are going to implement tracing functionality and circuit-breaker functionality to payment-service.

Implementing zipkin-service

Zipkin is the solution provided in Netflix OSS for distributed tracing of requests. To add it to your microservice setup, you need to implement the zipkin-service first and then configure other services to send tracing events to that service.

First add following two dependencies to each service’s pom file



org.springframework.cloud
spring-cloud-starter-sleuth



org.springframework.cloud
spring-cloud-sleuth-zipkin


Then you have to create the zipkin-service with following artifacts, similar to other microservices.

pom.xml file, refer : https://github.com/Buddhima/project-mss/blob/step-2/zipkin-service/pom.xml

Application.java file with @EnableZipkinServer annotation

package com.buddhima.example.service.zipkin;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import zipkin.server.EnableZipkinServer;

@SpringBootApplication
@EnableZipkinServer
public class Application {

    public static void main(String[] args) {
        new SpringApplication(Application.class).run(args);
    }
}

application.yml file

server:
  port: 9411

spring:
  application:
    name: zipkin-service

Following image depicts zipkin tracing of a request through microservices:

traces

Implementing Hystrix Circuit-Breaker

In short, a circuit-breaker is a design pattern which helps clients to be fault-tolerant with frequently failing endpoints. You will get a more detailed explanation on that concept through references [3], [4].

In this case I have added circuit-breaker functionality to the client-implementation of customerinfo-service at payment-service. In last article I have discussed how feign is used as the client. In addition to those attributes mentioned, we need to add hystrix dependencies in the pom file, implement fallback methods and point to those at feign client definition.

Following dependencies are required for hystrix and hystrix-dashboard

<!-- dependency for hystrix -->

org.springframework.cloud
spring-cloud-starter-hystrix


<!-- dependencies for hystrix-dashboard -->

org.springframework.boot
spring-boot-starter-actuator


org.springframework.cloud
spring-cloud-starter-hystrix-dashboard

Together with Hystrix, I have enabled Hystrix-Dashboard to view the status of each endpoint. This can be done via @EnableCircuitBreaker, @EnableHystrixDashboard annotations.

Then the client interface has been modified, specifying the fallback class as below:


package com.buddhima.example.service.payment.clients;

import org.springframework.cloud.netflix.feign.FeignClient;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;

@FeignClient(value = "customerinfo-service", path = "/customerinfo", fallback = CustomerInfoClientFallback.class)
public interface CustomerInfoClient {

@RequestMapping(method = RequestMethod.GET, value = "/name")
public String getName();

@RequestMapping(method = RequestMethod.GET, value = "/age")
public int getAge();
}

You can refer the implementation of CustomerInfoClientFallback at here: https://github.com/Buddhima/project-mss/blob/step-2/payment-service/src/main/java/com/buddhima/example/service/payment/clients/CustomerInfoClientFallback.java

By-default endpoint dashboard is per microservice, which is not usable when you have many microservices. Therefore Netflix has introduced a solution called Turbine which can aggregate the endpoint statuses of each microservice and construct an aggregated stream to be displayed at hystrix-dashboard.

You may refer source code of turbine-service at here: https://github.com/Buddhima/project-mss/tree/step-2/turbine-service

In the configuration file (application.yml), I have specified the names of microservices which need to be monitored


server:
port: 9060

spring:
application:
name: turbine-service

eureka:
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka/
instance:
preferIpAddress: true

turbine:
clusterNameExpression: new String("default")
appConfig: PAYMENT-SERVICE,CUSTOMERINFO-SERVICE,EDGE-SERVICE
combineHostPort: true

(Here I have configured hystrix in the payment-service only. You may add hystrix configs to other services as well)

hystrix-dashboard-e1517331780499.png
Hystrix Dashboard

You can check the git repository containing the complete source code at here [5].

References

[1] Getting start with Spring Boot : https://buddhimawijeweera.wordpress.com/2017/05/04/microservices-with-spring-boot/
[2] Netflix OSS with Spring Boot : https://buddhimawijeweera.wordpress.com/2017/12/29/microservices-with-netflix-stack-and-spring-boot/
[3] How Hystrix works : https://github.com/Netflix/Hystrix/wiki/How-it-Works
[4] Spring sample for Circuit-Breaker : https://spring.io/guides/gs/circuit-breaker/
[5] Source code : https://github.com/Buddhima/project-mss/tree/step-2

Microservices with Netflix stack and Spring Boot

Introduction

In my previous post on microservices [1], I have discussed how a microservicec can be implemented using Spring Boot. In this post I am going to discuss how a microservice implementation can be leveraged using Netflix OSS. Netflix can be considered as one of the early adopters of this trend. Therefore, they have launched released several projects that they have used in their implementation. Some of those outcomes are Eureka, Zuul, and Feign which I am going to use for the implementation.

Use case

For this implementation, I have chosen a simple use case where a payment-service is using customer-info-service to retrieve customer information. Apart from that customer-info-service can be directly invoked to retrieve data. Those two systems are exposed through a gateway service. Also there is a registry service which manage registrations of those services.

microservice-blog
Plan for the implementation

Implementation

For this implementation I have used spring boot version 1.5.2.RELEASE. Before starting up, make sure you have setup Java and configured it with your favorite IDE (in my case it’s IntelliJ). Also I have used maven as the build tool for this project. So let’s move to implementation of the use-case discussed.

Implementing the Registry Service

As the first step you need to create a maven project with your IDE and create a sub-module called “eureka-service” in it. In the src/java folder, create the package as you prefer (I used “com.buddhima.example.service.eureka”). In side that package create a new Java file called Application.java and add the following code.


package com.buddhima.example.service.eureka;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.server.EnableEurekaServer;

@SpringBootApplication
@EnableEurekaServer
public class Application {
 public static void main(String[] args) {
 new SpringApplication(Application.class).run(args);
 }
}

The annotation @EnableEurekaServer expresses that the microservice is going to be registry service. It requires to have “org.springframework.cloud:spring-cloud-starter-eureka-server” dependency in your pom file. Eureka is a project in Netflix stack which easily transform this microservice in to the registry of other projects. Other microservices can register to this as clients and this registry service will help to figure-out registered services’ locations when requested. You will find more information on Eureka project in its wiki page [2].

Implementing CustomerInfo service

This will be a typical microservice, which provides information about a customer (which is hardcoded for this sample). Without going further explanations, following are the relevant Application.java, CustomerInfoController.java, application.yml and pom.xml files.


package com.buddhima.example.service.customerinfo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;

@SpringBootApplication
@EnableEurekaClient
public class Application {

    public static void main(String[] args) {
        new SpringApplication(Application.class).run(args);
    }
}


package com.buddhima.example.service.customerinfo;

import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping(value = "/customerinfo")
public class CustomerInfoController {

    @RequestMapping(method = RequestMethod.GET, value = "/name")
    public String getName() {
        System.out.println("Name requested");

        return "foo";
    }

    @RequestMapping(method = RequestMethod.GET, value = "/age")
    public int getAge() {
        System.out.println("Age requested");

        return 28;
    }
}

server:
  port: 8000

spring:
  application:
    name: customerinfo-service

eureka:
  client:
    serviceUrl:
      defaultZone: http://localhost:8761/eureka/
  instance:
    preferIpAddress: true

customer-info service pom file : https://github.com/Buddhima/project-mss/blob/step-1/customerinfo-service/pom.xml

Special thing to note in the above code is the @EnableEurekaClient annotation which enables this microservice to register as a service in eureka registry.

Implementing Payment service with Feign client

Payment service is almost similar to customer-info service, but having a feign client which calls customer-info service. For that purpose, the CustomerInfoClient and PaymentController classes are as follows.


package com.buddhima.example.service.payment.clients;

import org.springframework.cloud.netflix.feign.FeignClient;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;

@FeignClient(value = "customerinfo-service", path = "/customerinfo")
public interface CustomerInfoClient {

    @RequestMapping(method = RequestMethod.GET, value = "/name")
    public String getName();

    @RequestMapping(method = RequestMethod.GET, value = "/age")
    public int getAge();
}


package com.buddhima.example.service.payment;

import com.buddhima.example.service.payment.clients.CustomerInfoClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.RequestMethod;

@RestController
@RequestMapping(value = "/payment")
public class PaymentController {

    @Autowired
    private CustomerInfoClient customerInfoClient;

    @RequestMapping(value = "/name", method = RequestMethod.GET)
    public String getName() {
        return customerInfoClient.getName();
    }

    @RequestMapping(value = "/age", method = RequestMethod.GET)
    public int getAge() {
        return customerInfoClient.getAge();
    }
}

To activate feign client, you need to add @EnableFeignClients annotation in Application.java class and add “org.springframework.cloud:spring-cloud-starter-feign” dependency.

Implementing Edge Service with Zuul

The purpose of edge service is closely similar to a software load-balancer. Zuul project [3] is focusing on dynamic routing of messages. It uses Eureka service to find and resolve the addresses for incoming requests and route them to proper microservice. So microservices at the backend can be scaled up/down without changing a single configuration in the rest of environment.

server:
  port: 6000

spring:
  application:
    name: edge-service

eureka:
  client:
    serviceUrl:
      defaultZone: http://localhost:8761/eureka/
  instance:
    preferIpAddress: true

zuul:
  debug:
    request: true
  routes:
    payment-service:
        path: /payment-service/**
        serviceId: payment-service
        stripPrefix: true
    customerinfo-service:
        path: /customerinfo-service/**
        serviceId: customerinfo-service
        stripPrefix: true

OK, that’s it! I have quickly gone through important points. I have added the complete source-code to github [4] for you to refer.

What you can do more

Well, this is not the end. There are so many paths you can take from here. I have highlighted few of those in the following list.

  1. Config-Service : creating a centralized place to manage all the configurations belongs to micrcoservices
  2. Circuit Breaker implementation : circuit breaker pattern avoids propagating backend failures to clients. Another project by Netflix called Hystrix [5], is very popular for this purpose. You may use Turbine project to aggregate multiple microservice information to a single dashboard.
  3. Docker and Kubenetes : Microservices deployments can be leveraged using docker and kubenetes to make it work in fault-tolerant manner.
  4. Analytics using ELK stack : You may heard of ELK stack [6] which provide various forms of support for analyzing data.

Where you can learn more

While doing the experiment, I came across numbers of resources which are written as tutorials for microservices. Some interesting ones are listed below.

  1. Fernando Barbeiro Campos’s Blog : https://fernandoabcampos.wordpress.com/2016/02/04/microservice-architecture-step-by-step-tutorial/
  2. Quimten De Swaef’s Blog : https://blog.de-swaef.eu/the-netflix-stack-using-spring-boot/
  3. Piotr’s TechBlog : https://piotrminkowski.wordpress.com/2017/02/05/part-1-creating-microservice-using-spring-cloud-eureka-and-zuul/
  4. Piggy Metrices, a POC app : https://github.com/sqshq/PiggyMetrics

 

References

[1] Microservices with Spring Boot : https://buddhimawijeweera.wordpress.com/2017/05/04/microservices-with-spring-boot/

[2] Eureka project : https://github.com/Netflix/eureka/wiki/Eureka-at-a-glance

[3] Zuul project : https://github.com/Netflix/zuul/wiki

[4] Github project : https://github.com/Buddhima/project-mss/tree/step-1

[5] Hystrix : https://github.com/Netflix/Hystrix/wiki

[6] ELK stack : https://www.elastic.co/

Simple Fault Statistic Dashboard for WSO2 API Manager with ELK

Introduction

WSO2 API Manager (WSO2 AM) [1] is a product in WSO2 stack which is fully open source and provides a complete API Management solution. It includes API creation, publishing and managing all aspects of an API and its lifecycle, and is ready for massively scalable deployments. Several products including WSO2 AM provides analytics support out of box via WSO2 analytic components.

On the other hand, ELK stack is very powerful and popular product in analytics domain. Specially Elasticsearch product, the heart of ELK stack, provides a convenient way to integrate with other systems with the help of other components in the stack. One key thing to remember when using ELK stack is, you need to stick with the same version across all products [2]. For this article I am using version 6.0.1 of the ELK stack.

image

Use case

I have done this as my first experiment with ELK stack. For this experiment I used following products in ELK stack.

  1. Filebeat
  2. Logstash
  3. Elasticsearch
  4. Kibana

WSO2 API Manager logs almost all errors in its main log (a.k.a. carbon log). Therefore this experiment is to identify faulty situations in the carbon log and use ELK stack to analyse those. So I decided to make it work with following frequent errors.

  1. Deployment failures of APIs : When there’s an incorrect syntax in synapse configurations.
  2. Message Building failures : When incoming message payload is not valid to the given content-type
  3. Backend connection failures : When API fails to establish connection with given backend service
  4. Authentication failures : When access token missing or invalid when calling the API.

In upcoming topics, I am going to show on how to use each component in ELK stack for implementing the use case.

Plan

Document 1

The carbon log keeps growing by the APIM server. In the same server, set up filebeat to read the carbon log. Filebeat push the logs to logstash to do filtering. After filtering logs, logstash pushes logs to elasticsearch for indexing. For visualizing purpose, kibana is set to retrieve data from elasticsearch. According to the plan I have discussed how each component in ELK stack fits together to provide a complete solution.

In next sections, I will describe relevant configurations for each component and the meaning. So that you can customize those according to your environment (here I am using the same host). I assume, you have already setup the WSO2 APIM, and have some understanding on carbon logs.

Setting up filebeat

The responsibility of filebeat is to read logs from the file and start the process. You can download filebeat from the website [3]. You can download filebeat manually or set it as a service. Depending on your setup, you need to place configuration file accordingly. Following are segments of filebeat.yml file need to modify. You will find a template file with name “filebeat.reference.yml”.


filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- type: log

# Change to true to enable this prospector configuration.
 enabled: true

# Paths that should be crawled and fetched. Glob based paths.
 paths:
 - /home/buddhima/telco/expreiments/elk/latest/wso2am-2.0.0/repository/logs/wso2carbon.log

. . .

#output.elasticsearch:
 # Array of hosts to connect to.
 #hosts: ["localhost:9200"]

. . .

output.logstash:
 # The Logstash hosts
 hosts: ["localhost:5044"]

In above configuration, log reading prospector is enabled and points to carbon log file of the WSO2 APIM. By default logstash output to elasticsearch is enabled. But according to the plan we need to send logs thorough logstash to filter. Therefore I commented the output.elasticsearch and host followed by that. The uncommented the output.logstash and the host followed by that. I let the hostname and ports remain the default as I have done this on the same machine. Once you complete it, you can start filebeat by following command (for standalone version). It will detect filebeat.yml as default configuration (which I have modified).


./filebeat run

Setting up logstash

Logstash plays a major role in this setup. Logstash is capable of filtering logs according to given criteria. You can download logstash at website [4]. Logstash uses input -> filter -> output order to process log inputs from filebeat. I created logstash-beat.conf file to write the configuration. Following is the logstash configuration to cater detection of above mentioned failures.

 

download
logstash-beat.conf

In the above config I have configured filebeat as the input and elasticsearch as the output. The filter section specifies how input logs should be handled before sending to output. If you have closely observed carbon logs, you will find Deployment failures and Message Build failures are recorded as error logs whereas Backend connection failures and Authentication failures are recorded as warning logs.

Within the filter, logs are initially separated according to log levels. Then filter logs according to subcategories. The grok filter is used to map some fields in the log message. mutate filter is used to add a new field “error_type” to the logs sent by filebeat. The drop filter is used to avoid forwarding unnecessary logs.

To start logstash with the configuration, execute the following command:


./logstash -f ../logstash-beat.conf

Setting up elasticsearch

Now you are getting closer to the final part. Elasticsearch is the most crucial component of this setup. However you no need to configure it for this experiment. Just download it from the website [5] and execute the following to start it.


./elasticsearch

Setting up kibana

Kibana is a powerful visualizing tool in the ELK stack. You can point kibana to different indexes in elasticsearch and visualize data. Though kibana provides view of processed data, it is capable of visualizing data periodically with the minimum of 5 seconds interval. Similar to elasticsearch, you no need to do any configuration changes to kibana for this experiment. You just need to download it from website [6] and execute following command to start it:


./kibana

Once it starts, you can access http://localhost:5601/ via web browser to access web UI. At the web UI, you can select the index as logstash-* as the index. Then you can go to visualizing section and add graphs you like and aggregate them to a dashboard. I’m going to stop here and let you to compose a dashboard you wish. Remember that elasticsearch is getting only the faulty logs according to logstash configuration. Following is a sample dashboard I have created based on 4 types on errors in WSO2 API Manager.

dashboard2
Configuring refresh time in Kibana
dashboard
Kibana dashboard view

Conclusion

This is my first experiment with ELK stack and I have shown how to compose an fault analysis dashboard with ELK stack for WSO2 API Manager. Here I used WSO2 APIM 2.0.0 version. I have discussed the reason for using each component in ELK stack and configurations relevant for those components. You will find configuration files at the following link [7]. Hope this will be your starting point and will help to build up a useful monitoring dashboard with ELK stack.

References

[1] WSO2 API Manager Documentation : https://docs.wso2.com/display/AM200/WSO2+API+Manager+Documentation

[2] ELK product compatibility : https://www.elastic.co/support/matrix#matrix_compatibility

[3] Filebeat website : https://www.elastic.co/products/beats/filebeat

[4] Logstash website : https://www.elastic.co/products/logstash

[5] Elasticsearch : https://www.elastic.co/products/elasticsearch

[6] Kibana : https://www.elastic.co/products/kibana

[7] Resources : https://www.dropbox.com/s/k8b7js9jbrobxjj/elk-resources.zip?dl=0

Analyzing Memory Usage of an Application

Introduction

Memory usage of an application is a key factor to monitor. Specially in production systems you need to set alarms to make sure that system is stable. Therefore memory usage can be considered as a probe to measure the health of the system. Usually production systems are installed on Linux servers. OS itself help in many ways to provide a clear view of application’s memory usage.

featured_image

In this post, I am going to discuss different commands and tools which can be used to measure memory usage of applications, specially Java applications. This post will guide you from higher-level to lower-level under following topics.

  1. Monitoring Overall System Memory Usage
  2. Monitoring Application’s Memory Usage
  3. Analyzing Java Application’s Memory Usage

Further information on commands and tools can be gained by going through the external links provided.

Monitoring Overall System Memory Usage

For this purpose I am going to discuss using “free” command in Linux. Free command gives an overview of complete system memory usage. Therefore please note that this is not an efficient way of measuring an applications performance. Because the system can host many applications and each application has its own boundaries. However, let’s look in to some usages of free command. (hint: use free -h to get a human readable output)

[root@operation_node ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          7873       7360        512          0         70        920
-/+ buffers/cache:       6369       1503
Swap:        11143        431      10712

In the above output, in the first line, you can see total physical memory is 7873 MB and 7360 MB is used. Because of that only 512 MB is remaining. However this does not imply that memory is completely being used. Linux OS is good at using memory effectively. Therefore it uses caching for make the memory access efficient. And that cached memory is showing as used in the first line.
What you should look at is the second line which removed the cache & buffer usage of the physical memory. In used column of second line shows actual use of memory without cache & buffer. In the free column of second row you can see 1503 MB of free memory, which is generated by accumulating free + cache & buffer. So actually you have 1503 MB of free physical memory for use. In addition, according to the third line, you have around 10 GB of swap memory ready for use. Please refer [1] for further information.

In modern versions of Linux kernels, the output of free command has changed. It would be look like something below.

me@my-pc ~ $ free -m

                  total        used        free      shared  buff/cache   available

Mem:           7882        3483         299         122        4100        3922

Swap:          9535           0        9535

In above case, the formula for calculation is as below [2] [3]:

total = used + free + buffers + cache

available : is the amount of memory which is available for allocation to a new process or to existing processes.

free : is the amount of memory which is currently not used for anything. This number should be small, because memory which is not used is simply wasted.

Monitoring Application's Memory Usage

In many cases we want to target monitoring of a single application than overall system. Overall memory usage reflects the use of memory including OS-level operations. Therefore we can use top command in Linux for this purpose. Following is a sample output of top command.

top-cmd-result
A result of a top command

What you should actually focus is the RES value and %MEM value of the application (for this, first you need to identify the process-id of an application using ps -aux | grep "application_name" command). You can use simple "e" to toggle the unit of displaying memory.

RES -- Resident Memory Size : The non-swapped physical memory a task has used.

%MEM -- Memory Usage (RES) : A task's currently used share of available physical memory (RAM).

According to above discussion, top command directly reveals the memory consumption of an application. For further information on top command, you may refer [4] [5].

Analyzing Java Application's Memory Usage

If your application is a Java application, then you might need to look at what objects consumes the high amount of memory. For that one option is taking a heap-dump of that Java application. You may use following command to take a heap dump of the application. Prior to that you should know the process-id of that running Java application.

jmap -dump:format=b,file=heap_dump.hprof <process_id>

Once you execute the command, you will get the file heap_dump.hprof  containing heap usage of Java program. Since the file is in binary format, you need to use a special tool to analyze it. Commonly using tool to inspect heap-dump is Eclipse Memory Analyzer Tool (MAT) [6], which is built on top of Eclipse platform. You just need to download the pack and extract it. Executing MemoryAnalyzer will open up a GUI application to analyze the heap-dump. When you open the heap-dump using MAT, tool will prompt you to generate reports based on heap-dump. You may interest about Leak Suspects Report which shows the large object which takes large portion of the memory.

MAT-leak_suspects
Memory Analyzer Tool with Leak Suspects Report

Another interesting view of this tool is the Dominator Tree, which shows large objects along with ones who keep them. According to the definition [7];

An object x dominates an object y if every path in the object graph from the start (or the root) node to y must go through x.

In the dominator tree view, you will see list of objects and the amount of memory they took when you take the heap-dump.

dominator_tree
Dominator Tree view of Memory Analyzer

In dominator tree view, you can go expanding each entry and see how they have composed. Two columns showing in this view are Shallow Heap and Retained Heap. By default the list is sorted by Retained Heap value, descending order. Following definition [8] clearly explain the meaning of those two values.

Shallow heap is the memory consumed by one object. An object needs 32 or 64 bits (depending on the OS architecture) per reference, 4 bytes per Integer, 8 bytes per Long, etc. Depending on the heap dump format the size may be adjusted (e.g. aligned to 8, etc...) to model better the real consumption of the VM.

Retained set of X is the set of objects which would be removed by GC when X is garbage collected.

Retained heap of X is the sum of shallow sizes of all objects in the retained set of X, i.e. memory kept alive by X.

Generally speaking, shallow heap of an object is its size in the heap and retained size of the same object is the amount of heap memory that will be freed when the object is garbage collected.

Therefore in case of out-of-memory or high memory indications, you should definitely focus on this Retained Heap values of the dominator tree view.

Conclusion

In this post I want to give an clear idea on using several tools to analyze the memory usage of an application running on Linux OS. I have used commands which comes with Linux itself. However you may find tools which can be installed to analyze memory. In the last section I spent on discussing how Eclipse Memory Analyzer can be used to examine heap usage of a Java program. Hope those will help you as well.

References

[1] Understanding Linux free memory : https://thecodecave.com/understanding-free-memory-in-linux/

[2] Usage of free memory : https://stackoverflow.com/questions/30772369/linux-free-m-total-used-and-free-memory-values-dont-add-up

[3] Ask Ubuntu clarification on free command : https://askubuntu.com/questions/867068/what-is-available-memory-while-using-free-command

[4] Super-user forum top command explanation : https://superuser.com/questions/575202/understanding-top-command-in-unix

[5] Linuxarea blog top command explanation : https://linuxaria.com/howto/understanding-the-top-command-on-linux

[6] Eclipse Memory Analyzer : https://www.eclipse.org/mat/

[7] MAT Dominator Tree : https://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.mat.ui.help%2Fconcepts%2Fdominatortree.html

[8] MAT Shallow Heap and Retained Heap explanation : https://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.mat.ui.help%2Fconcepts%2Fdominatortree.html

Inspecting Solr Index in WSO2 API Manager

Introduction

Apache Solr project [1] helps you to run a full-featured search server on a server. Also you can integrate Solr with your project to make searching faster. In WSO2 API Manager, Solr is using to make searching faster in store and publisher. In WSO2 API Manager, Solr indexing keeps the frequently using meta-data of APIs. Thereafter, to retrieve complete information about an API, API Manager uses its database. This mechanism makes searching faster and less burden on the databases.

adressenboek

However, in some situations things may go wrong. We have seen several cases Solr indexing does not tally with the information at database. Due to that, when displaying complete information about an API, you may see inconsistent information. In such situations you may want to inspect the Solr index at API Manager.

Setting Up Solr Server

Setting up the solr server is quite easy, as its a matter of downloading binary file from the project page [1]. The important thing here is to make sure you download the proper version. WSO2 API Manager 2.0.0 version is using Solr 5.2.1 version. I figured out it by going through the API Manager release tag pom, identify the registry version and searching registry pom file.

Once you download the binary package, extract it. You can start the Solr server by going to solr-5.2.1/bin directory and execute “./solr start“. Then Solr server will start as a background process. Then access its admin UI by location “http://localhost:8983/solr” in your browser.

Inspecting WSO2 API Manager Index

Before doing so, you must stop WSO2 API Manager and Solr server. To stop Solr server, execute command “./solr stop” inside bin directory. Then you need to copy Solr indexing configs and index from API Manager.

  • To copy configs, go to location “APIM_HOME/repository/conf/solr” and copy “registry-indexing” to “solr-5.2.1/server/solr” folder.
  • To copy indexed data, go to the location “APIM_HOME” and copy “solr” folder to same folder “solr-5.2.1” resides. This is done to comply the “dataDir” value at “solr-5.2.1/server/solr/registry-indexing/core.properties” file.

Now start the Solr server and go to admin UI. You should see a drop-down on the left pane and “registry-indexing” menu item in that. Select “registry-indexing” and now you will be able to query indexed data by going to Query section. To query Solr index you need to use specific query language, which is not actually difficult to understand. But in here I’m not going to discuss too much on query language, and it’s up to you to refer [2] and learn it. You can try-out those queries from admin UI directly.

registry-indexing in Solr admin UI

Writing a Java client to query information

In some cases, you may need to write a client to client which can talk to a Solr server and retrieve results. So here I am giving out an example Java code which you can use to retrieve results from a Solr server [3]. However, I am not going to explain the code in detail, because I believe it’s self-explanatory.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.buddhima.solr</groupId>
<artifactId>solr-testing</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
    <!-- https://mvnrepository.com/artifact/org.apache.solr/solr-solrj -->
    <dependency>
        <groupId>org.apache.solr</groupId>
        <artifactId>solr-solrj</artifactId>
        <version>7.1.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.solr/solr-common -->
    <dependency>
        <groupId>org.apache.solr</groupId>
        <artifactId>solr-common</artifactId>
        <version>1.3.0</version>
    </dependency>
</dependencies>
</project>
package com.solr.testing;

import java.io.IOException;

import org.apache.solr.client.solrj.SolrClient;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrDocumentList;

/**
 * Created by buddhima.
 */
public class SolrTesting {

    public static void main(String[] args) throws IOException, SolrServerException {
        // Default Solr port: 8983, and APIM using 'registry-indexing'
        SolrClient client = new HttpSolrClient.Builder("http://localhost:8983/solr/registry-indexing").build();

        SolrQuery query = new SolrQuery();
        query.setQuery("*:*");

        // Fields use for filtering as a list of key:value pairs
        query.addFilterQuery("allowedRoles:internal/everyone", "mediaType_s:application/vnd.wso2-api+xml");

        // Fields to show in the result
        query.setFields("overview_name_s", "overview_status_s", "updater_s", "overview_version_s", "overview_context_s");

        // Limit the query search space
        query.setStart(0);
        query.setRows(500);

        // Execute the query and print results
        QueryResponse response = client.query(query);
        SolrDocumentList results = response.getResults();
        for (int i = 0; i < results.size(); ++i) {
            System.out.println(results.get(i));
        }
    }
}

In addition to that you can refer [4] [5] for further learning on Solr syntax.

Conclusion

In this post I have discussed the use of Solr in WSO2 API Manager and how to investigate existing Solr index. In addition to that I have shown how to construct a Java client which can talk to a Solr server. I hope that the above explanation will help you to solve issues with Solr indexing.

Special thank goes to WSO2 support for providing guidance.

References

[1] Apache Solr project : http://lucene.apache.org/solr/

[2] Solr query syntax : http://www.solrtutorial.com/solr-query-syntax.html

[3] Using SolrJ : http://www.solrtutorial.com/solrj-tutorial.html

[4] Solr Query Syntax : http://yonik.com/solr/query-syntax/

[5] Solr df and qf explanation : https://stackoverflow.com/questions/17363677/solr-df-and-qf-explanation