Deploying WSO2 Streaming Integrator on AWS ECS in Minimum HA Mode

6 min readNov 2, 2020

In article, I have given a step by step guide on how to create a WSO2 Streaming Integrator docker images configured in HA mode and how to push those to the AWS ECR.

In this article, Let’s see how to deploy those two docker images in Amazon ECS.

Creating a VPC

We need both nodes to be in a single subnet to communicate with each other. Let’s navigate to the VPC dashboard create a VPC using the VPC wizard (Clcik on launch VPC wizard).

Select VPC with a Single Public Subnet

Give VPC name as si-ha-vpc and click on Create VPC

After successfully creating the VPC, you can view it in the VPC section

The created public subnet can view it in the Subnets section

A Relevant security group will be created in the Security group section. And for the time being, I have enabled all incoming traffic.

Setting up the Cluster

In the ECS Cluster page, Click on Create Cluster

Select EC2 Linux + Networking section and click on Next Step

Choose Create an empty cluster Enter Cluster name and click on Create

Creating a Task for Streaming Integrator’s First Node

Create a new Task Definition

Select on Fargate and click on Next Step

Enter Task definition name

Select task Memory and CPU capacities

Let’s add the container to the task in the same section. Click on Add Container

In the container configurations, enter a name for the container and copy the image URI as per the ECR

When you scroll down in the container configurations, Add the ports needed for the Streaming Integrator (9893, 9090, 9711, 9611, 7711, 7611, 7443, 7070, 9443, 8006)

Image URI can be taken from the ECR

And create the task.

Creating a Task for Streaming Integrator’s Second Node

Create a new task called `ha-node2-task` and enter all the other values as before other than the container details.

In the `Add container` enter the container name as `node2-ha-container` and select the 2nd Image URI from ECR as Image URI. And add the port mapping as mentioned above.

And create the task.

Create a service using the Task (ha-node1-task) for the First Node

Select the created cluster (si-ha-cluster) and create a service after selecting the Services tab

Select and enter the shown information and click on Next Step

Select the VPC and the Subnet which we created.

And then click on Edit to select the security group (I have selected the security group which was created by the VPC creation wizard). After that, scroll down and click on Next Step

Skip Set Auto Scaling and Review and Create the service.

Create a service using the Task (ha-node2-task) for the Second Node

All the information that should be selected and entered will be the same as when we created the 1st service other than the following information we entered when creating the service in the initial Step.

Task definition: ha-node2-task
Service name: ha-node2-service

Fill in the rest of the information. Review and Create the service.

If needed, the services can be run in two different clusters as well.

Viewing the Status of the servers

Goto the cluster and in the Tasks Tab, click on a running task.

Scroll down on the task information page, expand container and click on View logs in cloudwatch

The Cloudwatch will have logs similar to the following.

Checking on Streaming Integrator Cluster via the CloudWatch Logs

Active Node Logs

We can identify the Active node by the following 3 logs. This means, the node has started as active node and started persisting siddhi application statuses periodically.

[2020–03–20 20:18:48,283] INFO {org.wso2.carbon.streaming.integrator.core.ha.HAManager} — HA Deployment: Starting up as Active Node[2020–03–20 20:18:52,261] INFO {org.wso2.carbon.kernel.internal.CarbonStartupHandler} — WSO2 Streaming Integrator started in 44.893 sec[2020–03–20 20:19:46,595] INFO {org.wso2.carbon.streaming.integrator.core.persistence.PersistenceManager} — Siddhi apps persisted successfully

Passive Node Logs

If the 2nd node has started successfully, the following logs can be found in the Passive node.

[2020–03–20 20:38:14,390] INFO {org.wso2.carbon.streaming.integrator.core.ha.HAManager} — HA Deployment: Starting up as Passive Node[2020–03–20 20:38:18,287] INFO {org.wso2.carbon.kernel.internal.CarbonStartupHandler} — WSO2 Streaming Integrator started in 46.604 sec

Confirming HA functionality

If the HA configuration is working as expected, the following logs can be seen in the Active node.

[2020–03–20 20:38:15,409] INFO {org.wso2.carbon.streaming.integrator.core.ha.HAEventListener} — memberAdded event received for node id : wso2-si-2[2020–03–20 20:38:15,455] INFO {org.wso2.carbon.streaming.integrator.core.ha.HAEventListener} — Active node retrieved node details of passive node

And the following log should be printed without followed by Error traces. Which means the receiving events and the control messages are sent to the passive node via passive node’s event sync server.

[2020–03–20 20:38:46,587] INFO {org.wso2.carbon.streaming.integrator.core.persistence.PersistenceManager} — Siddhi apps persisted successfully

I hope you got know some information on how to run docker images in ECS and got a small knowledge on WSO2 Streaming Integrator Minimum HA functionality and how to figure it out using its logs.