-Xms -Xmx

GCP advantages & benefits over AWS/Azure

Google Cloud Platform or GCP has raised the bar for other cloud service providers since last 2 odd years. What was increasing looking like a two horse race between AWS and Azure, has now changed due to increased participation from GCP. A CxO had a binary choice until very recently,

spark

Spark Sql Optimisation - A data skew story

Performance Optimisation is hard, I had a few lessons the hard way with Elasticsearch Query Optimisation and OpenTSDB/HBase Tuning in the past. Spark Optimisation or Spark SQL Optimisation results in gazillions results, each one talking about how to tweak configurations and settings, How people solved a performance problem in

design-patterns

Configuration as Code

I recently got to work with something colleagues described as configuration as code. A search for configuration as code does not come up with many details but it is a thing, mentioned in airbnb blog. one (and only) formal definition is given as: Configuration as code is the formal migration

Health

Running: 0 to 5K, in 4 years!

Embed from Getty Images How many times you have seen someone running around near your office/park/home and told yourself "I want to be this guy/gal". Me too. But you are still not that guy/gal. This post is an honest account of me turning from

hadoop

Hadoop Tutorial for Beginners

Can you give me a quick Hadoop 101? or Can you give the team a tutorial on What is Hadoop? are some of the questions I get in almost all my engagements. These are the questions I get asked from friends, colleagues and everyone else who want to learn Hadoop.

kafka

Kafka Consumer Offset Management

Apache Kafka is the leading data landing platform. It is the de-facto standard for collecting and then streaming data to different systems. It is also used a filter system in many cases where messages from a topic are read and then put on a different topic after processing, much

hadoop

Hadoop Application Architecture Examples

There are several posts, blogs, and white papers on Hadoop Application Architectures. Patterns have been named: lambda, kappa etc. Jargons has developed, most notable is 'Data Lake'. Regardless of what a system is called, Hadoop and its ecosystem components are used in predictable ways. For example, * HDFS is

cloud computing

OpenTSDB/HBase - Improving Stability And Performance

OpenTSDB is a time series database based on HBase. It is used as a distributed monitoring system where metrics are collected and pinged to openTSDB from servers/machines. The metrics collected can be anything from CPU, RAM usage to Application Cache Hit ratios to Message Queue statistics. It can scale

elasticsearch

ElasticSearch Query: Performance Optimisation

In one of my previous posts on elasticsearch, i shared my understanding of elasticsearch configurations and best practices. That was mostly from an indexing perspective. There are several tweaks one can use to optimise query performance as well. Improving querying time can be even more challenging than trying to improve

cloud computing

Why Windows HDInsights is going nowhere

Everyone wants to do big data, Microsoft is no exception. Jumping on the bigdata bandwagon and cashing in is something no one wants to miss. We have so many players in the market: AWS, Cloudera, MapR, HortonWorks, IBM, Intel and of course open source Hadoop Ecosystem. Stakes are high and

Tools of the Trade

This is note to self post. I am listing here, some of the tools that i use on a daily basis. Some of these tools are just awesome, like powerline. The picture below is the current state of my terminal window :-) * Vi Commands * :set paste * :%le -> indent everything to

hadoop

Performance Tuning Data Load into Hadoop with Sqoop

Working with hadoop involves working with huge amounts of data. It also, at times, involves moving huge amounts of data from traditional data stores such as mysql and oracle. Apache Sqoop is an excellent tool that aids in migrating data to and from a hadoop cluster. Data migration into hadoop

database

Impala and Mysql: Comparing apples to oranges

Hadoop has become de-facto standard for big data and batch processing. Think of data pipeline and end up with Hadoop. The Hadoop eco system is changing and changing at a rapid pace. Hadoop serves very well with Hive for batch oriented tasks. But as the eco system move towards

Java

The Dependency Injection Debate

Hackernews is an awesome place. It keeps people involved and interested. There are articles, questions, answers and debates. One such debate going on is about Dependency Injection. It was declared as a non-virtue and subsequently some one else declared it a virtue. Virtue or not, one thing is clear

kafka

Comparison Matrix: Real time data processing systems

There are several tools/framework available that help process data as it arrives. I had done a comparative study of below four systems in the past: * Apache Kafka * Facebook Scribe * Cloudera Flume * Apache Chukwa KafkaScribeFlumeChukwaCurrent Version0.612.2?0.9.41,20.41Site & DocsAverageVery PoorGoodPoorTopologyP2PMaster/Slave3Master/Slave3, 4P2PCentral Node

aws

Hadoop Cluster on AWS VPC with Apache Whirr

Setting up hadoop cluster on cloud providers has been made relatively easy with tools such as apache whirr, cloudera manager, jclouds. Whirr uses jclouds internally. But what if one wanted to create a cluster thats not in the open public cloud? What if one wanted to create a cluster in

HBase

HBase Hive Integration

While working with hadoop related technologies one touches several tools/frameworks. Once we have a hadoop cluster running, the next thing we want is to update records and have sql-like features. HBase provides a way to fulfil the former and Hive fulfils the later. But what fun it would

aws

ElasticSearch Cluster: Configuration & Best Practices

I was recently working on setting up an elasticsearch cluster with apache whirr. Setting up a cluster is one thing and running it is entirely different. Running a cluster is far more complex than setting one up. Things are no different for an elasticsearch cluster. There are several things one

aws

Apache Whirr: ElasticSearch Cluster Setup on Amazon EC2

The last time i tried to create a cluster, i had several problems. I was creating a hadoop cluster on amazon ec2. Taking the learnings forward, i decided to create a elasticsearch cluster on ec2 with apache whirr. The idea of launching several machines with one command is really enticing

hadoop

Common Hadoop (YARN) Errors

While installing or running hadoop one gets different errors at different times. Here is a list of some that i could think of: 1. class org.apache.hadoop.mapred.ShuffleHandler not found - This errors comes in the YARN nodemanager. This is due to fact that the system could not find

aws

Apache Whirr: Create Hadoop Cluster Automatically

Creating a hadoop cluster is a task. It becomes all the more challenging because of ever changing/evolving nature of technologies involved. There was Apache Hadoop then we had Cloudera Hadoop and now we have very many (Hotronworks, MapR etc). Regardless of the flavour one chooses, getting a hadoop cluster

aws

AWS for Dummies

Here are some of the basic question that one comes across while working with AWS. I will try and keep adding more to the list. * Can i change security group of my EC2 instance? The answer is no. One can change security group of an instance that’s behind VPC

aws

No supported authentication methods available

While trying to SSH into AWS EC2 instance one may get this error No supported authentication methods available The most common reason is wrongly added private key into the .ssh/authorised_keys file. Look for: * multiple lines in your key, the key needs to be one line only. * comments or

Health

FitBit: Towards a -slightly- fitter tomorrow

Finally, after being long enough on my amazon wish list, Fitbit Ultra has now the distinction of being someone i cannot sleep without! Others being, obviously enough, my wife and my iPhone. The device costs, as of writing this, GBP 79.99 on amazon UK and costs USD 77.74

Java

Memory leaks and object pools

Java based development involves working with several open source software. It is very easy to start using a open source tool but it needs careful examination and evaluation. One of the things that i have done in recent past is working with GATE. GATE is an open source software for