Saturday, August 27, 2016

On Kafka

This blog captures steps to install and setup Kafka.

For instructions on how to install Kafka, visit: Installation Steps

After downloading:
untar bundle to /opt/apache-kafka
ln -s kafka_2.11-0.9.0.1 current
cd /opt/apache-kafka/current

At this point you can start Zookeper:

bin/zookeeper-server-start.sh config/zookeeper.properties&

To run Kafka on the current server, modify config/server.properties:

Uncomment:

port=9092
host.name=localhost

To allow for removing topics, add the following to config/server.properties:

# Enable topic deletion - only for development
delete.topic.enable=true

Now you can start Kafka:

bin/kafka-server-start.sh config/server.properties&

Kafka REST Proxy
In order to publish messages to Kafka via REST calls, you can use Kafka REST Proxy. To install it, visit Confluent Platform Quickstart.

Untar the downloaded bundle to /opt/confluent:

ln -s confluent-2.0.1 current
cd current
vi etc/kafka-rest/kafka-rest.properties

Uncomment

id=kafka-rest-test-server
schema.registry.url=http://localhost:8081
zookeeper.connect=localhost:2181

To run Kafka REST Proxy:
bin/schema-registry-start etc/schema-registry/schema-registry.properties &
bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties &

Some useful aliases to start/stop/view Kafka processes
alias zs="cd /opt/apache-kafka/current; nohup bin/zookeeper-server-start.sh config/zookeeper.properties 2>/dev/null &"
alias ks="cd /opt/apache-kafka/current; nohup bin/kafka-server-start.sh config/server.properties 2>/dev/null&"

alias krp1="cd /opt/confluent/current; nohup bin/schema-registry-start etc/schema-registry/schema-registry.properties 2>/dev/null&"
alias krp2="cd /opt/confluent/current; nohup bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties 2>/dev/null&"

alias pk="ps -aef|grep kafka"
alias topics="/opt/apache-kafka/current/bin/kafka-topics.sh  --list --zookeeper localhost:2181"

To create topics:
cd/opt/apache-kafka/current; bin/kafka-topics.sh --create --topic test --zookeeper YOUR.IP.ADDRESS.HERE:2181 --partition 1 --replication-factor 1

To list topics:
cd /opt/apache-kafka/current; bin/kafka-topics.sh --list --zookeeper YOUR.IP.ADDRESS.HERE:2181

To consume messages from a topic:
/opt/apache-kafka/current/bin/kafka-console-consumer.sh --zookeeper YOUR.IP.ADDRESS.HERE:2181 --topic MY_TOPIC 


1 comment:

  1. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.
    Digital Marketing Company in India

    ReplyDelete