Package org.apache.heron.spouts.kafka

Class KafkaSpout<K,V>

java.lang.Object

org.apache.heron.api.topology.BaseComponent

org.apache.heron.api.spout.BaseRichSpout

org.apache.heron.spouts.kafka.KafkaSpout<K,V>

Type Parameters:

K - the type of the key field of the Kafka record

V - the type of the value field of the Kafka record

All Implemented Interfaces:

Serializable, org.apache.heron.api.spout.IRichSpout, org.apache.heron.api.spout.ISpout, org.apache.heron.api.topology.IComponent, org.apache.heron.api.topology.IStatefulComponent<org.apache.kafka.common.TopicPartition,Long>

public class KafkaSpout<K,V>
extends org.apache.heron.api.spout.BaseRichSpout
implements org.apache.heron.api.topology.IStatefulComponent<org.apache.kafka.common.TopicPartition,Long>

Kafka spout to consume data from Kafka topic(s), each record is converted into a tuple via ConsumerRecordTransformer, and emitted into a topology

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
class	KafkaSpout.KafkaConsumerRebalanceListener

Constructor Summary

Constructors

Constructor	Description
KafkaSpout(KafkaConsumerFactory<K,V> kafkaConsumerFactory, Collection<String> topicNames)	create a KafkaSpout instance that subscribes to a list of topics
KafkaSpout(KafkaConsumerFactory<K,V> kafkaConsumerFactory, TopicPatternProvider topicPatternProvider)	create a KafkaSpout instance that subscribe to all topics matching the topic pattern

Method Summary

Modifier and Type	Method	Description
void	ack(Object msgId)
void	activate()
void	close()
void	deactivate()
void	declareOutputFields(org.apache.heron.api.topology.OutputFieldsDeclarer declarer)
void	fail(Object msgId)
Map<String,Object>	getComponentConfiguration()
ConsumerRecordTransformer<K,V>	getConsumerRecordTransformer()	return the consumer record transformer
void	initState(org.apache.heron.api.state.State<org.apache.kafka.common.TopicPartition,Long> aState)
void	nextTuple()
void	open(Map<String,Object> conf, org.apache.heron.api.topology.TopologyContext context, org.apache.heron.api.spout.SpoutOutputCollector aCollector)
void	preSave(String checkpointId)
void	setConsumerRecordTransformer(ConsumerRecordTransformer<K,V> consumerRecordTransformer)	set the Kafka record transformer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaSpout

public KafkaSpout(KafkaConsumerFactory<K,V> kafkaConsumerFactory,
                  Collection<String> topicNames)

create a KafkaSpout instance that subscribes to a list of topics

Parameters:

kafkaConsumerFactory - kafka consumer factory

topicNames - list of topic names

KafkaSpout

public KafkaSpout(KafkaConsumerFactory<K,V> kafkaConsumerFactory,
                  TopicPatternProvider topicPatternProvider)

create a KafkaSpout instance that subscribe to all topics matching the topic pattern

Parameters:

kafkaConsumerFactory - kafka consumer factory

topicPatternProvider - provider of the topic matching pattern

Method Detail

getConsumerRecordTransformer

public ConsumerRecordTransformer<K,V> getConsumerRecordTransformer()

return the consumer record transformer

Returns:

the Kafka record transformer instance used by this Kafka Spout

setConsumerRecordTransformer

public void setConsumerRecordTransformer(ConsumerRecordTransformer<K,V> consumerRecordTransformer)

set the Kafka record transformer

Parameters:

consumerRecordTransformer - kafka record transformer

open

public void open(Map<String,Object> conf,
                 org.apache.heron.api.topology.TopologyContext context,
                 org.apache.heron.api.spout.SpoutOutputCollector aCollector)

Specified by:

open in interface org.apache.heron.api.spout.ISpout

initState

public void initState(org.apache.heron.api.state.State<org.apache.kafka.common.TopicPartition,Long> aState)

Specified by:

initState in interface org.apache.heron.api.topology.IStatefulComponent<K,V>

preSave

public void preSave(String checkpointId)

Specified by:

preSave in interface org.apache.heron.api.topology.IStatefulComponent<K,V>

nextTuple

public void nextTuple()

Specified by:

nextTuple in interface org.apache.heron.api.spout.ISpout

activate

public void activate()

Specified by:

activate in interface org.apache.heron.api.spout.ISpout

Overrides:

activate in class org.apache.heron.api.spout.BaseRichSpout

deactivate

public void deactivate()

Specified by:

deactivate in interface org.apache.heron.api.spout.ISpout

Overrides:

deactivate in class org.apache.heron.api.spout.BaseRichSpout

ack

public void ack(Object msgId)

Specified by:

ack in interface org.apache.heron.api.spout.ISpout

Overrides:

ack in class org.apache.heron.api.spout.BaseRichSpout

fail

public void fail(Object msgId)

Specified by:

fail in interface org.apache.heron.api.spout.ISpout

Overrides:

fail in class org.apache.heron.api.spout.BaseRichSpout

close

public void close()

Specified by:

close in interface org.apache.heron.api.spout.ISpout

Overrides:

close in class org.apache.heron.api.spout.BaseRichSpout

declareOutputFields

public void declareOutputFields(org.apache.heron.api.topology.OutputFieldsDeclarer declarer)

Specified by:

declareOutputFields in interface org.apache.heron.api.topology.IComponent

getComponentConfiguration

public Map<String,Object> getComponentConfiguration()

Specified by:

getComponentConfiguration in interface org.apache.heron.api.topology.IComponent

Overrides:

getComponentConfiguration in class org.apache.heron.api.topology.BaseComponent