org.apache.samza.system.descriptors

Class InputDescriptor<StreamMessageType,SubClass extends InputDescriptor<StreamMessageType,SubClass>>

java.lang.Object

org.apache.samza.system.descriptors.StreamDescriptor<StreamMessageType,SubClass>

org.apache.samza.system.descriptors.InputDescriptor<StreamMessageType,SubClass>

Type Parameters:

StreamMessageType - type of messages in this stream.

SubClass - type of the concrete sub-class

Direct Known Subclasses:

GenericInputDescriptor

public abstract class InputDescriptor<StreamMessageType,SubClass extends InputDescriptor<StreamMessageType,SubClass>>
extends StreamDescriptor<StreamMessageType,SubClass>

An InputDescriptor can be used for specifying Samza and system-specific properties of input streams.

Stream properties provided in configuration override corresponding properties specified using a descriptor.

This is the base descriptor for an input stream. Use a system-specific input descriptor (e.g. KafkaInputDescriptor) obtained from its system descriptor (e.g. KafkaSystemDescriptor) if one is available. Otherwise use the GenericInputDescriptor obtained from a GenericSystemDescriptor.

Constructor Summary

Constructors

Constructor and Description
InputDescriptor(java.lang.String streamId, Serde serde, SystemDescriptor systemDescriptor, InputTransformer transformer) Constructs an InputDescriptor instance.

Method Summary

Modifier and Type	Method and Description
java.util.Optional<InputTransformer>	getTransformer()
SubClass	isBounded() If set, this stream will be considered a bounded stream.
SubClass	shouldBootstrap() If set, this stream will be processed as a bootstrap stream.
SubClass	shouldDeleteCommittedMessages() If set, and supported by the system implementation, messages older than the latest checkpointed offset for this stream may be deleted after the commit.
SubClass	shouldResetOffset() If set, when a Samza container starts up, it ignores any checkpointed offset for this particular input stream.
java.util.Map<java.lang.String,java.lang.String>	toConfig()
SubClass	withOffsetDefault(SystemStreamMetadata.OffsetType offsetDefault) If a container starts up without a checkpoint, this property determines where in the input stream we should start consuming.
SubClass	withPriority(int priority) If one or more streams have a priority set (any positive integer), they will be processed with higher priority than the other streams.

Methods inherited from class org.apache.samza.system.descriptors.StreamDescriptor

getPhysicalName, getSerde, getStreamId, getSystemDescriptor, getSystemName, withPhysicalName, withStreamConfigs

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

InputDescriptor

public InputDescriptor(java.lang.String streamId,
                       Serde serde,
                       SystemDescriptor systemDescriptor,
                       InputTransformer transformer)

Constructs an InputDescriptor instance.

Parameters:

streamId - id of the stream

serde - serde for messages in the stream

systemDescriptor - system descriptor this stream descriptor was obtained from

transformer - stream level input stream transform function if available, else null

Method Detail

shouldResetOffset

public SubClass shouldResetOffset()

If set, when a Samza container starts up, it ignores any checkpointed offset for this particular input stream. Its behavior is thus determined by the withOffsetDefault(org.apache.samza.system.SystemStreamMetadata.OffsetType) setting. Note that the reset takes effect every time a container is started, which may be every time you restart your job, or more frequently if a container fails and is restarted by the framework.

Returns:

this input descriptor

withOffsetDefault

public SubClass withOffsetDefault(SystemStreamMetadata.OffsetType offsetDefault)

If a container starts up without a checkpoint, this property determines where in the input stream we should start consuming. The value must be an OffsetType, one of the following:

• upcoming: Start processing messages that are published after the job starts. Any messages published while the job was not running are not processed.
• oldest: Start processing at the oldest available message in the system, and reprocess the entire available message history.

This property is for an individual stream. To set it for all streams within a system, see SystemDescriptor.withDefaultStreamOffsetDefault(org.apache.samza.system.SystemStreamMetadata.OffsetType). If both are defined, the stream-level definition takes precedence.

Parameters:

offsetDefault - offset type to start processing from

Returns:

this input descriptor

withPriority

public SubClass withPriority(int priority)

If one or more streams have a priority set (any positive integer), they will be processed with higher priority than the other streams.

You can set several streams to the same priority, or define multiple priority levels by assigning a higher number to the higher-priority streams.

If a higher-priority stream has any messages available, they will always be processed first; messages from lower-priority streams are only processed when there are no new messages on higher-priority inputs.

Parameters:

priority - priority for this input stream

Returns:

this input descriptor

shouldBootstrap

public SubClass shouldBootstrap()

If set, this stream will be processed as a bootstrap stream. This means that every time a Samza container starts up, this stream will be fully consumed before messages from any other stream are processed.

Returns:

this input descriptor

isBounded

public SubClass isBounded()

If set, this stream will be considered a bounded stream. If all input streams in an application are bounded, the job is considered to be running in batch processing mode.

Returns:

this input descriptor

shouldDeleteCommittedMessages

public SubClass shouldDeleteCommittedMessages()

If set, and supported by the system implementation, messages older than the latest checkpointed offset for this stream may be deleted after the commit.

Returns:

this input descriptor

getTransformer

public java.util.Optional<InputTransformer> getTransformer()

toConfig

public java.util.Map<java.lang.String,java.lang.String> toConfig()

Overrides:

toConfig in class StreamDescriptor<StreamMessageType,SubClass extends InputDescriptor<StreamMessageType,SubClass>>