org.apache.samza.table.descriptors

Class RemoteTableDescriptor<K,V,U>

java.lang.Object

org.apache.samza.table.descriptors.BaseTableDescriptor<K,V,RemoteTableDescriptor<K,V,U>>

org.apache.samza.table.descriptors.RemoteTableDescriptor<K,V,U>

Type Parameters:

K - the type of the key

V - the type of the value

U - the type of the update

All Implemented Interfaces:

TableDescriptor<K,V,RemoteTableDescriptor<K,V,U>>

public class RemoteTableDescriptor<K,V,U>
extends BaseTableDescriptor<K,V,RemoteTableDescriptor<K,V,U>>

Table descriptor for remote store backed tables

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	ASYNC_CALLBACK_POOL_SIZE
static java.lang.String	BATCH_PROVIDER
static java.lang.String	DEFAULT_RATE_LIMITER_CLASS_NAME
static java.lang.String	PROVIDER_FACTORY_CLASS_NAME
static java.lang.String	RATE_LIMITER
static java.lang.String	READ_CREDIT_FN
static java.lang.String	READ_CREDITS
static java.lang.String	READ_FN
static java.lang.String	READ_RETRY_POLICY
static java.lang.String	RL_READ_TAG Tag to be used for provision credits for rate limiting read operations from the remote table.
static java.lang.String	RL_WRITE_TAG Tag to be used for provision credits for rate limiting write operations into the remote table.
static java.lang.String	WRITE_CREDIT_FN
static java.lang.String	WRITE_CREDITS
static java.lang.String	WRITE_FN
static java.lang.String	WRITE_RETRY_POLICY

Fields inherited from class org.apache.samza.table.descriptors.BaseTableDescriptor

config, tableId

Constructor Summary

Constructors

Constructor and Description
RemoteTableDescriptor(java.lang.String tableId) Constructs a table descriptor instance

Method Summary

Modifier and Type	Method and Description
protected void	addTablePartConfig(java.lang.String tablePartKey, TablePart tablePart, Config jobConfig, java.util.Map<java.lang.String,java.lang.String> tableConfig) Helper method to add table part config items to table configuration
java.lang.String	getProviderFactoryClassName() Return the fully qualified class name of the TableProviderFactory
java.util.Map<java.lang.String,java.lang.String>	toConfig(Config jobConfig) Generate configuration for this table descriptor, the generated configuration should be the complete configuration for this table that can be directly included in the job configuration.
protected void	validate() Validate that this table descriptor is constructed properly; this method is used internally.
RemoteTableDescriptor<K,V,U>	withAsyncCallbackExecutorPoolSize(int poolSize) Specify the size of the thread pool for the executor used to execute callbacks of CompletableFutures of async Table operations.
RemoteTableDescriptor<K,V,U>	withBatchProvider(BatchProvider<K,V,U> batchProvider) Specifies a batch provider inorder to batch Table operations.
RemoteTableDescriptor<K,V,U>	withRateLimiter(RateLimiter rateLimiter, TableRateLimiter.CreditFunction<K,V> readCreditFn, TableRateLimiter.CreditFunction<K,V> writeCreditFn) Specify a rate limiter along with credit functions to map a table record (as KV) to the amount of credits to be charged from the rate limiter for table read and write operations.
RemoteTableDescriptor<K,V,U>	withRateLimiterDisabled() Disable both read and write rate limiter.
RemoteTableDescriptor<K,V,U>	withReadFunction(TableReadFunction<K,V> readFn) Use specified TableReadFunction with remote table and a retry policy.
RemoteTableDescriptor<K,V,U>	withReadRateLimit(int creditsPerSec) Specify the rate limit for table read operations.
RemoteTableDescriptor<K,V,U>	withReadRateLimiterDisabled() Disable the read rate limiter.
RemoteTableDescriptor<K,V,U>	withReadRetryPolicy(TableRetryPolicy retryPolicy) Use specified TableRetryPolicy with the TableReadFunction.
RemoteTableDescriptor<K,V,U>	withWriteFunction(TableWriteFunction<K,V,U> writeFn) Use specified TableWriteFunction with remote table and a retry policy.
RemoteTableDescriptor<K,V,U>	withWriteRateLimit(int creditsPerSec) Specify the rate limit for table write operations.
RemoteTableDescriptor<K,V,U>	withWriteRateLimiterDisabled() Disable the write rate limiter.
RemoteTableDescriptor<K,V,U>	withWriteRetryPolicy(TableRetryPolicy retryPolicy) Use specified TableRetryPolicy with the TableWriteFunction.

Methods inherited from class org.apache.samza.table.descriptors.BaseTableDescriptor

addTableConfig, getTableId, withConfig

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PROVIDER_FACTORY_CLASS_NAME

public static final java.lang.String PROVIDER_FACTORY_CLASS_NAME

See Also:

Constant Field Values

DEFAULT_RATE_LIMITER_CLASS_NAME

public static final java.lang.String DEFAULT_RATE_LIMITER_CLASS_NAME

See Also:

Constant Field Values

RL_READ_TAG

public static final java.lang.String RL_READ_TAG

Tag to be used for provision credits for rate limiting read operations from the remote table. Caller must pre-populate the credits with this tag when specifying a custom rate limiter instance through withRateLimiter(RateLimiter, TableRateLimiter.CreditFunction, TableRateLimiter.CreditFunction)

See Also:

Constant Field Values

RL_WRITE_TAG

public static final java.lang.String RL_WRITE_TAG

Tag to be used for provision credits for rate limiting write operations into the remote table. Caller can optionally populate the credits with this tag when specifying a custom rate limiter instance through withRateLimiter(RateLimiter, TableRateLimiter.CreditFunction, TableRateLimiter.CreditFunction) and it needs the write functionality.

See Also:

Constant Field Values

READ_FN

public static final java.lang.String READ_FN

See Also:

Constant Field Values

WRITE_FN

public static final java.lang.String WRITE_FN

See Also:

Constant Field Values

RATE_LIMITER

public static final java.lang.String RATE_LIMITER

See Also:

Constant Field Values

READ_CREDITS

public static final java.lang.String READ_CREDITS

See Also:

Constant Field Values

WRITE_CREDITS

public static final java.lang.String WRITE_CREDITS

See Also:

Constant Field Values

READ_CREDIT_FN

public static final java.lang.String READ_CREDIT_FN

See Also:

Constant Field Values

WRITE_CREDIT_FN

public static final java.lang.String WRITE_CREDIT_FN

See Also:

Constant Field Values

ASYNC_CALLBACK_POOL_SIZE

public static final java.lang.String ASYNC_CALLBACK_POOL_SIZE

See Also:

Constant Field Values

READ_RETRY_POLICY

public static final java.lang.String READ_RETRY_POLICY

See Also:

Constant Field Values

WRITE_RETRY_POLICY

public static final java.lang.String WRITE_RETRY_POLICY

See Also:

Constant Field Values

BATCH_PROVIDER

public static final java.lang.String BATCH_PROVIDER

See Also:

Constant Field Values

Constructor Detail

RemoteTableDescriptor

public RemoteTableDescriptor(java.lang.String tableId)

Constructs a table descriptor instance

Parameters:

tableId - Id of the table, it must conform to pattern [\\d\\w-_]+

Method Detail

withReadFunction

public RemoteTableDescriptor<K,V,U> withReadFunction(TableReadFunction<K,V> readFn)

Use specified TableReadFunction with remote table and a retry policy.

Parameters:

readFn - read function instance

Returns:

this table descriptor instance

withWriteFunction

public RemoteTableDescriptor<K,V,U> withWriteFunction(TableWriteFunction<K,V,U> writeFn)

Use specified TableWriteFunction with remote table and a retry policy.

Parameters:

writeFn - write function instance

Returns:

this table descriptor instance

withReadRetryPolicy

public RemoteTableDescriptor<K,V,U> withReadRetryPolicy(TableRetryPolicy retryPolicy)

Use specified TableRetryPolicy with the TableReadFunction.

Parameters:

retryPolicy - retry policy for the write function

Returns:

this table descriptor instance

withWriteRetryPolicy

public RemoteTableDescriptor<K,V,U> withWriteRetryPolicy(TableRetryPolicy retryPolicy)

Use specified TableRetryPolicy with the TableWriteFunction.

Parameters:

retryPolicy - retry policy for the write function

Returns:

this table descriptor instance

withRateLimiter

public RemoteTableDescriptor<K,V,U> withRateLimiter(RateLimiter rateLimiter,
                                                    TableRateLimiter.CreditFunction<K,V> readCreditFn,
                                                    TableRateLimiter.CreditFunction<K,V> writeCreditFn)

Specify a rate limiter along with credit functions to map a table record (as KV) to the amount of credits to be charged from the rate limiter for table read and write operations. This is an advanced API that provides greater flexibility to throttle each record in the table with different number of credits. For most common use-cases eg: limit the number of read/write operations, please instead use the withReadRateLimit(int) and withWriteRateLimit(int).

Parameters:

rateLimiter - rate limiter instance to be used for throttling

readCreditFn - credit function for rate limiting read operations

writeCreditFn - credit function for rate limiting write operations

Returns:

this table descriptor instance

withRateLimiterDisabled

public RemoteTableDescriptor<K,V,U> withRateLimiterDisabled()

Disable both read and write rate limiter. If the read rate limiter is enabled, the user must provide a rate limiter by calling withRateLimiter(RateLimiter, TableRateLimiter.CreditFunction, TableRateLimiter.CreditFunction) or withReadRateLimit(int). If the write rate limiter is enabled, the user must provide a rate limiter by calling withRateLimiter(RateLimiter, TableRateLimiter.CreditFunction, TableRateLimiter.CreditFunction) or withWriteRateLimit(int). By default, both read and write rate limiters are enabled.

Returns:

this table descriptor instance.

withReadRateLimiterDisabled

public RemoteTableDescriptor<K,V,U> withReadRateLimiterDisabled()

Disable the read rate limiter.

Returns:

this table descriptor instance.

withWriteRateLimiterDisabled

public RemoteTableDescriptor<K,V,U> withWriteRateLimiterDisabled()

Disable the write rate limiter.

Returns:

this table descriptor instance.

withReadRateLimit

public RemoteTableDescriptor<K,V,U> withReadRateLimit(int creditsPerSec)

Specify the rate limit for table read operations. If the read rate limit is set with this method it is invalid to call withRateLimiter(RateLimiter, TableRateLimiter.CreditFunction, TableRateLimiter.CreditFunction) and vice versa. Note that this is the total credit of rate limit for the entire job, each task will get a per task credit of creditsPerSec/tasksCount. Hence creditsPerSec should be greater than total number of tasks.

Parameters:

creditsPerSec - rate limit for read operations; must be positive and greater than total number tasks

Returns:

this table descriptor instance

withWriteRateLimit

public RemoteTableDescriptor<K,V,U> withWriteRateLimit(int creditsPerSec)

Specify the rate limit for table write operations. If the write rate limit is set with this method it is invalid to call withRateLimiter(RateLimiter, TableRateLimiter.CreditFunction, TableRateLimiter.CreditFunction) and vice versa. Note that this is the total credit of rate limit for the entire job, each task will get a per task credit of creditsPerSec/tasksCount. Hence creditsPerSec should be greater than total number of tasks.

Parameters:

creditsPerSec - rate limit for write operations; must be positive and greater than total number tasks

Returns:

this table descriptor instance

withAsyncCallbackExecutorPoolSize

public RemoteTableDescriptor<K,V,U> withAsyncCallbackExecutorPoolSize(int poolSize)

Specify the size of the thread pool for the executor used to execute callbacks of CompletableFutures of async Table operations. By default, these futures are completed (called) by the threads of the native store client. Depending on the implementation of the native client, it may or may not allow executing long running operations in the callbacks. This config can be used to execute the callbacks from a separate executor to decouple from the native client. If configured, this thread pool is shared by all read and write operations.

Parameters:

poolSize - max number of threads in the executor for async callbacks

Returns:

this table descriptor instance

withBatchProvider

public RemoteTableDescriptor<K,V,U> withBatchProvider(BatchProvider<K,V,U> batchProvider)

Specifies a batch provider inorder to batch Table operations.

getProviderFactoryClassName

public java.lang.String getProviderFactoryClassName()

Description copied from class: BaseTableDescriptor

Return the fully qualified class name of the TableProviderFactory

Specified by:

getProviderFactoryClassName in class BaseTableDescriptor<K,V,RemoteTableDescriptor<K,V,U>>

Returns:

class name of the TableProviderFactory

toConfig

public java.util.Map<java.lang.String,java.lang.String> toConfig(Config jobConfig)

Description copied from interface: TableDescriptor

Generate configuration for this table descriptor, the generated configuration should be the complete configuration for this table that can be directly included in the job configuration. Note: although the serdes may have already been set in this instance, their corresponding configuration needs to be generated centrally for consistency and efficiency reasons. Therefore the serde configuration for this table is expected to have already been generated and stored in the jobConfig.

Specified by:

toConfig in interface TableDescriptor<K,V,RemoteTableDescriptor<K,V,U>>

Overrides:

toConfig in class BaseTableDescriptor<K,V,RemoteTableDescriptor<K,V,U>>

Parameters:

jobConfig - job configuration

Returns:

table configuration

validate

protected void validate()

Description copied from class: BaseTableDescriptor

Validate that this table descriptor is constructed properly; this method is used internally.

Specified by:

validate in class BaseTableDescriptor<K,V,RemoteTableDescriptor<K,V,U>>

addTablePartConfig

protected void addTablePartConfig(java.lang.String tablePartKey,
                                  TablePart tablePart,
                                  Config jobConfig,
                                  java.util.Map<java.lang.String,java.lang.String> tableConfig)

Helper method to add table part config items to table configuration

Parameters:

tablePartKey - key of the table part

tablePart - table part

jobConfig - job configuration

tableConfig - table configuration