winton_kafka_streams package

Submodules

winton_kafka_streams.kafka_config module

Configuration values that may be set to control behaviour of Winton Kafka Streams

Configuration may either be set inline in your application using:

import kafka_config kafka_config.BOOTSTRAP_SERVERS = ‘localhost:9092’

or as a file in java properties format. The property names are identical to those used in the Java implementation for ease of sharing between both.

External files can be loaded using:

import kafka_config kafka_config.read_local_config(‘path/to/kafka.streams.config’)

winton_kafka_streams.kafka_config.APPLICATION_ID = 'wkstream.application.id'

The replication factor for changelog topics and repartition topics created by the application Default: 1 Importance: Low

winton_kafka_streams.kafka_config.APPLICATION_SERVER = ''

The maximum number of records to buffer per partition Default: 1000 Importance: Low

winton_kafka_streams.kafka_config.AUTO_OFFSET_RESET = 'earliest'

Consumer Default: false, see Consumer Auto Commit Importance: low

winton_kafka_streams.kafka_config.BOOTSTRAP_SERVERS = 'localhost:9092'

An identifier for the stream processing application. Must be unique within the Kafka cluster. It is used as:

  1. the default client-id prefix
  2. the group-id for membership management
  3. the changelog topic prefix.
winton_kafka_streams.kafka_config.BUFFERED_RECORDS_PER_PARTITION = 1000

An id string to pass to the server when making requests. (This setting is passed to the consumer/producer clients used internally by Winton Kafka Streams.) Default: “” Importance: Low

winton_kafka_streams.kafka_config.CACHE_MAX_BYTES_BUFFERING = 10485760

The number of standby replicas for each task Default: 0 Importance: Medium

winton_kafka_streams.kafka_config.CLIENT_ID = ''

The frequency with which to save the position (offsets in source topics) of tasks Default: 30000 (millisecs) Importance: Low

winton_kafka_streams.kafka_config.COMMIT_INTERVAL_MS = 30000

A list of classes to use as metrics reporters Default: [] Importance: Low

winton_kafka_streams.kafka_config.ENABLE_AUTO_COMMIT = 'false'

Consumer Default: Integer.MAX_VALUE Importance: low

winton_kafka_streams.kafka_config.LINGER_MS = 100

Producer Default: 10 Importance: low

winton_kafka_streams.kafka_config.MAX_POLL_INTERVAL_MS = 9223372036854775807

Consumer Default: 1000 Importance: low

winton_kafka_streams.kafka_config.MAX_POLL_RECORDS = 1000

Default serializer/deserializer class for record values, implements the Serde interface (see also key.serdes) Default: winton_kafka_streams.processor.serialization.serdes.BytesSerde Importance: Medium

winton_kafka_streams.kafka_config.METRICS_NUM_SAMPLES = 2

The highest recording level for metrics. Default: info Importance: Low

winton_kafka_streams.kafka_config.METRICS_RECORDING_LEVEL = 'info'

The window of time a metrics sample is computed over. Default: 30000 (millisecs) Importance: Low

winton_kafka_streams.kafka_config.METRICS_SAMPLE_WINDOW_MS = 30000

Partition grouper class that implements the PartitionGrouper interface Defatult: see Partition Grouper Importance: Low

winton_kafka_streams.kafka_config.METRIC_REPORTERS = []

The number of samples maintained to compute metrics. Default: 2 Importance: Low

winton_kafka_streams.kafka_config.NUM_STANDBY_REPLICAS = 0

The number of threads to execute stream processing Default: 1 Importance: Medium

winton_kafka_streams.kafka_config.NUM_STREAM_THREADS = 1

Timestamp extractor class that implements the TimestampExtractor interface Default: see Timestamp Extractor Importance: Medium

winton_kafka_streams.kafka_config.PARITION_GROUPER = None

The amount of time in milliseconds to block waiting for input Default: 100 (millisecs) Importance: Low

winton_kafka_streams.kafka_config.POLL_MS = 100

The amount of time in milliseconds to wait before deleting state when a partition has migrated Default: 60000 (millisecs) Importance: Low

winton_kafka_streams.kafka_config.REPLICATION_FACTOR = 1

Directory location for state stores Default: /var/lib/kafka-streams Importance: Low

winton_kafka_streams.kafka_config.RETRIES = 10

Consumer Default: earliest Importance: low

winton_kafka_streams.kafka_config.STATE_CLEANUP_DELAY_MS = 60000

Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default: 86400000 (millisecons) = 1 day Importance: Low

winton_kafka_streams.kafka_config.STATE_DIR = '/var/lib/kafka-streams'

Maximum number of memory bytes to be used for record caches across all threads Default: 10485760 (bytes) Importance: Medium

winton_kafka_streams.kafka_config.TIMESTAMP_EXTRACTOR = None

A host:port pair pointing to an embedded user defined endpoint that can be used for discovering the locations of state stores within a single Winton Kafka Streams application. The value of this must be different for each instance of the application. Default “” Importance: Low

winton_kafka_streams.kafka_config.VALUE_SERDE = 'winton_kafka_streams.processor.serialization.serdes.bytes_serde.BytesSerde'

Default serializer/deserializer class for record keys, implements the Serde interface (see also value.serdes) Default: winton_kafka_streams.processor.serialization.serdes.BytesSerde Importance: Medium

winton_kafka_streams.kafka_config.WINDOWSTORE_CHANGELOG_ADDITIONAL_RETENTION_MS = 86000000

linger.ms (low) Producer Default: 100 Importance: low

winton_kafka_streams.kafka_config.log = <Logger winton_kafka_streams.kafka_config (WARNING)>

A list of host/port pairs to use for establishing the initial connection to the Kafka cluster

winton_kafka_streams.kafka_config.read_local_config(config_file)

winton_kafka_streams.kafka_stream module

Module contents