APIs.md
July 25, 2026 · View on GitHub
Packages:
numaflow.numaproj.io/v1alpha1
Resource Types:
AWSAssumeRole
(Appears on: SqsSink, SqsSource)
AWSAssumeRole contains the configuration for AWS STS assume role authentication This can be used with any AWS service (SQS, S3, DynamoDB, etc.)
|
Field |
Description |
|---|---|
|
|
RoleARN is the Amazon Resource Name (ARN) of the role to assume. This is a required field when assume role is enabled. Example: “arn:aws:iam::123456789012:role/CrossAccount-Service-Role” |
|
|
(Optional)
SessionName is an identifier for the assumed role session. This appears in AWS CloudTrail logs to help identify the source of API calls. If not specified, a default session name will be generated based on the service context. |
|
|
(Optional)
DurationSeconds is the duration (in seconds) of the role session. Valid values: 900-43200 (15 minutes to 12 hours) Defaults to 3600 (1 hour) if not specified. The actual session duration is constrained by the maximum session duration setting of the IAM role being assumed. |
|
|
(Optional)
ExternalID is a unique identifier that might be required when you assume a role in another account. This is commonly used as an additional security measure for cross-account role access. |
|
|
(Optional)
Policy is an IAM policy document (JSON string) that you want to use as an inline session policy. This parameter is optional. When specified, the session permissions are the intersection of the IAM role’s identity-based policy and the session policies. This allows further restriction of permissions for the specific service operations. |
|
|
(Optional)
PolicyARNs is a list of Amazon Resource Names (ARNs) of IAM managed policies that you want to use as managed session policies. The policies must exist in the same account as the role. This allows attaching existing managed policies to further restrict session permissions. |
AbstractPodTemplate
(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, MonoVertexSpec, ServingSpec, SideInputsManagerTemplate, VertexTemplate)
AbstractPodTemplate provides a template for pod customization in vertices, daemon deployments and so on.
|
Field |
Description |
|---|---|
|
|
(Optional)
Metadata sets the pods’s metadata, i.e. annotations and labels |
|
|
(Optional)
NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ |
|
|
(Optional)
If specified, the pod’s tolerations. |
|
|
(Optional)
SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field. |
|
|
(Optional)
ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod |
|
|
(Optional)
If specified, indicates the pod’s priority. “system-node-critical” and “system-cluster-critical” are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ |
|
|
(Optional)
The priority value. Various system components use this field to find the priority of the pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ |
|
|
(Optional)
The pod’s scheduling constraints More info: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ |
|
|
(Optional)
ServiceAccountName applied to the pod |
|
|
(Optional)
RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the “legacy” RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class |
|
|
(Optional)
AutomountServiceAccountToken indicates whether a service account token should be automatically mounted. |
|
|
(Optional)
Set DNS policy for the pod. Defaults to “ClusterFirst”. Valid values are ‘ClusterFirstWithHostNet’, ‘ClusterFirst’, ‘Default’ or ‘None’. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to ‘ClusterFirstWithHostNet’. |
|
|
(Optional)
Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. |
|
|
(Optional)
ResourceClaims defines which ResourceClaims must be allocated and reserved before the Pod is allowed to start. The resources will be made available to those containers which consume them by name. |
AbstractSink
(Appears on: Sink)
|
Field |
Description |
|---|---|
|
|
(Optional)
Log sink is used to write the data to the log. |
|
|
(Optional)
Kafka sink is used to write the data to the Kafka. |
|
|
(Optional)
Blackhole sink is used to write the data to the blackhole sink, which is a sink that discards all the data written to it. |
|
|
(Optional)
UDSink sink is used to write the data to the user-defined sink. |
|
|
(Optional)
Serve sink is used to return results when working with a ServingPipeline. |
|
|
(Optional)
SQS sink is used to write the data to the AWS SQS. |
|
|
(Optional)
Pulsar sink is used to write the data to the Apache Pulsar. |
AbstractVertex
(Appears on: PipelineSpec, VertexSpec)
|
Field |
Description |
|---|---|
|
| |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Container template for the main numa container. |
|
|
(Optional)
Container template for all the vertex pod init containers spawned by numaflow, excluding the ones specified by the user. |
|
|
(Members of (Optional) |
|
|
(Optional) |
|
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, will override pipeline level settings |
|
|
(Optional)
Settings for autoscaling |
|
|
(Optional)
List of customized init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ |
|
|
(Optional)
List of customized sidecar containers belonging to the pod. |
|
|
(Optional)
Number of partitions of the vertex owned buffers. It applies to udf and sink vertices only. |
|
|
(Optional)
Names of the side inputs used in this vertex. |
|
|
(Optional)
Container template for the side inputs watcher container. |
|
|
(Optional)
The strategy to use to replace existing pods with new ones. |
|
|
(Optional)
Ordered enables ordered processing for this vertex, overrides pipeline-level setting. When enabled, messages will be processed in order based on their event time. Note: Reduce vertices ignore this setting as they are already partitioned. |
AccumulatorWindow
(Appears on: Window)
AccumulatorWindow describes a special kind of SessionWindow (similar to Global Window) where output should always have monotonically increasing WM but it can be manipulated through event-time by reordering the messages. NOTE: Quite powerful, should not be abused; it can cause stalling of pipelines and leaks.
|
Field |
Description |
|---|---|
|
|
Timeout is the duration of inactivity after which the state of the accumulator is removed. |
Authorization
(Appears on: HTTPSource, ServingSpec)
|
Field |
Description |
|---|---|
|
(Optional)
A secret selector which contains bearer token To use this, the client
needs to add “Authorization: Bearer |
Backoff
(Appears on: RetryStrategy)
Backoff defines parameters used to systematically configure the retry strategy.
|
Field |
Description |
|---|---|
|
|
(Optional)
Interval sets the initial retry duration, after a failure occurs. |
|
|
(Optional)
Steps defines the maximum number of retry attempts |
|
|
(Optional)
Interval is multiplied by factor each iteration, if factor is not zero and the limits imposed by Steps and Cap have not been reached. |
|
(Optional)
A limit on revised values of the interval parameter. If a multiplication by the factor parameter would make the interval exceed the cap then the interval is set to the cap and the steps parameter is set to zero. | |
|
|
(Optional)
The sleep at each iteration is the interval plus an additional amount
chosen uniformly at random from the interval between zero and
|
BasicAuth
(Appears on: NatsAuth)
BasicAuth represents the basic authentication approach which contains a user name and a password.
|
Field |
Description |
|---|---|
|
(Optional)
Secret for auth user | |
|
|
(Optional)
Secret for auth password |
Blackhole
(Appears on: AbstractSink)
Blackhole is a sink to emulate /dev/null
BufferFullWritingStrategy (string alias)
(Appears on: Edge)
BufferServiceConfig
(Appears on: GetServingPipelineResourceReq, InterStepBufferServiceStatus)
|
Field |
Description |
|---|---|
|
|
CombinedEdge
(Appears on: VertexSpec)
CombinedEdge is a combination of Edge and some other properties such as vertex type, partitions, limits. It’s used to decorate the fromEdges and toEdges of the generated Vertex objects, so that in the vertex pod, it knows the properties of the connected vertices, for example, how many partitioned buffers I should write to, what is the write buffer length, etc.
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
From vertex type. |
|
|
(Optional)
The number of partitions of the from vertex, if not provided, the default value is set to “1”. |
|
|
(Optional) |
|
|
To vertex type. |
|
|
(Optional)
The number of partitions of the to vertex, if not provided, the default value is set to “1”. |
|
|
(Optional) |
|
|
(Optional)
Determine whether ordered processing is enabled for |
Compression
(Appears on: InterStepBuffer)
Compression is the compression settings for the messages in the InterStepBuffer
|
Field |
Description |
|---|---|
|
|
(Optional)
Type is the type of compression to be used |
CompressionType (string alias)
(Appears on: Compression)
CompressionType is a string enumeration type that enumerates all possible compression types.
ConditionType (string alias)
ConditionType is a valid value of Condition.Type
Container
(Appears on: ServingStore, SideInput, UDF, UDSink, UDSource, UDTransformer)
Container is used to define the container properties for user-defined functions, sinks, etc.
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
(Optional) | |
|
(Optional) | |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
(Optional) |
ContainerTemplate
(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, MonoVertexSpec, ServingSpec, SideInputsManagerTemplate, VertexTemplate)
ContainerTemplate defines customized spec for a container
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
(Optional) | |
|
(Optional) | |
|
|
(Optional) |
|
|
(Optional) |
CronSchedule
(Appears on: CronScheduling)
|
Field |
Description |
|---|---|
|
|
Start of the cron window. Extended cron format (Second Minute Hour Dom Month Dow). |
|
|
End of the cron window. Same format as Start. Must differ from Start. |
|
|
Minimum replicas during this window. Overrides scale.min. |
|
|
Maximum replicas during this window. Overrides scale.max. |
CronScheduling
(Appears on: Scale)
CronScheduling defines cron-based autoscaling overrides.
|
Field |
Description |
|---|---|
|
|
(Optional)
Timezone for interpreting cron expressions. IANA Time Zone Database format. Defaults to UTC. |
|
|
Schedules are evaluated in order; the first active schedule takes precedence when windows overlap. |
DaemonTemplate
(Appears on: MonoVertexSpec, Templates)
|
Field |
Description |
|---|---|
|
|
(Members of (Optional) |
|
|
(Optional)
Replicas is the number of desired replicas of the Deployment. This is a pointer to distinguish between explicit zero and unspecified. Defaults to 1. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#what-is-a-replicationcontroller |
|
|
(Optional) |
|
|
(Optional) |
Edge
(Appears on: CombinedEdge, PipelineSpec)
|
Field |
Description |
|---|---|
|
| |
|
| |
|
|
(Optional)
Conditional forwarding, only allowed when “From” is a Source or UDF. |
|
|
(Optional)
OnFull specifies the behaviour for the write actions when the inter step buffer is full. There are currently two options, retryUntilSuccess and discardLatest. if not provided, the default value is set to “retryUntilSuccess” |
FixedWindow
(Appears on: Window)
FixedWindow describes a fixed window
|
Field |
Description |
|---|---|
|
|
Length is the duration of the fixed window. |
|
|
(Optional)
Streaming should be set to true if the reduce udf is streaming. |
ForwardConditions
(Appears on: Edge, MonoVertexBypassCondition)
|
Field |
Description |
|---|---|
|
|
Tags used to specify tags for conditional forwarding |
GSSAPI
(Appears on: SASL)
GSSAPI represents a SASL GSSAPI config
|
Field |
Description |
|---|---|
|
| |
|
| |
|
|
UsernameSecret refers to the secret that contains the username |
|
|
valid inputs - KRB5_USER_AUTH, KRB5_KEYTAB_AUTH |
|
|
(Optional)
PasswordSecret refers to the secret that contains the password |
|
|
(Optional)
KeytabSecret refers to the secret that contains the keytab |
|
|
(Optional)
KerberosConfigSecret refers to the secret that contains the kerberos config |
GeneratorSource
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Size of each generated message |
|
|
(Optional)
KeyCount is the number of unique keys in the payload |
|
|
(Optional)
Value is an optional uint64 value to be written in to the payload |
|
|
(Optional)
Jitter is the jitter for the message generation, used to simulate out of order messages for example if the jitter is 10s, then the message’s event time will be delayed by a random time between 0 and 10s which will result in the message being out of order by 0 to 10s |
|
|
(Optional)
ValueBlob is an optional string which is the base64 encoding of direct payload to send. This is useful for attaching a GeneratorSource to a true pipeline to test load behavior with true messages without requiring additional work to generate messages through the external source if present, the Value and MsgSize fields will be ignored. |
GetDaemonDeploymentReq
|
Field |
Description |
|---|---|
|
| |
|
| |
|
| |
|
|
GetJetStreamServiceSpecReq
|
Field |
Description |
|---|---|
|
| |
|
| |
|
| |
|
| |
|
|
GetJetStreamStatefulSetSpecReq
|
Field |
Description |
|---|---|
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
GetMonoVertexDaemonDeploymentReq
|
Field |
Description |
|---|---|
|
| |
|
| |
|
|
GetMonoVertexPodSpecReq
|
Field |
Description |
|---|---|
|
| |
|
| |
|
|
GetServingPipelineResourceReq
|
Field |
Description |
|---|---|
|
| |
|
| |
|
| |
|
|
GetSideInputDeploymentReq
|
Field |
Description |
|---|---|
|
| |
|
| |
|
| |
|
|
GetVertexPodSpecReq
|
Field |
Description |
|---|---|
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
GroupBy
(Appears on: UDF)
GroupBy indicates it is a reducer UDF
|
Field |
Description |
|---|---|
|
|
Window describes the windowing strategy. |
|
|
(Optional) |
|
|
(Optional)
AllowedLateness allows late data to be included for the Reduce operation as long as the late data is not later than (Watermark - AllowedLateness). |
|
|
Storage is used to define the PBQ storage for a reduce vertex. |
HTTPSource
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional)
Whether to create a ClusterIP Service |
|
|
(Optional)
Ports to listen on. HTTPS always runs on port 8443 by default. To enable plain HTTP, set ports.http explicitly. |
ISBSvcPhase (string alias)
(Appears on: InterStepBufferServiceStatus)
ISBSvcType (string alias)
(Appears on: GetDaemonDeploymentReq, GetSideInputDeploymentReq, GetVertexPodSpecReq, InterStepBufferServiceStatus)
IdleSource
(Appears on: Watermark)
|
Field |
Description |
|---|---|
|
|
Threshold is the duration after which a source is marked as Idle due to
lack of data. Ex: If watermark found to be idle after the Threshold
duration then the watermark is progressed by |
|
|
(Optional)
StepInterval is the duration between the subsequent increment of the
watermark as long the source remains Idle. The default value is 0s which
means that once we detect idle source, we will be incrementing the
watermark by |
|
|
IncrementBy is the duration to be added to the current watermark to progress the watermark when source is idling. |
|
|
(Optional)
InitSourceDelay is the duration after which, if source doesn’t produce any data, the watermark is initialized with the current wall clock time. |
InterStepBuffer
(Appears on: PipelineSpec, VertexSpec)
InterStepBuffer configuration specifically for the pipeline.
|
Field |
Description |
|---|---|
|
|
(Optional)
Compression is the compression settings for the InterStepBufferService |
InterStepBufferService
|
Field |
Description | ||
|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
| ||
|
| |||
|
|
(Optional) |
InterStepBufferServiceSpec
(Appears on: InterStepBufferService)
|
Field |
Description |
|---|---|
|
|
InterStepBufferServiceStatus
(Appears on: InterStepBufferService)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
| |
|
| |
|
| |
|
| |
|
|
JetStreamBufferService
(Appears on: InterStepBufferServiceSpec)
|
Field |
Description |
|---|---|
|
|
JetStream version, such as “2.7.1” |
|
|
JetStream StatefulSet size |
|
|
(Optional)
ContainerTemplate contains customized spec for NATS container |
|
|
(Optional)
ReloaderContainerTemplate contains customized spec for config reloader container |
|
|
(Optional)
MetricsContainerTemplate contains customized spec for metrics container |
|
|
(Optional) |
|
|
(Members of (Optional) |
|
|
(Optional)
Nats/JetStream configuration, if not specified, global settings in numaflow-controller-config will be used. See https://docs.nats.io/running-a-nats-service/configuration#limits and https://docs.nats.io/running-a-nats-service/configuration#jetstream. For limits, only “max_payload” is supported for configuration, defaults to 1048576 (1MB), not recommended to use values over 8388608 (8MB) but max_payload can be set up to 67108864 (64MB). For jetstream, only “max_memory_store” and “max_file_store” are supported for configuration, do not set “store_dir” as it has been hardcoded. |
|
|
(Optional)
Optional arguments to start nats-server. For example, “-D” to enable debugging output, “-DV” to enable debugging and tracing. Check https://docs.nats.io/ for all the available arguments. |
|
|
(Optional)
Optional configuration for the streams, consumers and buckets to be created in this JetStream service, if specified, it will be merged with the default configuration in numaflow-controller-config. It accepts a YAML format configuration, it may include 4 sections, “stream”, “consumer”, “otBucket” and “procBucket”. Available fields under “stream” include “retention” (e.g. interest, limits, workerQueue), “maxMsgs”, “maxAge” (e.g. 72h), “replicas” (1, 3, 5), “duplicates” (e.g. 5m). Available fields under “consumer” include “ackWait” (e.g. 60s) Available fields under “otBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). Available fields under “procBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). |
|
|
(Optional)
Whether encrypt the data at rest, defaults to false Enabling encryption might impact the performance, see https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/encryption_at_rest for the detail Toggling the value will impact encrypting/decrypting existing messages. |
|
|
(Optional)
Whether enable TLS, defaults to false Enabling TLS might impact the performance |
JetStreamConfig
(Appears on: BufferServiceConfig)
|
Field |
Description |
|---|---|
|
|
JetStream (NATS) URL |
|
| |
|
|
(Optional) |
|
|
TLS enabled or not |
JetStreamSource
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
URL to connect to NATS cluster, multiple urls could be separated by comma. |
|
|
Stream represents the name of the stream. |
|
|
(Optional)
Consumer represents the name of the consumer of the stream If not
specified, a consumer with name
|
|
|
(Optional)
The point in the stream from which to receive messages. https://docs.nats.io/nats-concepts/jetstream/consumers#deliverpolicy Valid options are: “all”, “new”, “last”, “last_per_subject”, “by_start_sequence 42”, “by_start_time 1753428483000”. The second value to “by_start_time” is unix epoch time in milliseconds. |
|
|
(Optional)
A set of subjects that overlap with the subjects bound to the stream to filter delivery to subscribers. https://docs.nats.io/nats-concepts/jetstream/consumers#filtesubjects |
|
|
(Optional)
TLS configuration for the nats client. |
|
|
(Optional)
Auth information |
JobTemplate
(Appears on: Templates)
|
Field |
Description |
|---|---|
|
|
(Members of (Optional) |
|
|
(Optional) |
|
|
(Optional)
ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won’t be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes. Numaflow defaults to 30 |
|
|
(Optional)
Specifies the number of retries before marking this job failed. More info: https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy Numaflow defaults to 20 |
KRB5AuthType (string alias)
(Appears on: GSSAPI)
KRB5AuthType describes the kerberos auth type
KafkaSink
(Appears on: AbstractSink)
|
Field |
Description |
|---|---|
|
| |
|
| |
|
|
(Optional)
SetKey sets the Kafka key to the keys passed in the Message. When the key is null (default), the record is sent randomly to one of the available partitions of the topic. If a key exists, Kafka hashes the key, and the result is used to map the message to a specific partition. This ensures that messages with the same key end up in the same partition. |
|
|
(Optional)
TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. |
|
|
(Optional) |
|
|
(Optional)
SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. |
KafkaSource
(Appears on: Source)
|
Field |
Description |
|---|---|
|
| |
|
| |
|
| |
|
|
(Optional)
TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. |
|
|
(Optional) |
|
|
(Optional)
SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. |
|
|
Lifecycle
(Appears on: PipelineSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
DeletionGracePeriodSeconds used to delete pipeline gracefully |
|
|
(Optional)
DesiredPhase used to bring the pipeline from current phase to desired phase |
|
|
(Optional)
PauseGracePeriodSeconds used to pause pipeline gracefully |
|
|
(Optional)
DeleteGracePeriodSeconds used to delete pipeline gracefully Deprecated: Use DeletionGracePeriodSeconds instead |
Log
(Appears on: AbstractSink)
LogicOperator (string alias)
(Appears on: TagConditions)
Metadata
(Appears on: AbstractPodTemplate)
|
Field |
Description |
|---|---|
|
| |
|
|
MonoVertex
|
Field |
Description | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
| ||||||||||||||||||||||||||||||||
|
|
| ||||||||||||||||||||||||||||||||
|
|
(Optional) |
MonoVertexBypassCondition
(Appears on: MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
Conditions to trigger bypass to primary sink component. Definition is similar to “conditions” in pipeline spec for edges. |
|
|
(Optional)
Conditions to trigger bypass to fallback sink component. Definition is similar to “conditions” in pipeline spec for edges. |
|
|
(Optional)
Conditions to trigger bypass to onSuccess sink component. Definition is similar to “conditions” in pipeline spec for edges. |
MonoVertexLifecycle
(Appears on: MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
DesiredPhase used to bring the MonoVertex from current phase to desired phase |
MonoVertexLimits
(Appears on: MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
Read batch size from the source. ReadBatchSize controls only how many
messages are fetched in a single read call from the source; it is not a
cap on how many messages may be in-flight (use |
|
|
(Optional)
ReadTimeout is the read timeout duration from the source. |
|
|
(Optional)
RateLimit for MonoVertex defines how many messages can be read from
Source. This is computed by number of |
|
|
(Optional)
Concurrency defines the maximum number of messages that can be actively
in-flight (read but not yet acknowledged) at any given time. With
read-ahead enabled, the data plane keeps reading new batches from the
source until the number of in-flight messages reaches
|
MonoVertexPhase (string alias)
(Appears on: MonoVertexLifecycle, MonoVertexStatus)
MonoVertexSpec
(Appears on: MonoVertex)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
| |
|
| |
|
|
(Optional) |
|
|
(Members of (Optional) |
|
|
(Optional)
Container template for the main numa container. |
|
|
(Optional) |
|
|
(Optional)
Limits define the limitations such as read batch size for the mono vertex. |
|
|
(Optional)
Settings for autoscaling |
|
|
(Optional)
List of customized init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ |
|
|
(Optional)
List of customized sidecar containers belonging to the pod. |
|
|
(Optional)
Template for the daemon service deployment. |
|
|
(Optional)
The strategy to use to replace existing pods with new ones. |
|
|
(Optional)
Lifecycle defines the Lifecycle properties of a MonoVertex |
|
|
(Optional)
Bypass defines the bypass destination and conditions to trigger bypass for the mono vertex components. If specified, the bypass will be triggered if the conditions are met at any of the components. The first level of the bypass spec specifies the destination to which the message will be forwarded to, and the next level specifies the conditions to trigger the said bypass. |
|
|
(Optional)
Streaming enables per-message, out-of-order source acknowledgement
instead of the default whole-batch barrier. When true, the source reads
continuously bounded by |
MonoVertexStatus
(Appears on: MonoVertex)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
(Optional) |
|
|
(Optional)
Total number of non-terminated pods targeted by this MonoVertex (their labels match the selector). |
|
|
(Optional)
The number of desired replicas. |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Time of last scaling operation. |
|
|
(Optional)
The generation observed by the MonoVertex controller. |
|
|
(Optional)
The number of pods targeted by this MonoVertex with a Ready Condition. |
|
|
The number of Pods created by the controller from the MonoVertex version indicated by updateHash. |
|
|
The number of ready Pods created by the controller from the MonoVertex version indicated by updateHash. |
|
|
If not empty, indicates the current version of the MonoVertex used to generate Pods. |
|
|
If not empty, indicates the updated version of the MonoVertex used to generate Pods. |
NatsAuth
(Appears on: JetStreamConfig, JetStreamSource, NatsSource)
NatsAuth defines how to authenticate the nats access
|
Field |
Description |
|---|---|
|
|
(Optional)
Basic auth which contains a username and a password |
|
(Optional)
Token auth | |
|
(Optional)
NKey auth |
NatsSource
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
URL to connect to NATS cluster, multiple urls could be separated by comma. |
|
|
Subject holds the name of the subject onto which messages are published. |
|
|
Queue is used for queue subscription. |
|
|
(Optional)
TLS configuration for the nats client. |
|
|
(Optional)
Auth information |
NoStore
(Appears on: PBQStorage)
NoStore means there will be no persistence storage and there will be data loss during pod restarts. Use this option only if you do not care about correctness (e.g., approx statistics pipeline like sampling rate, etc.).
OnFailureRetryStrategy (string alias)
(Appears on: RetryStrategy)
Ordered
(Appears on: AbstractVertex, CombinedEdge, PipelineSpec)
Ordered defines the ordered processing configuration.
|
Field |
Description |
|---|---|
|
|
(Optional)
Enabled toggles ordered processing. |
PBQStorage
(Appears on: GroupBy)
PBQStorage defines the persistence configuration for a vertex.
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
(Optional) | |
|
|
(Optional) |
PersistenceStrategy
(Appears on: JetStreamBufferService, PBQStorage)
PersistenceStrategy defines the strategy of persistence
|
Field |
Description |
|---|---|
|
|
(Optional)
Name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1 |
|
(Optional)
Available access modes such as ReadWriteOncePod, ReadWriteOnce, ReadWriteMany https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes | |
|
|
Volume size, e.g. 50Gi |
Pipeline
|
Field |
Description | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
| ||||||||||||||||||||
|
|
| ||||||||||||||||||||
|
|
(Optional) |
PipelineLimits
(Appears on: PipelineSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
Read batch size for all the vertices in the pipeline, can be overridden
by the vertex’s limit settings. ReadBatchSize controls only how many
messages are fetched in a single read call from the source/buffer; it is
not a cap on how many messages may be in-flight (use
|
|
|
(Optional)
BufferMaxLength is used to define the max length of a buffer. Only applies to UDF and Source vertices as only they do buffer write. It can be overridden by the settings in vertex limits. |
|
|
(Optional)
BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. Only applies to UDF and Source vertices as only they do buffer write. It will be overridden by the settings in vertex limits. |
|
|
(Optional)
Read timeout for all the vertices in the pipeline, can be overridden by the vertex’s limit settings |
|
|
(Optional)
RateLimit is used to define the rate limit for all the vertices in the
pipeline, it could be overridden by the vertex’s limit settings. For
source vertices, it will be set to rate divided by readBatchSize because
for source vertices, the rate limit is defined by how many times the
|
|
|
(Optional)
Concurrency defines the maximum number of messages that can be actively
in-flight (read but not yet acknowledged) at any given time across each
vertex of the pipeline. With read-ahead enabled, the data plane keeps
reading new batches from the source/buffer until the number of in-flight
messages reaches |
PipelinePhase (string alias)
(Appears on: Lifecycle, PipelineStatus)
PipelineResumeStrategy (string alias)
PipelineSpec
(Appears on: GetVertexPodSpecReq, Pipeline, ServingPipelineSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
InterStepBufferServiceName is the name of the InterStepBufferService to be used by the pipeline |
|
| |
|
|
Edges define the relationships between vertices |
|
|
(Optional)
Lifecycle define the Lifecycle properties |
|
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, they could be overridden by each vertex’s settings |
|
|
(Optional)
Watermark enables watermark progression across the entire pipeline. |
|
|
(Optional)
Templates are used to customize additional kubernetes resources required for the Pipeline |
|
|
(Optional)
SideInputs defines the Side Inputs of a pipeline. |
|
|
(Optional)
InterStepBuffer configuration specific to this pipeline. |
|
|
(Optional)
Ordered enables order-preserving processing for the entire pipeline. When enabled, messages will be processed in their arrival order (FIFO within each partition). This can be overridden at the vertex level. |
PipelineStatus
(Appears on: Pipeline)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
The generation observed by the Pipeline controller. |
|
|
(Optional)
Field to indicate if a pipeline drain successfully occurred, only meaningful when the pipeline is paused. True means it has been successfully drained. |
Ports
(Appears on: HTTPSource, ServingSpec)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
Probe
(Appears on: Container, ContainerTemplate)
Probe is used to customize the configuration for Readiness and Liveness probes.
|
Field |
Description |
|---|---|
|
|
(Optional)
Number of seconds after the container has started before liveness probes are initiated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes |
|
|
(Optional)
Number of seconds after which the probe times out. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes |
|
|
(Optional)
How often (in seconds) to perform the probe. |
|
|
(Optional)
Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1. |
|
|
(Optional)
Minimum consecutive failures for the probe to be considered failed after having succeeded. Defaults to 3. Minimum value is 1. |
PulsarAuth
(Appears on: PulsarSink, PulsarSource)
PulsarAuth defines how to authenticate with Pulsar
|
Field |
Description |
|---|---|
|
(Optional)
JWT Token auth | |
|
|
(Optional)
Authentication using HTTP basic https://pulsar.apache.org/docs/4.0.x/security-basic-auth/ |
PulsarBasicAuth
(Appears on: PulsarAuth)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
PulsarDeadLetterPolicy
(Appears on: PulsarSource)
|
Field |
Description |
|---|---|
|
|
Topic where messages exceeding the max redelivery count will be sent. |
|
|
Maximum number of redelivery attempts before routing to the dead letter topic. |
PulsarSink
(Appears on: AbstractSink)
|
Field |
Description |
|---|---|
|
| |
|
| |
|
| |
|
|
(Optional)
Auth information |
PulsarSource
(Appears on: Source)
|
Field |
Description |
|---|---|
|
| |
|
| |
|
| |
|
| |
|
|
Maximum number of messages that are in not yet acked state. Once this limit is crossed, futher read requests will return empty list. |
|
|
(Optional)
Auth information |
|
|
(Optional)
Consumer level dead letter policy. |
RateLimit
(Appears on: MonoVertexLimits, PipelineLimits, VertexLimits)
|
Field |
Description |
|---|---|
|
|
Max is the maximum TPS that this vertex can process give a distributed
|
|
|
Minimum TPS allowed during initial bootup. This value will be
distributed across all the replicas if a distributed |
|
|
RampUpDuration is the duration to reach the maximum TPS from the minimum TPS. The min unit of ramp up is 1 in 1 second. |
|
|
(Optional)
Store is used to define the Distributed Store for the rate limiting. We also support in-memory store if no store is configured. This means that every replica will have its own rate limit and the actual TPS will be the sum of all the replicas. |
|
|
(Optional)
RateLimiterModes is used to define the modes for rate limiting. |
|
|
(Optional)
ResumedRampUp is used to enable the resume mode for rate limiting.
This, if true, will allow the processor to resume the ramp-up process from the last known state of the rate limiter, i.e., if the processor was allowed X tokens before shutting down, it will be allowed X tokens again after the processor restarts.
The resumed ramp-up process will be allowed until TTL time after the processor first deregisters with the rate limiter. |
|
(Optional)
TTL is used to define the duration after which a pod is considered stale and removed from the pool of pods if it doesn’t sync with the rate limiter.
Furthermore, if the ResumedRampUp is true, then TTL also defines the amount of time within which, if a pod re-registers / registers with the same name, with the rate limiter, it will be assigned the same rate limit as the previous pod with that name. |
RateLimiterGoBackN
(Appears on: RateLimiterModes)
RateLimiterGoBackN is for the GoBackN mode. Releases additional tokens only when previously released tokens have been utilized above the configured threshold otherwise triggers a ramp-down. Ramp-down is also triggered when the request is made after quite a while.
|
Field |
Description |
|---|---|
|
|
(Optional)
CoolDownPeriod is the duration after which the rate limiter will start ramping down if the request is made after the cool-down period. |
|
|
(Optional)
RampDownStrength is the strength of the ramp-down. It is a value between 0 and 1. 0 means no ramp-down and 1 means token pool is ramped down at the rate of slope=(max - min)/duration. |
|
|
(Optional)
ThresholdPercentage specifies the minimum percentage of capacity, availed by the rate limiter, that should be consumed at any instance to allow the rate limiter to unlock additional capacity. For example, given the following configuration: - max = 100 - min = 10 - rampUpDuration = 10s i.e.–> slope = 10 messages/second - thresholdPercentage = 50 at t = 0, the rate limiter will release 10 messages and at least 5 of those should be consumed to unlock additional capacity of 10 messages at t = 1 to make the total capacity of 20. |
RateLimiterInMemoryStore
(Appears on: RateLimiterStore)
RateLimiterModes
(Appears on: RateLimit)
RateLimiterModes defines the modes for rate limiting.
|
Field |
Description |
|---|---|
|
|
(Optional)
Irrespective of the traffic, the rate limiter releases max possible tokens based on ramp-up duration. |
|
|
(Optional)
If there is some traffic, then release the max possible tokens. |
|
|
(Optional)
Releases additional tokens only when previously released tokens have been utilized above the configured threshold |
|
|
(Optional)
Releases additional tokens only when previously released tokens have been utilized above the configured threshold otherwise triggers a ramp-down. Ramp-down is also triggered when the request is made after quite a while. |
RateLimiterOnlyIfUsed
(Appears on: RateLimiterModes)
RateLimiterOnlyIfUsed is for the OnlyIfUsed mode. Releases additional tokens only when previously released tokens have been utilized above the configured threshold
|
Field |
Description |
|---|---|
|
|
(Optional)
ThresholdPercentage specifies the minimum percentage of capacity, availed by the rate limiter, that should be consumed at any instance to allow the rate limiter to unlock additional capacity.
Defaults to 50%
For example, given the following configuration: - max = 100 - min = 10 - rampUpDuration = 10s i.e.–> slope = 10 messages/second - thresholdPercentage = 50 at t = 0, the rate limiter will release 10 messages and at least 5 of those should be consumed to unlock additional capacity of 10 messages at t = 1 to make the total capacity of 20. |
RateLimiterRedisStore
(Appears on: RateLimiterStore)
|
Field |
Description |
|---|---|
|
|
Choose how to connect to Redis. - Single: use a single URL (redis://… or rediss://…) - Sentinel: discover the node via Redis Sentinel |
|
|
(Optional)
SINGLE MODE: Full connection URL, e.g. redis://host:6379⁄0 or rediss://host:port/0 Mutually exclusive with .sentinel |
|
|
(Optional)
SENTINEL MODE: Settings to reach Sentinel and the selected Redis node Mutually exclusive with .url |
|
|
(Optional)
COMMON: Optional DB index (default 0) |
RateLimiterRelaxed
(Appears on: RateLimiterModes)
RateLimiterRelaxed is for the relaxed mode. It will release the max possible tokens if there is some traffic.
RateLimiterScheduled
(Appears on: RateLimiterModes)
RateLimiterScheduled is for the scheduled mode. It will release the max possible tokens based on ramp-up duration irrespective of traffic encountered.
RateLimiterStore
(Appears on: RateLimit)
|
Field |
Description |
|---|---|
|
|
(Optional)
RedisStore is used to define the redis store for the rate limit. |
|
|
(Optional)
InMemoryStore is used to define the in-memory store for the rate limit. |
RedisAuth
(Appears on: RedisSentinelConfig)
|
Field |
Description |
|---|---|
|
|
(Optional)
For Redis 6+ ACLs. If Username omitted, password-only is also supported. |
|
|
(Optional) |
RedisSentinelConfig
(Appears on: RateLimiterRedisStore)
|
Field |
Description |
|---|---|
|
|
Required Sentinel “service name” (aka master name) from sentinel.conf |
|
|
At least one Sentinel endpoint; 2–3 recommended. Use host:port pairs. Example: [“sentinel-0.redis.svc:26379”, “sentinel-1.redis.svc:26379”] |
|
|
(Optional)
Auth to talk to the Sentinel daemons (control-plane). Optional. |
|
|
(Optional)
Auth to talk to the Redis data nodes (data-plane). Optional. |
|
|
(Optional)
TLS for Sentinel connections (if your Sentinels expose TLS). |
|
|
(Optional)
TLS for Redis data nodes (redis). Often enabled even if Sentinel is plaintext. |
RetryStrategy
(Appears on: Sink)
The RetryStrategy struct defines the configuration for handling operation retries in case of failures. It incorporates an Exponential BackOff strategy to control retry timing and specifies the actions to take upon failure.
|
Field |
Description |
|---|---|
|
|
(Optional)
BackOff specifies the parameters for the exponential backoff strategy, controlling how delays between retries should increase. |
|
|
(Optional)
OnFailure specifies the action to take when the specified retry strategy fails. The possible values are: 1. “retry”: start another round of retrying the operation, 2. “fallback”: re-route the operation to a fallback sink and 3. “drop”: drop the operation and perform no further action. The default action is to retry. |
RollingUpdateStrategy
(Appears on: UpdateStrategy)
RollingUpdateStrategy is used to communicate parameter for RollingUpdateStrategyType.
|
Field |
Description |
|---|---|
|
|
(Optional)
The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding down. Defaults to 25%. Example: when this is set to 30%, the old pods can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old pods can be scaled down further, followed by scaling up the new pods, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods. |
SASL
(Appears on: KafkaSink, KafkaSource)
|
Field |
Description |
|---|---|
|
|
SASL mechanism to use |
|
|
(Optional)
GSSAPI contains the kerberos config |
|
|
(Optional)
SASLPlain contains the sasl plain config |
|
|
(Optional)
SASLSCRAMSHA256 contains the sasl plain config |
|
|
(Optional)
SASLSCRAMSHA512 contains the sasl plain config |
|
|
(Optional)
OAuth contains the oauth config |
SASLOAuth
(Appears on: SASL)
|
Field |
Description |
|---|---|
|
|
ClientID refers to the secret that contains the client id |
|
|
ClientSecret refers to the secret that contains the client secret |
|
|
TokenEndpoint refers to the token endpoint |
SASLPlain
(Appears on: SASL)
|
Field |
Description |
|---|---|
|
|
UserSecret refers to the secret that contains the user |
|
|
(Optional)
PasswordSecret refers to the secret that contains the password |
|
|
SASLType (string alias)
(Appears on: SASL)
SASLType describes the SASL type
Scale
(Appears on: AbstractVertex, MonoVertexSpec)
Scale defines the parameters for autoscaling.
|
Field |
Description |
|---|---|
|
|
(Optional)
Whether to disable autoscaling. Set to “true” when using Kubernetes HPA or any other 3rd party autoscaling strategies. |
|
|
(Optional)
Minimum replicas. |
|
|
(Optional)
Maximum replicas. |
|
|
(Optional)
Lookback seconds to calculate the average pending messages and processing rate. |
|
|
(Optional)
After scaling down the source vertex to 0, sleep how many seconds before scaling the source vertex back up to peek. |
|
|
(Optional)
TargetProcessingSeconds is used to tune the aggressiveness of autoscaling for source vertices, it measures how fast you want the vertex to process all the pending messages. Typically increasing the value, which leads to lower processing rate, thus less replicas. It’s only effective for source vertices. |
|
|
(Optional)
TargetBufferAvailability is used to define the target percentage of the buffer availability. A valid and meaningful value should be less than the BufferUsageLimit defined in the Edge spec (or Pipeline spec), for example, 50. It only applies to UDF and Sink vertices because only they have buffers to read. |
|
|
(Optional)
DeprecatedReplicasPerScale defines the number of maximum replicas that can be changed in a single scale up or down operation. The is use to prevent from too aggressive scaling operations Deprecated: Use ReplicasPerScaleUp and ReplicasPerScaleDown instead |
|
|
(Optional)
ScaleUpCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling up. It defaults to the CooldownSeconds if not set. |
|
|
(Optional)
ScaleDownCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling down. It defaults to the CooldownSeconds if not set. |
|
|
(Optional)
ReplicasPerScaleUp defines the number of maximum replicas that can be changed in a single scaled up operation. The is use to prevent from too aggressive scaling up operations |
|
|
(Optional)
ReplicasPerScaleDown defines the number of maximum replicas that can be changed in a single scaled down operation. The is use to prevent from too aggressive scaling down operations |
|
|
(Optional)
Cron defines time-based autoscaling overrides. During active cron windows, the window’s min/max replace the base min/max for scaling decisions. |
ServeSink
(Appears on: AbstractSink)
ServingPipeline
|
Field |
Description | ||||
|---|---|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
| ||||
|
|
| ||||
|
|
(Optional) |
ServingPipelinePhase (string alias)
(Appears on: ServingPipelineStatus)
ServingPipelineSpec
(Appears on: ServingPipeline)
|
Field |
Description |
|---|---|
|
| |
|
|
ServingPipelineStatus
(Appears on: ServingPipeline)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
The generation observed by the ServingPipeline controller. |
ServingSource
(Appears on: Source)
ServingSource is the source vertex for ServingPipeline and should be used only with ServingPipeline.
ServingSpec
(Appears on: ServingPipelineSpec)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional)
Whether to create a ClusterIP Service |
|
|
(Optional)
Ports to listen on, default we will use 8443 for HTTPS. To start http server the http port should be explicitly set. |
|
|
The header key from which the message id will be extracted |
|
|
(Optional)
Request timeout in seconds. Default value is 120 seconds. |
|
|
(Optional) |
|
|
(Optional)
Container template for the serving container. |
|
|
(Optional)
Initial replicas of the serving server deployment. |
|
|
(Members of (Optional) |
ServingStore
(Appears on: ServingSpec)
ServingStore defines information of a Serving Store used in a pipeline
|
Field |
Description |
|---|---|
|
|
SessionWindow
(Appears on: Window)
SessionWindow describes a session window
|
Field |
Description |
|---|---|
|
|
Timeout is the duration of inactivity after which a session window closes. |
SideInput
(Appears on: PipelineSpec)
SideInput defines information of a Side Input
|
Field |
Description |
|---|---|
|
| |
|
| |
|
|
(Optional) |
|
|
SideInputTrigger
(Appears on: SideInput)
|
Field |
Description |
|---|---|
|
|
The schedule to trigger the retrievement of the side input data. It supports cron format, for example, “0 30 * * * *”. Or interval based format, such as “@hourly”, “@every 1h30m”, etc. |
|
|
(Optional) |
SideInputsManagerTemplate
(Appears on: Templates)
|
Field |
Description |
|---|---|
|
|
(Members of (Optional) |
|
|
(Optional)
Template for the side inputs manager numa container |
|
|
(Optional)
Template for the side inputs manager init container |
Sink
(Appears on: AbstractVertex, MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
(Optional)
Fallback sink can be imagined as DLQ for primary Sink. The writes to Fallback sink will only be initiated if the ud-sink response field sets it. |
|
|
(Optional)
OnSuccess sink allows triggering a secondary sink operation only after the primary sink completes successfully The writes to OnSuccess sink will only be initiated if the ud-sink response field sets it. A new Message crafted in the Primary sink can be written on the OnSuccess sink. |
|
|
(Optional)
RetryStrategy struct encapsulates the settings for retrying operations in the event of failures. |
SlidingWindow
(Appears on: Window)
SlidingWindow describes a sliding window
|
Field |
Description |
|---|---|
|
|
Length is the duration of the sliding window. |
|
Slide is the slide parameter that controls the frequency at which the sliding window is created. | |
|
|
(Optional)
Streaming should be set to true if the reduce udf is streaming. |
Source
(Appears on: AbstractVertex, MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
SqsSink
(Appears on: AbstractSink)
|
Field |
Description |
|---|---|
|
|
AWSRegion is the AWS Region where the SQS queue is located |
|
|
QueueName is the name of the SQS queue |
|
|
QueueOwnerAWSAccountID is the queue owner aws account id |
|
|
(Optional)
AssumeRole contains the configuration for AWS STS assume role. When specified, the SQS client will assume the specified role for authentication. |
SqsSource
(Appears on: Source)
SqsSource represents the configuration of an AWS SQS source
|
Field |
Description |
|---|---|
|
|
AWSRegion is the AWS Region where the SQS queue is located |
|
|
QueueName is the name of the SQS queue |
|
|
QueueOwnerAWSAccountID is the queue owner aws account id |
|
|
(Optional)
VisibilityTimeout is the duration (in seconds) that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request. Valid values: 0-43200 (12 hours) |
|
|
(Optional)
MaxNumberOfMessages is the maximum number of messages to return in a single poll. Valid values: 1-10 Defaults to 1 |
|
|
(Optional)
WaitTimeSeconds is the duration (in seconds) for which the call waits for a message to arrive in the queue before returning. If a message is available, the call returns sooner than WaitTimeSeconds. Valid values: 0-20 Defaults to 0 (short polling) |
|
|
(Optional)
EndpointURL is the custom endpoint URL for the AWS SQS API. This is useful for testing with localstack or when using VPC endpoints. |
|
|
(Optional)
AttributeNames is a list of attributes that need to be returned along with each message. Valid values: All | Policy | VisibilityTimeout | MaximumMessageSize | MessageRetentionPeriod | ApproximateNumberOfMessages | ApproximateNumberOfMessagesNotVisible | CreatedTimestamp | LastModifiedTimestamp | QueueArn | ApproximateNumberOfMessagesDelayed | DelaySeconds | ReceiveMessageWaitTimeSeconds | RedrivePolicy | FifoQueue | ContentBasedDeduplication | KmsMasterKeyId | KmsDataKeyReusePeriodSeconds | DeduplicationScope | FifoThroughputLimit | RedriveAllowPolicy | SqsManagedSseEnabled |
|
|
(Optional)
MessageAttributeNames is a list of message attributes that need to be returned along with each message. |
|
|
(Optional)
AssumeRole contains the configuration for AWS STS assume role. When specified, the SQS client will assume the specified role for authentication. |
Status
(Appears on: InterStepBufferServiceStatus, MonoVertexStatus, PipelineStatus, ServingPipelineStatus, VertexStatus)
Status is a common structure which can be used for Status field.
|
Field |
Description |
|---|---|
|
|
(Optional)
Conditions are the latest available observations of a resource’s current state. |
TLS
(Appears on: JetStreamSource, KafkaSink, KafkaSource, NatsSource, RedisSentinelConfig)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional)
CACertSecret refers to the secret that contains the CA cert |
|
|
(Optional)
CertSecret refers to the secret that contains the cert |
|
|
(Optional)
KeySecret refers to the secret that contains the key |
TagConditions
(Appears on: ForwardConditions)
|
Field |
Description |
|---|---|
|
|
(Optional)
Operator specifies the type of operation that should be used for conditional forwarding value could be “and”, “or”, “not” |
|
|
Values tag values for conditional forwarding |
Templates
(Appears on: PipelineSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
DaemonTemplate is used to customize the Daemon Deployment. |
|
|
(Optional)
JobTemplate is used to customize Jobs. |
|
|
(Optional)
SideInputsManagerTemplate is used to customize the Side Inputs Manager. |
|
|
(Optional)
VertexTemplate is used to customize the vertices of the pipeline. |
UDF
(Appears on: AbstractVertex, MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
UDSink
(Appears on: AbstractSink)
|
Field |
Description |
|---|---|
|
|
UDSource
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
UDTransformer
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
(Optional) |
UpdateStrategy
(Appears on: AbstractVertex, MonoVertexSpec)
UpdateStrategy indicates the strategy that the controller will use to perform updates for Vertex or MonoVertex.
|
Field |
Description |
|---|---|
|
|
(Optional)
Type indicates the type of the StatefulSetUpdateStrategy. Default is RollingUpdate. |
|
|
(Optional)
RollingUpdate is used to communicate parameters when Type is RollingUpdateStrategy. |
UpdateStrategyType (string alias)
(Appears on: UpdateStrategy)
UpdateStrategyType is a string enumeration type that enumerates all possible update strategies.
Vertex
(Appears on: VertexInstance)
|
Field |
Description | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
| ||||||||||||||||||
|
|
| ||||||||||||||||||
|
|
(Optional) |
VertexInstance
VertexInstance is a wrapper of a vertex instance, which contains the vertex spec and the instance information such as hostname and replica index.
|
Field |
Description |
|---|---|
|
| |
|
| |
|
|
VertexLifecycle
(Appears on: VertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
DesiredPhase used to bring the vertex from current phase to desired phase |
VertexLimits
(Appears on: AbstractVertex, CombinedEdge)
|
Field |
Description |
|---|---|
|
|
(Optional)
Read batch size from the source or buffer. It overrides the settings
from pipeline limits. ReadBatchSize controls only how many messages are
fetched in a single read call from the source/buffer; it is not a cap on
how many messages may be in-flight (use |
|
|
(Optional)
Read timeout duration from the source or buffer It overrides the settings from pipeline limits. |
|
|
(Optional)
BufferMaxLength is used to define the max length of a buffer. It overrides the settings from pipeline limits. |
|
|
(Optional)
BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. It overrides the settings from pipeline limits. |
|
|
(Optional)
RateLimit is used to define the rate limit for the vertex, it overrides
the settings from pipeline limits. For Source vertices, the rate limit
is defined by how many times the |
|
|
(Optional)
Concurrency defines the maximum number of messages that can be actively
in-flight (read but not yet acknowledged) at any given time. With
read-ahead enabled, the data plane keeps reading new batches from the
source/buffer until the number of in-flight messages reaches
|
VertexPhase (string alias)
(Appears on: VertexLifecycle, VertexStatus)
VertexSpec
(Appears on: Vertex)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
| |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Watermark indicates watermark progression in the vertex, it’s populated from the pipeline watermark settings. |
|
|
(Optional)
Lifecycle defines the Lifecycle properties of a vertex |
|
|
(Optional)
InterStepBuffer configuration specific to this pipeline. |
VertexStatus
(Appears on: Vertex)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
(Optional) |
|
|
(Optional)
Total number of non-terminated pods targeted by this Vertex (their labels match the selector). |
|
|
(Optional)
The number of desired replicas. |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Time of last scaling operation. |
|
|
(Optional)
The generation observed by the Vertex controller. |
|
|
(Optional)
The number of pods targeted by this Vertex with a Ready Condition. |
|
|
The number of Pods created by the controller from the Vertex version indicated by updateHash. |
|
|
The number of ready Pods created by the controller from the Vertex version indicated by updateHash. |
|
|
If not empty, indicates the current version of the Vertex used to generate Pods. |
|
|
If not empty, indicates the updated version of the Vertex used to generate Pods. |
VertexTemplate
(Appears on: Templates)
|
Field |
Description |
|---|---|
|
|
(Members of (Optional) |
|
|
(Optional)
Template for the vertex numa container |
|
|
(Optional)
Template for the vertex init container |
VertexType (string alias)
(Appears on: CombinedEdge)
Watermark
(Appears on: PipelineSpec, VertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
Disabled toggles the watermark propagation, defaults to false. |
|
|
(Optional)
Maximum delay allowed for watermark calculation, defaults to “0s”, which means no delay. |
|
|
(Optional)
IdleSource defines the idle watermark properties, it could be configured in case source is idling. |
Window
(Appears on: GroupBy)
Window describes windowing strategy
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
Generated with gen-crd-api-reference-docs.