Updating a streaming ingestion and replication task
Use the UpdateEntity resource to update a streaming ingestion and replication task. You can update streaming ingestion and replication tasks that use the following connectors: Amazon Kinesis, Amazon S3 V2, Microsoft Azure Event Hub, Microsoft Azure Data Lake Storage Gen2, Flat file, JDBC V2, JMS, Kafka, or MQTT.
MQTT sources
When the source connection of the task source is MQTT, you can include the following fields and key-value pairs in the config array of the source connection:
Key
Type
Required
Description
ClientID
String
-
Unique identifier of the connection between the MQTT source and the MQTT broker. The client ID is the file-based persistence store that the MQTT source uses to store messages while they are processed.
You can enter a string of up to 255 characters.
MaxQueueSize
Integer
-
The maximum number of messages that the processor can store in memory.
You can enter a value between 1 and 2147483647.
Topic
String
Yes
Name of the MQTT topic.
POST request example
To update a streaming ingestion and replication task with an MQTT source and a flat file target, you might send a request similar to the following example:
POST <serverUrl>/sisvc/restapi/v1/UpdateEntity/Documents('<document ID>') Content-Type: application/json Accept: application/json IDS-SESSION-ID: <SessionId>
When the source connection of the task source is JMS, you can include the following fields and key-value pairs in the config array of the source connection:
Key
Type
Required
Description
destinationType
String
Yes
Type of destination that the source service sends JMS messages to. Enter one of the following values:
- QUEUE. The JMS provider delivers messages to a single consumer who is registered for the queue.
- TOPIC. The JMS provider delivers messages to all active consumers that subscribe to the topic.
clientId
String
Yes
Unique ID of the JMS connection. You can enter a string of up to 255 characters.
sharedSubscription
String
Yes
Enables multiple consumers to access a single subscription. Applies to the TOPIC destination type. Enter one of the following values:
- True
- False
durableSubscription
String
Yes
When set to True, the JMS source service enables inactive subscribers to retain messages and then deliver them when the subscriber reconnects. Applies to the TOPIC destination type. Enter one of the following values:
- True
- False
subscriptionName
String
Yes
Name of the subscription. Applies to the TOPIC destination type, when the topic subscription type is shared, durable, or both.
JMS Destination
String
Yes
Name of the queue or topic that the JMS provider delivers messages to.
POST request example
To update a streaming ingestion and replication task with an JMS source and a flat file target, you might send a request similar to the following example:
POST <serverUrl>/sisvc/restapi/v1/UpdateEntity/Documents('<document ID>') Content-Type: application/json Accept: application/json IDS-SESSION-ID: <SessionId>
Microsoft Azure Data Lake Storage Gen2 (ADLS Gen2) targets
When the target connection of the task target is ADLS Gen2, you can include the following fields and key-value pairs in the config array of the target connection:
Key
Type
Required
Description
writeStrategy
String
Yes
The action to take when a file by the same name exists in the ADLS Gen2 storage.
Enter one of the following values:
- Append. Add data to the existing file.
- Overwrite. Replaces the existing file with the new file.
- Fail. Fail the request.
- Rollover. Close the current file and create a new file based on the configured rollover value.
rolloverSize *
Integer
-
Target file size, in KB, at which to trigger rollover. Applies to a Rollover write strategy.
You can enter a value between 1 and 2147483647.
rolloverEvents *
Integer
-
Number of events or messages to accumulate before a rollover. Applies to a Rollover write strategy.
You can enter a value between 1 and 2147483647.
rolloverTime *
Integer
-
Length of time, in milliseconds, after which to trigger a rollover. Applies to a Rollover write strategy.
You can enter a value between 1 and 2147483647.
filesystemNameOverride
String
-
Overrides the default file system name provided in the connection. This file system name is used write to a file at run time.
You can enter a string of up to 1,280 characters.
directoryOverride
String
-
Overrides the default directory path. The ADLS Gen2 directory path to write data to. If left blank, the default directory path is used.
You can enter a string of up to 1,280 characters.
compressionFormat
String
-
Compression format to use before the streaming ingestion task writes data to the target file.
Enter one of the following values:
- None
- GZIP
- BZIP2
- DEFAULT1
Enter this value to use the Zlib format.
- DEFAULT2
Enter this value to use the Deflate format.
File Name/Expression
String
Yes
ADLS Gen2 file name or a regular expression.
You can enter a string of up to 249 characters.
* Enter a value for at least one of the fields.
POST request example
To update a streaming ingestion and replication task with a flat file source and an ADLS Gen2 target, you might send a request similar to the following example:
POST <serverUrl>/sisvc/restapi/v1/UpdateEntity/Documents('<document ID>') Content-Type: application/json Accept: application/json IDS-SESSION-ID: <SessionId>
When the target connection of the task target is Amazon S3, you can include the following fields and key-value pairs in the config array of the target connection:
Key
Type
Required
Description
partitionTime
String
-
The time interval according to which the streaming ingestion task creates partitions in the Amazon S3 bucket.
Enter one of the following values:
- None
- 5min
- 10min
- 15min
- 20min
- 30min
- 1hr
- 1day
minUploadPartSize
Integer
-
Minimum part size when uploading a large file as a set of multiple independent parts, in megabytes. Use this property to tune the file load to Amazon S3.
You can enter a value between 50 and 5120.
multipartUploadThreshold
Integer
-
Multipart threshold when uploading objects in multiple parts in parallel.
You can enter a value between 50 and 5120.
Object Name/Expression
String
Yes
Amazon S3 target file name or a regular expression for the Amazon S3 file name pattern.
POST request example
To update a streaming ingestion and replication task with a flat file source and an Amazon S3 as target, you might send a request similar to the following example:
POST <serverUrl>/sisvc/restapi/v1/UpdateEntity/Documents('<document ID>') Content-Type: application/json Accept: application/json IDS-SESSION-ID: <SessionId>
When the target connection of the task target is Azure Event Hubs, you can include the following fields and key-value pairs in the config array of the target connection:
Key
Type
Required
Description
sasPolicyName
String
-
The name of the Event Hub Namespace Shared Access Policy.
You can enter a string of up to 255 characters.
sasPolicyPrimaryKey
String
-
The primary key of the Event Hub Namespace Shared Access Policy.
You can enter a string of up to 255 characters.
Event Hub
String
Yes
The name of the Azure Event Hubs.
You can enter a string up to 255 characters. The name can contain lower case characters, upper case characters, numbers, and the special characters - and _.
POST request example
To update a streaming ingestion and replication task with a flat file source and an Azure Event Hubs target, you might send a request similar to the following example:
POST <serverUrl>/sisvc/restapi/v1/UpdateEntity/Documents('<document ID>') Content-Type: application/json Accept: application/json IDS-SESSION-ID: <SessionId>
When the target connection of the task target is JDBC V2, you can include the following fields and key-value pairs in the config array of the target connection:
Key
Type
Required
Description
Table Name
String
Yes
Name of the table to insert data to in JSON format.
Enter a string of up to 988 characters.
POST request example
To update a streaming ingestion and replication task with a flat file source and a JDBC V2 target, you might send a request similar to the following example:
POST <serverUrl>/sisvc/restapi/v1/UpdateEntity/Documents('<document ID>') Content-Type: application/json Accept: application/json IDS-SESSION-ID: <SessionId>
When the source and target connection of the task is Amazon Kinesis Streams, you can include the following fields and key-value pairs in the config array of the source and target connection:
Key
Type
Required
Description
appendGUID
Boolean
Specifies whether or not to add a GUID as a suffix to the Amazon DynamoDB table name.
Enter one of the following values:
- true
- false
dynamoDB
String
Amazon DynamoDB table name where to store the checkpoint details of the Kinesis source data.
You can enter a string of up to 128 characters.
Stream
String
Yes
Name of the Kinesis Stream to read data from.
Enter a string of up to 128 characters.
Appears in the source node.
Stream Name/Expression
String
Yes
Kinesis Stream name or a regular expression to write data to.
Enter a string of up to 128 characters.
Appears in the target node.
POST request example
To update a streaming ingestion and replication task with an Amazon Kinesis Streams source and target, you might send a request similar to the following example:
POST <serverUrl>/sisvc/restapi/v1/UpdateEntity/Documents('<document ID>') Content-Type: application/json Accept: application/json IDS-SESSION-ID: <SessionId>
When the source and target connection of the task is flat file, you can include the following fields and key-value pairs in the config array of the source and target connection:
Key
Type
Required
Description
File
String
Yes
Absolute path and name of the source file. Enter the base directory for multiple files mode.
initialPosition
String
Yes
Starting position to read data from the file to tail. Enter one of the following values:
- Beginning of File. Read from the beginning of the file. Don't ingest any data that has already been rolled over.
- Current Time. Read from the most recently updated part of the file. Don't ingest data that was rolled over or data in the file that was written.
rolloverPattern
String
-
File name pattern for the file that rolls over.
If the file to tail rolls over, the Secure Agent uses the file name pattern to identify files that have rolled over. If the Secure Agent stops during a file rollover, when it restarts, it picks up the file where it was left off.
You can use asterisk (*) and question mark (?) as wildcard characters to indicate that the files are rolled over in the same directory. For example,${filename}.log.*. Here, asterisk (*) represents the successive version numbers that would be appended to the file name.
tailingMode
String
Yes
Tail a file or multiple files based on the logging pattern. Enter one of the following values:
- Single file. Tail one file.
- Multiple files. Tail all the files indicated in the base directory. You can enter a regular expression to indicate the files to tail.
File Name
String
Yes
The name of the target file.
interimDirectory
String
Yes
Path to the staging directory on the Secure Agent.
rolloverSize
Integer
Yes
The file size, in KB, at which the task moves the file from the staging directory to the target.
You can enter a value between 1 and 2147483647.
rolloverEvents
Integer
Yes
Number of events or messages to accumulate before a file rollover.
You can enter a value between 1 and 2147483647.
rolloverTime
Integer
-
Length of time, in milliseconds, after which the target file rolls over.
You can enter a value between 1 and 2147483647.
edges
Array
-
Sequence of dataflow execution.
POST request example
To update a streaming ingestion and replication task with a flat file source and target, you might send a request similar to the following example:
POST <serverUrl>/sisvc/restapi/v1/UpdateEntity/Documents('<document ID>') Content-Type: application/json Accept: application/json IDS-SESSION-ID: <SessionId>
When the source and target connection of the task is Kafka, you can include the following fields and key-value pairs in the config array of the source and target connection:
Key
Type
Required
Description
Topic
String
Yes
Kafka source topic name or a Java supported regular expression for the Kafka source topic name pattern to read the events from.
Enter a string of up to 249 characters.
consumerProperties
String
-
Provide a comma-separated list of optional consumer configuration properties. Specify the values as key-value pairs. For example, key1=value1, key2=value2 .
You can enter a string of up to 4000 characters.
producerProperties
String
-
The configuration properties for the producer.
Provide a comma-separated list and specify the values as key-value pairs.
You can enter a string of up to 4000 characters.
mdFetchTimeout
Integer
-
The time after which the metadata is not fetched.
Enter a value between 1 and 2147483647.
batchSize
Integer
-
The batch size of the events after which a streaming ingestion task writes data to the target.
Enter a value between 1 and 2147483647.
Topic Name/Expression
String
Yes
Kafka topic name or a Java supported regular expression for the Kafka topic name pattern.
You can enter a string of up to 249 characters.
POST request example
To update a streaming ingestion and replication task with a Kafka source and target, you might send a request similar to the following example:
POST <serverUrl>/sisvc/restapi/v1/UpdateEntity/Documents('<document ID>') Content-Type: application/json Accept: application/json IDS-SESSION-ID: <SessionId>