Frequently asked questions for Streaming Ingestion and Replication behavior

Are the checkpoints stored on the Secure Agent, cloud repository, source, or target?

Depending on the source and target, the checkpoints are stored on the Secure Agent, cloud repository, source, or target. The storage of checkpoints varies for different sources and targets.

For example, for an Amazon Kinesis Streams source, Streaming Ingestion and Replication creates a DynamoDB checkpoint table in a data warehouse such as Amazon S3.

Can I set the checkpoint after deploying or undeploying a task?

No, you can't set the checkpoint after deploying or undeploying a task.

In a streaming ingestion and replication task with a Kafka source and a Kafka target, can I rely on the offset stored in Kafka as a restart checkpoint?

Yes, you can treat the offset stored in Kafka as a restart checkpoint.

In a streaming ingestion and replication task with a Kafka source and a Kafka target, does the group.id property help to restart the checkpoint after the tasks are undeployed and deployed?

The group.id is a shared property in Kafka that helps in checkpoint recovery. When a Kafka source within a group processes a message successfully, it updates its offset in the topic partition. This offset is stored along with the group.id. When a Kafka dataflow transitions from the undeployed to the deployed status, and is up and running, it uses the same group.id and the stored offset to resume processing from where the dataflow stopped.

Can you guarantee At least once delivery allowing potential duplicates in the target after a task failure?

Yes, At least once delivery is guaranteed.

Is there any difference in delivery depending on the source and target connectors?

Each connector has a unique implementation, but all are designed to ensure At least once delivery guaranteed delivery.

How does recovery occur after a failure?

After the Secure Agent recovers from a failure, the runtime engine restarts, and the reconciliation process takes place.

Does Streaming Ingestion and Replication support a Secure Agent group with more than one Secure Agent?

Streaming Ingestion and Replication supports multiple Secure Agents under a single agent group. However, a dataflow can't run across multiple Secure Agents at the same time.

Does Streaming Ingestion and Replication support shared Secure Agent groups?

Streaming Ingestion and Replication doesn't support shared Secure Agent groups.

Why does the following warning message appear?

"The Secure Agent is either offline or not reachable."

You might receive the warning message because of the following reasons:

• - The Secure Agent is facing a network error.
• - The Channel Service in the agent is not working properly, preventing the tunnel from opening for communication.
• - There is an issue with Java heap memory.
• - Some old dataflows block other dataflows, making the agent unreachable.
• - A permission issue occurs while creating and starting the agent with different users.