Big Data Management Administrator Guide > Queuing > Persisted Queues
  

Persisted Queues

The Data Integration Service uses persisted queues to store deployed mapping jobs and workflow mapping tasks. Persisted queuing protects against data loss if a node shuts down unexpectedly.
When you deploy a mapping job or workflow mapping task, the Data Integration Service moves these jobs directly to the persisted queue for that node. The job state is "Queued" in the Administrator tool contents panel. When resources are available, the Data Integration Service starts running the job.
Every node in a grid has one queue. Therefore, if the Data Integration Service shuts down unexpectedly, the queue does not fail over when the Data Integration Service fails over. The queue remains on the Data Integration Service machine, and the Data Integration Service resumes processing the queue when you restart it.
Note: While persisted queues help prevent data loss, you can still lose data if a node shuts down unexpectedly. In this case, all jobs in the "Running" state are marked as "Unknown." You must manually run these jobs again when the node restarts.
By default, each queue can hold 10,000 jobs at a time. When the queue is full, the Data Integration Service rejects job requests and marks them as failed. When the Data Integration Service starts running jobs in the queue, you can deploy additional jobs.
Persisted queuing is available for certain types of jobs, but not all. When you run a job that cannot be queued, the Data Integration Service immediately starts running the job. If there are not enough resources available, the job fails.
The following job types cannot be queued:

Queuing Process

The Data Integration Service queues deployed jobs before running them in the native or Hadoop batch pool. On-demand jobs run immediately in the on-demand pool.
The following diagram shows the overall queuing and execution process:
The diagram shows a deployed job and an on-demand job. The deployed job goes to the queue, and then to the native or Hadoop batch pool. The on-demand job goes directly to the on-demand queue. Both jobs are completed after they are released from the pool.
When you deploy a mapping job or workflow mapping task, the Data Integration Service moves the job directly to the persisted queue for that node. If the queue is full, the Data Integration Service marks the job as failed.
You can cancel a job in the queue. A job is aborted if the node shuts down unexpectedly and the Data Integration Service is configured to discard all jobs in the queue upon restart.
When resources are available, the Data Integration Service moves the job to the execution pool and starts running the job. A deployed job runs in one of the following execution pools:
Native Batch Pool
Runs deployed native jobs.
Hadoop Batch Pool
Runs deployed Hadoop jobs.
You can cancel a running job, or the job may be aborted if the node shuts down unexpectedly. A job can also fail while running.
The Data Integration Service marks successful jobs as completed.
The Data Integration Service immediately starts running on-demand jobs. If you run more jobs than the On-Demand Pool can run concurrently, the extra jobs fail. You must manually run the jobs again when space is available.
The following table describes the mapping job states in the Administrator tool contents panel:
Job Status
Rules and Guidelines
Queued
The job is in the queue.
Running
The Data Integration Service is running the job.
Completed
The job ran successfully.
Aborted
The job was flushed from the queue at restart or the node shut down unexpectedly while the job was running.
Failed
The job failed while running or the queue is full.
Canceled
The job was deleted from the queue or cancelled while running.
Unknown
The job status is unknown.