REST API Reference > File Ingestion and Replication REST API > tasks resource
  

tasks resource

Use the tasks resource to create, update, delete, and view file ingestion and replication tasks.
Running and monitoring file ingestion and replication tasks involves a series of requests and responses. Use the followings methods to perform file ingestion and replication tasks:

View file ingestion and replication tasks

Use the GET request to view file ingestion and replication tasks.

GET request

To view the details of a particular file ingestion and replication task, include the file ingestion and replication in the following URI:
mftsaas/api/v1/mitasks/{{TASK-ID}}
To view the details for all file ingestion and replication tasks in the organization, omit the file ingestion and replication ID.
mftsaas/api/v1/mitasks
For example:
GET https://na1.dm-us.informaticacloud.com/mftsaas/api/v1/mitasks

GET response

Returns the task object if successful or an error object if errors occur.
The task object includes the following information about each of the file ingestion and replication tasks in the organization:
Field
Type
Description
taskId
String
ID number associated with the task.
taskName
String
Name of the task.
description
String
Description of the task.
location
String
Project and folder path where the task exists.
createTime
Date/time
Time when the task was created.
updateTime
Date/time
Time when the task was last updated.
Note: The create and update time in the response are in UTC time.

GET response example to view all file ingestion and replication tasks

The following sample response shows that there are three file ingestion and replication tasks in the organization:

{
"mitasks": [
{
"id": "1ONE5Vewzztl0tuKR0EDum",
"name": "A01_UMAR_MITASK2318",
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "",
"name": "",
"type": "local"
},
"targetConnection": {
"id": "0100000B000000000002",
"name": "ftps",
"type": "Advanced FTPS"
},
"agentGroupId": "01000025000000000002",
"updatedTime": "2019-01-30T11:17:49Z"
},
{
"id": "9D1tGkAxopJeFmUWoG4s48",
"name": "A01_UMAR_MITASK3354",
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "0100000B00000000000M",
"name": "AzureBlob",
"type": "Azure Blob"
},
"targetConnection": {
"id": "0100000B00000000000L",
"name": "SFTP_Conn",
"type": "Advanced SFTP"
},
"agentGroupId": "01000025000000000002",
"updatedTime": "2019-01-30T06:42:19Z"
},
{
"id": "4hcTFqKVOQrl1z4d6pGUMP",
"name": "A01_UMAR_MITASK5124",
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "0100000B0000000004IO",
"name": "S3",
"type": "AmazonS3"
},
"targetConnection": {
"id": "",
"name": "",
"type": "local"
},
"agentGroupId": "01000025000000000002",
"updatedTime": "2019-01-30T06:35:01Z"
}]
}

Get response example showing a file ingestion and replication task with file pattern as the file pickup option

The following sample response shows details of a file ingestion and replication task.
IDS-SESSION-ID:{{IDS-SESSION-ID}}
Accept:application/json
{
"id": "j9OLB12nqYObykdFSUMpO2",
"name": "FTPSrcTarget",
"location": {
"projectId": "dNC6zbp2lI8ghrKPo6hpwn",
"projectName": "Hardening"
},
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "0100000B00000000028M",
"name": "CCI_FTPS",
"type": "Advanced FTPS V2"
},
"targetConnection": {
"id": "0100000B0000000001JR",
"name": "CCI_FTP_Lin",
"type": "Advanced FTP V2"
},
"sourceParameters": {
"filePattern": "*.txt",
"sourceTransferMode": "AUTO",
"filePatternType": "WILDCARD",
"includeSubfolder": "false",
"sourceDirectory": "/root/suraj/qa/test/automation/RSFiles",
"checkDuplicate": "false",
"fileStability": "true",
"stabilityCheckInterval": "60",
"postPickupAction": "KEEP"
},
"targetParameters": {
"fileExistsAction": "APPEND_TIMESTAMP",
"targetDirectory": "/",
"targetTransferMode": "AUTO"
},
"agentGroupId": "01000025000000000003",
"createdTime": "2019-02-04T10:34:08Z",
"updatedTime": "2019-02-04T11:04:02Z",
"filePickupOption": "PATTERN"
}

GET response example showing a file ingestion and replication task with file list (file path) as the file pickup option

The following sample response shows a file ingestion and replication task with filePickupOption type as FILELIST and a filePickupFilePath in its sourceParameters, indicating that this task reads the designated pickup file to identify which files need to be processed.

{
"id": "aFHWKrr1RwycuBRBLTtt2t",
"name": "FilePath_CheckStability",
"location": {
"projectId": "0ggRhrI8ZziguyBxHBzuG0",
"projectName": "Default"
},
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "",
"name": "",
"type": "local"
},
"targetConnection": {
"id": "",
"name": "",
"type": "local"
},
"sourceParameters": {
"filePickupFilePath": "test.txt",
"sourceDirectory": "/root/test",
"checkDuplicate": "false",
"stabilityCheckInterval": "60",
"postPickupAction": "KEEP",
"filepickupByName": "FILEPATH",
"batchSize": "5",
"fileStability": "true",
"stabilityCheckInterval": "60"
},
"targetParameters": {
"fileExistsAction": "OVERWRITE",
"targetDirectory": "/root/testCheckStability"
},
"agentGroupId": "01001D25000000000002",
"createdTime": "2021-08-13T09:38:03Z",
"updatedTime": "2021-08-13T09:39:02Z",
"logLevel": "NORMAL",
"filePickupOption": "FILELIST"
}

GET response example showing a file ingestion and replication tasks with file list as the file pickup option

The following sample response shows a file ingestion and replication task with filePickupOption type as FILELIST, filepickupByName as LISTOFFILES, and a filePickupFileList in its sourceParameters, indicating that this task reads and identifies the designated pickup files to be processed.

{
"id": "2bTlAolXbAGlE7I5qauSAW",
"name": "DedupFilelist_pushdown",
"location": {
"projectId": "0ggRhrI8ZziguyBxHBzuG0",
"projectName": "Default"
},
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "01001D0B0000000005PP",
"name": "ADLSGen2",
"type": "Azure Data Lake Gen2"
},
"targetConnection": {
"id": "01001D0B0000000005PU",
"name": "AzureDW_Gen2",
"type": "Azure DW"
},
"sourceParameters": {
"sourceDirectory": "/B2B/MI",
"checkDuplicate": "true",
"postPickupAction": "KEEP",
"filepickupByName": "LISTOFFILES",
"blockSize": "8388608",
"filePickupFileList": "File1.txt,File2.txt",
"batchSize": "5",
"timeoutInterval": "60",
"fileStability": "true",
"stabilityCheckInterval": "60"
},
"targetParameters": {
"commandType": "auto",
"targetTableName": "test1234",
"isPushdown": "true",
"ingestionMethod": "polybase",
"targetSchemaName": "testing",
"isTruncateTarget": "true"
},
"agentGroupId": "01001D25000000000002",
"createdTime": "2021-04-29T08:47:57Z",
"updatedTime": "2021-04-29T08:47:57Z",
"logLevel": "NORMAL",
"filePickupOption": "FILELIST"
}

Create a file ingestion and replication task

Use a POST request to create a file ingestion and replication task.

POST request

To create a file ingestion and replication task through the API, use the following URI:
mftsaas/api/v1/mitasks
Include the following fields in the request:
Field
Type
Required
Description
name
String
Yes
Name of the task.
location
String
-
Location of the project.
projectId
String
-
ID number associated with the project.
projectName
String
-
Name of the project.
description
String
-
Description of the task.
sourceConnection
String
-
Directory from where files are transferred.
sourceType
String
Yes
Determines the type where files are transferred. Enter one of the following options:
  • - CONNECTION. Use connection as a source.
  • - FILELISTENER. Use file listener as a source.
includesubfolder
String
-
Determines whether to include the files in sub-folders in the transfer. Set the value to true to transfer files from all sub-folders under the defined source directory. Values are true or false.
checkDuplicate
String
-
Determines whether to check for duplicate files. Values are true or false. Set the value to true to check duplicate files and deny file transfer. If the value is set to false all files are transferred.
filePickupOption
String
Yes
Determines the file pickup method. Enter one of the following options:
  • - FILELIST. The file ingestion and replication task picks up files based on a file list.
  • - PATTERN. The file ingestion and replication task picks up files by pattern.
allowConcurrency
String
-
Determines whether to run multiple jobs concurrently. Set the value to true to run multiple jobs concurrently, else set the value to false.
Warning: Running concurrent jobs might cause unexpected results if the targets include duplicate files.
filePatternType
String
Yes
This applies when filePickupOption is PATTERN. File pattern type used to select files to transfer. Enter one of the following options:
  • - wildcard
  • - regex
filePattern
String
Yes
Enter file pattern types, depending on the file pattern that you have selected.
  • - wildcard. You can use the following wildcard character filters:
    • - An asterisk (*) matches any number of characters.
    • - A question mark (?) matches a single character.
  • - regex. Use regular expression to match the file pattern. Consider the following examples:
    • - Use the following syntax to listen to all files except for files with a name that contains out, foo, and baz: ^(?!.*(?:out|baz|foo)).*$ all except
    • - Use the following syntax to listen to all files with doc, docx, and pdf extensions: ([a-zA-Z0-9\s_\\.\-\(\):])+(.doc|.docx|.pdf)$
filepickupByName
String
Yes
This applies when filePickupOption is FILELIST. Enter one of the following options:
  • - filepath. Provide the path that contains the list of files to pick up and enter the file path.
  • - listoffiles. Provide the list of files to pick up and enter a comma-separated list of file names. Ensure there is no space before or after specifying the file name.
fileStability
Boolean
-
Determines if the task verifies whether the file is stable before picking it up. Enter one of the following values.
  • - true. The file ingestion and replication task verifies whether the file is stable before picking it up.
  • - false. The file ingestion and replication task does not verify whether the file is stable before picking it up.
Default is false.
stabilityCheckInterval
Int
-
Time in seconds that a file ingestion and replication task waits to check the file stability.
You can specify a value in the stabilityCheckInterval field only if the fileStability option is set to true.
The stability check interval ranges between 10 seconds to 300 seconds.
postPickupAction
String
-
Determines what to do with source files after the transfer of files. The following options are available:
  • - KEEP. Keep the files in the source directory.
  • - DELETE. Delete the files from the source directory.
  • - RENAME. Rename the files in the source directory. You must specify a file name suffix that File ingestion and replication adds to the file name when renaming the files.
  • - ARCHIVE. Archive the files to a different location. You must specify an archive directory.
targetConnection
String
Yes
Directory details to which files are transferred.
taskActions
String
-
Actions to process files in the file ingestion and replication task. If you add multiple actions, file ingestion and replication processes files in a sequence.
actions
File processing action.
Enter the following file processing actions:
  • - To compress files, enter Compression.
  • - To decompress files, enter Decompression.
  • - To encrypt files, enter Encryption.
  • - To dencrypt files, enter Decryption.
  • - To move files from multiple folders to a single folder and to rename file in the target directory, enter File Operations.
  • - To scan files for viruses by using the ICAP protocol, enter Virus Scan.
action type
Enter the action type depending on the action that you add.
To compress files use one of the following methods.
  • - Zip
  • - Tar
  • - Gzip
  • - Bzip2
To decompress files use one of the following methods.
  • - Unzip
  • - Untar
  • - Gunzip
  • - Bunzip2
To encrypt files add PGP. Enter the key ID in the properties.
Note: The file ingestion and replication task uses the PGP method to encrypt files. Generate a key ring using the CLI. Enter the key ring in the Key ID. For more information about the keyring CLIs, refer to key ring command reference in Tasks.
To decrypt files, add PGP. Enter the key passphrase in the properties.
Note: The file ingestion and replication task uses the PGP method to encrypt files. Generate the key passphrase using the CLI. Enter the key passphrase in the Key Passphrase. For more information about the keyring CLIs, refer to key ring command reference in Tasks.

POST request example

Use this sample as a reference to create a file ingestion and replication task with file pattern as the file pickup option
POST <serverURL>/public/core/v1/mitasks
Content-Type: application/json
Accept:application/json
Content-Type:application/json
IDS-SESSION-ID:{{IDS-SESSION-ID}}
{
"name": "Green Green v2",
"location": {
"projectId": "9JDNOBX9M31e2AD1dIUv6M",
"projectName": "Default"
},
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "",
"name": "",
"type": "local"
},
"sourceParameters": {
"filePattern": "*.txt",
"filePatternType": "WILDCARD",
"includeSubfolder": "false",
"sourceDirectory": "C:\\Monitor",
"checkDuplicate": "false",
"fileStability": "true",
"stabilityCheckInterval": "60",
"postPickupAction": "KEEP"

},
"targetConnection": {
"id": "0100000B000000000002",
"name": "ADLS",
"type": "Azure Data Lake"
},
"targetParameters": {
"adlsTargetLocation": "/satyen/green"

},
"agentGroupId": "01000025000000000002",
"filePickupOption": "PATTERN",
"logLevel": "NORMAL",
"allowConcurrency": "true",
taskActions":[
{
"action": "Compression",
"actionType": "Zip",
"properties": {}
}
]
}
Use this sample as a reference to create a file ingestion and replication task with file path as the file pickup option.
POST <serverURL>/public/core/v1/mitasks
Content-Type: application/json
Accept:application/json
Content-Type:application/json
IDS-SESSION-ID:{{IDS-SESSION-ID}}
{
"name": "FilePath_RestAPI1",
"location": {
"projectId": "0ggRhrI8ZziguyBxHBzuG0",
"projectName": "Default"
},
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "",
"name": "",
"type": "local"
},
"targetConnection": {
"id": "",
"name": "",
"type": "local"
},
"sourceParameters": {
"filePickupFilePath": "test.txt",
"sourceDirectory": "/root/test",
"checkDuplicate": "false",
"fileStability": "true",
"stabilityCheckInterval": "60",
"postPickupAction": "KEEP",
"filepickupByName": "FILEPATH",
"batchSize": "5"

},
"targetParameters": {
"fileExistsAction": "OVERWRITE",
"targetDirectory": "/root/testCheckStability"
},
"agentGroupId": "01001D25000000000002",
"logLevel": "NORMAL",
"filePickupOption": "FILELIST",
"allowConcurrency": "true"
}
Use this sample as a reference to create a file ingestion and replication task with file list as the file pickup option.
POST <serverURL>/public/core/v1/mitasks
Content-Type: application/json
Accept:application/json
Content-Type:application/json
IDS-SESSION-ID:{{IDS-SESSION-ID}}
{

"name": "DedupFilelist_RestAPI",
"location": {
"projectId": "0ggRhrI8ZziguyBxHBzuG0",
"projectName": "Default"
},
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "01001D0B0000000005PP",
"name": "ADLSGen2",
"type": "Azure Data Lake Gen2"
},
"targetConnection": {
"id": "01001D0B0000000005PU",
"name": "AzureDW_Gen2",
"type": "Azure DW"
},
"sourceParameters": {
"sourceDirectory": "/B2B/MI",
"checkDuplicate": "true",
"postPickupAction": "KEEP",
"filepickupByName": "LISTOFFILES",
"blockSize": "8388608",
"filePickupFileList": "File1.txt,File2.txt",
"batchSize": "5",
"timeoutInterval": "60",
"fileStability": "true",
"stabilityCheckInterval": "60"

},
"targetParameters": {
"commandType": "auto",
"targetTableName": "test1234",
"isPushdown": "true",
"ingestionMethod": "polybase",
"targetSchemaName": "testing",
"isTruncateTarget": "true"
},
"agentGroupId": "01001D25000000000002",
"logLevel": "NORMAL",
"filePickupOption": "FILELIST",
"allowConcurrency": "true"
}

POST response example

If the request is successful, you might receive a response similar to the following example:

{
"id": "cEMWKpibm44bNf5aMjbJ4U",
"name": "Green Green v2",
"location": {
"projectId": "9JDNOBX9M31e2AD1dIUv6M",
"projectName": "Default"
},
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "",
"name": "",
"type": "local"
},
"sourceParameters": {
"filePattern": "*.txt",
"filePatternType": "WILDCARD",
"includeSubfolder": "false",
"sourceDirectory": "C:\\Monitor",
"checkDuplicate": "false",
"fileStability": "true",
"stabilityCheckInterval": "60",
"postPickupAction": "KEEP"

},
"targetConnection": {
"id": "0100000B000000000002",
"name": "ADLS",
"type": "Azure Data Lake"
},
"targetParameters": {
"adlsTargetLocation": "/satyen/green"

},
"agentGroupId": "01000025000000000002",
"createdTime": "2018-08-27T07:03:32Z",
"updatedTime": "2018-08-29T12:14:58Z""
taskActions":[
{
"action": "Compression",
"actionType": "Zip",
"properties": {}
}
]
}
}
Note: The created and updated time in the response is displayed in the UTC time.

Update a file ingestion and replication task

Use a PUT request to update a file ingestion and replication task.

PUT request

To update a file ingestion and replication task, use the following URI:
mftsaas/api/v1/mitasks/<taskID>
Include the following fields in the PUT request:
Field
Type
Required
Description
id
String
-
ID number of the task.
name
String
Yes
Name of the task.
description
String
-
Description of the task.
sourceType
String
Yes
Determines the type where files are transferred. Enter one of the following options:
  • - CONNECTION. Use connection as a source.
  • - FILELISTENER. Use file listener as a source.
sourceConnection
String
-
Directory from where files are transferred.
includeSubfolder
String
-
Values are true or false. Set the value to true to transfer files from all sub-folders under the defined source directory.
checkDuplicate
String
-
Values are true or false. Set the value to true to check duplicate files and deny file transfer. If the value is set to false all files are transferred.
filePatternType
String
Yes
File name pattern used to select the files to transfer. Enter one of the following options:
  • - Wildcard
  • - Regex
filePattern
String
Yes
Enter pattern types, depending on the file pattern that you have selected.
  • - wildcard. You can use the following wildcard character filters:
    • - An asterisk (*) matches any number of characters.
    • - A question mark (?) matches a single character.
  • - Regex. Use regular expression to match the file pattern. Consider the following examples:
    • - Use the following syntax to listen to all files except for files with a name that contains out, foo, and baz: ^(?!.*(?:out|baz|foo)).*$ à all except
    • - Use the following syntax to listen to all files with doc and docx, pdf extensions: ([a-zA-Z0-9\s_\\.\-\(\):])+(.doc|.docx|.pdf)$ à
fileStability
Boolean
-
Determines if the task verifies whether the file is stable before picking it up. Enter one of the following values.
  • - true. The file ingestion and replication task verifies whether the file is stable before picking it up.
  • - false. The file ingestion and replication task does not verify whether the file is stable before picking it up.
Default is false.
stabilityCheckInterval
Int
-
Time in seconds that a file ingestion and replication task waits to check the file stability.
You can specify a value in the stabilityCheckInterval field only if the fileStability option is set to true.
The stability check interval ranges between 10 seconds to 300 seconds.
postPickupAction
String
-
Determines what to do with source files after the files transfer. The following options are available:
  • - KEEP. Keep the files in the source directory.
  • - DELETE. Delete the files from the source directory.
  • - RENAME. Rename the files in the source directory. You must specify a file name suffix that file ingestion and replication adds to the file name when renaming the files.
  • - ARCHIVE. Archive the files to a different location. You must specify an archive directory.
targetConnection
String
Yes
Directory details to which files are transferred.

PUT request example

Use this sample as a reference to update a file ingestion and replication task.
PUT <serverUrl>/public/core/v1/mitasks
Content-Type: application/json
Accept:application/json
Content-Type:application/json
IDS-SESSION-ID:{{IDS-SESSION-ID}}
{
"id": "cEMWKpibm44bNf5aMjbJ4U",
"name": "Green Green v2",
"description": "Green Green v2 Description",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "",
"name": "",
"type": "local"
},
"sourceParameters": {
"filePattern": "*.txt",
"filePatternType": "WILDCARD",
"includeSubfolder": "false",
"sourceDirectory": "C:\\Monitor",
"checkDuplicate": "false",
"fileStability": "true",
"stabilityCheckInterval": "60",
"postPickupAction": "KEEP"

},
"targetConnection": {
"id": "0100000B000000000002",
"name": "ADLS",
"type": "Azure Data Lake"
},
"targetParameters": {
"adlsTargetLocation": "/satyen/green"
},
"agentGroupId": "01000025000000000002
}

PUT response example

If the request is successful, you might receive a response similar to the following example:
{
"id": "cEMWKpibm44bNf5aMjbJ4U",
"name": "Green Green v2",
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "",
"name": "",
"type": "local"
},
"sourceParameters": {
"filePattern": "*.txt",
"filePatternType": "WILDCARD",
"includeSubfolder": "false",
"sourceDirectory": "C:\\Monitor",
"checkDuplicate": "false",
"fileStability": "true",
"stabilityCheckInterval": "60",
"postPickupAction": "KEEP"

},
"targetConnection": {
"id": "0100000B000000000002",
"name": "ADLS",
"type": "Azure Data Lake"
},
"targetParameters": {
"adlsTargetLocation": "/satyen/green"

},
"agentGroupId": "01000025000000000002",
"createdTime": "2018-08-27T07:03:32Z",
"updatedTime": "2018-08-29T12:14:58Z"
}

Note: The created and updated time in the response is displayed in the UTC time.

View the location of a file ingestion and replication task

Use the GET request to view the location of a file ingestion and replication task.

GET request

Use the following URI to get the location of a file ingestion and replication task.
/api/v1/mitasks?resolveLocation=true

GET response example

If the request to get the file location of a file ingestion and replication task is successful, you might receive a response similar to the following example:
{
"miTasks": [
{
"id": "1ONE5Vewzztl0tuKR0EDum",
"name": "A01_UMAR_MITASK2318",
"location": {
"folderId": "digFZU6HMo4gCKYghtQvgD",
"folderName": "A_01_UMAR",
"projectId": "503RTpKDSSLlwmkwTXL0Qx",
"projectName": "Default"
},
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "",
"name": "",
"type": "local"
},
"targetConnection": {
"id": "0100000B000000000002",
"name": "ftps",
"type": "Advanced FTPS"
},
"agentGroupId": "01000025000000000002",
"createdTime": "2019-01-28T09:54:53Z",
"updatedTime": "2019-01-30T11:17:49Z"
},
{
"id": "9D1tGkAxopJeFmUWoG4s48",
"name": "A01_UMAR_MITASK3354",
"location": {
"folderId": "digFZU6HMo4gCKYghtQvgD",
"folderName": "A_01_UMAR",
"projectId": "503RTpKDSSLlwmkwTXL0Qx",
"projectName": "Default"
},
"description": "",
"sourceType": "CONNECTION",
"sourceConnection": {
"id": "0100000B00000000000M",
"name": "AzureBlob",
"type": "Azure Blob"
},
"targetConnection": {
"id": "0100000B00000000000L",
"name": "SFTP_Conn",
"type": "Advanced SFTP"
},
"agentGroupId": "01000025000000000002",
"createdTime": "2019-01-30T06:36:28Z",
"updatedTime": "2019-01-30T06:42:20Z"
}
]
}

Delete a file ingestion and replication task

Use the DELETE request to delete a file ingestion and replication task.

DELETE request

To delete a file ingestion and replication include the task ID of the task through the API, in the following URI:
mftsaas/api/v1/mitasks/<taskID>