Use the code task API to submit Spark code written in Scala to an advanced cluster. You can use the CodeTask resource to create, start, and cancel a code task job. You can also access session logs, view job details and job status of a code task.
Consider the following guidelines when you use the code task resource:
•Write your code task in Scala.
•Submit your code in a JAR file using the code task APIs.
•Use an AWS serverless or non-serverless environment.
•Ensure that you have the permission to create, execute, and view the code task APIs.
•Use the following persisted variables as needed for the code task APIs:
- IDS_SESSION_ID
- ORG_ID
- CODE_TASK_ID
- CODE_TASK_JOB_ID
Note: If you use a tool such as Postman that automatically includes the HTTP version, do not enter the HTTP version in the URL. If the HTTP version appears twice in the URL, the request fails.
Complete the following tasks to submit Scala code in a JAR file and manage and monitor code task jobs:
•Send login information to get the session ID using Login.
Use this POST request to start the code task job. The response includes the code task job ID that you can use in subsequent API calls.
POST request
To start a code task, use the task ID. The code task ID is included in the response when you create a code task. For more information, see Create a code task .
Use the following URI to start a code task:
POST <server URL>/disnext/api/v1/CodeTask/Start
Use the following field in the POST request:
Field
Type
Required / Optional
Description
codeTaskId
String
Required
The code task ID from the create resource.
POST response
The following variable is set from the response attributes:
Name
Response Value
Note
CODE_TASK_JOB_ID
jobId
Used to cancel the code task job, or to get job status, session logs, or Spark task status.
POST request example
Use this sample as a reference to start a code task.
Use this GET request to view the code task configuration parameters.
GET request
To request the parameters of a code task, use the task ID. The code task ID is included in the response when you create a code task. For more information, see Create a code task .
Use the following URI to get the code task parameters:
GET <server URL>/disnext/api/v1/CodeTask/<Code task ID>
GET response
Returns the code task object for the requested task ID.
Returns the error object if errors occurred.
The following table describes the parameters in the response:
Field
Type
Description
codeTaskName
String
Name of the code task.
codeTaskId
Numeric
The code task identifier.
agentGroupId
String
Runtime environment that contains the Secure Agent used to run the code task.
overrideTaskTimeout
String
Overrides the code task timeout value for this execution. A value of -1 signifies no timeout.
logLevel
String
Log level for session logs, agent job log, Spark driver, and executor logs. Valid values are: none, terse, normal, verboseInitialization, or verboseData.
The default value is normal.
sparkMainClass
String
Entry point of the Spark application. For example:
org.apache.spark.examples.company.SparkExampleApp
sparkMainClassArgs
List<String>
Ordered arguments sent to the Spark application main class. For example:
--appTypeSPARK_PI_FILES_JARS--
classesToLoadcom.company.test.SparkTest1Class
sparkPrimaryResource
String
Scala JAR file that contains the code task.
sparkJars
List <String>
The directory and file name of the JAR file that is uploaded to the cluster and added to the Spark driver and executor classpaths.
sparkFiles
List <String>
The directory and file name of the Spark file that is uploaded to the cluster and available under the current working directory.
advancedCustomProperties
String
Spark properties or other custom properties that Data Integration uses.
GET request example
Use this sample as a reference to get the code task parameters.
GET <server URL>/disnext/api/v1/CodeTask/<Code task ID> Content-Type: application/json Accept: application/json IDS-SESSION-ID:{{IDS_SESSION_ID}}
GET response example
A successful GET response returns the code task parameters, similar to the following example:
Use this GET request to check the code task job status.
GET request
To request the status of a code task job, use the code task job ID. The code task job ID is included in the response when you start a code task. For more information, see Start a code task.
Use the following URI to get the code task job status parameters:
GET <server URL>/disnext/api/v1/CodeTask/JobStatus/<Code task job ID>
GET response
Returns the code task object for the requested job ID.
Returns the error object if errors occurred.
The following table describes the parameters in the response:
Field
Type
Description
codeTaskName
String
Name of the code task.
codeTaskId
Numeric
The code task identifier.
agentGroupId
String
Runtime environment that contains the Secure Agent used to run the code task.
executionState
String
The state of the job: QUEUED, RUNNING, SUCCEEDED, FAILED, CANCELLED Job status. A job can have one of the following statuses:
- Starting. The job is starting.
- Running. The job is either queued or running.
- Success. The job completed successfully.
- Failed. The job did not complete because it encountered errors.
sessionLogUrl
String
URL of the session log file.
assignedServerlessComputeUnits
Double
Number of serverless compute units per hour that the task requested.
You can view the number of requested compute units if the task runs in a serverless runtime environment.
consumedServerlessComputeUnits
Double
Total number of serverless compute units that the task consumed.
You can view the number of consumed compute units if the task runs in a serverless runtime environment.
firstErrorMessage
String
Error message, if any, that is associated with the job.
GET request example
Use this sample as a reference to get the code task job status.
An error status GET response returns the code task job status, similar to the following example:
{ "status": "FAILED", "startTime": "2022-03-10T17:23:29.000", "updateTime": "2022-03-10T17:23:34.000", "endTime": "2022-03-10T17:23:34.000", "instanceName": "Demo-2", "assetName": "Demo", "runId": 2, "orgId": "4nuRA6NIsI6lvUBYamL76P", "startedBy": "coder@examplecompany.com", "runtimeEnvId": "01000025000000000003", "codeTaskId": 477, "errorMessage": "WES_internal_error_Failed to start cluster for [01000025000000000003]. Error reported while starting cluster [404 {\"code\":\"CONFIG.NOT_FOUND_id\",\"message\":\"Internal error. Cannot find an advanced configuration with ID 01000025000000000003. Contact Informatica Global Customer Support.\",\"debugMessage\":\"Internal error. Cannot find an advanced configurati[truncated]. For more information about the failure, check the application log.If the problem persists, contact Informatica Global Customer Support.", "sessionLogUrl": "logservice/api/v1//jobs/6pqqt9KTgi3l9jyVnAajei/logs", "agentJobLogUrl": null, "advancedLogLocation": null, "advancedLogDownloadUrl": null }
Cancel a code task
Use this PUT request to cancel the code task job.
PUT request
To cancel a code task job, use the code task job ID. The code task job ID is included in the response when you start a code task. For more information, see Start a code task.
Use the following URI to cancel a code task job:
PUT <server URL>/disnext/api/v1/CodeTask/Cancel
Use the following field in the PUT request:
Field
Type
Required / Optional
Description
jobId
String
Required
The code task job ID from the start resource.
PUT response
Returns the code task cancellation state for the requested job ID.
Returns the error object if errors occurred.
PUT request example
Use this sample as a reference to cancel a code task job.
A successful PUT response returns the code task job cancellation information, similar to the following example:
{ "state": "Code Task marked for cancellation.", "jobId": "6pqqt9KTgi3l9jyVnAajei" }
Session logs for a code task
Use the jobs resource to access the code task session logs.
GET request
To access the session logs of a code task, use the code task job ID. The code task job ID is included in the response when you start a code task. For more information, see Start a code task.
Use the following URI to get code task session logs:
GET <server URL>/logservice/api/v1/jobs/<Code task job ID>/logs
You can use the Postman Send and Download option.
GET response
Returns a link to the code task ZIP file for the requested job ID.
Returns the error object if errors occurred.
GET request example
Use this sample as a reference to access code task session logs.
Use the jobs resource to access the Spark task results for the code task.
GET request
To access the Spark task results of a code task, use the code task job ID. The code task job ID is included in the response when you start a code task. For more information, see Start a code task.
Use the following URI to access code task Spark task results:
GET <server URL>/logservice/api/v1/jobs/<Code task job ID>/callback
GET response
Returns the code task Spark task details for the requested job ID.
Returns the error object if errors occurred.
GET request example
Use this sample as a reference to access code task Spark task results.