Monitoring Data Integration Jobs > Data Integration job log files > Viewing log files for advanced cluster subtasks
  

Viewing log files for advanced cluster subtasks

When you monitor a subtask that runs on an advanced cluster, you can view the execution plan, the session log, the agent job log, and the Spark driver and Spark executor logs. If an initialization script runs on the cluster, you can also view the init script logs and cloud-init logs.
To access a specific log file for the job, you might have to download the log from Data Integration or from your cloud platform.
The following table indicates whether you can download each log from Data Integration, your cloud platform, or both:
Log
Download location
Execution plan
Data Integration
Session log
Data Integration
Agent job log
Data Integration, cloud platform¹
Spark driver log
Data Integration, cloud platform¹
Spark executor log
Data Integration², cloud platform¹
Initialization script log
Cloud platform
Cloud-init log
Cloud platform
Spark event log
Cloud platform
1 If you use a serverless runtime environment, downloading from your cloud platform is not available.
2 If you do not use a serverless runtime environment, downloading from Data Integration is not available. Use the displayed log location to download the log from your cloud platform.

Downloading the execution plan and session log

You can download the execution plan and the session log from the job results in Data Integration.

Downloading advanced logs

You can download the agent job log and the Spark driver log in the Spark task details in Data Integration. You can also copy the advanced log location and navigate to the location on your cloud platform. You must navigate to the advanced log location to download Spark executor logs.
If you use a serverless runtime environment, you can download the advanced logs from Data Integration. The Download option becomes available when the job completes.

Downloading init script and cloud-init logs

To download an initialization script log or a cloud-init log, navigate to the following location on your cloud platform:
<Log location>/ClusterLogs/<cluster ID>/<timestamp>/<cluster node instance ID>/
The log location is the location for log files that is specified in the advanced configuration.
Note: You can download cloud-init logs only in an AWS environment.

Downloading Spark event logs

To download the Spark event log, navigate to the following location:
<Log location>/eventLogs/atscaleagent
The log location is the location for log files that is specified in the advanced configuration.
If the job fails to start, you see the following sample text in the event log file:
{"Event":"SparkListenerLogStart","Spark Version":"3.0.0"}
For more information about the job failure, see the Spark driver log.

Rules and guidelines

Refer to the following rules and guidelines when you view log files: