Monitor > Data Integration job log files > Viewing log files for advanced cluster subtasks

Viewing log files for advanced cluster subtasks

When you monitor a subtask that runs on an advanced cluster, you can view the execution plan, the session log, the agent job log, and the Spark driver and Spark executor logs. If an initialization script runs on the cluster, you can also view the init script logs and cloud-init logs.

To access a specific log file for the job, you might have to download the log from Monitor or from your cloud platform.

The following table indicates whether you can download each log from Monitor, your cloud platform, or both:

Log	Download location
Execution plan	Monitor
Session log	Monitor
Agent job log	Monitor, cloud platform¹
Spark driver log	Monitor, cloud platform¹
Spark executor log	Monitor², cloud platform¹
Initialization script log	Cloud platform
Cloud-init log	Cloud platform
Spark event log	Cloud platform
1 If you use a serverless runtime environment, downloading from your cloud platform is not available. 2 If you do not use a serverless runtime environment, downloading from Monitor is not available. Use the displayed log location to download the log from your cloud platform.

Downloading the execution plan and session log

You can download the execution plan and the session log from the job results in Monitor.

Downloading advanced logs

You can download the agent job log and the Spark driver log in the Spark task details in Monitor. You can also copy the advanced log location and navigate to the location on your cloud platform. You must navigate to the advanced log location to download Spark executor logs.

If you use a serverless runtime environment, you can download the advanced logs from Monitor. The Download option becomes available when the job completes.

Downloading init script and cloud-init logs

To download an initialization script log or a cloud-init log, navigate to the following location on your cloud platform:

<Log location>/ClusterLogs/<cluster ID>/<timestamp>/<cluster node instance ID>/

The log location is the location for log files that is specified in the advanced configuration.

Note: You can download cloud-init logs only in an AWS environment.

Downloading Spark event logs

To download the Spark event log, navigate to the following location:

<Log location>/eventLogs/atscaleagent

The log location is the location for log files that is specified in the advanced configuration.

If the job fails to start, you see the following sample text in the event log file:

{"Event":"SparkListenerLogStart","Spark Version":"3.0.0"}

For more information about the job failure, see the Spark driver log.

Rules and guidelines

Refer to the following rules and guidelines when you view log files:

•If you download the Spark driver and agent job logs while the job is running, the files reflect the current state of the job.
•If you view a log file for a subtask that runs on an advanced cluster with a mapping name that begins with a number or a space, the mapping name is prefixed with an underscore in the log file.

For example, when a log file references a mapping named 123Mapping, the log files use the name _123Mapping.