External Hadoop Cluster Deployment

• •Solr version 5.2.1
• •HBase version 0.98
• •Scanner components

Prerequisites for the External Cluster

• •OpenSSL version on the cluster nodes is openssl-1.0.1e-30.el6_6.5.x86_64 or later. Make sure that you do not use versions in the 1.0.2 branch.
• •Ensure that you install JDK 1.8 on all cluster nodes.
• •Verify that the secure path in the /etc/sudoers file has the /usr/bin directory location at the start.
• •On each host machine, verify that you have the zip and unzip utilities available.
• •You have the Read, Write, and Execute permissions for owners, groups, and others on HDFS directories.
• •Verify that the maximum number of open file descriptors is 10,000 or more. Use the ulimit command to verify the current value and change the value if required.
• •When you create the Catalog Service that connects to an SSL-enabled external cluster, verify that you configure the following properties:
• - A keytab file that contains all the users in LDAP.
• - Kerberos domain name.
• - HDFS namenode and YARN Resource Manager service principals
• - Path to Solr keystore file and password.
• - Import the Hadoop cluster certificates to the Informatica domain truststore.
• •Before you deploy Enterprise Data Catalog on clusters where Apache Ranger is enabled, make sure that the Informatica domain user has the required permission to submit applications to the YARN queue.
• •If the cluster is enabled for SSL, it is recommended to enable SSL for the Informatica domain and the Catalog Service.
• •Verify that you install the following prerequisite packages before you enable Kerberos:
• - krb5-workstation
• - krb5-libs
• - krb5-auth-dialog
• •Create the service-logs directory under /informatica/ldm/<service cluster name>/ and assign the ownership of the directory to the service cluster user if the cluster is enabled for Kerberos.
Note: If the cluster is not enabled for Kerberos, create the service-logs directory under /informatica/ldm/<domain user name>/ and assign the ownership of the directory to the domain user.
• •If the cluster is not enabled for Kerberos. create the directory <domain user name> under /user and assign the ownership of directory to the domain user.
Note: If the cluster is enabled for Kerberos, create the directory <service cluster name> under /user and assign the ownership of the directory to the service cluster user.
• •Ensure that you do not create the Informatica domain on a node in the existing Hadoop cluster.
• •If you want to enable Kerberos authentication for Enterprise Data Catalog deployed on a multi-node Informatica domain, make sure that you complete the following prerequisites:
• - Make sure that all the domain nodes include the krb5.conf file in the following directories:
• ▪ $INFA_HOME/services/shared/security/
• ▪ /etc/
• - Make sure that the /etc/hosts file of all cluster nodes and domain nodes include the krb hosts entry and a host entry for other nodes.
• - Install krb5-workstation in all domain nodes.
• - Make sure that the keytab file is present in a common location on all domain nodes.
• •If you want to enable SSL authentication for Enterprise Data Catalog deployed on a multi-node Informatica domain, make sure that you complete the following prerequisites:
• - Export the Default.keystore of each node to the infa_truststore.jks on all nodes.
• - Make sure that the Default.keystore is unique for each host node.
• - Copy the Default.keystore to a unique location of each node.
• - If Informatica Cluster Service and Catalog Service are on different nodes, then export the Apache Ambari server certificate to the infa_truststore.jks on all nodes.

Preparing the External Hadoop Cluster Environment

• •Create the following directories in HDFS before you create the Catalog Service:
• - /Informatica/LDM/<service cluster name>
• - /user/<user name>
Where <service cluster name> is the name of the service cluster that you need to enter when you create the Catalog Service and <user name> is the username of the Informatica domain user.
• •Make <user name>who is the Informatica domain user is the owner of the /Informatica/LDM/<service cluster name> and /user/<user name> directories.

Kerberos and SSL Setup for an External Cluster

Prerequisites for SSL Authentication

• •Informatica domain is configured in the SSL mode.
• •The cluster and YARN REST endpoints are Kerberos-enabled.
• •Create a keystore file for the Apache Solr application on all nodes in the cluster. Import public certificates of Apache Solr keystore files on all the hosts into all the truststore files configured for HDFS and YARN. This step is required for Apache Spark and scanner jobs to connect to the Apache Solr application.
• •Import the public certificates of Apache Solr and YARN applications into the truststore file of the Informatica domain. This step is required for Catalog Service to connect to YARN and Solr applications.
• •Import the public certificates of Informatica domain and the Catalog Service into the YARN truststore.
• •Import the public certificate of the Catalog Service into the Informatica domain truststore.
• •If you plan to deploy Enterprise Data Catalog on an existing Hortonworks version 2.5 cluster that does not support SSL authentication, perform the following steps:
1. 1. Configure the following properties in the /etc/hadoop/conf/ssl-client.xml file: ssl.client.truststore.location and ssl.client.truststore.password.
2. 2. Ensure that the ssl.client.truststore.location value is set to /opt directory and not /etc directory. Verify that you configure the full path to the truststore file for the ssl.client.truststore.location property. For example, you can set the value similar to /opt/truststore/infa_truststore.jks.
3. 3. Export the keystore certificate used in the Informatica domain.
4. 4. Import the keystore certificate into the Informatica domain truststore file.
5. 5. Place the domain truststore file in all the Hadoop nodes in the /opt directory. For example, /opt/truststore/infa_truststore.jks.
6. 6. Open the /etc/hadoop/conf/ssl-client.xml file.
7. 7. Modify the ssl.client.truststore.location and ssl.client.truststore.password properties.

Prerequisites for Kerberos Authentication

• •Create the following users in the LDAP security domain where <user name> is the service cluster name.
• - <user name>@KERBEROSDOMAIN.COM
• - <user name>/<hostname>@KERBEROSDOMAIN.COM
Note: Create the user ID for all the hosts in the cluster.
• - HTTP/<host name>@KERBEROSDOMAIN.COM
Note: Create the user ID for all the hosts in the cluster.
• - Create a keytab file with credentials for all these users created in LDAP. You can create keytab files for each one of the users in KDC server and merge them using the ktutil command to create single keytab file.
• - Create the following folders in HDFS that Enterprise Data Catalog uses as data directories for the Catalog Service: /Informatica/LDM/<user name> and /user/<user name>.
• - Change the owner of these two folders to <user name>.
• - Create a local user with username as <user name> on all the hosts in the cluster. This step is required to launch the application on YARN as the user configured for the Catalog Service.
• •Set up the udp_preference_limit parameter in the krb5.conf Kerberos configuration file to 1. This parameter determines the protocol that Kerberos uses when it sends a message to the KDC. Set udp_preference_limit = 1 to always use TCP. The Informatica domain supports only the TCP protocol. If the udp_preference_limit parameter is set to any other value, the Informatica domain might shut down unexpectedly.