Database Ingestion and Replication > Database Ingestion and Replication sources - preparation and usage > PostgreSQL sources
  

PostgreSQL sources

To use PostgreSQL sources in database ingestion and replication tasks, first prepare the source database and then review the usage considerations.

Source preparation

To use PostgreSQL sources in database ingestion and replication tasks, complete the following source preparation tasks.
On the Secure Agent system, install the ODBC driver that is appropriate for your operating system.
Note: If you use the PostgreSQL ODBC driver, use psqlODBC driver version 13.x or earlier. Beginning with psqlODBC 15.x, the PostgreSQL ODBC driver transitioned to using OpenSSL 3.0.x, which is incompatible with the OpenSSL 1.1.x version that Data Ingestion and Replication uses.
On the PostgreSQL database system, perform the following configuration steps:
  1. 1For incremental load and initial and incremental load jobs, ensure that the PostgreSQL postgresql.conf configuration file specifies the wal_level=logical parameter.
  2. This parameter determines how much information PostgreSQL writes to the Write-Ahead Log (WAL). The setting of logical adds information that is required to support logical decoding.
    To set wal_level to logical on Amazon Aurora PostgreSQL or Amazon RDS for PostgreSQL sources, set the rds.logical_replication parameter to 1 in the cluster parameter group. For Azure Database for PostgreSQL - Flexible Server, set the wal_level parameter to logical on the Server Parameters page in the Azure portal.
    For Cloud SQL for PostgreSQL sources, perform the following actions:
    1. aConnect to the database by using the public IP.
    2. Note: Ensure that you add the required IPs under Authorized networks in the Google Cloud console.
    3. bCreate a Cloud SQL for PostgreSQL database instance replica.
    4. cIn the Cloud Shell, as a local admin user, run the following commands:
    5. alter database postgres set default_transaction_read_only = off;
      gcloud sql connect database_replica --user=postgres --quiet;
      ALTER USER postgres WITH REPLICATION;
      CREATE USER replication_user WITH REPLICATION IN ROLE cloudsqlsuperuser LOGIN PASSWORD 'password';
      ALTER USER postgres WITH REPLICATION;
    6. dIn the Google Cloud console, add the following database flags:
    7. eRestart the database instance.
  3. 2If you use the DataDirect ODBC for PostgreSQL driver, ensure that the database does not use the SCRAM-SHA-256 authentication method. Use another authentication method, such as MD5.
  4. Note: The PostgreSQL ODBC driver supports the SCRAM-SHA-256 authentication method. In PostgreSQL 13, this authentication method became the default method.
  5. 3To deploy and run a database ingestion and replication task that includes a PostgreSQL source, the source connection must specify a database user who has the required privileges. Create the user and grant privileges to that user in the following ways:
  6. 4If you plan to use the wal2json plugin for logical decoding output for incremental load or initial and incremental load jobs, install the plugin.
  7. 5If you plan to use the pgoutput plugin for incremental load or initial and incremental load jobs, use the following SQL statement to create publications for database ingestion jobs:
  8. CREATE PUBLICATION publication_name [FOR TABLE [ONLY] table_name [*] [,...] | FOR ALL TABLES ];
    Ensure that the publication includes all tables that you want to replicate to the target.
  9. 6For incremental load and initial and incremental load jobs, use the following function to create a PostgreSQL logical replication slot:
  10. SELECT pg_create_logical_replication_slot('slot_name', 'plugin_type');
    Where the plugin_type is either the pgoutput plugin or the wal2json plugin.
    Note: If you capture change data from a PostgreSQL replica database, ensure that you create the replication slot on the replica.
  11. 7For incremental load and initial and incremental load jobs, ensure that the max_replication_slots parameter in the postgresql.conf configuration file has a value greater than or equal to the number of concurrent database ingestion jobs that you plan to use.
  12. Important: All replication slots must be unique across all concurrent jobs.
  13. 8For incremental load and initial and incremental load jobs, ensure that the PostgreSQL sources use the UTF-8 encoding.
  14. 9Ensure that the PostgreSQL sources use the UTF-8 encoding. If you use another encoding for the source database, your initial load, incremental load, and combined initial and incremental load jobs might fail.

Usage considerations