Audit Trail Configuration

Prerequisite

Before you can start with this chapter, you need to have finished the following parts:

Configure Audit Trail in the Product 360 Application

The Audit Trail feature is delivered as part of the Server and Desktop Client package. By default the Audit Trail functionality is disabled. To enable this feature you have to configure the following properties in <PIM_SERVER_INSTALLATION_ROOT>\configuration\HPM\server.properties file:

Enable Audit Trail

General settings

Mandatory

system.name

images/s/o7yjop/8703/51k4y0/_/images/icons/emoticons/check.svg

This is a mandatory property. It specifies the name of the system, e.g. Test System /Productive System / Demo / Poad, etc.
Blanks will be replaced with _. Best practice: use 0-9A-Za-z.-

audittrail.rest.url

images/s/o7yjop/8703/51k4y0/_/images/icons/emoticons/check.svg

URL path to the Elasticsearch REST server.

E.g. audittrail.rest.url = http://localhost:9200,http://localhost:9201

audittrail.rest.user

images/s/o7yjop/8703/51k4y0/_/images/icons/emoticons/error.svg

Login name of the Elasticsearch REST server.

audittrail.rest.password

images/s/o7yjop/8703/51k4y0/_/images/icons/emoticons/error.svg

Login password of the Elasticsearch REST server.

audittrail.rest.allow.self-signed.certificate

images/s/o7yjop/8703/51k4y0/_/images/icons/emoticons/error.svg

Allows self-signed certificate only if you use https.

audittrail.mode

images/s/o7yjop/8703/51k4y0/_/images/icons/emoticons/check.svg

The Audit Trail can be set up with the below modes

Mode

PRODUCTION

(Recommended) All Audit Trail data will remain intact even if Product 360 - Server is restarted.

CLEAN_SLATE

(Only for testing, use with caution) All Audit Trail data will be erased when Product 360 - Server is restarted.

DISABLED

The Audit Trail will be disabled.
This setting is only for development environments! With disabled audit trail the delivery
of trigger messages to Apache Active MQ is not guaranteed. The audit trail documents are used in failover
or high-load/overload scenarios to guarantee that they are delivered.

audittrail.installation.type

images/s/o7yjop/8703/51k4y0/_/images/icons/emoticons/check.svg

The Audit Trail can have below installation types

Installation type

Configuration folder

elastic-standalone

conf/audittrail/elastic-standalone

On-premise installation of Elasticsearch for PROD

elastic-aws

conf/audittrail/elastic-aws

AWS Elasticsearch Service

elastic-test

conf/audittrail/elastic-test

On-premise installation of Elasticsearch for DEV, QA

audittrail.threadpool.size

images/s/o7yjop/8703/51k4y0/_/images/icons/emoticons/check.svg

This property should have a value that is the same as db.default.pool.maxPoolSize

audittrail.backup.restoration.mode

images/s/o7yjop/8703/51k4y0/_/images/icons/emoticons/error.svg

This setting will synchronize the Product 360 records in the relational database and their corresponding Audit Trail data in Elasticsearch.

Default: false

NOTE: Set to "true" only when Product 360 - Server starts after recovering from a disaster.

For AWS managed Elasticsearch

policy_id in template is deprecated in the AWS Elasticsearch update of April 2021 which is used by Product 360 for Index Lifecycle Management. To make the application compatiable, the below flag is introduced in the plugin_cusotmization.ini file

com.heiler.ppm.audittrail.server/auditTrail.aws.policyIdDeprecated=true

This should only be made false in case April 2021 update is not applied to AWS Elasticsearch.

Note : This flag is introduced from Product 360 versions 10.1.0.00.12, 10.1.0.01.05, 10.1.0.02.01

Configure Audit Trail view

To limit the amount of data displayed, there are the following properties in <PIM_SERVER_INSTALLATION_ROOT>\configuration\HPM\plugin_customization.ini file.

Property

Value

com.heiler.ppm.audittrail.server/historyDetailTab.pageSize

The count of documents to be displayed initially in the history detail tab for single/multi selected objects or audit trail search results, default is 5

com.heiler.ppm.audittrail.server/historyDetailTab.content.maxChildren

The maximum number of child nodes within a parent node in the history detail tab, default is 70.

Start Product 360 Server

In case of successful Audit Trail setup you should see the following log messages for each root entity which is configured for Audit Trail:

  • Lifecycle for entity '<entity identifier>' is running in Elasticsearch

Repository Based Configuration

Short Identifier

In order to generate and store the changes in the entity item change document, it is important to provide the Short Identifier for entities and fields.

Audit Trail Settings

Audit Trail can be activated for any root entity in the repository. An activation/deactivation on the sub-entity level is not supported. The setting of the root entity will always be used, independent of the setting for the sub-entities.

  1. Open the repository editor,

  2. Go to Custom section and select a root entity

  3. If "Audit trail settings" is not already available for that entity then right-click → select "New child" → click "Audit trail settings"

  4. Audit trail settings have the below properties

    Property

    Value

    Datasource

    Datasource of Entity, which can be any of the following - MAIN, MASTER, or SUPPLIER

    Documentation

    Retention Policy

    There are 3 types of retentions can be chosen from the drop-down

    Retention type

    NO_RETENTION

    If no auditing is needed for the entity. No change information will be recorded in the Elasticsearch for this entity

    LONG_RETENTION

    If auditing is needed to be stored for the entity for a considerable amount of time, e.g. a few years.

    The longterm_policy and longterm_template from the audittrail configuration folder will be used to create policy and index

    SHORT_RETENTION

    If auditing is needed to be stored for the entity for a relatively short amount of time, e.g. a few months.

    The shortterm_policy and shortterm_template from the audittrail configuration folder will be used to create policy and index

    Revision setting

    Below revision settings are available to choose from for auditing

    Revision

    ALL_REVISIONS

    All revisions changes will be recorded in the Audit Trail

    HEAD_REVISION_ONLY

    Only head revisions changes will be recorded in the Audit Trail

    NON_HEAD_REVISION_ONLY

    All the revisions changes except head revision will be recorded in the Audit Trail

    Use Own Index

    Below values can be set for this property

    Own index

    true

    This will create a separate index in Elasticsearch for this particular entity

    false

    This will use a shared index in Elasticsearch for multiple entities.

Configuring Audit Trail for entities is a manual process.

Customers migrating to this version must manually configure the "Audit trail settings" for each entity.

Supports Audit Trail Property

If this flag is set to false for a field/entity then any changes done on the field/entity are not recorded in the Audit Trail. However a corresponding change document is stored recording the transaction but with empty change summary.

  • Entity level : If set at an entity level then all it's sub-entities and fields also inherit this behavior.

  • Field level : If set at a field level then only that field is affected. Any change done with respect to this field is not recorded in change summary of the document.

Index policies and templates

Elasticsearch provides OOB functionality to handle index lifecycle management which helps indices to automatically rollover in different phases after defined durations. This lifecycle management makes it easier to manage indices in hot-warm-cold architectures.

Audit Trail policies and templates are available at the following location in <PIM_SERVER_INSTALLATION_ROOT>\configuration\HPM\audittrail.

There are folders available for each audittrail.installation.type configured in server.properties. E.g. Folder elastic-standalone will have all the configurations for installation type elastic-standalone.

Based on the repository configuration for each entity, policy and template will be applied to particular indices.

Lifecycle Policies

There are shortterm_policy and longterm_policy in Product 360.

Each has a different configuration for the number of days an index will be in one phase. These can be configured based on customer needs and Product 360 will take care of applying these for indices in Elasticsearch after Product 360 server restart.

Each phase is explained below -

Phases

Hot

This phase is used for both creation and search of data by the Audit Trail.

The index will be in the hot phase once created and stay in this phase until rollover. Only hot indices are writable indices.

When indices rollover, there is a potential chance that indices might be in an inconsistent state.

To make sure indices never go to an inconsistent state, Product 360 has a job running to rollover indices from the hot phase to the warm phase.

Other phases are taken care of by Elasticsearch lifecycle management.

This rollover job is deprecated from Product 360 versions 10.1.0.00.08, 10.1.0.01.03, 10.1.0.02.00

Rollover of indices from the hot phase to warm phase using the below settings -

  • Default policy rollover is set to 60 days in policy template but the actual rollover happens based on plugin_customization.ini settings

    • com.heiler.ppm.audittrail.server/rolloverWatchdog.longPolicyRolloverInDays=30

    • com.heiler.ppm.audittrail.server/rolloverWatchdog.shortPolicyRolloverInDays=30

These rollover settings are deprecated from Product 360 versions 10.1.0.00.08, 10.1.0.01.03, 10.1.0.02.00

Policy rollover time must be more than double of rollover time given in plugin_customization.ini so that Product 360 has sufficient time to attempt a proper rollover of the index.

If Product 360 is unable to perform the rollover, then the default rollover defined in the Elasticsearch policy will execute.

Warm

This phase is used only for search by the Audit Trail. The index will be in a warm phase once the rollover is complete.

We can define actions to be taken in the warm phase for changing shards and replicas and also to shrink the index.

Delete

Delete phase deletes the index once it reaches the time defined in min_age.

Customers should set the min_age of this phase based on how long they want to retain Audit Trail data of a particular entity.

For on-prem Elasticsearch installation, in every phase, the min_age means the index will wait for that amount of time in the previous phase before entering the current phase.

For example, if the min_age in the delete phase has 10d then the index will be in the warm phase for at least 10 days and then it will be deleted.

Read more at https://www.elastic.co/guide/en/elasticsearch/reference/6.8/_timing.html

Index Templates

There are 3 index templates in the Audit Trail configuration folder. Each has index settings JSON which contains configurations like number_of_shards, number_of_replicas, etc.

template

longterm_template

This defines the index settings for the LONG_RETENTION entity based on the repository's "Audit trail settings".

shortterm_template

This defines the index settings for the SHORT_RETENTION entity based on the repository's "Audit trail settings".

migration_template

If the customer migrates data from the old Audit Trail to the new Audit Trail, a new index will be created using this template configuration.

Configure Audit Trail logs in Product 360

Logs for different components of the Audit Trail can be enabled in log4j2.xml located at <PIM_SERVER_INSTALLATION_ROOT>\configuration\HPM\log4j2.xml

All components can be configured to have different logging levels based on need. Below are the components.

Audit Trail modules

AUDIT_TRAIL

Any logging level set to this component will apply to all the other components of the Audit Trail. By default, this is set to INFO.

AUDIT_TRAIL.LIFECYCLE

This component will show logs of all the activities related to the lifecycle like index creation, index rollover, etc.

AUDIT_TRAIL.WATCHDOG

This component logs activity related to internal Audit Trail jobs which synchronizes data between Elasticsearch and RDBMS.

AUDIT_TRAIL.PERSISTENCE

This component logs activity related to the persistence of data in the Audit Trail

AUDIT_TRAIL.SEARCH

This component logs activity related to the search of data from the Audit Trail

Example of enabling lifecycle component debug logs
<Logger name="AUDIT_TRAIL.LIFECYCLE" level="DEBUG" additivity="false">
<AppenderRef ref="AuditTrailAppender" />
<AppenderRef ref="StdFileAppender" />
</Logger>