Repository

Please ensure the repository has all needed audit trail configurations in place. This is important because only data for audit trail enabled root entities will be migrated, all other data will be skipped.

In addition to the root entity audit trail settings, the "Supports Audit Trail" setting for repository fields will also be checked. Only audit trail enabled fields and logical keys will be migrated.

Furthermore, the mandatory attribute "short identifier" must have a valid value for all active entities, logical keys and fields. You can check your repository by starting a validation in the repository editor.

audittrail.migration.server.properties

The access to the legacy audit trail database should be configured in <PIM_SERVER_INSTALLATION_ROOT>\configuration\HPM\audittrail\migration\audittrail.migration.server.properties file.
This file can be created from a corresponding template file. The essential settings here are the host and schema configuration. You can take the settings from the old configuration for audit trail.

plugin_customization.ini

In order to use the migration you have to configure the time period for which you want to migrate changes. In addition to that, you can configure system resources to be used. The affected configuration file is <PIM_SERVER_INSTALLATION_ROOT>\configuration\HPM\plugin_customization.ini.

migration_template.json

In addition to the audit trail index policies and templates used for Elasticsearch in <PIM_SERVER_INSTALLATION_ROOT>\configuration\HPM\audittrail, there is a similar template that is used for audit trail migration indexes, migration_template.json. For details, please refer to the Audit Trail Configuration documentation, chapter "Index policies and templates".

Parallel to normal P360 server operation

The first option is to start the migration after the P360 system has been updated. Because the migration first processes the latest data and then goes backwards in time, the data most likely to be needed is available almost immediately. It is important to be aware that not all of the system resources should be used for the migration, otherwise the P360 server will not run smoothly for your everyday tasks. To limit the system resources used, you can configure the values for the attribute migration.maxThreads and migration.fetch.size, see .AuditTrail Migration v10.1#plugin_customization.ini.
Alternatively, for a multi-server system, one of the job servers could be used exclusively for the migration. Please have a look at the priority job server feature (available since10.1.0.00.05). With this feature you can define certain jobs as priority jobs, so those will only be executed on the priority job servers.

In advance of system upgrade

The second option is to do the migration in advance of the system upgrade. For this purpose, a separate P360 system (including a copy of the database) is needed, for example the QA or test system. As the migration would be the only process running on that system, you can use all available system resources for it, see .AuditTrail Migration v10.1#plugin_customization.ini.

Step 1

Determine the number of root entity changes to be migrated totally and the number of root entity changes to be migrated in a time frame (e.g. first month). Use following statements:

Step 2

Analyze the migration protocol for the same time frame as in SQL statement above and determine the duration of the migration process.
In our example it took 133 minutes.

Step 3

Determine the total duration using the following formula

time_total = root_entities_total * time_for_migration_in_time_frame/root_entities_in_time_frame

Example

In our example migration we determined following numbers:

Delete migrated data

In order to remove the already migrated entries from elastic you have to delete the corresponding migration indexes.

The index names follow a pattern, dependent on the settings in the repository. The migration indexes have the same names, but with the suffix '_migrate', e.g.

• Changes of products of the Master catalog are stored in <system_name>.audit_product2g_master_migrate

• Changes of items of supplier catalogs are stored in <system_name>.audit_article_supplier_migrate

• Changes of other objects are usually stored in <system_name>.audit_longterm_main_migrate

You can find the system name in the server.properties file.

To delete the indexes, either use an interface for elastic like kibana or use the following commands:

Related CURL commands are:

To find out related indices in Elasticsearch you can use:

You have to replace: server, port and user with Elasticsearch URL and credentials.

For further information see Elasticsearch Delete index API.

Delete progress information

Finally, remove the progress.json file from the subfolder audittrailMigration of the configured shared folder and you can try again. The P360 server doesn't need to be restarted.

Decreasing the memory usage

There is a possibility to decrease the memory consumption of the audit trail migration process by adjusting the value of com.heiler.ppm.audittrail.migration.server/migration.fetch.size, see .AuditTrail Migration v10.1#plugin_customization.ini.

Please note that the time of migration will increase, if you decrease the fetch size.

Decreasing the number of threads

Retrieving and processing migration data from the old audit trail database is the crucial part of the audit trail migration process, so all available CPU cores on the audit trail migration server are used by default. If the audit trail migration is running in parallel with a running P360 system on the same machine, the number of threads used by the migration must be limited to ensure fluid use of the P360 system.

This can be done by changing the property com.heiler.ppm.audittrail.migration.server/migration.maxThreads, see .AuditTrail Migration v10.1#plugin_customization.ini.

Adjustments of elastic settings (Exception: 'Unable to parse response body')

There can be huge change documents created and stored in Elasticsearch, e.g. during migration of changes made by import or merge.

To avoid HTTP/1.1 413 Request Entity Too Large errors that are logged to the migration protocol as "Exception during migration: Unable to parse response body" errors, add and adjust the setting http.max_content_length in the Elasticsearch configuration. The default value is 100MB which might not be enough for some migration cases.

Please note: Before you make any changes, you should always read the Elasticsearch Installation documentation.

If PIM runs in a hosted environment like AWS, an AWS ES limitation can cause such issue. e.g. m5.large.elasticsearch instance has a Maximum Size of HTTP Request Payloads of 10MB. That may be not enough for any audit trail migration request to Elasticsearch.

See 'Network limits' in https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html

Elasticsearch Exception (type=circuit_breaking_exception)

If the audit trail migration stops with an Elasticsearch exception with type=circuit_breaking_exception and reason=Data too large, data for [<http_request>] ...

then the memory settings (-Xmx) in Elasticsearch config/jvm.options file is too low.

Please note, if the memory for Xmx is > 1GB and you are using JDK8 on windows, then you should use an odd number (1, 3, 5... GB) as value (https://github.com/elastic/elasticsearch/issues/47384#issuecomment-543952442).

Elasticsearch Exception (type=illegal_argument_exception)

If the audit trail migration stops with an Elasticsearch exception with type=illegal_argument_exception and reason=all shards failed,
please look for any nested exceptions. If the nested exception is also an Elasticsearch exception with type=search_phase_execution_exception, please have a look to the section below.

Elasticsearch Exception (type=search_phase_execution_exception)

If the audit trail migration stops with an Elasticsearch exception with type=search_phase_execution_exception and reason=Result window is too large, from + size must be less than or equal to ...
then the Elasticsearch setting max_result_window has to be increased.

The maximum number of search results for an index is limited to 10000 by default. We recommend to use 50000 for this property. But be careful, search requests take heap memory and time proportional to max_result_window value, so it should not be increased too much.

You can change the value for the corresponding index directly, for example using Kibana (Management → Index Management → Edit settings) or by using the update index settings API of Elasticsearch (see https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html).

Preparation

During the database setup you specify a server.properties file. This usually is the one you use for your sever.
However, if you want to update the object identifiers in the audit trail migration indexes, you have to prepare a special server.properties file. The best way is to copy the original server.properties file and adjust it.
The following changes will cause the Database Setup Client to run in "single version only" mode, which means that only one database update will be processed. If it is finished successfully, the corresponding version will be written to the database so that the same update script will not run a second time. (As this update is optional, your server will startup without the update as well.)

Add the following lines to your copy of the server.properties file:

• For MS SQL Server

db.update.only.schema = MAIN
db.update.only.version = 794

• For Oracle

db.update.only.schema = MAIN
db.update.only.version = 795

Starting the database setup

Now you can start the Database Setup Client and choose the adjusted server.properties file.

Logging

By default the logging will only tell you that the update is executed.

You can get more information if you adjust the settings in the log4j2.xml file of the Database Setup Client by setting the log level to DEBUG:

The log will contain the names of the indexes that were found, the number of processed documents and the task id of the task updating the documents in elasticsearch.

If you want to get more information about the progress of the update, you can activate the performance log:

In this mode, a log entry will be written every five minutes that provides information about the number of documents that have already been processed.

Whenever elasticsearch has finished processing a single index, you will see a summary in the log as well.

Further information

Please note that the duration of the update depends on the number of documents in the migration indexes. For our tests we found that 100000 documents take around one minute to be processed. The overall time of the update is determined by the biggest index because all indexes are processed in parallel.

Not touched documents mean these documents match our query for the update but couldn't be processed by the script to correct the identifier because it is something really unusual we cannot handle (e.g 'item'@'123'@'MASTER'). These identifiers won't be changed.

Technical details

For developers, there is a way to further enhance the log. Set com.heiler.ppm.dbsetup.core.PERF to log level TRACE.

For the index updates this will add the average duration for 100000 documents. For the finish summary of each index it will add the average duration of 100000 processed documents as well as the number of updated and not touched documents. Note that for technical reasons the number of updated documents in the finish summary is the number from the last update, not the total number of processed documents.

You can use the Kibana Console to execute the following REST request to see the status of the tasks in elasticsearch: