This article describes how to rebuild the censhare embedded database (cdb). the description is based on using a Linux operating system.
General information
A cdb rebuild is necessary if:
the configuration of the cdb or their indexes are changed
new calculated features are added
In case of no production critical issue, schedule the rebuild outside the main working hours.
The cdb rebuilds on basis of the oracle database. ("currversion" of table "asset" - triggered by the table asset_ccn_counter)
Do not update the jvm/cdb configuration while censhare Server is running at main working hours
Important hint: deactivate the automatic CCN sync in censhare Admin-client during cdb rebuild Configuration|Modules|Administartion|Embedded Database|Embedded database sync using CCNs (automatic)
Important hint: Make sure to not cross any do deletion or archiving jobs while rebuilding or do a server shutdown otherwise your cdb will be incomplete and you have to start the rebuild from scratch.
Important hint: Since censhare 4.6 the old formular 2*nodecache + 2*datacache +1GB = JVM max is not valid anymore.
Important hint: Since censhare 4.8 cdb rebuilds can cause the deletion of content of online channel websites.
Reason: Since the new online channel versions use the app servers cdb instead of the oracle database for synchronizing the content of the web servers and the rebuild of the cdb will lead to an initially empty cdb the content on the delivery servers will also be deleted und gradually reappear while the rebuild proceeds.
Solution: Disconnect the rmi connection between the online channel satellite and the app server during the rebuild to prevent the deletion of the content on the web servers. This can be archived by changing the url in the satellite configuration file on the web servers to an invalide domain. Note that the functionality of the web servers might be limited during that time because neither updates can be written (e.g. user generated content) nor can commands be called from the web servers.
Note down the old configuration as you have to change them back after the rebuild.
The embedded DB infrastructure uses these three services:
AssetStoreService
This service manages local storage of assets in cdb
AssetQueryService
This service manages the retrieval of assets from local storage via queries
IIndexService
This service manages direct access to inverted index storage for additional embedded database users like translation and terminology
Disabling the flag “Contentfulltext” to improve speed for cdb rebuilds
The Content (full text) feature indexes the content of asset files (for example, text documents). It can be searched from a field in the General tab of the Detail search in both web client and Javaclient, called “Content (full text)“. To reduce time for the cdb rebuild, it is possible to make a change to this feature which can later berestored:
In the censhare Admin Client, activate the Admin mode.
Open the Master data/Features table and search the censhare:text.content Feature
To open the feature configuration dialog, double click the entry.
At the bottom of the feature configuration dialog, under Custom configuration / Fulltext index disable the flag “Content fulltext”
After the cdb was rebuilt (and synched to remote servers, if applicable)
Re-enable the flag “Content fulltext”
Enable the server action “Embedded database fulltext index sync” under Configuration /Administration / Embedded database on the Master server
Run the server action from the Master server, then from the other servers, enabling and disabling it on each server as you go. Note: It may be required to log on and off again to see the server action on the remote server. Make sure to sync the config across to the Remote servers. Alternatively, it’s also possible to synch the cdb from the Master server once the fulltext index sync is finished.
Increase jvm heap space
Before changing the jvm and cdb settings you need to check for the available RAM.
free -g | head -2 total used free shared buffers cached Mem: 15 4 11 0 0 0
We do have 11GB free RAM, so 10GB for jvm is sufficient.
Adapt the jvm settings
vi ~/cscs/app/config/launcher.$CSS_ID.xml
Search for the active preconfigured jvm settings, marked with "current". For example:
<!-- current: machines with 8 or more threads 64Bit Java, RAM greater-equal 4GB -->
active configuration:
<jvmarg value="-Xms2556m" enabled="true"/> <jvmarg value="-Xmx2556m" enabled="true"/>
new configuration:
<jvmarg value="-Xms10g" enabled="true"/> <jvmarg value="-Xmx10g" enabled="true"/>
Increase data-cache and node-cache
Use up to half of the jvm for data-cache and 1/4 for node-cache.
vi ~/cscs/app/services/assetstore/config.$CSS_ID.xml
A good value for data-cache and node-cache would be 5GB and 2.5GB.
default configuration:
node-cache-size-mb="300" data-cache-size-mb="800"
new configuration:
node-cache-size-mb="2500" data-cache-size-mb="5000"
Save cdb. Set prerequisites
Stop the censhare Server.
censhare.rc stop
Create a working copy by moving the folder aside. Server will automatically create ~work/cdb if it does not exist during server start
mv ~/work/cdb ~/work/cdb.savedbeforerebuild
Disabling the flag “Contentfulltext” to improve speed forcdb rebuilds
The Content (full text) feature indexes the content of asset files (for example, text documents). It can be searched from a field in the General tab of the Detail search in both web client and Javaclient, called “Content (full text)“. To reduce the time for the cdb rebuild, it is possible to make a change to this feature which can later be restored:
In the censhare Admin Client, activate the Admin mode.
Open the Master data/Features table and search the censhare:text.content Feature
To open the feature configuration dialog, double click the entry.
At the bottom of the feature configuration dialog, under Custom configuration / Fulltext index disable the flag “Content fulltext”
After the cdb was rebuilt (and synched to remote servers, if applicable)
Re-enable the flag “Content fulltext”
Enable the server action “Embedded database fulltext index sync” under Configuration /Administration / Embedded database on the Master server
Run the server action from the Master server, then from the other servers, enabling and disabling it on each server as you go. Note: It may be required to log on and off again to see the server action on the remote server. Make sure to sync the config across to the Remote servers. Alternatively, it’s also possible to synch the cdb from the Master server once the fulltext index sync is finished.
Start rebuild
Start the censhare Server, and the cdb is rebuilding itself.
censhare.rc start
Watch rebuild
Monitor the rebuild
tail -F ~/work/logs/server-0.0.log | grep -iE "updater wrote|cdb"
AssetStoreUpdater: AssetStoreService: AssetStoreUpdater: Updater wrote 200 (200 high) updates in 200ms, 5000 tasks in update queue, 64 tasks in buildQueue, 1000 tasks in fulltextBuildQueue, 2594183 events in queue.
Explanation AssetStoreUpdater log entries
Updater wrote 200 (high) updates in 200ms
The first part gives information about how many updates are written in how many milliseconds. Generally, the AssetStoreUpdater needs 1-2ms for 1 update. So 1 update per ms is a good value. The (high) stands for high priority assets, there can be assets with low priority as well. max. value = 200
5000 tasks in update queue
The update queue contains write instructions for the cdb. If this queue is regularly full, it hints for a too low CDB cache-memory or slow hard drive. If the update queue is empty but the event queue is still having updates this means that the database serves not enough updates for the update queue. max. value = 5000
64 tasks in buildQueue
and
1000 tasks in fulltextBuildQueue
The buildQueue and fulltextBuildQueue are containing tasks for the calculation logic. This logic splits assets such as text (fulltextBuildQueue) into data fragments for the cdb. max. value each = 1000
2594183 events in queue
This number describes the current version of assets to be updated. A high amount of events is normal for a rebuild.
As soon as the rebuild is finished (0 tasks in the queues)
AssetStoreUpdater: AssetStoreService: AssetStoreUpdater: Updater wrote 1 updates (1 high) in 0ms, 0 tasks in update queue, 0 tasks in buildQueue, 0 tasks in fulltextBuildQueue, 0 events in queue.
wait for the next checkpoint. A checkpoint is going to be written every 10 minutes.
AssetStoreUpdater: CDBEnvironment: AssetStoreUpdater: writing checkpoint AssetStoreUpdater: CDBEnvironment: AssetStoreUpdater: Evicted 0 bytes from data cache temp AssetStoreUpdater: CDBEnvironment: AssetStoreUpdater: Evicted 0 bytes from node cache temp AssetStoreUpdater: CDBEnvironment: AssetStoreUpdater: verified read node cache size OK: 16548411 AssetStoreUpdater: CDBEnvironment: AssetStoreUpdater: verified temp node cache size OK: 0 AssetStoreUpdater: CDBEnvironment: AssetStoreUpdater: verified total node cache size: 16548411, target max size 314572800 AssetStoreUpdater: CDBEnvironment: AssetStoreUpdater: verified read data cache size OK: 63141939 AssetStoreUpdater: CDBEnvironment: AssetStoreUpdater: verified temp data cache size OK: 0 AssetStoreUpdater: CDBEnvironment: AssetStoreUpdater: verified total data cache size: 63141939, target max size 838860800 AssetStoreUpdater: AssetStoreService: AssetStoreUpdater: 44 Transactions done, checkpoint finished in 82ms.
Rollback settings
As soon as the checkpoint is finished, stop the censhare Server, restore the changes made to the config files and start again.
Troubleshooting
// cdb update during runtime
With the server action within censhare Admin Client at Configuration|Modules|Administration|Embedded Database|Embedded database sync using CCNs it is possible to sync the oracle database table with the censhare embedded database during runtime.
The difference between the rebuild from scratch via moving the ~/work/cdb folder aside and using the server action is the server action does not delete the existing cdb files it only appends new ones. This means that the cdb folder size is bigger afterwards.
If you have changed assets via SQL within the oracle database you have to insert the changed asset ids into the asset_ccn_counter table. Afterwards you have to execute the above-mentioned server action:
SQL> INSERT INTO asset_ccn_counter (asset_id) VALUES ([asset-ids]); SQL> commit;
// After restart: AssetStore Service not available
The AssetStore Service must read all cdb files until the last checkpoint. If this was performed the AssetStore Service is available. This can take up to several minutes depending on the cdb folder size. Until this time warning/error messages are possible. The necessary time can be reduced if the filling level of the cdb files (cleaner-usage-threshold) will be increased because then less cdb files exist and must be read.
// After restart: Client fuzzy search not possible
Even if the AssetStore Service was started successfully the fuzzy search is not working. Important for a working fuzzy search is the KnownValues . This can take up to several minutes depending on the cdb folder size. Until this time warning/error messages are possible. The following line will be shown within the server log file if the KnownValues were successfully loaded:
LoadKnownValues: AssetStoreService: no-context: async: loadKnownValues done.
// Show and delete cdb statistics
With the server action within censhare Admin Client at Configuration|Modules|Administration|Embedded Database|Embedded database statistics it is possible to show and delete the current statistics within a window.
// Interpret cdb statistics
// Decrease disk usage by changing cleaner-usage-threshold parameter
The parameter is named cleaner-usage-threshold and can only be changed within the xml file at Configuration|Services|Embedded Database. This means: if only 30% of the files of a cdb file are used, the 30% will be moved into the last created cdb file and the unnecessary file will be deleted which will free up disk space.
Formula: totalFileUsage% : cleaner-usage-threshold = factor Current cdb folder size x factor = maximum cdb folder sizeExample: If totalFileUsage 60% = 30GB used disk space and cleaner-usage-threshold = 10: 60 : 10 = 6 30GB x 6 = 180GB