An introduction to backend modules.
- Implement server actions
- Implement automated processes
- Implement import/export/synchronization interfaces
Note: Frequently XSLT based solutions are appropriate or even more appropriate than Java development (not covered here).
Development setup
- Use release distribution or git (export) distribution
- When using git distribution, perform
bin/build.sh init jar
to generate classes and jars - Open project in Eclipse or setup IntelliJ (or use vi...)
- Possible to start server within Eclipse but not recommended
- Start server locally (does run on macOS and Windows), needs Oracle
- Use Remote Java Application Debug setup in Eclipse (or IntelliJ)
In launcher.xml
enable remote debugging:
<jvmarg value="-Xrunjdwp:transport=dt_socket,server=y,address=8003,suspend=n" enabled="true"/>
Note
Performance/Memory profiler like YourKit highly recommended!
<jvmarg value="-agentpath:/Applications/YourKit_Java_Profiler_2015_build_15074.app/Contents/Resources/bin/mac/libyjpagent.jnilib=listen=all" enabled="true"/>
Modules
- Are in the following directories:
- censhare-product/censhare-Server/app/modules directory (default)
- censhare-product/censhare-Custom/censhare-Server/app/modules (custom)
- Consist of:
- XML description file ("command-xml")
- Java (source) code
- Java properties files for localization
- Hot deployable, Java source code is compiled and loaded:
- Use UI admin client to "refresh" or
- Use shell admin client
bin/AdminClient.sh -b refresh
Persistence Architecture
- Relational database has tables
- In Java need „objects“ and/or XML
- Like to have Plain Old Java Object (POJO)
- Use as data transfer object (DTO)
- Serializable
- Should support changes / transactions
- Like to have XML
Class stack:
- AXml (abstract base class)
- XML node (we are not using org.w3c.dom)
- CXml / BXml implementation
- CXml is object instance based
- BXml is byte[] based (CDB, online channel)
- DXml adds change tracking / transactions
- DataObject adds (database) persistence
- AssetBase (automatically generated from db-schema)
- Asset (additional methods)
Memory demand for a 1,9 MB XML file
- When setting the asset name using the setName method, the corresponding XML node attribute gets updated:
// Asset asset = ... asset.setName("Asset name").setAnnotation("comment"); // In generated AssetBase class contains: public static final String ATTR_NAME = "name"; public T setName(String value) { xml.setAttr(ATTR_NAME, value); return (T) this; }
<asset name="Asset name" annotation="comment".... />
- How to know what to tell the database?
- Database wants insert / update / delete
- In transactional „mode“ DXml intercepts and notes the changes.
- corpus:dto_flags
new
- must insertdelete
- should deletemodified
- must updatetransactional
- is in transactional modepersistent
- is in database already
asset.setName("Asset name"); <asset name="Old name" new-val:name="Asset name" .... corpus:dto_flags="tm"/>
- How to set the transactional mode?
- You can do this manually (see below)
- But typically the persistence manager does that for you
Asset asset = new Asset(); asset.setName("old name"); asset.setTransactionalRecursive(true); asset.setName("new name"); System.out.println(asset.xml());
<asset corpus:dto_flags="tm" name="old name" new-val:name="new name"/>
Typical pattern:
DBTransactionManager tm; PersistenceManager pm; AssetManagementService am; tm = command.getTransactionManager(); pm = tm.getDataObjectTransaction().getPersistenceManager(); am = ServiceLocator.getStaticService(AssetManagementService.class); tm.begin(); Asset asset = Asset.newInstance(); asset.setName("new name"); asset.setType(AssetType.PICTURE.getDBValue()); asset.setApplication("default"); pm.makePersistent(asset); am.checkInNew(tm, asset); tm.end();
tm = command.getTransactionManager(); pm = tm.getDataObjectTransaction().getPersistenceManager(); am = ServiceLocator.getStaticService(AssetManagementService.class);
Note: tm and pm have a one-to-one relationship („separation of concerns“...)
tm.begin(); Asset asset = Asset.newInstance(); asset.setName("new name"); asset.setType(AssetType.PICTURE.getDBValue()); asset.setApplication("default"); pm.makePersistent(asset); am.checkInNew(tm, asset); tm.end();
tm.begin
- Starts a new transaction
- Marks objects in pm as transactional
tm.begin(); Asset asset = Asset.newInstance(); asset.setName("new name"); asset.setType(AssetType.PICTURE.getDBValue()); asset.setApplication("default"); pm.makePersistent(asset); am.checkInNew(tm, asset); tm.end();
pm.makePersistent
- Creates primary keys
- Attaches object to pm (put's it into pm)
tm.begin(); Asset asset = Asset.newInstance(); asset.setName("new name"); asset.setType(AssetType.PICTURE.getDBValue()); asset.setApplication("default"); pm.makePersistent(asset); am.checkInNew(tm, asset); tm.end();
tm.end
- Performs SQL statements but no commit
- Marks objects in pm as non-transactional
tm.commit
additionally does SQL commit and issues eventstm.commit
is done automatically on command completion- Objects stay in pm!
Common mistakes
- pm is like a bag that keeps all objects
- Some APIs attach (insert) assets to the pm, others don‘t. (E.g.:
AssetCacheService cacheGetAsset
vs.cacheGetAssetNoAttach
) - For bulk changes you need to manage that manually to avoid an
OutOfMemoryException
(E.g. calltm.end
andpm.reset
) - New assets don‘t have
new-val
Missing or too frequent am.update.
tm.begin(); asset.setName(newName); if (asset.isNewDeletedOrModifiedRecursive()) am.update(tm, asset); tm.end();
The persistence layer does write the change in the database. But:
- doesn't know how to handle files added to an asset
- doesn't create asset events that notify the clients
- cannot perform checkout / checkin
- doesn't update versioned
asset_relation
/asset_element_relation
table
Cannot perform checkout / checkin in one step, must use
tm.begin
/tm.end
- Events are posted with
tm.commit()
, don't commit every single change! - To avoid deadlocks if working with multiple assets in one transaction the assets must be processed in ascending asset id order !!! (E.g. use TreeSet / TreeMap to collect assets first)
tm.begin(); Asset asset = am.checkOut(tm, Asset.newUn(id, Asset.VERSION_CURRENT)); tm.end(); tm.begin(); asset.setName(newName); am.checkIn(tm, asset); tm.end();
- Use bulk operations for performance:
- to get 10 assets using
cacheGetAsset(DBTransactionManager tm, AssetCacheKey key)
: costs 10 x about 12 sql statements - if not using CDB - to get 10 assets using
cacheGetAsset(DBTransactionManager tm, AssetCacheKey[] keys)
: costs about 12 sql statements - if not using CDB
- to get 10 assets using
- If transaction is "dirty" – first SQL insert/update statement executed – CDB no longer used to allow for read after write
Notes on CDB
- Stores current asset versions
- Almost all queries for current versions are handled through it (if transaction not dirty)
- All returned assets are checked to be „up to date“ using the ccn with a fallback to Oracle (use
queryAssetsLazy
if not required) asset.ccn
is a change control number- tcn incremented with each update
- to allow for change detection without locking assets and updating their tcn (e.g. new asset relation)
asset.ccn = tcn + (select count(*) from asset_ccn_counter where id = asset_ccn_counter.asset_id
Asset API
- Hidden in
asset.struct()
to avoid messy code completion suggestions
StorageItem si = asset.struct().getStorageItem(StorageItemKey.MASTER.getDBValue()); for (ParentAssetRel rel : asset.struct().getParentAssetRelIter(AssetRelType.VARIANT)) { long parentId = rel.getParentAsset(); ... }
Query API
- Common API for embedded DB and Oracle
- Compatibility for legacy implementations
- XPath queries
Query Builder
- Fluent interface for creating queries - with types (refactoring!)
- Works together with schema created classes
- Works stand alone!
QBuilder qb = new QBuilder(QSchemaDef.SCHEMA_DEF); QAsset a1 = qb.table(QAsset.class); QChildAssetRel r1 = qb.table(QChildAssetRel.class); QSelect stm = qb.select() .table(a1) .join(r1, r1.join(qb, a1)) .columns(a1.ID, a1.VERSION, a1.CURRVERSION, r1.HAS_UPDATE_CHILD_GEOMETRY) .where( qb.expr(a1.NAME, "Hello") ); System.out.println(SQLQueryBuilder.toString(stm)); QBuilder qb = new QBuilder(); QTable t1 = qb.tableAdHoc("something"); QSelect stm = qb.select().table(t1).columns(t1.defString("name")); System.out.println(SQLQueryBuilder.toString(stm));
Asset Query Service
StringBuilder sb = new StringBuilder(); AssetQueryService qservice = ServiceLocator.getStaticService(AssetQueryService.class); AssetQBuilder qb = qservice.prepareQBuilder(); QAsset tAsset = (QAsset) qb.getMainTable(); // qb.setSqlFallback(true); // force oracle QSelect select = qb.select(tAsset). where(qb.and( qb.expr(tAsset.CURRVERSION, 0), qb.expr(tAsset.ID, ">=", 1000), qb.expr(a.TYPE, "picture"), qservice.preparePermissions(qb, tm.getTransactionContext().getUserId()) )).orderBy(tAsset.NAME); int count = 0; for (Asset asset: qservice.queryAssets(select, tm)) { count++; if (command.isCanceled()) return Command.CMD_CANCELED; // or throw Exception sb.append(asset.getId()); sb.append("\t"); sb.append(asset.getName()); sb.append("\n"); if (count % 1000 == 0) logger.info("at: " + count + ", " + asset); } logger.info("done at: " + count); cmdXml.put("report", sb.toString());
AssetAutomation
- AssetAutomation allows combined code for server actions and automated processes
- See
modules.lib.asset_automation.AssetAutomationSample
- All "AAxxx" modules are implemented that way
Last Updated: 5 June 2019