An introduction to backend modules.

  • Implement server actions
  • Implement automated processes
  • Implement import/export/synchronization interfaces

Note: Frequently XSLT based solutions are appropriate or even more appropriate than Java development (not covered here).

Development setup

  • Use release distribution or git (export) distribution
  • When using git distribution, perform bin/build.sh init jar to generate classes and jars
  • Open project in Eclipse or setup IntelliJ (or use vi...)
  • Possible to start server within Eclipse but not recommended
  • Start server locally (does run on macOS and Windows), needs Oracle
  • Use Remote Java Application Debug setup in Eclipse (or IntelliJ)

In launcher.xml enable remote debugging:

<jvmarg value="-Xrunjdwp:transport=dt_socket,server=y,address=8003,suspend=n" enabled="true"/>

Note

Performance/Memory profiler like YourKit highly recommended!

<jvmarg value="-agentpath:/Applications/YourKit_Java_Profiler_2015_build_15074.app/Contents/Resources/bin/mac/libyjpagent.jnilib=listen=all" enabled="true"/>

Modules

  • Are in the following directories:
    • censhare-product/censhare-Server/app/modules directory (default)
    • censhare-product/censhare-Custom/censhare-Server/app/modules (custom)
  • Consist of:
    • XML description file ("command-xml")
    • Java (source) code
    • Java properties files for localization
  • Hot deployable, Java source code is compiled and loaded:
  • Use UI admin client to "refresh" or
  • Use shell admin client bin/AdminClient.sh -b refresh

Persistence Architecture

  • Relational database has tables
  • In Java need „objects“ and/or XML
    • Like to have Plain Old Java Object (POJO)
    • Use as data transfer object (DTO)
    • Serializable
    • Should support changes / transactions
    • Like to have XML

Class stack:

  • AXml (abstract base class)
    • XML node (we are not using org.w3c.dom)
  • CXml / BXml implementation
    • CXml is object instance based
    • BXml is byte[] based (CDB, online channel)
  • DXml adds change tracking / transactions
  • DataObject adds (database) persistence
  • AssetBase (automatically generated from db-schema)
  • Asset (additional methods)

Memory demand for a 1,9 MB XML file

  • When setting the asset name using the setName method, the corresponding XML node attribute gets updated:
// Asset asset = ...
asset.setName("Asset name").setAnnotation("comment");

// In generated AssetBase class contains:
public static final String ATTR_NAME = "name";

public T setName(String value) {
    xml.setAttr(ATTR_NAME, value);
    return (T) this;
}
<asset name="Asset name" annotation="comment".... />
  • How to know what to tell the database?
  • Database wants insert / update / delete
  • In transactional „mode“ DXml intercepts and notes the changes.
  • corpus:dto_flags
    • new - must insert
    • delete - should delete
    • modified - must update
    • transactional - is in transactional mode
    • persistent - is in database already
asset.setName("Asset name");
<asset name="Old name" new-val:name="Asset name" .... corpus:dto_flags="tm"/>
  • How to set the transactional mode?
  • You can do this manually (see below)
  • But typically the persistence manager does that for you
Asset asset = new Asset();
asset.setName("old name");
asset.setTransactionalRecursive(true);
asset.setName("new name");
System.out.println(asset.xml());
<asset corpus:dto_flags="tm" name="old name" new-val:name="new name"/>

Typical pattern:

DBTransactionManager tm;
PersistenceManager pm;
AssetManagementService am;

tm = command.getTransactionManager();
pm = tm.getDataObjectTransaction().getPersistenceManager();
am = ServiceLocator.getStaticService(AssetManagementService.class);

tm.begin();

Asset asset = Asset.newInstance();
asset.setName("new name");
asset.setType(AssetType.PICTURE.getDBValue());
asset.setApplication("default");

pm.makePersistent(asset);
am.checkInNew(tm, asset);

tm.end();
tm = command.getTransactionManager();
pm = tm.getDataObjectTransaction().getPersistenceManager();
am = ServiceLocator.getStaticService(AssetManagementService.class);

Note: tm and pm have a one-to-one relationship („separation of concerns“...)

tm.begin();

Asset asset = Asset.newInstance();
asset.setName("new name");
asset.setType(AssetType.PICTURE.getDBValue());
asset.setApplication("default");

pm.makePersistent(asset);
am.checkInNew(tm, asset);

tm.end();

tm.begin

  • Starts a new transaction
  • Marks objects in pm as transactional
tm.begin();

Asset asset = Asset.newInstance();
asset.setName("new name");
asset.setType(AssetType.PICTURE.getDBValue());
asset.setApplication("default");

pm.makePersistent(asset);
am.checkInNew(tm, asset);

tm.end();

pm.makePersistent

  • Creates primary keys
  • Attaches object to pm (put's it into pm)
tm.begin();

Asset asset = Asset.newInstance();
asset.setName("new name");
asset.setType(AssetType.PICTURE.getDBValue());
asset.setApplication("default");

pm.makePersistent(asset);
am.checkInNew(tm, asset);

tm.end();

tm.end

  • Performs SQL statements but no commit
  • Marks objects in pm as non-transactional
  • tm.commit additionally does SQL commit and issues events
  • tm.commit is done automatically on command completion
  • Objects stay in pm!

Common mistakes

  • pm is like a bag that keeps all objects
  • Some APIs attach (insert) assets to the pm, others don‘t. (E.g.: AssetCacheService cacheGetAsset vs. cacheGetAssetNoAttach)
  • For bulk changes you need to manage that manually to avoid an OutOfMemoryException (E.g. call tm.end and pm.reset)
  • New assets don‘t have new-val

Missing or too frequent am.update.

tm.begin();

asset.setName(newName);
if (asset.isNewDeletedOrModifiedRecursive())
    am.update(tm, asset);

tm.end();

  • The persistence layer does write the change in the database. But:

    • doesn't know how to handle files added to an asset
    • doesn't create asset events that notify the clients
    • cannot perform checkout / checkin
    • doesn't update versioned asset_relation / asset_element_relation table

  • Cannot perform checkout / checkin in one step, must use tm.begin / tm.end

  • Events are posted with tm.commit(), don't commit every single change!
  • To avoid deadlocks if working with multiple assets in one transaction the assets must be processed in ascending asset id order !!! (E.g. use TreeSet / TreeMap to collect assets first)
tm.begin();
Asset asset = am.checkOut(tm, Asset.newUn(id, Asset.VERSION_CURRENT));
tm.end();

tm.begin();
asset.setName(newName);
am.checkIn(tm, asset);
tm.end();
  • Use bulk operations for performance:
    • to get 10 assets using cacheGetAsset(DBTransactionManager tm, AssetCacheKey key): costs 10 x about 12 sql statements - if not using CDB
    • to get 10 assets using cacheGetAsset(DBTransactionManager tm, AssetCacheKey[] keys): costs about 12 sql statements - if not using CDB
  • If transaction is "dirty" – first SQL insert/update statement executed – CDB no longer used to allow for read after write

Notes on CDB

  • Stores current asset versions
  • Almost all queries for current versions are handled through it (if transaction not dirty)
  • All returned assets are checked to be „up to date“ using the ccn with a fallback to Oracle (use queryAssetsLazy if not required)
  • asset.ccn is a change control number
    • tcn incremented with each update
    • to allow for change detection without locking assets and updating their tcn (e.g. new asset relation)
asset.ccn = tcn + (select count(*) from asset_ccn_counter where id = asset_ccn_counter.asset_id

Asset API

  • Hidden in asset.struct() to avoid messy code completion suggestions
StorageItem si = asset.struct().getStorageItem(StorageItemKey.MASTER.getDBValue());

for (ParentAssetRel rel : asset.struct().getParentAssetRelIter(AssetRelType.VARIANT)) {
    long parentId = rel.getParentAsset();
    ...
}

Query API

  • Common API for embedded DB and Oracle
  • Compatibility for legacy implementations
  • XPath queries

Query Builder

  • Fluent interface for creating queries - with types (refactoring!)
  • Works together with schema created classes
  • Works stand alone!
QBuilder qb = new QBuilder(QSchemaDef.SCHEMA_DEF);
QAsset a1 = qb.table(QAsset.class);
QChildAssetRel r1 = qb.table(QChildAssetRel.class);
QSelect stm = qb.select()
    .table(a1)
    .join(r1, r1.join(qb, a1))
    .columns(a1.ID, a1.VERSION, a1.CURRVERSION, r1.HAS_UPDATE_CHILD_GEOMETRY)
    .where(
        qb.expr(a1.NAME, "Hello")
    );
System.out.println(SQLQueryBuilder.toString(stm));

QBuilder qb = new QBuilder();
QTable t1 = qb.tableAdHoc("something");
QSelect stm = qb.select().table(t1).columns(t1.defString("name"));
System.out.println(SQLQueryBuilder.toString(stm));

Asset Query Service

StringBuilder sb = new StringBuilder();
AssetQueryService qservice = ServiceLocator.getStaticService(AssetQueryService.class);
AssetQBuilder qb = qservice.prepareQBuilder();
QAsset tAsset = (QAsset) qb.getMainTable();
// qb.setSqlFallback(true); // force oracle

QSelect select = qb.select(tAsset).
        where(qb.and(
                qb.expr(tAsset.CURRVERSION, 0),
                qb.expr(tAsset.ID, ">=", 1000),
                qb.expr(a.TYPE, "picture"),
                qservice.preparePermissions(qb, tm.getTransactionContext().getUserId())
                )).orderBy(tAsset.NAME);

int count = 0;
for (Asset asset: qservice.queryAssets(select, tm)) {
    count++;
    if (command.isCanceled())
        return Command.CMD_CANCELED; // or throw Exception
    sb.append(asset.getId()); sb.append("\t");
    sb.append(asset.getName()); sb.append("\n");
    if (count % 1000 == 0)
        logger.info("at: " + count + ", " + asset);
}
logger.info("done at: " + count);
cmdXml.put("report", sb.toString());

AssetAutomation

  • AssetAutomation allows combined code for server actions and automated processes
  • See modules.lib.asset_automation.AssetAutomationSample
  • All "AAxxx" modules are implemented that way

Last Updated: 5 June 2019