Job seems to be stuck at “Build_Master” mjoiner. This job run’s every 15mins. But seems like job is not moving forward. Any idea how do we resolve it?
This dataflow has transient target tables present in them which are being used as SOURCES in the dataflows. Any reason the transient table counts are too high when compared to data being extracted from source for ‘s_usgci_mst’ job.
As the Transient SOURCES having too much data present in them. The MJOINER is stuck for longer time.
I suspect the CREATE TARGET and was missed as part of the 4.x upgrade due to which huge data was loaded into transient tables and joiner was taking a very long time almost 42 hours to complete. After enabling ‘create target‘ and ‘recreate target’ options in target transient data object properties for the ‘s_usgci_mstr’ job. Job is running within the expected SLA. Need to monitor the Job.
The functionality of transient tables to automatically drop after each run. For this to happen we need to enable the CREATE TARGET and RECREATE TARGET option in the Transient Targets.
Once you enable the above properties and then try to run it. The dataflow should run within specified timelines. As the transient tables are not getting dropped, JOINER is trying to perform the inner join or left join on all the ingested data which is causing the performance issue. These transient tables should be dropped and re-created periodically in the dataflow.