Open topic with navigation
DMExpress job log files contain information about a given job run, including job start/end time, source/target size and record count, and execution status. Collating these data points is useful for detailed job status reporting, auditing, and troubleshooting.
DMExpress itself can be used to process these job logs, consolidating information about the job execution into a single record for easy review and storage.
The attached DMExpress log processing job takes one or multiple DMExpress text log files as input, and outputs a one-line status report for every individual process.
The log processing job extracts the highlighted data points in the following snippet of a DMExpress text log file:
and outputs them in the following pipe-delimited report format:
The Files field is a grouping of multiple elements – Source or Target file, filename, record count and file size – and this set repeats for each file to be reported. Any intermediate files generated by multi-task processes are not included in the report.
The log processing job, J_JobReport.dxj, consists of two subjobs:
This job parses the log files to extract the information to be reported, and tags the source and target files to be reported by performing a lookup on the list of files created by J_GetFileNames. Any task level information, like source and target record count and file sizes, is aggregated first at the task level and then at the job level. Other information, like job start time, end time, etc., is produced at the job level. The two sets are then joined to produce a summarized snapshot at the job level, which is written to the JobReport output file.
The log processing job requires the following environment variables to be defined in the DMExpress Server dialog if running from the GUI, or in the shell environment if running from the command line:
|LOG_DIR||the full path to the folder where the log files to be processed are stored. When running from the GUI, select the option in the Run Job dialog to copy the job log to a specified location, and then specify that same location in this variable.|
|LOG_SRC||the single or wildcarded (‘*’) filename of the file(s) to be processed in the LOG_DIR folder|
|LOG_REPORT_DIR||the full path to the folder in which to store the log processing report, named “JobReport”|
Following is an example invocation from the command line:
dmxjob /run J_JobReport.dxj /export LOG_DIR=/DMXJobs/Logs LOG_SRC=MyJobLog.txt LOG_REPORT_DIR=/DMXJobs/Reports
49_JobLogProcessing.zip, compatible with DMExpress version 8.0.5 or higher
The log processing job does not process MapReduce logs or database sources/targets, and it only processes UNIX (LF terminated) text logs.
DMExpress has an option to output log information in either XML or text format, defaulting to XML in the Run Job dialog, and to text when running via the command line (dmxjob or dmexpress).
See Running the job from a command prompt, Run Job dialog, and DMExpress Server dialog in the DMExpress Help for details on setting environment variables and running jobs.
Copyright © 2016 Syncsort All rights reserved.