The life of a system administrator can be full of surprises. Things you have taken for granted, turn out to be completely different.

Let me get directly to the point. All jobs being started on IBM i are logged in the history log, right?
If you want to know the answer, you need to read on.

When using Performance Data Investigator (PDI) after we noticed some spikes in the CPU usage, we did notice that one particular job was using a lot of CPU:

It was obvious which task did take the most CPU, for readability it has it’s name first. So what I did was run the command listed below, to check if the task was still running:

wrkjob job(721674/QSECOFR/QP0ZSPWP)

The result was the message: Job 721674/QSECOFR/QP0ZSPWP not found.

So the job was no longer active, so I did run the command shown below: 

DSPLOG JOB(721674/QSECOFR/QP0ZSPWP)

Again no result:

After diving into the IBM i documentation I discovered the reason why in some cases the starting and stopping is not available in the history log. This had to do with the “Spawn Process”, this link will deals with that subject: spawn()–Spawn Process.

he following steps:

1. Start a 5250 emulation session
2. Enter the command: QSH
3. In this session, run the CL command by using the syntax: “system dspjob”
4. Scroll up, to find the name of the job
5. Your jobname will be “QP0ZSPWP”
6. Go to a second 5250 emulation and run the command: DSPJOB QP0ZSPWP
7. You will notice that your job, with the jobuser and jobnumber can be found
8. Now run the command DSPLOG with the information you found before

So know we know that the history log is missing information, so what to do collect all the information of all jobs running on IBM i? The answer to this question, is to implement Auditing.
When you include the job tasks (*JOBDTA) in the audit actions you are good to do.

To get started with Auditing this link will get help you: Using the security audit journal.

With the growing use of open source on IBM i, it is my guess that the History Log will be missing more and more starting and stopping of tasks. The good thing is that with Auditing you miss nothing of the action. The History Log might not catch anything, the Audit journal will not miss any start or stop of a task on IBM i.

In order to determine if you have jobs being started or stopped without the History Log missing them, you need to setup Auditing first. After you have done that you can run the SQL statement show below to check if you are missing information. It is also here where the DB2 for i Services does play a big role to get this information from the Audit journal:

As you can see, in this example the task QUMEPRVAGT is not in the History Log. This task is part of the CIMON TCP/IP server, which comes with the operating system.

In order to get you started, below the SQL statement in text:

WITH jaudhrn_js AS (
SELECT ENTRY_TIMESTAMP, JOURNAL_ENTRY_TYPE, JOURNAL_CODE,
TRIM(JOB_NUMBER) || ‘/’ || TRIM(JOB_USER) || ‘/’ || TRIM(JOB_NAME) AS FullJob,
CHAR(SUBSTR(ENTRY_DATA, 1, 1), 1) AS Detailed_Entry,
CASE
WHEN CHAR(SUBSTR(ENTRY_DATA, 1, 1), 1) = ‘S’ THEN ‘CPF1124’
WHEN CHAR(SUBSTR(ENTRY_DATA, 1, 1), 1) = ‘E’ THEN ‘CPF1164’
ELSE ‘OEPS’
END AS Search_For_MSG
FROM TABLE (
QSYS2.DISPLAY_JOURNAL(
‘QSYS’, ‘QAUDJRN’, STARTING_TIMESTAMP => CURRENT TIMESTAMP – 2 HOURS,
JOURNAL_ENTRY_TYPES => ‘JS’)
)
WHERE (CHAR(SUBSTR(ENTRY_DATA, 1, 1), 1) IN (‘S’, ‘E’))
)
SELECT A.ENTRY_TIMESTAMP, a.FullJob AS Job_In_AuditJRN, a.Detailed_Entry,
b.from_job AS Job_In_QHSTLOG, a.Search_For_MSG, b.MESSAGE_TIMESTAMP, b.MESSAGE_ID,
b.MESSAGE_TEXT
FROM jaudhrn_js A
LEFT JOIN TABLE (
qsys2.history_log_info(
START_TIME => CURRENT TIMESTAMP – 125 MINUTES,
END_TIME => A.ENTRY_TIMESTAMP + 1 MINUTE)
) b
ON a.FullJob = b.from_job AND a.Search_For_MSG = b.MESSAGE_ID
ORDER BY A.ENTRY_TIMESTAMP DESC;

If you have Auditing already implemented, you can directly start with this SQL statement. Please be aware the DB2 for i Service we are using “QSYS2.DISPLAY_JOURNAL” has been enhanced in the latest Technology Refresh of September 2021 for both 7.3 and 7.4. If we are using any of that enhanced functionality in the SQL statement we are using, is something I do not know for sure. My motto is as always, stay current with IBM i.