Why is Printing Delayed Although Resources are Available?
Kühne + Nagel (AG & Co.) KG in Hamburg, Germany, ran into serious performance problems when print jobs started piling up on the job queues of their iSeries server. Consignment documents, invoices, etc., that normally were printed within seconds, started having delays of several hours, obviously causing severe problems for the business. The print jobs had been working fine for years - and no changes had been made.
The server was in no way overloaded - an average CPU usage of around 65% meant that 4 or 5 of the 16 CPUs were idling most of the time, and also the disk I/O rates were well within the recommended values. Many batch jobs were running in parallel to produce the print, but they showed close to no use of resources - in fact, they did not really seem to move for prolonged periods of time.
It can be tough to locate the cause of performance problems when jobs are eating up a lot of resources, but it is often more difficult to find the cause when close to no resources are used. A record or object lock was an obvious guess, but did not seem to be the case. Moreover, the application was designed to allow many parallel jobs (and not cause any locks), and with the number of transactions almost unchanged such locks would have happened long before.
Fortunately the company had an ace up its sleeves: The software package GiAPA from iPerformance. GiAPA (Global i Performance Analyzer) includes options for analyzing where jobs are delayed, when they do not move. GiAPA Trace Job Analysis showed that the delays, sometimes exceeding 30 seconds, always occurred when a data base file or member in QTEMP was created or deleted, i.e. within IBM supplied programs like QDBCRTME (Data Base Create Member).
With this documentation in hand the ball could be played to IBM who proved that seize waits (seize = operating system internal lock) during update of the list of owned objects for the user profile QDBSHR were causing the problem. (IBM subsequently decided to issue a PTF to remove the problem for the current version of the operating system, and will of course include this change in the new OS version.)
In the meantime the problem could be avoided by clearing and reusing instead of creating and deleting the files in QTEMP - - and the print started to appear within seconds again.