Why the Right Data Matters More Than More Expertise

When people talk about IBM i application performance problems, the conversation often turns quickly toward expertise. Who is experienced enough to find the cause? Who knows which tool to use? Who can look at the right job at the right moment? Experience certainly matters. But in many cases, the real obstacle is not lack of expertise. It is lack of the right data.

Why performance investigations so often take too long

One of the common frustrations in application performance work is that problems rarely arrive in a convenient form. A job may be delayed only occasionally. A transaction may suddenly run much longer than normal and then return to normal again. A batch job may complete successfully, but still waste many hours of runtime for reasons nobody can immediately see. In such situations, people often assume the difficult part is the analysis itself. In my experience, that is often not true.

The difficult part is that by the time someone starts investigating, the necessary data is either incomplete or missing.

Without the right documentation of what the job was actually doing, even a very capable specialist may be forced to guess. This is one reason performance analysis has historically been seen as a specialist discipline. You do not just need data. You need the right data. And the type of data required depends on the nature of the problem. If the issue is CPU-related, one set of information may be enough. If the issue involves file access, locking, SQL, memory pressure, or interactions with other jobs, you may need something else. That creates a circular problem: you need data to understand the problem, but you often need some understanding of the problem before you know which data to collect.

A different starting point

A much better starting point is to ensure that the most useful performance data is already available when the problem occurs. That changes everything.

Instead of beginning with “what should we collect?”, you can begin with “what actually happened?” This sounds simple, but in practice it changes application performance analysis from a reactive exercise into a much more practical one.

What data solves most problems?

Based on many years of performance work, a useful rule of thumb is that more than 80% of common application performance issues can be understood if you have resource usage data for all jobs at regular intervals, call stack data for jobs using significant CPU, and open file data for jobs using significant CPU. That combination is important because it gives both breadth and detail. Breadth means you can see what is happening across the whole workload. Detail means you can see what the jobs that matter were actually doing. Without breadth, you risk looking only where you already suspect the problem may be. Without detail, you may see that a job used resources, but still not understand why.

Why this matters in real cases

Consider a transaction that normally completes in about one second, but occasionally takes 30 seconds. If the right data is not already there, the investigation may begin too late. The problem has come and gone.

Or consider a batch job that runs for more than 83 hours. If you only know that it performs many reads, you may conclude that the system is handling the workload quite well. But if you also have detailed open file data, it may become visible that a small QTEMP file is being read repeatedly by key inside a main processing loop. The difference is not intelligence. The difference is visibility.

Expertise becomes more useful when the data already exists

This does not mean expertise becomes irrelevant. Quite the opposite. The right expertise becomes far more useful when it is applied to good data instead of bad assumptions. A skilled analyst is always more effective when the evidence is already present. And programmers, operations staff, and managers can also make better decisions when the explanation is based on actual job behavior rather than guesswork.

Why continuous collection matters

Of course, this only works if the data can be collected continuously without creating a new performance problem in the process. That is the practical challenge. If the collection mechanism itself is too heavy, nobody will run it all the time. But if the collection can run continuously with very low overhead, then the data is there when it is needed. And when the data is already there, performance analysis stops being a search in the dark. It becomes a much more direct process of explanation, prioritization, and improvement.

Better data leads to better priorities

There is another important consequence. When the right data is available, you do not just identify what went wrong. You also get a much better basis for deciding which issue matters most, what can realistically be improved, how much time or CPU may be saved, and where effort should be spent first. This is one of the main reasons the right data matters more than simply having more tools or more expert hours available. Better evidence leads to better priorities.

Final thought

IBM i application performance has often been treated as something slow, difficult, and dependent on rare expertise. But much of that reputation comes from a very practical problem: the right data is usually missing when the explanation is needed. When that problem is solved, the rest often becomes much easier. Not because expertise no longer matters, but because good data allows expertise to be used where it creates the most value.

This article is part of GiAPA’s guide: Why IBM i Application Performance Analysis Feels Hard.

If you want the broader context — why IBM i performance problems often seem difficult, what data makes them easier to explain, and how real cases point to practical improvements — the full guide brings the pieces together.