OpsIQ

Runtime Issues ​

Problems you might run into while using OpsIQ, after it is installed and running. For problems during installation, see Installation & Setup Issues.

This page is a starting point

These are the runtime issues seen most often. It will grow as the pilot surfaces more. If you encounter something that is not here, contact your Fresche contact.

Checking the services' health ​

When something is wrong but it is not clear what, ask the services themselves. The tools server and the AI server each answer a health check at /health that runs their own checks and names the one that failed. In a PASE shell on the IBM i, after the PATH line from PASE shell setup, run curl -k https://127.0.0.1:8445/health for the tools server and curl -k https://127.0.0.1:3939/health for the AI server. 8445 and 3939 are the installer's default ports; substitute yours if they differ. On a split install, run the AI server's check on the app host, where the AI server runs; on a Linux or macOS app host any shell will do. If curl is not found on the IBM i, install it with yum install curl. -k is needed because both use the certificates generated at install, which curl does not trust; use -k only against 127.0.0.1 as shown. Each answers with a line of JSON: an overall status, the version, and a checks list whose entries each carry a name, a status of ok or error, and a detail.

ServiceCheckWhat it tells you
Tools serverlicenceWhether the license key is valid, and if not, why. The name is spelled licence in the output. This check answers even when the license has expired, although every other request is then refused.
Tools serverdb2Whether the tools server can connect to Db2 for i and run a trivial query.
Tools serverconfigHow many of its queries are enabled.
AI serverchat-databaseWhether the database holding chat history and sessions answers. If this fails, the AI server's overall status is error.
AI servertoolsWhether the AI server can reach the tools server and whether that reports itself healthy. When the two services' caller keys differ, it fails with HTTP 401 (caller key rejected), and the AI server's error log adds that TOOLS_API_KEY does not match tools-api.key. Absent when the AI server has no tools server configured.
AI serverai-credentialWhether Anthropic accepts the API key. rejected by the provider (HTTP 401) or (HTTP 403) means the key is wrong or revoked.

The AI server reports degraded rather than error when only the tools or ai-credential check fails, because it can still serve the web UI. If curl cannot connect at all, that service is not running: see A service restarts over and over and Managing the Services.

"API key is invalid" in chat or investigations ​

Symptom
Chatting or running an investigation fails with a message that the API key is invalid.
Cause
OpsIQ's Anthropic API key is missing, incorrect, or has been revoked. This is the key that powers the AI, and it is separate from your OpsIQ license.
Fix
Check the AI server's ai-credential health check (see Checking the services' health). rejected by the provider (HTTP 401) or (HTTP 403) confirms the key itself is refused, and ANTHROPIC_API_KEY is not set — chat and investigation are off means there is none. Confirm the key is still valid with whoever manages your Anthropic account. Once you have a working key, set it as ANTHROPIC_API_KEY in the AI server's .env.local (see Configuration Options) and restart the server (see Managing the Services). Your Fresche contact can help if needed.

License expired ​

Symptom
Nobody can sign in, and the sign-in page reports Authentication service temporarily unavailable. Users who were already signed in find that chat, investigations and the dashboard tiles fail, often with errors that refer to the tools server being unavailable.
Cause
The OpsIQ license key has expired or cannot be read. The tools server then refuses every request with HTTP 503 and the reason, except its health check. Signing in is one of those requests, because OpsIQ checks your IBM i password through the tools server, so sign-in fails along with everything that queries the system.
Fix
Check the tools server's licence health check (spelled that way in the output; see Checking the services' health), which names the reason. Then obtain a renewed key from your Fresche contact and apply it — see Renewing a license.

An investigation says the audit journal could not be read ​

Symptom
A security investigation reports that it could not read the audit journal, or that the profile was not authorized to read it. Other evidence in the same investigation is present.
Cause
OpsIQ runs each evidence query under the profile of the signed-in user, so it can never see more than that person can. Reading the system audit journal QSYS/QAUDJRN requires the *AUDIT special authority (*ALLOBJ also grants it). Ordinary object authority to the journal is not enough, and IBM i refuses the read with message CPF9802.
Fix
Grant *AUDIT to the profiles that run security investigations, for example CHGUSRPRF USRPRF(<user>) SPCAUT(<their existing list> *AUDIT). CHGUSRPRF replaces the list rather than adding to it, so include the authorities the profile already has. Until then, treat the authority-failure part of the investigation as unavailable rather than as an all-clear: OpsIQ read no journal entries, which is not the same as there being none.

The browser warns that the connection is not private ​

Symptom
Opening the web UI shows a certificate warning — "Your connection is not private", NET::ERR_CERT_AUTHORITY_INVALID, or a name mismatch — and you have to click through it to reach the login page.
Cause
OpsIQ installs with a self-signed certificate unless a CA-signed one was supplied, and no browser trusts a self-signed certificate. A name mismatch instead means a certificate is installed but does not cover the address being used. A warning that appears on a system that worked before usually means the certificate expired.
Fix
Install a certificate from your own certificate authority, and confirm it covers the address people type. The whole procedure, for a system that is already running, is in Replacing the TLS Certificate.

Cannot sign in or stay signed in ​

Symptom
You cannot sign in with a profile you know is valid, or you are signed out unexpectedly and asked to sign in again.
Cause
OpsIQ signs you in with your IBM i user profile, so most sign-in problems trace back to the profile itself (disabled, or an expired password), to the OpsIQ server not being reachable, or to the tools server refusing requests. Authentication service temporarily unavailable means OpsIQ could not check your password at all, most often because the tools server is down, its license has expired, or it is turning the AI server away because their caller keys do not match (see Sign-in fails with a 503 after install).
A session also ends on its own, within an hour of your last activity and sometimes after as little as 15 minutes. OpsIQ works on your behalf with an IBM i profile token that lasts one hour and can only be renewed while it is still valid, so this limit is fixed and no setting lengthens it. An open OpsIQ tab counts as activity, because it checks in with the server every ten minutes and again when you return to it, and that keeps the token renewed. So you are signed out once every OpsIQ tab has been closed, or your computer has slept, for longer than that. Signing out ends the session on the server straight away, so other open tabs of the same browser are signed out the next time they contact the server.
Fix
Confirm the profile works by signing on to the IBM i directly — a disabled profile or an expired password blocks OpsIQ sign-in as well. If sign-in hangs or errors rather than rejecting your password, check the services' health (see Checking the services' health) and restart what is down (see Managing the Services). If you were simply signed out after being away, sign back in — that is expected.

Chat messages fail with "Too many requests" ​

Symptom
After a burst of activity, the chat panel answers a message with an error bubble reading Too many requests and a retry button (the AI server returned HTTP 429). A minute later chat works again.
Cause
The AI server accepts at most 20 chat requests a minute for each sign-in. Each message counts, and so does the request for suggested follow-up questions that the chat panel makes after an answer, so sending many messages in quick succession can reach the limit.
Fix
Wait a minute and send the message again. The limit is fixed and is not a setting.

OpsIQ does not start after a system restart (IPL) ​

Symptom
After an IPL, the OpsIQ web UI will not load at all. (On a tools-only box the symptom differs — see the note in Managing the Services.)
Cause
OpsIQ's services do not yet restart automatically after a full system restart. This is a known limitation, and automatic startup is planned.
Fix
Start the services again using the procedure in After a system restart. Until automatic startup is available, plan to do this after every IPL.

The services did not start, and PM2 lists nothing ​

Symptom
You started OpsIQ, but the web UI does not load and pm2 list reports no services at all rather than showing them stopped or errored.
Cause
Most often the PM2_HOME setup line was not run, in which case pm2 list reports an empty list even while the services are running perfectly well. Failing that, the start did not get far enough to bring PM2 up at all: on a managed install it runs as a submitted job, and if that job could not run the start script there is no process list to report.
Fix
Run the two PASE shell setup lines and check pm2 list again. If it is still empty, read the start log and then the submitted job's printed output, as described in When a managed start fails. Between them they name the cause. If the command reports Permission denied rather than an empty list, see pm2 reports "Permission denied" on rpc.sock instead.

pm2 reports "Permission denied" on rpc.sock ​

Symptom
A pm2 command fails with [PM2][ERROR] Permission denied, check permissions on <install-root>/.pm2/rpc.sock.
Cause
On a managed install the services run under a dedicated profile (OPSIQ by default), and PM2 accepts commands only from the profile that owns them. A PM2 is running at the PM2_HOME named in the message, and your profile is not allowed to command it. Unlike an empty list, this does mean something is running there, though the path in the message is still worth checking in case it is not the instance you meant.
Fix
Whether PM2 accepts your commands depends on your authority, not on the install. A profile with *ALLOBJ can drive it directly; *JOBCTL on its own cannot. See Whether PM2 will answer you. Opening the PM2 log files directly is a plain IFS file read rather than a pm2 command, so it needs no authority and stays available to you (see Viewing the logs). The AI server's and web UI's own log files are the exception, since only the service profile or *ALLOBJ can read them (see their log files). Note that pm2 logs is not a way around this, since it reads through PM2 and is refused the same way. Diagnose from the files, then get someone with the authority to act. Do not loosen the permissions on the socket or the PM2 home to get around it.

A service restarts over and over ​

Symptom
A service is listed by pm2 list, but the ↺ (restart count) column keeps climbing, and OpsIQ is unreliable or drops out intermittently.
Cause
PM2 restarts a service automatically whenever it exits. A service that cannot stay up, usually because of a configuration problem, is therefore restarted repeatedly instead of simply staying down. After repeated quick crashes PM2 gives up and lists it as errored.
Fix
Read that service's error log to find why it exits, using pm2 logs <service-name> --err --nostream --lines 100 (see PASE shell setup for the setup lines). Without --nostream the command streams until interrupted, which never ends when it is run as a submitted job. Reading the error log file directly, as described in Viewing the logs, avoids that entirely. The log names the failure. If it points at a setting, correct it in the configuration and restart the service (see Configuration Options).

A log file has grown very large ​

Symptom
A file under <install-root>/.pm2/logs/ is far larger than the rest, or the file system it lives on is filling up.
Cause
PM2 rotates a log once it passes 10 MB, and again at midnight if it has grown at all, so a single file should not run away. Seeing one far larger than that, or several rotated copies piling up, points at a service writing continuously — a crash-loop is the usual reason — or at log rotation being off: check pm2 list for opsiq-logrotate-<instance> (see What is installed where).
Fix
Find out why the file is growing before reclaiming the space, since a crash-loop is the usual reason (see A service restarts over and over). Then empty the file in place rather than deleting or renaming it, because the service holds it open: : > <path>. See Viewing the logs for the paths and for pm2 flush.

An investigation returns little or no evidence ​

Symptom

Investigations complete but come back thin, or a particular kind of question never produces evidence, while OpsIQ is otherwise healthy and other investigations work.

Cause

OpsIQ gathers evidence by running read-only queries against IBM i system services. Which of those services exist, and which your profile is allowed to read, varies from system to system: some are supplied by optional licensed programs, some were added in later IBM i releases, and some are restricted by object authority. We avoid depending on optional services where we can, and test against the IBM i releases OpsIQ supports, but no test covers every combination of release, installed options and authority, so some configurations do slip past us. A query that cannot run on your system fails, and the AI receives the failure instead of evidence. When IBM i refuses the query for lack of authority, the failure reads Not authorized: the signed-in profile lacks an authority this tool's query requires.; any other query failure is reported as an internal server error. Either way the investigation has to carry on without that evidence.

Fix

Ask OpsIQ which of its queries work on this system. From a PASE shell (see Getting a PASE shell), in the tools server's directory under your install root:

sh
cd /QOpenSys/opsiq/opsiq/apps/ibm-opsiq-tools
./ibm-opsiq-tools --check-sql

It prints a line for each value a query accepts, then a summary. Queries that take no values print no line, and a few queries run a second preparatory statement that is listed separately, so the line count will not match the query count; that is expected. The lines that matter are the ones marked WARN: each names a query that cannot run here, and why.

WARN  get_command_help    could not prepare: COMMAND_INFO in QSYS2 type *FILE not found.

69 statement(s) checked, 175 marker(s) described, 0 disabled tool(s) skipped.
2 of those could not be prepared (see WARN above); their markers were not checked.

What to do next depends on the reason each WARN gives:

  • It mentions authority. The profile is not allowed to read that service. Whoever administers security on this system can grant it read access; re-run the check afterwards and the query should come back clean.
  • It says an object or column is "not found". The service does not exist at this IBM i release, or its optional licensed program is not installed. Send the output to your Fresche contact. There is nothing to change on your side, and the output tells us exactly which queries are affected on your system, which is the part we would otherwise have to guess at.

If the check reports no WARN lines at all, the thin investigation has some other cause. Send the output anyway and say what you were investigating.

Either way, only the investigations relying on those queries are affected; the rest of OpsIQ keeps working.

Run this as the profile that signs in to OpsIQ

OpsIQ runs each query under the profile of the person signed in, not under the service profile. Run the check as the profile that is actually seeing the problem. Running it as a profile with *ALLOBJ hides the very authority problems you are looking for, and two people with different authority can legitimately get different results.

The output is safe to send as-is

The check prepares each query without ever running it, so none of your data is read and nothing on your system is changed. The output is limited to the names of OpsIQ's own queries and their parameters, the SQL types they resolved to, and, on a WARN line, the message IBM i gave for why one of those queries could not run (which names the IBM i service involved, as in the example above). Nothing needs redacting before you send it.

The check needs no downtime and interrupts nobody: it opens its own short-lived database connection and leaves the running services alone.

Unfamiliar jobs running under the OpsIQ profile ​

Symptom
Active jobs on the system include jobs named QP0ZSPWP running under a profile such as OPSIQ, with nothing to indicate what they are or whether they can safely be ended.
Cause
That is simply how OpsIQ's services appear on the IBM i. They are PASE programs, so each one runs as a QP0ZSPWP job under the service profile rather than under a job name of its own, and a small churn of short-lived jobs under the same profile is normal.
Fix
No action is needed, as OpsIQ is running normally. See How OpsIQ appears on the IBM i to work out which job belongs to which service. Do not end these jobs: OpsIQ is started and stopped through PM2, as described in Restarting the services.