Managing the Services

OpsIQ's long-running programs are supervised by PM2, a widely used process manager for Node.js applications. Instead of each program being started by hand and left to fend for itself, they are handed to PM2 once. PM2 keeps a list of the processes it owns, starts them, restarts any that exit unexpectedly, and collects their output into log files. Nearly everything on this page is therefore a pm2 command; the ones OpsIQ needs are collected in the pm2 command reference, and the PM2 documentation covers the rest.

Two details of how PM2 works shape everything below:

• There is one PM2 per install. Its process list lives in a directory called the PM2 home, one per OpsIQ instance, and the PM2_HOME variable is what tells pm2 which instance you mean. Point it elsewhere and PM2 will answer accurately about something other than the instance you had in mind.
• PM2 belongs to the profile that started it. On a managed IBM i install that is a dedicated service profile rather than you, so whether PM2 accepts your commands comes down to your authority. See Whether PM2 will answer you.

This page covers checking status, restarting and stopping the services, reading their logs, bringing them back after a system restart, what to do when a start fails, and what they look like natively on the IBM i.

What is installed where

OpsIQ is installed in one of three shapes, and which one you have decides both what you should expect to see and which parts of this page apply. Service names end in the instance name, opsiq by default.

What was installed	Services you should see	Where they run
Everything on the IBM i (--role all, the common case)	opsiq-tools-opsiq, opsiq-server-opsiq, opsiq-web-ui-opsiq, opsiq-logrotate-opsiq	IBM i
The tools server only (--role tools)	opsiq-tools-opsiq, opsiq-logrotate-opsiq	IBM i
A split install (--role tools here, --role app elsewhere)	opsiq-tools-opsiq and opsiq-logrotate-opsiq on the IBM i, opsiq-server-opsiq, opsiq-web-ui-opsiq and opsiq-logrotate-opsiq on the other host	both

opsiq-logrotate-opsiq rotates this instance's PM2 logs; it has no port and no health check. If it is missing, the other services start without it and the start prints [opsiq] log rotation is off for this start (on a managed install, that line is in the start log). The PM2 logs then grow without limit, so re-run the installer for the instance to restore log rotation, and contact your Fresche contact if the message persists.

If only the tools server was installed, that single service is the whole picture. The AI server and the web UI are not missing or failed; they were never installed on this machine. See What to install (roles) for the choice that was made at install time.

If part of OpsIQ runs on a Linux host

The AI server and the web UI can be installed on a separate Linux machine, usually a VM, that reaches the IBM i tools server over the network. The tools server itself always runs on the IBM i, because reading the IBM i is its whole purpose.

A split install therefore means administering two hosts, and this page applies to both, differently:

• The tools server, on the IBM i. Everything on this page applies as written.
• The AI server and web UI, on the Linux host. The pm2 commands are identical, but nothing IBM i specific applies: no SBMJOB, no submitted jobs, no spooled files, no WRKACTJOB.

	IBM i	Linux host
Setup lines	PATH and PM2_HOME	PM2_HOME only
Install root	/QOpenSys/opsiq/<instance> by default	the --install-root you chose; there is no default
Service profile	managed (OPSIQ) or direct	always the profile you connect as
Who PM2 takes commands from	a profile with *ALLOBJ	the owning user, root, or that user's group
Recovering after a restart	after an IPL	after a reboot

A Linux install is always a direct install, because the managed service profile is an IBM i mechanism. The services run as the user who installed them, so if that is you, sign in as that user, export PM2_HOME, and every command in the pm2 command reference works as written. No privilege escalation is involved, and the delegated helper is an IBM i thing you can ignore.

If the person troubleshooting is not the one who installed it, which is common enough, Whether PM2 will answer you has the rule for both platforms and shows how to act as the owning user.

opsiq-install status reports on one host at a time, so run it against each host in turn.

Installed everything on the IBM i?

Then the rest of this page is written for your setup as it stands. Continue with Checking service status.

Checking service status

From your workstation, in the folder you unzipped the installer into (see Get the installer):

macOS / Linux

./opsiq-install status --host your-ibm-i.example.com --ssh-user YOURPROFILE

Windows (PowerShell or Command Prompt)

.\opsiq-install.exe status --host your-ibm-i.example.com --ssh-user YOURPROFILE

Windows (Git Bash)

./opsiq-install.exe status --host your-ibm-i.example.com --ssh-user YOURPROFILE

Or, on the box itself in a PASE shell, with the two setup lines first, as described next:

PATH=/QOpenSys/pkgs/lib/nodejs22/bin:/QOpenSys/pkgs/bin:$PATH; export PATH
PM2_HOME=/QOpenSys/opsiq/opsiq/.pm2; export PM2_HOME
pm2 list

That works when you are signed in as the profile the services run under. On a managed install they run under a dedicated profile (OPSIQ by default), and PM2 refuses a pm2 command issued by anyone else:

[PM2][ERROR] Permission denied, check permissions on /QOpenSys/opsiq/opsiq/.pm2/rpc.sock

Read that message narrowly. It says that a PM2 is running at the PM2_HOME you pointed at and that your profile is not allowed to command it. It does not confirm that this is the instance you wanted, only that something is there and it is not yours to drive. Check the path in the message first, then see Whether PM2 will answer you, which sets out what decides this and what you can still do without the authority.

Two things to establish first

Which install do you have?

Every procedure below forks on whether the install is managed (the services run under a dedicated profile) or direct (they run under the profile that installed them). The install root's owner is the answer, because that is the profile the services run as:

ls -ldL /QOpenSys/opsiq/opsiq

The -L matters: if the install root is a symlink, -d on its own reports whoever made the link rather than the profile that owns the tree. The installer's own probe uses -L for the same reason.

If the owner shown is your own profile, the install is direct. If it is another profile, OPSIQ by default, the install is managed, and whether you can drive it depends on your authority rather than on the install (see Whether PM2 will answer you). The same directory holds .opsiq-install.json, which records the instance name, the role that was installed, and the ports:

cat /QOpenSys/opsiq/opsiq/.opsiq-install.json

Getting a PASE shell

The pm2 program is an open-source program, so it runs in PASE, the AIX-compatible environment on the IBM i, rather than from a 5250 command line. Two ways in:

• CALL QP2TERM from a 5250 command line, which opens a PASE session in your 5250 job.
• SSH from your workstation, which lands you in a PASE shell directly. This is the better option for reading logs, since you can scroll and copy.

Both give you the same shell. The SBMJOB commands on this page are the exception: those are CL, and they are run from a 5250 command line, not from PASE.

PASE shell setup

Every pm2 command on this page needs these two lines run first, once per PASE session, and they last only for that session (adjust the path if you installed a non-default instance):

IBM i (PASE)

PATH=/QOpenSys/pkgs/lib/nodejs22/bin:/QOpenSys/pkgs/bin:$PATH; export PATH
PM2_HOME=/QOpenSys/opsiq/opsiq/.pm2; export PM2_HOME

Linux host

PM2_HOME=/opt/opsiq/opsiq-opsiq/.pm2; export PM2_HOME

• PATH — the open-source programs pm2 and yum live under /QOpenSys/pkgs, which is not on the default PASE PATH. Without this line pm2 is simply reported as not found, even though it is installed.
• PM2_HOME — PM2 keeps its process list per home directory, so without it pm2 list reports an empty list even while the services are running fine.
• Your authority matters too. On a managed install these two lines are not always enough on their own, because PM2 will only take commands from a profile allowed to reach it. See Whether PM2 will answer you.

On a Linux host only PM2_HOME is needed, because node and pm2 are already on the login PATH there. The path is the one you gave the installer as --install-root, since a Linux install has no default location. The example above follows the layout the installer suggests, /opt/opsiq/opsiq-<instance>, for an instance named opsiq.

Use the two-step VAR=…; export VAR form

The IBM i PASE default shell is bsh, which rejects the combined export VAR=value form — it fails with bsh: VAR=value: is not an identifier and then abandons the rest of the block, so nothing after it runs either. Assign first, then export, exactly as above. (In bash or ksh the combined form works, but the two-step form is safe in all of them.)

Whether PM2 will answer you

PM2 accepts commands only from a user allowed to reach its control socket. On a managed IBM i install the services run under a dedicated profile, so that comes down to your authority. On a Linux host they run as whoever installed them, so it comes down to which user you are.

Where	Who you are	Can run pm2 commands
IBM i	A profile with *ALLOBJ, which whoever installed OpsIQ has	yes
IBM i	*JOBCTL, a common operator authority	no
IBM i	In the same group as the service profile	no
IBM i	No special authority	no
Linux	The user that owns the PM2 home, or root	yes
Linux	A member of that user's group	yes
Linux	Any other user	no

The IBM i rows were measured on a managed install, one profile per authority level. *ALLOBJ is what makes the difference there. It is what lets a profile reach another profile's socket. *JOBCTL does not help, so a conventional operator profile is no better placed than an ordinary user. This covers every pm2 command, pm2 logs included, since that reads the log through PM2 rather than off disk.

The group row differs between the two platforms, which matters if you are deciding how to give a colleague access. On Linux, PM2 creates its socket group-writable and carrying the installing user's group, so adding someone to that group works. On the IBM i it does not, because the IFS directories are setgid and everything PM2 creates there lands in group 0 rather than in the service profile's group.

If PM2 will answer you, there is nothing more to it. Export the setup lines and use pm2 directly, on a managed install as readily as on a direct one. That covers every procedure on this page except starting an instance from cold, which cannot go through PM2 because PM2 is not running yet, and so still depends on how the instance was installed. PM2's own daemon does the work, and it runs as the owning user, so a service you restart keeps running as that user rather than as you.

On a Linux host, sudo is the practical way to act as the owning user. It clears the environment, so pass PM2_HOME through rather than exporting it beforehand.

sudo -u opsiq env PM2_HOME=/opt/opsiq/opsiq-opsiq/.pm2 pm2 list

pm2 list reports the caller, not the owner

After you restart a service, PM2's user column shows your profile rather than the one the service runs under. That column records whoever last issued the command. The process itself is unchanged, which you can confirm with the port query under How OpsIQ appears on the IBM i.

When PM2 refuses you

The message names its control socket:

[PM2][ERROR] Permission denied, check permissions on /QOpenSys/opsiq/opsiq/.pm2/rpc.sock

You cannot work around that by loosening permissions on the socket or the PM2 home, and on the IBM i you cannot set up the delegated route either, because that needs *ALLOBJ as well.

What you can still do is read the PM2 log files themselves. They are ordinary files, readable by any user on the box, so opening one has nothing to do with PM2 and needs no authority. That is usually enough to identify what is wrong before handing it to someone who can act on it. See Viewing the logs. The AI server's and web UI's own log files are the exception: they are private to the service profile (see The AI server's and web UI's own log files).

Delegating control without *ALLOBJ

Skip this unless you need to let an operator control the services without giving them *ALLOBJ. It is the one case the direct route above does not cover.

The route is a small helper script, prepared once by someone who does have *ALLOBJ, and afterwards submitted by the operator as a job that runs under the service profile. The operator needs *USE to the service profile. That is an explicit grant, since public authority to a user profile is *EXCLUDE by default.

1. Prepare the helper, as a profile with *ALLOBJ. Adjust the paths if you installed a non-default instance:

cat > /QOpenSys/opsiq/opsiq/opsiq-pm2.sh <<'EOF'
exec > /QOpenSys/opsiq/opsiq/opsiq-pm2.log 2>&1
umask 022
PATH=/QOpenSys/pkgs/lib/nodejs22/bin:/QOpenSys/pkgs/bin:/QOpenSys/usr/bin:/usr/bin:$PATH; export PATH
QIBM_MULTI_THREADED=Y; export QIBM_MULTI_THREADED
PM2_HOME=/QOpenSys/opsiq/opsiq/.pm2; export PM2_HOME
HOME=/QOpenSys/opsiq/opsiq; export HOME
eval "$@"
EOF
chmod 700 /QOpenSys/opsiq/opsiq/opsiq-pm2.sh
chown OPSIQ /QOpenSys/opsiq/opsiq/opsiq-pm2.sh

exec redirects everything the command prints into a log file, because a submitted job has no screen to write to. umask 022 matters whenever the command starts PM2 itself, such as pm2 resurrect after an IPL: a submitted job otherwise creates files writable by everyone, and PM2 would then create its logs that way. The four environment lines are what the installer sets up for its own submitted jobs, and a submitted job starts with a fresh environment, so all four are needed: PATH and PM2_HOME are the setup lines, QIBM_MULTI_THREADED=Y is the thread-capable flag PASE Node needs, and HOME keeps the job out of the service profile's own home. Then eval "$@" runs whatever command is passed in at step 2.

The chmod and the chown both matter. The helper runs whatever it is handed, under the service profile, so anyone who can write to the file can run a command as that profile. Set the mode before changing the owner, and keep the helper in the install root: /tmp is world-writable, and it is cleared at IPL.

2. Run a command with it, as the operator. This is a CL command, so it goes on an IBM i command line (a 5250 command entry) rather than in a PASE shell. Clear the previous log first, so what you read afterwards can only be from this run:

SBMJOB CMD(STRQSH CMD('/QOpenSys/usr/bin/sh /QOpenSys/opsiq/opsiq/opsiq-pm2.sh "pm2 list"'))
       JOB(OPSIQPM2) JOBD(QGPL/OPSIQJOBD) USER(OPSIQ) INLLIBL(*JOBD)

Any command from the pm2 command reference goes in the double quotes: "pm2 stop all", "pm2 restart opsiq-server-opsiq", "pm2 logs opsiq-server-opsiq --err --nostream --lines 100". Adjust JOBD and USER if you installed with a non-default service identity.

Spell out /QOpenSys/usr/bin/sh

Under STRQSH the bare name sh is QShell, which writes redirected output in the job's EBCDIC character set, and the log then reads back as unusable characters over SSH. The PASE shell at /QOpenSys/usr/bin/sh writes UTF-8, which is why the full path is given.

3. Read the output, once the job has ended:

cat /QOpenSys/opsiq/opsiq/opsiq-pm2.log

The submitted job runs asynchronously, so the SBMJOB returns as soon as the job is accepted rather than when pm2 has finished. Give it a few seconds. The helper only overwrites its log once it starts, so a job that dies before reaching the helper leaves the previous run's output on disk, where it reads as a success for a command that never ran. Remove the log before submitting and that cannot mislead you. If the log is missing afterwards, the job could not run the helper at all, and When a managed start fails covers where that evidence goes instead.

Restarting the services

Run the PASE shell setup lines, then restart a single service, for example after a configuration change:

pm2 restart opsiq-server-opsiq

Or restart everything in the instance with pm2 restart all. The command is the same on a managed install, on a direct install and on a Linux host. What differs is only whether PM2 will take it from you, which comes down to your authority on the IBM i and to which user you are on Linux. Both are covered under Whether PM2 will answer you. Either way PM2's daemon respawns the service, so it keeps running as whoever owns the daemon rather than as you.

If you are the operator working through the delegated helper, send "pm2 restart opsiq-server-opsiq" as the quoted command instead.

Starting an instance from cold

pm2 restart needs PM2 to be running already. When it is not, after an IPL or after pm2 kill, use the start script the installer leaves in the install root:

IBM i, direct install

sh /QOpenSys/opsiq/opsiq/.opsiq-pm2-start.sh

IBM i, managed install, from a 5250 command line

SBMJOB CMD(STRQSH CMD('/QOpenSys/usr/bin/sh /QOpenSys/opsiq/opsiq/.opsiq-pm2-start.sh'))
       JOB(OPSIQSTRT) JOBD(QGPL/OPSIQJOBD) USER(OPSIQ) INLLIBL(*JOBD)

Linux host

sh /opt/opsiq/opsiq-opsiq/.opsiq-pm2-start.sh

Adjust the paths, JOBD, and USER if you installed with non-default values. If you are unsure, your Fresche contact can confirm the exact command for your setup.

The managed form submits a job to run as the service profile, so it needs *USE to that profile, the same explicit grant described under Delegating control. Without it the SBMJOB is rejected on the spot with CPD1616 Not authorized to user profile OPSIQ, and no job is created, so nothing reaches the start log.

The installer writes that script into the install root when it installs the instance, so it is already there and is not something you create. It is built for that one instance, with its paths, ports and service names baked in, which is also why it is not a file to copy between boxes or between instances.

Use it rather than calling pm2 start ecosystem.config.cjs yourself. It exports the environment that file reads, the instance name, the ports and the server URL, and without it PM2 starts a second set of services under unsuffixed names that collide with the running ones.

The start script restarts every service, not just a missing one

Re-running it hands PM2 the whole service list again, and PM2 restarts the entries that are already running as well as starting the ones that are not. Treat it as a brief outage for the whole instance. To bring back one service that is merely stopped, use pm2 start <service-name>.

After changing a TLS certificate

pm2 restart (and pm2 resurrect after an IPL) replays the environment PM2 already stored, and does not re-read ecosystem.config.cjs. That replay is described in full under After changing a secret, and it applies here too: if you added, replaced or removed the web UI's operator.crt / operator.key, the service has to be recreated from the file rather than restarted:

pm2 delete opsiq-web-ui-<instance>

then run the start script above. Note that pm2 delete removes the service from PM2 rather than stopping it, so pm2 start <service-name> will not bring it back, and the start script restarts the whole instance as described above. Replacing the TLS Certificate sets out the whole procedure, verification included.

After changing a secret

Changing a value in apps/opsiq-server/.env.local — an Anthropic API key, a Jira API token — takes effect on the next restart of the AI server. Put the change in .env.local, not the .env beside it: the installer rewrites .env on every upgrade, and .env.local wins over it. See Configuration Options. Two things are worth knowing.

PM2 keeps a copy of the environment it started each service with. It takes that copy when a service is started from ecosystem.config.cjs, saves it, and replays it on every pm2 restart and after every pm2 resurrect. If a variable was in the shell at that moment, PM2's copy of it lives on. The AI server therefore treats its own configuration files — .env, with .env.local over it — as the authority for its own secrets, and for the settings that decide where it connects and writes: on startup it overrides any inherited copy of ANTHROPIC_API_KEY, AUTH_TOKEN, JIRA_API_TOKEN, JWT_SECRET, SESSION_TOKEN_KEY, TOOLS_API_KEY, AUTH_IBM_OPSIQ_TOOLS_URL, TOOLS_PROXY_URL, OPSIQ_DB_URL, TLS_CERT_FILE and TLS_KEY_FILE with what those files say, and logs which ones it had to override. If you see that message, the saved environment is stale, so clear it with pm2 delete plus the start script (as under Starting an instance from cold), not a plain restart.

For any other setting, the saved environment still wins over .env. The startup log names those too, as the inherited environment overrides .env for…. Clearing it takes the same pm2 delete plus start-script cycle. A value in .env.local is the exception: it wins over both files and over the saved environment, so it needs only a restart.

Do not export a secret in the shell you start the services from. Three things now stop it becoming a problem, and it is still the wrong habit:

1. ecosystem.config.cjs tells PM2 to drop variables whose names look like credentials (KEY, TOKEN, SECRET, PASSWORD, LICEN…) as it reads the file, so an exported secret never enters PM2's saved copy at all — however the services were started.
2. The installer restricts .pm2/dump.pm2 to its owner, in case something lands there anyway.
3. The AI server takes its own secrets from its configuration files (.env, with .env.local over it) rather than the inherited environment, as described above.

Everything else under .pm2 stays readable exactly as before, including the log files Viewing the logs tells you to open.

One consequence worth knowing: a setting whose name contains one of those words cannot be passed to the services by export any more. For the AI server, put it in apps/opsiq-server/.env.local, which it reads directly and which no upgrade rewrites.

Stopping the services

Stopping is for a planned pause: a maintenance window, a save or restore of the install tree, or freeing a port. It leaves the install in place. To remove an instance altogether, see Uninstalling instead.

While the services are stopped, a full install's web UI will not load, and on a tools-only box any remote OpsIQ application that uses this machine's tools server will report failed investigations or tool queries, often as a 503.

pm2 stop opsiq-server-opsiq

Use pm2 stop all for every service in the instance, and pm2 start <service-name> or pm2 start all to bring them back. A stopped service stays registered with PM2, so starting it again brings back exactly what you stopped, without the whole-instance restart that the start script performs.

Run pm2 save only if the stop is meant to outlast a restart

pm2 save records the current list, stopped services included, as the one pm2 resurrect restores. So save after stopping only when you want the services to stay down through a pm2 resurrect. If the stop is temporary, leave the saved list alone, so pm2 resurrect after an IPL still brings everything back. The start script is not bound by the saved list: it starts every installed service regardless and then saves again, which replaces a list you saved with services stopped.

What pm2 save writes, and why its mode matters

pm2 save rewrites PM2's saved list, .pm2/dump.pm2, and copies the previous one to .pm2/dump.pm2.bak. Those two files hold each service's whole start-up environment, so the installer restricts both to their owner and re-asserts that on every run of the start script. PM2 preserves an existing file's mode when it rewrites it, so a pm2 save you run yourself keeps them closed rather than reopening them.

If you ever find a dump file readable by anyone else, run the start script, which sets the mode again. See After changing a secret for what ends up in there and why it matters.

pm2 kill is a different thing from pm2 stop: it also ends the PM2 process that supervises the services. Reach for it only when you mean to shut the instance down completely, and remember the way back up is then a cold start, not pm2 start.

Viewing the logs

Each service writes two files in the instance's PM2 home, one for ordinary output and one for errors:

<install-root>/.pm2/logs/<service-name>-out.log
<install-root>/.pm2/logs/<service-name>-error.log

For a default IBM i install, the AI server's error log is therefore /QOpenSys/opsiq/opsiq/.pm2/logs/opsiq-server-opsiq-error.log.

Reading them is the same on every install: open the file. These are ordinary files, so this is a plain read rather than a pm2 command, and it is the one thing on this page that needs no authority at all. PM2 creates them readable by everyone, under directories everyone can traverse, so any user on the box can read them. That makes it the whole of what an operator without *ALLOBJ can do unaided, and often enough to find the problem. The files are UTF-8, so they read correctly over SSH and in DSPF.

pm2 logs reads the same content through PM2 instead, which means it needs the authority any other pm2 command needs (see Whether PM2 will answer you). Opening the file is simpler and always available, which is why the examples below do that.

IBM i, PASE shell

tail -n 100 /QOpenSys/opsiq/opsiq/.pm2/logs/opsiq-server-opsiq-error.log

IBM i, 5250 command line

DSPF STMF('/QOpenSys/opsiq/opsiq/.pm2/logs/opsiq-server-opsiq-error.log')

Linux shell

tail -n 100 /opt/opsiq/opsiq-opsiq/.pm2/logs/opsiq-server-opsiq-error.log

The AI server's and web UI's own log files

The AI server and the web UI also write every log entry, one JSON object per line, to dated files of their own:

<install-root>/apps/opsiq-server/logs/opsiq-server-<YYYY-MM-DD>.log
<install-root>/apps/opsiq-web-ui/.next/standalone/logs/opsiq-web-ui-<YYYY-MM-DD>.log

A day that outgrows one file continues in opsiq-server-<YYYY-MM-DD>.1.log, .2.log and so on. These files are the more complete record. Each entry reaches the file before the service moves on, whereas the AI server's copy in the PM2 logs is written in the background and can lose the last entries when the service crashes. After a crash, read these files rather than the PM2 logs.

Unlike the PM2 logs, these files are private to the profile the services run under: the logs directory and the files in it are closed to everyone else. On a managed install, reading them therefore takes that profile or *ALLOBJ.

By default each service keeps 30 days of these files, at most five files of 20 MB per day. For the AI server the limits are adjustable, as is how much it logs; see Configuration Options.

The start script keeps a separate log of its own, described under When a managed start fails.

How the logs are bounded

Each instance runs one more PM2 process, opsiq-logrotate-<instance>, which rotates every log in its PM2 home. That covers the services' logs and PM2's own pm2.log. A file is rotated when it passes 10 MB, and every file with content is rotated at midnight. The rotated copy sits beside the live file as <service-name>-out__<date>_<time>.log, and the seven most recent copies of each file are kept. The live file keeps its name, so the paths above stay the ones to open.

To reclaim space sooner, empty a file in place rather than deleting or renaming it. The service holds the file open and would otherwise keep writing to a file you can no longer read:

: > /QOpenSys/opsiq/opsiq/.pm2/logs/opsiq-server-opsiq-out.log

That command writes nothing to the file, which truncates it to zero bytes while leaving it open for the service still writing to it. Check the path carefully before running it: there is no confirmation prompt, and it empties whatever you name.

Reading a log needs no authority, but emptying one does. The logs are writable only by the profile the services run under. On the IBM i a profile with *ALLOBJ can also empty one; on a Linux host, root can.

pm2 flush empties every log in the instance in one step, under the same rule as any other pm2 command, so your profile has to be one PM2 will answer. Do not reach for it while you are investigating a failure, because it discards the evidence along with the space.

The tools server's caller key

The tools server answers only callers that present its key, and the AI server is the one caller that holds it. The key lives in a file the installer creates beside the tools server's configuration:

<install-root>/apps/ibm-opsiq-tools/tools-api.key

The AI server holds the same value as TOOLS_API_KEY in its .env. Both are the installer's to manage: an upgrade keeps the key, and re-running the installer puts the two back in step if they ever differ. Do not edit the file, and never paste the key into a ticket, email or chat. The one reason to copy it is a split install run from two workstations, described under What to install (roles).

The tools server does not start without a valid key file. Under PM2 the service exits as soon as it starts, so pm2 list shows opsiq-tools-<instance> restarting or errored. Its error log, opsiq-tools-<instance>-error.log (see Viewing the logs), then has a line containing server.api key unusable; refusing to start, with the file's path and the reason. When the key loads, the log records server.api key loaded instead. The log never contains the key itself. See The tools server does not start for the fix.

Checking the tools server by hand

Only /health answers without the key, so it is the one address to probe directly:

curl -k https://127.0.0.1:8445/health

Substitute your instance's tools port. Any other address answers 401 to a request that does not carry the X-Opsiq-Tools-Key header, even when the service is healthy. /health also answers 401 to a request that carries the header with the wrong key. The AI server asks it with its own copy of the key, so when the two copies differ, the AI server's /health reports its tools check as failed with HTTP 401 (caller key rejected), and its error log says TOOLS_API_KEY does not match tools-api.key.

After a system restart (IPL)

OpsIQ's services do not yet restart automatically after a full system restart. This is a known limitation, and automatic startup is planned. Until then, start the services again after every IPL:

• Run the start script for your install. It starts every service installed on this box from the ecosystem file, whatever the saved list holds, then saves the list afresh, and it works whether or not PM2 is already running.
• Or, if you prefer, run the setup lines and then pm2 resurrect, which restores the saved list as it stood at the last pm2 save, including any service that was stopped when it was saved.

What you see if the services are not running after an IPL depends on what is installed on the box:

• Full install: the web UI will not load at all.
• Tools-only install: the AI server and web UI run on another machine, so the local symptom is different — that remote OpsIQ application cannot reach this box's tools server, which shows up there as failed investigations or tool queries (often a 503) until the tools service is restarted here.

Telling side-by-side instances apart

Side-by-side instances (see Install location & instances) are separate installs, each with its own PM2 daemon, and PM2_HOME is what selects the one you are talking to. Three things follow:

• pm2 list shows a single instance, the one whose PM2_HOME you exported. Services belonging to other instances never appear, and their absence is not a fault.
• all is scoped to that instance too. pm2 restart all, pm2 stop all and pm2 kill reach only the instance you selected, so they are safe on a shared box, as long as PM2_HOME points where you think it does. Run pm2 list first and check the names that come back.
• Service names carry the instance name, opsiq-server-dev for an instance called dev, so the list itself confirms which install you reached.

Where to point PM2_HOME:

Install	Install root	PM2_HOME
IBM i, current default	/QOpenSys/opsiq/<instance>	/QOpenSys/opsiq/<instance>/.pm2
IBM i, installed before grouped roots	/QOpenSys/opsiq-<instance>	/QOpenSys/opsiq-<instance>/.pm2
Linux host	the --install-root you chose	<install-root>/.pm2

To see which roots exist on an IBM i box, both layouts at once:

ls -d /QOpenSys/opsiq/* /QOpenSys/opsiq-* 2>/dev/null

pm2 command reference

Every command below assumes the setup lines and a profile PM2 will answer. <service> is a full name such as opsiq-server-opsiq; substituting all acts on every service in the instance.

Command	What it does
pm2 list	The instance's services, with status, restart count, uptime and memory
pm2 describe <service>	Everything PM2 knows about one service, its log paths included
pm2 logs <service>	Streams its output. --err for errors only, --nostream --lines 100 for a single dump
pm2 restart <service>	Stops and starts it, replaying the environment PM2 already stored
pm2 stop <service>	Stops it but leaves it registered, so pm2 start can bring it back
pm2 start <service>	Starts a registered service that is currently stopped
pm2 save	Records the current list as the one pm2 resurrect will restore
pm2 resurrect	Restores the saved list, after an IPL or a reboot
pm2 flush	Empties every log file in the instance
pm2 kill	Stops the services and PM2 itself, for this instance only

Two commands to leave to the procedures above

pm2 start ecosystem.config.cjs and pm2 delete <service> both belong to the TLS certificate procedure under Restarting the services, and outside it they are easy to get wrong. Starting the ecosystem file by hand creates a second set of services under unsuffixed names that collide with the running ones, which is why .opsiq-pm2-start.sh exists: it exports the environment that file expects. pm2 delete removes a service from PM2 altogether rather than stopping it, so it will not come back with pm2 start <service>, only by re-running the start script. Use pm2 stop when you simply want a service to pause.

When a managed start fails

On a managed install the start runs as a submitted job on the IBM i, which changes where the evidence lives when it fails. Check the process list first, either with opsiq-install status from your workstation, or with pm2 list from a profile PM2 will answer. Two answers are easy to misread, and neither means the services are down:

• An empty list usually means PM2_HOME was not exported, so PM2 reported on a different and empty home instead of on OpsIQ's.
• Permission denied, check permissions on ...rpc.sock means the reverse: the right instance, reached by the wrong profile.

If pm2 list shows a service as errored or stopped, then PM2 did start, and that service's own error log is the fastest lead. Open the file, as described under Viewing the logs.

If the list is genuinely empty, the start never got as far as running PM2. Where the evidence lives then depends on how far it did get, so work through these in order:

1. Whether the job was created at all. If the SBMJOB was rejected, that happened on your 5250 screen at the time, typically with CPD1616 Not authorized to user profile OPSIQ. No job means no start log entry and no spooled file, so rule this out first or the two sources below will look untouched and tell you nothing.
2. The start log, .opsiq-pm2-start.log in the install directory (/QOpenSys/opsiq/opsiq/.opsiq-pm2-start.log for the default install). The start script sends everything it does to that file, replacing it on each run. A successful start ends with a table of the running services followed by Successfully saved. If the log stops short of that, its last lines name what went wrong.
3. The submitted job's printed output. If the start script could not be run at all, because the path is wrong or the service profile cannot read it, then nothing reaches the start log and the error goes to a QPRINT spooled file belonging to the submitted job instead. List them with WRKSPLF SELECT(OPSIQ), substituting your own service profile.

Usually there is no joblog to check

The default job description leaves message logging at *NOLIST, which suppresses the joblog whenever a job ends normally, and a start script that merely fails does end its job normally. In the usual case there is therefore no joblog to find, and the two sources above are the whole record. A joblog is produced only when the job ends abnormally, so it is worth a look when neither source explains the failure, and if you installed with your own --service-jobd its logging settings may differ.

How OpsIQ appears on the IBM i

Everything above manages OpsIQ through PM2, which is how it is meant to be run. This section covers the separate question of what OpsIQ looks like when you review the system natively, during a capacity review or a performance investigation, or when you or a colleague find jobs you do not recognize.

OpsIQ's services, and the PM2 process that supervises them, are PASE programs, so on the IBM i each one runs as a job named QP0ZSPWP under the service profile. In WRKACTJOB a full install looks like this:

Subsystem/Job  User     Number   Type  Function          Status
  QP0ZSPWP     OPSIQ    319777   BCI   PGM-node          SELW
  QP0ZSPWP     OPSIQ    319785   BCI   PGM-node          SELW
  QP0ZSPWP     OPSIQ    319786   BCI   PGM-ibm-opsiq-    TIMW
  QP0ZSPWP     OPSIQ    319787   BCI   PGM-node          SELW

A few consequences follow from that:

• The long-running services have no OpsIQ-specific job names. QP0ZSPWP is the standard job name for every PASE program on the system, so it is the user that identifies them as OpsIQ: the service profile (OPSIQ by default on a managed install, or your own profile on a direct install). See Service identity.
• A full install is four jobs, not three, and you cannot tell them apart from this view. Three of them are Node programs showing identically as PGM-node: the AI server, the web UI, and PM2's own supervisor process. The fourth is the tools server, whose name is truncated to PGM-ibm-opsiq-. To identify the AI server and the web UI, use their ports (below). The supervisor is then the Node job left over, because it does not listen on any of OpsIQ's ports. A tools-only install is two jobs: the tools server and the supervisor.
• The subsystem depends on how OpsIQ was started, so do not rely on a fixed one. A managed install is submitted as a batch job and normally runs in QBATCH; a direct install is started from your sign-on session and normally runs in QUSRWRK. Either can differ if the job description or job queue has been changed on your system.
• OpsIQ does not create a subsystem, an autostart job entry, or a job schedule entry. A managed install does create the service profile and a job description (QGPL/OPSIQJOBD by default), and it is that job description which decides where the services run.
• Short-lived jobs are normal. The tools server starts a separate child process for each connection, so a small churn of brief jobs under the service profile is expected behavior, not a leak. On a managed install, running an installer command also submits a short-lived job named OPSIQCHK, OPSIQSTRT, OPSIQSTOP, OPSIQSTAT, or OPSIQLOG, which ends on its own.

Finding the job for a particular service

Identify a service by the port it listens on, not by a job number, because job numbers change every time a service restarts. Your instance's ports were shown when it was installed; the defaults are listed under Ports & networking. Run this from Run SQL Scripts or STRSQL, substituting the port you are interested in:

SELECT DISTINCT j.SUBSYSTEM, j.JOB_NAME, j.FUNCTION
  FROM QSYS2.NETSTAT_JOB_INFO n
  JOIN TABLE(QSYS2.ACTIVE_JOB_INFO()) j
    ON j.JOB_NAME = n.JOB_NAME
 WHERE n.LOCAL_PORT = 3939
   AND n.REMOTE_PORT = 0

The REMOTE_PORT = 0 condition restricts the result to the listening service itself. Without it you also get a row for every client currently connected to that port.

Start and stop OpsIQ through PM2, not ENDJOB

Ending a service's job does not stop the service: PM2 notices and starts it again. Ending the PM2 job that supervises them does not stop them either. The services carry on running with nothing supervising them, still holding their ports, and then have to be cleaned up by hand. Either way you end up worse off than when you started, so use the procedures in Restarting the services instead. If a service will not respond to PM2 at all, contact your Fresche contact rather than ending jobs by hand.