OpsIQ

Replacing the TLS Certificate ​

OpsIQ installs with a self-signed certificate, generated on the box at install time. It encrypts every connection from the first minute, but no browser trusts it, so everyone who opens the web UI is met with a security warning and has to click past it. This page covers replacing that certificate with one issued by your own certificate authority, on a system that is already installed and running.

If you have not installed OpsIQ yet, you can supply your certificate during the install instead and skip this page entirely — see Your own certificate for the web UI.

Which certificate you are replacing ​

A full OpsIQ install has three TLS servers, and each one got its own generated certificate. Only the first is yours to replace:

ComponentWhat it servesCertificate
Web UI (opsiq-web-ui)the browser-facing site, port 3000 by defaultreplaceable — this page
AI server (opsiq-server)requests from the web UI, over 127.0.0.1generated, leave it alone
Tools server (ibm-opsiq-tools)requests from the AI servergenerated, leave it alone

The two internal hops dial https://127.0.0.1:<port> and verify the generated certificates directly. A CA-signed certificate is issued for a host name, and a public CA is not permitted to issue one for a loopback address at all, so pointing those hops at your certificate breaks them, and login stops working. Nothing verifies the web UI's certificate, which is exactly why that one can be swapped freely.

The AI server's port is not browser-facing

If a certificate warning on port 3939 is what prompted this, you are looking at the internal AI server. Browsers do not need to reach it. To present a public certificate there anyway, front it with a reverse proxy that terminates TLS. That is outside the installer, and your Fresche contact can advise.

Before you start ​

You need two files, on the machine you run the installer from:

  • The certificate, PEM encoded, holding the full chain: your server certificate (the leaf) first, then any intermediates. The root is optional and is normally left out. A leaf-only file installs, with a warning, but browsers that do not already hold your issuing CA reject it.

  • The private key, PEM encoded and unencrypted. The services start unattended and cannot answer a passphrase prompt. Decrypt a protected key first:

    bash
    openssl pkey -in key.pem -out key-plain.pem

The certificate itself must meet these conditions. The installer checks them before it touches the box, and refuses a pair that cannot work:

RequirementWhy
A subjectAltName that covers the name people type in the browserBrowsers have ignored commonName since 2017. A certificate with no subjectAltName is refused. One whose names do not include the --host you install with only draws a warning, because people may reach the web UI by another name, such as a DNS alias or a load balancer.
A leaf certificate, not a CA certificateA CA certificate cannot serve, and its key does not belong on a customer box.
Inside its validity windowAn expired or not-yet-valid certificate fails in every browser.
RSA of 2048 bits or more, or EC on P-256, P-384 or P-521Anything weaker makes the web UI refuse to start; other key types are rejected by browsers.
Certificate and key in separate files, and a matching pairThe two flags take one file each. A combined PEM is rejected.

Plan the name before you request the certificate. Whatever people type to reach OpsIQ — opsiq.example.com, an alias, a load-balancer name — must appear in the certificate. Prefer a certificate issued for this host over a wildcard.

Method 1 — re-run the installer ​

This is the supported path, and the one to use whenever you still have the installer. Run the same install command you used originally, with the two TLS flags added:

bash
./opsiq-install install --host your-ibm-i.example.com --ssh-user YOURPROFILE \
  --tls-cert ./fullchain.pem --tls-key ./privkey.pem --skip-license
powershell
.\opsiq-install.exe install --host your-ibm-i.example.com --ssh-user YOURPROFILE `
  --tls-cert .\fullchain.pem --tls-key .\privkey.pem --skip-license
bash
./opsiq-install.exe install --host your-ibm-i.example.com --ssh-user YOURPROFILE \
  --tls-cert ./fullchain.pem --tls-key ./privkey.pem --skip-license

Pass both flags or neither. The paths are on your own machine and are resolved against the current directory; a leading ~ is not expanded, so write the path out in full. You can set OPSIQ_TLS_CERT and OPSIQ_TLS_KEY instead of the flags, or tlsCertPath and tlsKeyPath in an answer file.

What the run does:

  1. Reads and validates the pair on your machine, before anything on the target changes. A pair that cannot work is refused while the box is still untouched.
  2. Recognizes the existing instance and performs an in-place upgrade: it keeps the current ports, snapshots the install directory, and restarts the services.
  3. Delivers the pair to the web UI and restarts it from its configuration file, so the new certificate is actually picked up.
  4. Verifies that the web UI serves your certificate, not merely that the files arrived. A failed health check rolls the instance back.

--skip-license tells the installer to leave the license key already on the box alone, so you do not need to supply one again. Chat history, the license, and the sign-in secrets survive the run.

The Anthropic API key is the exception: the installer rewrites the AI server's configuration on every run, so it needs the key again. Set OPSIQ_AI_API_KEY in the shell you run the installer from, as for the original install (see The Anthropic API key), or answer the installer's prompt for it. Without one an interactive run keeps asking, and an unattended run stops before connecting with Missing required config in non-interactive mode: aiApiKey. If you set ANTHROPIC_API_KEY in .env.local, that key still wins — see Upgrading OpsIQ.

This restarts the whole instance

An install run stops and restarts all of OpsIQ's services, so treat it as a short, planned outage rather than a change you make during business hours.

Method 2 — replace the files on the box ​

Use this when re-running the installer is not practical — the installer is not at hand, or the certificate must be swapped without an upgrade. It does the same thing the installer does, by hand, and it skips the validation and the automatic verification, so work through Verify the new certificate afterwards.

There is no configuration file to edit — the file name is the setting

OpsIQ looks for a certificate and a key named operator.crt and operator.key in the web UI's own directory. When both are readable it serves them; when they are absent it serves the generated server.crt and server.key. You do not point anything at them, and there is no setting that can name a different path.

That choice is made once, when the web UI service is created under PM2, and PM2 then stores it. Putting the files in place therefore changes nothing on its own: step 3 recreates the service so the choice is made again. That, and not a configuration edit, is what puts your certificate into service.

Throughout, <root> is the install root (/QOpenSys/opsiq/opsiq for a default IBM i install; on a Linux host it is the --install-root chosen at install time, which has no default), <instance> is the instance name (opsiq by default), and <service-profile> is the profile the services run under: the service profile (OPSIQ by default) on a managed install, or the profile you installed as on a direct install. Which install do you have? tells the two apart.

On a split install, work on the host that runs the web UI

In a split install the web UI and the AI server run on a separate Linux host, and the certificate you are replacing is on that machine, not on the IBM i. Everything below still applies, with two differences: a Linux install is always a direct install, so the IBM i authority rules in the next box do not apply and the Linux rows of Whether PM2 will answer you do instead; and there is no PASE shell, so the pm2 commands run in an ordinary Linux shell after you export PM2_HOME (see PASE shell setup, which carries the Linux form).

You need enough IBM i authority for this method

On a managed IBM i install, both steps 2 and 3 act on files and processes that belong to another profile: chown to a different owner needs *ALLOBJ, and PM2 answers only a profile with *ALLOBJ. Being in the service profile's group does not help on the IBM i, and *JOBCTL does not either — see Whether PM2 will answer you. On a direct install the services are yours, so no special authority is involved. If you do not have that authority, use Method 1 or ask whoever administers the system.

  1. Upload the certificate and the key from your workstation to the web UI's directory, under these exact names. scp over the same SSH access the installer uses is the simplest way:

    bash
    scp ./fullchain.pem YOURPROFILE@your-ibm-i.example.com:<root>/apps/opsiq-web-ui/operator.crt
scp ./privkey.pem   YOURPROFILE@your-ibm-i.example.com:<root>/apps/opsiq-web-ui/operator.key

    Write the real install root in place of <root>, and name the host that runs the web UI — on a split install that is the Linux host, not the IBM i.

    Any other transfer works as well, as long as the files land at exactly those two paths and arrive unaltered. Transfer them as binary, never as an FTP text transfer: a PEM file translated to EBCDIC is unreadable to the web UI.

    Nothing in OpsIQ ever generates those two names, so an upgrade cannot overwrite them. The generated server.crt and server.key stay beside them, untouched and unused.

  2. Make the service profile the owner, and close the files to everyone else. Substitute your own <service-profile>, as described above — a chown to the wrong profile leaves the web UI unable to read its own certificate.

    bash
    chown <service-profile> <root>/apps/opsiq-web-ui/operator.crt <root>/apps/opsiq-web-ui/operator.key
chmod 600 <root>/apps/opsiq-web-ui/operator.crt <root>/apps/opsiq-web-ui/operator.key

    On a direct install you already own the files, so only the chmod is needed.

    The web UI serves your pair only when both files are readable to it. If one is missing or unreadable, it falls back to the generated certificate and writes operator TLS pair unusable to its log.

  3. Recreate the web UI service, so that the check described above runs again and finds your pair. Remove the service from PM2, then start the instance from its start script. Run both as a user PM2 will answer, after the shell setup for your platform:

    bash
    pm2 delete opsiq-web-ui-<instance>

    Then start the instance again, with the form that matches your install:

    bash
    sh <root>/.opsiq-pm2-start.sh
    text
    SBMJOB CMD(STRQSH CMD('/QOpenSys/usr/bin/sh <root>/.opsiq-pm2-start.sh'))
       JOB(OPSIQSTRT) JOBD(QGPL/OPSIQJOBD) USER(<service-profile>) INLLIBL(*JOBD)
    bash
    sh <root>/.opsiq-pm2-start.sh

    A managed install must start the script as the service profile, which is what the SBMJOB form does. Adjust the path, the JOBD and the USER if you installed with values other than the defaults.

pm2 restart does not apply a certificate change

PM2 keeps a copy of the environment each service started with and replays it on pm2 restart — and after pm2 resurrect following an IPL. That replay does not re-read the configuration file that decides which certificate the web UI serves, so a plain restart keeps serving the old one. The pm2 delete and start-script cycle above is what applies the change. Certificates are read only when the process starts, so the same applies to every later renewal.

The start script restarts every service in the instance

It hands PM2 the whole service list again, so the AI server and the tools server are restarted as well. Expect a brief outage for the whole instance. See Starting an instance from cold.

Verify the new certificate ​

Do all three checks. The first proves the server presents the certificate, the second proves your browsers accept it, and the third proves nothing else broke.

The examples name your-ibm-i.example.com, which is right for the common install where everything runs on the IBM i. On a split install, put the host that runs the web UI in its place — the Linux host — in all three checks.

  1. Ask the server what it presents. Use the openssl form in any POSIX shell — macOS, Linux, Git Bash on Windows, or a PASE shell on the IBM i itself — or the PowerShell form on Windows, which needs nothing installed:

    bash
    openssl s_client -connect your-ibm-i.example.com:3000 \
  -servername your-ibm-i.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName
    powershell
    $client = [Net.Sockets.TcpClient]::new('your-ibm-i.example.com', 3000)
$stream = [Net.Security.SslStream]::new($client.GetStream(), $false, { $true })
$stream.AuthenticateAsClient('your-ibm-i.example.com')
$cert = [Security.Cryptography.X509Certificates.X509Certificate2]::new($stream.RemoteCertificate)
$cert | Format-List Subject, Issuer, NotBefore, NotAfter
$cert.Extensions | Where-Object { $_.Oid.FriendlyName -eq 'Subject Alternative Name' } | ForEach-Object { $_.Format($false) }
$client.Close()

    The PowerShell form accepts any certificate on purpose, so that it can report on one the machine does not yet trust. It tells you what is served; step 2 is what proves a browser accepts it.

    The issuer must be your CA, the dates must bracket today, and the subjectAltName must list the name people use. An issuer equal to the subject means the generated self-signed certificate is still being served.

  2. Open the web UI in a browser, at https://your-ibm-i.example.com:3000. There must be no certificate warning. Use a fresh private window: a browser that was shown the old certificate can hold it for the rest of its session.

  3. Confirm the services are healthy. From your workstation:

    bash
    ./opsiq-install status --host your-ibm-i.example.com --ssh-user YOURPROFILE
    powershell
    .\opsiq-install.exe status --host your-ibm-i.example.com --ssh-user YOURPROFILE
    bash
    ./opsiq-install.exe status --host your-ibm-i.example.com --ssh-user YOURPROFILE

    All installed services must be online. status reports on one host at a time, so on a split install run it against the web UI's host for this check, and against the IBM i as well if you want the whole picture. If the web UI is not online, read its log — see Viewing the logs.

Renewing the certificate later ​

A renewal is the same procedure with newer files, by either method. Nothing else changes: the file names stay the same, and the certificate is read only when the web UI process starts, so the restart is the step that must not be skipped. Put a reminder in place well before the expiry date — a certificate that expires while OpsIQ is running keeps being served, and every browser then rejects it.

Going back to the generated certificate ​

Delete the two files, then restart the web UI from its configuration file exactly as in step 3 of Method 2 — the start script differs between a managed and a direct install:

bash
rm <root>/apps/opsiq-web-ui/operator.crt <root>/apps/opsiq-web-ui/operator.key
pm2 delete opsiq-web-ui-<instance>

Then run the start script for your install: sh <root>/.opsiq-pm2-start.sh on a direct install, whether on the IBM i or on a Linux host, or the SBMJOB form on a managed IBM i install.

The generated server.crt is still in place, so the web UI serves it again and the browser warning comes back. This is a complete revert.

Looking after the private key ​

The key sits unencrypted on the box, because the services start unattended. Three things follow:

  • The installer stores it mode 600, owned by the service profile. Keep it that way if you place the file by hand.
  • Any profile with *ALLOBJ on the partition can read it regardless of the mode.
  • Every pre-upgrade snapshot (<root>.backup.<timestamp>, three kept by default) contains a copy, and uninstalling does not remove those snapshots.

If you are replacing the certificate because the old key was exposed, delete the snapshots as well, and have the old certificate revoked by your CA.

Troubleshooting ​

The installer rejects the certificate or the key ​

Symptom
The run stops before anything on the box changes, naming a problem with --tls-cert or --tls-key.
Cause
The pair is validated first, precisely so this is caught while the system is untouched. The message names the defect: the two flags are swapped, both point at the same combined PEM, the key is encrypted, the key does not match the certificate, the leaf is a CA certificate, the certificate has no subjectAltName, or the key is too small.
Fix
Correct the material as the message describes and run again. See Before you start for what each file must contain.

The browser still warns after the change ​

Symptom
The web UI serves a certificate, but the browser reports it as untrusted or issued for the wrong name.
Cause
Most often one of four things: the name in the address bar is not in the certificate's subjectAltName; the file held only the leaf and this browser does not know your issuing CA; the browser cached the old certificate; or the web UI never restarted and still serves the old one.
Fix
Check what is actually served with the openssl s_client command under Verify the new certificate. If it is still the self-signed certificate, redo the restart in step 3 of Method 2. If it is yours, compare the subjectAltName with the address you use, and re-issue the file with the full chain if only the leaf is present.

The web UI logs "operator TLS pair unusable" ​

Symptom
The web UI starts and serves the self-signed certificate again, and its log holds operator TLS pair unusable.
Cause
Only one of the two files is present, or the service profile cannot read them. This is deliberate: an incomplete pair falls back to a working certificate rather than leaving the web UI down.
Fix
Confirm both files exist under the names in Method 2, then re-apply the ownership and mode in step 2 and restart with step 3.

The web UI does not start at all after the change ​

Symptom
pm2 list shows opsiq-web-ui-<instance> as errored, or it restarts repeatedly.
Cause
The pair is readable but cannot be served — an encrypted key is the usual reason, a key below 2048 bits the other.
Fix
Read the service's error log (see Viewing the logs). Decrypt the key with openssl pkey, or obtain a certificate with a stronger key, and repeat the swap. To restore service immediately, revert to the generated certificate.

The AI server and the tools server still show self-signed certificates ​

Symptom
A port scan or a security review reports self-signed certificates on the AI server's and tools server's ports (3939 and 8445 by default).
Cause
This is by design. Those two ports are internal hops that dial 127.0.0.1 and verify the generated certificates, which no public CA can issue for. See Which certificate you are replacing.
Fix
No action. If an internal policy requires a public certificate on the AI server's port, put a reverse proxy in front of it; your Fresche contact can advise.