4. Start and stop operations

By the end of this chapter: You choose the start and stop methods that fit the situation, and you start GritDB again safely after an abnormal termination.

4.1 Start sequence and completion check

The start command and its expected output are in QuickStart 4.1. gritdb-start goes through the steps below in order.

• If a step goes over its time limit, the start fails and GritDB shuts itself down. In that case, see QuickStart GDB-BOOT-005.

• The steps without a time limit are GritDB engine recovery, catch-up, and PostgreSQL recovery after an abnormal termination. During these steps, gritdb-stop does not stop GritDB (4.2). The supervisor process handles a kill (SIGTERM) it receives as a quick stop after the start finishes.

• gritdb-start waits without output until it finishes. It takes about 30 seconds even with little data, so do not interrupt it.

• To watch the progress, run the command below in another terminal. It shows the current step name.

  grep 'GritAdminPhaseRunner: ===' ~/gritdb1/log/grit_admin_main.log | tail -1
  

• GritDB started successfully appears after accept connections (step 9). When this line appears, you can connect right away.

If you connect during the start, the connection is rejected with one of the messages below. In that case, wait.

• pg_isready shows rejecting connections or no response.

Confirm the start. Run the command again after 20 seconds to catch a shutdown right after the start.

~/gritdb1/bin/pg_isready -p 5432 -d postgres

• It is normal if pg_isready shows accepting connections both times.
• With -d postgres, the server log gets no connection failure lines.

Starting shell:

• Start GritDB from a shell without environment variables that start with PG and without LD_LIBRARY_PATH. The environment of the starting shell passes to the GritDB processes.

  env | grep -E '^(PG|LD_LIBRARY_PATH)'
  

  No output is normal.

• If you set TMPDIR in 1.9 step 10, pass the same value at every start (1.4 ④). If you put the line export TMPDIR=<directory> in the service account’s ~/.profile, you do not forget it at a manual start. For start at boot, use Environment=TMPDIR in 4.6. To check, run echo "${TMPDIR:-/tmp}". It returns the value you set.

  • The TMPDIR of the installation script must be a directory where programs can run (1.4 ④). So if that directory is noexec, pass a different value only when you install.

• Do not run start commands at the same time. The screen output log (log/grit_admin_stdout.log) can be emptied.

• In 1.1, there are no gritdb-start options for operation. All commands and options are in A.1.

4.2 Stop methods and time

• A normal stop shuts PostgreSQL down with Smart Shutdown (SIGTERM). A quick stop uses Immediate Shutdown (SIGQUIT).
• If the screen shows [WARN] Client command failed: …, [WARN] Falling back to direct signal..., and [INFO] Sending SIGTERM to GritAdmin, the normal stop command did not get through, so GritDB went down with a quick stop. In that case, the next start takes the recovery path (4.3 step 5, ABNORMAL_RESTART).
• A normal stop takes about 1 minute when there are no sessions. A quick stop takes about 10 to 15 seconds. If the steps use their full time limits, the stop takes up to about 11 minutes.

For the stop completion check, all of the following must be true. Do not judge by GritDB stopped. on the screen alone. This line also appears during a start.

1. There are 0 processes. To check, run the stop check in QuickStart 4.5. It shows no output.

2. State files and lock

   ls ~/gritdb1/run/
   cat ~/gritdb1/run/grit_admin_clean_shutdown
   flock -n ~/gritdb1/run/gritdb.lock true && echo "lock released = stopped"
   

   • It is normal if ls shows no grit_admin.pid or grit_admin.sock and the last line is lock released = stopped.
   • After a normal stop, cat returns CLEAN_SHUTDOWN. After a quick stop or an abnormal termination, this file does not exist.

Disconnect remaining sessions before the stop. After the stop begins, new connections are blocked, but open sessions keep running SQL.

1. Stop the applications, reports, BI tools, CDC tools, and ETL tools.
2. Connect with ~/gritdb1/bin/psql -p 5432 -d postgres and run the query below.
3. Run the same query again and check that it returns 0 rows.
4. Run ~/gritdb1/bin/gritdb-stop.

SELECT pg_terminate_backend(pid) FROM pg_stat_activity
WHERE backend_type = 'client backend' AND pid <> pg_backend_pid()
  AND application_name NOT LIKE 'gritdb\_catalog\_%';

• Connections that start with gritdb_catalog_ are GritDB internal connections, so do not disconnect them.
• If you stop without ending them, the end of the output shows [WARN] Client command failed: ERROR stop failed … and [WARN] Falling back to direct signal.... These lines mean that GritDB went down with a quick stop. In that case, the clean-shutdown marker run/grit_admin_clean_shutdown does not exist, and the next start goes through recovery (4.3).
• Function and view: pg_terminate_backend, pg_stat_activity

Stop commands during a stop or a start:

• While a normal stop is in progress, gritdb-stop --quick from another terminal cannot cut in. It waits until the first stop finishes.
• During a start, gritdb-stop does not stop GritDB. The PID file does not exist yet (GritAdmin PID file not found), so it sends nothing. The screen shows GritDB stopped., but GritDB keeps running after the start finishes. So stop GritDB again after the start finishes. The screen of the stop command you ran during the start also shows the warnings POSIX shared memory segments remaining … and Remove them manually only when no GritDB process is running. The GritDB that is starting uses this shared memory, so do not delete it (the check in QuickStart GDB-SHUTDOWN-001). To check whether GritDB is starting, see “Starting” in 4.5.
• If you must shut down urgently, use gritdb-stop --quick from the beginning.

Automation:

• gritdb-stop has no wait time limit, so set a time limit in your script. Make it longer than the maximum normal stop time (about 11 minutes), for example 15 minutes.

• If the time limit passes, the stop continues inside GritDB. An additional --quick cannot cut in. So a person investigates as described in QuickStart 7.4.

  timeout 900 ~/gritdb1/bin/gritdb-stop || echo "STOP TIMEOUT"
  

• The supervisor process handles a kill (SIGTERM) sent directly to it as a quick stop. So do a normal stop only with gritdb-stop. gritdb-stop also sends this signal when the stop command does not get through.

• To restart, run gritdb-stop first, then gritdb-start. There is no separate restart command (QuickStart 4.6).

4.3 Behavior on a crash and starting again

• If one GritDB process terminates abnormally, all of GritDB stops. The same applies when one PostgreSQL backend terminates abnormally (including a forced kill by the OS). There is no automatic restart. It is the same as PostgreSQL with restart_after_crash turned off (5.3).
• Even if you end only the GritDB engine process or only the GritDB catalog process, all of GritDB stops immediately.
• Do not send the HUP signal (kill -HUP) to GritDB processes. If you do, the process terminates.

Steps after an abnormal termination:

1. Finish the recording and diagnostic data collection in QuickStart 7.4. The next start overwrites log/grit_admin_stdout.log. Check the size of log/GRIT_SURFER_ASSERT.log. If it is larger than 0, there is an abort record.

2. For up to 1 minute, run the stop check in QuickStart 4.5 to confirm that all GritDB processes are gone. If any remain after 1 minute, see QuickStart 7.4 and contact support (chapter 0).

3. Run ~/gritdb1/bin/gritdb-stop to clean up the remaining PID file. If the screen shows the warning GritAdmin PID file exists but process not running or GritAdmin PID file not found, that is normal.

4. Run ~/gritdb1/bin/gritdb-start.

5. Check the start path.

   grep DetectScenario ~/gritdb1/log/grit_admin_main.log | tail -1
   

   It is normal if the line contains ABNORMAL_RESTART. It means that GritDB started on the recovery path after an abnormal termination.

6. Verify with 8.5 (apply check and row count comparison).

The next start runs recovery on the abnormal termination path. Recovery has no time limit (4.1).

• During PostgreSQL recovery (redo), log/postgresql.log gets these lines in order: redo starts at <LSN> → every 10 seconds redo in progress, elapsed time: <seconds> s, current LSN: <LSN> → redo done at <LSN>. The time is proportional to the WAL that built up after the last checkpoint. The tuning block sets checkpoint_timeout to 1h and max_wal_size to 64GB (5.2), so it can take longer than with the PostgreSQL defaults. While current LSN moves forward, wait. If you force it to end, the next start runs redo again from the beginning.
  • To check, run grep -E 'redo (starts|in progress|done)' ~/gritdb1/log/postgresql.log | tail -3. log_startup_progress_interval sets the interval (default 10 seconds).
• Watch the recovery progress in log/grit_recovery_worker_process.log. Every 5 seconds, it gets a [WARN] level line RecoveryWorker: recovery queue not drained after <ms> ms — Plug/Unify pipeline may be blocked while catalog writes are deferred. The whole line, through may be blocked … at the end, is fixed text. While <ms> keeps growing, recovery is in progress, so do not interrupt the start (4.1).
• The start is complete when log/grit_admin_main.log has a PID file written line after === OpenSessionGate ===, and then the screen shows GritDB started successfully.

4.4 Clean up remaining processes and shared memory

What gritdb-stop cleans up by itself:

• Deletes the remaining PID file.
• Ends the GritDB engine and PostgreSQL processes that remain after the supervisor process goes down (it matches executable names).
• For the remaining shared memory of this installation, reports only the count and does not delete it. For details, see QuickStart GDB-SHUTDOWN-001.

Check what remains after the stop.

1. Check the processes with the stop check in QuickStart 4.5. If there is output, wait 1 minute and check again. If processes still remain, do not end them one by one with kill. In that case, see QuickStart 7.4 and contact support (chapter 0).
2. Shared memory remains when the supervisor process is forced to end. If the remaining shared memory warning appears after the stop, confirm that there are 0 processes. Then clean up as described in QuickStart GDB-SHUTDOWN-001.
3. If the same warning repeats even after a normal stop or a quick stop, see QuickStart 7.4 and contact support (chapter 0).

Process names:

• Identify processes by the comm name (pgrep -x). If the installation directory path is short, the command-line name can appear as the executable path.
• If there are several installations, the names are the same, so tell them apart with /proc/<PID>/exe (2.9).

4.5 Status check

There is no status command. Judge the state from the lock, run/grit_admin.pid, the process names, and pg_isready together.

flock -n ~/gritdb1/run/gritdb.lock true && echo "lock released = stopped"
while IFS=: read -r k p || [ -n "$k" ]; do
  printf '%-12s %-8s %s\n' "$k" "$p" "$(ps -p "$p" -o comm= || echo DEAD)"
done < ~/gritdb1/run/grit_admin.pid
~/gritdb1/bin/pg_isready -p 5432 -d postgres

• If the state is normal (running), the first line shows no output. The 4 PID lines show GritAdmin, GritSurfer, postgres, and GritCatalog once each, with no DEAD. The last line is accepting connections.
• If GritDB is stopped, the first line is lock released = stopped. The second line is …/grit_admin.pid: No such file or directory (shell error). The last line is no response.
• If GritDB is starting, the first line shows no output. The second line is the same No such file or directory. The last line is rejecting connections. In that case, check again a little later.
• If there are several installations, tell them apart with /proc/<PID>/exe instead of the process names (2.9).

4.6 Start at server boot

The product does not include a service unit file. If you use systemd, create /etc/systemd/system/gritdb.service based on the example below. Do this as root.

[Unit]
Description=GritDB
After=network-online.target
Wants=network-online.target
RequiresMountsFor=<data volume real path>

[Service]
Type=oneshot
RemainAfterExit=yes
User=gritdb
Environment=TMPDIR=<directory in the data volume>
ExecStart=<installation directory>/bin/gritdb-start
ExecStop=<installation directory>/bin/gritdb-stop
TimeoutStartSec=infinity
TimeoutStopSec=20min
LimitNOFILE=65536
LimitCORE=<core limit>

[Install]
WantedBy=multi-user.target

• The shell environment that systemd starts has no variables that start with PG. This meets the starting shell conditions in 4.1.
• Even when GritDB runs as a service, shared memory can be removed when someone logs in as gritdb and then logs out. So do 1.4 ⑤ (RemoveIPC) first.
• Service example for PostgreSQL: Starting the Database Server

Enable and start (root):

systemctl daemon-reload
systemctl enable gritdb
systemctl start gritdb

• If a GritDB that you started by hand is running, first stop it with gritdb-stop, then run systemctl start.
• During a start, when systemctl status shows activating, do not run systemctl stop gritdb. Stop it after it becomes active.
• Check that the status line of systemctl status gritdb is active (exited). GritDB keeps running after the start command ends (Type=oneshot). Also check with 4.5.
• If you started GritDB with systemd, also stop it with systemctl stop gritdb. After an abnormal termination, do 4.3 steps 3 and 4 with systemctl stop gritdb and systemctl start gritdb, respectively.
• If you start GritDB from a job scheduler or a CI job runner, GritDB can terminate when that job ends. So start it with the service manager (systemd).

4.7 Planned shutdown (OS patch, reboot)

1. For an analytics setup, first check how long it can stay stopped (below).
2. Collect a baseline before the work (7.7).
3. Stop the applications, disconnect the remaining sessions, and then do a normal stop (4.2).
4. Confirm that GritDB stopped (4.5 and the stop check in QuickStart 4.5).
5. Do the OS work.
6. Check glibc again (QuickStart 2.4 checklist item 4, min_glibc or later). Also check these 1.4 items again: open file limit, kernel.core_pattern, time synchronization, clocksource, and RemoveIPC.
7. Start GritDB (4.1). Then check with 4.5. If you enabled start at boot (4.6), check with 4.5 after the reboot.
8. Check the “At patching” items in 7.3. For an analytics setup, check that it catches up (7.5).

The time the analytics setup can stay stopped is the operational database’s max_slot_wal_keep_size divided by its WAL generation rate. For details, see 6.4.1 step 1 and 6.4.3. If max_slot_wal_keep_size is -1, divide the free disk space of the operational database instead. If the stop goes over, take the tables again from the start (8.6).

To get the WAL generation rate, measure twice on the operational database with an interval (for example, 1 hour during a busy period).

SELECT pg_current_wal_lsn();

SELECT pg_size_pretty(pg_wal_lsn_diff('<later value>', '<first value>'));

• The result is the size of the WAL that built up during that interval. The functions are described in pg_current_wal_lsn and pg_wal_lsn_diff.

If a problem occurs

• See the GDB-BOOT and GDB-SHUTDOWN entries in QuickStart, chapter 7, and TroubleShooting.

Learn more

• For monitoring metrics, see chapter 7. For logs, see chapter 9.
• PostgreSQL 17 documentation: Shutting Down the Server, Starting the Database Server, Managing Kernel Resources (RemoveIPC), and Server Signaling Functions