7. When problems occur

By the end of this chapter: You solve common errors yourself. For problems you cannot solve, you keep the evidence and hand them over correctly.

Time: As needed

7.1 How to read errors

Where errors appear

How to find an entry — Search this chapter for part of the error text (for example, jq is required). Then go to the GDB- entry.

Action levels

7.2 Symptom map

Installation fails ─────────────── [ERROR] · ERROR: text on the screen ───▶ GDB-INSTALL-001 to 003 · 005
Settings are gone after a reinstall or upgrade ───────────────────────────▶ GDB-INSTALL-004
GritDB does not start ──────────── gritdb-start output ───────────────────▶ GDB-BOOT-001 to 006
Table registration fails ─────────────────────────────────────────────────▶ GDB-RUNTIME-001
Disk is full · cannot open files ─────────────────────────────────────────▶ GDB-RUNTIME-002 · 003
A process died · hangs · data looks wrong ────────────────────────────────▶ GDB-RUNTIME-004 → 7.4
Warnings remain after the stop · stop does not finish ────────────────────▶ GDB-SHUTDOWN-001 · 002

7.3 Common errors

GDB-INSTALL-001 · Installing as root

Symptom — installation screen

▶ 3.1 Initializing PostgreSQL data directory               ... FAIL

Install failed at step: 3.1 Initializing PostgreSQL data directory
See full log at: /root/gritdb1/log/install_gritdb_<timestamp>.log
  | ERROR: running as root: --user <service-account> is required (initdb refuses root).

Cause: PostgreSQL does not create a data directory as root.

Fix

1. /root/gritdb1 is the directory that the failed installation just created. It has no data. As root, delete it.
2. su - gritdb
3. Install again with chapter 3. If you have a reason to install as root, see Admin, chapter 2.

Level: L1 · Type: Procedure

GDB-INSTALL-002 · jq is missing

Symptom

[ERROR] jq is required to read the manifest. Install jq first.

Cause: The installation script needs jq to read the product information (manifest.json).

Fix

1. As root, install jq from the OS installation media or your organization’s internal repository (on Ubuntu, apt install jq).
2. Run the installation command again.

Level: L1 · Type: Environment

GDB-INSTALL-003 · Delivered files fail the integrity check

Symptom — one of these three

gritdb-<...>.tar.gz: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

ERROR: hash mismatch: <file>
ERROR: dist verification FAILED. Do not install this tree.

  <file> missing!  FAIL

Cause: Damage during transfer, a SHA256SUMS of another version, or files changed after extraction.

Fix

1. Stop the installation.
2. Delete the extracted directory (~/pkg/gritdb-*/).
3. Get the installation file and SHA256SUMS again. Get SHA256SUMS through a different channel from the installation file.
4. Start again from section 2.3.

Level: L1 · Type: Environment

GDB-INSTALL-004 · Damaged managed block in postgresql.conf

Symptom — After a reinstall or upgrade, settings that you put in postgresql.conf are gone. The installation screen shows no error.

Cause: Someone deleted or edited the start or end marker line of the managed block by hand. Then the installation command ran again.

Check — Run the same command before an installation or upgrade, too.

grep -n "gritdb managed" ~/gritdb1/datafiles/pgdata/postgresql.conf

It is normal if each of the two lines below appears once.

<line number>:# >>> gritdb managed begin
<line number>:# <<< gritdb managed end

Fix

1. Before an installation or upgrade, restore the missing or changed marker lines exactly as shown above. Then run the installation command. The start marker is the first line of the block. The end marker is the line after the last line of the block.
2. If settings are already gone, add them again with ALTER SYSTEM (Admin, chapter 5).
3. If you do not know which settings are gone, contact support (see 7.4).

Level: L1 · Type: Procedure

GDB-INSTALL-005 · Installation directory path is too long

Symptom

[ERROR] Prefix path is too deep: socket path would exceed the 108-byte unix socket limit.
[ERROR] Choose a shorter --prefix (e.g. $HOME/gritdb1).

Cause: The installation directory path followed by /run/grit_admin.sock exceeds the operating system limit. The installation directory path must be 80 characters or fewer.

Fix: Install in the standard path without options. If the home directory path itself is long, see Admin, chapter 2 to install in a shorter path.

Level: L1 · Type: Procedure

GDB-BOOT-001 · Starting from a directory where GritDB is not installed

Symptom

ERROR: not an installed GritDB tree (missing .gritdb_installed): <path>
       Run install_gritdb.sh first.
[ERROR] Run install_gritdb.sh first.

When it happens: You ran bin/gritdb-start in a directory that you only extracted (~/pkg/gritdb-*/), or the installation failed partway.

Cause: The path does not have the install anchor .gritdb_installed, so the start script refuses it.

Fix

1. Look at <path> in the message.
2. If it is ~/pkg/..., run ~/gritdb1/bin/gritdb-start instead.
3. If it is ~/gritdb1, the installation did not finish. In that case, install again with chapter 3.

Level: L1 · Type: Procedure

GDB-BOOT-002 · Already running

Symptom

[WARN] GritAdmin is already running (lock held: /home/gritdb/gritdb1/run/gritdb.lock)
[WARN] Use gritdb-stop to stop it first

Cause: GritDB is already running.

Fix: Confirm that it is normal with section 4.3. If you want to start it again, restart it with 4.6.

Level: L1 · Type: Procedure

GDB-BOOT-003 · PostgreSQL started with pg_ctl stops after 60 seconds

Symptom: PostgreSQL that you started with pg_ctl start shuts itself down after about 60 seconds. log/postgresql.log records a failure to connect to the GritDB shared memory.

Cause: PostgreSQL started without the supervisor process (grit_admin). This falls under prohibition 1.

Fix

1. Wait about 1 minute. PostgreSQL shuts itself down.
2. Run command 1 in section 4.3 and confirm that there is no output. If processes remain after 2 minutes, see 7.4.
3. ~/gritdb1/bin/gritdb-start

Level: L1 · Type: Procedure

GDB-BOOT-004 · glibc version is too old

Symptom — during installation or start

<...>/libc.so.6: version `GLIBC_2.<n>' not found (required by <...>)

Cause: The glibc on the server is older than the version that GritDB requires.

Check: Use item 4 of the checklist in chapter 2.

Fix: You cannot install GritDB on this server. Change to an OS whose glibc is the required version or later.

Level: L1 · Type: Environment

GDB-BOOT-005 · Supervisor process exits during startup

Symptom — one of these two

[ERROR] GritAdmin process exited unexpectedly during startup
[ERROR] Check logs: /home/gritdb/gritdb1/log/grit_admin_stdout.log
[ERROR] Check logs: /home/gritdb/gritdb1/log/grit_admin_main.log

[ERROR] GritAdmin process exited during initialization
[ERROR] Check logs: /home/gritdb/gritdb1/log/grit_admin_main.log

Check — Read the end of the three logs.

tail -20 ~/gritdb1/log/grit_admin_stdout.log
grep "\]\[ERROR\]\[" ~/gritdb1/log/grit_admin_main.log | tail -20
tail -20 ~/gritdb1/log/postgresql.log

Level: L1 to L3, depending on the check result

GDB-BOOT-006 · Cannot start after only pgdata was replaced

Symptom: The start fails, and the logs show both ResolveStartLsn and unreadable.

When it happens: Someone deleted and recreated only datafiles/pgdata, or replaced it with a pgdata from another location (prohibition 2).

Cause: The point that datafiles/checkpoint refers to does not exist in the new pgdata.

Fix — It depends on whether you must keep the data.

1. If this is a production environment and the data matters, leave everything as it is and contact support (see 7.4).

2. If this is a test environment and you can discard the data, reinitialize. The existing data stays under a new name (Admin, chapter 3).

   cd ~/pkg/gritdb-*/
   ./install_gritdb.sh --reinit-data
   

Level: L3 for production, L2 for test · Type: Procedure

GDB-RUNTIME-001 · Table registration is rejected

Symptom — psql

ERROR:  Table public.<table> must have REPLICA IDENTITY FULL. Current setting: DEFAULT. Run: ALTER TABLE public.<table> REPLICA IDENTITY FULL;

Or

ERROR:  Relation public.<table> not found

Cause: The table does not meet the registration condition (REPLICA IDENTITY FULL), or there is a typo in the schema or table name.

Fix

1. If the first error appears, run the ALTER TABLE ... REPLICA IDENTITY FULL; that the message shows. Then register again.
2. If the second error appears, correct the names in register_gritsurfer_table('<schema>', '<table>').

Level: L1 · Type: Procedure

GDB-RUNTIME-002 · Not enough disk space

Symptom: The logs or the SQL screen show No space left on device, or usage in check 3 of chapter 5 is close to 100%.

Check

df -h ~/gritdb1/datafiles ~/gritdb1/log
du -sh ~/gritdb1/datafiles/* ~/gritdb1/log

Fix

1. Clean up files outside GritDB or extend the volume to get more space.
2. ⚠ Do not delete files in datafiles/ (prohibition 2). Do not delete the replication slot either (prohibition 3).
3. If log/ is large, clean it up with the procedure in Admin, chapter 9.
4. If datafiles/pgdata/pg_wal is large, do not delete it. Contact support instead (see 7.4).
5. After you get more space, run gritdb-stop → gritdb-start. If GritDB hangs with no response, see 7.4 before you restart.

Level: L2 · Type: Environment

GDB-RUNTIME-003 · Open file limit exceeded

Symptom: Too many open files appears in the logs.

Check

ulimit -n

Fix

1. As root, raise the limit of the gritdb account to 65536 or more (Admin, chapter 1).
2. Log in again as gritdb and check ulimit -n.
3. gritdb-stop → gritdb-start
4. If the error appears again after you raise the limit, contact support (see 7.4).

Level: L2 · Type: Environment

GDB-RUNTIME-004 · Process exit · no response · data problem

Symptom — any one of these

• Check 1 in chapter 5 shows fewer than four processes.
• SQL does not respond.
• log/GRIT_SURFER_ASSERT.log has a new record.
• Query results differ from what you expect.

Cause: It is an internal engine error. The support organization analyzes the cause.

Check — First look for traces of a recent reinitialization.

ls -d ~/gritdb1/datafiles.old-* 2>/dev/null

• If output appears, someone may have reinitialized, and the previous data may remain under a new name. In that case, consider a restore with Admin, chapter 3.
• If there is no output, go to the fix.

Fix: Follow the procedure in 7.4. Collect diagnostic data first, before you restart or reinitialize.

Level: L3 · Type: Engine

GDB-SHUTDOWN-001 · Shared memory remains after the stop

Symptom — at the end of the gritdb-stop output

[WARN] POSIX shared memory segments remaining: <count> (see /dev/shm, suffix _<instance ID>)
[WARN] Remove them manually only when no GritDB process is running

Check — Confirm that GritDB stopped completely.

pgrep -u gritdb -l -x 'GritAdmin|GritSurfer|GritCatalog|postgres'

No output means that GritDB stopped completely.

Fix — Do this only when the check shows no output.

ID=$(~/gritdb1/bin/grit_instance_id ~/gritdb1)
ls /dev/shm | grep "_${ID}$"
rm -f /dev/shm/*_"${ID}"

• Delete only the segments that end with the instance ID of this installation. Do not touch the shared memory of other GritDB installations.
• If processes remain, do not delete anything. In that case, see 7.4.
• [WARN] Killing orphaned ... means that gritdb-stop cleaned up the remaining processes. Check once more with the check command.

Level: L1 · Type: Environment

GDB-SHUTDOWN-002 · Stop does not finish

Symptom: gritdb-stop stays at Sending: grit_admin_client stop for a long time. Or the end of the output shows [WARN] Client command failed: ERROR stop failed … and [WARN] Falling back to direct signal....

Cause: A normal stop waits up to 5 minutes for running statements to end. If open sessions (idle ones included) remain, it switches to a quick stop after about 30 seconds.

Check — In another terminal, look at the remaining sessions. Lines that show a user name and a DB name are connected sessions.

ps -u gritdb -o pid,args | grep "postgres: "

Fix

1. First end the connected applications. To end the remaining sessions, see Admin 4.2.
2. If you cannot end them, wait. The stop then switches to a quick stop. Work in progress is cut off, and the next start runs recovery.

• While a stop is in progress, gritdb-stop --quick in another terminal waits until the first stop finishes.

Level: L1 · Type: Procedure

7.4 If the problem remains — collect diagnostic data and contact support

1. Stop what you are doing — Do not reinitialize (--reinit-data), delete files in datafiles/, or delete the replication slot. These actions destroy the evidence of the cause.

2. If GritDB does not respond, contact support first — Some information is available only while GritDB hangs. Call the support organization before you restart.

3. Record — the time of occurrence, the last action, the error text on the screen, and the matching GDB- number

4. Collect diagnostic data

   cd ~
   S=gritdb_support_$(date +%Y%m%d_%H%M%S)
   mkdir -p "$S"
   { date; uname -a; free -g; df -h; ulimit -a; } > "$S/system.txt" 2>&1
   for p in $(cut -d: -f2 ~/gritdb1/run/grit_admin.pid 2>/dev/null); do echo "== $p"; cat /proc/$p/limits; done >> "$S/system.txt" 2>&1
   ps -u gritdb -o pid,etime,args > "$S/process.txt" 2>&1
   ls -l /dev/shm > "$S/shm.txt" 2>&1
   cp ~/gritdb1/share/gritdb/manifest.json ~/gritdb1/datafiles/pgdata/postgresql.conf ~/gritdb1/datafiles/pgdata/postgresql.auto.conf ~/gritdb1/config/grit_*.config "$S/" 2>/dev/null
   tar -czf "$S.tar.gz" "$S" -C ~/gritdb1 log
   ls -lh "$S.tar.gz"
   

   • If a core file exists, report its location too. Check the location with cat /proc/sys/kernel/core_pattern.

5. ⚠ Review before you send data out — Logs and settings can contain original SQL text, data values, and paths. Review and mask them under the data export rules of your organization, then send them. For the masked bundle ($S.masked.tar.gz), see Admin, chapter 9. Do not send the unmasked $S.tar.gz.

6. Contact support — The contact is Datagrit. If you bought GritDB from a distributor or partner, contact that company. Send the collected file and the record from step 3.

7. Restart — After the collection, if the service must recover quickly, run gritdb-stop → gritdb-start. Reinitialize only when the support organization tells you to.

Learn more

• For the full error list, in-depth diagnosis, and details of the collection procedure, see TroubleShooting.