9. Log management

By the end of this chapter: You know which logs build up where, and you manage logs and core files so that they do not fill the disk.

9.1 Log file map

All logs are in ~/gritdb1/log/.

9.2 Format and levels

• The GritDB log format is [timestamp][level][tid:thread][location] message. The levels are INFO, WARN, and ERROR. The format and levels are the same as in QuickStart 5.1.
• A line is at most 4096 bytes. Longer lines are cut and end with ....
• The first line of a new file after rotation is [GritSurferLogger] Log rotated - old file size: <bytes> bytes. This line has no prefix fields.
• The PostgreSQL log (postgresql.log) uses the PostgreSQL default format. The line prefix is <timestamp> <time zone> [<PID>], the default of log_line_prefix. LOG:, ERROR:, FATAL:, PANIC:, and so on come after it. A failed statement is recorded on the next line, STATEMENT: (9.7). Messages of the GritDB extension (GritSurfer: …) are also written to this file.
• GritDB logs always write at the INFO level. INFO, WARN, and ERROR lines are written. The level keys (*.log_level) in config/grit_*.config do not change the level in 1.1. Do not delete the keys (5.6).
• The log size key (grit_unify.delta.log_file_max_byte) is also not used in 1.1. The rotation size is the fixed value in 9.3. Do not delete the key (5.6).

9.3 Rotation and size

• You can delete or move rotated files (.1 to .20) while GritDB runs.
• When all rotating GritDB logs are full, log/ reaches tens of GB. For disk planning, see 1.3. For the daily size record, see 7.4.

9.4 PostgreSQL log rotation

With the installation default, postgresql.log does not rotate. One file keeps growing, so the operator sets up rotation. The PostgreSQL side is explained in Log file maintenance.

Rotate it with the OS logrotate. The log location and name do not change. Create /etc/logrotate.d/gritdb as root.

<installation directory>/log/postgresql.log {
    su gritdb gritdb
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

• Use an absolute path for <installation directory> (for example, gritdb1 under the home directory of the service account).

• rotate 30 and daily are examples. Match them to your retention period (9.5).

• copytruncate copies the file, then empties it. It rotates without stopping GritDB. A few lines written between the copy and the truncation can be lost. GritDB opens and writes postgresql.log in append mode, so no empty space appears at the front after the truncation.

• To check, run sudo logrotate -d /etc/logrotate.d/gritdb. The output has no error lines. The next day, run ls -l <installation directory>/log/postgresql.log*. It shows a rotated file.

• For the ERROR count on the day of a rotation (QuickStart 5.2, 7.4), count both files together with the command below. logrotate runs once a day (on Ubuntu 24.04, within 1 hour after midnight). Today’s lines written before the rotation move to postgresql.log.1. delaycompress means that .1 is not compressed.

  cat ~/gritdb1/log/postgresql.log.1 ~/gritdb1/log/postgresql.log 2>/dev/null | grep -c "^$(date +%F).*ERROR:"
  

You can also rotate with the PostgreSQL log collector (logging_collector). In that case, the log location changes.

⚠ If you turn on logging_collector, the location and name of the PostgreSQL log change

• Reason: New log lines go to the file that log_directory and log_filename set. The postgresql.log commands in QuickStart 5.2 and chapter 7 do not see the new lines.
• Instead: Use logrotate. If you turned on the collector, write the new location and name in the operations record. In that case, also change the file name in the check commands.

9.5 Retention and cleanup

• For the retention period, follow the log retention rules of your organization.
• From the time of a failure until the support case is closed, do not delete the logs of that period. This is because QuickStart 7.4 collects all of log/.

Delete rotated files past the retention period as the service account. First check the targets with -print.

find ~/gritdb1/log -name '*.log.[0-9]*' -mtime +<days> -print
find ~/gritdb1/log -name '*.log.[0-9]*' -mtime +<days> -delete

Archive the 2 shared logs while GritDB is stopped.

for f in grit_surfer_main.log grit_plug_producer_processor.log; do
  mv ~/gritdb1/log/"$f" ~/gritdb1/log/"$f.$(date +%Y%m%d)"
done

• To check, run ls -l ~/gritdb1/log/. It shows the files with a date suffix. After the next start, check that new files with the original names appear.
• The renamed files are also cleaned up by the find commands above.

9.6 Core files

A core file is a file to which a process writes its memory when it ends abnormally. A GritDB engine core file is as large as the engine memory (tens of GB).

As the service account, check when core files are created and where they go.

ulimit -Hc
cat /proc/sys/kernel/core_pattern

• If ulimit -Hc is 0, no core file is created. The GritDB engine raises its soft limit up to this hard value. The setting is in 1.4.
• If core_pattern is a path, the core file is written there. If it starts with |, the core collection program of the OS receives it. In that case, check the location and configuration of that program.
• If the core file cannot be fully written within 1 minute, it is cut off. Put it on a fast disk.
• For the core location, use a disk with as much free space as the engine memory. If it is on the same file system as datafiles/, a core file can fill the data disk.
• Do not delete a core file until the support case is closed. Delete it after that (7.6).
• When you contact support, report the location and size of the core file (QuickStart 7.4).
• A core file contains the data values that were in memory. For export, see 9.7.

9.7 Export

• ⚠ Logs, core files, and the analytic query text file can contain original SQL text and data values.

Steps before export:

1. As the file to send, choose $S.masked.tar.gz, which you make in step 3 below. It consists of the QuickStart 7.4 collection, the configuration actually in use, and masked logs. $S.tar.gz from QuickStart 7.4 contains unmasked logs, so do not send it. Allow only the service account to read it, or delete it (11.7).

2. Review the places in the table above under the data export rules of your organization.

3. Mask the logs. Make the bundle again from copies without the original SQL text lines. Run the commands in the same shell as QuickStart 7.4 (cd ~, $S).

   for f in ~/gritdb1/log/postgresql.log*; do
     zcat -f "$f" | awk '/(STATEMENT|DETAIL|CONTEXT|QUERY):/ || /LOG: +(duration: [0-9.]+ ms +)?(statement|execute|bind|parse)[^:]*:/ {skip=1; next} skip && /^\t/ {next} {skip=0; print}' > "$S/masked_$(basename "$f" .gz)"
   done
   grep -c -E '(STATEMENT|DETAIL|CONTEXT|QUERY):|LOG: +(duration: [0-9.]+ ms +)?(statement|execute|bind|parse)[^:]*:' "$S"/masked_*
   cp ~/gritdb1/datafiles/pgdata/postgresql.auto.conf "$S/"
   cp -r ~/gritdb1/config "$S/"
   tar -czf "$S.masked.tar.gz" --exclude='postgresql.log*' "$S" -C ~/gritdb1 log
   tar -tzf "$S.masked.tar.gz" | grep -c '/postgresql\.log'
   

   • The lines left out are statements and details attached to errors (STATEMENT:, DETAIL:, CONTEXT:, QUERY:), and statements logged by log_statement and log_min_duration_statement (statement:, execute, bind, parse). Their continuation lines, which start with a tab, are left out too.
   • Rotated files (.1, .2.gz … in 9.4) are masked too. zcat -f reads uncompressed files as they are. The bundle does not include the unmasked postgresql.log* (--exclude).
   • Check that the first grep -c returns 0 for each file and that the last line returns 0.
   • postgresql.auto.conf (ALTER SYSTEM values, 5.3) and config/ (GritDB engine configuration, 5.6) are the configuration actually in use, so send them together with the logs.
   • ERROR: lines can also contain values, so check them in the step 2 review.

4. Core files cannot be masked, so decide with your organization whether to send them. If you cannot send them, say so when you contact support.

5. Record the files sent, the date, and the recipient.

• For detailed collection and export procedures, see TroubleShooting.

If a problem occurs

• If there is not enough disk space, see GDB-RUNTIME-002 in QuickStart, chapter 7, and TroubleShooting.

Learn more

• For routine cleanup, see 7.6. For security, see 11.7.
• PostgreSQL 17 documentation: Log settings, Log file maintenance