A. Reference
By the end of this chapter: You find options, settings, SQL functions, log files, directories, diagnostic queries, and terms in one place.
A.1 Commands and options
- The authoritative source for installation options is the table in 2.1 of chapter 2.
gritdb-start and gritdb-stop options:
| Command | Option | Meaning |
|---|---|---|
gritdb-start |
(none) | Starts GritDB (4.1). There are no options for production use |
gritdb-start |
-h, --help |
Shows the usage |
gritdb-stop |
(none) | Normal stop (4.2) |
gritdb-stop |
-q, --quick |
Quick stop (4.2). Unlike SIGKILL to Engine in the usage text (--help), the GritDB engine also receives SIGQUIT first. So core files can be created (9.6). Only processes that do not end within 60 seconds are killed |
gritdb-stop |
-h, --help |
Shows the usage |
- Other options in the
gritdb-startusage (--asan,--valgrind,--crash-policy) are for development and testing, so do not use them. If you pass--crash-policy, the start fails. - When
gritdb-startreceives an unknown option, it printsUnknown option: <option>and does not start. gritdb-stopignores unknown options and runs a normal stop. So for a quick stop, check the spelling (--quick,-q).- Neither command has a time limit on how long it waits to finish (4.1, 4.2).
Exit codes:
| Command | Exit code | Meaning |
|---|---|---|
gritdb-start |
0 |
GritDB started successfully! means that GritDB is ready to accept connections (4.1) |
gritdb-start |
1 |
GritDB did not start. Causes: a path where GritDB is not installed (Run install_gritdb.sh first., QuickStart GDB-BOOT-001), already running (GritAdmin is already running, QuickStart GDB-BOOT-002), the supervisor process exited during the start (GritAdmin process exited unexpectedly during startup, QuickStart GDB-BOOT-005), or an unknown option |
gritdb-stop |
0 |
Ran to the end. It is 0 even if GritDB was already stopped or was starting. So use 4.2 and 4.5 to judge whether GritDB stopped |
gritdb-stop |
1 |
A path where GritDB is not installed (Run install_gritdb.sh first.) |
- An abnormal termination after the start completes does not show in the
gritdb-startexit code. So recheck after 20 seconds, as in 4.1.
Environment variables:
- There are no environment variables for the customer to set. GritDB commands do not read
GRITDB_HOME,PGDATA, orPGPORT. They find the installation directory from the location of the executable. - Start GritDB from a shell without environment variables that start with
PGand withoutLD_LIBRARY_PATH(4.1). - The
TMPDIRvalue in the starting shell becomes the temporary file location of the GritDB engine (1.4, 7.6).
| Category | bin/ commands |
|---|---|
| Use | gritdb-start, gritdb-stop, psql, pg_isready, pg_dump, pg_dumpall, grit_instance_id (with multiple installations, 2.9) |
| Do not run directly | grit_admin, grit_admin_client, grit_catalog_process, grit_surfer_process, postgres (including single-user mode; to start GritDB, see 4.1), initdb (for reinitialization, see 3.1) |
| Do not use | pg_ctl, pg_upgrade, pg_resetwal, pg_rewind, pg_basebackup (for backups), clusterdb, vacuumdb --full, pg_createsubscriber (for the analytics setup, see 6.4), pg_checksums (not tested with GritDB) |
| Only when support asks for it | gritdb_cf |
- Use
bin/commands not in this table (createdb,pg_controldata,pg_waldump, and so on) as the PostgreSQL 17 documentation describes. Do not run operations that rewrite registered tables (6.7).
A.2 Settings
| Name | Location | Default | Range and unit | Takes effect | How to change |
|---|---|---|---|---|---|
shared_preload_libraries |
Required block in the managed block | 'gritsurfer' |
— | Restart | Do not change |
wal_level |
Required block in the managed block | logical |
— | Restart | Do not change |
search_path |
Required block in the managed block | '"$user", public, gritdb' |
— | reload | Do not change. The service account creates objects with the schema name (11.5) |
port |
Managed block | Value set at installation (default 5432) | 1 to 65535 | Restart | install_gritdb.sh --pgport (2.5) |
shared_buffers |
Tuning block in the managed block | Calculated at installation (5.5) | — | Restart | superuser, ALTER SYSTEM (5.3) |
wal_buffers |
Tuning block in the managed block | 64MB |
— | Restart | superuser, ALTER SYSTEM |
max_wal_size |
Tuning block in the managed block | 64GB |
— | reload | superuser, ALTER SYSTEM |
min_wal_size |
Tuning block in the managed block | 2GB |
— | reload | superuser, ALTER SYSTEM |
checkpoint_timeout |
Tuning block in the managed block | 1h |
— | reload | superuser, ALTER SYSTEM |
bgwriter_lru_maxpages |
Tuning block in the managed block | 1000 |
— | reload | superuser, ALTER SYSTEM |
effective_cache_size |
Tuning block in the managed block | Calculated at installation (5.5) | — | reload | superuser, ALTER SYSTEM |
work_mem |
Tuning block in the managed block | Calculated at installation (5.5) | — | reload | superuser, ALTER SYSTEM. In a session, SET |
restart_after_crash |
PostgreSQL (fixed by GritDB) | off |
— | — | Cannot be changed. If you write it in postgresql.conf, GritDB does not start (5.3) |
grit_surfer.catalog.read_threads |
config/grit_surfer.config |
4 |
1 to 64 (count) | Restart | Service account, file (5.6) |
grit_surfer.channel.shm_limit_mb |
config/grit_surfer.config |
512 |
16 to 65536 MB | Restart | Service account, file (5.6) |
grit_burst.session.connect_timeout_ms |
config/grit_burst.config |
300000 |
1 to 3600000 ms | Restart | Service account, file (5.6) |
grit_unify.delta.log_level, grit_plug.consumer.log_level, grit_burst.query.log_level |
config/grit_unify.config, grit_plug.config, grit_burst.config |
info |
— | Not used in 1.1 (9.2) | Do not delete |
grit_unify.delta.log_file_max_byte |
config/grit_unify.config |
10485760 |
Bytes | Not used in 1.1 (9.2) | Do not delete |
gritsurfer_session_mode |
Extension setting | pg |
pg, surfer |
At connection only | All accounts, connection option (6.6). Do not set it globally |
gritsurfer_session_fallback |
Extension setting | on |
on, off |
Session (SET) |
All accounts |
gritsurfer_write_visibility_guard |
Extension setting | on |
on, off |
Session | superuser only |
- Meaning of the settings (PostgreSQL):
shared_preload_libraries,wal_level,search_path,port,shared_buffers,wal_buffers,max_wal_size,min_wal_size,checkpoint_timeout,bgwriter_lru_maxpages,effective_cache_size,work_mem,restart_after_crash - The managed block is rewritten at every installation, upgrade, and reinitialization. So set values that you want to keep with
ALTER SYSTEM(5.3). - To check when a setting takes effect, run
SELECT name, setting, context, pending_restart FROM pg_settings WHERE name = '<setting>';(5.4). - Other PostgreSQL settings that this guide tells you to change: number of workers (6.4.1 step 5); number of tables copied at the same time during the initial copy,
max_sync_workers_per_subscription(6.8); number of connections,max_connections(5.4, 10.2); time zone (5.3);listen_addresses(11.4); audit logging (11.9) - The tables in 5.2 and 5.6 of chapter 5 are the authoritative source. The authoritative source for extension settings is 6.6.
A.3 SQL functions
| Function | Returns | What it does | Section |
|---|---|---|---|
register_gritsurfer_table(schema_name text, table_name text) |
void |
Registers a table | 6.3 |
unregister_gritsurfer_table(schema_name text, table_name text) |
void |
Unregisters a table | 6.3 |
gritsurfer_version() |
text |
Extension SQL version, not the GritDB version number. For the version number, see chapter 0 | 6.5 |
get_gritsurfer_*() (table below) |
Table | Returns status as a table for processing with SQL | 6.5 |
show_gritsurfer_tables(), show_gritsurfer_coldfiles(), show_gritsurfer_unify_checkpoints(), show_gritsurfer_active_queries(), show_gritsurfer_query_history(max_rows integer DEFAULT NULL, since timestamptz DEFAULT NULL) |
SETOF text |
Returns the same content as human-readable text | 6.5 |
- With the installation defaults, every account can run these functions. Narrow the privileges in 11.5. Only the service account uses the register and unregister functions, so do not grant them to roles. After you complete 11.5, regular accounts cannot run them even if you grant them.
- Status functions return only the objects of the connected database. Only
get_gritsurfer_unify_checkpoints()is common to all databases. - The arguments of the history function can differ by installation package version. So check them with
\df get_gritsurfer_query_historybefore you use it (3.5, 6.5). - The tables in 6.3 and 6.5 of chapter 6 are the authoritative source.
| Function | Output columns |
|---|---|
get_gritsurfer_tables() |
16 columns: table_id, extra_id, table_type, schema_name, table_name, column_count, constraint_count, index_count, register_ts, partition_strategy, is_partition, parent_table_id, partition_key_attnums, partition_bound, space_id, file_id |
get_gritsurfer_columns(), get_gritsurfer_constraints(), get_gritsurfer_indexes() |
28, 15, and 12 columns, respectively. Column, constraint, and index information from the PostgreSQL catalog (linked by table_id) |
get_gritsurfer_coldfiles() |
17 columns: table_oid, cold_group_id, type, schema_name, table_name, stem, num_cold_chunks, num_cold_segments, row_count, file_size_bytes, committed_lsn, min_lsn, max_commit_ts, schema_version, min_cold_group_id, max_cold_group_id, status |
get_gritsurfer_unify_checkpoints() |
3 columns: ckpt_lsn, ckpt_ts, created_at |
get_gritsurfer_active_queries(), get_gritsurfer_query_history(max_rows, since) |
17 columns: seq, pid, query_start_time, query_hash, q_seq, session_start_time, query_text (first 1024 characters), status, 9 stage times (dispatched_time, burst_received_time, coord_started_time, exec_started_time, worker_taken_time, first_chunk_time, pg_first_recv_time, burst_end_time, pg_end_time) |
Frequently used columns:
| Function | Column | Meaning |
|---|---|---|
get_gritsurfer_tables() |
table_id |
Table OID (same as pg_class.oid) |
get_gritsurfer_tables() |
register_ts |
Registration time. A TOAST table registered along with the table also appears as a row |
get_gritsurfer_coldfiles() |
(row) | One row is one file. One cold file set (files that the GritDB engine stores by column for analytic queries) is 3 rows with different type values. To see it per set, use show_gritsurfer_coldfiles() |
get_gritsurfer_coldfiles() |
status |
STAGED, ACTIVE, FROZEN, INACTIVE (6.5) |
get_gritsurfer_coldfiles() |
row_count, file_size_bytes |
Rows and bytes in the file |
get_gritsurfer_unify_checkpoints() |
ckpt_lsn |
WAL position of the unify checkpoint. The unify checkpoint is the WAL position up to which the GritDB engine has applied changes durably. It is not the PostgreSQL checkpoint. Use it for the apply check (6.6.1) |
get_gritsurfer_unify_checkpoints() |
created_at |
Time recorded. The age is now() - max(created_at) (A.6) |
| Both query functions | pid |
PID of the session that sent the query. Match it with pg_stat_activity.pid (6.9) |
| Both query functions | query_text |
First 1024 characters of the original query text. The full text is in datafiles/query_text_archive.jsonl |
| Both query functions | 9 stage times | The difference from query_start_time shows the wait time and the execution time (6.9) |
| Both query functions | seq |
Row number in the history for history rows, and row number in the result for running queries |
Query status (status):
| Value | Meaning |
|---|---|
CREATED |
Received and waiting to run |
EXECUTING |
The GritDB engine is running it |
EXECUTION_DONE |
The GritDB engine finished running it and is passing the results |
COMPLETED |
The results reached PostgreSQL |
FAILED |
Ended with an error. A query that was run again in PostgreSQL also has this value (6.6) |
CANCELLED |
Canceled |
UNKNOWN |
Unknown value |
- Functions in the
gritdbschema that are not in this section are for GritDB internal use. Do not call them. Revoke their execute privileges in 11.5.
A.4 Log files
- The table in 9.1 of chapter 9 is the authoritative source.
A.5 Directories
Installation directory (~/gritdb1):
| Directory and file | Content | Kind |
|---|---|---|
bin/ |
Commands (A.1) | Installed files |
lib/ |
Shared libraries | Installed files |
share/ |
PostgreSQL shared files, extensions, share/gritdb/manifest.json (installed version and file list) |
Installed files |
etc/ |
Path scripts and configuration templates (templates/) |
Installed files |
include/, licenses/, docs/ |
Headers, open source licenses, RELEASE-NOTES.md |
Installed files |
config/ |
4 GritDB engine configuration files (.config of grit_burst, grit_plug, grit_surfer, and grit_unify). Created only at the first installation (5.6) |
Configuration |
datafiles/ |
Data (table below). A symlink if relocated (2.4) | State |
log/ |
Logs (9.1) | State |
run/ |
Runtime files (table below) | State |
.gritdb_installed |
Install anchor. Without it, GritDB does not start (QuickStart GDB-BOOT-001) | State |
- Installation and upgrade change the installed files. Upgrade leaves configuration and state as they are (3.5).
- A development package also has
run/gritdb/,tools/,.bashrc, andbuild/(chapter 0).
Inside datafiles/:
| Location | Content |
|---|---|
pgdata/ |
PostgreSQL data, WAL (pg_wal), recovery replication slot, postgresql.conf, pg_hba.conf |
checkpoint/ |
unify checkpoint position (grit_unify.cp). Do not delete or edit it (QuickStart 6.1, prohibition 2) |
queue/ |
GritDB engine queue files. Their apparent size is large, with much empty space. For copying, see 8.2 |
cold_file/ |
cold files. There is one directory per registered table that has applied rows. The name ends with _<database OID>_<table OID>. GritDB deletes it after unregistration or DROP (7.6). Directories whose names end with _0_0_<number> belong to GritDB internal tables and are not in the registration list |
buffer_spill/ |
Temporary files that overflowed GritDB engine memory. Deleted at a normal stop (7.6) |
query_text_archive.jsonl |
Full original text of analytics session queries (one line per query). Not deleted automatically (9.5, 11.1) |
- Relocation symlinks (2.4):
datafiles→--datafilesreal path,datafiles/pgdata→--pgdatareal path - To check, run
readlink -f ~/gritdb1/datafiles. It returns the real path set at installation.
Inside run/:
| File | Content |
|---|---|
gritdb.lock |
Runtime lock. Do not delete it (4.5) |
grit_admin.pid |
Created after the start completes. It has 4 lines: PID:, ENGINE_PID:, BASE_DB_PID:, CATALOG_PID: (4.5) |
grit_admin.sock |
Stop command socket. A user who can write to it can stop GritDB (11.7) |
grit_admin_clean_shutdown |
Normal stop marker (CLEAN_SHUTDOWN, 8.2) |
A.6 Diagnostic queries
Copy them into ~/gritdb1/bin/psql -p 5432 -d <db> opened as the service account. Queries not marked “in each database” give the same result in any database.
WAL retained by the recovery replication slot:
SELECT slot_name, active, wal_status,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots WHERE slot_name = 'gritsurfer_slot';
- View and function:
pg_replication_slots,pg_current_wal_lsn() - It is normal if
activeisf,wal_statusisreserved, andretained_waldoes not keep growing.
Count and size of cold files by status (in each database):
SELECT schema_name, table_name, status, count(*) AS files,
pg_size_pretty(sum(file_size_bytes)) AS size
FROM get_gritsurfer_coldfiles()
GROUP BY schema_name, table_name, status
ORDER BY schema_name, table_name, status;
- It is normal if
STAGEDdoes not keep growing andINACTIVEdisappears soon (7.8).
unify checkpoint age:
SELECT max(ckpt_lsn) AS ckpt_lsn, now() - max(created_at) AS age
FROM get_gritsurfer_unify_checkpoints();
- It is normal if
agestays short (7.2). You can also see the same thing in the file withstat -c '%y' ~/gritdb1/datafiles/checkpoint/grit_unify.cp. The value is the last time the unify checkpoint advanced.
Long transactions:
SELECT pid, usename, application_name, state,
now() - xact_start AS xact_age, left(query, 60) AS query
FROM pg_stat_activity
WHERE xact_start IS NOT NULL AND pid <> pg_backend_pid()
ORDER BY xact_start LIMIT 10;
- It is normal if there are no long-open transactions and no
idle in transaction. If there are any, end them in that application.
Remaining prepared transactions (PREPARE TRANSACTION):
SELECT gid, prepared, owner, database FROM pg_prepared_xacts;
- It is normal if
pg_prepared_xactshas 0 rows.
Registered table list (in each database):
SELECT schema_name, table_name, register_ts FROM get_gritsurfer_tables() ORDER BY 1, 2;
- TOAST tables registered along with the tables also appear.
Recent analytic queries (in each database; check the arguments for each version, 6.5):
SELECT query_start_time, status, pg_end_time - query_start_time AS total,
left(query_text, 60) AS query
FROM get_gritsurfer_query_history(100, now() - interval '1 hour')
ORDER BY query_start_time DESC;
Evidence of a restart:
SELECT pg_postmaster_start_time();
- It is normal if the value is the same as at the previous check. If it changed, a restart happened (7.9). The function is described in
pg_postmaster_start_time().
Other queries:
| What to check | Where |
|---|---|
Apply check (pg_current_wal_lsn() vs max(ckpt_lsn)) |
6.6.1 |
| GritDB workers and internal connections | 7.1 |
| Analytics setup subscription (slot on the operational database side and status on the GritDB side) | 7.5 |
| Transaction ID age and dead rows | 7.8 |
| Security checks | 11.10 |
A.7 Oracle and Tibero mapping
| What you know | GritDB |
|---|---|
startup, tbboot |
gritdb-start |
shutdown normal, tbdown |
gritdb-stop. It switches to a quick stop if sessions remain (4.2) |
shutdown immediate, tbdown immediate |
Terminate the remaining sessions (4.2, pg_terminate_backend) → gritdb-stop |
shutdown abort, tbdown abort |
gritdb-stop --quick |
| alert log | log/grit_admin_main.log and log/postgresql.log |
init parameters, ALTER SYSTEM |
ALTER SYSTEM, managed block, config/grit_*.config |
V$ dynamic views |
pg_stat_*, get_gritsurfer_*() |
Choosing In-Memory objects (ALTER TABLE … INMEMORY) |
register_gritsurfer_table |
- This table continues the table in QuickStart 1.3.
A.8 Glossary
| Term | Meaning |
|---|---|
| Installation directory | The directory where GritDB is installed (~/gritdb1) |
| Supervisor process | grit_admin — the process that starts and watches postgres, grit_surfer_process, and grit_catalog_process |
| Transaction engine · GritDB engine | The transaction engine is PostgreSQL. The GritDB engine is the engine that handles analytics (PLUG, UNIFY, BURST). It is the same thing as the analytics engine in the architecture part of the website |
| Registered table | A table put under GritDB management with register_gritsurfer_table |
| Regular session · analytics session | A regular session is a connection in which PostgreSQL runs all queries. An analytics session is a connection that sends queries on registered tables to the GritDB engine |
| Analytics setup · operational database · built-in replication | The analytics setup is a setup that receives the changes of the operational database and handles analytics. The operational database is the database that the customer uses for business. Built-in replication is PostgreSQL logical replication |
| cold file | A file in which the GritDB engine stores registered table data by column for analytic queries. Where to see it: datafiles/cold_file/, get_gritsurfer_coldfiles() (6.5) |
| compaction | GritDB engine work that merges small cold files into large cold files. No setting changes it. Where to see it: log grit_unify_cold_store_compactor_gc_processor.log (9.1), cold file status ACTIVE (6.5) |
| unify checkpoint | The WAL position up to which the GritDB engine has applied changes durably. It is not the PostgreSQL checkpoint. Where to see it: get_gritsurfer_unify_checkpoints() (6.5), datafiles/checkpoint/grit_unify.cp |
| Recovery replication slot | gritsurfer_slot — the point from which the GritDB engine continues to receive WAL |
| Managed block | The GritDB-only settings section at the end of postgresql.conf |
| Instance ID | An identifier set for each installation directory. It is the suffix of shared memory names |
| Reinitialization | Creating the data anew. The existing data is kept under a new name |
| Replica identity | The columns that the operational database writes to WAL to identify the rows to update or delete (primary key, unique index, or whole row) |
| Initial copy | Copying the current rows of a table once at the start, when you create a subscription or add the table |
| Apply check | Checking by WAL position whether a change you just committed is visible in analytics sessions (6.6.1) |
| Apply lag | The time until a change you just committed shows in analytics session queries |
| Worker slots | The number of background workers that PostgreSQL can start (max_worker_processes) |
| publication · subscription | A publication is the set of tables that the operational database publishes. A subscription is the setting with which GritDB receives that set (PostgreSQL logical replication, 6.4) |
| Taking a table again | In an analytics setup, receiving one table or all tables again from the operational database, starting from the initial copy (6.4.2, 8.6) |
| Security barrier view | A view with security_barrier turned on. Analytics sessions in 1.1 do not apply it (11.6) |
| Importance | The size of the impact that can occur if you skip a check (7.3). It is not the same as the severity level in the subscription contract |
| Installation script | install_gritdb.sh — the script that runs installation, upgrade, reinitialization, and cleanup (chapters 2 and 3) |
| Contact support | The contact path in chapter 0 (Datagrit, distributor, or partner) |