Hi,
My TSDB was shut down over the weekend, Dec 5th. When I tried to resume it this morning (Dec 8th), it stucks in the resume state. Could you help restart it?
service ID: f40awjlq3x
project ID: nyy1m9q7ci
Log:
2025-12-05 16:09:03 UTC-6 [386625]: LOG: database system is shut down
2025-12-05 16:09:02 UTC-6 [386627]: LOG: shutting down
2025-12-05 16:09:02 UTC-6 [386639]: FATAL: terminating walreceiver process due to administrator command
2025-12-05 16:09:02 UTC-6 [386656]: FATAL: terminating connection due to administrator command
2025-12-05 16:09:02 UTC-6 [386657]: FATAL: terminating connection due to administrator command
2025-12-05 16:09:02 UTC-6 [386625]: LOG: aborting any active transactions
2025-12-05 16:09:02 UTC-6 [386625]: LOG: received fast shutdown request
2025-12-05 16:09:02 UTC-6 [386629]: LOG: new target timeline is 7
2025-12-05 16:09:02 UTC-6 [386674]: LOG: statement: SET log_statement TO 'none';
2025-12-05 16:09:02 UTC-6 [386629]: LOG: restored log file "00000007.history" from archive
2025-12-05 16:09:02 UTC-6 [386674]: LOG: statement: CHECKPOINT;
2025-12-05 16:09:01 UTC-6 [386664]: LOG: could not receive data from client: Connection reset by peer
2025-12-05 16:09:01 UTC-6 [386629]: LOG: restored log file "00000007.history" from archive
2025-12-05 16:09:01 UTC-6 [386639]: LOG: replication terminated by primary server
2025-12-05 16:09:01 UTC-6 [386639]: DETAIL: End of WAL reached on timeline 6 at 4/D00000A0. 2025-12-05 22:09:01 - on_role_change.sh - /etc/postgresql/scripts/timescaledb/on_role_change.sh - I was called with the following parameters on_role_change replica f40awjlq3x-an 2025-12-05 22:09:00,989 INFO: Lock owner: f40awjlq3x-an-1; I am f40awjlq3x-an-0 2025-12-05 22:09:01,287 INFO: switchover: demote in progress 2025-12-05 22:09:01,388 INFO: establishing a new patroni heartbeat connection to postgres 2025-12-05 22:09:01,391 INFO: establishing a new patroni restapi connection to postgres
2025-12-05 16:09:01 UTC-6 [386639]: LOG: started streaming WAL from primary at 4/D0000000 on timeline 6
2025-12-05 16:09:01 UTC-6 [386639]: LOG: fetching timeline history file for timeline 7 from primary server
2025-12-05 16:09:00 UTC-6 [386629]: LOG: consistent recovery state reached at 4/D00000A0
2025-12-05 16:09:00 UTC-6 [386629]: LOG: invalid record length at 4/D00000A0: expected at least 24, got 0
2025-12-05 16:09:00 UTC-6 [386625]: LOG: database system is ready to accept read-only connections /var/run/postgresql:5432 - accepting connections /var/run/postgresql:5432 - accepting connections
2025-12-05 16:09:00 UTC-6 [386629]: LOG: entering standby mode
2025-12-05 16:09:00 UTC-6 [386629]: LOG: restored log file "00000006.history" from archive
2025-12-05 16:09:00 UTC-6 [386629]: LOG: database system was shut down at 2025-12-05 22:08:54 UTC
2025-12-05 16:09:00 UTC-6 [386625]: LOG: listening on Unix socket "/home/postgres/pgdata/.s.PGSQL.5432"
2025-12-05 16:09:00 UTC-6 [386625]: LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-12-05 16:08:59 UTC-6 [386625]: LOG: listening on IPv4 address "0.0.0.0", port 5432
2025-12-05 16:08:59 UTC-6 [386625]: LOG: starting PostgreSQL 17.7 (Ubuntu 17.7-3.pgdg22.04+1) on aarch64-unknown-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0, 64-bit
2025-12-05 16:08:59 UTC-6 [386625]: LOG: pgaudit extension initialized
2025-12-05 16:08:56 UTC-6 [1216]: LOG: database system is shut down 2025-12-05 22:08:56,751 INFO: Leader key released 2025-12-05 22:08:56,751 INFO: Lock owner: f40awjlq3x-an-1; I am f40awjlq3x-an-0 2025-12-05 22:08:56,751 INFO: switchover: demote in progress 2025-12-05 22:08:56,752 INFO: Lock owner: f40awjlq3x-an-1; I am f40awjlq3x-an-0 2025-12-05 22:08:56,752 INFO: switchover: demote in progress 2025-12-05 22:08:56,759 INFO: Lock owner: f40awjlq3x-an-1; I am f40awjlq3x-an-0 2025-12-05 22:08:56,759 INFO: switchover: demote in progress 2025-12-05 22:08:56,863 INFO: Lock owner: f40awjlq3x-an-1; I am f40awjlq3x-an-0 2025-12-05 22:08:56,864 INFO: switchover: demote in progress 2025-12-05 22:08:56,987 INFO: Lock owner: f40awjlq3x-an-1; I am f40awjlq3x-an-0 2025-12-05 22:08:56,987 INFO: switchover: demote in progress 2025-12-05 22:08:58,755 INFO: Local timeline=6 lsn=4/D0000028 2025-12-05 22:08:58,755 INFO: closed patroni connections to postgres 2025-12-05 22:08:59,899 INFO: postmaster pid=386625 /var/run/postgresql:5432 - no response
2025-12-05 16:08:56 UTC-6 [386612]: FATAL: the database system is shutting down
2025-12-05 16:08:56 UTC-6 [386611]: FATAL: the database system is shutting down
2025-12-05 16:08:56 UTC-6 [386610]: FATAL: the database system is shutting down
2025-12-05 16:08:56 UTC-6 [386609]: FATAL: the database system is shutting down
2025-12-05 16:08:56 UTC-6 [386608]: FATAL: the database system is shutting down 2025-12-05 22:08:56,196 INFO: Got response from f40awjlq3x-an-1 http://100.115.37.62:8008/patroni: {"state": "running", "postmaster_start_time": "2025-12-05 22:08:42.881985+00:00", "role": "replica", "server_version": 170007, "xlog": {"received_location": 20669530112, "replayed_location": 20669530112, "replayed_timestamp": "2025-12-05 22:08:53.115588+00:00", "paused": false}, "timeline": 6, "replication_state": "streaming", "dcs_last_seen": 1764972529, "database_system_identifier": "7574841356511735935", "patroni": {"version": "4.0.3", "scope": "f40awjlq3x-an", "name": "f40awjlq3x-an-1"}} 2025-12-05 22:08:56,197 INFO: Member f40awjlq3x-an-1 exceeds maximum replication lag
2025-12-05 16:08:56 UTC-6 [386607]: FATAL: the database system is shutting down
2025-12-05 16:08:55 UTC-6 [386606]: FATAL: the database system is shutting down
2025-12-05 16:08:55 UTC-6 [386595]: FATAL: the database system is shutting down
2025-12-05 16:08:55 UTC-6 [386592]: FATAL: the database system is shutting down
2025-12-05 16:08:55 UTC-6 [386594]: FATAL: the database system is shutting down
2025-12-05 16:08:55 UTC-6 [386593]: FATAL: the database system is shutting down
2025-12-05 16:08:55 UTC-6 [386591]: FATAL: the database system is shutting down
2025-12-05 16:08:55 UTC-6 [386602]: FATAL: the database system is shutting down
2025-12-05 16:08:54 UTC-6 [386590]: FATAL: the database system is shutting down
2025-12-05 16:08:54 UTC-6 [386587]: FATAL: the database system is shutting down
2025-12-05 16:08:54 UTC-6 [386586]: FATAL: the database system is shutting down
2025-12-05 16:08:54 UTC-6 [386585]: FATAL: the database system is shutting down
2025-12-05 16:08:54 UTC-6 [386588]: FATAL: the database system is shutting down
2025-12-05 16:08:54 UTC-6 [386584]: FATAL: the database system is shutting down
2025-12-05 16:08:54 UTC-6 [1224]: LOG: checkpoint complete: wrote 53 buffers (0.2%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.147 s, sync=0.055 s, total=0.511 s; sync files=17, longest=0.025 s, average=0.004 s; distance=9692 kB, estimate=24125 kB; lsn=4/D0000028, redo lsn=4/D0000028
2025-12-05 16:08:54 UTC-6 [386578]: FATAL: the database system is shutting down
2025-12-05 16:08:53 UTC-6 [1224]: LOG: checkpoint starting: shutdown immediate
2025-12-05 16:08:53 UTC-6 [1224]: LOG: shutting down 2025-12-05 22:08:53 - wal_archive_command_pgbackrest - archiving pg_wal/0000000600000004000000CF
2025-12-05 16:08:53 UTC-6 [1216]: LOG: background worker "TimescaleDB Background Worker Scheduler" (PID 1491) exited with exit code 1
2025-12-05 16:08:53 UTC-6 [1216]: LOG: background worker "logical replication launcher" (PID 1290) exited with exit code 1
2025-12-05 16:08:53 UTC-6 [1216]: LOG: background worker "TimescaleDB Background Worker Launcher" (PID 1288) exited with exit code 1
2025-12-05 16:08:53 UTC-6 [386553]: FATAL: terminating connection due to administrator command
2025-12-05 16:08:53 UTC-6 [1300]: FATAL: terminating connection due to administrator command
2025-12-05 16:08:53 UTC-6 [1250]: FATAL: terminating connection due to administrator command
2025-12-05 16:08:53 UTC-6 [1216]: LOG: aborting any active transactions
2025-12-05 16:08:53 UTC-6 [384871]: FATAL: terminating connection due to administrator command
2025-12-05 16:08:53 UTC-6 [384868]: FATAL: terminating connection due to administrator command
2025-12-05 16:08:53 UTC-6 [1491]: FATAL: terminating background worker "TimescaleDB Background Worker Scheduler" due to administrator command
2025-12-05 16:08:53 UTC-6 [1288]: FATAL: terminating background worker "TimescaleDB Background Worker Launcher" due to administrator command
2025-12-05 16:08:53 UTC-6 [384870]: FATAL: terminating connection due to administrator command
2025-12-05 16:08:53 UTC-6 [384869]: FATAL: terminating connection due to administrator command
2025-12-05 16:08:53 UTC-6 [382413]: FATAL: terminating connection due to administrator command
2025-12-05 16:08:53 UTC-6 [382409]: FATAL: terminating connection due to administrator command
2025-12-05 16:08:53 UTC-6 [1216]: LOG: received fast shutdown request