Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: postgresql-cfbot/postgresql
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: cf/6487~1
Choose a base ref
...
head repository: postgresql-cfbot/postgresql
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: cf/6487
Choose a head ref
  • 5 commits
  • 17 files changed
  • 2 contributors

Commits on Jun 23, 2026

  1. Pipelined Recovery - Producer Related Code 

    This includes the producer specific code for the producer-consumer
architecture for WAL replay that separates WAL decoding from the startup
process, enabling parallel processing between differemt steps of replay.

The producer includes a background worker that reads and decodes WAL records,
then send them to the startup process for the redo. IPC happens via shared
memory message queues (shm_mq), allowing the decoder to run ahead of
the apply process.

This will give some performance improvements in recoveries for CPU-bound
workloads.

New GUC: wal_pipeline (default: off)

Author: Imran Zaheer <imran.zhir@gmail.com>
Idea by: Ants Aasma <ants@cybertec.at>
    @imranzaheer612
    imranzaheer612 authored and Commitfest Bot committed Jun 23, 2026
    Configuration menu
    Copy the full SHA
    4218451 View commit details
    Browse the repository at this point in the history

  2. Pipelined Recovery - Consumer Related Code 

    This includes the consumer-specific code for the producer-consumer
architecture for WAL replay that separates WAL decoding from the
recovery process, enabling parallel processing between different steps
 of replay.

The startup process act as consumer and will initiate the pipeline before
starting the redo loop inside PerformWalRecovery() and will connect to
the shm mq to receive the decoded records directly. Subsequently,
it will apply those records and keep on fetching new decoded records in
the main redo loop.

Finally, it will receive a shutdown message from the producer
(based on the consistency checks, the producer will take care of whether
to stop decoding or not). The consumer will exit the loop and later will
ask to fully shutdown the pipeline workers in FinishWalRecovery().

Author: Imran Zaheer <imran.zhir@gmail.com>
Idea by: Ants Aasma <ants@cybertec.at>
    @imranzaheer612
    imranzaheer612 authored and Commitfest Bot committed Jun 23, 2026
    Configuration menu
    Copy the full SHA
    39d52b4 View commit details
    Browse the repository at this point in the history

  3. Pipelined Recovery - Decoupling startup and producer states. 

    Before this patch, most states/variables were statically defined
under xlogrecovery.c as they would only be needed by the startup process.
However, to implement pipelining, the producer (parallel decoder)
should also be aware of these static states. Some of these states stays
constant throughout decoding/recovery, while others may change during
recovery.

To address this, WalPipelineParams contains a set of all local states
that the startup process had before it started the pipeline. These
states are passed to the producer, which updates them accordingly
before entering the decoding loop. This is done using
WalPipeline_ExportRecoveryState() and WalPipeline_ImportRecoveryState().

WalPipeline_ExportRecoveryState --> saves all the local startup proc states
WalPipeline_ImportRecoveryState --> apply those states to the producer context

To manage the states that may keep changing during the recovery, we
added them to the shared memory in XLogRecoveryCtlData. Now whenever
the startup proc updates any of such local states the shared state
is also upated accordingly,  so that the pipeline stay aware of this
update during the decoding.
    @imranzaheer612
    imranzaheer612 authored and Commitfest Bot committed Jun 23, 2026
    Configuration menu
    Copy the full SHA
    1842c0e View commit details
    Browse the repository at this point in the history

  4. Pipelined Recovery - Add Tap test 

    Some basic tap test to ensure pipeline is working as expected.

Most of the testing is done by running all the tests under the recovery suite
with pipeline on by default i.e. PG_TEST_INITDB_EXTRA_OPTS="-c wal_pipeline=on"
    @imranzaheer612
    imranzaheer612 authored and Commitfest Bot committed Jun 23, 2026
    Configuration menu
    Copy the full SHA
    c8f3ec9 View commit details
    Browse the repository at this point in the history

  5. [CF 6487] v4 - Pipelined Recovery 

    This branch was automatically generated by a robot using patches from an
email thread registered at:

https://commitfest.postgresql.org/patch/6487

The branch will be overwritten each time a new patch version is posted to
the thread, and also periodically to check for bitrot caused by changes
on the master branch.

Patch(es): https://www.postgresql.org/message-id/CA+UBfamzeXcdEbmhdOdjWn5X_YVt9n8xUpH5ZwmA7S8VWvaoXw@mail.gmail.com
Author(s): Imran Zaheer
    Commitfest Bot committed Jun 23, 2026
    Configuration menu
    Copy the full SHA
    aba1a98 View commit details
    Browse the repository at this point in the history
Loading