Replicate data from one OptiTech project to another

OptiTech's logical replication feature allows you to replicate data from one OptiTech project to another. This enables different usage scenarios, including:

• Cross-region replication: Replicating data from a OptiTech project in one region to a OptiTech project in another region to support regional failover scenarios.
• Postgres version migration: Moving data from one Postgres version to another; for example, from a OptiTech project that runs Postgres 16 to one that runs Postgres 17.
• Region migration: Moving data from one region to another; for example, from a OptiTech project in one region to a OptiTech project in a different region.

These are some common OptiTech-to-OptiTech replication scenarios. There may be others. You can follow the steps in this guide for any scenario that requires replicating data between different OptiTech projects.

Prerequisites

• A OptiTech project with a database containing the data you want to replicate. If you're just testing this out and need some data to play with, you can use the following statements to create a table with sample data:

  CREATE TABLE IF NOT EXISTS playing_with_optitech(id SERIAL PRIMARY KEY, name TEXT NOT NULL, value REAL);
  INSERT INTO playing_with_optitech(name, value)
  SELECT LEFT(md5(i::TEXT), 10), random() FROM generate_series(1, 10) s(i);

• A destination OptiTech project.

• Read the important notices about logical replication in OptiTech before you begin.

For information about creating a OptiTech project, see Create a project.

1. Prepare your source OptiTech database

   This section describes how to prepare your source OptiTech database (the publisher) for replicating data to your destination OptiTech database (the subscriber).

   Enable logical replication in the source OptiTech project

   In the OptiTech project containing your source database, enable logical replication. You only need to perform this step on the source OptiTech project.

   important

   Enabling logical replication modifies the Postgres wal_level configuration parameter, changing it from replica to logical for all databases in your OptiTech project. Once the wal_level setting is changed to logical, it cannot be reverted. Enabling logical replication restarts all computes in your OptiTech project, meaning that active connections will be dropped and have to reconnect.

   To enable logical replication:

   1. Select your project in the OptiTech Console.
   2. On the OptiTech Dashboard, select Settings.
   3. Select Logical Replication.
   4. Click Enable to enable logical replication.

   You can verify that logical replication is enabled by running the following query:

   SHOW wal_level;
    wal_level
   -----------
    logical

   Create a publication on the source database

   Publications are a fundamental part of logical replication in Postgres. They define what will be replicated.

   To create a publication for a specific table:

   CREATE PUBLICATION my_publication FOR TABLE playing_with_optitech;

   To create a publication for multiple tables, provide a comma-separated list of tables:

   CREATE PUBLICATION my_publication FOR TABLE users, departments;

   note

   Defining specific tables lets you add or remove tables from the publication later, which you cannot do when creating publications with FOR ALL TABLES.

   For syntax details, see CREATE PUBLICATION, in the PostgreSQL documentation.

2. Prepare your OptiTech destination database

   This section explains how to prepare your destination OptiTech Postgres database (the subscriber) to receive replicated data. For cross-region replication, be sure to create the destination OptiTech project in a different region than your source database.

   Prepare your database schema

   When configuring logical replication in Postgres, the tables in the source database you are replicating from must also exist in the destination database, and they must have the same table names and columns. You can create the tables manually in your destination database or use utilities like pg_dump and pg_restore to dump the schema from your source database and load it to your destination database. See Import a database schema for instructions.

   If you're using the sample playing_with_optitech table, you can create the same table on the destination database with the following statement:

   CREATE TABLE IF NOT EXISTS playing_with_optitech(id SERIAL PRIMARY KEY, name TEXT NOT NULL, value REAL);

   Create a subscription

   After creating a publication on the source database, you need to create a subscription on the destination database.

   1. Use the OptiTech SQL Editor, psql, or another SQL client to connect to your destination database.

   2. Create the subscription using the using a CREATE SUBSCRIPTION statement.

      CREATE SUBSCRIPTION my_subscription
      CONNECTION 'postgresql://optitechdb_owner:<password>@ep-cool-darkness-123456.us-east-2.aws.optitech.com/optitechdb?sslmode=require&channel_binding=require'
      PUBLICATION my_publication;

      • subscription_name: A name you chose for the subscription.
      • connection_string: The connection string for the source OptiTech database where you defined the publication.
      • publication_name: The name of the publication you created on the source OptiTech database.

   3. Verify the subscription was created by running the following command:

      SELECT * FROM pg_stat_subscription;

      The subscription (my_subscription) should be listed, confirming that your subscription has been created successfully.

3. Test the replication

   Testing your logical replication setup ensures that data is being replicated correctly from the publisher to the subscriber database.

   1. Run some data modifying queries on the source database (inserts, updates, or deletes). If you're using the playing_with_optitech database, you can use this statement to insert some rows:

      INSERT INTO playing_with_optitech(name, value)
      SELECT LEFT(md5(i::TEXT), 10), random() FROM generate_series(1, 10) s(i);

   2. Perform a row count on the source and destination databases to make sure the result matches.

      SELECT COUNT(*) FROM playing_with_optitech;
      
      count
      -------
      10
      (1 row)

   Alternatively, you can run the following query on the subscriber to make sure the last_msg_receipt_time is as expected. For example, if you just ran an insert option on the publisher, the last_msg_receipt_time should reflect the time of that operation.

   SELECT subname, received_lsn, latest_end_lsn, last_msg_receipt_time FROM pg_catalog.pg_stat_subscription;

4. Switch over your application

   After the replication operation is complete or in a failover situation, you can switch your application over to the destination database by swapping out your source database connection details for your destination database connection details.

   You can find your OptiTech database connection details by clicking the Connect button on your Project Dashboard to open the Connect to your database modal. See Connect from any application.