Sitemap

Real-Time Data Replication between Azure PostgreSql databases with Debezium

4 min readJan 8, 2026

--

In this post, I’ll walk through how I set up real-time data replication from an Azure PostgreSQL database into Apache Kafka using Debezium, and then into a second PostgreSQL database. While Debezium simplifies Change Data Capture (CDC), working with Azure’s managed PostgreSQL service brings unique challenges. I’ll break down each step from configuration to deployment.

🧱 Architecture Overview

Press enter or click to view image in full size
Architecture overview

Debezium’s PostgreSQL Kafka Source Connector enables Change Data Capture (CDC) by streaming changes from the PostgreSQL Write-Ahead Log ( WAL) into Kafka topics . Each change then replicated to kafka topics.
From Kafka, we use Debezium’s JDBC Sink Connector to replicate the changes into the target PostgreSQL database.

✅ Prerequisites

  • Azure PostgreSQL instance (Flexible or Single server)
  • Kafka cluster (local Docker or managed)
  • Basic understanding of Kafka Connector
  • Debezium connector setup
  • PostgreSQL settings (logical replication, slot etc.)

⚙️ Configuring Azure PostgreSQL for CDC

For source db these configurations has to be set to:

  • wal_level=logical
  • session_replication_role = replica

🔍 Note: On Azure, some of these settings are managed via the portal or CLI under “Server Parameters”.

🚀 Setting up Kafka + Debezium

I assume you are familiar with kafka, if not you can take a look at https://kafka.apache.org/quickstart. Debezium Kafka Connector is a wrapper of Kafka Connector plugin so we have to download Debezium libs to make available for kafka connector before start. So lets download Debezium Sink(JDBC) and Source Connectors and put them in to where kafka connect directory is, lets call this directory debezium and now we have to point to it in kafka connect settings. You can use preexisting or make a new config file, for example take a look at the connect-distributet.properties file under config directory in your kafka installation, it would look something like below. Add plugin path to the file.

key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
config.storage.topic=connector-config.json
offset.storage.topic=connector-offset.json
status.storage.topic=connector-status.json
database.history.kafka.topic=connector-dbhistory.json
database.history.kafka.recovery.poll.interval.ms=1000
offset.flush.interval.ms=10000
plugin.path=/kafka/debezium
bootstrap.servers=localhost:9093
group.id=someGroupId

🔌 Start Debezium and configure database publishing

Start connector

/your kafka installation/bin/connect-distributed.sh /your kafka installation/config/connect-distributed.properties

Deploy source connector with a PUT request:

curl -s -X PUT -H  "Content-Type:application/json" http://localhost:8083/connectors/formuesgoder-bifrost-source/config \
-d "{
\"name\": \"name of the connector\",
\"connector.class\": \"io.debezium.connector.postgresql.PostgresConnector\",
\"tasks.max\": \"1\",
\"topic.prefix\": \"your topic prefix\",
\"database.hostname\": \"psqlf-pgflex-some-db.postgres.database.azure.com\",
\"database.port\": \"5432\",
\"database.user\": \"database user name\",
\"database.password\": \"db password\",
\"database.dbname\": \"db name\",
\"plugin.name\": \"pgoutput\",
\"snapshot.mode\": \"when_needed\",
\"publication.autocreate.mode\": \"filtered\",
\"publication.name\": \"dbz_publication\",
\"slot.name\": \"debezium\",
\"schema.include.list\": \"schema name in db\",
\"table.include.list\": \"list of the table if you dont want everything\"
}"

You should now start seeing changes appear in your Kafka topics. Now we will configure sink connector to forward data in topic to target database

curl -s -X PUT -H  "Content-Type:application/json" http://localhost:8083/connectors/formuesgoder-bifrost-sink/config \
-d "{
\"name\": \"name of the connector\",
\"group.id\": \"kafka group id\",
\"connector.class\": \"io.debezium.connector.jdbc.JdbcSinkConnector\",
\"connection.url\": \"jdbc:postgresql://psqlf-pgflex-your.db.postgres.database.azure.com:5432/your-db\",
\"connection.username\": \"debezium\",
\"auto.create\": \"false\",
\"auto.evolve\": \"false\",
\"connection.password\": \"target db password\",
\"dialect.name\": \"PostgreSqlDatabaseDialect\",
\"insert.mode\": \"upsert\",
\"tasks.max\": 1,
\"topics.regex\": \"some regex for your topic\",
\"collection.name.format\": \"\${source.table}\",
\"delete.handling.mode\": \"rewrite\",
\"primary.key.mode\": \"record_value\",
\"primary.key.fields\": \"id\"
}"

that is it. By now your data is supposed to being captured from source and replicated to target database.

📊 Use Case or Next Steps

  • Integration with other services (Elasticsearch, Flink, etc)
  • Data warehousing or real-time analytics
  • Optional: transforming messages with Kafka Streams or Kafka Connect SMTs for enrichment of the data.

✅ Conclusion

Debezium makes it possible to implement Change Data Capture (CDC) with minimal effort, enabling zero-downtime, real-time data replication across platforms. With a few adjustments, it integrates well even with managed services like Azure PostgreSQL.

This article is written by Kubilay Karayilan

Kubilay is a full-stack developer and senior consultant in the SystemDev & Design department. With a BSc in Data Engineering from OsloMet and 13 years of experience in system and cloud development, Kubilay — often called “Kubi” — has contributed to a wide range of projects involving on-premises to cloud migrations, cloud-native solutions, and modernization initiatives. He has extensive experience with modern tools and agile ways of working, including OKRs and Scrum, as well as technologies such as microservices, Kafka, Kubernetes, AWS, and Azure.

🛠 Resources

--

--

Crayon Consulting
Crayon Consulting

Written by Crayon Consulting

Helping businesses utilize the power of Data, AI, and Tech. Advisors, techies, designers, and developers with a passion for innovation. www.crayonconsulting.no