Release Notes

MySQL and MariaDB binlog-based change data capture (CDC) for Node.js, originally created by Nevill Dutt.

@vlasky/zongji has been tested working with MySQL 5.7, 8.0 and 8.4, and MariaDB 11.8.

It leverages mysql2 for connections and authentication, while using zongji's binlog parsing and event pipeline.

Release Notes

See the CHANGELOG for the full release history.

Version 0.6.0 was a major modernisation that rewrote the codebase to use the mysql2 module, ES6 syntax and ESM exports, added TypeScript definitions, and added official support for MySQL 8.4.

Version 0.5.9 is the last release that supports Node.js versions below 18 and CommonJS.

Quick Start

import ZongJi from '@vlasky/zongji';

const zongji = new ZongJi({ /* ... MySQL Connection Settings ... */ });

// Each change to the replication log results in an event
zongji.on('binlog', function(evt) {
  evt.dump();
});

// Binlog must be started, optionally pass in filters
zongji.start({
  includeEvents: ['tablemap', 'writerows', 'updaterows', 'deleterows']
});

GTID example

zongji.on('binlog', function(evt) {
  const type = evt.getTypeName();
  if (type === 'Gtid' || type === 'AnonymousGtid') {
    console.log('GTID:', evt.gtid, 'SID:', evt.sid, 'GNO:', evt.gno);
  }
});

TypeScript

TypeScript definitions are included:

import ZongJi from '@vlasky/zongji';

For a complete implementation see example.js...

Module format

Since v0.6.0 this package is ESM-only. CommonJS projects can load it on Node.js >= 20.17 or >= 22.12, where require() of ES modules is supported, or on any supported version via dynamic import():

const { default: ZongJi } = await import('@vlasky/zongji');

Installation

Requires Node.js v18+. Actively supported and tested on Node.js 22, 24 and 26; version 0.7.0 was also verified on Node.js 18 and 20 at release, but those lines are end-of-life and no longer tested.
```
$ npm install @vlasky/zongji
```

Enable the binlog in my.cnf, restart the server after making the changes.

Binlog checksum is enabled by default in all supported MySQL and MariaDB versions. Zongji can work with it, but it doesn't verify the checksum. On MariaDB, binlog_format defaults to MIXED, so setting it to row explicitly matters there in particular.

# Must be unique integer from 1-2^32
server-id        = 1
# Row format required for ZongJi
binlog_format    = row
# Directory must exist. This path works for Linux. Other OS may require
#   different path.
log_bin          = /var/log/mysql/mysql-bin.log

binlog_do_db     = employees   # Optional, limit which databases to log
expire_logs_days = 10          # Optional, purge old logs
max_binlog_size  = 100M        # Optional, limit log size

# Recommended on MySQL 8.0+ / MariaDB 10.5+: makes binlog events carry
#   complete column metadata (names, signedness, charsets, enum/set
#   values), so ZongJi decodes rows without querying INFORMATION_SCHEMA
#   (see below)
binlog_row_metadata = FULL

Create an account with replication privileges, e.g. given privileges to account zongji (or any account that you use to read binary logs)
```
GRANT REPLICATION SLAVE, REPLICATION CLIENT, SELECT ON *.* TO 'zongji'@'localhost'
```

ZongJi Class

The ZongJi constructor accepts one argument of either:

An object containing MySQL connection details in the same format as used by package mysql2
Or, a mysql2 Connection or Pool object that will be used for querying column information.

If a Connection or Pool object is passed to the constructor, it will not be destroyed/ended by Zongji's stop() method.

Binlog row values follow the same mysql2 connection options as query results, so CDC events and queries on the same connection agree:

Option	Effect on row values
`dateStrings`	`DATE`, `DATETIME` and `TIMESTAMP` columns are returned as strings instead of `Date` objects.
`timezone`	Applied when converting `DATETIME` and `TIMESTAMP` values to `Date` objects.
`decimalNumbers`	`DECIMAL` columns are returned as exact strings by default (e.g. `'-123.4500'`); set `decimalNumbers: true` to receive Numbers (may lose precision beyond 15 significant digits).
`jsonStrings`	`JSON` columns are returned as parsed JavaScript values by default; set `jsonStrings: true` to receive JSON strings.

Table metadata source

To decode row events, ZongJi needs each table's column metadata. It obtains this in one of two ways:

From the binlog itself (MySQL 8.0+ or MariaDB 10.5+ with binlog_row_metadata=FULL): every TableMap event carries column names, signedness, character sets and enum/set value lists. ZongJi decodes rows entirely from the stream: no INFORMATION_SCHEMA queries, no pausing the binlog connection, and no risk of the metadata being stale after an ALTER TABLE (each event describes the schema in force when it was written). Enum/set values containing commas or quotes also decode correctly only in this mode.
From INFORMATION_SCHEMA (all other cases, including MySQL 5.7, the MySQL default binlog_row_metadata=MINIMAL and the MariaDB default NO_LOG): on the first event for each table, ZongJi briefly pauses the stream and fetches column metadata over its control connection. If the table is altered between the binlog write and the fetch, decoded rows may be misinterpreted until the next TableMap refresh.

Setting binlog_row_metadata=FULL on the server is therefore recommended. It is a dynamic global variable (SET GLOBAL binlog_row_metadata=FULL) with negligible binlog size overhead for typical schemas. Even under MINIMAL, MySQL 8.0+ binlogs carry exact integer signedness, which ZongJi uses in preference to parsing the column definition.

MariaDB caveats under FULL: the binlog cannot distinguish UUID, INET4, INET6 and VECTOR columns from BINARY/VARBINARY, so in this mode their values arrive as raw Buffers rather than the formatted values the INFORMATION_SCHEMA path produces; and tables containing classic temporal type codes (which may hide MariaDB 5.3-era "hires" columns the binlog cannot describe) automatically fall back to the INFORMATION_SCHEMA path.

Each instance includes the following methods:

Method Name	Arguments	Description
`start`	`options`	Start receiving replication events, see options listed below
`stop`	None	Disconnect from MySQL server, stop receiving events
`on`	`eventName`, `handler`	Add a listener to the `binlog` or `error` event. Each handler function accepts one argument.

Some events can be emitted in different phases:

Event Name	Description
`ready`	This event occurs right after ZongJi successfully establishes a connection, sets up replica (slave) status, and sets the binlog position.
`binlog`	Once a binlog is received and passes the filter, it will bubble up with this event.
`error`	Every error will be caught by this event.
`stopped`	Emitted when ZongJi connection is stopped (ZongJi#stop is called).

Always attach an error listener. Errors that occur before a listener attaches (for example, a connection failure in the same tick as construction) are buffered and re-delivered to the first error listener. If no listener is ever attached, the buffered errors are thrown, following Node's default behaviour for unhandled 'error' events.

Options available:

Option Name	Type	Description
`serverId`	`integer`	Unique number (1 - 2³²) to identify this replication slave instance. Must be specified if running more than one instance of ZongJi. Must be used in `start()` method for effect. Default: `1`
`startAtEnd`	`boolean`	Pass `true` to only emit binlog events that occur after ZongJi's instantiation. Must be used in `start()` method for effect. Default: `false`
`filename`	`string`	Begin reading events from this binlog file. If specified together with `position`, will take precedence over `startAtEnd`.
`position`	`integer`	Begin reading events from this position. Must be included with `filename`.
`gtidSet`	`string`	Resume from executed GTIDs (e.g. a persisted `zongji.gtidSet`): a MySQL GTID set (`'uuid:1-27'`; tagged transactions print as `'uuid:1-27:tag_a:1'` on MySQL 8.3+) or a MariaDB GTID position (`'0-1-1234'`), matching the server flavour. The server locates the correct binlog file itself and skips transactions already processed, so a checkpoint remains valid across failover to another server in the same replication topology. Requires `gtid_mode=ON` on MySQL; always available on MariaDB. Pass `''` to stream the entire available history. Takes precedence over `filename`/`position` and `startAtEnd`.
`requestAnnotateRows`	`boolean`	MariaDB only: ask the server to send `annotaterows` events (the SQL statement text preceding each row operation's events). Off by default; note the statement text may contain sensitive literals that the row images alone would not expose. Ignored on MySQL. Default: `false`
`includeEvents`	`[string]`	Array of event names to include Example: `['writerows', 'updaterows', 'deleterows']`
`excludeEvents`	`[string]`	Array of event names to exclude Example: `['rotate', 'tablemap']`
`includeSchema`	`object`	Object describing which databases and tables to include (Only for row events). Use database names as the key and pass an array of table names or `true` (for the entire database). Example: `{ 'my_database': ['allow_table', 'another_table'], 'another_db': true }`
`excludeSchema`	`object`	Object describing which databases and tables to exclude (Same format as `includeSchema`) Example: `{ 'other_db': ['disallowed_table'], 'ex_db': true }`

By default, all events and schema are emitted.
excludeSchema and excludeEvents take precedence over includeSchema and includeEvents, respectively.
Calling start() while a previous start() is still initialising is ignored and the first call completes. The exception is after an intervening stop(): the new start() then restarts cleanly, and exactly one binlog stream is opened.
Calling start() while ZongJi is already running does not reconnect; it only updates the event and schema filters from the given options.

Supported Binlog Events:

Event name	Description
`unknown`	Catch any other events
`query`	Insert/Update/Delete Query
`intvar`	Autoincrement and LAST_INSERT_ID
`rotate`	New Binlog file. The `filename` and `position` resume properties stay updated across rotations whether or not this event is included; include it only if you want to observe rotations yourself.
`format`	Format Description
`xid`	Transaction ID
`gtid`	MySQL GTID event with `gtid`, `sid`, `gno` properties. Tagged GTIDs (MySQL 8.3+, `GTID_NEXT='AUTOMATIC:tag'`) arrive as the same event with `tag` set and `gtid` in `uuid:tag:gno` form
`anonymousgtid`	MySQL anonymous GTID event (same shape as `gtid`)
`previousgtids`	MySQL Previous GTIDs event with `gtidSet` and `sids`
`mariadbgtid`	MariaDB GTID event with `gtid` (`'domain-server-sequence'`), `domainId`, `serverId`, `seqNo` and flags (`standalone`, `isDdl`, XA states). Replaces the `BEGIN` query event of a transaction.
`mariadbgtidlist`	MariaDB GTID list event (start of every binlog file): the last GTID per domain and server seen so far, MariaDB's analogue of `previousgtids`
`binlogcheckpoint`	MariaDB binlog checkpoint event naming the oldest binlog file still needed for crash recovery (informational)
`annotaterows`	MariaDB: the SQL `statement` that produced the row events that follow (the analogue of MySQL's rows-query event). Only sent when `start()` was given `requestAnnotateRows: true`; useful for audit trails and statement attribution in CDC records
`rowsquery`	MySQL: the SQL `statement` that produced the row events that follow (the analogue of MariaDB's `annotaterows`). Sent to every consumer while the server runs with `binlog_rows_query_log_events=ON` (default `OFF`); no `start()` option is involved. The same sensitivity note applies: statement text may contain literals
`startencryption`	MariaDB encrypted-binlog marker (informational: the server decrypts before sending, the stream itself is cleartext)
`xaprepare`	XA PREPARE event (MySQL 5.7+ and MariaDB) terminating an XA-prepared event group, with `xidFormatId`, `xidGtrid`, `xidBqual` and `onePhase`
`heartbeat`	Sent by the server instead of real events: while the connection idles, and in place of transactions skipped server-side during a GTID resume. Carries `binlogName`; `nextPosition` holds the advanced position.
`tablemap`	Before any row event (must be included for any other row events)
`writerows`	Rows inserted, row data array available as `rows` property on event object
`updaterows`	Rows changed, row data array available as `rows` property on event object
`deleterows`	Rows deleted, row data array available as `rows` property on event object
`transactionpayload`	Compressed transaction from MySQL 8.0.20+ servers with `binlog_transaction_compression=ON`. ZongJi cannot decode the row events inside it, so it emits an `error` (once per instance) naming the server setting responsible. (MariaDB's equivalent, `log_bin_compress=ON`, IS supported: its compressed events decode transparently as ordinary `query` and row events.)
`partialupdaterows`	Partial JSON update from MySQL 8.0+ servers with `binlog_row_value_options=PARTIAL_JSON`. ZongJi cannot decode the JSON diff format, so it emits an `error` (once per instance) naming the server setting responsible.

When the server runs with gtid_mode=ON (MySQL), or always on MariaDB, every emitted event carries a gtid property ('uuid:sequence' on MySQL, 'domain-server-sequence' on MariaDB) identifying the transaction it belongs to, tracked internally even if GTID events are excluded by includeEvents. It is undefined for anonymous transactions, for events seen before the stream's first GTID event, and for events that belong to no transaction (e.g. a rotate arriving between one transaction's commit and the next transaction's GTID event; the commit event itself still carries its transaction's gtid). One exception: a DDL statement has no commit marker of its own, so its gtid persists on such trailing events until the next transaction begins. Use it for checkpointing and deduplication. The same value is available as zongji.lastGtid while an event is being delivered; it is only coherent when read synchronously inside a 'binlog' handler, so checkpointing code should prefer event.gtid.

File and position resume

zongji.options.filename and zongji.options.position always hold a safe resume point for the connected server. The position advances only at transaction boundaries: it freezes when a transaction's first TableMap arrives and jumps to the commit marker's end position, so a persisted pair never points inside a transaction (where row events could be separated from the table metadata they need and silently dropped). Resuming from a pair persisted mid-transaction therefore replays that whole transaction: consumers should treat delivery as at-least-once and make their processing idempotent. Read the pair synchronously inside a 'binlog' handler when checkpointing per event.

GTID-based resume

zongji.gtidSet exposes the executed GTIDs as a string: a MySQL GTID set (e.g. '3e11fa47-…:1-27') or a MariaDB GTID position (e.g. '0-1-1234'), depending on the connected server. It is the value ZongJi started from, extended as each observed transaction commits, so a persisted value never claims a transaction whose row events were still in flight. Persist it alongside (or instead of) filename/position and resume with:

zongji.start({ gtidSet: savedGtidSet, serverId: 5 });

Unlike file+position checkpoints, a GTID checkpoint is valid on any server in the same replication topology, which makes resuming across a failover possible. The server performs the skipping itself, so already-processed transactions are not resent.

The GtidSet class behind it is exported for consumers that need to work with GTID sets themselves, e.g. testing whether an incoming event.gtid was already covered by a snapshot:

import { GtidSet } from '@vlasky/zongji';

const snapshot = GtidSet.parse(savedGtidSet); // or @@gtid_executed
if (!snapshot.contains(event.gtid)) {
  applyChange(event);
}

Its MariaDB counterpart MariadbGtidPosition is exported too, with a covers() method so consumers do not have to compare sequence numbers themselves (seqNo values above Number.MAX_SAFE_INTEGER arrive as exact strings, and comparing them as Numbers silently rounds; internally GtidSet is exact over MySQL's full GNO range of 1 to 2^63-1 and MariadbGtidPosition over MariaDB's full u64 sequence range):

import { MariadbGtidPosition } from '@vlasky/zongji';

const snapshot = MariadbGtidPosition.parse(savedGtidPos); // or @@gtid_binlog_pos
snapshot.covers(event.gtid);                      // at or before the position?
snapshot.covers(event.gtid, { inclusive: false }); // strictly before it?

The default (inclusive) form answers "was this transaction's commit at or before the position?", which is what snapshot-barrier checks want: a recorded position covers the whole of its final transaction. Redelivery watermarks must pass { inclusive: false }: all events of a transaction share one GTID, so an inclusive check would treat the watermark transaction's own replayed events as already seen and drop every one after the first. A server id different from the position's entry for that domain reports not covered, even for a lower sequence number: after a failover, sequence numbers can regress under the new server id, so ordering against another server's watermark is not meaningful, and for at-least-once consumers an idempotent redelivery is recoverable where a skip is not. A domain the position does not list is likewise not covered.

zongji.gtidSet is available when start() was given a gtidSet, with startAtEnd (seeded from the server's gtid_executed on MySQL, gtid_current_pos on MariaDB), or when reading from the start of a binlog file (seeded from its Previous_gtids or MariaDB GTID list event). It is undefined when resuming from an arbitrary mid-file file+position checkpoint, where no exact value can be known. Tagged GTIDs (MySQL 8.3+) are fully supported: tagged transactions appear in the set as uuid:…:tag:intervals sections, and a checkpoint containing tags resumes correctly (its wire encoding is only understood by MySQL 8.3+ servers).

On MySQL, GTID resume requires gtid_mode=ON and a set is a per-source-UUID collection of intervals. On MariaDB, GTIDs are always on and a position is a single watermark per replication domain (domain-server-sequence): resuming replays each listed domain after its watermark, and domains not listed replay from the beginning. If the server has purged binlogs your checkpoint still needs, start() emits an error (code 1236) naming the problem; MariaDB additionally reports a checkpoint ahead of its binlog as error 1945/1955. A fresh snapshot is then required.

Event Methods

Neither method requires any arguments.

Name	Description
`dump`	Log a description of the event to the console
`getEventName`	Return the name of the event

Important Notes

🌟 All MySQL column types are supported, with type casting similar to mysql2.
MariaDB-specific types and features are supported: UUID, INET4 and INET6 decode to their canonical text forms and VECTOR to a raw Buffer of little-endian float32s (matching mysql2 query results); MariaDB JSON is a LONGTEXT alias and yields strings; COMPRESSED columns are decompressed transparently; compressed binlog events (log_bin_compress=ON) decode as ordinary query/row events; and MariaDB 5.3-era "hires" temporal columns (tables written under mysql56_temporal_format=OFF) decode correctly.
🙊 64-bit integers are decoded exactly using native BigInt (see #108). If an integer is within the safe range of JS numbers (-2^53, 2^53), a Number is returned, otherwise an exact String. This also applies to 64-bit integers inside JSON columns.
👉 TRUNCATE statement does not cause corresponding DeleteRows event. Use unqualified DELETE FROM for same effect.
When using fractional seconds with DATETIME and TIMESTAMP data types, only millisecond precision is available due to the limit of Javascript's Date object.
Binlog checksums (e.g. CRC32) are supported; zongji will detect and ignore the checksum bytes at the end of row events.

Run Tests

Install Docker
Run docker-compose up -d to start the MySQL and MariaDB containers
Run npm test to execute the test suite against MySQL 8.4 (port 3306)
Run TEST_MYSQL_PORT=3308 npm test to execute it against MariaDB 11.8; MySQL-only tests skip automatically, and the MariaDB-specific suite in test/mariadb.js runs (it no-ops against MySQL)

References

The following resources provided valuable information that greatly assisted in creating ZongJi:

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 290 Commits
.github/workflows		.github/workflows
lib		lib
scripts		scripts
test		test
.gitignore		.gitignore
.npmignore		.npmignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
docker-test.sh		docker-test.sh
eslint.config.js		eslint.config.js
example.js		example.js
index.d.ts		index.d.ts
index.js		index.js
jsconfig.json		jsconfig.json
package-lock.json		package-lock.json
package.json		package.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Release Notes

Quick Start

GTID example

TypeScript

Module format

Installation

ZongJi Class

Table metadata source

File and position resume

GTID-based resume

Important Notes

Run Tests

References

License

About

Uh oh!

Releases 3

Packages

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

Release Notes

Quick Start

GTID example

TypeScript

Module format

Installation

ZongJi Class

Table metadata source

File and position resume

GTID-based resume

Important Notes

Run Tests

References

License

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 3

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Packages