KNet: .NET suite for Apache Kafka™

KNet is a comprehensive .NET suite for Apache Kafka™ that provides direct access to all Apache Kafka™ APIs and features: Producer, Consumer, Admin, Streams, Connect, and KRaft backend support.

KNet client-side features are also compatible with any broker that implements the Kafka wire protocol — see Backend compatibility below.

Libraries and Tools

KNet	KNetCLI	KNet.Templates	KNetPS	KNetConnect

Pipelines

Project disclaimer

KNet is a suite for Apache Kafka™, maintained by MASES Group and open to community contributions. Its primary scope is to support other MASES Group projects — both open-source and commercial — though it is freely available for any use. Dedicated community and commercial subscription plans are available. The repository and releases may contain bugs. The release cycle follows the Apache Kafka™ release cycle, with additional releases driven by critical issues or enhancement requests from this or other dependent projects.

Looking for Apache Kafka™ expertise? MASES Group can help you design, build, deploy, and manage Apache Kafka™ clusters and streaming applications. Find out more.

---

Scope of the project

This project aims to create a set of libraries and tools to directly access, from .NET, all the features available in the Apache Kafka™ binary distribution.

There are many client libraries written to manage communication with Apache Kafka™. Conversely, this project uses the Java™ packages released from The Apache Foundation directly, providing several benefits:

• all implemented features are available at no extra implementation costs — see KNet usage;
• avoids any third-party communication protocol implementation;
• access to all features made available from Apache Kafka™: the most important are Apache Kafka™ Streams and Apache Kafka™ Connect, which have no native C# implementation;
• measured high performance in many operating conditions.

Currently the project tries to support, at our best, the supported Apache Kafka™ binary distribution:

KNet	State	Apache Kafka™	Branch	.NET Framework	.NET	JVM™
3.x.x	Active	4.x.x	master
2.9.*	Active	3.9.x	release/2.9.X
2.8.*	Deprecated	3.8.*	release/2.8.X
2.7.*	Deprecated	3.7.*	release/2.7.X
2.6.*	Deprecated	3.6.*	release/2.6.X

The Apache Kafka™ packages are downloaded from:

kafka-streams-scala_2.13	kafka-tools	kafka-shell

connect-mirror	connect-file	connect-basic-auth-extension	trogdor

---

Backend compatibility

KNet uses the official Apache Kafka™ Java client packages directly. This architecture has a direct impact on backend compatibility: not all KNet features have the same compatibility scope.

Client-side features — Producer, Consumer, Admin Client, Kafka Streams, KNet Streams SDK, KNet Connect SDK, KNetPS scriptable cmdlets — communicate with the broker exclusively through the Kafka wire protocol and are therefore compatible with any broker that implements it, not only Apache Kafka™ itself.

Examples of compatible brokers: Redpanda, Amazon MSK, Confluent Platform / Cloud, Aiven for Apache Kafka™, IBM Event Streams, WarpStream, AutoMQ, and others.

Important: KNet also includes server-side features to start and manage Apache Kafka™ broker nodes, ZooKeeper™ nodes, and KRaft controllers (via KNetCLI, KNetPS, and Docker images). These features are specific to Apache Kafka™ and are not applicable to alternative brokers.

See Supported Backends for the full compatibility matrix covering all KNet feature areas.

---

Community and contributions

If you find KNet useful:

• Leave a ⭐ on the repository
• Open issues to report bugs 🐛 or request features
• Submit Pull Requests to improve the project

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to coc_reporting@masesgroup.com.

Summary

• Roadmap
• Current state
• Supported backends
• Performance
• Connect SDK
• Streams SDK
• RocksDB configuration
• KNet usage
• KNet APIs extensibility
• KNet Serializer/Deserializer
• KNet CLI usage
• KNet Connect usage
• KNet Docker usage
• KNet PowerShell usage
• KNet Template usage
• How to build from scratch

Recent changes

• V3.0.3+ / 2.9.4+: KNet Connect SDK now supports JVM hosted runtime — KNet Connect SDK connectors can be deployed and used like any other JVM-based connector in a standard Kafka Connect cluster; here full usage.
• V2.7.0+:
  • KNetProducer, KNetConsumer, and KNet Streams SDK manage JVM™ counter-part types;
  • serializers support data exchange based on byte array and ByteBuffer;
  • version 2.7.2 introduces ISerDesSelector to optimize serialization selection based on byte array or ByteBuffer.
• V2.5.0+: two breaking changes — uses Java.Lang.String instead of string (System.String) in generated classes; KNet Streams SDK manages JVM™ counter-part types.
• V2.4.0+: new KNet Streams SDK available.
• V2.0.0+: code base fully reflected from the Apache Kafka™ JARs downloaded from Maven; some developed classes remain alongside the specific KNet implementations.
• V1.5.4+: new packages dedicated to KNet Serializer/Deserializer.
• V1.4.7+: new KNetConnect project to execute Apache Kafka™ Connect related jobs; here full usage.
• V1.4.4+: new KNetPS project — write PowerShell client scripts for an Apache Kafka™ cluster and more; here full usage.

---

Runtime engine

KNet uses JNet, and indeed JCOBridge with its features, as its runtime engine to bridge the JVM™ and the .NET CLR within the same process:

• Cyber-security:
  • JVM™ and CLR, or CoreCLR, run in the same process, but are isolated from each other;
  • JCOBridge does not make any code injection into JVM™;
  • JCOBridge does not use any other communication mechanism than JNI;
  • .NET (CLR) inherits the cyber-security levels of the running JVM™ and Apache Kafka™;
• Direct access to the JVM™ from any .NET application:
  • Any Java™/Scala class behind Apache Kafka™ can be directly managed: Consumer, Producer, Administration, Streams, Connect, and server-side classes;
  • No need to learn new APIs: the same APIs are exposed in C# style;
  • No extra validation cycle on protocol and functionality: bug fixes, improvements, and new features are immediately available;
  • Documentation is shared between Java™ and .NET;
• Dynamic code: enables writing Java™/Scala/Kotlin code seamlessly within a standard .NET application in C#/VB.NET — see this simple example and KNet APIs extensibility.

Note

JCOBridge 2.6.* can be used for free without any obligations. A commercial license must be purchased — or the software uninstalled — if you derive direct or indirect income from its usage.

JCOBridge resources

JCOBridge	2.5.* series	2.6.* series
KNet	> 1.5.* series	> 3.0.* series and latest 2.9.* series
Release notes	Link	Link
Community Edition	Conditions	Conditions
Commercial Edition	Information	Information

Latest release:

KAFKA is a registered trademark of The Apache Software Foundation. KNet has no affiliation with and is not endorsed by The Apache Software Foundation.