Abstract
A message-oriented middleware (MOM) with an associated system built on top of it is targeting scenarios where the action–reaction cycle is desirable to be separated both in space and time. This allows clients and servers to be detached from each other, and follow their intrinsic life cycle. Apache Kafka is a high performant and fault-tolerant MOM.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
A client/server model is an abstraction in distributed systems, where at a given moment in time, we can identify a pair of collaborating components; one as a service provider (server), and the other as its user (client). In MOMs, it is better to speak of peers, as the roles of clients and servers can change quite rapidly.
- 2.
By an event we assume a compact, well-structured, and machine-processable message .
- 3.
Establishing direct connections with all types of data processors wouldn’t scale. The only allowable direct connection is between a smart meter and a broker (a component that facilitates event publication). By using a MOM any temporal coupling is eliminated by letting the producer emit a message even if nobody is listening on the other side of a channel. Referential decoupling means that a producer doesn’t know or care who will consume the message. Of course, you might have higher level temporal dependencies and ordering constraints in a distributed system, but we are talking here about a single request/response cycle.
- 4.
A configuration with three partitions per topic with a single broker is only good for educational purposes (we use this setup to showcase our custom partitioner).
- 5.
Our docker-compose.yaml file is set up for a single broker (the port number on the host is fixed; notice the bold part in the listing). To create multiple brokers on a single machine (to boost performance on a multicore machine) you should delete that bolded 9092: part (we are assuming that at this moment the cluster isn’t yet created). You would then be able to scale the running cluster (after starting it up) with a simple command of docker-compose scale kafka=3 (for three brokers). Don’t forget that the port numbers on the host for those brokers will be dynamically allocated (they are visible after executing docker-compose ps). Therefore, you will need to alter the bootstrap.servers property inside the configuration file for both a producer and a consumer.
- 6.
In our case we don’t bother about an ordering issue. However, if you want to keep proper ordering of records with multiple retries, then you should also set the max.in.flight.requests.per.connection producer configuration parameter to 1.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2016 Ervin Varga
About this chapter
Cite this chapter
Varga, E. (2016). Apache Kafka as a Messaging Hub. In: Creating Maintainable APIs. Apress, Berkeley, CA. https://doi.org/10.1007/978-1-4842-2196-9_12
Download citation
DOI: https://doi.org/10.1007/978-1-4842-2196-9_12
Published:
Publisher Name: Apress, Berkeley, CA
Print ISBN: 978-1-4842-2195-2
Online ISBN: 978-1-4842-2196-9
eBook Packages: Professional and Applied ComputingProfessional and Applied Computing (R0)Apress Access Books