The Software Architects' Newsletter September 2020

A monthly overview of things you need to know as an architect or aspiring architect. [InfoQ]( The Software Architects' Newsletter September 2020 [View in browser]( Our thirty-eighth issue of the Architects’ Newsletter again focuses on the topic of “streaming data pipelines, modern ETL, and data mesh.” We believe that these topics span several phases of the diffusion of the innovation curve. For example, in our latest [Architecture and Design InfoQ Trends Report (April 2020)](, we placed data mesh in the innovator phase, and event-driven architecture and streaming in the early majority phases. We believe that understanding all the emerging patterns, antipatterns, and technologies related to these topics is essential for a software architect. News The Best of Kafka Summit 2020 The Lenses team [summarized their attendance at this year’s virtual Kafka Summit](. The topics of the top five talks they covered include: reducing data platform storage costs; scaling Kafka at a retail bank; Kafka at the center of a security data platform; using Kafka to cure COVID-19; Automating and governing your data platform deployments with Git. From Monolith to Event-Driven: Finding Seams in Your Future Architecture In this recent InfoQ article by [Jayson Go](, principal in software development and engineering for a fintech company focused on investment technology, he argued [that event-driven architectures (EDAs) aren’t new but we may find that the practices]( we use in implementing them often distract from the core principles. This consequently causes us to overlook that EDAs are really about the events’ primary role of being a unit of change within the system. Go also discussed that the differences between commands and queries are more fundamental in CQRS than the separation itself, and event sourcing’s separation of events from states can provide the necessary abstraction of the system’s transitional state. The Evolution of Data Architecture: Moving to a Data Mesh Recently, the Skiplist team led a webinar in the [Kakou CxO Community]( that focused on the topic of the "[Evolution of Data Architecture: Moving to a Data Mesh](". The hosts examined the benefits of a data mesh, how to get started, how to make the transition, and how to mitigate the risks with a data mesh. Principles for Microservice Design: Think IDEALS, Rather Than SOLID In this recent popular InfoQ article "[Principles for Microservice Design: Think IDEALS, Rather Than SOLID](", [Paulo Merson](, a developer at the Brazilian Federal Court of Accounts and visiting scientist with the SEI, presented a new design model for microservices. He proposed that developers follow the "IDEALS": interface segregation, deployability (is on you), event-driven, availability over consistency, loose-coupling, and single responsibility. In regard to event-driven architectures, he argued that an important benefit is improved scalability and throughput. This benefit stems from the fact that message senders aren’t blocked waiting for a response, and the same message/event can be consumed in parallel by multiple receivers in a publish-subscribe fashion. How Banks Can Overcome SQL and NoSQL Related Obstacles with Apache Kafka [Madhvi Mavadiya](, writer/editor at Finextra, recently published an article focused on "[Event Stream Processing: How Banks Can Overcome SQL and NoSQL Related Obstacles with Apache Kafka](". Topics discussed included: event stream processing (ESP) in the context of finance, the role of change data capture (CDC), and the relationship from these technologies to the existing extract-transform-load (ETL) and online transaction processing (OLTP) systems. Case Study John DesJardins on In-Memory Data Grids, Stream Processing, and App Modernization In a [recent InfoQ podcast](, [John DesJardins](, field CTO and VP solution architecture at Hazelcast, sat down with InfoQ podcast co-host Daniel Bryant. Topics discussed included: how in-memory data grids have evolved, use cases at the edge (IoT, ML inference), integration of stream processing APIs and techniques, and how data grids can be used within application modernization. An in-memory data grid (IMDG) is a set of clustered computers that pool together their memory to let applications share data with other applications running in the cluster. An IMDG can be deployed on virtual machines (VMs), cloud platforms, and container orchestration frameworks such as Kubernetes. Many cloud vendors and IMDG vendors provide a hosted "as-a-service" offering. IMDGs have seen a resurgence in usage for dealing with edge (IoT) and machine learning workloads. Two popular use cases include stream processing and “gluing” together existing and new systems. Data stored within an IMDG can be accessed with low latency and can also be continuously analyzed using stream processing APIs and techniques, e.g. using [Apache Beam](. Hazelcast’s IMDG offering also allows Java developers to use the Java Streams APIs. Organizations that are modernizing their applications can use an IMDG as the data storage and processing "glue" between the old and new worlds. This can also facilitate testing. For example, the developers who are writing to the data grid can be testing their part of the system and verifying that this is working. Developers responsible for consuming events or data can verify the payload contracts and correspondingly test their applications in isolation. The complete podcast and show notes can be found on InfoQ: "[John DesJardins on In-Memory Data Grids, Stream Processing, and App Modernization](". To get notifications when InfoQ publishes content on these topics, follow "[architecture](" and "[streaming](" and "[ETL](" and "[data mesh]("on InfoQ. Missed a newsletter? You can find all of the [previous issues]( on InfoQ. Event Introducing QCon Plus by InfoQ: A Virtual Conference for Architects and Senior Software Engineers Developing our technical and non-technical skills remains our priority as architects or senior software engineers. Since many of us are unable to travel to attend conferences and support our development, the team behind InfoQ and QCon created an alternative virtual event: [QCon Plus]( (Nov 4-18), comprising of 3 weeks of live software engineering content designed around your schedule. [QCon Plus]( covers the trends, best practices, and solutions leveraged by the world's most innovative software shops. Taking place between November 4-18, the event is thoughtfully designed with shorter, [focused technical sessions]( spread over 3 weeks. Here’s what you can expect: - Work the event around your schedule. Plan your schedule around a couple of hours a day, two to three days a week for three weeks. - Learn from practitioners creating the future. Uncover trends and tools that are helping organizations tackle change and uncertainty. Validate your project roadmap during challenging times. You’ll learn from 54 speakers and [4 keynotes]( across [18 tracks](. - Peer learning groups & interactive problem-solving. Connect with a small group of peers and get answers to your problems through [highly interactive sessions]( including Q&As, AMAs, breakouts, and real-time collaborative action. - It’s not just about the tech. It’s about you too. Build relationships and network with a global developer community. Get together with peers in a safe setting where everyone is able to express themselves. Focus on non-tech topics from managing change to becoming the leader you want to be. This is your opportunity to spend quality time learning actionable insights from software leaders to help you adopt the right technologies and practices. [Find out more about the event and save your spot now](. This edition of The Software Architects' Newsletter is brought to you by: [NGINX]( Eliminating Microservices Complexity and Tool Sprawl Microservices-based architectures are giving enterprises the agility and scalability to deliver customer experiences championed in the digital economy, but the resulting tool sprawl and complexities are driving [an urgent need to consolidate and simplify](. As organisations begin to run into these challenges, the tendency can be to patch over the issues by deploying lots of point tools that solve different parts of the problem, resulting in tool sprawl, growing fragility and even more complexity. A great deal of expertise and interdependencies are required for just the operational environment, let alone the application itself. InfoQ strives to facilitate the spread of knowledge and innovation within this space, and in this newsletter we aim to curate and summarise key learnings from news items, articles and presentations created by industry peers, both on InfoQ and across the web. We aim to keep readers informed and educated about emerging trends, peer-validated early adoption of technologies, and architectural best practices, and are always keen to receive feedback from our readers. We hope you find it useful, but if not you can unsubscribe using the link below. [Unsubscribe]( Forwarded email? Subscribe and get your own copy. [Subscribe]( Follow InfoQ.com on [Twitter]( [Facebook]( [LinkedIn]( [Youtube]( You have received this email because you subscribed to "The Architects' Newsletter". To stop receiving the Architects' Newsletter, please click the following link: [Unsubscribe]( - - - C4Media Inc. (InfoQ.com), 2275 Lake Shore Boulevard West, Suite #325, Toronto, Ontario, Canada, M8V 3Y3