The Software Architects' Newsletter December 2023

[InfoQ Live Logo] The Software Architects' Newsletter December 2023 [View in browser]( Welcome to the InfoQ Software Architects' Newsletter! Each month, we bring you essential news and experience on emerging patterns and technologies from industry peers. In this final month of the year, we present the highlights from the newsletter's past year, covering topics such as modern data processing, edge computing, event-driven architecture, and the state of the art in enterprise programming languages. We aim to explore interesting emerging trends through the lens of a software architect, and the yearly [InfoQ Software Architecture and Design Trends Report]( often acts as the catalyst for identifying many interesting technologies, techniques, and processes. News Resilient Real-Time Data Streaming across the Edge and Hybrid Cloud In this QCon Plus talk recording published in January, [Kai Waehner]( explores different architectures and their trade-offs for [transactional and analytical workloads](. Real-world examples include financial services, retail, and the automotive industry. The Wonders of PostgreSQL Logical Decoding Messages In an InfoQ article published in March, [Gunnar Morling](, Senior Staff Software Engineer at Decodable, explores how [PostgreSQL can emit messages into its write-ahead log (WAL)]( without updating actual tables. Logical decoding messages can be read using change data capture tools like Debezium. Stream processing tools like Apache Flink can be used to process these messages, e.g., enrich, transform, and route. There are several use cases for logical decoding messages, including providing audit metadata, application logging, and microservices data exchange. However, there is no fixed schema for logical decoding messages; the application developer must define, communicate, and evolve the schema. Interested readers can learn more about data engineering trends via the "[InfoQ AI, ML, and Data Engineering Trends Report 2023](" and the [accompanying podcast](. Living on the Edge: Boosting Your Site's Performance with Edge Computing [Erica Pisani](, senior software engineer at Netlify, presented at [QCon London]( on [what edge is](, how running code and serving data on the edge can improve site performance, and how to leverage these options effectively in a site to maximize site performance with minimal architectural changes. Pisani started explaining what Edge is, presenting the concepts of cloud regions and their availability zones, taking AWS as an example (the same applies to Google Cloud and Microsoft Azure). Pisani describes "The Edge" as data centers outside the availability zones. Huawei Open Sources Kuasar, a Rust-Based Container Runtime During the [KubeCon + CloudNativeCon EU 2023]( event, Huawei announced that it had open-sourced [Kuasar](, a Rust-based container runtime that supports multiple types of sandboxes, including microVMs, Linux containers, app kernels, and WebAssembly (Wasm) runtimes that can be valuable for edge use cases. The topics of containers and Wasm were explored further in the [InfoQ Cloud and DevOps Trends 2023 podcast](, where the panelists noted, "We are seeing interesting adoption of cloud native frameworks and tooling across a broad range of systems, such as WebAssembly (Wasm), [and] eBPF". The Guardian Optimizes Mobile Push-Notification Delivery Architecture The technology team at the Guardian has used event-driven architecture to make [mobile push notifications faster]( to improve readers' experiences. The original architecture, optimized for concurrency, has been suffering from delays in notification delivery. The engineers used improved observability to make significant gains through experimentation. Guardian readers can use the mobile app to access content and register to receive breaking news alerts via push notifications. The event-driven architecture (EDA) behind that has been operating since 2009. Over time, notification delivery speeds have decreased, taking more than five minutes for some users, missing their goal of delivering notifications to 90% of the intended audience within two minutes. Grab Reduces Traffic Cost for Kafka Consumers on AWS to Zero Grab took advantage of the ability of Apache Kafka consumers to connect to the broker node in the same availability zone (AZ) introduced in Kafka 2.3 and reduced the traffic cost on AWS to zero for reconfigured consumers. The change has [substantially reduced overall infrastructure costs]( for running Apache Kafka on AWS. For readers interested in learning more about cloud cost control, Daniel Bryant sat down with Roi Ravhon on the InfoQ podcast in September to discuss "[FinOps, Application Unit Economics, and Cloud Cost Optimization](". Getting to Know Dapr with Mark Fussell and Yaron Schneider In a recent episode of the InfoQ podcast, [Thomas Betts]( talked to [Mark Fussell]( and [Yaron Schneider]( about the [Distributed Applications Runtime, Dapr](. In the latest InfoQ Architecture and Design Trends Report, Dapr is one of the early adopters of design ideas for portability and cloud-bound applications. Dapr provides APIs that abstract away the infrastructure details for modern applications, providing secure, best-practice implementations that work across any cloud, framework, and language. In related news, Wes Reisz summarized the [latest Dapr release that was announced at AppDeveloperCon](, a co-located event at KubeCon + CloudNativeCon NA 2023 that ran in November in Chicago. Datadog Creates Scalable Data Ingestion Architecture Datadog created a [dedicated data ingestion architecture]( offering exactly-once semantics for their third-generation event store, Husky. The event-driven architecture (EDA) can accommodate bursts in traffic in the multi-tenant platform with reasonable ingestion latency and acceptable operational costs. The architecture of Husky separates data ingestion, data compaction, and data reading workloads, which allows them to be scaled independently. All three workloads leverage a shared metadata store built on [FoundationDB]( and a blob storage service that uses [AWS S3](. The data ingestion workload uses [Apache Kafka]( to deliver events into the storage platform and route them internally to data writers. Related to this topic, Thomas Betts recently sat down with Lily Mara, Engineering Manager at OneSignal, for an InfoQ podcast, "[Building a Reliable Kafka Data Processing Pipeline with Lily Mara](". Java 21, the Next LTS Release, Delivers Virtual Threads, Record Patterns, and Pattern Matching Oracle has released [version 21 of the Java programming language]( and virtual machine. As the first release offering long-term support (LTS) since JDK 17 in 2021, there are 15 JEPs in this final feature, including Virtual Threads, Record Patterns, and Pattern Matching for switch. At [QCon San Francisco](, held in October, [Paul Bakker]( presented "[How Netflix Really Uses Java](". A large part of this talk was dedicated to Netflix’s upgrade path to JDK 17 and the planning for migration to JDK 21. For readers interested in the Java ecosystem, the [InfoQ Java Trends Report 2023]( was published in November. Mike Redlich, and the InfoQ Java team also [publish a weekly overview of key news from the Java space]( on InfoQ. Keeping Go "Boring" in Go 1.21: How Google Grants Backward Compatibility In a recent article, Google engineer [Russ Cox detailed what Google does]( to ensure each new Go release honors Go’s backward-compatibility guarantee. This includes generalizing GODEBUG in Go 1.21 to cover even subtle incompatibility cases. Introduced in Go 1, Go's backward-compatibility guarantee ensures all correct Go programs will continue to work with future language releases. Cox explains that this goal entails two major efforts: checking that each API change does not break anything and extensive testing to catch subtler incompatibility cases. Sponsored [Amazon Web Services Cookbook: Recipes for Success on AWS (By O’Reilly) - Sponsored by DoiT International]( [Amazon Web Services Cookbook: Recipes for Success on AWS (By O’Reilly)]( This cookbook provides practical insights into cloud challenges and success strategies, focusing on Amazon Web Services (AWS). Structured into technology-focused chapters, the book offers concise, self-contained “recipes” with clear problem statements, solutions, and discussions. Catering to both novices and experts, it uses real-world examples to demonstrate AWS applications, making it a valuable resource for career advancement. [Download Now](. Download the “[AWS Cookbook](”, sponsored by DoIT International Case Study Tales of Kafka at Cloudflare: Lessons Learned on the Way to 1 Trillion Messages In this summary of a related [QCon London]( talk, Matt Boyle and Andrea Medda discuss how [Kafka clusters are used at Cloudflare]( to process large amounts of data, with a general-purpose message bus cluster developed to decouple teams, scale effectively, and process trillions of messages. There are four lessons to be learned: - Always find the right balance between flexibility and simplicity: while a configurable setup may offer more flexibility, a simpler one allows standardization across different pipelines. - Visibility: adding metrics to the SDK as soon as possible can help teams understand how the system behaves and make better decisions, especially during incidents. - Contracts: enforcing one strong, strict contract gives great visibility into what is happening inside a topic, allowing one to know who is writing and reading from it. - Document the good work you do so you don't have to spend time answering questions or helping people debug production issues. This can be achieved through channels like Google Chat and wikis. By following these rules, Boyle and Medda could improve their systems and make their customers happy, even in high-stress situations. This content is an excerpt from a recent InfoQ article by Matt Boyle and Andrea Medda, "[Tales of Kafka at Cloudflare: Lessons Learnt on the Way to 1 Trillion Messages](". To get notifications when InfoQ publishes content on these topics, follow "[Architecture and Design](", "[Development](", and "[AI, ML, and Data Engineering](" on InfoQ. Missed a newsletter? You can find all of the [previous issues]( on InfoQ. Sponsored [Kubernetes Best Practices, 2nd Edition (By O’Reilly) - Sponsored by Microsoft]( [Kubernetes Best Practices, 2nd Edition (By O’Reilly)]( Get blueprints to build, deploy, and manage a successful multitier app in Kubernetes. The revised chapters in this eBook provide you with the tools to integrate Kubernetes clusters into your enterprise environment. You will learn how to deploy and manage clusters at scale using Kubernetes, streamline the testing and debugging experience with practice code samples and exercises, and solve common development challenges with insights from experts and developers. [Download Now](. Download the “[Kubernetes Best Practices](”, sponsored by Microsoft Upcoming Events QCon: For practitioners, by practitioners [Enhance your knowledge and stay ahead in the world of distributed systems at QCon London 2024]( This track, hosted by Thomas Betts, delves into the complexities of distributed systems, covering the pivotal roles of APIs, protocols, and observability. Gain insights from experienced professionals who have built, maintained, and evolved connections in distributed systems. [Find out more](. [Explore all 15 tracks at QCon London 2024](. [InfoQ Dev Summit Boston 2024, June 24-25]( InfoQ is delighted to announce a new two-day conference, InfoQ Dev Summit Boston 2024, taking place June 24-25, 2024. Discover transformative insights that will directly impact your immediate development roadmap and decision-making processes. [Register now with early bird tickets](. [QCon San Francisco 2024, Nov 18-22]( QCon San Francisco will be back, November 18-22, 2024. Early access to group discounts is available now. [Save your place with our early bird launch pricing until March 12](. About InfoQ Senior software developers rely on the InfoQ community to keep ahead of the adoption curve. One of the main reasons software architects and engineers tell us they keep coming back to InfoQ is because they trust the information provided and selected by their peers. We’ve been helping software development teams adopt new technologies and practices for over 15 years through InfoQ articles, news items, podcasts, tech talks, trends reports, and QCon software development conferences. We hope you find this newsletter useful. If not, you can unsubscribe using the link below. [Unsubscribe]( Forwarded email? Subscribe and get your own copy. [Subscribe](   Follow InfoQ on: [Twitter]( [Facebook]( [Youtube]( [LinkedIn]( You have received this email because you subscribed to "The Architects' Newsletter". To stop receiving the Architects' Newsletter, please click the following link: [Unsubscribe]( - - - C4Media Inc. (InfoQ.com), 705-2267 Lake Shore Blvd. West, Toronto, Ontario, Canada, M8V 3X2