Building Event-Driven Microservices

Building Event-Driven Microservices

by Adam Bellemare
Released July 2020
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781492057895

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon Buy on ebooks.com
Start your free trial

Book description

Organizations today often struggle to balance business requirements with ever-increasing volumes of data. Additionally, the demand for leveraging large-scale, real-time data is growing rapidly among the most competitive digital industries. Conventional system architectures may not be up to the task. With this practical guide, you’ll learn how to leverage large-scale data usage across the business units in your organization using the principles of event-driven microservices.

Author Adam Bellemare takes you through the process of building an event-driven microservice-powered organization. You’ll reconsider how data is produced, accessed, and propagated across your organization. Learn powerful yet simple patterns for unlocking the value of this data. Incorporate event-driven design and architectural principles into your own systems. And completely rethink how your organization delivers value by unlocking near-real-time access to data at scale.

You’ll learn:

  • How to leverage event-driven architectures to deliver exceptional business value
  • The role of microservices in supporting event-driven designs
  • Architectural patterns to ensure success both within and between teams in your organization
  • Application patterns for developing powerful event-driven microservices
  • Components and tooling required to get your microservice ecosystem off the ground

Publisher resources

View/Submit Errata

Table of contents

  1. Preface
    1. Conventions Used in This Book
    2. O’Reilly Online Learning
    3. How to Contact Us
    4. Acknowledgments
  2. 1. Why Event-Driven Microservices
    1. What Are Event-Driven Microservices?
    2. Introduction to Domain-Driven Design and Bounded Contexts
      1. Leveraging Domain Models and Bounded Contexts
      2. Aligning Bounded Contexts with Business Requirements
    3. Communication Structures
      1. Business Communication Structures
      2. Implementation Communication Structures
      3. Data Communication Structures
      4. Conway’s Law and Communication Structures
    4. Communication Structures in Traditional Computing
      1. Option 1: Make a New Service
      2. Option 2: Add It to the Existing Service
      3. Pros and Cons of Each Option
      4. The Team Scenario, Continued
      5. Conflicting Pressures
    5. Event-Driven Communication Structures
      1. Events Are the Basis of Communication
      2. Event Streams Provide the Single Source of Truth
      3. Consumers Perform Their Own Modeling and Querying
      4. Data Communication Is Improved Across the Organization
      5. Accessible Data Supports Business Communication Changes
    6. Asynchronous Event-Driven Microservices
      1. Example Team Using Event-Driven Microservices
    7. Synchronous Microservices
      1. Drawbacks of Synchronous Microservices
      2. Benefits of Synchronous Microservices
    8. Summary
  3. 2. Event-Driven Microservice Fundamentals
    1. Building Topologies
      1. Microservice Topology
      2. Business Topology
    2. The Contents of an Event
    3. The Structure of an Event
      1. Unkeyed Event
      2. Entity Event
      3. Keyed Event
    4. Materializing State from Entity Events
    5. Event Data Definitions and Schemas
    6. Microservice Single Writer Principle
    7. Powering Microservices with the Event Broker
      1. Event Storage and Serving
      2. Additional Factors to Consider
    8. Event Brokers Versus Message Brokers
      1. Consuming from the Immutable Log
      2. Providing a Single Source of Truth
    9. Managing Microservices at Scale
      1. Putting Microservices into Containers
      2. Putting Microservices into Virtual Machines
      3. Managing Containers and Virtual Machines
    10. Paying the Microservice Tax
    11. Summary
  4. 3. Communication and Data Contracts
    1. Event-Driven Data Contracts
      1. Using Explicit Schemas as Contracts
      2. Schema Definition Comments
      3. Full-Featured Schema Evolution
      4. Code Generator Support
      5. Breaking Schema Changes
    2. Selecting an Event Format
    3. Designing Events
      1. Tell the Truth, the Whole Truth, and Nothing but the Truth
      2. Use a Singular Event Definition per Stream
      3. Use the Narrowest Data Types
      4. Keep Events Single-Purpose
      5. Minimize the Size of Events
      6. Involve Prospective Consumers in the Event Design
      7. Avoid Events as Semaphores or Signals
    4. Summary
  5. 4. Integrating Event-Driven Architectures with Existing Systems
    1. What Is Data Liberation?
      1. Compromises for Data Liberation
      2. Converting Liberated Data to Events
    2. Data Liberation Patterns
    3. Data Liberation Frameworks
    4. Liberating Data by Query
      1. Bulk Loading
      2. Incremental Timestamp Loading
      3. Autoincrementing ID Loading
      4. Custom Querying
      5. Incremental Updating
      6. Benefits of Query-Based Updating
      7. Drawbacks of Query-Based Updating
    5. Liberating Data Using Change-Data Capture Logs
      1. Benefits of Using Data Store Logs
      2. Drawbacks of Using Data Base Logs
    6. Liberating Data Using Outbox Tables
      1. Performance Considerations
      2. Isolating Internal Data Models
      3. Ensuring Schema Compatibility
      4. Capturing Change-Data Using Triggers
    7. Making Data Definition Changes to Data Sets Under Capture
      1. Handling After-the-Fact Data Definition Changes for the Query and CDC Log Patterns
      2. Handling Data Definition Changes for Change-Data Table Capture Patterns
    8. Sinking Event Data to Data Stores
    9. The Impacts of Sinking and Sourcing on a Business
    10. Summary
  6. 5. Event-Driven Processing Basics
    1. Composing Stateless Topologies
      1. Transformations
      2. Branching and Merging Streams
    2. Repartitioning Event Streams
      1. Example: Repartitioning an Event Stream
    3. Copartitioning Event Streams
      1. Example: Copartitioning an Event Stream
    4. Assigning Partitions to a Consumer Instance
      1. Assigning Partitions with the Partition Assignor
      2. Assigning Copartitioned Partitions
      3. Partition Assignment Strategies
    5. Recovering from Stateless Processing Instance Failures
    6. Summary
  7. 6. Deterministic Stream Processing
    1. Determinism with Event-Driven Workflows
    2. Timestamps
      1. Synchronizing Distributed Timestamps
      2. Processing with Timestamped Events
    3. Event Scheduling and Deterministic Processing
      1. Custom Event Schedulers
      2. Processing Based on Event Time, Processing Time, and Ingestion Time
      3. Timestamp Extraction by the Consumer
      4. Request-Response Calls to External Systems
    4. Watermarks
      1. Watermarks in Parallel Processing
    5. Stream Time
      1. Stream Time in Parallel Processing
    6. Out-of-Order and Late-Arriving Events
      1. Late Events with Watermarks and Stream Time
      2. Causes and Impacts of Out-of-Order Events
      3. Time-Sensitive Functions and Windowing
    7. Handling Late Events
    8. Reprocessing Versus Processing in Near-Real Time
    9. Intermittent Failures and Late Events
    10. Producer/Event Broker Connectivity Issues
    11. Summary and Further Reading
  8. 7. Stateful Streaming
    1. State Stores and Materializing State from an Event Stream
    2. Recording State to a Changelog Event Stream
    3. Materializing State to an Internal State Store
      1. Materializing Global State
      2. Advantages of Using Internal State
      3. Disadvantages of Using Internal State
      4. Scaling and Recovery of Internal State
    4. Materializing State to an External State Store
      1. Advantages of External State
      2. Drawbacks of External State
      3. Scaling and Recovery with External State Stores
    5. Rebuilding Versus Migrating State Stores
      1. Rebuilding
      2. Migrating
    6. Transactions and Effectively Once Processing
      1. Example: Stock Accounting Service
      2. Effectively Once Processing with Client-Broker Transactions
      3. Effectively Once Processing Without Client-Broker Transactions
    7. Summary
  9. 8. Building Workflows with Microservices
    1. The Choreography Pattern
      1. A Simple Event-Driven Choreography Example
      2. Creating and Modifying a Choreographed Workflow
      3. Monitoring a Choreographed Workflow
    2. The Orchestration Pattern
      1. A Simple Event-Driven Orchestration Example
      2. A Simple Direct-Call Orchestration Example
      3. Comparing Direct-Call and Event-Driven Orchestration
      4. Creating and Modifying an Orchestration Workflow
      5. Monitoring the Orchestration Workflow
    3. Distributed Transactions
      1. Choreographed Transactions: The Saga Pattern
      2. Orchestrated Transactions
    4. Compensation Workflows
    5. Summary
  10. 9. Microservices Using Function-as-a-Service
    1. Designing Function-Based Solutions as Microservices
      1. Ensure Strict Membership to a Bounded Context
      2. Commit Offsets Only After Processing Has Completed
      3. Less Is More
    2. Choosing a FaaS Provider
    3. Building Microservices Out of Functions
    4. Cold Start and Warm Starts
    5. Starting Functions with Triggers
      1. Triggering Based on New Events: The Event-Stream Listener
      2. Triggering Based on Consumer Group Lag
      3. Triggering on a Schedule
      4. Triggering Using Webhooks
      5. Triggering on Resource Events
    6. Performing Business Work with Functions
    7. Maintaining State
    8. Functions Calling Other Functions
      1. Event-Driven Communication Pattern
      2. Direct-Call Pattern
    9. Termination and Shutdown
    10. Tuning Your Functions
      1. Allocating Sufficient Resources
      2. Batch Event-Processing Parameters
    11. Scaling Your FaaS Solutions
    12. Summary
  11. 10. Basic Producer and Consumer Microservices
    1. Where Do BPCs Work Well?
      1. Integration with Existing and Legacy Systems
      2. Stateful Business Logic That Isn’t Reliant Upon Event Order
      3. When the Data Layer Does Much of the Work
      4. Independent Scaling of the Processing and Data Layer
    2. Hybrid BPC Applications with External Stream Processing
      1. Example: Using an External Stream-Processing Framework to Join Event Streams
    3. Summary
  12. 11. Heavyweight Framework Microservices
    1. A Brief History of Heavyweight Frameworks
    2. The Inner Workings of Heavyweight Frameworks
    3. Benefits and Limitations
    4. Cluster Setup Options and Execution Modes
      1. Use a Hosted Service
      2. Build Your Own Full Cluster
      3. Create Clusters with CMS Integration
    5. Application Submission Modes
      1. Driver Mode
      2. Cluster Mode
    6. Handling State and Using Checkpoints
    7. Scaling Applications and Handling Event Stream Partitions
      1. Scaling an Application While It Is Running
      2. Scaling an Application by Restarting It
      3. Autoscaling Applications
    8. Recovering from Failures
    9. Multitenancy Considerations
    10. Languages and Syntax
    11. Choosing a Framework
    12. Example: Session Windowing of Clicks and Views
    13. Summary
  13. 12. Lightweight Framework Microservices
    1. Benefits and Limitations
    2. Lightweight Processing
    3. Handling State and Using Changelogs
    4. Scaling Applications and Recovering from Failures
      1. Event Shuffling
      2. State Assignment
      3. State Replication and Hot Replicas
    5. Choosing a Lightweight Framework
      1. Apache Kafka Streams
      2. Apache Samza: Embedded Mode
    6. Languages and Syntax
    7. Stream-Table-Table Join: Enrichment Pattern
    8. Summary
  14. 13. Integrating Event-Driven and Request-Response Microservices
    1. Handling External Events
      1. Autonomously Generated Events
      2. Reactively Generated Events
    2. Handling Autonomously Generated Analytical Events
    3. Integrating with Third-Party Request-Response APIs
    4. Processing and Serving Stateful Data
      1. Serving Real-Time Requests with Internal State Stores
      2. Serving Real-Time Requests with External State Stores
    5. Handling Requests Within an Event-Driven Workflow
      1. Processing Events for User Interfaces
    6. Micro-Frontends in Request-Response Applications
    7. The Benefits of Microfrontends
      1. Composition-Based Microservices
      2. Easy Alignment to Business Requirements
    8. Drawbacks of Microfrontends
      1. Potentially Inconsistent UI Elements and Styling
      2. Varying Microfrontend Performance
      3. Example: Experience Search and Review Application
    9. Summary
  15. 14. Supportive Tooling
    1. Microservice-to-Team Assignment System
    2. Event Stream Creation and Modification
    3. Event Stream Metadata Tagging
    4. Quotas
    5. Schema Registry
    6. Schema Creation and Modification Notifications
    7. Offset Management
    8. Permissions and Access Control Lists for Event Streams
    9. State Management and Application Reset
    10. Consumer Offset Lag Monitoring
    11. Streamlined Microservice Creation Process
    12. Container Management Controls
    13. Cluster Creation and Management
      1. Programmatic Bringup of Event Brokers
      2. Programmatic Bringup of Compute Resources
      3. Cross-Cluster Event Data Replication
      4. Programmatic Bringup of Tooling
    14. Dependency Tracking and Topology Visualization
      1. Topology Example
    15. Summary
  16. 15. Testing Event-Driven Microservices
    1. General Testing Principles
    2. Unit-Testing Topology Functions
      1. Stateless Functions
      2. Stateful Functions
    3. Testing the Topology
    4. Testing Schema Evolution and Compatibility
    5. Integration Testing of Event-Driven Microservices
    6. Local Integration Testing
      1. Create a Temporary Environment Within the Runtime of Your Test Code
      2. Create a Temporary Environment External to Your Test Code
      3. Integrate Hosted Services Using Mocking and Simulator Options
      4. Integrate Remote Services That Have No Local Options
    7. Full Remote Integration Testing
      1. Programmatically Create a Temporary Integration Testing Environment
      2. Testing Using a Shared Environment
      3. Testing Using the Production Environment
    8. Choosing Your Full-Remote Integration Testing Strategy
    9. Summary
  17. 16. Deploying Event-Driven Microservices
    1. Principles of Microservice Deployment
    2. Architectural Components of Microservice Deployment
      1. Continuous Integration, Delivery, and Deployment Systems
      2. Container Management Systems and Commodity Hardware
    3. The Basic Full-Stop Deployment Pattern
    4. The Rolling Update Pattern
    5. The Breaking Schema Change Pattern
      1. Eventual Migration via Two Event Streams
      2. Synchronized Migration to the New Event Stream
    6. The Blue-Green Deployment Pattern
    7. Summary
  18. 17. Conclusion
    1. Communication Layers
    2. Business Domains and Bounded Contexts
    3. Shareable Tools and Infrastructure
    4. Schematized Events
    5. Data Liberation and the Single Source of Truth
    6. Microservices
    7. Microservice Implementation Options
    8. Testing
    9. Deploying
    10. Final Words
  19. Index

Product information

  • Title: Building Event-Driven Microservices
  • Author(s): Adam Bellemare
  • Release date: July 2020
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781492057895