"I love CodeOpinion - Software Architecture & Design"

2022-05-07 11:11:14

Skip to main contentCodeOpinionToggle navigationConsultingTalksYouTubeTwitterLinkedInDo you need help with Software Architecture & Training?Inquire Consulting &Training ServicesReal-World Event Driven Architecture! 4 Practical Examples May 4, 2022May 4, 2022 Derek ComartinLeave a commentEvent Driven Architecture has many benefits. Loose and temporal coupling, scaling, resilience, and more. But what does that mean? What are some actual use-cases? I’m going to describe 4 practical examples of real-world event driven architecture.YouTubeCheck out myYouTube channelwhere I post all kinds of content that accompanies my posts including this video showing everything that is in this post.https://www.youtube.com/watch?v=dhdLxW0vDro&list=PLThyvG1mlMzm2FyVpKDiU2c7VtrB2Zezg&index=36Video can’t be loaded because JavaScript is disabled: Event Driven Architecture in the Real World! 4 Practical Examples (https://www.youtube.com/watch?v=dhdLxW0vDro&list=PLThyvG1mlMzm2FyVpKDiU2c7VtrB2Zezg&index=36)External IntegrationsThe first practical use-case for real-world event driven architecture is integrating with external 3rd party services. In a lot of systems, when something occurs (an event) you need to interact and exchange data with a 3rd party service. This is where Event Driven Architecture can really shine.As an example, let’s say we have a Sales boundary that when an order is placed we need to send the customer an order confirmation email, send them a text message (SMS), and possibly also have a Webhooks system to make calls to other HTTP APIs.The beauty of event driven architecture is that each one of these concerns (email, SMS, webhooks) is decoupled from the action of placing an order. Each consumer has their own responsibility for handling each concern.One consumer is responsible for generating an email and sending it to AWS Simple Email Service (SES).Another consumer, concurrently but in complete isolation can be responsible for making HTTP calls to external systems in the form of webhooks.Lastly, there could be another consumer responsible for sending a text message (SMS) to Twilio.Each one of these consumers is independent of each other as well as the initial publisher (Sales) of when an order was placed. Each is executed independently without interference from the other. Since you don’t own external services, it’s hard to guarantee any type of availability. This means if one consumer is having availability issues downstream (AWS SES, Twilio, etc) all other consumers responsible for different external integrations are completely unaffected.This allows you to integrate all different ways in a plug-in-play fashion. Need to add some new functionality? Add a new consumer.WorkflowLong-running business processes and workflows can be difficult to manage. Event Driven Architecture allows many different logical boundaries that perform specific behaviors apart of a long-running business process.Let’s say the Sales boundary publishes an Order Placed event.Our Billing service needs to know when this occurs so that it can Bill the customer or create an invoice.Once the order has been billed, we then want to allocate the product in the Warehouse and create a shipping label. For this to happen the Billing service will publish an Order Billed event.The Warehouse will consume that Order Billed event to then allocate the product and create a shipping label.The last portion of this workflow now is for Sales to know when the Shipping Label has been created by the warehouse, so it can update the Order Status from “Order Placed” to “Order Ready to Ship”.There are different ways to achieve this, but specifically outlined above is called Event Choreography. You can also manage this workflow with Orchestration.Each service is responsible for a portion of the workflow, without realizing they are. The Billing service doesn’t care about a shipping label being created within the Warehouse. All it cares about is when an Order Placed event occurs from Sales, it needs to Bill the customer and publish an Order Billed event. Each service boundary is doing its part in this long-running business process.State TransferChanging the shape of data for different use cases can be really helpful as an example of real-world event driven architecture. We can leverage Event Driven Architecture to tell consumers when data has changed so they can update their copy of the data. Let’s say we have three different types of data stores. One is a cache, another is a data warehouse we use for Reporting and BI, and the other is a data lake we dump data to for future use.In order to keep all these data stores updated, we can publish an event that contains the data that changed.From there we can have various consumers that take the event and mutate the data in their respective data stores. We may have a cache that needs to be invalidated as well as a data warehouse that needs to be updated.Also independently we can have a consumer that dumps that in a data lake for future use.What this allows you to do is be more real-time about various read models that your system uses. You likely won’t always have a single type of database as a system grows and has different needs and requirements. A common example is OLAP for reporting and OLTP for business processes.Temporal CouplingLastly, removing temporal coupling between your own internal systems is a major benefit to Event Driven Architecture.When making synchronous blocking request-response calls between services, we’re adding temporal coupling. When an order is placed in the system, if we’re using HTTP calls or gRPC to call between services, we’re temporally coupled. This means that all the services need to be available at the same time in order for everything to be functioning correctly.In the example above, Sales needs to call Billing to charge the customer when an order is being placed.After that is done, it then has to make another blocking synchronous call to the Warehouse. But what happens if that call fails or the Warehouse service is unavailable? We’ve already charged the customer in the previous call to the Billing service. We have no distributed transaction so there is no rollback.This is why in the example earlier I illustrated the workflow using a message broker because we remove the temporal coupling. Each service operates independently without being constrained by the availability of other services.When an order is placed, if the Billing service is unavailable, nothing fails. When it comes back online and picks up the Order Placed event from the broker, it will continue the workflow.Once you remove temporal coupling, you start treating your internal services, just like the external services I was mentioning at the very beginning. When you move to an event driven architecture because everything is independent, it doesn’t require everything to be online and available to complete long-running business processes and workflows.Real-World Event Driven ArchitectureHopefully, this illustrated the different ways you can leverage an event driven architecture. Integrating with 3rd party external services, handling long-running business processes and workflows, changing the shape of data in real-time for analytics and reporting, and removing temporal coupling to add resilience.Join!Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design. As well as access to source code for any working demo application that I post on my blog or YouTube. Check out theYouTube Membership or Patreonfor more info.You also might likeBuilding a Webhooks System with Event Driven ArchitectureEvent Choreography & Orchestration (Sagas)Cache invalidation isn’t a hard problem Follow @CodeOpinion on TwitterADVICE to my younger self as a Software Developer April 20, 2022April 20, 2022 Derek ComartinLeave a commentWhat advice would I give a software developer? Software development, design, and architecture can feel overwhelming. Especially when you’re just trying to get into it as there is so much to learn. What exactly should you focus on? A particular framework, language, or library? After over 20 years of software development as a career writing line of business and enterprise systems, here are the top 5 pieces of advice I’d tell my younger self.YouTubeCheck out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.https://www.youtube.com/watch?v=pSMXDfRfyEc&list=PLThyvG1mlMznuNW2tITIGmgQqJikLBqab&index=17Video can’t be loaded because JavaScript is disabled: ADVICE to my younger self as a Software Developer (https://www.youtube.com/watch?v=pSMXDfRfyEc&list=PLThyvG1mlMznuNW2tITIGmgQqJikLBqab&index=17)Business KnowledgeIf you’re writing Line of Business or Enterprise Software, having a good understanding of the business domain your writing software for is critical. This is why it’s perplexing to me that the software industry is so focused on frameworks, libraries, and platforms. Yes, technical concerns matter, but the sheer lack of emphasis on business concepts is amazing to me.The best business app developers I know are equally business savvy as they are technical. I'd argue that there needs to be way less technology X CRUD 101 tutorials and way more domain experience reports.— Derek Comartin (@codeopinion) June 21, 2020By far the best developers I know have a good understanding of business concepts and the specific business domain they are in. This gives them the ability to speak to a varied audience. From both spectrums, they can speak with domain experts all the way to other software developers.Having experience in many different business domains only helps your understanding of designing a system. There is a common thread that generally is involved in most business domains that you can then relate to others.My advice to my younger self is don’t get so caught up entirely on the technical aspects of being a software developer. I wish I would of put more focus on understanding various business concepts earlier in my carrier.Logical BoundariesThe more you understand how various businesses domains operate, the better you can understand how to design a system by decomposing it into logical boundaries.Defining boundaries is one of the most difficult things to do, yet is one of the most important.When you don’t really know a particular domain, it’s like walking into a dark room with only a small flashlight. As you point the flashlight around the room, you get a better sense of how big the room is, what’s on the floor, how tall the ceilings are, etc. It takes time and it takes a lot of exploration to get a better perspective.Logical boundaries own a set of business capabilities and the data behind those capabilities. Within a logical boundary, there are long-running business processes and workflows. These also might span multiple logical boundaries.CRUD (Create-Read-Update-Delete) has its place, but not the heart of the system which generally has a lot of complexity. The outer edge of your system has boundaries that are more in a supporting role for reference data can often be CRUD or be an external system/service that you integrate with. Define logical boundaries that isolate the business capabilities your system provides.Coupling & CohesionFundamentally understand Coupling & Cohesion. Seemingly every design decision revolves around these two concepts. coupling is the degree of interdependence between software modules; a measure of how closely connected two routines or modules arehttps://en.wikipedia.org/wiki/Coupling_(computer_programming)cohesion refers to the degree to which the elements inside a module belong together.https://en.wikipedia.org/wiki/Cohesion_(computer_science)Circling back to the earlier section about defining Logical Boundaries is exactly about trying to achieve a high level of functional cohesion. Logical boundaries will have some degree of coupling because they will need to interact with each other.Ian said it perfectly. It’s all about trade-offs.Coupling & Cohesion. Yin & Yang. It's a push/pull relationship where you try and find a balance for a given context.— Derek Comartin (@codeopinion) January 18, 2022ComplexityIf you’re in a complex business domain, managing complexity can be difficult. But don’t make it even worst by adding unneeded technical complexity. This happens way too often by developers wanting to use new technology/library/framework without any idea if it solves a problem they actually have.Complexity also comes in the form of logical boundaries as well as coupling & cohesion.Defining logical boundaries is trying to prevent you from creating a giant turd pile (big ball of mud). Ultimately you’re trying to build a set of smaller turd piles. This allows you to isolate the complexity to a smaller logical boundary that is limited in scope.As much as you think the system you’re building is going to be perfect, over time different insights and gaining new knowledge will mean that the overall design will be less ideal than you want it to be. And that’s OK!Isolate complexity the best you can give your understanding at the time of making a decision. Understand that you won’t get it completely right. However, DO NOT add unneeded technical complexity because a discovered a new design pattern, library, framework, etc that you think will solve all of your problems. Also, DO NOT add unneeded complexity based on assumptions or “what ifs” that you think might be needed to make the system more flexible possibly in the future. Give yourself options but not at a high cost of complexity.Know what you don’t knowThere is so much to learn in software it’s pretty much impossible to know everything. However, it is helpful to have a surface-level knowledge or just be aware of the current landscape. That doesn’t mean you need to be an expert at all, rather the opposite. Just being aware of a new tool/framework/platform and the problems it solves can be really helpful if you find yourself with that problem.The key is acknowledging that you aren’t an expert but rather only have entry-level knowledge.Don’t be an Expert Beginner. My software developer advice is to realize that there is a mountain of knowledge you just don’t have and that you don’t know what you don’t know. That’s OK! Don’t take yourself too seriously.Join!Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design. As well as access to source code for any working demo application that I post on my blog or YouTube. Check out the YouTube Membership or Patreon for more info.You also might likeHighly COHESIVE Software Design to tame ComplexityWrite Stable Code using Coupling MetricsContext is King: Finding Service Boundaries Follow @CodeOpinion on TwitterAsynchronous Request-Response Pattern for Non-Blocking Workflows April 13, 2022April 20, 2022 Derek ComartinLeave a commentWhat’s the asynchronous request-response pattern for? We’re used to synchronous communication. You make a request to another service and get a response. When you move to asynchronous communication, you often think of messages as fire-and-forget. You send a command to a queue, and the consumer handles it asynchronously. Because of the temporal decoupling, you don’t know when the message was processed or what the result was. However, there is a solution! Let me explain how you can use the request-response pattern to reply sender once a message is processed.YouTubeCheck out myYouTube channelwhere I post all kinds of content that accompanies my posts including this video showing everything that is in this post.https://www.youtube.com/watch?v=6UC6btG3wVIVideo can’t be loaded because JavaScript is disabled: Was that message processed? Asynchronous Request-Response Pattern (https://www.youtube.com/watch?v=6UC6btG3wVI)Blocking Synchronous Most developers are familiar with blocking synchronous calls. If you make an HTTP call to a service, you’re making a blocking synchronous call.When Service A makes a blocking synchronous call to Service B, it must wait to get the response (or acknowledgment) that the request is completed. There are many situations where this is entirely appropriate, such as in client or UI code. UI code reaching out to get data from a service is naturally blocking request-response.However, service to service communication using blocking synchronous calls has complexity. As an example, if an order is placed within the Sales service, it makes a blocking synchronous call to the Billing service to charge the customer’s credit card or create an invoice.Once that blocking call succeeds, then needs to make a blocking synchronous call to the warehouse service in order to allocate the product and create the shipping label.This entire process is blocking and is temporally coupled. Both Billing and Warehouse services need to be available in order for the Sales service to place an order. Also since we lack a distributed transaction, we can be left with an inconsistent expected state if there is a failure. If the request to Billing was successful, but the request to the Warehouse failed, we now need to develop some resiliency to make another call back to the Billing service to refund the order. But what if that request fails? This isn’t resilient at all.Asynchronous Request-ResponseOne solution to this problem is to remove the temporal coupling by using asynchronous request-response. This way each service can operate independently without requiring the other service to be available and able to process requests. To do this we can leverage asynchronous messaging and also apply the request-response pattern.Here’s how asynchronous request-response works. First, the producer will send a message to a queue.The consumer will process that message asynchronously.Once the consumer has processed the message, it will create a reply message and send it to an entirely different queue.The producer will then consume the reply message from the reply queue.So now let’s jump back to the example of Sales, Billing, and Warehouse services. Using asynchronous messaging and the request-response pattern, we remove the temporal coupling and have each service work independently.When an order is placed in the sales service, we’ll have an orchestrator that will send a Bill Order command to queue on our message broker.Once Billing consumes that message, it will then send an Order Billed reply message that the Orchestrator will consume.Once the Orchestrator consumes the Order Billed reply message, it will then create and send a Create Shipping Label command that the Warehouse service will consume.Once the Warehouse has processed the message it will create a Shipping Label Created reply message that the Orchestrator will consume.Once the Orchestrator has processed the reply, it changes the status of the order that was placed to Ready To Ship.Here’s what our Orchestrator (NServiceBus Saga) looks like. It’s handling the appropriate reply messages and sending new commands (messages) to the broker as an asynchronous workflow.View the code on Gist.Because all of this work is done using non-blocking asynchronous messaging, none of the services must available when an order is placed. They are all working independently processing messages in isolation.The asynchronous request-response pattern allows you to tell a sender that the message has been processed and what the outcome or result was. You can leverage this to then build workflows to involve many different services all in a non-blocking way.Join!Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design. As well as access to source code for any working demo application that I post on my blog or YouTube. Check out the YouTube Membership or Patreon for more info.You also might likeEvent Choreography & Orchestration (Sagas)Data Consistency Between Microservices Follow @CodeOpinion on TwitterPosts navigation12…123Older posts Follow @CodeOpinion on Twitter#mc_embed_signup{background:#fff;clear:left;font:14px Helvetica,Arial,sans-serif}Categories.NET (215).NET Core (93)ALM (5)Architecture (61)ASP.NET (47)ASP.NET Core (67)AWS (4)Azure (3)CQRS (54)Data (4)DocumentDB (1)Domain Driven Design (9)Event Sourcing (13)F# (5)HTTP API (12)Lean (2)Messaging (29)Object Oriented Programming (7)Orleans (7)Owin (3)Patterns (10)PHP (1)Practices (16)Principles (5)Programming Languages (3)Random (10)Roundup (67)SharePoint (1)SignalR (7)Software Design (1)SQL (2)Troubleshooting (3)Uncategorized (16)Visual Studio Online (1)sparklingTheme by Colorlib Powered by WordPress