Mm-hm Yeah ooh all right so So when we talk about world order. Yeah no. Would you decide or we have hybrid and she also has gotten me to join us. All right screen going. So alright so let's see. So Okay. So we're going to talk about events >> And so we're trying to put this in a kind of advanced software which is what we do. And so I suppose that in your classes you don't have a program that's pretty far from making software. Nowadays software just make a really regular programming programs here and there. And then maybe you should think because >> It's a great program. I got variables right you see the day. When you write software you can say variable static variables very temporary variable. But that's still very romance larger software. We're going to immediately after processing. So here are some books which are. So this is the first book is this guy. So he wrote a book. Yet is event processing stuff. I guess you that LinkedIn LinkedIn with the guys who wrote this program back someone's Kafka program. So he wrote this book really different enterprise integration pattern. This is about making large programs and how they fit together or they both actually different. Thankfully we talked about this either. All right so first let's talk about enterprise integration. I guess the word. So I say software components. Right so when we when we re actually whenever you attack a problem you're always going to be a big problem and that's how you always do. That's advice why you write functions that way. So so you do that right because c is followed problems can never stood right that's what I do that Also when developing that you can you can break it into smaller problems and different developers can then work on those individual problems. Yeah maybe even the details of the solution that the label is no. Part of it is if you fall in loopholes that. Now a lot of text on the screen because I like that person >> Well let me not small or okay or ok. So then all these solutions to the individual component the detailed solutions >> Or maybe not depending on the details of the other solutions to the other components right the people who work on these different components and they don't have to interact with the details. And now I think you probably you probably have talked about this this idea when to object oriented programming the idea of class yet because it's an interface. And the ADA changed changed all the functions in the class. To give you the interface the same and that right. So see you develop. Different people can develop classes. People just use the interface Between the Buttons. All right so this is the idea here which isn't learn best. And then I'll see a flexibility to right now. You can improve individual phonons. Maybe you can you can integrate off-the-shelf much like you do that again right right so if you do a good job of earthquake Cappella Superman coming to import other things and use those rates we use those etiquette books. Right so you don't make one long punchy. But instead you make lots of functions where each function solves the initial parts. All right so we know that we're all agree that we'd like to have software components right okay then the software have to interact right so now this is the problem so we need to interact with interaction requires coordination coordination complex. So we could plan things. A small system a right depends on system B system. A nice way. This isn't a design for devices to be to understand. So the V2 Is a system a requires me. Hey you're really lucky. You probably when you're making these functions make up your writing code and I go okay I need to do something you write that. Let me go back and use that when you're writing system day after versus day after you get off figured out. Hey guys an organism. Yet you do a lot of planning is there. And then you have the ability to anticipate what's going to come back for. So this is the a >> A very serious problem in software development. We know fathers hey browser today. But now we have this interaction. Play an independence. Great. Yeah this relay. So it's a liability for example. Right so the reliability is the worse. Worse worse than the worst are about liability. So here's a map. So this a system a breaking broken with probability 0.5 it'll be probably half the time it just right and that's the full system is functioning only when both a and B are functioning. Properly at the system function is point you vibrate >> They both have to be working for that to work but we're saying that the system is only o if both at one year olds. Right yeah. Yeah and yeah so that's worse than >> This is a 0.5 early fire hoses much worse. Yet more systems. So we have tenses those each Greenberg and finally five probability will be 0.5 raise to ten. Lineage about working decreasing exponentially with the number. As reading these guys they have 1500 assistance videos isn't. So whenever the probability of 0.9 raised 1500 with that >> And yet when I do do that yeah the 69th. So it's never ever ever get. Green eyes still et al and worthy Reais right so you'll never get it working. So rate but that's not harvested as not houses. So that's not a GUI. Catholics systems where system is going to we're only able component or components. So up to now writing programs we call functions and those functions. It's all been brokers in the whole thing works or doesn't work. You know. So you function function a we have two functions even programmers do functions and they say wonderful works and what doesn't work. And your whole function worrying doesn't work. Writing functions up. Now it is like this that everything has to work or doesn't work. Let's not have you use where. So think about like email. You sent an email. Don't send it to. You don't need everything to be working. So that we send email to a friend that person's laptop isn't aware that person's email server doesn't work. You can still send your email send there. And then you can tell right now he says is not working. And then later they turn on their laptop. Right so when you use systems that are systems can be on at different times and they work just fine. So far programs mostly they both afterward. So this is the deal. How reduces my barrier. So the antivirus software technology making software work together is called integration or maybe antivirus immigration. You know try to integrate these different components which are a. So we're all components are o. So we need to figure out how to make them work together. About this way not where everything has to work. Right so this is a it is the Achilles heel of components. And I was I was thinking about integration between the bones and how they gaze of things where we used is loosely coupled. So please write very Philippe Greek one stream made them invisible and she held his ankle so that we get any of that. So >> So everybody says yes we need to have loose coupling and that'll that'll solve component interaction problem. Really is hardening. Effect that you know I think Roddenberry software this time it's going to be doing all the time and you won't be thinking. But we shouldn't think that asked me she reports loose coupling for how to do that. And everybody said so yeah so it's a very common thing Albanians loose coupling and say. So what we are trying to do are suffering points to be as independent as possible. So that you really have to think about when you're making the software you really want to be >> I actually hear your monitors make things independent of us. Right otherwise you're here. Yeah both isms I'd have to work. And I always think about making independent of path right and of course it's objective. But it's kind of April. So the opposite of loose coupling or tight coupling. Right so when you have a Java class or another classmate and those are tightly coupled >> Okay >> Definitely working on Java code than developers right you're all working towards our development happening in the same key are all working on this. Pretty commonly be coordinating closely. You're all working on the same code which is interacting. Not. Yeah I think that that's true. You have to be on the same team is decoupled FB like coordination between the team building which part of the very common way gray codes that one giant piece of java code. Best on tight coupling. Similar little bit little better. Instead of working on the same same piece of code you can work on different code. One code can call me undergo so that things we use every nipples remote procedure calls or remote method invocation or you've HDB. We call functions. So this is definitely better than working on one piece of java code because Mao developers be different teams may either between companies. Now you have a little bit about independence. They were still black complexity because you still have to get all the interfaces right. Now. Yeah exactly >> Note that this is a tricky one even location. So we say advice GDP or NPO. Or if you say you know what that surveys that's being called the API guess effects. Also they maybe for doing single server that's handling this. Sort of once you use art and see that's it. And I use RPC forever right because now everybody who's your MCAT or change we're going to change how we do that. Now you have to Everybody has to change from RPC to something else. And you pick one of these. Now change it as children and a lot of places >> That tells you that tightly coupled with you making a change of luck. Again is low. So as the ADB server is not working days isn't down. Alright so we're back to the situation where in order for things to work everything has to be wrong. But that's not a problem. We shouldn't eat important. Nothing over more take couple of people who've these tight coupling for a long time. For us to see very widely use was nine and that's a bad thing. But you know if you can have these loop you have a loose coupling that's bad right and then you get all the benefits of software components >> It can be easier to achieve loose. So like you have to do you loose coupling and stuff but okay so one of the chief ways that we do lose coupling is with asynchronous. So what this means is that you have you have a caller calling function and a call the call and call and wait for the answer. So that doesn't work. So usually when you're writing Java code right here we're going to call some function of opportunity return. Whatever terms you use here >> That's fine so that unless you know sometimes that's really the only way to do it. But if it's not you can do other way you can call something and they go on and do something else. Then or wait for the answer or some other way. Now you can use of asynchronous. That's the entirely voice you usually Java class and call some other class gets the answer keeps on going. But what if you could just push me or I see your email >> So you use all the time the email was asynchronous writing press send. And then what do you know you go out and do the thing. You don't even check if it says set. You're off doing something else. And then eventually it'll say sand set right and then eventually the valorize the other side is completely asynchronous. So UCB. So that's great to have some asynchronous. Right so you press and way way way rival things read right because this is completely asynchronous >> That's great. Right. Tweet was posted. You don't have to wait your thumb or e-commerce web site. That can you do that in an asynchronous way >> That was hardly anything else. Yeah people said a student's. Alright so that's exactly how you think it is. So what what came actually ok. So we always know e-cigarettes are bad. So then the question is can we make the whole thing work asynchronous recommend payment right where the payment right where he was presented with it with the payment. Payment was for example. So the press or right and actually their credit card. So that so maybe you can do other way merchandising that avail credit card isn't valid things. So maybe one way to do it is you submit it. And then you have a status page. And eventually the status page without the sword events right but we are examined inventories screwed up. You'd have to You we ask them if these two are probably somehow from alcohol and you'd have to use a data inventory is some sort of tie right and then you can permanently sub a. So maybe >> Maybe you could make all these systems from P100 asynchronous. And that would be great passwords. So and then you have loose coupling. And then you say okay hey so very proud thing. Yeah. So business I'll not post-treatment 0. Yeah that's right. I guess it depends on where you first. Take the credit by first grade right ok. Yeah yeah you're figuring out if we're. Yeah so so yes I'm Jerry I'm sure that people make many of the things asynchronous. So you really want for yeah yeah you have to work it through and he decided you can ingest ileus weather data. Entry rejected energy pretty neat. So I just like blob of data ingestion. Other things. Yes. Okay oh yeah I can give them other things where you can get where a software program where you can just ingest or just basic us versus them email. So web server not right that OF $70 >> That's that's not so easy to see. But you can kind of something we post something. And then you wait for that or some other place. Oh something and give it some weight some work. That's another place for that instead of going back and forth. Right you just split it somewhere and then you go to the status place and say yeah that is an asynchronous or servers synthesis. Depending up on each line something like this. There are triggers and who's serving. Right yes yes yes. Girls is asynchronous and the edges are where you say yeah so it looks like it's against but really smooth edges ageism asynchronous system and then retrieve it. So it looks like oh I submitted a request. I'm waiting for the response and get the response. Looks asynchronous but actually inside. Yeah so yeah I think I'll call it function. Any wafer response. But what that function is doing is it's actually Smitty press Search. They Kashmir census asynchronous string excuse me. The cash has to stay for a while. Yeah yeah. Yeah so you make things that look synchronous an actuary. But let's say I want to get you guys unless a patient record. Hey what are you getting things on on-demand radio guests again you make you can make things a lot easier. Here a network firewall access but it's also going to LA to do wishing I arrow that May 1 say well I guess if they're not in such a network firewall we can't do this. You can't make out oh is it okay if it was time to go through this okay the fact that not everything not everything but you don't get carried away by. It can be asynchronous. And yeah. Low latency versus asynchronous. So they're not exact. So they're related to alias for years appear here as explicit race a virus injected right here's a smoke. Here we detect the fire smoke detector. Right so I say as he was no value to me there's a senior assistant. Since I see. So if we made this part synchronous what what's the what's the use of making this nothing and do essay if it fails right so it sends a message i textbook into this system as opposed to making Yes I think that's right. So this is the fires. It definitely doesn't have any of this system over here. The smoke detector doesn't have anything you can do to to prove the relatively it didn't work So what you do is you can make it anything but a message into here. And then you wait some time. And then it times out. Oh I'm sorry. Okay so I take the message the textbook to get it on the slide. So the message and then you put it into this queue messages mostly that current statuses I0. Then why do these alarms pulls the message out so the state is now active with a timestamp. But nothing happens somehow the way we were away and didn't quite work so it timed out. So we're back to smoke detector. This alarm didn't go off right and pull. The passage couldn't actually sounding models. We've a backup system pulls message. Hey I'm the state goes to some time. Now the alarm is actually raised right. Sounds to me out. There's one word. Now we give the message messages handlers for adolescence has done >> So that's that's the way we can handle this kind of thing that again flatter. So right and this one the text message hexes mode. What is this he'll pull the methods. One method we said some timer. We Wait wait wait our waiting list for this alarm to go off and say everything yes it has been handled but it didn't work. So we cleared we reset reset state. And then there's other one pulls a message. And I always say it's active >> This stuff works. Let me say it messages handle. Now does it just handles and now we're done. That's way better than something like this right we have the smoke detector a dx. The 9000 favourite technique works with this alarm right but which one is when you know we're allowed laryngitis and 12 and orphans this alarm ready with all you'd have is largest though that works right that's the detector that can work with a certain alarm systems. Which one's right. Then how do you figure out which one we have to get some of the other ones. And they are very difficult to hook these things together. Right so we so this is what you really don't want to do. You don't want to have these these these components directly communicate. This is tight coupling where these are where this one detects the smoke tells this thing with this one rejects it has offline or something like that. Then this one. This one takes over. You don't have to build that here. You don't want to have that honest dependence There are some things do have to be lazy right so even if I have like asynchronous very so so ancient times log in right after scientists if it's right that one is probably good infrastructure management right so machine and starting out decided that okay or not. Okay and we talk about network security. So some things you really have very low latency. So then synchronously. So let's go back to this. And this are the things that we see. Messages we see there is there's a thing in the middle here. Right that was that was that that was a key message queue and these messages are sending timers. And these acts and the retry or less that this is what we're going to do to make this whole whole thing work and this will give us this loose coupling smoke detector goes directly work with the alarms absolutely loose coupling. And and and you see that it's asynchronous yes they definitely see how this is loosely coupled. And these are the things you've managed is the thing. Right so here's the generic system where we have some system some universal message as the thing in the middle. And we have these things to be a publisher to publisher or sorry sources of information sources smoke detector. All the different kinds of detectors that he's made all kinds of arms. And maybe you have something which is a message happens right since a this one this one can pull out the message and then an empowered as it can put that message back in. But all these things are now loosely coupled because you have this system in the middle there. Yeah many types of messages many pictograph right so so this is very good. Why do we have in this picture we said one message and one type of assay. But you can imagine here. If you want to add in another alarm system as long as I can work with this message queue then you're all set. All right so so that that's somewhat loose coupling. Allows that you can add things later and other processors and other sources. That means loose coupling means that we didn't have to do all this planning ahead of time. I think. Yeah any questions here. So why don't you only have really think that you only have one type of data. What can you really need is complicated as I said I don't know. So so Maybe you don't maybe you can build variants of it. Anyway to go. On the other hand nowadays these systems are are pretty good. That if you're going to glue this thing to this thing. Now we have the cloud such standard systems to glue things together. That maybe it's easier to instead of having this one directly it might be easier to get directly that when these two are tightly coupled. So usually two things. One the loosely-coupled. You might as well get. One of these sounds sort of event five misses >> And people like it better because they think these are pools of uses. Okay so what do we have here so we have these things where like the convent processors or message handlers or message brokers Cubase based and my base. We have these publish-subscribe protocols are different solutions. One is Google Cloud Pub Sub yet very popular kafka. Ok so we have a string >> So let me go back here. This is one particular thing where a single event. But another way that this thing that we can have lots of events coming here lot of Q and you have a class that didn't exist and very symbol a single event where you can also handle streams can be tweaks as can be different things that are handling. Things doesn't really matter a destroyed invasiveness system. Passes them. Gradually edX components. 06 extremely sorts. Where does where does that where does this inputs come from >> Okay so where does the data come from the most cases actually the data constant software we go. We usually are building software receiving the inquiry driven >> If it's right because we already have so search theory data's coming in after such a huge array of any data formatting think that perhaps one of the examples of days of ingesting that email tweet sensors IOT scale that was once theta's sample. What they think and true control probably yeah. What does SU >> And so their industrial control systems they have some plant which is go through some faster your advisory completely unimportant. So these are the are the right. Imagine a factory where they have lots of things that are controlling all that equipment. What are the what are the thing that they could fall on the following equipment they are increasing pressure open about those about what's all that stuff. So they so the all those all that. I already got hardware-software handing industrial systems are called endpoint and their computer control via network are or escape raises the big worry about. Like their electric grid skated devices which are on the Internet and somebody hacks into skated devices then they'd bring down there. And right there is that the name of that when somebody attacks as I already said they had the centrifuges to Jerry to purify uranium and somebody was able to infiltrate those. So what they were able trade was the SCADA systems. There is revealed by Seaman Willie offline but there was no network astronauts still there. They were able to re-invent what that was was was that scales. Right so we can have >> Write data from our *** right from our vein or actually we have our own emails or tweets that they've pushed into our systems or maybe from somewhere else. And it will push this data to ways of testing. Right so we have. So the data is distributed. Then we have the sources or adding network latency predictive length maybe data becoming too fast >> Let's see so here's a picture. Right I guess forgivable for pizza now he hasn't better names. We have so producers and consumers to producers. They published the data sanity but producers are produced that. Now if I look at that in a producer's publish consumers subscribe bizarre but these are the words we use producers and consumers >> And they publishes yeah I've never seen sender recipient served. I think maybe that might be hopefully better. Now welcome publish-subscribe around for awhile. So I mostly do things new genes. But these frameworks are backed by today are either there's variation. But the ideas have been around for a while. Okay so producers consumers publishers and skirts. How do we how do we do the integration for once they have a direct messaging communicate things. Talk about the tight coupling and indirectly where we write to a database for example file or or or the or the keys. So this is this is the this is what we're I guess is yes. So I guess it is three. It should be a third one is indirectly includes other cues. So we have directly which is tight coupling to find something indirectly we're heading producers as fiber. And there's different ways of producing or you can put that data in some place and he was like Thank you put it in the database. But one common ways with it with the messages. Okay so there we have the asynchronous because here's a picture producer Hugh and consumers. And we have and we have a buffer here. So this buffer is should be should be persistent data. So should be this thing this Q should be highly reliable way so is usually put into hard-drive. Right so this actually is not clearly says is putting pieces verse. That's right. This is the noise in the data >> The stories of the ribose there's no loss of data also the buffer to be super huge building. Because the hard drive. So so everything should be should be good events from an average. But I think usually less excuse and not a single system usually have made systems. You have three or more systems. So when you send the data into the system it goes into many places For example in GCP they have some very large number system. We said that there this send the face view. This is a secret as call you send it and that is okay. Right so so this so this part synchronise this part from here to here. So this synchronous part here when you send that in it or copy the data too many systems. And as soon as I said there's more than half of them responded that they have the data and they saved the data successfully to the hard drive or ones that are the response is safe then this thing finishes. So this is so a lot of things are hidden but extremely reliable. So self-assured versus them very very resistant but they definitely zeta. I think they both see you on the desk the same memory they have many backups so that the average. And then over here q. So the message it is saved like you until it's processed or somehow knew. Okay so safely stored in the heel and we have these we have a tiny order is everything has to be transactional. And actually you can do something completes and then you're done. So this will be great when you buy something when you transfer money in the bank I should be a transaction bank should be either money or money there where you know it's about basically do beforehand wasn't there there wasn't this they have to know these things interventions should be should be I guess building aspect of it. One moment nobody money's not there. And then the next number that is I don't know when that moment is is that at a time and this will be tiny guarantee them ordering guarantees. Maybe these message should be process. Maybe exactly once by a consumer. Maybe the event was either that retrieval. I guess I can me Japan messageQueue that depends on. Maybe depends on my content defense. Usually is first-in-first-out varies. When you say guarantee they just briefly guaranteed Write a check to make sure that in order to actually seek some will they will guarantee or they get right so so yeah they do allow. For example. There are now two things happen at the same time. You do something. And then you know then you can definitely. So yeah absolutely. That's one of the issues that these things right so if we do finish except yet then after will be after nasa thing databases. In other words do you live and then there's some cloud solutions. The accuser that's easily that be your first. What are the helpful thing you can get here. They were very simple but you pull it out though. So you don't have to go to the more elaborate few insufficient greatness coupling. We work with reliable. Okay that's a heel more copy and way to do it is message brokers. This is I will call it heavyweight cases and now we have solutions. You can build your own system. But this is this. Connects these producers to consumers and producers receives a message from producer pushes them because you've already done. Right you have usually at last producers and my messages messages might be removed. Didn't weigh different goods series. That when you're when Arlington seniors receive the messages are supposed to live in the messages are limited. Or maybe they're saved. They said flexibility depends on different types of consumers >> I will get different different types of consumers hearing the same message isn't peice of producers and different types of different types. And so he's always saying I think this was sending tweets as well as sending stock prices is one is to smoke glass. So they can be messy general Message Broker. This one can be consuming different types of messages here. And consumers can receive multiple kind of a general system >> You've got lots of things to have the same high-rate. So maybe two these things are both. Well suggesting truth is going to be different. So just send me in the tweets and the prophecy that maybe the Boston St. wherever there. Maybe every potable general general place home season. And we'll talk a lot. You read the rest of the class. Synchronous push data high holiday time. Bullying highway consumer has out-of-date because he was stopword embrace code crashes all the time >> That's okay they still being pushed into your system. Or your system starts working again or the process is not great. But you know everything is going to still work. So you know how is your username Gita humour. And I thought about Barbara thing. Yeah improved development agility ice and now producers can be added for different kinds of dairy. So initially we were like oh we just need a smoke alarm. We're just gonna produce this detection of slope. Why do we need this thing and then we also any poisonous vapors or CO2 things. Okay so first thing is once a day. Later I you you cited more events. Easy. You don't have to plan ahead talked about Integration components and one of the browsers there's too much planning to make the whole thing work. Here you say look I don't know what we're going to deal with the data. I know going to have some sort of data you build it and just putting in >> Through this thing now you can build consumers okay over the data this way or this way over God send that customer legally messages Lawson voice. You can add that later. If you go and new types of pursuing Yeah so you have improved agility. If you have one of these message assistance scalability we looked at that in more detail scalability producers and consumers. But it depends we'll look at this liability. Sooner crashes everything's a message handling system is should be very reliable. So you can see you can always should recover from consumer crashes or a. Okay so now let's look at a feel good. So to not omega systems not really play or be very scalable you can use one of these systems and they will they ask yet feel good adult in looking at this idea load balancer yeah. A tweet >> Enlightened system right you get a monitors a tweets for traffic jams right you're going to give traffic updates iMac in traffic jams for them right definitely want and actually the tweets arrive and push it into your system. You saw that the consumer re re using messages. And so when I take the tweet processor take the tweets look complicated to get to see the pull them in and you put them in this system. They come in. And now the consumer is going to pull the tweets and way scan for words that say something about patents >> Right so you pull the tree out you look you scan over the tax and look for words like cars or tracker that hey it does have words. Then you gotta put it into some neural network and I'm going to try to figure out where in the traffic jam is actually really is talking about yeah or or wet. Okay and then you results are correlated across other tweet right and try to figure out is it really time again I'm hopeful people are tweeting about this traffic jams nine okay there's really is a real-time Ajahn breakthrough. That's how you are going to say let's suppose that it takes ten milliseconds that profit the tweet because there's no preferred works. But it does take time worth anything I'm saying. Did you all correlation neural network I have them. Right >> Recent wars arrive at a very high rate. So how how well how do you go for the system you can build with their life and how you make these things work faster Nathaniel. During rush hour a lot of words about without factory where it's about to happen in a. So now the traffic where between consumers and I think you're ingesting system as some yeah a year or two busy because now they have a long path. Right so we just we have many detectors >> This load balancing. Each each of these during the same same processing say Rossi and they're running a parallel system to be able to handle this kind of thing. And this and this is Lovell's who these consumers are not completely independent. And we can move them as maybe as Russia or columns. You can add more. And then I am going to add them as the demand. And then the message system could do what I guess you could do the same for the omega0 center in a round robin fashion than the first one here segue everywhere. That's not very good but maybe a hues that. So when a consumer is reading it cause the next message off. And then when this 100 would that be a good approach then if one of numerous takes a long time you want the other one. Said that both for many of the tweets >> I'm very fast ten milliseconds but every now and then they take the same. So if this one is Property Law where you should send another one right so it shouldn't statute called the 23rd. You only accept tweets that way so they can they can they will have some of them will be busy doing along with him and some of them are not new as it passes the short life. So so there are these they're not all going to be busy right now to them than me doing log processing. A lot of them are very busy. And these ones are doing the path prophesying. This one finishes the vast prophecy. And then this one gets a long-winded process but this would be the load which is which is which is varying over time right some case law. That says you have enough of these load will be spread. Cuba right so this so they can fill in this one gets busy. This one fill in for this one is delay. The other one >> Hey I sub a important idea which is different from around a round robin is busy. Then I don't know what you do is you do a lot less that you really want to have real load balancing where the other ones can take up the slack. Okay another one is the ingestion monitoring this up the tweets as they arrive. Isn't reasonable >> If they arrive at a high rate okay how are we getting this one machine how many teeth are there so many trees are coming into the system produces about saving web managers and they're all pulling tweets in different ways. And so they're pulling the same message that hey so this is our solution. So far we have made gestures May variances producing the same data and many consumers consuming the same. So this is logout >> I could do the same thing. Same data coming here saying that goes like this. You have multiple producers sometimes going this way. And he had a message broker neighbors as consumers make guesses and different things. Fanout fanout different logout. Now we're partners in two different ways. One is 21 way or the other way where consumer buying trends are totally. Different teams are doing essentially we have this whole pre-processing ingestion system. Why don't we do more things than just what we can do that. Now the tweets are coming past that but now yeah completely different types of consumers. Seeing different kind of desert. Logout fan out different types. Of course this will be lowbrow off. Many of the man may leave. This can be very hard to do a completely different approach single Tweet and he passes by both different from here. Here single tweak if possible by this one or this one or this one. Not by all Fan out versus that mouth taste consumers right and they're put there pulling or pushing up two cases we can we can push the data sewers or we can pull. So only when you pull it. So so type one because they were pulls the baton the axon eta. So no other type one consumer is an important message. And then and then I'll type to pull a message maybe failed and pulls or they can be typed into the kind of very separate systems. Now you have almost two separate system to them. Detector a consumer trend monitoring or they're both fascinating and the both reliable but more or less independent. Load on digit message comes in. They get sent him consumer type one anchors everything to. And saw the best food for producing the same type data again and now it goes to two to one message delivery of water. So we asked the messages actually looked at that in the fire. So one of them didn't confirm the message. So the message was a recent announcement of the message that indicates that the message has some process. If it has not received in time the delivery with retry hey nothing great snafu. I will write out every word. What do we do with reordered read delivery right you have read liberating asked can I have reordered we have messages being processed or message twice like this. This is a very young isn't hard to handle. Right so here we have we have multiple consumers were doing was logout. Okay so the first message to find a mosquito but he's too busy place. And the back of the neck comes then. So positive messages three and it fails >> So then what happens so that message three day passes here. Hey fail now about thing but a little bit happened. Now we did message for first then message three. Or what happens if this message pretty much finished. So she went to was processing and pretty much finished. And just before I finish a crash or a pow-wow or something something that he broke. So it's finished processing. And then a dig at the back and he says doesn't act when you're done. You can get the ACK back. So now this one bus now too big brow >> We're an out-of-order messes with the four than three. And we get message three twice. C Well yeah yeah so that but the Add-ins and not something else somewhere else. But yesterday that we have we have a problem. So yeah how to how to avoid this problem. So what one thing is proud to leave. If we didn't have this one is a profit message one done Republicans to three. And then if you fail to start back up again and then again I guess they're pretty easy to get messages. Three again. Pretty easy of a single system to figure out if you process three array. And then you do four and so on. So one solution is to get rid of this this the load balancing. The load balancing is we saw our great it's great but it causes problems with water. I sell one number just to get rid of. Yeah so for example we can do is you can have a single consumer and then break the consumers into little pieces. So you had lost your loss of consumes right here set of I see this problem but let's just make it that consumer to only process the messages. These messages are related through the passage through whenever Gloria. I shouldn't take the stream and divide it into smaller pieces. The Hindu forever so GCP for example it allows only a tenth of a message types. You can have each traces each person who's generally tweak either all message high vitamin that far fluid only if we if we have one consumer passes passes one message type that means ten hasn't received that effectiveness is quite a big number. So pursuing the fast enough that we should get a 10 thousand machines for example. To handle messages right so let me go back to we're trying to get rid of this and load balancing for a single consumer >> All right and then we won't have this out-of were duplicates. So one way to do that. One way to do that while it's not a giant single consumer that's not gonna work. It's not going to go to him or another way to do it. If you take this stream of events you break into different types. So now he's type is this >> The low data rate and now you can see we really only need a single consumer for each year that GCP. There'll be a limit. There are ten thousand ten thousand paving number. It's up to you. You can think about making systems where you Udi has only a single consumer single consumer. And yeah and you're laughing types of consumers by each customer. For example each of your customers is single consumer. That's okay legally if and how the customers customers every day that could be okay. I think they may do that. But yeah it still might not be possible way. Please arrive very fast. And that's something that people there people do. Also load balancing because I'm just for food but also sterility or y system breaks. There is another one. They got the. Why does it crashes or it can take a long time to process it some message. So so love owls you also good for liability. Also we haven't said yet. So investors take a long hard process another consumer pre-process that messages. So I keep slaves as well. And that persistence sometimes right you need to deal with the out-of-order of things because the banks are entries with an outward really has nothing to do about that >> Okay so for out of messages brakes they really are unavoidable with what to do with them. Once they just you know just passes them in order. And that basically usually usually are in or maybe maybe does not worry about it. As long as it doesn't completely break. Maybe okay. Or another thing is if you can you can detect the MS ignore. It depends on your a. So things are really out of words that it's not your fault that the customer is not going to mine that much >> A lot of other things going on in the systems of numbers. So if you misuse a message secret numbers right then you can wait for how long. Now the case is due to ignore our awareness. So we can do all of these some combination and multiple or or it's good to write your code. Or usually what people do You didn't handle slightly out of order with very outward and you ignore it. So it doesn't systems >> Handle out of water and they add up to one hour after one hour I think north. So that is a problem. With these classes we had low if you have a low balance you've got to deal with with our messages and potentially do the same thing. He said it processes should be but I didn't potent. Either both these that you read process has no impact. This is super important. All yes I remember my first refers doing complicated coding. We didn't do that now over oil >> Often somebody processes at the same time going back. So we insert into a database. Is a database you always make your you're always update or insert. A theories are developed. You can guess the sequence numbers we drop duplicates. So so systems. So systems will do for you but first choice. So yes next time we'll talk about Dataflow and as the Amazon. All right so let me integrate systems. I know there's probably this is all to be sure you have heard a and gathering ideas. Have you ever impose hoses right yeah no hours our inspecting that. We all very new. But yeah there. You will see that you have this brown eyes and mouth choosing very foreign 8-10. Mm-hm. Yeah yes Yeah yeah oh Mm-hm wave. Mm-hm All right so okay so publishers publish subscribe. So this is a very subtler GitHub for publishers. And then progress consumers are pulley. Will probably I will be talking about so far. And then the way some more turnout was that he had topics. So events are grouped into three different kinds of producers. A breezy different topics is the work that happens. Though of it. So you can use different worthy event type data type but the real word topic kind of login using. So for now I will try to use the word. Topic. Topic means different event types. We can group the hierarchy the higher your topics one topic and then suddenly by that into other topics like that >> But that depends on the system. So we're going to look a little bit more closely at GCDs described system passes all build them. Rock solid handles all this for you. And that's a whole scribe and they don't have a single layer. You have topics and that's it. There's no subtopics. Or Catholics or. Here's the idea it's not GCP but for content-based topics where you can look at the content itself and figure out that that's minus v. So that's not so important >> But you can also make that hey you can have one topic of Anthology one topic that goes into consumers receivers and parse the data. And they and and based on the data in January they produce data about different topics. So you can easily take a a not content-based and make it into a picture all different data sources bargain about last class itself. So really tall. What do we need to use all of these I think we have time for email events. If you're if you're if you're tracking logs and so it's very easy to go through. Okay so let's look at TCP's futures rates. So they have thoughts is well I want to layer up topics and also highly scalable. It's also blowin push a message in any region and pull it. Many of you hear anything. You can push and pull data out or have data pushed to you. The data is only held for seven days. That's it. That's the unique feature of TCP actually many of the solutions. They only hold the data. After seven days that something which is kind of not that long you can make systems that will take the data and push it into a database. And there's very efficient systems to do that. But that is the power drawing out. Something that one has to consider. And then you have to act. And they tap and the timeout you asthma. Method that we'll talk a little bit more about timeout By default when you get a message you consumer you're gonna mess up your ten minutes about receives number four get around that. We're talking about. Ease inscription is guaranteed to get. So we have so consumer's eyes description each speech consumer. Each description is guaranteed message at least once or twice. But you always get at least once. You can have a suit. So so >> But it is possible to get it more than once as possible. Be reordered. They also have this message replay feature and you can go in and say take all the messages for days and and and replay. So I know that consumers have acknowledge them. But they'll remove the dolphin. Every run that's really good for bike racing you test your code change when a rerun everything for past say 40 hours. That's very useful for debugging but you only but any going back >> About 70 new divisive snapshot snapshot. And then you can rerun that snapshot. In a separate message effort separately >> Okay so what do you do with a time out so this could be a problem of timers. Sure what's the tire out be set to one would think he sent him out large right at that way right so just on a single a single timeout message you say yeah to process the message in whatever that time. The attendance was not process can be read delivered rather consumer. So when I approached us and said that timer to be a lot to be a large value. That way if it takes process takes I then doesn't spuriously high Mao and send the message to consumer theory. When we the people. Yeah that's right. That's right versus a small fails reprocessed quickly. And I am the length. So BED is tomorrow. Tomorrow these are 22 times. So one is a heartbeat. You need to send a heartbeat. Every whatever you say I say every 30 seconds at least And then so you just keep on set. So if you have possibly takes a long time they just keep on saying. Alright so if you could sue because he crashes then there's very quickly there's no heartbeat that a message is sent somewhere else. Possibly take long times keep up me. Saying herpes heartbeats are are are much better than I know. So what do you do when your code is use have heartbeats all of your code. You're always handy. Piece of code to say things that are being processed in rp. You're only required to make sure that no single step of your code takes a long time. You have to really make sure that it's no longer. So if you're going to communicate with external system maybe making a web request to the database and you have to add something in there to make sure someone else will do that. Tuesday. Udp are the DNAs. You've to thread one thread. Database that other thread is sitting RPs. So you're always unhappy cases some some play. But that yes then Brothers. If the herpes C or aloo you said your heart and retired from the loop and you're stuck in a loop or break out in the name sending heartbeats. But he's any Harvey forever. So you sort of another timer that says regardless and parties massive impossibly high should never be over seas and RB. Every every say ten seconds by the massive processing time is 30 minutes say recover quickly and you and your hierarchically deposits for a long time. That three other things. Consumers can get busy to track. If residue to old inscriptions can time out and just be archived passages. Anything that one of the other approach that was that was pops up as other state. Another growth is Kafka which is very popular. And I thought this isn't very interesting shows jobs based on keywords. So it looks so you'll post jobs and some some website looking at keywords. So most highly ranked currently is James frequently is Agile's offered well doesn't mean that it's also this isn't UK UK's. Most. So they found. This is 12 by the jugular vein. So our sprite right but so it's a little bit vague here and you can So yes boundaries is very much a placebo and adjust their C-sharp interesting. Very fascinating Azure. Amazon does. So you can get a sense of why it is hot in terms of jobs. When Hafez right pretty high so it has rank two Gcv. Gcv Gcv. You talked about Kubernetes higher grades and a lot of things here. Well what else maybe some hear some pumps you talked about but is a three here. So when the task is a very popular. Okay so vague as well so it's open source. But it was relatively limited as so as having no solution in the cloud provider and having them support is great. Let's see. Here's a passive. Google recently had a Kafka. And they did this maze of ga. Ga ga beta beta beta beta GA in general. So that they have not yet made not that Azure as November a year ago that you're still. I didn't knew where all the all the vendors have their own sort of thing as far as the solution. If b carefully because E is different a virus you don't know how well they support it. Because they have a very rock solid whereas GCP Azure is processing systems are those can be extremely mature. So okay so popular. Because okay so this is different from from the South. They hope they these logs. I say Lincoln Lincoln Lincoln's company. Oh yeah let me go back to that For this one the other one that you these are the kind of the standard systems that have been around for a long time. So rather than q i this message based systems there been around for a long and very solid. This is the oldest and live in Europe. You see these although these are various solid solutions. Kafka is far more far high-ranking jocks. Definitely trending >> And the thing is so so he asked Lincoln Febby do things quite differently. So instead of using this idea of message Hapsburg may use this idea of Lars message broker. So much that the message is consumed as live but that's not true GCP. We saw first of all there's many kinds of messages EthernetServer inoculated into all the types of consumers. Have I believe them. And GCP doesn't lead to a host of her 17th. So Bailey lives are different are different saves it saves the sequential block >> And they're usually saved to disk. So so the metadata that SB safe can be very long. So it's not. That's why the RNA message processing system but it just has logs. And then you can read from the log wherever you want. Right and then what about load balancing so we talked about sorting and again low balancing comparator. That little bouncy is where you have looked at before if multiple machines doing the same tasks to servers web servers and they can handle customers. And if one gets busy other shorting is where presumably of data by independent jumps day one customer I loaded our ID and that one day it is and the other customers on the sorry the law balancing or related but a little bit different. Because the shortest independent pieces dependencies this is having more love Elsevier mobile systems that can do the same thing. Right and then if you have a message that we saw that that that you have load balancing. Then message ordering is a problem that messages can get slot one. Another buffer shorting is not a problem like that. Okay so Cs words that you'll see over and over again when you look at Kafka topics and partition. So GCP has single hierarchy so that I can single layer just have topics. Kafka has two layers of topics and then partition the words are. I'm not sure why they have such different words that should lead to topic subtopic. They call topic and partition. The topic is the first layer hierarchy after that is partition And he sent that message you send it to a Popper suggested by a single machine or a single system you get backed up like that. So the talk's going to a single single cluster different topics can I go to a different cluster so load balancing is achieved by having different topics. And then I guess that each part partitions are handled by each partition is handled by a single consumer. Good. Okay let's go back here. Ok so we have so Kafka talking about load off in a certain set of consumers and consumer with sadness some messages and see what happens. When we take a topic we braided Partitions. Partitions distributed across like this division. Partition a partition is consumed by exactly one consumer. So partition is consumed by exactly one consumer. And consumer can consume no more than one. Ok so more than one hearing picture. Here here's a topic. I have three partitions we break it up into three pieces. So this consumer is doing this partition is considering these two parties are like that. This is not okay. Each partition can only be handled by one symbol. This consumer purchases made my business because he was handling two partitions. This Is that okay because we have this partition being handled by two consumers last matlab. Ok. So each you have a topic you break it up into little slices consumers and then that different consumers can handle parties. He says that is not load balancing. Is is is sorry take data and slice it into pieces. And now each piece is handled by a different system is also very important that each partition has to be handled by this. This is okay. Yeah this is not so useful. You haven't actually consumer here which is not being used. But you can you you'll have three partitions. You only use three consumers. This is not so this is an OK but not very good for partitions. But one partition is very busy and The parties are spread so the purchases are spreads each consumer has duplication but that must have gotta be better to have this provision maybe go over here. So love it partitions versus versus upset right so here we have basically here we have this two layer. Two we have topics. We break it up into slices like this whereas this Vs and pumps up we have single Popper and his team applied. It's being processed by logout load balancing across multiple. Here we have shown. So what happens then is these fees get busy right then we raise about sues Nobel analysis for that. So then you have to make sure that they're not they don't get too busy. So 11 thing you can do is you can have you can pay the partitions very very small. You should make a revision is very small. Then if something advocate if there's going to be you can add in another consumer. And now you see how you can reduce the load because now these partitions partitions are being assigned soon. So that's how you can do load balancing. Here when load balancing we just added another sewer. Here if one load balancing we always make our consumers partition small we can add a will have three partitions. You can't do that with your Watson little marketing and keep on things >> So OK so I guess it's okay but you know one of the big selling points of this whole approach is that is that half guy doesn't reorder. But he doesn't reorder things within a partition which make a Camry or across partitions where he had many little partitions. What's the point of yes I know you're not that reordering within a partition but you'd have your reorient across participants and I'm you know apriori across the data you. So what these nine why is he don't have I reorder but they're everywhere >> But to make it scale and the ethnic partitions very small. So an egg every ordering you are so you can have reorder your crop rotation. So there's whole reorder that you don't really achieve. An ovary worry right I mean you can re bases like this not reordered unless that's okay. Hey this doesn't cases that he'd also been at one Subramanian amounting to really achieve load balancing. And that's the problem to really issue about vascular Joe. Thanks. So you can scale. Yes. Okay yeah. Laughs and little Parkinson's. And I think you're actually gonna have A reordering Right so that's a brow so that that to me that's a defensive way. When do that to me is a huge drawback is that you have to have any partition and you have to scale it out. So the scale is very very injury. Okay so this is let me be honest AND these words. So we have topic. And then we have partitions Take a topper and we partition these little little slices. One consumer PR per partition. But the consumer can have many more than one partition goes to one. Read in all the hops up scaling where you know we had multiple consumers as many as you as you want to be more are getting one topic. But this is that this one will play play play mission is that is that we didn't Isn't this logout you talk about fanout. Fanout break breaks a fair means that you have different types of consumers. So we have consumer pay X Y. So they can write and this is the system monitoring for traffic jams. Monitoring for your sewer trench to bloody husband assistance and they can both do that. I get the of events. Okay so this is called a consumer group consumer group for that so that so that's a fan out. Alright so that's how this is. This is some key things for them. Yeah I always had files around a little bit. Confusing topics partitions. And then you have consumer groups. Alright so can partition be consumed by more than one can soon. And I partition because my whirlwind consumer consumer the different consumer groups would answer yes but within a consumer drew. These partitions consumed by a different person with who say Yes yes or no. The better way to look at is can you have load balancing or how do you give about balancing that Maslow politicians not by having my consumers and consumers small partitions. And that you have loaded. All right. All right so that's that's I can do it. And then there's this idea that they have this offset. So it's a law created data producers are policy and law. So you just continually appending data at the end And then what coffee does it tracks these offsets where is it being read being read you iceberg consumers pull it off for some width one consumer group. And this partition five processes Number five hours. But I ask that because to six it says that as long as you're to the end and you're reading it from some of that course that you process it. And then you crash right and that nasa then offset is not the problem. How do you get a feeling of loss of super about Kafka all thorough backup system. Yes. Yeah. Kafka As you can see the data for as long as one goes. All right so let's see. So then this is a huge area. So many ties of systems to GCP as if there is an event hub. She's like Kafka or Amazon had their own something rather than who you are that which is the thing you have simple queue service. Here it was a huge area. We'll touch on a couple of key ones. And I think the most useful one is the rarest like TCP photo. Yeah so it's hard to see so many solutions. This is massive comparison of Kafka versus half as source. These are the reach of the virus and have their own system. And then all of the best guarantees or guarantees it through but we'll go through. Maybe we can just do a quick Craig left TCP to apply TCP. Udp So for me I like to do it do it. Neither free absolutely free. So yeah so I'll write that video and that whole thing we're going to do and I'll show you that shall we don't use it laughter versus one that's awfulness that gives you the account. When you do this when you run your command prompt is good to check this. I found apartment. Let me do that. This will probably live. So you give your your project ID. Command line always check that because it's very easy to get wrong. Anyway so let's get into it is that the Council and we're going to make up a topic where we throw down eventually you get into that. Our retiree topper how bag. Right because inscription You do it absolutely instructions. Hafez crazy. This description. This this topic is this description. I have the same name product out pull and push. The village leaders. Have always talked about the acknowledgement deadline tension. Grammar. These are the set of the heartbeat one or B would not retain a golf and so so this is going to retain it for seven days no problem. Saturdays hierarchy. Even if I do process it overeating all cost money tell the story about my rice. Now my description is there. Okay. Okay. Yeah. Good. Question. Publisher. Application creates and sends messages to a line. Describe their cases like copper towards the exit polls Jesus Grace Hopper subscriber subscription. This is asynchronous message isn't designed to be scalable and highly reliable scale through pressure. All right great. All right so now let's publish a message. Should be messages All right so that was at the Council. Man why he wanted to do it in Python video shows here. One of the things you have to do is here too fragile servers account he saved to a file. Now you have this is this is the key that will allow you were your post code and describe your policy producers and consumers. You gotta use these branches. Sure I don't have it. Totally finalizing try. The prescriber code evolving in the same way to the other one. The change saves a life. Same as the file name you say name. Financials. As I say the credentials and ended up being my project ID. Capsules that they hear hear driver. The appointment ID. And wouldn't they neighbor prescription No your name. Alright lets go. Crazy Beersheba. We've already created a subscription for the message but there's no messages there. Let me pose another message unfold. So this will just stop here because you create support really building that. We just put it too small. So I may I may make this primer that putting the project ID description name And then for the message. So I bought one on one or more messages me good whole bunches of things. So this GitHub just employing one message and you wanted to method. And then from the AC. And that's that's this scribe publish. It's also online. Same idea for free that the topic make the name of African and polish the message. Goes really easy dark all the stuff. No union is trivial. It's super above main clients. And so yeah very existence is built in. So how I'll put up the code are young GitHub. I'll make a video. Alright so any questions. All right. Okay so next time I guess I'm going to try to do this events. Fast-food thing. Google has very low. Herder Do do do is I guess for generations. So latest greatest. Apache Beam and Google supports that Dataflow little bit more complicated than this. But we're trying to do that. And then you can pretty much that. And then a little bit of we did cover database goodwill with databases that you can build your neither the quarters with the communities and pumps. And image processing can go a lot of very powerful systems. Okay I'll put this all on live birth. Say. Okay. Yeah well everything Oh yeah for now looking for for new stuff. Started another startup. Yeah yeah. As people who are now everybody know Yeah

cpeg473-010-20191107-183000.mp4

From Andrew Novocin November 08, 2019

10 plays 0 comments You unliked the media.

Video Created by UD Capture Classroom Recording in Interdis Science Engr 205 on 2019-11-07 18:30:00.

Appears In: CloudCompAndSec

Comments

Add a comment