Now let's walk through the phases of developing publishing a machine learning model. Think of MLOps as a lifecycle management discipline for machine learning. Its goal is a balanced process for approach to the management of resources, data, code, time, and quality to achieve business objectives in meet regulatory concerns. Some of the concepts from DevOps translate directly to MLOps. When software developers work on a project, they don't all work on the same code at the same time. Instead, they check out the code they intention workout from a code safe, they merge it back with their task is finished. Before the code is returned to the safe, the developer checks that nothing has changed in the main version and then unit test the updates before merging the code back together. The more frequently these changes are merged with the main code, the less chance that is of divergence. This process is called Continuous Integration or CI. In a busy development team, this happens tens of times a day. Another process favored by developers is Continuous Delivery or CD. This is a method for building, testing in releasing software in short cycles. Done this way, the main development code is almost always production ready, it can be released into the light environment at anytime. If it is not done this way, the main code is like a race car with its wheels off and its engine out, you can go fast, but only after is put back together. Continuous delivery can be done either manually or automatically. Continuous integration of source code, unit testing, integration testing in continuous delivery of the software to production, or important processes in machine learning operations too. But there is another important aspect to MLOps, that's right, data. Unlike conventional software that can be relied on to do the same thing every time on the ML model, can go off. By this we mean that its projective power wins as data profile changes, which they inevitably do. So we can build on continuous integration and continuous delivery, and introducing new term Continuous Training or CT. Continuous training is the process of monitoring, measuring, retraining and serving the models. MLOps differs from DevOps in important ways too. Continuous integration is no longer only about testing, invalidating code and components, but also about testing and invalidating data, data schemas and models. It is no longer about a single software package or service, but a system. The ML training pipeline that should automatically deploy another service, the model prediction service. Uniquely, ML is also concerned with automatically monitoring, retraining and serving the models. Another concept that transfers well from software development machine learning is technical debt. Software developers are familiar with time, resources, and quality trade offs. They talk about technical debt, which is the backlog of free work that builds up, because sometimes they have to compromise on quality in order to develop code quickly. They understand that, although there may have been good reasons to do this, they have to go back and fix things later. This is an engineering version of the common saying, putting off until tomorrow what is better than today, there is a price to pay. Machine learning could arguably be considered the high interest credit card of technical debt. These means that developing and deploying in the ML system can be relatively fast and cheap, but maintaining it overtime can be difficult and expensive. The real challenge isn't building an ML model, it is building an integrated an ML system and continuously operating it in production. Just like a high interest credit card, that technical debt with machine learning compounds, it can be incredibly expensive and difficult to pay down. Machine learning systems can be thought of as a special type of software system. So operationally they have all the challenges of software development, plus a few of their own. Some of these include, multi-functional teams, because ML projects will probably have developers in data scientists working on data analysis, model development and experimentation. Much in functional teams can create it all management challenges. Machine learning is experimental in nature. You must constantly try new approaches with the data, the models, and parameter configuration. The challenge is tracking what worked and what didn't in maintaining reproducibility while maximising causerie usability. Another consideration is attaching and the mouse system is more involved than passing other software systems. Because you're validating data, parameters, and code together in a system instead of unit testing methods and functions. ML systems deployment isn't a simple as deploying enough line train, ML Model as production service. ML systems can require you to deploy a multistep pipeline to automatically retrain and deploy models. And finally, concerns with concept drift and consequent model decay should be addressed. Data profiles constantly change, if something changes in the data input, the projective power of the modeling production will likely change with it. Therefore you need to track summary statistics of the data and monitor the online performance of your model to send notifications, or rollback with values deviate from your expectations. Technical batch builds up in the ML system for many reasons. So we'll be looking at ways to mitigate that throughout this course.
