Now, let's walk through the phases of developing and publishing a machine learning model. Think of MLOps as a lifecycle management discipline for machine learning. Its goal is a balanced, processful approach to the management of resources, data, code, time, and quality to achieve business objectives and meet regulatory concerns. Some of the concepts from DevOps translate directly to MLOps. When software developers work on a project, they don't all work on the same code at the same time. Instead, they check out the code they intend to work on from a code safe, then merge it back when their task is finished. Before the code is returned to the safe, the developer checks that nothing has changed in the main version, and then unit test their updates before merging the code back together. The more frequently these changes are merged with the main code, the less chance there is of divergence. This process is called Continuous Integration, or CI. In a busy development team, this happens tens of times a day. Another process favored by developers is Continuous Delivery, or CD. This is a method for building, testing, and releasing software in short cycles. Done this way, the main development code is almost always production-ready, and can be released into the live environment at any time. If it is not done this way, the main code is like a race car with its wheels off and its engine out, you can go fast, but only after it's put back together. Continuous delivery can be done either manually or automatically. Continuous integration of source code, unit testing, integration testing, and continuous delivery of the software to production, are important processes in Machine Learning Operations too. But there is another important aspect to MLOps. That's right, data. Unlike conventional software that can be relied on to do the same thing every time, an ML model can go 'off.' By this we mean that its predictive power wanes as data profile changes, which they inevitably do. So we can build on continuous integration and continuous delivery, and introduce a new term, Continuous Training, or CT. Continuous training is the process of monitoring, measuring, retraining, and serving the models. MLOps differs from DevOps in important ways too. Continuous integration is no longer only about testing and validating code and components, but also about testing and validating data, data schemas, and models. It is no longer about a single software package or service, but a system - the ML training pipeline - that should automatically deploy another service: the model prediction service. Uniquely, ML is also concerned with automatically monitoring, retraining, and serving the models. Another concept that transfers well from software development to machine learning is 'technical debt.' Software developers are familiar with time, resources, and quality trade-offs. They talk about technical debt, which is the backlog of re-work that builds up because sometimes they have to compromise on quality in order to develop code quickly. They understand that, although there may have been good reasons to do this, they have to go back and fix things later. This is an engineering version of the common saying, 'putting off until tomorrow what is better done today.' There is a price to pay. Machine learning could arguably be considered 'the high-interest credit card of technical debt.' This means that developing and deploying an ML system can be relatively fast and cheap, but maintaining it over time can be difficult and expensive. The real challenge isn't building an ML model; it is building an integrated ML system and continuously operating it in production. Just like a high-interest credit card, the technical debt with machine learning compounds and it can be incredibly expensive and difficult to pay down. Machine learning systems can be thought of as a special type of software system. So operationally they have all the challenges of software development, plus a few of their own. Some of these include: multi-functional teams, because ML projects will probably have developers and data scientists working on data analysis, model development, and experimentation. Multi-functional teams can create their own management challenges. Machine learning is experimental in nature. You must constantly try new approaches with the data, the models, and parameter configuration. The challenge is tracking what worked and what didn't and maintaining reproducibility while maximising code reusability. Another consideration is that testing an ML system is more involved than testing other software systems, because you're validating data, parameters, and code together in a system instead of unit-testing methods and functions. ML systems deployment isn't as simple as deploying an offline-trained ML model as a prediction service. ML systems can require you to deploy a multi-step pipeline to automatically retrain and deploy models. And finally, concerns with concept drift and consequent model decay should be addressed. Data profiles constantly change. If something changes in the data input, the predictive power of the model in production will likely change with it. Therefore, you need to track summary statistics of the data and monitor the online performance of your model to send notifications, or roll back when values deviate from your expectations. Technical debt builds up in the ML system for many reasons, so we'll be looking at ways to mitigate that throughout this course.
