The term ‘MLOps’ is appearing more and more. Many from a traditional DevOps background might wonder why this isn’t just called ‘DevOps’. In this article we’ll explain why MLOps is so different from mainstream DevOps and see why it poses new challenges for the industry.

We'll see that the key differences between DevOps and MLOps come from how machine learning uses data. We'll see that the need to handle data volume, transformation and quality affects the whole MLOps lifecycle.

“MLOps”一词越来越多地出现。许多有传统DevOps背景的人可能会问，为什么这不这是叫做“DevOps”。在本文中，我们将解释为什么MLOps与主流的DevOps如此不同，并了解为什么它会给行业带来新的挑战。

我们将看到DevOps和MLOps之间的关键区别来自于机器学习如何使用数据。我们将看到处理数据量、转换和质量的需求会影响整个MLOps生命周期。

DevOps is a well-established set of practices to ensure smooth build-deploy-monitor cycles. It is based around CI/CD and infrastructure. The space of tools includes git, Jenkins, Jira, docker, kubernetes etc.:

DevOps是一套完善的实践，可以确保构建-部署-监控周期的顺利进行。它基于CI/CD和基础设施。工具的空间包括git、Jenkins、Jira、docker、kubernetes等：

MLOps has not achieved the same level of maturity. As much as 87% of machine learning projects never go live.

ML infrastructure is complex and workflows extend beyond production of artifacts to include data collection, prep and validation. The types of hardware resources involved can be specialised (e.g. GPUs) and require management. The data flowing through the model and the quality of predictions can also require monitoring, resulting in a complex MLOps landscape:

MLOps 尚未达到相同的成熟度。 多达 87% 的机器学习项目从未上线。ML基础设施复杂，工作流程超出工件的生产范围，包括数据收集、准备和验证。所涉及的硬件资源的类型可以专门化(例如gpu)并且需要管理。在模型中流动的数据和预测的质量也可能需要监控，从而导致复杂的MLOps场景:

The driver behind all these differences can be found in what machine learning is and how it is practised. Software performs actions in response to inputs and in this ML and mainstream programming are alike. But the way actions are codified differs greatly.

Traditional software codifies actions as explicit rules. The simplest programming examples tend to be ‘hello world’ programs that simply codify that a program should output ‘hello world’. Further control structures can then be added to add more complex ways to perform actions in response to inputs. As we add more control structures, we learn more of the programming language. This rule-based input-output pattern is easy to understand in relation to older terminal systems where inputs are all via the keyboard and outputs are almost all text. But it also true of most of the software we interact with, though the types of inputs and outputs can be very diverse and complex.

所有这些差异背后的驱动因素可以从什么是机器学习以及如何进行机器学习中找到。软件根据输入执行操作，在这种ML和主流编程中是相似的。但行为的编纂方式却大不相同。

传统的软件将行为编成明确的规则。最简单的编程例子往往是' hello world '程序，它只是简单地编写一个程序应该输出' hello world '。然后可以添加进一步的控制结构，以添加更复杂的方式来执行响应输入的操作。随着我们添加更多的控制结构，我们会学到更多的编程语言。与旧的终端系统相比，这种基于规则的输入输出模式很容易理解，旧的终端系统的输入都是通过键盘，输出几乎都是文本。尽管输入和输出的类型可能是非常多样化和复杂的，但我们所接触的大多数软件也是如此。

ML does not codify explicitly. Instead rules are indirectly set by capturing patterns from data. This makes ML more suitable for a more focused type of problem that can be treated numerically. For example, predicting salary from data points/features such as experience, education, location etc. This is a case of a regression problem, where the aim is to predict the value of a variable (salary) from the values of other variables by use of previous data. Machine learning is also used for classification problems, where instead of predicting a value for a variable, instead the model outputs a probability that a data point falls into a particular class. Example classification problems are:

Given hand-written samples for numbers, predict which number is which.

Classify images of objects according to category e.g. types of flowers

We don’t need to understand all the details here of how ML is done. However, it will help to have a picture of how ML models are trained. So let’s consider at a high level what is involved in a regression problem such as predicting salary from data for experience, education, location etc. This can be addressed by programmatically drawing a line through the data points:

The line is embodied in an equation:

ML没有明确地进行编码。相反，规则是通过从数据中捕获模式来间接设置的。这使得ML更适合于可以用数值方法处理的更集中类型的问题。例如，根据经验、教育程度、工作地点等数据点/特征预测薪水。这是一个回归问题的例子，其目的是通过使用以前的数据从其他变量的值预测一个变量(工资)的值。机器学习也被用于分类问题，在这里，模型输出的不是预测变量的值，而是数据点属于特定类别的概率。分类问题的例子有:

给出手写的数字样本，预测哪个数字是哪个。

根据类别对物体的图像进行分类，例如花的种类。

这里我们不需要了解ML是如何完成的所有细节。然而，它将有助于了解ML模型是如何训练的。所以，让我们从更高的层次来考虑回归问题中涉及到什么，比如根据经验、教育程度、地点等数据来预测工资。这可以通过编程在数据点上画一条线来解决:

这条线体现在一个方程中:

The coefficients/weights get set to initial values (e.g. at random). The equation can then be used on the training data set to make predictions. In the first run the predictions are likely to be poor. Exactly how poor can be measured in the error, which is the sum of the distances of all the output variable (e.g. salary) samples from the prediction line. We can then update the weights to try to reduce the error and repeat the process of making new predictions and updating the weights. This process is called 'fitting' or 'training' and the end result is a set of weights that can be used to make predictions.

So the basic picture centres on running training iterations to update weights to progessively improve predictions. This helps to reveal how ML is different from traditional programming. The key points to take away from this from a DevOps perspective are:

The training data and the code together drive fitting.

The closest thing to an executable is a trained/weighted model. These vary by ML toolkit (tensorflow, sc-kit learn, R, h2o, etc.) and model type.

Retraining can be necessary. For example, if your model is making predictions for data that varies a lot by season, such as predictions for how many items of types of clothing will sell in a month. In that case training on data from summer may give good predictions in summer but will not give good predictions in winter.

Data volumes can be large and training can take a long time.

The data scientist’s working process is exploratory and visualisations can be an important part of it.

This leads to different workflows for traditional programming and ML development.

系数/权重被设置为初始值(例如随机)。然后，可以在训练数据集上使用该方程来进行预测。第一轮的预测可能会很糟糕。误差到底有多差可以衡量，误差是所有输出变量(如工资)样本与预测线之间距离的总和。然后，我们可以更新权值来尝试减少错误，并重复进行新的预测和更新权值的过程。这个过程被称为“拟合”或“训练”，最终结果是一组可以用来进行预测的权重。

因此，基本图像的中心是运行训练迭代，以更新权值来逐步改善预测。这有助于揭示ML与传统编程的区别。从DevOps的角度来看，要点如下:

(1)、训练数据和代码一起驱动拟合。

(2)、最接近可执行文件的是经过训练的/加权模型。这些根据ML工具包(tensorflow, sc-kit learn, R, h2o等)和模型类型而有所不同。

(3)、再训练可能是必要的。例如，如果您的模型正在预测随季节变化很大的数据，例如预测一个月内将销售多少种服装。在这种情况下，利用夏天的数据进行训练可能会在夏天给出很好的预测，但在冬天却不会给出很好的预测。

(4)、数据量可能很大，训练可能需要很长时间。

(5)、数据科学家的工作过程是探索性的，可视化可以是其中重要的一部分。

这导致了传统编程和ML开发的不同工作流。

Consider a traditional programming a workflow:

User Story

Write code

Submit PR

Tests run automatically

Review and merge

New version builds

Built executable deployed to environment

Further tests

Promote to next environment

More tests etc.

PROD

Monitor - stacktraces or error codes

考虑一个传统的编程工作流:

(1)、用户故事

(2)、编写代码

(3)、提交PR

(4)、自动运行测试

(5)、审查和合并

(6)、新版本的构建

(7)、构建可执行文件部署到环境中

(8)、进一步的测试

(9)、提升到下一个环境

(10)、更多的测试等。

(11)、生产线

(12)、监视器-堆栈跟踪或错误代码

The trigger for a build is a code change in git. The packaging for an executable is normally docker.

With machine learning the driver for a build might be a code change. Or it might be new data. The data likely won’t be in git due to its size.

Tests on ML are not likely to be a simple pass/fail since you’re looking for quantifiable performance. One might choose to express performance numerically with an error level. The level can vary a lot by business context. For example, consider a model that predicts a likelihood of a financial transaction being fraudulent. Then there may be little risk in predicting good transactions as fraudulent so long as the customer is not impacted directly (there may be a manual follow-up). But predicting bad transactions as good could be very high risk.

触发构建的是git中的代码更改。可执行文件的包装通常是docker。

通过机器学习，构建的驱动程序可能是代码更改。也可能是新的数据。数据可能不会在git中，因为它的大小。

ML上的测试不太可能是简单的通过/失败，因为您正在寻找可量化的性能。人们可能会选择用错误级别以数字方式表达性能。该级别可能因业务环境而有很大差异。例如，考虑一个预测金融交易欺诈可能性的模型。然后，只要客户不直接受到影响（可能需要手动跟进），将良好交易预测为欺诈的风险就很小。但将坏交易预测为好交易可能会带来很高的风险。

The ML workflow can also differ depending on whether the model can learn while it is being used (online learning) or if the training takes place separately from making live predictions (offline learning). For simplicity let’s assume the training takes place separately.

A high-level MLOps workflow could look like:

Data inputs and outputs. Preprocessed. Large.

Data scientist tries stuff locally with a slice of data.

Data scientist tries with more data as long-running experiments.

Collaboration - often in jupyter notebooks & git

Model may be pickled/serialized

Integrate into a running app e.g. add REST API (serving)

Integration test with app.

Rollout and monitor performance metrics

ML 工作流程也可能有所不同，具体取决于模型是否可以在使用时学习（在线学习），或者训练是否与实时预测分开进行（离线学习）。为了简单起见，让我们假设训练是单独分开进行的。

一个高级的MLOps工作流程应该是这样的:

(1)、数据输入和输出。预处理。大。

(2)、数据科学家用一片数据在本地尝试一些东西。

(3)、数据科学家尝试使用更多的数据作为长期运行的实验。

(4)、协作-经常在jupyter notebooks和git中

(5)、模型可以被pickle /序列化

(6)、集成到一个正在运行的应用中，例如添加REST API(服务)

(7)、与应用程序集成测试。

(8)、展示和监视性能指标

The monitoring for performance metrics part can be particularly challenging and may involve business decisions. Let’s say we have a model being used in an online store and we’ve produced a new version. In these cases it is common to check the performance of the new version by performing an A/B test. This means that a percentage of live traffic is given to the existing model (A) and a percentage to the new model (B). Let’s say that over the period of the A/B test we find that B leads to more conversions/purchases. But what if it also correlates with more negative reviews or more users leaving the site entirely or is just slower to respond to requests? A business decision may be needed.

The role of MLOps is to support the whole flow of training, serving, rollout and monitoring. Let's better understand the differences from mainstream DevOps by looking at some MLOps practices and tools for each stage of this flow.

性能指标的监视部分可能特别具有挑战性，并且可能涉及业务决策。假设我们有一个模型正在网上商店中使用，并且我们已经制作了一个新版本。在这些情况下，通常通过执行A/B测试来检查新版本的性能。这意味着现有模型(A)和新模型(B)分别获得一定比例的实时流量。假设在A /B测试期间，我们发现B能够带来更多的转化率/购买。但如果这也与更多的负面评论有关，或者更多的用户完全离开网站，或者只是响应请求较慢有关，该怎么办？可能需要一个业务商业决策。

MLOps的作用是支持训练、服务、展示和监控的整个流程。让我们通过查看这个流程每个阶段的一些MLOps实践和工具，让我们更好地理解它与主流DevOps的区别。

Initially training jobs can be run on the data scientist’s local machine. But as the size of the dataset or processing grows then a tool will be needed that can leverage specialised cloud hardware, parallelize steps and allow long-running jobs to run unattended. One tool for this is kubeflow pipelines:

最初的训练工作可以在数据科学家的本地机器上运行。但随着数据集或处理规模的增长，就需要一种工具来利用专门的云硬件，并行化步骤，并允许长时间运行的作业在无人看管的情况下运行。其中一个工具就是kubeflow管道:

Steps can be broken out as reusable operations and run in parallel. This helps address needs for steps to split the data into segments and apply cleaning and pre-processing on the data. The UI allows for inspecting the progress of steps. Runs can be given different parameters and executed in parallel. This allows data scientists to experiment with different parameters and see which result in a better model. Similar functionality is provided by MLFlow experiments, polyaxon and others.

Many training platforms can be hooked up with Continuous Integration. For example, a training run could be triggered on a commit to git and the model could be pushed to make live predictions. But deciding whether a model is good for live use can involve a mixture of factors. It might be that the main factors can be tested adequately at the training stage (e.g. model accuracy on test data). Or it might be that only initial checks are done at the training stage and the new version is only cautiously rolled out for live predictions. We’ll look at rollout and monitoring later - first we should understand what live predictions can mean.

步骤可以分解为可重复使用的操作并并行运行。这有助于解决将数据分割为多个段并对数据应用清理和预处理的步骤的需求。UI允许检查步骤的进度。运行可以被赋予不同的参数并并行执行。这使得数据科学家可以用不同的参数进行实验，看看哪个模型更好。MLFlow实验、polyaxon等也提供了类似的功能。

许多训练平台都可以与持续集成相连接。例如，可以在提交到git时触发一个训练运行，并且可以推动模型进行实时预测。但是，决定一个模型是否适合实际使用需要考虑多种因素。可能主要因素可以在训练阶段进行充分的测试(例如测试数据的模型准确性)。或者可能是在训练阶段只进行了初步的检查，新版本只是谨慎地推出实时预测。稍后我们将讨论部署和监控——首先我们应该理解实时预测的含义。

For some models the predictions are made on a file of data points or a new file each week. This kind of scenario is called offline predictions. In other cases predictions need to be made on demand. For this live use-cases typically the model is made available to respond to HTTP requests. This is called real-time serving.

One approach to serving is to package a model by serializing it as a python pickle file and hosting that for the serving solution to load it. For example, this is serving manifest for kubernetes using the Seldon serving solution (a tool on which I work):

apiVersion: machinelearning.seldon.io/v1alpha2

kind: SeldonDeployment

metadata:

  name: sklearn

spec:

  name: iris

  predictors:

  - graph:

      children: []

      implementation: SKLEARN_SERVER

      modelUri: gs://seldon-models/sklearn/iris

      name: classifier

    name: default

    replicas: 1

对于一些模型，预测是每周对数据点文件或新文件进行的。这种情况称为离线预测。在其他情况下，需要按需进行预测。对于这个实时用例，通常可以使用模型来响应HTTP请求。这被称为实时服务。

提一种提供服务的方法是通过将模型序列化为 python pickle 文件并将其托管以供服务解决方案加载来打包模型。例如，这是使用Seldon服务解决方案(我使用的一个工具)为kubernetes服务的清单:

apiVersion: machinelearning.seldon.io/v1alpha2

kind: SeldonDeployment

metadata:

  name: sklearn

spec:

  name: iris

  predictors:

  - graph:

      children: []

      implementation: SKLEARN_SERVER

      modelUri: gs://seldon-models/sklearn/iris

      name: classifier

    name: default

    replicas: 1

The ‘SeldonDeployment’ is a kubernetes custom resource. Within that resource it needs to be specified which toolkit was used to build the model (here sci-kit learn) and where to obtain the model (in this case a google storage bucket).

Some serving solutions also cater for the model to be baked into a docker image but python pickles are common as a convenient option for data scientists. Submitting the serving resource to kubernetes will make an HTTP endpoint available that can be called to get predictions. Often the serving solution will automatically apply any needed routing/gateway configuration needed, so that data scientists don’t have to do so manually.

“SeldonDeployment”是kubernetes的一个自定义资源。在该资源中，需要指定使用哪个工具箱来构建模型(在这里scikitlearn)以及从哪里获取模型(在本例中是谷歌的storage bucket)。

一些服务解决方案也可以将模型放入docker映像中，但python pickle通常是数据科学家的方便选择。将服务资源提交给kubernetes将使一个HTTP端点可用，可以调用该端点来获得预测。通常，服务解决方案会自动应用所需的任何路由/网关配置，因此数据科学家不必手动这样做。

Self-service for data scientists can also be important for rollout. This can need careful handling because the model has been trained on a particular slice of data and that data might turn out to differ from live. The key strategies used to reduce the risk of this are:

1) Canary rollouts

With a canary rollout a percentage of the live traffic is routed to the new model while most of the traffic goes to the existing version. This is run for a short period of time as a check before switching all traffic to the new model.

2) A/B Test

With an A/B test the traffic is split between two versions of a model for a longer period of time. The test may run until a sufficient sample size is obtained to compare metrics for the two models. For some serving solutions (e.g. Seldon, KFServing) the traffic-splitting part of this can be handled by setting percentage values in the serving resource/descriptor. Again, this is to enable data scientists to set this without getting into the details of traffic-routing or having to make a request to DevOps.

3) Shadowing

With shadowing all traffic is sent to both existing and new versions of the model. Only the existing/live version of the model’s predictions are returned as responses to live requests. The non-live model’s predictions are not returned and instead are just tracked to see how well it is performing.

数据科学家的自助式服务对于推广也很重要。这可能需要谨慎处理，因为模型是针对特定的数据片进行训练的，而该数据可能与实际数据有所不同。降低这种风险的关键策略有:

1)、Canary的发布

随着canary的推出，一部分实时流量被流向到新模型，而大部分流量则被流向到现有版本。在将所有流量切换到新模型之前，这将运行一小段时间作为检查。

2) 、A / B测试

通过A/B测试，流量在一个模型的两个版本之间分配更长的时间。测试可能会一直运行，直到获得足够的样本大小来比较两个模型的指标。对于某些服务解决方案(例如Seldon, KFServing)，流量分流部分可以通过在服务资源/描述符中设置百分比值来处理。同样，这是为了让数据科学家能够在不涉及流量路由细节或向DevOps提出请求的情况下进行设置。

3)、跟踪

通过跟踪，所有流量都被发送到模型的现有版本和新版本。只有模型预测的现有/实时版本将作为实时请求的响应返回。非实时模型的预测不会被返回，而是被跟踪以查看其表现如何。

Deciding between different versions of a model naturally requires monitoring.

With mainstream web apps it is common to monitor requests to pick up on any HTTP error codes or an increase in latency. With machine learning the monitoring can need to go much deeper into domain-specific metrics. For example, for a model making recommendations on a website it can be important to track metrics such as how often a customer makes a purchase vs chooses not to make a purchase or goes to another page vs leaves the site.

It can also be important to monitor the data points in the requests to see whether they are approximately in line with the data that the model was trained on. If a particular data point is radically different from any in the training set then the quality of prediction for that data point could be poor.

It is termed an ‘outlier’ and in cases where poor predictions carry high risk then it can be valuable to monitor for outliers.

If a large number of data points differ radically from the training data then the model risks giving poor predictions across the board - this is termed ‘concept drift’.

Monitoring for these can be advanced as the boundaries for outliers or drift may take some experimenting to decide upon.

在模型的不同版本之间做出决定自然需要监控。

在主流的web应用程序中，监控请求以获取任何 HTTP 错误代码或延迟增加是很常见的。通过机器学习，监控可能需要更深入地研究特定领域的指标。例如，对于一个在网站上做推荐的模型来说，跟踪一些指标是很重要的，比如客户购买的频率，选择不购买的频率，或者访问另一个页面的频率，离开网站的频率。

监控请求中的数据点也很重要，以查看它们是否与模型所训练的数据大致一致。如果一个特定的数据点与训练集中的任何数据点完全不同，那么该数据点的预测质量可能很差。

它被称为“异常值”，在糟糕的预测带有高风险的情况下，监测异常值是有价值的。

如果大量的数据点与训练数据完全不同，那么模型就有可能在所有方面给出糟糕的预测——这被称为“概念漂移”。

由于异常值或漂移的边界可能需要一些实验来确定，因此可以进一步监测这些异常值。

For metrics that can be monitored in real-time it may be sufficient to expose dashboards with a tool such as grafana. However, sometimes the information that reveals whether a prediction was good or not is only available much later. For example, there may be a customer account opening process that flags a customer as risky. This could lead to a human investigation and only later will it be decided whether the customer was risky or not. For this reason it can be important to log the entire request and the prediction and also store the final decision. Then offline analysis run over a longer period can provide a wider view of how well the model is performing.

Support for custom metrics, request logging and advanced monitoring varies across serving solutions. In some cases a serving solution comes with out of the box integrations (e.g. Seldon) and in other cases the necessary infrastructure may have to be setup and configured separately.

对于可以实时监控的指标，使用诸如grafana之类的工具公开仪表板可能就足够了。然而，有时揭示预测好坏的信息要很久以后才能得到。可能存在将客户标记为有风险的客户开户流程。这可能会导致人工调查，只有稍后才能确定客户是否有风险。由于这个原因，记录整个请求和预测并存储最终决策可能很重要。然后，在更长的时间内运行离线分析可以更广泛地了解模型的执行情况。

对自定义指标、请求日志记录和高级监控的支持因服务解决方案而异。在某些情况下，提供服务的解决方案带有开箱即用的集成(例如Seldon)，而在其他情况下，可能必须单独设置和配置必要的基础设施。

If something goes wrong with running software then we need to be able to recreate the circumstances of the failure. With mainstream applications this means tracking which code version was running (docker image), which code commit produced it and the state of the system at the time. That enables a developer to recreate that execution path in the source code. This is reproducibility.

Achieving reproducibility for machine learning involves much more. It involves knowing what data was sent in (full request logging), which version of the model was running (likely a python pickle), what source code was used to build it, what parameters were set on the training run and what data was used for training. The data part can be particularly challenging as this means retaining the data from every training run that goes to live and in a form that can be used to recreate models. So any transformations on the training data would need to be tracked and reproducible.

如果软件运行出现问题，那么我们需要能够重新创建失败的情况。对于主流应用程序，这意味着跟踪正在运行的代码版本(docker 镜像)，哪个代码提交产生了它，以及当时的系统状态。这使开发人员能够在源代码中重新创建该执行路径。这是再现性。

实现机器学习的可再现性涉及更多。它涉及到知道发送了什么数据(完整的请求日志记录)，模型的哪个版本正在运行(可能是python pickle)，使用什么源代码来构建它，在训练运行中设置了什么参数，以及在训练中使用了什么数据。数据部分可能特别具有挑战性，因为这意味着要保留每一次运行的训练的数据，并且以一种可以用于重建模型的形式保存这些数据。因此，训练数据上的任何转换都需要被跟踪和重现。

The tool scene for tracking across the ML lifecycle is currently dynamic. There are tools such as ModelDB, kubeflow metadata, pachyderm and Data Version Control (DVC), among others. As yet few standards have emerged as to what to track and how to track it. Typically platforms currently just integrate to a particular chosen tool or leave it to the users of the platform to build any tracking they need into their own code.

There are also wider governance challenges for ML concerning bias and ethics. Without care models might end up being trained using data-points that a human would consider unethical to use in decision-making. For instance, a loan approval system might be trained on historic loan repayment data. Without a conscious decision about which data points are to be used, it might end up making decisions based on Race or Gender.

Given concerns about bias, some organisations are putting an emphasis on being able to explain why a model made the prediction that it did in a given circumstance. This goes beyond reproducibility as being able to explain why a prediction was made can be a data science problem in itself ('explainability'). Some types of models such as neural networks are being referred to as ‘black box’ as it is not easy to see why a prediction would come about from inspecting their internal structure. There are black-box explanation techniques emerging (such as Seldon's Alibi library) but for now many organisations for whom explainability is a key concern are currently sticking to white box modelling techniques.

用于跨ML生命周期跟踪的工具场景目前是动态的。有诸如ModelDB、kubeflow元数据、pachyderm和数据版本控制(DVC)等工具。迄今为止，关于跟踪什么以及如何跟踪的标准还很少。通常，目前的平台只是集成到特定选择的工具，或者让平台用户将他们需要的任何跟踪构建到他们自己的代码中。

ML在偏见和道德方面也面临更广泛的治理挑战。如果没有护理模型，最终可能会使用人类认为在决策中使用不道德的数据点进行训练。例如，可以根据历史贷款还款数据训练贷款审批系统。如果没有关于要使用哪些数据点的有意识的决定，它可能最终会根据种族或性别做出决定。

考虑到对偏见的担忧，一些组织正在强调能够解释为什么模型会在给定情况下做出预测。这超越了可重复性，因为能够解释为什么做出预测本身就是一个数据科学问题(“可解释性”)。神经网络等一些类型的模型被称为“黑匣子”，因为很难通过检查它们的内部结构来了解为什么会产生预测。虽然出现了一些黑箱解释技术(例如 Seldon 的 Alibi 库)，但目前许多将可解释性视为关键问题的组织目前都坚持使用白盒建模技术。

MLOps is an emerging area. MLOps practices are distinct from mainstream DevOps because the ML development lifecycle and artifacts are different. Machine learning works by using patterns from training data - this makes the whole MLOps workflow sensitive to data changes, volumes and quality.

There are a wide range of MLOps tools available but most are young and compared with mainstream DevOps the tools may not yet interoperate very well. There are some initiatives towards standardisation but currently the landscape is quite splintered with big commercial players (including major cloud providers) each focusing primarily on their own end-to-end ML platform offering. Large organisations are having to choose whether an end-to-end offering meets their machine learning platform needs or if they instead want to assemble a platform themselves from individual (likely open source) tools.

MLOps是一个新兴的领域。MLOps 实践与主流 DevOps 不同，因为 ML 开发生命周期和工件是不同的。机器学习通过使用训练数据中的模式来工作——这使得整个MLOps工作流对数据的变化、数量和质量都很敏感。

有多种MLOps工具可用，但大多数都很年轻，与主流的DevOps相比，这些工具可能还不能很好地互操作。有一些标准化举措，但目前的格局与大型商业参与者（包括主要云提供商）相当分裂，每个参与者主要专注于自己的端到端 ML 平台产品。大型组织不得不选择端到端的产品是满足他们的机器学习平台需求，或者他们想用单独的（可能是开源的）工具自己组装一个平台。