Pipelines. You heard us mention pipelines quite a bit throughout the course,

so let's take a few minutes to discuss what they are.

Pipelines can be thought of as a conveyor belt in a factory.

Along the line, there are different assembly stations.

These assembly stations are stages.

Depending on what we want to accomplish,

we may have only one stage,

or we may have many stages.

Pipelines work like this.

Documents represented by these squares

enter our pipeline and begin to flow into our first stage.

This stage is called match which we'll be introduced to you very soon.

We set this stage up so that only red and blue squares make it through.

Next, they flow through our pipeline and enter the second stage.

In this example, we'll call this stage project.

We set the stage up to transform our squares into circles.

This is a small representation of the power of the aggregation framework offers.

We can transform our data in almost any way we desire.

We'll be covering project stage in great detail in later lessons.

This stage represents one of the many powerful analysis stages available,

and it is called group.

Here, we have configured the stage to gather all of the documents that are flowing into

it and produce a single document that gives us the ratio of red to blue circles.

We'll cover group in many other powerful data analysis stages later in the course.

There you have it, a high-level overview of what pipelines are.

At the most basic level,

there are a composition of stages from one to

many that we can arrange and configure in almost any way we like.

The aggregation framework drives

many stages to allow us to filter and transform our data.

All we have to do is compose the pipeline.

Some key takeaways to remember.

Pipelines are a composition of stages.

Stages are configurable to produce desired transformations.

Documents flow through stages like parts in an assembly line or water through a pipe.

Finally, with only a few exceptions which we'll cover later,

stages can be arranged in any way we like and as many as we require.

The Concept of Pipelines