This course answers the questions, What is data visualization and What is the power of visualization? It also introduces core concepts such as dataset elements, data warehouses and exploratory querying, and combinations of visual variables for graphic usefulness, as well as the types of statistical graphs, tools that are essential to exploratory data analysis.
Exploratory Querying
Loading...
來自 Arizona State University 的課程
Introduction to Data Exploration and Visualization
6 個評分
從本節課中
Exploratory Querying and Visual Variables Used in Data Exploration and Visualization
與講師見面
Ross MaciejewskAssociate Professor at Arizona State University in the School of Computing, Informatics & Decision Systems Engineering and Director of the Center for Accelerating Operational Efficiency
School of Computing, Informatics & Decision Systems Engineering
K. Selcuk CandanProfessor of Computer Science and Engineering
Director of ASU’s Center for Assured and Scalable Data Engineering (CASCADE)
Well, welcome to the fourth set of slides of
the lecture on Introduction to Data Exploration.
So far, we have defined what we mean by data exploration.
We have discussed data challenges
that are underlying data exploration systems and data exploration tools.
And we have tried to have some sense about what are
the data models and how the data models relate to data exploration.
In this set of slides,
we will focus on the different modalities for data exploration.
So, let's revisit data exploration and let's
revisit the exploratory search that we had defined in earlier lectures.
If you remember, we said in the very first lecture is that exploratory search
involves acquiring new knowledge and revealing new facts using several techniques.
And we defined these techniques as analysis,
that is identifying common patterns or outliers, comparison,
that is being able to compare different data elements
and decide whether they are similar or different, aggregation,
which is taking a data collection and summarizing them
into groups based on the similarity or distance of the data elements,
transformation, which was taking data and
transforming it into a more convenient representation, and visualization.
So, what we will do in this lecture is we will expand on this a little bit more detail.
In particular, we are going to define a set of exploration modalities,
and I will provide examples for each of these specific data exploration modality.
And while they are doing this,
we will see that these will rely on analysis,
comparison, aggregation, transformation, and
visualization techniques that we have just introduced.
Okay, so, let's start with similarity queries or ranked queries,
one of the simple modalities of data exploration.
In similarity queries or ranked queries,
the way the users explore the data is very simple.
Users provides and what we call the query example,
for example query, and the system
returns potential matches to this query from the database.
So here, we have the users,
and on the other side,
we have the database.
The user specifies what he or she is interested in,
and the database identifies potential,
potential is important, potential matches.
Now, if you take a look at this slide,
you will see that the user provided one query,
the user is interested in images that contains Fuji mountain and a lake,
and the system has identified several potential matches to the user's query.
However, that's not the entire story.
If you look at the results more closely,
you will see that for each of the matches,
there are different degrees of matching to the user specification.
For example, in one of the candidate results,
the lake and the mountain have been found however,
the special relationship is not perfectly matched.
In fact in this case, the special relationship
is completely opposite of what the user has asked.
In another one, the special relationship is perfectly matched however,
the mountain and the lake matches are not perfect.
In another application, we have the mountain matched a bit well,
special match is okay, but on the other hand,
one of the conditions which is having a lake in the result is not matched at all.
So, the question then becomes from a data exploration perspective.
Which of these results should be presented to user first?
Because the user specified to some degree what he or she is interested.
And it is the database management systems or
exploration system's responsibility to
first identify how well different subgoals are satisfied,
and then rank these results according to some objective or subjective criteria.
Now this is very important.
This is very important because ranking are based on
some objective criteria is easy. I will discuss that.
We'll discuss what we mean by ranking results based on objective criteria.
The major difficulty is ranking results based on the subjective criteria.
That is yes, we know that the user is interested in Fuji mountain.
Yes, we know that user is interested in a lake,
but we don't know in this example,
how much importance user gives to
either the Fuji mountain condition or the lake condition.
The same way we also don't know how much user give importance to the special condition.
These are important because unless I know
how much important the user gives to these different sub-goals,
it is very difficult for me to create a ranking to support effective data exploration.
Now, let's first focus on the objective criteria.
One way in which
many database management system or data exploration tools
represent the data internally is the vector representation.
And if you remember last lecture,
we talked briefly about vector representation.
We said that the vector representation is a way to
represent non-structural data to support the exploration.
In this light, I'm giving a hint about how vector representation looks like.
In the following lectures,
we'll actually provide a more formal definition of
a vector representation and we'll look at these vector representations in more detail.
But for now, let's assume that we are given a set of images.
Let's assume that from these images,
we have extracted a set of image features.
These features for example could be colors.
This could be amount of blue an image has.
This could be the amount of red the image has.
And this could be amount of green the image has.
Or it could be other features.
For now, it doesn't matter.
For what we have done is that we have taken the set of images in our database,
and now we represented each of these images as a collection of features.
This is what we will call as a vector space representation of our data.
Now given this, if the user provides us with a query,
in this case the user says, "Well,
find in our database or images that are similar to this image."
What we can do is we can use the objective criteria of the user's query,
what we can do is we can go and quantify the amount of blue this image has,
the amount of green the image has,
the amount of red this image has,
and we can map the query in the same space as the original dataset.
Now, this is great,
because we have our database in a vector space,
we transform our database into vectors in a vector space.
Not only that, we have also mapped our query,
the query example that the user provided into the same space.
In this combined space,
now what we can do is we can do what is known as
nearest-neighbor search to identify
other images in the database that have similar composition or features.
Now, we will see that this definition of
similar objects in the database
required us to be able to compare this future compositions.
And we'll see that there are different ways of doing that.
There are different ways of measuring
the similarity or distance between objects in the same space.
But nevertheless, given a set of features,
and given a distance function,
I have an objective way of comparing
a given query to all the data in the database and identifying in this example,
the two most similish images in the database.
Great. So you know how to deal with objective criteria.
The difficulty of course is the subjective criteria because in this example,
I know for each object how much blue,
how much green, and how much red the images contain.
But I do not know how much importance the user gives to the blue color,
how much importance the user gives to the green color,
and how much importance the user gives to the red color.
In this example, if the red color is important
then this shouldn't be a result because it doesn't have any red in it.
On the other hand, if the color is not important,
if the red color is not important maybe this
is a very good match to the given query example.
So, the next is actually a question is how do we go over,
how do we overcome this subjectivity gap?
We also refer to this as semantic gap.
This is a common problem in data exploration,
because when the user comes to explore the database,
the system does not know what the user cares about.
The system does not know
what the user's subjective criteria is relative to the particular exploration task.
In fact in many cases,
the user may not know it ahead of the time.
Remember when we the defined data exploration,
we said that user may not have a precise idea of what he or she would like to find.
In this example, while the user starts his querying,
the user may not know whether the color blue is important for her ahead of the time.
And while doing the exploration,
the user may decide well,
maybe the red color is important for me because I want to
play fair red flowers on my web page.
So, this essentially means that usually,
data exploration requires us to be able to use what is known as
relevance feedback to learn
through user interactions the subjective criteria
that is applicable to a given exploration task.
In this case, for example,
the user provides the query object,
the system provides a set of ranked results.
The user declares which of the results are appropriate,
which of the results are better than the others,
and the system next goes and analyze the user's relevance feedback and learns,
"For this user, looks like the color blue is not important,
but for this other user,
it looks like the color blue is very important.
So, to be able to do that,
the system needs to support what we call the relevance feedback.
And again, we will not get into further details in this particular set of slides,
but we'll talk about relevance feedback in the future as appropriate.
Now, we have discussed so far what we call similarity query,
but this is not the only way data is explored.
Another data exploration tool that often used in databases
or data warehouses is called drill-down/Roll-up data access.
So let's see an example.
In this case, let's assume that we are given a structured data.
In this case, the structured data is a data table which we discussed earlier.
We have a table,
the table has three attributes,
name, age and location.
And we have one,
two, three, four, five,
six rows or six data entries that are described in this name,
age, location table, very simple, very structured.
Now, let's assume that on top of this table we are also given additional meta data.
We did not use the term meta data,
I'm using the term meta data the first time.
Meta data is essentially data and about data or data that is supporting data.
In this case, I have an age meta data described in the form of a hierarchy.
So what this age meta data is doing is enabling me to cluster or to
group each entries into higher level, less precise representations.
In addition to the age meta data in this application,
I also have a location meta data.
Again, location meta data is provided in the form of a hierarchy.
For example, Phoenix is in Arizona,
Arizona in Southwest and Southwest is in US.
We're are talking about Southwest of the US.
The same way Baltimore is in Maryland,
and Maryland is in US.
Note that, essentially, what the meta data is
doing is helping us interpret the results that are in the database.
So, we are given a data table,
we are given an age meta data,
we are given a location meta data.
So, the question is that,
how do we use this drill-down or roll-up operations to explore the data?
Now, let's see an example.
The first example that we will do is roll-up on age.
So roll-up essentially means,
going higher in the hierarchy.
If you take a look at the original table,
the entries in the age column or age attribute,
they're all at the leaf level of the hierarchy,
the lowest level of the hierarchy.
When we roll-up on age,
essentially what we are doing is we are going
one level high in this semantic representation.
In this case, what we have done is we have gone to the age column
and we have replaced all of the entries with the second level representation.
Note that effectively what this is doing is that
this is reducing role-up operation effectively,
reducing the amount of detail we have.
We have more detail in the original data,
we have less detail in the roll-up data.
In this case, three of the rows are all aggregated with the same age representation,
one star, and three of the rows
have all been aggregated with the same representation, two star.
Are we done? Well, no.
Maybe I want, to reduce the data detail a little bit more.
Remember, I'm doing data exploration.
Remember, I'm trying to maybe get these large patterns,
big patterns instead of details in the data.
So, what I can do is I can further roll the data.
In this case, I'm doing a roll-up on location.
So, roll-up on location means,
I'm going one step higher in the location hierarchy for all of the rows in my database.
What happened is that I had one row in Phoenix that got replaced with Arizona.
I lost details.
I lost details, but I learned something.
In the original database,
I didn't know how many rows that were in Maryland, I had no idea,
I couldn't know of that,
but then the new rolled up table I can see that there are
three rows that are all in the state of Maryland.
So, if that is important for me,
now I have more information about that.
So roll-up essentially, gives us the ability to go higher down in the hierarchy.
But that's not the only thing.
So, let's assume that basically I'm exploring the data,
I rolled up on age,
I rolled up on location and I have obtained this table.
What I can do next is I can decide to further drill down on age,
but I can limit my drill-down operation to one star.
So I'm not drilling down in the entire age hierarchy,
I am focusing on the one third entry,
and I am coming down,
I'm adding more details in the one star position of the hierarchy.
Basically, let's assume that I want to get
more information about the teenagers who are using my system.
So, what I want to do is one star was good,
it was telling me before that there were three teenagers who were using my system.
Now, I know that I can get more information,
and I can see that there are two teenagers, 19 year old,
you were on my system, and there is one user,
a 12 age who was accessing my system.
So, drilling down gives me the ability to look at more details,
to add more details where it is needed.
If I roll down, if I drill down, am I done?
No, I can continue exporting my data.
I'm looking at my data,
and I see that's, okay, I have Arizona,
California Maryland, can I roll this up more?
In this case I can say, "Well,
why don't you roll-up location Arizona more,
because I want to see where Arizona is in the US."
In this example, my original table had Arizona in only one entry,
and when I roll-up in Arizona,
I get one entry in southwest.
So I know that one of my users is in the southwest.
So if you can see,
drill-down and roll-up is a tool that I can use
to add either more details to my database or my data set,
or to remove details from my data.
This way, I can either get better more higher level patterns,
I can see more high level patterns in my data,
or I can see more details in my data.
This is a very strong data exploration tool.
But, roll-up and drill-down requires that in addition to the data table,
you also have access to metadata
hierarchies that can support either roll-up or that can support drill-down.
Next, let's take a look at frequent items, sketches or summaries.
In this course, in the set of slides,
we'll mainly focus on summaries.
We'll revisit frequent items and sketches in the future. So what is a summary?
Remember when we were doing roll-up operation,
we were clustering objects together.
Now, when we were clustering objects together,
we were giving specific information about what we want to roll-up,
or what we want to drill-down.
For example, in the last example, we said, "Well,
why don't we take the Arizona entry,
and then why don't we roll-up on Arizona."
So, we gave very specifications about what we want to roll-up,
how we want to cluster.
In the case of generating summaries however,
we are again given possibly a data table and possibly additional metadata,
but instead of being stated ahead of the time,
how we want to roll-up,
how we want to go high in the hierarchy,
we are given a target number of rows.
So, in this example,
what I'm saying is, "Well you know what,
my original data table had one,
two, three, four, five, six, rows.
That's too much data for me.
Why don't you summarize it down to only two rows, can you do that?"
So, I'm user, I tell the system,
"I'm exploring the data.
I need help. Please summarize the data into a smaller set of rows."
Now, in this case the system has,
I didn't tell the system how to summarize.
So this system can do different things.
So, in this example,
we see one summarization of the table.
There are two tuples,
so what the system had decided to do was,
keep this entry intact,
it kept that entry intact as you can see,
and summarized all of the other remaining entries into one single row.
Now, let's take a look into that.
The name, I cannot,
I'm summarizing the data so the different rows have different names,
so I cannot really tell what the name is.
So, I'm going to replace it with a placeholder value, in this case a dash.
For age, as you can see,
I am going up to summarize,
to put all of this in the same row.
I have to put 12, 19,
22, 22, and 27 into one single row.
So, the only way I can do that is if I go all the way up in the age metadata.
Which means that I have to replace the age entry with a star.
So I'm going all the way to the upper level.
What about location? So let's take a look into that.
For the location, once again,
I have to put Phoenix, San Diego,
Baltimore, Frederick, and another Baltimore entry into one single row.
And the only way I can do it is if I roll-up all the way to the US entry.
So for this summary,
I have replaced the entry with US.
Essentially, in this case this row marked with
a blue arrow correspond to five rows in my original table.
This second row might be the orange arrow
correspond to only one entry in my original table. A summary.
So, the summary in this case is created by the system.
I only told the system, "Well,
create a summary to me with two entries."
And the system created it.
Now, as you will see,
this is one summary,
but there can be other summaries as well.
So for example, there's an alternative summary of the same table.
Again, I have two rows,
but in this example,
each row correspond to three tuples.
Instead of one row corresponding to five tuples in my database,
and the other one corresponding to only one tuple, in this case,
I have three rows in one row,
and three rows in the other summarized row.
So, how does the system achieve this?
Well, essentially what the system had to do is for the row marked in blue,
it had to roll-up to one star,
and roll-up to Southwest.
And for the row represented with the orange arrow,
this system had to roll-up to two star in Asia attribute,
and Maryland in the location attribute,
and that was sufficient.
Now, hope with this you can see these two summarizations are not identical.
And in fact, one summerization may be more desirable than the other summarization.
In this case, if I had to choose,
I would probably choose this summarization as a better summarization.
Because instead of having one row with a lot of data in it,
summarize a lot of data and it and another one that summarize enough,
I get the same amount of summarization in both rows, in the output.
So, how do we summarize the data effectively?
It is the data system's responsibility to
find a good summarization to support effective exploration.
We will not discuss this anymore in this class,
but data clustering, data summarization,
is an important challenge.
And in the machine learning lecture that we are covering,
courses that we are covering,
we'll discuss more about how to cluster the data together, more effectively.
What about aggregate queries and iceberg queries, what are they?
So let's basically take a look into them a little bit more carefully.
Let's go back to our last example,
where we had taken our data with six tuples,
and we had summarized them into a summarized table with only two rows in them.
By the way I'm using tuples and rows interchangeably,
so they are the same thing.
In a relational database,
usually the row are called tuples.
So I'm using them interchangeably.
Okay. So, this is a summarized data table.
What I can do is, I can add to
this summarized data table additional aggregation operations.
So for example, I can take a look at my summarized table,
here I see that age is one star,
for the other row age is two star.
But this doesn't tell me much about what is the age distribution inside?
I don't know for example what is the maximum age inside.
It could be 12, it could be 19.
If summarized, I don't know what's inside.
So what aggregation operations are doing,
if given a summary,
given a cluster, or given a grouping of the data,
they enable me to get more information about the data distribution in that cluster,
data distribution in that grouping,
or data distribution in the summary.
In this example, if I asked the system to tell me about the maximum age,
the system is telling me, "Well,
in this grouping, the maximum age was 19.
In this grouping, the maximum age was 27." Look what happened.
I still have summarized data,
but about the summaries,
I have more information.
So this we call aggregation.
What about iceberg queries?
Iceberg queries are one more step further.
In this case, what I had done was,
I told the system, "Well,
please create me a summary,
and for each group give me the maximum age."
Great. But let's assume in this example that I have only two tuples, right?
So but let's assume that somehow I had after doing this,
I want to reduce the data more.
And what impact to get what I want is that,
I want to get the results where the maximum age is greater than 20.
So this is called an iceberg query.
Why is it called iceberg query?
Well, essentially what I'm doing if you think about it, I have a condition.
The condition clusters from the data,
and I want to basically put a constraint on the condition,
and I want to basically find the results where the maximum age is greater than 20.
So it's like creating an iceberg if you sort of think about it.
So in this example,
my iceberg query returns to me only one row,
and that row is the second row of my summary.
The first row of my summary got eliminated.
Why did it get eliminated?
Because my condition is,
maximum age is greater than 20.
And in this example,
there's only one row,
marked with the orange arrow where the maximum age is greater than 20.
So Iceberg queries essentially combine,
summarization, aggregation, and additional filtering.
In one query and that basically potentially
help me reduce the data so that I can more effectively explore the data.
Great. Now finally for this lecture,
let's look at Skyline queries and let's define what
Skyline queries are and how they are used
in decision support and how they are used in data exploration.
Now in summarization we have seen that what we are
doing is we are reducing the data by telling the system we want,
say, five summarized tuples or two summarized tuples.
Skylines as you will see also reduce data.
So our goal is similar.
We are reducing the data, but instead of telling
the system we want these many results in the output,
we give another condition and I will explain what that condition is in just a second.
So let's take a look at another example.
This example is a little bit different and instead of having a table with hierarchies,
we have a table represented in the form of a vector space again.
So in this vector space,
in this case, we have a table of restaurants,
we have a number of restaurants and for each restaurant I have a record of
the average menu price and I have also a record of the rate.
For each restaurant, I know what is the sort of how much I will spend in
a dinner for a person let's say and what is the average rating as of late.
Obviously, the higher the rating the better the restaurant.
I want to go to a highly rated restaurant.
But again obviously the cheaper the price the better.
I don't want to spend too much money.
If I can get a good food at a cheaper price obviously I want that.
Right. So if that is the case,
if I'm given restaurants with the rating information and average price and if
I'm also given the information that
the high rating the better and the cheaper the food the better,
can you reduce the data
such that it only shows me the restaurants that I should consider.
This is called the Skyline Query.
Essentially what this Skyline Query is trying to
do is it tries to eliminate from the datasets
those restaurants that are both expensive and don't serve good food.
So in this example for instance,
this restaurant over here may not be very attractive.
Why? Well, because it has a low rating and it is also quite expensive.
What about this restaurant here.
It's pretty good, because it has a very high rating and it's also quite cheap.
So what the Skyline query is doing is to have me decide where I should go for dinner.
It helps more effectively explore the data by eliminating restaurants that are
not good in terms of their price and rating relationship.
So, by the way,
this is also known as the Maximum Vector Problem and it was coined by
Skylines in a 2001 paper by Börzsönyi, Kossman and Stocker.
But essentially, what this kind of an operation does is it
identifies a subset of
the objects in the database that dominates the other objects,
but are not dominated by each other.
So for example, let's see,
for this object is not in the Skyline. It should be eliminated.
Why? Well, because this restaurant or here is more
expensive and has less average rating than this restaurant over here.
The same way this restaurant that's called restaurant A is more
expensive and has a poorer rating than this restaurant over here.
What we call this is domination.
So what you are saying is that the restaurant is dominated by
other restaurants which means that the restaurant A should be eliminated,
shouldn't even be considered.
On the other hand, let's focus on say a restaurant B over here.
Restaurant B is not dominated by any other restaurant in our dataset.
There is no other restaurant in our database that has a better rating price relationship.
Yes there are some restaurants such as
restaurants C which has a better rating. That is true.
However restaurant C is more expensive than restaurant B.
So in terms of their price rating relationship they are not compatible.
I might decide to go to restaurant B or restaurant C depending
on whether I want to eat better food or whether I want to eat cheaper food.
What about restaurant D over here.
Well if you take a look at restaurant D you will see that it is
very expensive but it stores very good food.
Once again it is not comparable to Restaurant B or
restaurant C. Yes restaurant B and restaurant C are serving cheaper food,
however if I really want to have
an excellent dinner I might decide to go to restaurant D. So,
this is what we mean by Skylines.
Essentially a Skyline means give me the dataset,
give me a decision criteria,
in this case two decision criteria.
And what I will do is I will find the set
of data elements that are dominated by others and I will eliminate them.
This means that my decision now is only limited to a small number of elements.
I need to explore on a small number of elements to be able to decide.
Very powerful tool and is used very
effectively and very often in decision support systems.
Great. Finally, as I
had mentioned earlier there are data sketches as well.
So what I will do next is I will give just a hint of what we mean by a data sketch.
I will not provide the definition.
I will also not provide a very detailed example,
but that you will get some sense about what the data sketch is.
And this is the example that I'm sure that you are
familiar with and or you have seen it as before.
So let's assume that we have a document collection.
From this document collection we can create what is known as a tag cloud or a term cloud.
Essentially what we are doing is we are looking at the keywords that are
occurring in the documents and I'm just counting them.
These counts gives us essentially some indication as to what are the common keywords,
common terms or common tags in the database.
So we can use this term cloud or tag cloud as a sketch of our document collection.
Great. But how can I use this for exploring my data.
Well I can use this for exploring my data for example,
what I can do is I can link the terms in my term cloud to the documents in the database.
For example, in this case I can link the vacation and I
can allow the user to use this vacation term as a filter condition.
So in this case word user says vacation,
I can eliminate from the collection those documents that do not have that term.
Now if the user is adding another condition,
another filter condition such as Japan,
this enables me further eliminate documents from my collection.
So essentially what I am doing is that I'm using
the sketch as a way to explore the data in
my document collection and to potentially reduce the number
of objects or documents that I need to further explore.
A very simple but a very powerful tool.
We call this data sketch and there are different forms of data sketches.
Some of these we've all learned throughout the data exploration course.
See you in the next lecture.
