Before you can work with data you have to get some. This course will cover the basic ways that data can be obtained. The course will cover obtaining data from the web, from APIs, from databases and from colleagues in various formats. It will also cover the basics of data cleaning and how to make data “tidy”. Tidy data dramatically speed downstream data analysis tasks. The course will also cover the components of a complete data set including raw data, processing instructions, codebooks, and processed data. The course will cover the basics needed for collecting, cleaning, and sharing data.
Reading XML
Loading...
來自 Johns Hopkins University 的課程
Getting and Cleaning Data
5915 個評分
從本節課中
Week 1
In this first week of the course, we look at finding data and reading different file types.
與講師見面
Jeff Leek, PhDAssociate Professor, Biostatistics
Bloomberg School of Public Health
Roger D. Peng, PhDAssociate Professor, Biostatistics
Bloomberg School of Public Health
Brian Caffo, PhDProfessor, Biostatistics
Bloomberg School of Public Health
This lecture is about reading XML.
So if you don't know about XML, it's
Extensible Markup Language, is what it stands for.
So it's frequently used to score structured data.
It's particularly widely used in internet applications.
So you'll see it a lot when you're doing
things like web scraping or trying to get data from
an internet API or trying to download data from say,
open data of websites, say like an open government website.
So, extracting HTML, XML is actually the basis
for most of the web-scraping that you'll see.
There are two components to an XML file.
There's the markup.
That's the labels that give the text structure.
So you can imagine if you just started typing, you
would end up with sort of an unstructured text file.
The markup is the way that you add labels so that the file ends being structured.
And then the content is the actual text that you type in,
in between sort of the labels that give the structure to the text.
So to be a little bit more concrete, we'll talk about tags, elements and attributes.
So tags correspond to the labels, that there are labels that are going to
be applying to particular parts of the text so that it will be structured.
So, there are start tags.
And so, start tags look like this.
So, they start with an open caret on one side and then they
have a, a phrase and then a closed caret on the other side.
So, you'll have that at the beginning of a certain part of the text.
So, that will start them, that text as a, a labelled section called section.
And then it ends with an end tag which looks like this.
It has the same phrase as before, but it
starts with a forward slash inside of the two carets.
And so the empty tags are tags that
correspond to lines where you don't necessarily need
both an open and then a close tag to deliminate a certain part of a text.
You might just want to denote one specific line, say, as a line break.
And so what you do is, you have the same sort of structure.
Only you have the forward slash just over here at the right end.
So elements are specific examples of tags.
So, for example, one element of the XML
markup could be, the tag greeting, with the end
tag greeting over here on the right hand side
and in between it the content, the hello world.
So, attributes are components of the label.
So, you can actually add to the tags components.
And so, for example, you might have an image tag.
So, what you might want to have corresponding to
that image is an attribute that's the source.
That's where the image is actually coming from.
And then an alternate phrase for that tag, and that could be instructor, for example.
So another example here is suppose you have a tag that's
a step tag, and you might have multiple steps, so tag
number step number one might be tagged with number equal 1
and then two and then this is number three step three.
And then it needs to end with an end tag just like it did before.
If you want to know a lot more about XML and
it's probably Probably a good idea before doing a whole bunch
more with this lecture, if you could go and see, the
Wikepedia entry on XML, which is actually quite comprehensive and nice.
So this is an example XML file.
It's a little bit hard to see, but you can see it okay if
you go actually to the web link here at the bottom of the page.
And so the idea here is you have a text file,
and that text file has lots of tags and content in it.
So for example here you have a tag that says food and then
here is a close tag for that tag that says food down here.
So you can see they're indented at the same level within this file.
And then for this particular food element you actually have a name, so there's
the open tag for name and there's the close tag for name, and this particular.
Element is named sort of Belgian Waffle, delicious.
And then there's other tags that correspond
to other components of this particular food element.
And then you go down and there's another
food elements with another names and so forth.
So that's what an xml file look at.
It looks like and I encourage you to go look the
the bigger version of that when you go to that web link.
So you can read the file into R, you can do this
with the XML package so we're loading the XML package all caps XML.
And then you give it the url.
So in this case, we're going to give it the url from
the previous XML file we were looking at on that previous screen.
And then we can use the function XML three parts to parse out that XML file.
So what this does is.
It loads the document into a memory, into a R memory in a way
that you can then parse it and get access to different parts of it.
So, within R, it's still a structured object, so we have to
be able to use different functions to access different parts of that object.
So, the first thing that we want to look at is the root node.
So, the root node you can get with XML root.
And so what that is, is it's sort of the wrapper,
you can think of it as the wrapper for the entire document.
And so if you execute this xml root command, you'll
have access to that particular element to that xml file.
And then if you want to get the name out, you can actually use the xmlname
applied to that node to get the name out, in this case, it's the breakfast menu.
You can also look at the names of the root node.
And so, when you do names of the root node what it's actually
telling you is what all the nested elements within that root node are.
So, the root node wraps the whole document
and the whole document is a breakfast menu.
And then there are five different breakfast items on this
menu, and each one is wrapped within a food element.
So, you get five food elements, if you look at names of the root node.
So,
the next thing that you could use is
actually directly access parts of the XML document.
And you can do it in a little bit in the same that way you access a list in R.
And so, for example, if you want to look at.
You, you have this root node element that we've extracted.
And we can look at the first component of that or the
first element of that and we do it by using double brackets here.
That's how you access the first element, say in a list.
And so, what you get out, actually, is the first food element, like this.
So it contains the food tags, and it also
contains all the information about the first food element.
Then, if you want to keep drilling down into smaller and smaller parts of
the XML document, you can do sub-setting, just like you do with lists.
So first, this exit rootNode and the first bracket one, will again give you this.
Element here with the entire food component and then if you
want to take the first element of that, you'll just repeat the
process again and then you'll be looking at just the name
the belgian waffles because that's the
first subcomponent of the first subcomponent.
You can actually programmatically extract different parts of the file.
So you can do that with the XML Xsupply command,
and so what you do is you pass that a
parsed XML object and then you tell it what function
you like to apply; so in this case XML value.
So what that's going to do is, it's going to
loop through all of the elements of the
root node and get the XML value And
by default this is going to do this recursively.
So, if you apply it in this way,
since rootNode contains the entire document, it's going to
go through and get every single value of
every sing tagged element in the entire document.
And so you just get a bunch of text all
strung together that's all the text that was in that document.
You might be a little bit more specific
and get a specific component of the document.
And you can do that using the XPath language, so this a whole other language
that you have to learn in addition to XML to be able to access the data.
But if you learn just a few of the
components then you'll be a long way along the way.
I'm going to talk a little bit about this, but
there's actually really nice set of lecture notes that are
available here that go into this in a little
bit more depth, if you find yourself a little bit.
Lost after this lecture.
So the first thing that you're going to be looking at is the top
level node of each element, and that you get with a forward slash node.
The node at any level is double slash node.
And then you can actually extract specific nodes with specific attributes.
And we'll talk about how to do that in the next couple of slides.
[SOUND] So the first thing that you're going to want to be able to do is
get the items on the menu and their prices and so the way that
we're going to do that is we're again
going to use this xpathSApply but now we're
going to be a little bit more targeted
about which data we're going to be extracting.
So we're giving you, again, [INAUDIBLE] the rootNode.
That's the entire document, and so then what we're going to be doing
is we're going to be using This xpack element here is //name.
And so, what that's going to do is it's
going to go through and get all of the nodes
that correspond to an element with title name and
then it's going to get the xmlvalue of those nodes.
So, what it's going to do is it's going to take
out basically all of the elements of that XML file that
are tagged with name and so you can see, you end
up with the names of all of the items on the menu.
You can also do the same thing.
You send the exact same rootNode to xpathSApply, only now you
can say I actually just want the nodes that correspond to price.
And again I want their xmlValue.
And so it will plot out the xmel values which the prices of the values and now
we extract from that xmel file the names of the menu items as well as their prices.
To give you an example of, it's a
little more complicated I am actually going to use
a website of the Baltimore Ravens which is an
American football team that is based here on Baltimore.
So, this the homepage of that team on ESPN.
Which is a sports channel in the U.S. and so
what we're going to do is actually look at the source code.
So, if you right click on the page and say view
source what you'll see is a source document that looks like this.
It's actually quite complicated.
That's the source code that actually
demonstrates, that's process to show you the
website that you actually see when you go navigate your browser to that website.
So we're going to actually drill into this website
source code and see if we can extract some information.
So again I'm going to pass the file URL.
This is the URL of that website, if you go back to the previous page.
So now, since I'm parsing an HTML file, instead of an XML
file, I'm going to use HTML tree parse instead of XML tree parse.
Um,the difference is our, it's different enough you want to
use HTML tree parse when you're parsing an HTML file.
And then I'm going to pass the command use internal equals true, so
that I can get all the different nodes inside of that, that file.
So now what I'm going to do is, I'm again going
to use the xpathSApply to programatically extract some components of this document.
So I'm going to start with the whole document again.
And now what I'm going to try to do is, I'm going to
again extract the XML value to the value inside of certain elements.
But I want to find very specific kinds of elements.
So here's what I'm going to do.
I'm going to look for elements.
That are list items li, and that have a particular class.
So they have class equal to score.
And so, what this is going to do is it's
going to go through the entire document, and any time it
sees a tag that is a, a list item,
it's going to check and see if it's class is score.
And if it's class is score it'll return the XML value.
So it turns out if you go back to this website and you look very very
carefully, That you can see for example there are these list items and
the class in this class is equal to the team name and so the
next element that I'm going to be extracting
on this page is actually the team name.
So it's the same thing.
I look for a list item with class equal
to team name and it will extract those team names.
So the way this works is you go to the website and
you find the tag and any attributes that you want to extract,
and then you need to write them into this xpath language to
extract only the data for those specific elements with xpath, as applied.
So, if I do this, I end up with the scores for each of the games and the teams.
So, I've scraped from that website information directly using
This is a very brief introduction to XML.
The xml packaging r and xml in general.
If you go and check out the xml tutorials for the
xml package there's a short one that's really good and a long
one that's extremely extensive and should only be read after reading the
short one so you get an idea of what is going on.
And then there's really outstanding guide to the XML
package that's linked to here as well that will
give you a lot of information about how you
can use XML to programatically extract information from websites.
