This course (The English copy of "用Python玩转数据" <https://www.coursera.org/learn/hipython/home/welcome>) is mainly for non-computer majors. It starts with the basic syntax of Python, to how to acquire data in Python locally and from network, to how to present data, then to how to conduct basic and advanced statistic analysis and visualization of data, and finally to how to design a simple GUI to present and process data, advancing level by level. This course, as a whole, based on Finance data and through the establishment of popular cases one after another, enables learners to more vividly feel the simplicity, elegance, and robustness of Python. Also, it discusses the fast, convenient and efficient data processing capacity of Python in humanities and social sciences fields like literature, sociology and journalism and science and engineering fields like mathematics and biology, in addition to business fields. Similarly, it may also be flexibly applied into other fields. The course has been updated. Updates in the new version are : 1) the whole course has moved from Python 2.x to Python 3.x 2) Added manual webpage fetching and parsing. Web API is also added. 3) Improve the content order and enrich details of some content especially for some practice projects. Note: videos are in Chinese (Simplified) with English subtitles. All other materials are in English.
2 Network Data Retrieval
Loading...
來自 Nanjing University 的課程
Data Processing Using Python
107 個評分
從本節課中
Data Acquisition and Presentation
Welcome to learn Module 02 “Data Acquisition and Presentation”! After learning this module, you can master the modes of acquiring local data and network data in Python and use the basic and yet very powerful data structure sequence, string, list and tuple in Python to fast and effectively present data and simply process data.
與講師見面
ZHANG Liassociate professor
Department of Computer Science
In addition to local data for utilization,
there are large quantities of data generated over the Internet every day,
inside which there exist many contents with great values.
You might wonder,
whether it is troublesome to acquire those network data.
Indeed, nope,
Python provides many functions
to conveniently acquire network data.
However, contents on some web pages may be dynamically generated,
say, dynamically generated by "js".
At this time, the source codes of web page
do not correspond to the contents displayed on the web pages,
like the data and information of Dow Jones Industrial Average stocks on Yahoo Finance.
Data on this website often change.
In Python, how can we acquire network data?
Acquisition of network data is also known as "crawl",
divided into two stage.
Stage One is to scrape
and Stage Two is to parse.
For scraping, we used to adopt
such a built module "urllib",
especially its "request" module,
which can conveniently scrape web page contents.
This module has been gradually replaced with the Requests third party library.
We'll mention the Requests third party library later.
It is suitable for development of medium-sized and small web crawlers.
For developing large web crawlers,
it's necessary to use Scrapy,
which is a highly popular open-source crawler framework.
Let's look at a concept here.
The process of scraping is actually
the client computer sends a request to the server.
Responseand the server will return a response.
After getting a response,
we'll parse it.
At present, popular parser tools include the BeautifulSoup library
and the regular expression module.
We'll briefly introduce them later.
Apart from those scraping and parsing modes,
sometimes we may turn to a third party API to
more conveniently scrape and parse webpage contents.
In this way, the third party API has automatically
scraped and parsed for us
with regular updating.
Let's look at the popular Requests library for webpage scraping first.
Compared with many previous libraries,
this library is indeed simple, convenient and human-oriented.
Its basic method is "get()",
corresponding to the "get" method in the http protocol,
which may request and acquire the resources at designated url locations.
Well, let's look at its official website first.
This is rightly the official website of Requests library.
It's worth noticing that,
we should get used to using
some officially-provided help information.
For example, when we're under the Python environment before,
we would often use "dir()" and "help()".
As for third party libraries like Requests,
their official websites generally provide quite rich information,
supporting multiple languages.
It will be OK if we rely on
its website.
Well, let's have a look.
This is a brief introduction to the Requests library.
Quite interesting.
This first sentence says
Requests is the only Non-GMO HTTP library
for Python,
safe for human consumption.
Some warning information as well.
Recreational use of the Python standard library for HTTP
may result in dangerous side-effects,
including security vulnerabilities, verbose code,
headaches, or even death.
Is that interesting?
Let's feel the charm of the Requests library.
Quite easy. Use the "get()" method,
followed by the webpage address to be scraped,
then a response object is acquired,
Next, view the status code with "status_code".
If it is 200, it means everything is OK.
Then, view the webpage contents through the text attributes.
These statements are most common.
Is it easy? Let's give it a go.
Suppose we're to scrape,
the short comments on a certain page of the Little Prince at book.douban.com ,
what we're going to scrape is the source codes of webpage.
Not difficult.
Through the "get()" method in the Requests library, we can scrape them.
It's worth mentioning that,
before scraping webpage of a website,
check whether this website has any Robots Protocol.
For instance, the Robots Protocol(The Robots Exclusion Protocol) of Douban website is like this.
How to view a Robots Protocol?
Some websites provide such a file, robots.txt.
If so, it means the website has its own Robots Protocol.
For example, let's have a look.
We're now scraping the directory, "subject".
It's not prohibited here.
So we can scrape it.
Sure, if you want to scrape several pages,
pay attention to its delay,
here, say, it's 5 seconds.
OK, let's have a try.
The Requests library has been preloaded in Anaconda,
we can directly import it.
If not installed,
do not worry,
install it through pip install,
then use its "get()" method
to put the url of our requested webpage.
It's scraping now.
Alright, check the status code,
200. Everything's OK.
That's the scraped page contents.
Quite easy, right?
However, contents on some web pages may be dynamically generated.
When directly scraped,
contents of these webpages may not include our desired data.
For instance, we'd like to scrape
the basic information of Dow Jones Industrial Average stocks.
In its source code, as we see a lot of
our desired data can not be acquired through
directly scraping this url,
but dynamically generated by js.
Of course, website generation modes
are not always the same.
Some data, say,
can be quite easily acquired sometimes.
We can look for some non-dynamically generated pages for scraping,
or acquire data with the API we'll talk about later.
There're also some existing data sets,
or data directly saved online.
For conveniently acquiring,
the basic information of Dow Jones Industrial Average stocks,
like company code, long name and its closing price etc.
we can utilize such a website, similar to CNN.
Sure, please note that
the modes of data generation of this website may also be changed.
With url, we can use
the "get()" method in the Requests library
to send a request,
and then acquire a response object.
This object includes the request information
and the server's response information.
And Requests will automatically decode
information from the server.
Suppose a webpage is in the "json" format,
then we can utilize the
embedded "json" decoder in the Requests library for decoding.
In a similar way like this,
Suppose the response contents are in a binary form,
then we can use the "content",
attribute to decode,
like, returning a picture created with binary data.
Besides, we also introduced
the more common "text" attribute.
It may automatically guess the text codes and decode them.
Well, we might as well use
the "encoding" attribute to modify text codes.
The frequently used one is utf-8.
Such webpages contents we've acquired in this way,
if parsed in a way we'll introduce soon,
can produce the real data inside.
For example, if we take a part of it,
the effect is like this.
When introducing the dictionary later,
we'll talk about,
how to use the arguments in the "get()" method in the Requests library
to submit inquires at the keyword inquiry interface of search engines.
So far,
we have got
a little skill of scraping webpages.
For more complex inquiries,
like submitting forms,
we still need the "post()" method,
which is not frequently used.
After acquiring the source codes of webpages,
we need to parse the data contents of webpage,
such as the short comments on
the Little Prince, which we introduced just now.
All its comments
are marked with this tag group.
Such data parsing is quite suitable for
a library like BeautifulSoup.
Beautiful Soup is
an HTML and XML parser.
It can very conveniently
extract data from this kind of files.
Suppose in another situation.
Let's see,
this is the score of a comment.
We'll find
this data structure is relatively complex.
To extract this kind of data,
with complex details,
it is more suitable to adopt the regular expression module.
Let's look at Beautiful Soup first.
This is the official website of Beautiful Soup.
Have a look.
You might have found
it is similar to the Requests library.
Its official website provides abundant contents,
as well as some instances for beginners,
supporting multiple languages.
Let's look at it briefly.
For using BeautifulSoup,
import the library first,
next, use the BeautifulSoup() function
to create a BeautifulSoup object.
Here, you may send in a string or
a file handle.
Moreover, pay attention to selecting the parser.
A lot of parsers are available.
The official website also compare them in details.
For HMTL, the most frequently used parser is LXML.
This parser operates at a high speed,
with strong capacity of document fault tolerance.
We often recommend it.
Let's have some examples.
BeautifulSoup. Have a look at
the basic use of Beautiful Soup.
Since Anaconda has been
preloaded with the BeautifulSoup library,
we only need to directly import it.
Use the upper case please.
Then, we define a string,
use the BeautifulSoup() function to send in this string,
generate a BeautifulSoup object "soup".
It's worth noticing that
there are four BeautifulSoup objects.
Tag, NavigableString, BeautifulSoup and Comment.
Tag is just the tag in HTML or XML documents,
like this,
providing some decorations for text contents.
And BeautifulSoup,
in most cases it is indeed Tag,
so we can treat them as Tag objects in most cases.
NavigableString is indeed strings in Tag,
like the Little Prince here,
while comment is a sub-class of NavigableString.
OK let's go on.
All tag contents can be acquired through
BeautifulSoup object .Tag .
For example, we can acquire the contents of Tag b.
Check its type.
It's a tag.
The most essential attributes of tag object include "name" and "attribute".
Each tag acquires its name through the "name" attribute.
Say, here, its name is "p".
A tag may includes many attributes.
We can acquire corresponding attributes
through these attribute names.
Besides, operations of tag attributes are
the same as the dictionary we'll talk about later.
So, we can acquire attributes in such a way.
Also, let's have a look.
NavigableString objects can be expressed with the string attribute.
For instance, do we get the non-attribute string in tag?
Let's check its type.
It is NavigableSting.
It will turn out to be very common in our later sections.
In BeautifulSoup, there is another
common method find_all().
For instance, if we write like this,
we can find all Tag b contents.
If we only need to find the first tag contents,
we can use the find() method.
Sure, apart from using tag as an argument, the find_all() method
can also be followed by the attribute name.
Next, let's have a look at
the task of parsing book reviews of the Little Prince we introduced before.
Let's see how can we
use BeautifulSoup to parse its contents.
Let's look at the program.
Use the object we got before as an argument.
Send it into the BeautifulSoup() function.
We get a BeautifulSoup object, "soup".
Then, use the find_all() method to find
the line of the review.
As we saw before,
the characteristic of review line is Tag span.
The attribute content is short.
The find_all() method returns a list.
We traverse the list.
We'll talk about this mode soon.
Have a brief look.
For each item in this list,
we only need to output the string attribute of this object,
and we'll get the string.
Execute this program.
Have we acquired those short comments?
After acquiring the comments, let's look at
how to get such scores.
As we mentioned before,
such a way of acquiring details is more appropriate
to adopt regular expression.
What is a regular expression?
It's often used to retrieve and replace a text complying with
certain rules or modules.
For example, 0 to 9 means any number between them.
. means any characters except the newline character,
* means repetition for 0 or several times.
Adding parentheses means to group.
Many authoritative websites provide introductions to regular expressions.
They are also quite complex.
So we just briefly
introduce them here.
Only one example here.
In webpage parsing,
there is one form very very frequently used.
Suppose we're to find
a string like this,
followed by a string at the back,
a string in the middle.
Then, very simply, we can
express this string as (.*?),
and then use the compile() method
to compile the string into a pattern instance.
Have a try.
The execution of compiling a string into a pattern instance is fast.
So we often use it.
Then, we use the most frequently used
findall() function in the regular expression module
to match such a model in the source code.
The return result is a list p.
Then, calculate the sum of scores.
It is originally a string,
so we need to convert it into the int type first,
and then output the sum.
OK, let's run it.
Finally, we see the sum is 680.
We might have a look at the value of p.
It is a list.
Scores are all here.
This example has realizes the scraping of a single webpage,
and parsing of some webpage contents.
We can also repeatedly scrape many webpages
based on the characteristics of url of these webpages.
In such a way, a small crawler is created.
So far, we have mastered the basics
of developing a small web crawler.
Let's look back.
Use the Requests library to scrape webpages,
use the BeautifulSoup library and the regular expression module
to parse webpage contents.
You can practice them more in future ,
and improve your skills, of course.
But do not just focus on Douban.
When scraping webpages,
Do follow the Robots Protocol.
