Hello everybody. Welcome to Python for Everybody. We are doing some code walk-throughs. If you want to follow along with the code, you can download the source code from Python for Everybody website. So, the code we're playing with today is twfriends.py. This is a step beyond the simple twspider. It is a restartable spider. But we're going to date the model things a little bit differently. We're going to have two tables and we're going to have a many-to-many relationship accepted. It's a many-to-many relationship between the same table, which is okay. Twitter friends are a directional relationship. So, we start out here in twfriends.py. Remember, that the file hidden.py, I'll show it to you, but I'm not going to open it because I've got my keys and secrets in it. So, this hidden.py file, you've got to edit that, and you got to go to apps.twitter.com and get your keys and put them in there. Otherwise, these things won't work. But if you have Twitter and you set your API keys up, and you put them in the hidden.py, then all these things will work. It's fun actually, and impressive. Not hard to do actually. So, the Twitter URL, that's my library, that reads hidden.py and augments the URL does all the OAuth stuff, JSON and SSL because Python doesn't accept any certificates even if they're good certificates. So, we crush that. Here's our friends list that we're going to hit. We're going to make a database friends.sqlite. Now, here we're doing a create table if not exists. So, what this really is saying is, I want this to be a restartable process and I don't want to lose the data. We're starting out, we do not have any SQLite files. So, this is going to create the database and create these tables. But the second time we run it, we're not going to recreate the tables. We're not going to be able to restart this, because we're going to run out of rate limit before we finish this. So, we just have to wait, however long the time it takes to reset, and we'll watch the rate limit go down. So, we're going to have a people table, and we're going to have a primary key and the name. The name is going to be unique, and whether or not we've retrieved it. That's from a previous one. Then there's the who follows who, the from ID to, to ID. So, this is a direction and we're going to put a uniqueness constraint in, just like we do in many-to-many, that basically says, the combination of from ID and to ID has got to be unique. We don't allow ourselves to put duplicates of the combination. So, from ID can be one in many records, and to ID can be one in many records, but one, one is only allowed once. This is the crud we have to do to convince Python to accept the Twitter certificate. So, this is similar to some of the other stuff that we've done. We're going to enter a Twitter account or quit. If we enter by itself, then we will actually go and retrieve a record that was not yet retrieved. Now, we're actually pulling out two values, ID and name. So, we will grab, fetch one, is going to give us a two tuple basically. We're going to store that in ID, in account. Of course, that's like, this is coming back with a two tuple, first of which is the ID from the database. Limit one, means we're only going get one of these or zero of these. If there's zero these, that means there are no unretrieved Twitter accounts. Retrieved equals zero, well, you'll see in a second, that all the new accounts we put in, are the ones for which we haven't retrieved. Again, given that our rate limit, we want to know which ones we've retrieved. Okay? So, what we're going do next is, we're going to check to see if the person that we just checked, which means the length of the account is greater than what we just entered. We're going to check to see if they're already there. Okay? We're going to select ID from people where name equals. So, that's the one we just entered. We're going to fetch one and grab the first thing, because we only got one thing in the select statement here. If this person that we just asked to see is not in the table, that means this is going to fail, we're going to do an insert or ignore. This our ignore is redundant because we just checked to see if it was there. But we'll put that in just to be safe. We're going to put the name in as the new account that we're looking at, indicating that retrieved is zero. So, that we will know that we haven't retrieved it yet. You'll see that we'll update that in a second. We commit it, so that later selects we'll see this. So, you got to do the commit. This later select wouldn't see the one we just inserted. We're going to ask how many rows were affected, and if it's not equal to one, then, we're going to complain about- we inserted it, and we're going to do this thing. We are going to ask, "Hey remember there was an ID up there?" Right here, ID integer, primary key, and we did not insert this here, but we want to know what that ID is. Every time I was showing you that in lectures, I was saying it's really easy in Python to do this. That's what we're saying. This cursor did the insert, but one of the things happens is after the insert, we're going to grab the last row ID, which is the primary key that was assigned by SQL. Okay? So, that means, that one way or another, coming through this code here in line 45, one way or another, we're either going to know the ID of the user that was there before, or we just inserted one and so we're going to know the primary key of the current user. You'll see why we need that. So, ID, is the primary key of the current user that we entered right here. Okay? Now, what we're going to do is do the Twitter URL augment with OAuth, and all the keys, and the secrets in hidden.py. Instead, we're going go through let's count 1,000. Let's go count, what the heck. Let's go 200. Up to 200 friends. No, let's do 100. We'll keep it that way. Then, we're going to retrieve it and we're retrieving the account. We're not going to print the nasty URL out, we could. Then we're going open the URL with a connection, and then we're going to read that, and we're going get the UTF-8 data from this, and then we're going to decode that, and we're going to have the unicode data. So, the data in string is a internal Python string, with all that data representing all the wonderful characters. Of course, we're going to ask URL open to give us back the headers as a dictionary, using this call. We can see how many we have left for the remaining. Right? What's the remaining rate limit that we have? So, then we're going to parse the data with JSON load S. Wait, I need to continue in here. Continue. Okay. Save. If we're going to parse this data, we'll print it out, right? So, that means that this died, which means it's not syntactically correct, json basically. Who knows if we're ever going to see that, but at least, when it blows up it'll print this data out. We'll have to catch it, and then it'll continue. Actually, I'll make this a break, because if that's blown up that bad, we should quit. Now, I don't yet know what happens when this rate limit says you can't have it, and so but I do know that I expect when it's successful, that there will be a key of users in this outer dictionary that we're going to get. If this outer dictionary, that if users is not in the parsed dictionary, then I'm going to dump out this data so that at least I can debug what happens when I've got some broken json. So, the difference between this code, this code is going to fail when the json is syntactically bad, meaning a curly brace isn't right or whatever. This code will trigger when I get good json, but I don't have a user's key in it, okay? So, then, once we've retrieved it, we will be pretty happy with it. We're going to update for our account that we're retrieving. We're going to set this is one of our retrieved accounts, okay? Then, what we're going to do is write a loop that goes through all the friends of this particular user that we're asking, and gets their screen name, prints it out, and then we're going to check to see if this one is already in our people database, because this is a spider, we're grabbing accounts. So, we'll do a friend ID, and do a fetch one, grab the subzero thing. If that works, if this person not in there, this fetch one is going to blow up which means we're going to drop down to the except code. But if it does work, we have friend ID, that if they're there, and they're already in our database. They just weren't retrieved. Okay. So now, if the friend ID wasn't there, we're going to do an insert into, setting retrieved to zero, and then we're going to commit. Now, remember, row count is how many rows were affected by this last transaction code at row count, and we're going to die if that insert doesn't work. This is unlikely, unless somehow we've ran out a disc drive or something, and we're going to grab the friend ID as the key, the last row that was inserted. We're only going insert one row, so it's basically the primary key of the row that we just inserted. So, if you look at this code right here, it comes out the bottom, one way or another with friend ID successful. Friend ID is either they are already in our database or they're not, and if we insert them, then we have it. So now, this count new and count old is just so I can print out a nice print out. Now, we are going to insert into the friend's table which is called the follows table in this case, from ID and to ID. Those are the two outward pointing foreign keys, and we have the ID of the account that we're retrieving the friends of, and then this particular friend. So, we're inserting the connection from this person to that person then we commit it. We want to commit these again so that later selects when the loop goes back up, later selects get all of that data that's going on. So, we do want to commit from time to time, and then we close the cursor at the very end. So, let's run this and see what happens. So, python twfriends.py. Of course, I am a refugee from python two, so I always forget to type python three. So, we're going to start. If we take a look right now, I'm going to start another tab over here, and ls-l *sqlite. Now, that sqlite file is there, and it's actually made the tables. If you go up here, it ran all this stuff, create the tables, yada, yada, and we're sitting right here at this line. As a matter of fact, I think without causing too much trouble, I can open that database, and get into this database right here. There is no data in the Follows table, and there is no data in the People table. It's completely empty. So, we're waiting for the first one, and I'll go with mine, Doctor Chuck. So, it's retrieving the 100 friends, and they all were brand new. They are all inserted. So now, if I hit Refresh, we will see that Doctor Chuck is retrieved. Who follows? So, these are all the people I follow, for one follows two. So, if we look at here, we see that Doctor Chuck follows Stephanie Teasley. Because we grabbed the followers of Dr. Chuck, we're going to have a record in all of the follows for all the ones that I did, right? So, these are all the people I followed, and we put them in. So, we can go back and we can, let's see grab somebody. Let's go grab Stephanie Teasley, and let's pull out her friends. So, we grabbed 100 of her folks. I got 14 left. That's my x-rate limit. I did Stephanie Teasley, so let's go back here. So, you'll notice there's 101, that row is going to be 182. That's interesting. So, we've retrieved Doctor Chuck and Stephanie Teasley, and let's go take a look in the Friends table, the Follows table. So, we have all the people I follow, now all the people Stephanie follows. So, there we go. Let's go ahead and do somebody else. Let's see here. I think we both follow Tim McKay. Where is Tim McKay? Let's follow Tim McKay. Let's see who to Tim follows, If we can get like an overlap. We revisited some. Let's see if we can see this in the Follows. Let's see People. So, we've got Dr. Chuck retrieved. Tim McKay's somewhere down here. It might take us a while before we get any really good overlaps. Let's see. Let's do a database called. Let's see. Let's do a database SQL. Select, count. Yeah. Okay. So, let's just run this some more. It's clearly working. Now, one thing I can do here is I can hit "Enter", and it will just pick one randomly. So, it grabbed Liveedutv. Let's see how many I got left. We got 12 left. Now, I can hit "Enter" again. It picks another one. That was the next one. It's picking them in order. Is it picking them in order? Let's go to "People". Yeah, it's picking these. So, we can see that it's going to just do the first unretrieved person, who's Nancy. Let it retrieve Nancy. So, it grabbed Nancy, new. So, we're finding some and this table's getting really big. So, if we look at the "People" table, we now have 455 people and we have 467 following records. So, there we go. Hit "Enter", does another one, and away we go. So, you get the idea. I can type quit to finish. Just to give you a little interesting bit of code to show you how to do selects, I'm going to do this twjoin. Now, you'll notice that we're not talking, let's show you one thing, ls minus l star sqlite. So, this database has it. So, I can restart this process and run it again. The database is still there. So, we just grab, swear_trek. So, we can keep doing this. So, this data, it keeps extending. So, this is a restartable process. I can run it. Then, tell it to grab the next unretrieved one. So, away we go. So, that's part of it. I've got eight left. How many do I have left, really? Let's keep going. How many do I got left? I got five left. Okay. Wait, I guess we'll just run it out. So, I got four left. You know what I should do? I can't change the code. Yes, I can change the code. I can stop the code and I can quit the code. So, what I'm going to do is I'm going to change this code a little bit really quick and I'm going to print the headers of rate limiting at the beginning and at the end. So, now I can run it again. I changed the code. Hopefully, I didn't make a Python error. Tell it to go get another one, Ana Navarro. So, I got three left. We'll see what happens when I run out of rate limit. Run out of rate limit. So, we have one left. Hit "Enter" and control K, OpenSource.org. So, we have zero left, that worked. Now, let's see what happens. I don't know what happens next. We blew up. Too many requests. So, we got a HTTP Error 429. So, that means that going from Marc Cuban, that was in line 48, so the right thing to do would be in line 48. We should really put this in a "try except" block. "try except" block because it gives us an error. Print, fiddlesticks. How do I print the exception message? I always am forgetting. Print "Failed to Retrieve". So, we'll put that in. Now, if I run it. Then, I have to put a "break" here because that's not good. Break. Fail to Retrieve. See, I never know how to print out the error message. Yeah. So, see that's the weird thing about stuff is, that I don't ever remember enough. I don't remember the syntax, what I say here, to print the error message out. So, I'm going to go to Google and I'm going to say, "Print out the exception message in Python." Python 3, hello. So, let's go find it here in the documentation. Except. Is this it? Is this what I say? I just want to print out the message. That's it. Except. Let's try this. So, this is part of Python programming is, for me at least, because I'm just not a genius expert at this stuff. This is one thing I like about Python is you can guess stuff and sometimes you guess right. So, there we go. We got the error, we got the nice little error message, and we see error 429: Too Many Requests. So, that cleans that up nicely. So, we have run out of requests. On that, it is a good time to say thanks for listening. I hope that you found this valuable.
