网站调用新浪微博内容(新版新浪微博模拟登陆请看：本文后面的解决动态加载的程序)

　　网站调用新浪微博内容(新版新浪微博模拟登陆请看：本文后面的解决动态加载的程序)

　　新版新浪微博模拟登录请看：

　　本文后面解决动态加载的程序还是有效的

　　再次编辑，出了点问题

　　第一个模块是模拟登录新浪微博，创建一个weiboLogin.py文件，输入如下代码：

　　#! /usr/bin/env python

# -*- coding: utf-8 -*-

import sys

import urllib

import urllib2

import cookielib

import base64

import re

import json

import hashlib

class weiboLogin:

cj = cookielib.LWPCookieJar()

cookie_support = urllib2.HTTPCookieProcessor(cj)

opener = urllib2.build_opener(cookie_support, urllib2.HTTPHandler)

urllib2.install_opener(opener)

postdata = {

'entry': 'weibo',

'gateway': '1',

'from': '',

'savestate': '7',

'userticket': '1',

'ssosimplelogin': '1',

'vsnf': '1',

'vsnval': '',

'su': '',

'service': 'miniblog',

'servertime': '',

'nonce': '',

'pwencode': 'wsse',

'sp': '',

'encoding': 'UTF-8',

'url': 'http://weibo.com/ajaxlogin.php?framelogin=1&callback=parent.sinaSSOController.feedBackUrlCallBack',

'returntype': 'META'

}

def get_servertime(self):

url = 'http://login.sina.com.cn/sso/prelogin.php?entry=weibo&callback=sinaSSOController.preloginCallBack&su=dW5kZWZpbmVk&client=ssologin.js(v1.3.18)&_=1329806375939'

data = urllib2.urlopen(url).read()

p = re.compile('\((.*)\)')

try:

json_data = p.search(data).group(1)

data = json.loads(json_data)

servertime = str(data['servertime'])

nonce = data['nonce']

return servertime, nonce

except:

print 'Get severtime error!'

return None

def get_pwd(self, pwd, servertime, nonce):

pwd1 = hashlib.sha1(pwd).hexdigest()

pwd2 = hashlib.sha1(pwd1).hexdigest()

pwd3_ = pwd2 + servertime + nonce

pwd3 = hashlib.sha1(pwd3_).hexdigest()

return pwd3

def get_user(self, username):

username_ = urllib.quote(username)

username = base64.encodestring(username_)[:-1]

return username

def login(self,username,pwd):

url = 'http://login.sina.com.cn/sso/login.php?client=ssologin.js(v1.3.18)'

try:

servertime, nonce = self.get_servertime()

except:

print 'get servertime error!'

return

weiboLogin.postdata['servertime'] = servertime

weiboLogin.postdata['nonce'] = nonce

weiboLogin.postdata['su'] = self.get_user(username)

weiboLogin.postdata['sp'] = self.get_pwd(pwd, servertime, nonce)

weiboLogin.postdata = urllib.urlencode(weiboLogin.postdata)

headers = {'User-Agent':'Mozilla/5.0 (X11; Linux i686; rv:8.0) Gecko/20100101 Firefox/8.0 Chrome/20.0.1132.57 Safari/536.11'}

req = urllib2.Request(

url = url,

data = weiboLogin.postdata,

headers = headers

)

result = urllib2.urlopen(req)

text = result.read()

p = re.compile('location\.replace\(\'(.*?)\'\)')

try:

login_url = p.search(text).group(1)

urllib2.urlopen(login_url)

print "Login success!"

except:

print 'Login error!'

　　然后创建 main.py 文件并输入以下代码：

　　#!/usr/bin/env python

# -*- coding: utf-8 -*-

import weiboLogin

import urllib

import urllib2

username = '你的微博用户名'

pwd = '你的微博密码'

WBLogin = weiboLogin.weiboLogin()

WBLogin.login(username, pwd)

　　注意：如果登录失败，可能是您的账号登录时需要输入验证码！您可以在网页上登录您的帐户并尝试。您可以在账户设置中设置某些区域不输入验证码。

　　参考：

　　接下来考虑爬取微博内容的实现。

　　这时，我遇到了困难。当我抓取指定网址的微博时，最初只显示了15条。后者是延迟显示（在ajax中称为延迟加载？）。也就是说，当滚动条第一次拖到底部时，会显示第二部分，然后再拖到底部，会显示第三部分。此时，一个页面的微博就完成了。因此，要获取一个微博页面的所有微博，需要访问该页面3次。创建getWeiboPage.py文件，对应代码如下：

　　#!/usr/bin/env python

# -*- coding: utf-8 -*-

import urllib

import urllib2

import sys

import time

reload(sys)

sys.setdefaultencoding('utf-8')

class getWeiboPage:

body = {

'__rnd':'',

'_k':'',

'_t':'0',

'count':'50',

'end_id':'',

'max_id':'',

'page':1,

'pagebar':'',

'pre_page':'0',

'uid':''

}

uid_list = []

charset = 'utf8'

def get_msg(self,uid):

getWeiboPage.body['uid'] = uid

url = self.get_url(uid)

self.get_firstpage(url)

self.get_secondpage(url)

self.get_thirdpage(url)

def get_firstpage(self,url):

getWeiboPage.body['pre_page'] = getWeiboPage.body['page']-1

url = url +urllib.urlencode(getWeiboPage.body)

req = urllib2.Request(url)

result = urllib2.urlopen(req)

text = result.read()

self.writefile('./output/text1',text)

self.writefile('./output/result1',eval("u'''"+text+"'''"))

def get_secondpage(self,url):

getWeiboPage.body['count'] = '15'

# getWeiboPage.body['end_id'] = '3490160379905732'

# getWeiboPage.body['max_id'] = '3487344294660278'

getWeiboPage.body['pagebar'] = '0'

getWeiboPage.body['pre_page'] = getWeiboPage.body['page']

url = url +urllib.urlencode(getWeiboPage.body)

req = urllib2.Request(url)

result = urllib2.urlopen(req)

text = result.read()

self.writefile('./output/text2',text)

self.writefile('./output/result2',eval("u'''"+text+"'''"))

def get_thirdpage(self,url):

getWeiboPage.body['count'] = '15'

getWeiboPage.body['pagebar'] = '1'

getWeiboPage.body['pre_page'] = getWeiboPage.body['page']

url = url +urllib.urlencode(getWeiboPage.body)

req = urllib2.Request(url)

result = urllib2.urlopen(req)

text = result.read()

self.writefile('./output/text3',text)

self.writefile('./output/result3',eval("u'''"+text+"'''"))

def get_url(self,uid):

url = 'http://weibo.com/' + uid + '?from=otherprofile&wvr=3.6&loc=tagweibo'

return url

def get_uid(self,filename):

fread = file(filename)

for line in fread:

getWeiboPage.uid_list.append(line)

print line

time.sleep(1)

def writefile(self,filename,content):

fw = file(filename,'w')

fw.write(content)

fw.close()

　　在刚才的main.py中添加相应的内容，完整内容为：

　　#!/usr/bin/env python

# -*- coding: utf-8 -*-

import weiboLogin

import getWeiboMsg

import urllib

import urllib2

username = '你的微博用户名'

pwd = '你的微博密码'

WBLogin = weiboLogin.weiboLogin()

WBLogin.login(username, pwd)

WBmsg = getWeiboMsg.getWeiboMsg()

url = 'http://weibo.com/1624087025?from=otherprofile&wvr=3.6&loc=tagweibo'

WBmsg.get_firstpage(url)

WBmsg.get_secondpage(url)

WBmsg.get_thirdpage(url)

　　参考：

　　执行python main.py，应该可以运行了，结果保存在./output/文件夹中，这个文件夹是提前创建的。

　　昨天我们整整过了一个下午，还有很多事情没有做。欢迎留言。