爬虫 on 搁浅St的blog

ubuntu下使用scrapy抓取cnblogs

Sun, 26 Apr 2015 16:52:28 +0000

今天在伯乐在线上看到一篇翻译的博客，讲的是使用scrapy来抓取stackoverflow上的问题，刚好好久没用这个，于是一并捡起来玩一下。

软件安装

我的环境是：ubuntu 14.04 lts 需要安装相关软件

scrapy

pip install Scrapy

PyMongo

pip install pymongo

Mongodb

上面安装的是python使用Mongodb的接口，很显然，我们要安装Mongodb才能使用

sudo apt-get install mongodb-server

至此，要使用的软件都已经安装完毕

使用scrapy新建工程

使用scrapy新建工程很简单，如下所示，我们新建一个stack的项目，他会在你的当前目录新建一个stack文件夹

scrapy startproject stack

并且会建成如下所示的目录树结构

chen@chen-P31:~$ tree stack
stack
├── stack
│   ├── __init__.py
│   ├── items.py
│   ├── pipelines.py
│   ├── settings.py
│   └── spiders
│       └── __init__.py
└── scrapy.cfg

接下来，我们修改items.py的内容，这个文件用于定义存储“容器”，用来存储将要抓取的数据。

from scrapy.item import Item,Field

class StackItem(Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = Field()# 我们添加两个字段，我们等会儿会抓取一个标题和url两个字段
    url = Field()

接着，还有一个很重要的东西，对，就是我们的蜘蛛，我们在spider目录下，新建一个stack_spider.py文件。顾名思义，这就是我们的蜘蛛。我们需要定义我们爬虫的起点，爬虫的规则等等

from scrapy import Spider
from stack.items import StackItem  # 导入我们上面定义的容器类
class StackSpider(Spider):
    name = 'stack'   # 定义我们爬虫的名字
    allowed_domains = ["cnblogs.com"]   # 规定爬虫爬取的域名
    start_urls = ['http://www.cnblogs.com/geqianst/p/',]   # 爬虫工作的起点

    def parse(self, response):# 爬虫用来做数据解析的
        questions = response.xpath('//div[@id="myposts"]//a[@id]')
        # xpath选择器，这里的含义是取所有id为myposts的div，在它下面找所有带id的超链接a
        # 实际结果是这样的
        # [<Selector xpath='//div[@id="myposts"]//a[@id]' data=u'<a id="PostsList1_rpPosts_TitleUrl_0" hr'>,
        # <Selector xpath='//div[@id="myposts"]//a[@id]' data=u'<a id="PostsList1_rpPosts_TitleUrl_1" hr'>,
        # <Selector xpath='//div[@id="myposts"]//a[@id]' data=u'<a id="PostsList1_rpPosts_TitleUrl_2" hr'>,
        # <Selector xpath='//div[@id="myposts"]//a[@id]' data=u'<a id="PostsList1_rpPosts_TitleUrl_3" hr'>,
        # <Selector xpath='//div[@id="myposts"]//a[@id]' data=u'<a id="PostsList1_rpPosts_TitleUrl_4" hr'>,
        # <Selector xpath='//div[@id="myposts"]//a[@id]' data=u'<a id="PostsList1_rpPosts_TitleUrl_5" hr'>,
        # <Selector xpath='//div[@id="myposts"]//a[@id]' data=u'<a id="PostsList1_rpPosts_TitleUrl_6" hr'>,
        # <Selector xpath='//div[@id="myposts"]//a[@id]' data=u'<a id="PostsList1_rpPosts_TitleUrl_7" hr'>,
        # <Selector xpath='//div[@id="myposts"]//a[@id]' data=u'<a id="PostsList1_rpPosts_TitleUrl_8" hr'>,
        # <Selector xpath='//div[@id="myposts"]//a[@id]' data=u'<a id="PostsList1_rpPosts_TitleUrl_9" hr'>]
        #

        for question in questions:
            item = StackItem()
            item['title'] = question.xpath(
                'text()').extract()[0]
            item['url'] = question.xpath(
                '@href').extract()[0]
            print item
            yield item

测试

ok，上述工作基本完成，我们来测试一下

scrapy crawl stack

还可以这样测试一下，使用shell命令妈蛋，我的竟然出错了，输出如下

chen@chen-P31:~/stack$ scrapy crawl stack
2015-04-26 16:49:11+0800 [scrapy] INFO: Scrapy 0.24.6 started (bot: stack)
2015-04-26 16:49:11+0800 [scrapy] INFO: Optional features available: ssl, http11
2015-04-26 16:49:11+0800 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'stack.spiders', 'SPIDER_MODULES': ['stack.spiders'], 'BOT_NAME': 'stack'}
Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 143, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 89, in _run_print_help
    func(*a, **kw)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 150, in _run_command
    cmd.run(args, opts)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/crawl.py", line 60, in run
    self.crawler_process.start()
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 92, in start
    if self.start_crawling():
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 124, in start_crawling
    return self._start_crawler() is not None
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 139, in _start_crawler
    crawler.configure()
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 46, in configure
    self.extensions = ExtensionManager.from_crawler(self)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/middleware.py", line 50, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/middleware.py", line 29, in from_settings
    mwcls = load_object(clspath)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/misc.py", line 42, in load_object
    raise ImportError("Error loading object '%s': %s" % (path, e))
ImportError: Error loading object 'scrapy.telnet.TelnetConsole': No module named conch

这是什么gui？还好我有stackoverflow，google一番，找到解决办法（其实这不是最后的解决办法，请往后看）网上说是twisted的问题，重新安装一下就好了，ok，走起

chen@chen-P31:~/stack$ sudo apt-get install twisted
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package twisted
chen@chen-P31:~/stack$ sudo apt-get install twisted.conch
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'python-twisted-conch' for regex 'twisted.conch'
Note, selecting 'python2.7-twisted-conch' for regex 'twisted.conch'
Note, selecting 'python-twisted-conch' instead of 'python2.7-twisted-conch'
The following packages were automatically installed and are no longer required:
  cli-common dockmanager freepats gstreamer1.0-plugins-bad-faad
  gstreamer1.0-plugins-bad-videoparsers libbotan-1.10-0:i386 libcdaudio1
  libdbus-glib2.0-cil libdbus2.0-cil libdbusmenu-glib4:i386
  libdbusmenu-gtk4:i386 libegl1-mesa:i386 libegl1-mesa-drivers:i386 libflite1
  libfluidsynth1 libgbm1:i386 libgconf2.0-cil libgdiplus libgif4
  libgles2-mesa:i386 libglib2.0-cil libgme0 libgmp10:i386
  libgnome-desktop-2-17 libgnome-keyring1.0-cil libgnomedesktop2.20-cil
  libgstreamer-plugins-bad0.10-0 libgstreamer-plugins-bad1.0-0 libgtk2.0-cil
  libicu52:i386 libmimic0 libmms0 libmono-addins0.2-cil libmono-cairo4.0-cil
  libmono-corlib4.0-cil libmono-corlib4.5-cil libmono-data-tds4.0-cil
  libmono-i18n-west4.0-cil libmono-i18n4.0-cil libmono-posix4.0-cil
  libmono-security4.0-cil libmono-sharpzip4.84-cil libmono-sqlite4.0-cil
  libmono-system-configuration4.0-cil libmono-system-core4.0-cil
  libmono-system-data4.0-cil libmono-system-drawing4.0-cil
  libmono-system-enterpriseservices4.0-cil
  libmono-system-runtime-serialization-formatters-soap4.0-cil
  libmono-system-security4.0-cil libmono-system-transactions4.0-cil
  libmono-system-web-applicationservices4.0-cil
  libmono-system-web-services4.0-cil libmono-system-web4.0-cil
  libmono-system-xml-linq4.0-cil libmono-system-xml4.0-cil
  libmono-system4.0-cil libmono-web4.0-cil libmpg123-0 libnotify0.4-cil
  libofa0 libopenal-data libopenal1 libopenvg1-mesa:i386 libqrencode3:i386
  libqt5core5a:i386 libqt5dbus5:i386 libqt5gui5:i386 libqt5network5:i386
  libqt5widgets5:i386 libqtshadowsocks:i386 librsvg2-2.18-cil libslv2-9
  libsoundtouch0 libspandsp2 libsrtp0 libssl1.0.0:i386 libv4l-0:i386
  libv4lconvert0:i386 libvo-aacenc0 libvo-amrwbenc0 libwayland-client0:i386
  libwayland-egl1-mesa:i386 libwayland-server0:i386 libwildmidi-config
  libwildmidi1 libwnck2.20-cil libxcb-icccm4:i386 libxcb-image0:i386
  libxcb-keysyms1:i386 libxcb-randr0:i386 libxcb-render-util0:i386
  libxcb-shape0:i386 libxcb-util0:i386 libxcb-xfixes0:i386 libxcb-xkb1:i386
  libxkbcommon-x11-0:i386 libxkbcommon0:i386 libzbar0:i386 mono-4.0-gac
  mono-gac mono-runtime mono-runtime-common mono-runtime-sgen python-mpd
  python-mutagen python-twisted-names
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  python-pyasn1
The following NEW packages will be installed:
  python-pyasn1 python-twisted-conch
0 upgraded, 2 newly installed, 0 to remove and 6 not upgraded.
Need to get 286 kB of archives.
After this operation, 1,793 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://mirrors.ustc.edu.cn/ubuntu/ trusty/main python-pyasn1 all 0.1.7-1ubuntu2 [44.2 kB]
Get:2 http://mirrors.ustc.edu.cn/ubuntu/ trusty/main python-twisted-conch all 1:13.2.0-1ubuntu1 [242 kB]
Fetched 286 kB in 0s (1,595 kB/s)         
Selecting previously unselected package python-pyasn1.
(Reading database ... 359746 files and directories currently installed.)
Preparing to unpack .../python-pyasn1_0.1.7-1ubuntu2_all.deb ...
Unpacking python-pyasn1 (0.1.7-1ubuntu2) ...
Selecting previously unselected package python-twisted-conch.
Preparing to unpack .../python-twisted-conch_1%3a13.2.0-1ubuntu1_all.deb ...
Unpacking python-twisted-conch (1:13.2.0-1ubuntu1) ...
Processing triggers for doc-base (0.10.5) ...
Processing 1 added doc-base file...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up python-pyasn1 (0.1.7-1ubuntu2) ...
Setting up python-twisted-conch (1:13.2.0-1ubuntu1) ...

安装总算完成，再试一次，妈蛋，又来一个新错误，这是什么gui？？？

chen@chen-P31:~/stack$ scrapy crawl stack
2015-04-26 16:50:44+0800 [scrapy] INFO: Scrapy 0.24.6 started (bot: stack)
2015-04-26 16:50:44+0800 [scrapy] INFO: Optional features available: ssl, http11
2015-04-26 16:50:44+0800 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'stack.spiders', 'SPIDER_MODULES': ['stack.spiders'], 'BOT_NAME': 'stack'}
Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 143, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 89, in _run_print_help
    func(*a, **kw)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 150, in _run_command
    cmd.run(args, opts)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/crawl.py", line 60, in run
    self.crawler_process.start()
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 92, in start
    if self.start_crawling():
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 124, in start_crawling
    return self._start_crawler() is not None
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 139, in _start_crawler
    crawler.configure()
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 46, in configure
    self.extensions = ExtensionManager.from_crawler(self)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/middleware.py", line 50, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/middleware.py", line 29, in from_settings
    mwcls = load_object(clspath)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/misc.py", line 42, in load_object
    raise ImportError("Error loading object '%s': %s" % (path, e))
ImportError: Error loading object 'scrapy.contrib.memusage.MemoryUsage': No module named mail.smtp

最后的最后，我在我们万能的github上找到答案，原来是我们没有安装python-twisted，安装一下，世界都美好了

sudo apt-get install python-twisted

输出到文件

为了更直观的看到结果，我们将结果输出到一个json文件

scrapy crawl stack -o items.json -t json

噢耶，第一个爬虫成功

存储到mongodb

接下来，我们做最后一件事，我们将结果存储到mongodb的数据库中在这里，我遇到一个大坑，无论是伯乐在线翻译的博客还是网上搜索到的一般教程，都是使用pymongo.Connection来连接数据库，可是妈蛋，你使用pip install pymongo安装的版本都是最新版本3.0.1，那个Connection的写法已经不支持，被丢弃了，擦。我们来看一下版本，我学到一个新命令pip show pymongo，用来查看某一个包的版本的。在pymongo 3.0的版本中，已经不再支持pymongo.Connection，而是使用pymongo.MongoClient来替代。

第一步

创建一个用来保存我们抓取数据的数据库。打开settings.py,指定管道，然后加入数据库的相关设置


BOT_NAME = 'stack'

SPIDER_MODULES = ['stack.spiders']
NEWSPIDER_MODULE = 'stack.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
# USER_AGENT = 'stack (+http://www.yourdomain.com)'

ITEM_PIPELINES = ['stack.pipelines.MongoDBPipeline', ]
# 关于mongodb的相关设置，包括服务器的ip，端口号，数据库名，表名，
# 我也是第一次使用mongodb竟然不需要用户验证信息，而且这表名确实奇怪，叫做MONGODB_COLLECTION
MONGODB_SERVER = "localhost"
MONGODB_PORT = 27017
MONGODB_DB = "stackoverflow"
MONGODB_COLLECTION = "questions"

DOWNLOAD_DELAY = 5  # 抓取的延迟

第二步

我们已经能够爬取和解析html数据了，而且已经配置了数据库，接下来，我们通过pipelines.py中建立一个管道去连接这两个部分。我们首先来完成数据库的连接部分

import pymongo

from scrapy.conf import settings
from scrapy.exceptions import DropItem
from scrapy import log


class MongoDBPipeline(object):

    def __init__(self):
        connection = pymongo.MongoClient(
            settings['MONGODB_SERVER'],
            settings['MONGODB_PORT']
        )
        db = connection[settings['MONGODB_DB']]
        self.collection = db[settings['MONGODB_COLLECTION']]

接下来定义一个处理函数

    def process_item(self, item, spider):
        for data in item:
            if not data:
                raise DropItem("Missing data!")
        self.collection.update({'url': item['url']}, dict(item), upsert=True)
        log.msg("Question added to MongoDB database!",
                level=log.DEBUG, spider=spider)
        return item

ok,搞定，我们再测试一把

scrapy crawl stack

执行效果如下

参考文献

1 ImportError: Error loading object ‘scrapy.contrib.memusage.MemoryUsage’: No module named mail.smtp 2 http://stackoverflow.com/questions/8671071/error-to-execute-python-scrappy-module 3 Python下用Scrapy和MongoDB构建爬虫系统（1）

python学习笔记--python和beautifulsoup遇到的编码问题

Sun, 29 Mar 2015 14:43:45 +0000

在刚开始使用github pages的时候,我用python写了一个爬虫,计划是从csdn和cnblogs等博客网站上,把自己之前写的博客爬取下来,然后再转换成hexo用的markdown格式,样就可以直接添加到我的github pages.

项目主页: https://github.com/Jackroyal/blog2markdown

最近刚好在学习python,刚好就把它给优化了一下,顺便做了个跨平台(哈哈,win和ubuntu都可以跑哈),昨天遇到很蛋疼的问题,一它给了我很多思路帮助我定位问题,.直搞到凌晨两点才弄好. 非常非常感谢【总结】Python 2.x中常见字符编码和解码方面的错误及其解决办法作者做了一个很好的总结,帮助我们定位问题.

1 编码类型

首先确定好你的编码类型,比如一般推荐用utf-8.当确定编码类型后,就要保持统一,不要又弄些GBK的编码在里面.

1.1 编辑器编码
- 我们有时候会犯一个错误,我在py文件的头部声明当前文件是按照utf-8来编码.但是文件实际保存的编却不是utf-8,这样也会导致乱码.建议使用可以查看当前文件编码的编辑器,比如sublime text或者notepad++ 或者pycharm.在sublime下如图所示
- 文件编码声明,我们要在py文件的头部添加一行# -*- coding: utf-8 -*-,表明我接下来要使用utf-8编码
1.2 python解释器
- 如果是Python的IDLE，如果你没修改defaultencoding，那么就使用默认的字符编码可以通过sys.getdefaultencoding()而获得，比如此处获得是：ascii
1.3 执行python代码
- 其中，很常见的几种动作是：
  - 打印print对应的所获得的字符
    - 对于字符串打印,Python的逻辑:
      - 如果是Unicode字符串,则可以,自动地,编码为对应的终端所用编码,然后正确的显示出来
      - 比如unicode的字符串,输出到windows的默认编码为GBK的cmd中,则Python可以自动将Unicode编码为GBK,然后输出到cmd中
      - 个别特殊情况,也会出错:
        
        当此unicode字符串中包含某特殊字符,而目标终端的编码集合中,没有此字符,则很明显也是无法实现将Unicode编码为对应的特定编码的字符串,无法正确显示的
      - 如果是某种编码类型的str,则需要该str的编码类型,和目标终端编码匹配
        
        比如GBK的字符串,输出到windows的默认编码为GBK的cmd,则是可以正常输出的
        
        此处后来经过代码测试，就发现一个有趣或者说诡异的问题，虽然我们python文件声明的UTF-8编码，但是实际上实际上是用GBK编码，而此时，文件中的字符串，很明显是用GBK存储的，所以，将此GBK字符，输出到GBK的cmd中，是可以正常输出的。即，此处字符串的类型，很明显只和文件所用的实际编码有关，而和文件所声明的代码无关。
      - 如果是UTF-8的字符串,输出到windows的默认编码为GBK的cmd,就会出错
        
        对相应的字符，进行编码（为某种特定类型的字符str），或解码（为对应的unicode类型的字符）
        
        比如将当前的某种编码的字符串，解码为Unicode字符串
        
        很明显，也是要保证，你字符串本身的编码和所指定的编码，两者之间要一致的
        
        比如：decodedUnicode = someUtf8Str.decode(“UTF-8”)
        
        而如果用这样的：decodedUnicode = someGbkStr.decode(“UTF-8”)，那就会出现错误

2 常用方法

2.1 encode和decode

encode() unicode编码->其他编码

decode() 其他编码->unicode编码

使用这两个方法的前提是,你要知道当前是什么编码.然后用对应的编码去进行解码比如对于s字符串可以用

s.encode('utf-8') # 将s由unicode转码成utf-8
s.decode('GBK') # s是GBK编码,将s转换成unicode

2.2 isinstance()

isinstance(s , unicode) # 检测s是否是unicode编码

isinstance(s , str) # 检测s是否是str格式

3 beautifulsoup编码问题

Beautiful Soup 会按顺序尝试不同的编码将你的文档转换为Unicode：

可以通过from_encoding参数传递编码类型给soup的构造器
通过文档本身找到编码类型：例如XML的声明或者HTML文档http-equiv的META标签。
如果Beautiful Soup在文档中发现编码类型，它试着使用找到的类型转换文档。 +
但是，如果你明显的指定一个编码类型，
并且成功使用了编码：这时它会忽略任何它在文档中发现的编码类型。
通过嗅探文件开头的一下数据，判断编码。如果编码类型可以被检测到，
它将是这些中的一个：UTF-*编码，EBCDIC或者ASCII。
通过chardet库,嗅探编码，如果你安装了这个库。
UTF-8
Windows-1252

一般来说,bs的自动识别,是不会有问题的,但是在我这里除了问题,具体原因不太清楚我的网页上已经声明了是utf-8编码原来代码如下

# 这是原来的编码,在win下面乱码
self.soup = bs((response.read()))
print self.soup.originalEncoding   # 此处结果竟然返回Windows-1252

修改后代码如下

# win下乱码的关键在这里,beautifulsoup解析的编码不对,我们这里直接指定编码
self.soup = bs((response.read()), from_encoding='utf-8')
print self.soup.originalEncoding   # 修正后代码正确返回'utf-8'

看来beautifulsoup的自动识别编码不能全部依赖. 好不容易才定位到这里的问题,折腾了一天啊这里用了一个方法来检测编码.就是soup.iriginalEncoding属性

print self.soup.originalEncoding   # 修正后代码正确返回'utf-8'

这篇博客好水,毕竟不是很懂,所以说不出来

update:2015-03-30

4 新技能get

之前都没好好理解原作者的博客,觉得没办法一个程序在win和ubuntu中不更改正常运行,现在发现,如果把编码改为unicode格式输出,那么系统会自动转换,这样就不存在utf-8编码在windows下cmd乱码了.

参考文献

1 【总结】Python 2.x中常见字符编码和解码方面的错误及其解决办法 2 【已解决】python中文字符乱码（GB2312，GBK，GB18030相关的问题）参考资料太多,贴不过来啊,主要都是cifan的博客,里面资料很多,一步一步都有过程,非常好用,谢谢cifan作者