如何将response中的metadata传递给item about ruia

[howie6879]

能描述清楚一点么？

[YISION]

我在一个页面在抓取到urllist，在这个页面上可以看到该url的简介，然后我再请求这个url的详情，想把简介和详情信息组合起来，我现在是利用response.metadata传过去了,不知道这样合不合理

from ruia import *
from ruia_ua import middleware
import aiofiles

class CatalogueItem(Item):
target_item = TextField(css_select="div.info")
title = TextField(css_select="div.title",default="")
link = AttrField(css_select="a", attr="href")

class PageItem(Item):
target_item = TextField(css_select='div.msg')
info = TextField(css_select='span>label',many=True,default="")

class LianjiaSpider(Spider):
start_urls = ["https://bj.lianjia.com/chengjiao/pg"+str(i)+"/" for i in range(1,2)]
request_config = {"RETRIES": 3, "DELAY":1, "TIMEOUT": 20}
concurrency = 20

async def parse(self, response: Response):
    catalogue = []
    async for cat in CatalogueItem.get_items(html=response.html):
        catalogue.append(cat)
    urls = [page.link for page in catalogue]
    async for response in self.multiple_request(urls, is_gather=True):
        title = catalogue[response.index].title
        link = catalogue[response.index].link
        response.metadata.update({"title":title,"link":link})
        yield self.parse_page(response)

async def parse_page(self, response):
    async for item in PageItem.get_items(html=response.html):
        self.metadata.update(response.metadata)
        yield item

async def process_item(self, item):
    print (self.metadata)
    print (item.info)

if name == "main":
LianjiaSpider.start(middleware=middleware)

[howie6879]

self.metadata是什么东西。

async def parse(self, response: Response):
    catalogue = []
    async for cat in CatalogueItem.get_items(html=response.html):
        catalogue.append(cat)
    urls = [page.link for page in catalogue]
    async for response in self.multiple_request(urls, is_gather=True):
        title = catalogue[response.index].title
        link = catalogue[response.index].link
        response.metadata.update({"title":title,"link":link})
        yield self.parse_page(response)

async def parse_page(self, response):
    async for item in PageItem.get_items(html=response.html):
        first_info = response.metadata
        second_info = item.info

这样就行了

[YISION]

因为想在process_item里面一起处理 后面入库

[howie6879]

因为想在process_item里面一起处理 后面入库

按照让上面说的那样也可以实现，如果非要到process_item，可以将response.dat赋值给item的属性

[YISION]

嗯好的 学习了