rmax / dirbot-mysql

Scrapy project based on dirbot to show how to use Twisted's adbapi to store the scraped data in MySQL.

Highlights

👉🏽 Watch my video on How Syntheric Media is Changing the World

👉🏽 Watch my own journey From Passion to Purpose with Python 🎬

👉🏽 Watch my epic story of The Genesis of ChatGPT 🎬

👉🏽 Read my poem to The Majestic Illimani 🗻

👉🏽 Read my vision of The Titikaka Lake in the year 3012 🔮

👉🏽 Read my epic tale of Tupac Katari🗿

👉🏽 Read my epic tale of St Patrick 🍀

👉🏽 Follow me on Twitter at @rmaxdev 🐥

About

🙋🏽‍♂️ Max is a highly skilled software engineer with experience in building production services for AI systems. He is currently working at an AI company in the Netherlands, where he is known for his ability to stay current with the latest advances in the field of software engineering and his dedication to delivering high-quality results. In addition to his technical expertise, Max is also committed to maintaining a healthy work-life balance through activities like capoeira and yoga.

👨🏽‍💻 As a successful software engineer with over 10 years of experience, he specialize in machine learning, distributed systems, and large scale data processing using Python. He has a proven track record of delivering successful projects and is passionate about using his skills and knowledge to help organizations to achieve their ambitious goals.

🚀 Max has been a founding member of successful startups and have contributed to open source projects and Linux communities. He keeps a deep understanding of the latest trends and best practices in the field and is always looking for new opportunities to learn and grow.

💪🏽 In his current role, Max coordinates with key stakeholders and product owners to ensure that projects are delivered on time and to the highest standards. He a strong communicator and works well in both team and individual settings. Max is also highly adaptable and can quickly learn and master new technologies and concepts.

🙌🏽 If you are looking for a dedicated and experienced software engineer with a focus on AI systems, please don't hesitate to reach out to Max. He will be excited to hear from you and discuss how AI can change the world for better.

👉🏽 Keywords: #Python #PyTorch #SoftwareEngineering #DistributedSystems #MachineLearning #MLOps #DeepNeuralNetworks #LargeModels

dirbot-mysql's Issues

Populating DBs; roxy_ui vs memex_uncharted

Not so much of an issue, but more a question about how to best begin getting data into the environment.

I'm looking at getting a scraper / pipeline set up to begin populating the DB. From looking at the schemas, it looks like roxy_ui is for scraped data whereas memex_uncharted is more for rules and context. Is this correct? Any advice would be much appreciated!

AttributeError: 'list' object has no attribute 'items'

scrapy list
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/bin/scrapy", line 11, in
sys.exit(execute())
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/cmdline.py", line 108, in execute
settings = get_project_settings()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/utils/project.py", line 60, in get_project_settings
settings.setmodule(settings_module_path, priority='project')
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/settings/init.py", line 285, in setmodule
self.set(key, getattr(module, key), priority)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/settings/init.py", line 260, in set
self.attributes[name].set(value, priority)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/settings/init.py", line 55, in set
value = BaseSettings(value, priority=priority)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/settings/init.py", line 91, in init
self.update(values, priority)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/settings/init.py", line 317, in update
for name, value in six.iteritems(values):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/six.py", line 581, in iteritems
return iter(d.items(**kw))
AttributeError: 'list' object has no attribute 'items'

ImportError: Error loading object 'googledir.pipelines.FilterWordsPipeline': No module named exceptions

Hi, its not working anymore.

Traceback (most recent call last):
File "./scrapy-ctl.py", line 7, in
execute()
File "/usr/lib/python2.6/site-packages/scrapy/cmdline.py", line 143, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/usr/lib/python2.6/site-packages/scrapy/cmdline.py", line 88, in _run_print_help
func(_a, *_kw)
File "/usr/lib/python2.6/site-packages/scrapy/cmdline.py", line 150, in _run_command
cmd.run(args, opts)
File "/usr/lib/python2.6/site-packages/scrapy/commands/crawl.py", line 46, in run
spider = self.crawler.spiders.create(spname, **opts.spargs)
File "/usr/lib/python2.6/site-packages/scrapy/command.py", line 34, in crawler
self._crawler.configure()
File "/usr/lib/python2.6/site-packages/scrapy/crawler.py", line 44, in configure
self.engine = ExecutionEngine(self, self._spider_closed)
File "/usr/lib/python2.6/site-packages/scrapy/core/engine.py", line 63, in init
self.scraper = Scraper(crawler)
File "/usr/lib/python2.6/site-packages/scrapy/core/scraper.py", line 66, in init
self.itemproc = itemproc_cls.from_crawler(crawler)
File "/usr/lib/python2.6/site-packages/scrapy/middleware.py", line 50, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "/usr/lib/python2.6/site-packages/scrapy/middleware.py", line 29, in from_settings
mwcls = load_object(clspath)
File "/usr/lib/python2.6/site-packages/scrapy/utils/misc.py", line 40, in load_object
raise ImportError, "Error loading object '%s': %s" % (path, e)
ImportError: Error loading object 'googledir.pipelines.FilterWordsPipeline': No module named exceptions

Problem with the order of the parameters

I was trying the MySQL pipeline and It wasn't working for me.
In the process_item you pass as first parameter the item and second the spider, and scrapy(at least 0.14) expect : item, spider.
http://doc.scrapy.org/en/0.14/topics/item-pipeline.html#process_item
def process_item(self, item, spider):
instead of
def process_item(self, spider, item):

Anyway, thanks for share this code.

Commands out of sync

I am using your code snippet and getting next error:

--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/enterprise/adbapi.py", line 452, in _runInteraction
    conn.rollback()
  File "/usr/lib/python2.7/dist-packages/twisted/enterprise/adbapi.py", line 49, in rollback
    self._connection.rollback()
_mysql_exceptions.ProgrammingError: (2014, "Commands out of sync; you can't run this command now")

Maybe you know how can i solve this?