Coder Social home page Coder Social logo

rybodiddly / kijiji-reposter Goto Github PK

View Code? Open in Web Editor NEW
32.0 6.0 11.0 320 KB

Kijiji Automated Reposting and Message Replying Utility written in Python with a Flask based GUI

License: MIT License

Python 64.90% HTML 25.44% CSS 9.66%
kijiji reposter automation ad listing

kijiji-reposter's Introduction

Kijiji-Reposter

Kijiji Automated Reposting and Replying Utility written in Python (version 3). The reposter is completely api driven (i.e. no webscraping). Utilizes Flask, to run a local server designed to manage the GUI interface and reposting / scheduling and auto reply functions. Viable for both server and desktop environments, but if desktop, system must run 24/7 and have sleep functions disabled.

Recent Updates:

  • fixed ad geo location
  • fixed eBay image upload api token
  • added search functionality
  • dynamic categories, locations & attributes
  • added support for optional attributes (fullfilment, cashless etc.)
  • added token retention system
  • added conversations support
  • fixed message auto replier bugs
  • absolute paths incorporated for wider compatibility
  • retry failed attempts

Requirements:

apscheduler (if you get pytz error, it can be ignored, but you can avoid by using tzlocal 2.1)
flask
flask-wtf (Install WTForms 2.3.3 first, else it will break html5 import)
httpx
pgeocode
xmltodict
urllib3

Usage:

Edit the secret key argument on line 25 in server.py. Then run:

python server.py

Connections:

Once server is running, connect to either of the addresses listed below in a web browser. Or if running on a network, connect to the ip and port 5000 of computer running the kijiji reposter server.

localhost:5000/
127.0.0.1:5000/

Accounts:

Login using an existing kijiji account. Or if you do not have an account, create one at kijiji.ca.

Reposting:

To create a reposting schedule, begin creating an ad by clicking the 'Post' icon at the top of the home screen. While entering the ad details, make sure to check the repost checkbox and enter the reposting times (eg. 07:00 am, 1:30 pm). Currently, only 8 reposting slots have been implemented. But you can edit the server.py code to allow for more.

Auto Replier:

The auto replier scans your account for new messages, and if a new message is found and contains any word or phrase (not case sensitive) defined as a 'rule', it will automatically send the associated 'response'. When setting up a new rule, enter the desired, rule and response along with your password, as it will be required by the auto replier when logging in to check recent messages. Example usage:

Rule:     Hi, is this still available?
Response: Yes, it's still available.

Rule:     Is the price negotiable?
Response: No, it's not negotiable.

Currently the auto replier checks your messages every 25 minutes if rules have been created. You can adjust the timing by editing the following code on line 1690:

sched.add_job(messageAutoReplier,'cron',minute='*/25')

Change the */25 to any number of your choosing, example: */6

Please note that using the auto replier will mark messages as read, meaning that when checking your messages manually they will not appear as new / unread. If you do not desire this functionality, and would like to deactivate the autoreplier, simply comment out line 1690 as shown in the code below:

#sched.add_job(messageAutoReplier,'cron',minute='*/25')

Token Retention System:

It was discovered that tokens remain valid almost indefinitely. To avoid congestion and redundant login attempts, tokens are retained for one day. This setting can be altered by editing the math on line 88 in kijijiapi.py:

expiryTime = int(time.time()) + (24 * 60 * 60) # 24hrs, 60mins, 60secs = 1 day

Force Post Ad from File:

If you require the ability to force post an ad from file due to botched reposting, accidental deletion or other strange circumstances, you can access the force post function at localhost:5000/force. Note that you will first need to have an ad file saved in the users folder, both of which would have been created when initially posting an ad with a reposting schedule, and two, you will also be required to manually update the 'current_ad_id' field in the schedules.json file after the forced repost if a reposting schedule exists.

ToDo:

  • implement async
  • impliment notification functionality
  • basic bug fixes / improvements

kijiji-reposter's People

Contributors

rybodiddly avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

kijiji-reposter's Issues

ModuleNotFoundError: No module named 'pgeocode'

I recently did a fresh installation on Windows 10 and also on a Raspberry Pi

For both installations, I had an error on the first execution. (error: ModuleNotFoundError: No module named 'pgeocode')

pi@raspberrypi:~/Kijiji-Reposter $ python server.py
Traceback (most recent call last):
  File "/home/pi/Kijiji-Reposter/server.py", line 5, in <module>
    import pgeocode
ModuleNotFoundError: No module named 'pgeocode'
pi@raspberrypi:~/Kijiji-Reposter $

I resolve the issue by installation the package 'pgeocode' with the following command:
pip3 install pgeocode

I think the pgeocode package should probably be added in requirements.txt.

AutoReplier raised an exception

Job "messageAutoReplier (trigger: cron[minute='*/1'], next run at: 2022-08-31 09:39:00 EDT)" raised an exception
Traceback (most recent call last):
  File "C:\Users\Vostro\AppData\Local\Programs\Python\Python310\lib\site-packages\apscheduler\executors\base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "C:\Users\Vostro\Desktop\UI-Reposter\server.py", line 398, in messageAutoReplier
    conversationID = item['@uid']
TypeError: string indices must be integers

Kijiji Max Attribute Lengths

When posting an ad an error is received if the title is greater than Kijiji's max length. There is a minimum of 8 characters, and a maximum of 64.

Idea for expanding project

Would you be willing to accept a push to include functions for automatically scraping and filtering listings from Kijiji and notifying end user of new listings which pass the filter? You can see this more in-depth in my project if you want. I'll rewrite the code to Python and I think it'd be a cool idea to have this tool that can do all these useful features.

[FEATURE] Running via script

This is an awesome project! Would love to see something thats runnable via script instead of having to bring up the entire server.

E.g reposting an ad by python3 repost my_ad.yml. The content of my_ad.yml could be just be all the attributes that get submit when using kijijis website (can post the ad once manually, look at the network console, and copy everything over). If all of that is converted into your submitFunction that would be really slick.

Another approach would be to document what the payload of the submitFunction should look like so people can create their own custom ads to pass to it. However, since different ads have different attributes it would be hard to cover all the bases

Coverting in No-SQL Database

Hi,

I want to convert all the data coming from XML and JSON files data into MongoDB. I think that will more efficient to handle data. Do you have any idea how I am gonna start with this one??

New Kijiji Update Broke LoginFunction?

I was writing my API and playing around with it when I noticed that login through mobile API no longer works. I tried this function using this repo and the same error happened. It said 'Sorry, your username or password is incorrect. Please try again.'. I tried logging in through Kijiji manually with the same creds and it worked just fine (not mobile, website instead). It's possible this is only on my end but give it a shot with your creds and see if the same error message occurs. On a side note, my token was invalidated which is why I tried to do this and it failed. Also if you know the endpoint for checking token validity please let me know what it is.

TypeError: __init__() got an unexpected keyword argument 'connect_timeout'

Hello, running ubuntu 18. Using Python 3.6.9
This looking to be related to http2.0 anyone else having a problem with this?

$ python server.py
Traceback (most recent call last):
  File "server.py", line 450, in <module>
    timeout = httpx.Timeout(15.0, connect_timeout=30.0)
TypeError: __init__() got an unexpected keyword argument 'connect_timeout'

Getting Error in post tab

can't retrieve the post data. here is error msg:

File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/flask/app.py", line 2464, in call
return self.wsgi_app(environ, start_response)
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/flask/app.py", line 2450, in wsgi_app
response = self.handle_exception(e)
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/flask/app.py", line 1867, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functionsrule.endpoint
File "/Users/catalina/Workspace/Python/Kijiji-Reposter/server.py", line 274, in home

if no notification is sent after some time and there's some thread already stopped.

File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/flask/templating.py", line 140, in render_template
ctx.app,
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/flask/templating.py", line 120, in _render
rv = template.render(context)
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/Users/catalina/Workspace/Python/Kijiji-Repost-Headless/venv/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "/Users/catalina/Workspace/Python/Kijiji-Reposter/templates/home.html", line 2, in top-level template code
{% extends 'layout.html' %}
File "/Users/catalina/Workspace/Python/Kijiji-Reposter/templates/layout.html", line 22, in top-level template code
{% block content %}{% endblock %}
File "/Users/catalina/Workspace/Python/Kijiji-Reposter/templates/home.html", line 37, in block "content"

{% if item['@id']|checkSchedule == true %}{% endif %} File "/Users/catalina/Workspace/Python/Kijiji-Reposter/server.py", line 1134, in checkSchedule info.pydev_step_stop = None FileNotFoundError: [Errno 2] No such file or directory: 'static/schedules.json'

on posting ads

TypeError
TypeError: 'NoneType' object is not subscriptable

Traceback (most recent call last)
File "d:\Code\kjjreposter\venv\lib\site-packages\flask\app.py", line 2548, in call
return self.wsgi_app(environ, start_response)
File "d:\Code\kjjreposter\venv\lib\site-packages\flask\app.py", line 2528, in wsgi_app
response = self.handle_exception(e)
File "d:\Code\kjjreposter\venv\lib\site-packages\flask\app.py", line 2525, in wsgi_app
response = self.full_dispatch_request()
File "d:\Code\kjjreposter\venv\lib\site-packages\flask\app.py", line 1822, in full_dispatch_request
rv = self.handle_user_exception(e)
File "d:\Code\kjjreposter\venv\lib\site-packages\flask\app.py", line 1820, in full_dispatch_request
rv = self.dispatch_request()
File "d:\Code\kjjreposter\venv\lib\site-packages\flask\app.py", line 1796, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "d:\Code\kjjreposter\Kijiji-Reposter\server.py", line 978, in submit
adID = parsed['ad:ad']['@id']
TypeError: 'NoneType' object is not subscriptable
The debugger caught an exception in your WSGI application. You can now look at the traceback which led to the error.
To switch between the interactive traceback and the plaintext one, you can click on the "Traceback" headline. From the text traceback you can also create a paste of it. For code execution mouse-over the frame you want to debug and click on the console icon on the right side.

You can execute arbitrary Python code in the stack frames and there are some extra helpers available for introspection:

dump() shows all variables in the frame
dump(obj) dumps all that's known about the object

Posting new ad fails with using certain locations

When posting a new ad, choosing certain geographic locations fails to post the ad.

This seems to happen because some locations are nested up to 4 levels deep (Country > Province > Metro Area > City) and others are just 3 levels deep (Country > Province > City). For example, nested under the "Ottawa / Gatineau Area" location are "Gatineau" and "Ottawa" locations.

The UI doesn't let you select the the most nested location (e.g. just "Ottawa" won't be an option), and when it attempts to submit the new ad to Kijiji, it rejects it and the API error message says that you must select a location. It would appear that it wants the most nested location IDs and not an ID further up. For example, if you attempt to post an ad to the "Ottawa / Gatineau Area" location, it will fail.

urllib.error.URLError

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>

I get that error every time I try to make a post

Add tags to the listing

Category: Services -> Tutoring

I cannot add any tags to my listing using the Post page.

Tags are crucial to pop up in the Kijiji search! Also, a one repost button would be helpful too!

Incorrect Location via Postal Code

So the ad posts perfectly fine and includes the postal code, But when you open the ad and click on the postal code that is there it directs you to the middle of the ocean.

Manually updating the postal code corrects the issue and i see that it includes the and province as well, not sure if some extra validation needs to take place

Unable to post ads with pictures

Upon posting the ad I get a TypeError: 'NoneType' object is not subscriptable. In the terminal I see this message:

OrderedDict([('api-base-error', OrderedDict([('@xmlns', 'http://www.ebayclassifiedsgroup.com/schema/types/v1'), ('@http-status-code', '400'), ('api-errors', OrderedDict([('api-error', OrderedDict([('message', 'One of your images is not on an accepted service. Please upload your images again.')]))])), ('api-debug-errors', None), ('api-field-errors', None)]))])
127.0.0.1 - - [21/Aug/2022 23:47:14] "←[35m←[1mPOST /submit HTTP/1.1←[0m" 500 -

Reposting doesn’t work

Ad posted fine, but reposting at 8:00 AM and 5:00 PM both missed by the app. Any particular fix?

Unable to post pictures

When I try to create an ad with pictures I get a TypeError. Without pictures everything works properly. Do the pictures have to be in a specific directory? Thanks.

Cannot post new ad; API error "Cannot unmarshal content"

I attempted to post a new ad from the /post page after logging into the web interface, but this fails and Flask dumps the stack trace.

The problem is after during the call to submitFunction, the Kijiji site returns an API error message: "Cannot unmarshal content" with HTTP status code 400.

[FR] Docker

Any chance of having this available in a docker container?

Raspberry Pi error: numpy.ndarray size changed

I tried to install Kijiji-Reposter on a Raspberry Pi.

I get the following error when I start the program:
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 44 from C header, got 40 from PyObject

Details:

pi@raspberrypi:~/Kijiji-Reposter $ python server.py
Traceback (most recent call last):
  File "/home/pi/Kijiji-Reposter/server.py", line 5, in <module>
    import pgeocode
  File "/home/pi/.local/lib/python3.9/site-packages/pgeocode.py", line 14, in <module>
    import pandas as pd
  File "/home/pi/.local/lib/python3.9/site-packages/pandas/__init__.py", line 22, in <module>
    from pandas.compat import (
  File "/home/pi/.local/lib/python3.9/site-packages/pandas/compat/__init__.py", line 15, in <module>
    from pandas.compat.numpy import (
  File "/home/pi/.local/lib/python3.9/site-packages/pandas/compat/numpy/__init__.py", line 7, in <module>
    from pandas.util.version import Version
  File "/home/pi/.local/lib/python3.9/site-packages/pandas/util/__init__.py", line 1, in <module>
    from pandas.util._decorators import (  # noqa
  File "/home/pi/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 14, in <module>
    from pandas._libs.properties import cache_readonly  # noqa
  File "/home/pi/.local/lib/python3.9/site-packages/pandas/_libs/__init__.py", line 13, in <module>
    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 44 from C header, got 40 from PyObject
pi@raspberrypi:~/Kijiji-Reposter $

I suppose that the Raspberry Pi was probably not a 'target device'. I just want to report the problem in case a simple/quick fix exists...
It is probably related to processor architecture (32 bits versus 64 bits).

Type error when posting new ad

Hello,

This looks great for reposting Kijiji ads but I am getting an error when trying to add a new add/post.

I am running this on a windows server in an Anaconda Python 3.7 environment with the required dependencies.

The error is:

TypeError: 'NoneType' object is not subscriptable

Stack trace:

File "C:\Users\Administrator\anaconda3\envs\Kijiji2\lib\site-packages\flask\app.py", line 2464, in __call__ return self.wsgi_app(environ, start_response) File "C:\Users\Administrator\anaconda3\envs\Kijiji2\lib\site-packages\flask\app.py", line 2450, in wsgi_app response = self.handle_exception(e) File "C:\Users\Administrator\anaconda3\envs\Kijiji2\lib\site-packages\flask\app.py", line 1867, in handle_exception reraise(exc_type, exc_value, tb) File "C:\Users\Administrator\anaconda3\envs\Kijiji2\lib\site-packages\flask\_compat.py", line 39, in reraise raise value File "C:\Users\Administrator\anaconda3\envs\Kijiji2\lib\site-packages\flask\app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "C:\Users\Administrator\anaconda3\envs\Kijiji2\lib\site-packages\flask\app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "C:\Users\Administrator\anaconda3\envs\Kijiji2\lib\site-packages\flask\app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "C:\Users\Administrator\anaconda3\envs\Kijiji2\lib\site-packages\flask\_compat.py", line 39, in reraise raise value File "C:\Users\Administrator\anaconda3\envs\Kijiji2\lib\site-packages\flask\app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "C:\Users\Administrator\anaconda3\envs\Kijiji2\lib\site-packages\flask\app.py", line 1936, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "P:\Scripts\Kijiji2\Kijiji-Reposter\server.py", line 555, in submit Open an interactive python shell in this frameadID = parsed['ad:ad']['@id']

Listing fails to stay active after a 'successful' repost

For some listings, the application attempts to repost them, and it successfully does so (it even gets the new listing id from the server) but after a few seconds, it is taken down from kijiji.

It only happens with some listings, under the same account. It is very difficult to tell what is going on because again, the listing does get posted successfully for a couple of seconds. Right now failing under the category 15, but category 760 seems to be working fine.

Does anybody know what could be causing this issue? As far as the application is concerned, the listing gets reposted succesfully so its very difficult to debug.

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired

I tried to install Kijiji-Reposter on Windows 10 (64 bits)

I get the following error when I post my first add.

urllib.error.URLError
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)>

image

For now, as a quick ugly fix, I added the following two lines in the server.py

At line 19:
import ssl

At line 27 (just after app.secret_key):
ssl._create_default_https_context = ssl._create_unverified_context

I'm not sure how to resolve this issue to correctly use SSL with Windows 10.
Any suggestions ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.