kishanbagaria / dabot

🤖 CLI tool to automate stuff on DeviantArt.com

dAbot

CLI tool to automate stuff on DeviantArt.com

Installation

1. Install Python 3 (Python 2 will work too but it's not recommended)
2. Open Terminal/Command Prompt and enter pip install dAbot
   If that doesn't work, try python -m pip install dAbot
   If that doesn't work either, you're likely using Windows and you should reinstall Python making sure that the "Add in PATH" option is checked.
3. Once dAbot is installed, you can use any of the commands listed below by entering them in Terminal/Command Prompt.

To upgrade it later, run pip install dAbot --upgrade

Usage

dAbot <cookies_txt_path> [-v] llama      give          random        (deviants|groups|exchangers)
dAbot <cookies_txt_path> [-v] llama      give          msgs          (activity|replies)           [--trash_msgs]
dAbot <cookies_txt_path> [-v] llama      give          file          (dev_names|dev_ids)          <file_path>
dAbot <cookies_txt_path> [-v] llama      give          group_members <group>                      [--reversed]
dAbot <cookies_txt_path> [-v] llama      give          url           <url>
dAbot <cookies_txt_path> [-v] llama      give          traders
dAbot <cookies_txt_path> [-v] llama      give          traders_random
dAbot <cookies_txt_path> [-v] llama      give          <deviant>
dAbot <cookies_txt_path> [-v] points     give          <deviant>     <amount>                     [<message>]
dAbot <cookies_txt_path> [-v] points     balance
dAbot <cookies_txt_path> [-v] devwatch   (add|remove)  <deviant>
dAbot <cookies_txt_path> [-v] msgs       trash         (activity|bulletins|notices|replies|comments)
dAbot <cookies_txt_path> [-v] comment    <deviant>     <comment>
dAbot <cookies_txt_path> [-v] logout
dAbot <cookies_txt_path> [-v] exec       <code>
dAbot <cookies_txt_path> [-v] llama      stats         <deviant>
dAbot <cookies_txt_path> [-v] llama      hof           group         <group_name> [--reversed]
dAbot <cookies_txt_path> [-v] llama      hof           file          <file_path>
dAbot <cookies_txt_path> [-v] llama      hof           <deviant_names>...
dAbot <cookies_txt_path> [-v] badges     hof           <deviant_names>...
dAbot <cookies_txt_path> [-v] save       random        (deviants|groups|exchangers)               <quantity>
dAbot <cookies_txt_path> [-v] save       group_members <group>
dAbot <cookies_txt_path> [-v] save       dev_ids       <dev_names_file_path>                      [--if_llama_given]

1. Install a browser extension that can export a cookies.txt file. Here's one for Chrome.
2. Go to DeviantArt.com and export the cookies.txt file for that tab only (or for DeviantArt.com only).
3. For <cookies_txt_path>, supply the path where the cookies.txt was exported/saved.
   This should look something like C:\Users\you\Downloads\cookies.txt or /Users/you/Downloads/cookies.txt
4. After exporting cookies from your browser, you can clear your browser cookies for DeviantArt.com (or all cookies) so that your cookies aren't shared between the browser and the bot (which might create issues). You'll then have to login on your browser again and this will give your browser a fresh set of cookies.

If some command doesn't work, disable Eclipse and try again.

Example

This will give me a llama:

dAbot /Users/you/Downloads/cookies.txt llama give Kishan-Bagaria

Disclaimer

I don't intend to maintain this tool (except for some occasional changes). I developed this as a personal project back in 2014 and three years later, it wasn't doing much good sitting around, so I put it out here.
It can be refactored a lot and sped up 100x using asynchronous connections. If you're a developer interested in hacking the code, you should definitely look into that.

dabot's Issues

Using command in Python

Ok. I'm bad a programming.
How can I run this through python?

os.system("say hi")       #Works
os.system("dAbot <MyUser> <MyPass> -v llama give <Username>")       #Nope. <>s have actual info

Can i send llama to my/other deviant watchers ?

1.I have a question if I can send a badge to my / other profile watchers on deviantart?
If yes - tell me how to do it ^^

2.What is the "hof"command ?

login not working

All features of dAbot have stopped working for me. Now, when attempting to login after "downloading login page" it literally outputs all html, css, js data of the dA page into the terminal and quits with the following:

Traceback (most recent call last):
  File "dAbot.py", line 803, in <module>
    main()
  File "C:\Python27\lib\site-packages\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "C:\Python27\lib\site-packages\retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "C:\Python27\lib\site-packages\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "C:\Python27\lib\site-packages\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "dAbot.py", line 667, in main
    if not login(username, password):
  File "dAbot.py", line 205, in login
    'validate_token' : get_validate_token(login_html),
  File "dAbot.py", line 178, in get_validate_token
    return r.group(1)
AttributeError: 'NoneType' object has no attribute 'group'
Press any key to exit...

🤖 Info on bot detection bypassing

I have written a patch for dAbot in #22 that allows you to bypass the bot detection.

General information

What you see with the message Please confirm you are human or similar errors, is actually part of a security measure by PerimeterX that internally uses reCAPTCHA to combat bots on DeviantArt.

This is a recent security measure introduced by Wix -- the parent company of DeviantArt -- that triggers on a score-based basis, which means you could have either made more realistic requests, or slowed down the bot enough that you wouldn't trigger the automated detection.

However, this detection seems to get more aggressive each day, and you can't do anything but to bypass it using a reCAPTCHA token.

While trying to debug my own bot, dADroid, I ran into the similar frustrating obstacle, so using a bit of reverse-engineering and looking into the PerimeterX code, I devised a simple method to bypass this piece of crap by providing the token directly to the bot.

I've ported my solution to python for dAbot, so users could also bypass the message.

How this works

You need to get a response token by solving a reCAPTCHA that has been originated on a DeviantArt page.

I have written a userscript to make this process easier, but in the meantime, you can use the manual method to gain the token.

⚠ NOTE: You need the edits in #22 in order to make this work.

HOW TO GET A reCAPTCHA TOKEN MANUALLY

These instructions are written for Chrome on Windows. You need to adapt it for your own browser.

1. Open a new incognito window (Ctrl+Shift+N)
2. Type this into address bar
   https://www.deviantart.com/users/wrong-password?url=%2fusers%2flogin&uuid=perimeterx-token&vid=
3. Hit F12 → Open Console
   (If you see a "What's new?" tab, just hit ESC e.g. Escape button.)
4. Type this and hit Enter:

window.handleCaptcha = function(token) { document.write("<pre>" + token + "</pre>"); }

5. Solve the captcha
6. Copy the response token into either dADroid or dAbot

HOW TO GET A reCAPTCHA TOKEN AUTOMATICALLY

WORK-IN-PROGRESS I am developing a userscript that allows you to acquire the token with just one click.

1. Open a new tab (doesn't need to be incognito)
2. Go to the following page:
   https://www.deviantart.com/perimeterx
3. (Once) Install the required user-script if you haven't
4. Solve the captcha
5. Copy the response token into either dADroid or dAbot

Conclusions

Please post comments to this issue if you need additional help, or encounter any errors.

I'd be happy to hear your feedback if this method proves to be successful and solves your issue!

Llamas to group members

When i try send lama to group_members:

Traceback (most recent call last):
File "d:\python27\lib\runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "d:\python27\lib\runpy.py", line 72, in run_code
exec code in run_globals
File "D:\Python27\Scripts\dAbot.exe_main.py", line 9, in
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 812, in main
run()
File "d:\python27\lib\site-packages\retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "d:\python27\lib\site-packages\retrying.py", line 206, in call
return attempt.get(self._wrap_exception)
File "d:\python27\lib\site-packages\retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "d:\python27\lib\site-packages\retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 710, in run
for m in get_group_members(args[''], args['--reversed']):
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 459, in get_group_members
end_offset = int(re.search(regex['group_last_offset'], html).group(1))
AttributeError: 'NoneType' object has no attribute 'group'
Press any key to exit...

So the bot doesn't work anymore right?

Seems DeviantArt has recently blocked all tools like this one. I get a message "Access to this page has been denied because we believe you are using automated tools to log into and browse DeviantArt."

Like in an old issue here: #13 and here: #19

Is there anything we can do?

Traceback (most recent call last):

Hi!
Today i have this problem:
I reinstaling python, dabot and trying on another machine. Thanks for help.

Traceback (most recent call last):
File "d:\python27\lib\runpy.py", line 174, in run_module_as_main
"main", fname, loader, pkg_name)
File "d:\python27\lib\runpy.py", line 72, in run_code
exec code in run_globals
File "D:\Python27\Scripts\dAbot.exe_main.py", line 9, in
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 811, in main
init()
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 671, in init
load_data()
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 594, in load_data
LlamaTransactions = read_llama_transactions()
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 567, in read_llama_transactions
return set(json.load(f))
File "d:\python27\lib\json_init.py", line 287, in load
return loads(fp.read(),
EOFError: compressed file ended before the logical end-of-stream was detected

dA did it again...

Just tried running the bot and dA managed to find out that it is not a browser doing the requests again.
Tried changing the UserAgent strings in the code to see if it would fix the issue like last time but it seems to be something different now.
Interestingly i was able to login in using an incognito tab with disabled JS...
The HTML of the login form didn't show any sign of google recaptcha apart from an attribute data-sigil="recaptcha-form" on the form tag and references in js files.
Google's ReCaptcha can't be used without Javascript enabled though (at least the Invisible one) so i don't have a single clue what's going on here...
Maybe the HTML that the bot gets upon GETting the login form is already different from what browsers users will get?

Regarding bypass chegg recapcha

I developed a simmillar bot for scraping answers and used 2capcha to bypass recapcha , there was a callback function call "handlecapcha" which i used to callback and bypass capcha. This doesnt seem to work now. any idea Why?

Script stopped working with specific username

I've been using this for giving llamas for a while and it was working fine, but now it throws me the following error:

Traceback (most recent call last):
File "c:\python27\lib\runpy.py", line 174, in run_module_as_main
"main", fname, loader, pkg_name)
File "c:\python27\lib\runpy.py", line 72, in run_code
exec code in run_globals
File "C:\Python27\Scripts\dAbot.exe_main.py", line 9, in
File "c:\python27\lib\site-packages\dAbot\dAbot.py", line 817, in main
init()
File "c:\python27\lib\site-packages\dAbot\dAbot.py", line 677, in init
load_data()
File "c:\python27\lib\site-packages\dAbot\dAbot.py", line 598, in load_data
LlamaTransactions = read_llama_transactions()
File "c:\python27\lib\site-packages\dAbot\dAbot.py", line 571, in read_llama_transactions
return set(json.load(f))
File "c:\python27\lib\json_init.py", line 291, in load
**kw)
File "c:\python27\lib\json_init_.py", line 339, in loads
return _default_decoder.decode(s)
File "c:\python27\lib\json\decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "c:\python27\lib\json\decoder.py", line 380, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting object: line 1 column 221152 (char 221151)
Press any key to exit...

The error happens only when I enter my username. I tried typing different usernames and in those cases the script works fine again, it just throws the "wrong password" error which is to be expected. It isn't a problem after logging in either because if I type my username with a wrong password, it crashes again before even trying to log in. I can run the script properly in another computer too (same internet IP), so I'd guess it's not something from deviantart but an issue with the script.

I've tried reinstalling Python and the script again but that doesn't fix it either. Any idea what could the issue be?

"Please confirm you are human." message blocks bots

DeviantArt's "Please confirm you are human." message prevents dAbot from working. It appears when you try to login. The full message is:

"Access to this page has been denied because we believe you are using automated tools to log into and browse DeviantArt.

This may happen as a result of the following:

• Javascript is disabled or blocked by an extension. Please click here to learn how to re-enable Javascript, and avoid using extensions and plugins that hinder necessary parts of the website from loading.
• Your browser does not support cookies or you have disabled them. Click here for information on how to re-enable cookies.

Please make sure Javascript and cookies are enabled on your browser in order to browse DeviantArt."

User collector

Can dAbot collect dev username and save to file?

Thanks !!

Spam Filter

Hey there i've been a faithful user of the OCLB for a while now and just found out about this bot.
Been running it for a few hours now to give llamas to random deviants and so far it looks like it is able to throw out about 19 llamas every half an hour.

Before i knew about this nifty CLI tool i've developed a GreaseMonkey script which would run through a memberlist of a group and automatically click every .oclb-give and continue to the next page when done.
However my script clicked a single button per minute and waited for 5 minutes if it reached the limit.

Unfortunately i didn't really calculate my script's 'llamas per hour' for comparison.
I was wondering if you did a bit of research wether you are getting 'punished' if you hit the spam limit.
Maybe there's a chance that putting in delays between the giving of llamas results in a higher llamas/hour rate.

About "llama give random exchangers"

So i've been running that command for a bit now and i came up with an idea to improve that process.
Currently i'm running it in 2 windows simultanously because it's kinda slow to find exchangers (which didn't receive a llama from me yet) and that way i'm actually sometimes able to hit the spam limit.
I guess this would be a rather drastic change to the codebase but the whole thing could be more effective if there'd be multiple processes running: One (or more) finding suitable exchangers and putting them into a queue and the other one actually giving llamas to the people in the queue.
This came to my mind when i saw that dAbot stopped doing anything for half an hour even though it could theoretically search for exchangers in the meantime.

urlparse

In dAbot.py, there's a line that says "import urlparse"

After the new Python update, urlparse has been moved to urllib.parse

So..... now what?

Also, I have python 2 and 3, and it chooses to run with 3. How do I make it run with 2?

@KishanBagaria

KeyError: 'location'

When running 'llama give random deviants' it throws this error:

Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Programs\Python\Python38-32\Scripts\dAbot-script.py", line 11, in
load_entry_point('dAbot==2019.9.24.2', 'console_scripts', 'dAbot')()
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\dAbot\dAbot.py", line 812, in main
run()
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\retrying.py", line 206, in call
return attempt.get(self._wrap_exception)
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\six.py", line 703, in reraise
raise value
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\dAbot\dAbot.py", line 680, in run
while True: give_llama_to_deviant(get_random('deviant'))
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\dAbot\dAbot.py", line 197, in get_random
parsed = urlparse(get_redirected_url(url['random'] + what))
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\dAbot\dAbot.py", line 194, in get_redirected_url
return dA.head(url).headers['Location']
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\requests\structures.py", line 52, in getitem
return self._store[key.lower()][1]
KeyError: 'location'

Issues - gathering User names

I understand that dabot can not find username in urls,
Because deviantart change username.deviantart.com to deviantart.com/username
This is also visible in the source of the page, on the list with deviants

I find this in code:
'dev_names' : r'([A-Za-z0-9-]+).deviantart.com',

Its not correct i think... could you help me change this? (i know i sucks)
'dev_names' : r'\deviantart.com([A-Za-z0-9-]+)',

Quick Questions

@KishanBagaria
How would I
a) edit watch settings for a user and
b) fetch the last date a user was online
through code?
C) Also I think the server is backed up

Propably captcha ?

Access to this page has been denied because we believe you are using automated tools to log into and browse DeviantArt.

This may happen as a result of the following:

• Javascript is disabled or blocked by an extension. Please click here to learn how to re-enable Javascript, and avoid using extensions and plugins that hinder necessary parts of the website from loading.
• Your browser does not support cookies or you have disabled them. Click here for information on how to re-enable cookies.

Please make sure Javascript and cookies are enabled on your browser in order to browse

Slowing down?

Since the last 2 months, the speed that the code runs on seemed to... slow down by about 4 times?
It's not a wifi or computer issue, as it's the same on other devices and other networks

Script halting when working with usernames containing hyphens on linux.

Over the last months running dAbot on linux i ran into an issue regarding usernames that have hyphens at the beginning or end of their usernames. The script halted with the console saying this:
Retrying... ConnectionError(MaxRetryError("HTTPSConnectionPool(host='shaft-.deviantart.com', port=443): Max retries exceeded with url: /badges/llama/ (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9fba1693d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))",),)
After investigating the issue i found out that deviantArt's subdomain system for usernames is actually producing invalid subdomains according to RFC1035 ("They must start with a letter, end with a letter or digit, and have as interior characters only letters, digits, and hyphen.").
While Windows doesn't seem to have an issue with resolving domains like these, linux does.
As a simple solution to this issue while preserving the ability to theoretically give users with invalid subdomains llamas i'd like to propose a try-catch in the loops (https://github.com/KishanBagaria/dAbot/blob/master/dAbot/dAbot.py#L697) which would ignore the "invalid" users and continue with the next one without halting.

Bug due to https-migration

When i logged into my computer this morning i saw several errors in my terminals stating that they couldn't request https:///someuser.deviantart.com
I've looked a bit through the code and saw that the username variables had a leading slash due to incorrect substringing here.

Replacing the 7 (http://) with an 8 (https://) fixed it for me.

DAThrottlingError()

12 Jul 2017
07:37:37.076 PM [Llama Transactions] 0
07:37:37.076 PM Checking if logged in as kay-ra
07:37:37.435 PM Downloading login page
07:37:38.826 PM Logging in as kay-ra
07:37:39.390 PM Retrying... DAThrottlingError()

What does it mean and how can it be solved?
Thank you !