defaultnamehere / zzzzz Goto Github PK

Have other experienced unexpected crashes of fetcher.py?

when I load facebook.com/pull It says Page not found, neither I get client_id in browser GET request, am I missing something?

Here is the full error:

/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

Is it just me or everyone is having this?

Hi there guys,

I'm trying this out, but don't have any programming experience.. I've installed phyton 3.9, but never used this before. Can someone explain me step-by-step how to get this script working? As for the secrets.txt, how should the cookie information in this txt file like like..maybe an example?

I've ran the fetcher.py script but with next output error:
Traceback (most recent call last):
File "\fetcher.py", line 6, in
import requests
ModuleNotFoundError: No module named 'requests'

any idea?

thank for your patience ;-)

MK

As title says, would be nice to have an extra script that just grabs your friend list and dumps it to a file for easy Name->UID mapping.

Many projects include a Dockerfile which allows other folks to quickly run the project in a controlled environment.

Provide one for this project.

Tasks:

• select base OS.
• discover system packages which are required for this package to run.
• define setup process.
• provide working examples (copy-n-paste-ready) of this package in action.
  • if API keys or authentication tokens, etc are required, then provide detailed instructions on how to acquire them, and where they should be applied for a successful run of this software.

Traceback (most recent call last):
  File "./fetcher.py", line 182, in <module>
    f.start_request()
  File "./fetcher.py", line 97, in start_request
    resp = self.make_request()
  File "./fetcher.py", line 54, in make_request
    response_obj = requests.get(url, params=self.params, headers=self.REQUEST_HEADERS)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 447, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: SysCallError(104, 'ECONNRESET')",)

This should not result in a crash but in a retry

So when I run python fetcher.py I just repeatedly get this line in the console:

Response:{u'reason': 110, u't': u'refresh', u'seq': 0}

Can anyone shed any light on what it is and how to fix it? Nothing is going into the log directory at all.

If I fiddle around with the SECRETS.txt file and make something invalid it fails for different reasons, so I can only assume the cookie and IDs are working ok.

This is on Windows 7 if that makes a difference...

Hello,

you should note python-dev and libssl-dev as requirement. I had some SSL issues which I got rid by installing both and going for

pip install 'requests[security]'

afterwards. This got me going.

However now I get the following response. Not sure where the request is failing.

#python fetcher.py
>
Got error from request, restarting...
>
Got error from request, restarting...
>
Got error from request, restarting...
>
Got error from request, restarting...
^CTraceback (most recent call last):
  File "fetcher.py", line 177, in <module>
    time.sleep(SLEEP_TIME)
KeyboardInterrupt

I would debug the HTTP requests but it is all TLS1.2 encrypted. Any idea how to proceed?

unable to obtain client id, gives authentication error.

Probably same result can be achieved via official api in similar way Pidgin uses.

I keep getting the "Got error from request, restarting..."
and facebook seems like changed the response

"buddyList":{"id":{"lat":1462613641},"id":{"lat":1462687695},"id":{"lat":1462694839},"id":{"lat":0},"id":{"lat":1462798728,"p":2,"vc":10}

this is the error message:

Traceback (most recent call last):
  File "./fetcher.py", line 180, in <module>
    f.start_request()
  File "./fetcher.py", line 122, in start_request
    self._log_lat(uid, str(item["overlay"][uid]["la"]))
KeyError: 'la'

I "fixed" this by checking whether the key exists first. But I'm not sure whether this might break something down the road e.g. in graph.py

Hello! Can you please include a screen shot in your github so we all know what kind of graphs you can create? Thanks!

Hi,
uhm..
Do you have an example for this cookie?
I have several entries in my "facebook cookie" ;)

Thanks,

What I get from the pull request is a list of friends with the following information:
"<user_id>":{ "lat":1481207023, "p":2, "vc":10 }

someone knows what the fields "p" and "vc" stand for?

In README.md, clarify that cookie in SECRETS.txt needs single-quotes

Hey ! I see that implemented a blacklist for contacts you don't want to see. However, to be really usable for people with many facebook friends, I think it could be better to have a configurable 'whitelist', to include only selected friends
Thank you for your time !

I made a PHP version of this, perhaps I could get a link here as the PHP fork of this project.

https://github.com/Dygear/zzzzz-php

When I run python app.py and then visit http://0.0.0.0:5000/, I can see the input form. However, giving a user ID there, it then goes to a 405 Method Not Allowed page. Not sure what I am doing wrong.

After producing output for a few minutes, fetcher.py crashes with this error:

Traceback (most recent call last):
  File "fetcher.py", line 175, in <module>
    f.start_request()
  File "fetcher.py", line 124, in start_request
    user_data.append(json.dumps(item["overlay"][uid]["p"]))
KeyError: 'p'

When I run pip install -r requirements.txt
gives this error

my requirements.txt:
requests==2.9.1
Flask==0.10.1

Downloading/unpacking requests==2.9.1 (from -r requirements.txt (line 1))
Downloading requests-2.9.1-py2.py3-none-any.whl (501kB): 501kB downloaded
Downloading/unpacking Flask==0.10.1 (from -r requirements.txt (line 2))
Downloading Flask-0.10.1.tar.gz (544kB): 544kB downloaded
Running setup.py (path:/tmp/pip_build_deploy/Flask/setup.py) egg_info for package Flask

warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'examples'
warning: no previously-included files matching '*.pyo' found under directory 'examples'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'

Downloading/unpacking Werkzeug>=0.7 (from Flask==0.10.1->-r requirements.txt (line 2))
Downloading Werkzeug-0.11.4-py2.py3-none-any.whl (305kB): 305kB downloaded
Downloading/unpacking Jinja2>=2.4 (from Flask==0.10.1->-r requirements.txt (line 2))
Downloading Jinja2-2.8-py2.py3-none-any.whl (263kB): 263kB downloaded
Downloading/unpacking itsdangerous>=0.21 (from Flask==0.10.1->-r requirements.txt (line 2))
Downloading itsdangerous-0.24.tar.gz (46kB): 46kB downloaded
Running setup.py (path:/tmp/pip_build_deploy/itsdangerous/setup.py) egg_info for package itsdangerous

warning: no previously-included files matching '*' found under directory 'docs/_build'

Downloading/unpacking MarkupSafe (from Jinja2>=2.4->Flask==0.10.1->-r requirements.txt (line 2))
Downloading MarkupSafe-0.23.tar.gz
Running setup.py (path:/tmp/pip_build_deploy/MarkupSafe/setup.py) egg_info for package MarkupSafe

Installing collected packages: requests, Flask, Werkzeug, Jinja2, itsdangerous, MarkupSafe
Found existing installation: requests 2.2.1
Not uninstalling requests at /usr/lib/python2.7/dist-packages, owned by OS
Can't roll back requests; was not uninstalled
Cleaning up...
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run
requirement_set.install(install_options, global_options, root=options.root_path)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1436, in install
requirement.install(install_options, global_options, _args, *_kwargs)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 672, in install
self.move_wheel_files(self.source_dir, root=root)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 902, in move_wheel_files
pycompile=self.pycompile,
File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 206, in move_wheel_files
clobber(source, lib_dir, True)
File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 193, in clobber
os.makedirs(destsubdir)
File "/usr/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/requests-2.9.1.dist-info'

Storing debug log for failure in /home/deploy/.pip/pip.log

I might be missing something here, but shouldn't there be a requirements.txt file?

Running pip install -r requirements.txt gives me:

Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

I have the fetcher continuously updating in the background, and running graph.py gets me great csv's, but when I run app.py the web app does not use the updated csv's. I tried deleting the generated_graphs folder and then re-generating it with graph.py, but that didn't help. How do I make the web app use the latest data?

I have not registered any activity for these stati. Maybe their key has changed?

I tried:
python /home/openhabian/zzzzz/fetcher.py &
But it didn't work. Even
python /home/openhabian/zzzzz/fetcher.py
says
SECRETS.txt: no such file or directory
I have to manually navigate to zzzzz folder using cd command and then run
python fetcher.py
in order to make it run
Same goes for other scripts. How to run them in background?

I can't find it -_-

I had no problem finding the 2 IDs, but I can't find the cookie.

I checked logs, and my router was unreachable for 5 minutes but because the script crashed - I lost about a few hours of potential data. Here is the trace:

Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "fetcher.py", line 61, in make_request
    data = json.loads(raw_response)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "fetcher.py", line 176, in <module>
    f.start_request()
  File "fetcher.py", line 93, in start_request
    resp = self.make_request()
  File "fetcher.py", line 64, in make_request
    print(content)
NameError: name 'content' is not defined

I started the fetcher using

python2.7 fetcher.py

After two failed attempts I quit the script usinc CTRL+C.
It showed this Traceback:

>
Got error from request, restarting...
>
Got error from request, restarting...
>
^CTraceback (most recent call last):
  File "fetcher.py", line 176, in <module>
    f.start_request()
  File "fetcher.py", line 93, in start_request
    resp = self.make_request()
  File "fetcher.py", line 49, in make_request
    response_obj = requests.get(url, params=self.params, headers=self.REQUEST_HEADERS)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 784, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 252, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 300, in ssl_wrap_socket
    cnx.do_handshake()
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1441, in do_handshake
    result = _lib.SSL_do_handshake(self._ssl)
KeyboardInterrupt

Probably an issue with SECRETS.txt.
the file looks like this now: (with the numbers letters altered and the cookie redacted)

uid=100000232323239
cookie=datr=JuOSKadJDmdiU8_gn3djr4B; locale=en_US; c_user=[Redacted cookie]wd=882x749
client_id=88c9g39b

Terminal outputs:

Felipes-MacBook-Air:zzzzz-master felipe$ python fetcher.py
Traceback (most recent call last):
File "fetcher.py", line 8, in
import graph
File "/Users/felipe/Downloads/zzzzz-master/graph.py", line 6, in
import flap
File "/Users/felipe/Downloads/zzzzz-master/fbapi.py", line 4, in
import fetcher
File "/Users/felipe/Downloads/zzzzz-master/fetcher.py", line 25, in
class Fetcher():
File "/Users/felipe/Downloads/zzzzz-master/fetcher.py", line 31, in Fetcher
'cookie': secrets.cookie,
AttributeError: 'Namespace' object has no attribute 'cookie'

Hello Guys,

The code is running well, but most of the time when I check the status of the 'fetcher.py' I find that the connection is interrupted with the following response. Do anyone have the same issue? if yes, did you try to overcome it?

Response:{u'ms': [{u'type': u'buddylist_overlay', u'realtime_viewer_fbid': 100002051235984L, u'overlay': {u'701918867': {u'a': 2, u'ol': 0, u'vc': 8}}}], u'u': 100002051235984L, u't': u'msg', u'seq': 1330}
Traceback (most recent call last):
File "fetcher.py", line 179, in
f.start_request()
File "fetcher.py", line 121, in start_request
self._log_lat(uid, str(item["overlay"][uid]["la"]))
KeyError: 'la'

The code below showed up
Traceback (most recent call last): File "fetcher.py", line 179, in <module> f.start_request() File "fetcher.py", line 95, in start_request resp = self.make_request() File "fetcher.py", line 69, in make_request print("Response:" + str(data)) File "C:\Users\TKGMoMoSheep\AppData\Local\Programs\Python\Python35-32\lib\encodings\cp437.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_map)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u6bdb' in position 1413: character maps to <undefined>

Running fetcher.py yields this.

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:31 5: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause t he server to present an incorrect TLS certificate, which can cause validation fa ilures. For more information, see https://urllib3.readthedocs.org/en/latest/secu rity.html#snimissingwarning.
SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:12 0: InsecurePlatformWarning: A true SSLContext object is not available. This prev ents urllib3 from configuring SSL appropriately and may cause certain SSL connec tions to fail. For more information, see https://urllib3.readthedocs.org/en/late st/security.html#insecureplatformwarning.
InsecurePlatformWarning
Got error from request, restarting...

How do I get it to access the facebook api?

_
Sorry, I've never written code or anything, I literally have no programming experience and this is the first creation that I decided to attempt to use.
_

I ran python graph.py
then python app.py
then used chrome and mozilla firefox to access both 0.0.0.0:5000 and localhost:8000; got the "This webpage is unavailable" error for both.

What did I do wrong?

I think I got through all the steps (unless SECRET.txt has to be formatted in some different way?); all the information is right and cmd is no longer giving me an error.

Trying to run
python graph.py
yields the following error
Traceback (most recent call last): File "graph.py", line 64, in <module> g.generate_all_csvs(start_time=now - 3 * ONE_DAY_SECONDS, end_time=now) File "graph.py", line 56, in generate_all_csvs self.to_csv(uid, start_time, end_time) File "graph.py", line 27, in to_csv status_history = history.StatusHistory(uid) File "/Users/vinted/Documents/zzzzz-master/history.py", line 18, in __init__ with open("log/{uid}.txt".format(uid=uid)) as f: IOError: [Errno 2] No such file or directory: 'log/.txt'
A workaround is to replace {uid} with a concrete uid

fetcher.py is throwing this error.

python fetcher.py

Traceback (most recent call last):
  File "fetcher.py", line 8, in <module>
    import graph
  File "/home/user/apps/Stalky/graph.py", line 6, in <module>
    import fbapi
  File "/home/user/apps/Stalky/fbapi.py", line 4, in <module>
    import fetcher
  File "/home/user/apps/Stalky/fetcher.py", line 17, in <module>
    setattr(secrets, vals[0].lower(), vals[1])
IndexError: list index out of range

I have Python2.7 and have this error in both virtualenv and outside virtualenv.
I renamed the app folder "Stalky", added "SECRETS.txt with appropriate info as far as I can tell.

With python2.7:

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Got error from request, restarting...

python3:

Traceback (most recent call last):
  File "fetcher.py", line 6, in <module>
    import requests
ImportError: No module named requests

Yes, I read the page recommended for python2.7. No, I am not not able to fix it with the recommended solutions.
Yes, I ran all that "pip install requests" etc, and the rest you can find here.

using my raspberry 2 with debian.

edit:

got a little further with: sudo apt-get install python3-requests

Traceback (most recent call last):
  File "fetcher.py", line 194, in <module>
    f.start_request()
  File "fetcher.py", line 110, in start_request
    resp = self.make_request()
  File "fetcher.py", line 52, in make_request
    raw_response = response_obj.text
  File "/usr/lib/python3/dist-packages/requests/models.py", line 831, in text
    content = str(self.content, encoding, errors='replace')
TypeError: str() argument 2 must be str, not None

Hey, cool project. I used it a monthish ago to gather some data and it work great. However, now when I try to use it (after replacing my cookie with a more recent one), I get a json blob with what I assume is the expected data and then immediately the error:

Traceback (most recent call last):
  File "fetcher.py", line 175, in <module>
    f.start_request()
  File "fetcher.py", line 124, in start_request
    user_data.append(json.dumps(item["overlay"][uid]["p"]))
KeyError: 'p'

Thoughts? Any tips on how to fix this or where to look? I'm using it for a data vis project and while I'm too busy to fix it right now, I could submit a PR in the near future.