itzmestar / octoparse Goto Github PK

Thank you for the great package!

Is there a way to use the get_not_exported_data() function for more than 1,000 rows? I would think there would be an offset parameter that would allow me to iterate over the first 1,000 then skip 1,000 for the next batch and iterate over the whole set. If I want to export the unexported rows between 2,000 and 3,000 from a task is there a way to do so?

Get Data by Offset: where user can fetch only a subset of data from the server.

I believe this to be a bug.
Steps to reproduce on master branch from Octoparse/octoparse/:

 $ python
Python 3.8.5 (default, Jul 21 2020, 10:42:08)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import octoparse
>>> octo = octoparse.Octoparse()
Enter Octoparse Username: [USERNAME]
Password: [PASSWORD]
>>> octo.list_all_task_groups()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mcline/dev/octoparse-orig/octoparse/octoparse.py", line 241, in list_all_task_groups
    response = _get_request(self._get_url(path), self._get_access_token())
  File "/Users/mcline/dev/octoparse-orig/octoparse/octoparse.py", line 124, in _get_access_token
    timedelta = datetime.now() - self.token_entity['datetime']
KeyError: 'datetime'

I believe a fix would be to this line should be added to the refresh_token() method:
fea6a7e#diff-52e2adcfc102844f5c18a891b46001dff96ea89b3a767ce60083de88aa69bc31R134

When I use get_task_data() for whatever reason the code gets totally stuck and just never stops running, even if I set a size of 1, which should be instant. Not having this problem with any of the other functions. I can provide more details about my configuration, but I would be curious to hear if everything works as expected on your end and if anything in particular would be helpful in troubleshooting.

Thank you!

Error encountered while running update_data_status():

Traceback (most recent call last):
File "", line 1, in
File "D:\py_game\venv38\lib\site-packages\octoparse\octoparse.py", line 305, in update_data_status
response = _post_request(self._get_url(path), self._get_access_token(), params=params)
TypeError: _post_request() got an unexpected keyword argument 'params'

implement a generator for task data fetching