API issue? about toggl2clockify

[Coimbra1984]

Thanks a lot!
I've just updated the code and .exe file. They should work now for more than 50 clients, tags and projects.

I tested it on my clockify test account at least and got results as I would expect.
While testing I sometimes got error code 500 on loading my user. 500 means "internal server error" maybe this is the same problem that you observed as well.

[Coimbra1984]

Hi acseven ,

thank you for using our tool.
I havve just added a new commit which outputs the clockify API error code when trying to load the user. According to the API (https://clockify.me/developers-api) it should be any of:
401 Unauthorized
403 Forbidden
404 Not Found
Maybe you can run the script again and post the new output in order to further investigate.

As for the second error, there might be one problem. How many clients do you have in your workspaces? Per default, the clockify API returns only the first 50. If you have more, than I need to enhance the logic of fetching clients from clockify.

Best regards,

Markus

[acseven]

Hi, thanks for the prompt reply. In this current migration I have 69 clients, so that's probably the current issue.

After some hours of pause, it now stops at the last error I mentioned:

> 2019-06-09 10:26:39,345 - INFO - Clue - adding project ABC (532 of 1604 projects)
> Traceback (most recent call last):
>   File "toggl2clockify.py", line 209, in <module>
>   File "Clue.py", line 126, in syncProjects
>   File "ClockifyAPI.py", line 331, in addProject
>   File "ClockifyAPI.py", line 212, in getClientID
> RuntimeError: Client ANS not found in workspace COMPANY
> [24736] Failed to execute script toggl2clockify

Also, is the timeout option worth trying, and to you have a suggestion on a value? Is it in miliseconds?

Thanks!

[acseven]

Maybe you can run the script again and post the new output in order to further investigate.

I tried the script just a few times since yesterday, and the only error that pops up is with the Clients. I'd be thankful if you could have a look into it, to see if I can get to the time entries process, which I guess might be even more time consuming.

Edit: I have to say I've tested the other tool available (https://github.com/mikerourke/toggl-to-clockify-web) and yours works - by far - much better, even with this issue considered.

[acseven]

Thanks! I'm testing it right now, it's moving past where it got stuck, I'll let you know how it went. One other thing I've noticed though, I believe we have some time entries for which there are no longer users, either active or disabled. Do you think these time entries will be properly migrated?

[Coimbra1984]

Oh, if I understand right, that will probably let the tool crash. The tool fetches all entries from toogl.

Each toggl time entry has a userID field. If this userID is not found in toggl's user list, the time entry is skipped (not migrated to clockify and a WARNING message will be seen on console, something like: user ID %s not in toggl workspace, skipping entry %s...), this happened on our dataset where we had a user in toggl with time entries but left the company and thus was no longer an active toggl user.

Now I'm not sure if I understand your situation. You have toggl users from past which are not part of the new clockify workspace? This is a situation that is not handled at the moment.
I guess there are two possibilities:

1. skip these entries (not migrate)
2. assign these entries to a dummy user (you would need to create that dummy user in clockify as all time entries need a "user").

[acseven]

In our current procedure in Toggl we can see active and inactive users; if a user leaves the company the admin/manager will disable their account. I've created accounts in Clockify for all of these users, so those should be fine. The issue is that I think there are some other users (in Toggl) from the early days where the manager simply deleted the accounts altogether. But I think that warning should suffice, although that means I need to go through the log in the end to get those emails and add them manually to Clockify..
Thanks

[Coimbra1984]

Yes, please go through the log. The tool generates a file called "log.txt" in the folder where it is invoked. Go through the file and search for "warning". If you're not sure about the message, just post it here...
If you understand all warning messages and accept them, your migration should be complete (don't forget to run the tool with
bin/toggl2clockify.exe --skipClients --skipProjects --skipEntries --skipTags --doArchive
at the very end, when you're sure that all data has been properly migrated.

[acseven]

Ah, there it is:

2019-06-09 19:32:45,331 - WARNING - Clue - user ID 1880___ not in toggl workspace, skipping entry ...

Could there be a way of displaying the user's email as well/instead?

[Coimbra1984]

can you try again. I found a "user" field in toggl's time entries. Hopefully this gives more information than user id...

[acseven]

Ok, the name appears, and that's useful. In the meantime I added one of those users to see what the tool does; basically it skips the entry altogether because it doesn't find the user in the Toggl workspace:

2019-06-09 20:04:35,782 - WARNING - Clue - user ID 1880___ (name=John Doe) not in toggl workspace, skipping entry Task 1...

This happens when "John Doe" is already added to Clockify...

[Coimbra1984]

I see, this is because user matching works in the following way:
(toggl) get User ID from toggl
(toggl) get Usermail from UserID
(clockify) find user with that Usermail
(clockify) get user id from mail
(clockify) add timeentry with user id

... anyway I added a second logic now, if the user is not found in toggl workspace, check if a user with the exact same name is already present in clockify, if so, than use this user.

You should see the output
"user ID %s (name=%s) not in toggl workspace, but found a match in clockify workspace %s..."
If this kind of matching happens now, which is hopefully what you describe above

[acseven]

Hmm, it doesn't seem to be working; I've checked the user's name field in Clockify and it is exactly the same as the log output, but the warning message is still the earlier one:

2019-06-09 20:25:55,497 - WARNING - Clue - user ID 1880___ (name=John Doe) not in toggl workspace, skipping entry Task 1...

[Coimbra1984]

hmm, strange...
okay I modified the tool to be a bit more verbose. It generates now a file called clockify_users.json can you cross check whether you find the user in that file (something like "name": "John Doe")
I also added a new print line:
user '%s' found in clockify workspace as ID=%s"
is this output line shown?

[acseven]

Yup, I think that does it:

2019-06-09 21:17:19,652 - INFO - Clue - adding entry , project: None (13 of 5791)
2019-06-09 21:17:19,658 - INFO - Clue - user 'John Doe' found in clockify workspace as ID=5cfd----
2019-06-09 21:17:19,660 - INFO - Clue - user ID 1880___ (name='John Doe') not in toggl workspace, but found a match in clockify workspace [email protected]...
2019-06-09 21:17:19,906 - WARNING - ClockifyAPI - no project in entry

Thanks a bunch, it will now take quite a few hours to complete the migration. I'll let you know if something else came up.

[acseven]

I have almost 5800 time entries and around entry 1000, the error below appeared. I can't say if it wasn't the network connection that went down, but I wouldn't believe much that's the case.

Also, a suggestion if it's possible/useful: the tool verifies if an entry exists only after processing tags and processing that user check, and then skipping it. If it does indeed exist then it shouldn't matter checking its tags or user, right? This could save a lot of requests/time in reruns like the one I'm doing

Traceback (most recent call last):
  File "site-packages\urllib3\connection.py", line 159, in _new_conn
  File "site-packages\urllib3\util\connection.py", line 57, in create_connection
  File "socket.py", line 733, in getaddrinfo
socket.gaierror: [Errno 11001] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages\urllib3\connectionpool.py", line 600, in urlopen
  File "site-packages\urllib3\connectionpool.py", line 343, in _make_request
  File "site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
  File "site-packages\urllib3\connection.py", line 301, in connect
  File "site-packages\urllib3\connection.py", line 168, in _new_conn
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x0000017D498FEEB8>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages\requests\adapters.py", line 449, in send
  File "site-packages\urllib3\connectionpool.py", line 638, in urlopen
  File "site-packages\urllib3\util\retry.py", line 399, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='clockify.me', port=443): Max retries exceeded with url: /api/v1/workspaces/5cf18327f15c985e3841018e/tags?page=1&page-size=50 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000017D498FEEB8>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "toggl2clockify.py", line 225, in <module>
  File "Clue.py", line 240, in syncEntries
  File "TogglAPI.py", line 125, in getReports
  File "TogglAPI.py", line 161, in _getReports
  File "Clue.py", line 217, in onNewReports
  File "ClockifyAPI.py", line 498, in addEntry
  File "ClockifyAPI.py", line 419, in getTagName
  File "ClockifyAPI.py", line 386, in getTags
  File "ClockifyAPI.py", line 143, in multiGetRequest
  File "site-packages\requests\api.py", line 75, in get
  File "site-packages\requests\api.py", line 60, in request
  File "site-packages\requests\sessions.py", line 533, in request
  File "site-packages\requests\sessions.py", line 646, in send
  File "site-packages\requests\adapters.py", line 516, in send
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='clockify.me', port=443): Max retries exceeded with url: /api/v1/workspaces/5cf18327f15c985e3841018e/tags?page=1&page-size=50 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000017D498FEEB8>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed',))
[8268] Failed to execute script toggl2clockify

[acseven]

Again:

2019-06-10 02:48:19,638 - INFO - Clue - ...ok
2019-06-10 02:48:19,638 - INFO - Clue - adding entry , project: None (1079 of 5791)
2019-06-10 02:48:20,103 - WARNING - ClockifyAPI - no project in entry
2019-06-10 02:48:21,247 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:22,667 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:24,044 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:25,425 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:26,851 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:28,247 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:29,653 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:31,033 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:32,429 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:33,792 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:35,156 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:36,559 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:37,972 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:39,337 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:40,719 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:42,096 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:43,474 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
2019-06-10 02:48:44,871 - INFO - ClockifyAPI - finsihed getting clockify tags, saving results to clockify_tags.json
Traceback (most recent call last):
  File "site-packages\urllib3\contrib\pyopenssl.py", line 456, in wrap_socket
  File "site-packages\OpenSSL\SSL.py", line 1716, in do_handshake
  File "site-packages\OpenSSL\SSL.py", line 1449, in _raise_ssl_error
OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages\urllib3\connectionpool.py", line 600, in urlopen
  File "site-packages\urllib3\connectionpool.py", line 343, in _make_request
  File "site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
  File "site-packages\urllib3\connection.py", line 344, in connect
  File "site-packages\urllib3\util\ssl_.py", line 345, in ssl_wrap_socket
  File "site-packages\urllib3\contrib\pyopenssl.py", line 462, in wrap_socket
ssl.SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages\requests\adapters.py", line 449, in send
  File "site-packages\urllib3\connectionpool.py", line 638, in urlopen
  File "site-packages\urllib3\util\retry.py", line 399, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='clockify.me', port=443): Max retries exceeded with url: /api/v1/workspaces/5cf18327f15c985e3841018e/tags?page-size=50&page=1 (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "toggl2clockify.py", line 225, in <module>
  File "Clue.py", line 240, in syncEntries
  File "TogglAPI.py", line 125, in getReports
  File "TogglAPI.py", line 161, in _getReports
  File "Clue.py", line 217, in onNewReports
  File "ClockifyAPI.py", line 498, in addEntry
  File "ClockifyAPI.py", line 419, in getTagName
  File "ClockifyAPI.py", line 386, in getTags
  File "ClockifyAPI.py", line 143, in multiGetRequest
  File "site-packages\requests\api.py", line 75, in get
  File "site-packages\requests\api.py", line 60, in request
  File "site-packages\requests\sessions.py", line 533, in request
  File "site-packages\requests\sessions.py", line 646, in send
  File "site-packages\requests\adapters.py", line 514, in send
requests.exceptions.SSLError: HTTPSConnectionPool(host='clockify.me', port=443): Max retries exceeded with url: /api/v1/workspaces/5cf18327f15c985e3841018e/tags?page-size=50&page=1 (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),))
[29072] Failed to execute script toggl2clockify

[Coimbra1984]

Also, a suggestion if it's possible/useful: the tool verifies if an entry exists only after processing tags and processing that user check, and then skipping it. If it does indeed exist then it shouldn't matter checking its tags or user, right? This could save a lot of requests/time in reruns like the one I'm doing

Not sure if I understood your point. Before adding an entry, we have to enable the corresponding user by an API call. Only then we can query the clockify database for time entries of this user. After we loaded the user, the script fetches time entries from clockify with the same "start", "billable", "description", "projectID", "end", and "tagIds" fields. If we already have the same entry in clockify than we don't add the entry again.

I found one line of code which looks strange to me, when I loaded a user, I marked all internal tags as invalid, and the tool will than fetch all tags again. This will indeed create a lot of unnecessary requests.

I fixed that and uploaded a new version. The errors which you received are some kind of network errors, it should be better now, with that modified version

[acseven]

Hi,

This time I believe it's something with the project name string. I could change this in Toggl, but more like can appear:

2019-06-11 01:37:05,055 - INFO - Clue - adding entry Vários, project: GEST\xc3O DE PROJECTO (PM) (2623 of 5818)
Traceback (most recent call last):
  File "toggl2clockify.py", line 225, in <module>
  File "Clue.py", line 240, in syncEntries
  File "TogglAPI.py", line 125, in getReports
  File "TogglAPI.py", line 161, in _getReports
  File "Clue.py", line 194, in onNewReports
  File "Clue.py", line 180, in timeToUtc
  File "site-packages\dateutil\parser\_parser.py", line 1358, in parse
  File "site-packages\dateutil\parser\_parser.py", line 646, in parse
  File "site-packages\dateutil\parser\_parser.py", line 722, in _parse
  File "site-packages\dateutil\parser\_parser.py", line 207, in split
  File "site-packages\dateutil\parser\_parser.py", line 76, in __init__
TypeError: Parser must be a string or character stream, not NoneType
[13196] Failed to execute script toggl2clockify

[Coimbra1984]

Okay, I'll have to check that. Can you paste the original toggl project name here, so that I can setup a test case.

[Coimbra1984]

Okay I just tested on my workspace with some ugly names like äöü1§µáuroé and testPRJ1ää@ê€á. This worked as expected.
But from your error messages it looks like this is an entry without END time, which is perfectly fine but was not handled properly. I modified the handling of these entries, maybe you can check again.

You're becoming the most challenging user of the tool, that's cool. I hope you're still patient. Sorry for the inconveniences. When we migrated we had like 4800 entries which were migrated without major problems.

[acseven]

No worries, thank you for such prompt replies.
This current version has failed just one time due to "too many retries" connecting at some point, but other that it looks it's running properly.
This will take a bit longer though because only now I've noticed the first 5800 time entries were just for one year, so it look we should have a few tens of thousands to go. But I can't say for sure as there are many entries across multiple years in clockify already.
It could be nice to have a summary when beginning the script of how many of each entity we have to process with the current config settings.
I'll let you know how it goes in the meantime.

[acseven]

Yup, I think this time it's done.

I'll do another round for the archives and afterwards cross-check data directly from the platforms' reports.

2019-06-11 20:07:10,689 - INFO - toggl2clockify - -------------------------------------------------------------
2019-06-11 20:07:10,689 - INFO - toggl2clockify - Phase 4 of 5 (Import entries) completed (entries=26871, ok=20212, skips=6561, err=0)
2019-06-11 20:07:10,689 - INFO - toggl2clockify - -------------------------------------------------------------
2019-06-11 20:07:10,689 - INFO - toggl2clockify - -------------------------------------------------------------
2019-06-11 20:07:10,689 - INFO - toggl2clockify - Phase 5 of 5: Archiving projects
2019-06-11 20:07:10,689 - INFO - toggl2clockify - -------------------------------------------------------------
2019-06-11 20:07:10,689 - INFO - toggl2clockify - ... skipping phase 5
2019-06-11 20:07:10,689 - INFO - toggl2clockify - -------------------------------------------------------------
2019-06-11 20:07:10,689 - INFO - toggl2clockify - Phase 5 of 5 (Archiving  projects) completed (entries=0, ok=0, skips=0, err=0)
2019-06-11 20:07:10,705 - INFO - toggl2clockify - -------------------------------------------------------------
2019-06-11 20:07:10,705 - INFO - toggl2clockify - finished importing workspace 'COMPANY'

Thanks for all the help!

[acseven]

One heads up: if a user is inactive in Clockify, the tool won't run past user's projects check:

2019-06-11 23:36:11,127 - INFO - ClockifyAPI - synchronizing clockify projects for user (inactive user here)...
Traceback (most recent call last):
File "toggl2clockify.py", line 225, in
File "Clue.py", line 243, in syncEntries
File "TogglAPI.py", line 125, in getReports
File "TogglAPI.py", line 161, in _getReports
File "Clue.py", line 220, in onNewReports
File "ClockifyAPI.py", line 446, in addEntry
File "ClockifyAPI.py", line 277, in getProjectID
File "ClockifyAPI.py", line 260, in getProjects
File "ClockifyAPI.py", line 157, in multiGetRequest
RuntimeError: get on url https://clockify.me/api/v1/workspaces/---/projects failed with status code 403
[25232] Failed to execute script toggl2clockify

[Coimbra1984]

Sorry, I missed this comment.
You mean the user is disabled in clockify? But I think this might be a bad idea to inactivate users in clockify before migrating the data, or what would be the use case behind that?

[acseven]

You mean the user is disabled in clockify? But I think this might be a bad idea to inactivate users in clockify before migrating the data, or what would be the use case behind that?

We have a lot of inactive users in Toggl. After the bulk of the data was migrated I deactivated those same users in Clockify; when trying to run the tool again it fails.

[Coimbra1984]

Okay, I have added a check on user's status at the very beginning. If a user is not active, the tool outputs an error message and you have to activate the user in the web frontend, because there is no API call to do that automatically.