Importing from Trello does not work properly about vikunja

[kolaente]

It was clearly a permission error.

I love that comic, I have it printed on a wall in the office.

Some cards show that they have an attachment, with the correct filename, but list them as a 33-byte file. Downloading the file gives you a broken file. The file size also corresponds with some of the files in the list below.

What does that look like when you open the file? (for example with cat?

I have the whole import as a debug-log, if you want it. I don't want to put it here since it's quite massive, and I'm unsure about privacy concerns.

I'd love to take a look at the log. You could send it to me via email: [email protected]

[NoLooseEnds]

I removed the one board that got imported, and started again.

This time it imported another board – from a workspace(!). It imported all the "buckets", but only imported one card (of 24 spread over different buckets).

The cards do have images, also the one card that got imported, but the image did not get imported. But that might be another limitation of the import tool?

[kolaente]

Could you enable debug logging on the server and try again? There should be something there, this clearly looks like a bug.

Maybe we're running in Trellos rate limit.

[NoLooseEnds]

Enabled debug now, should have done it in the first place. Everything regarding parsing through all the boards, cards, etc. seems to work fine. Until it wants to save an attachment. Here is the relevant log:

2021-09-08T18:41:03.761760066+02:00: DEBUG	▶ migration/insertFromStructure 2226 [creating structure] Creating 1 attachments
2021-09-08T18:41:03.763807113+02:00: ERROR	▶ migration/InsertFromStructure 2228 [creating structure] Error while creating structure: open files/7: permission denied
2021-09-08T18:41:03.763864623+02:00: ERROR	▶ handler/Migrate 2229 open files/7: permission denied

But the /app/vikunja/files directory is writable, and running "touch testfile" from inside the docker works fine.

/app/vikunja/files # touch testfile
/app/vikunja/files # ls -alh
total 5K
drwxr-xr-x    2 root     root           3 Sep  8 18:45 .
drwxr-xr-x    1 vikunja  root        4.0K Sep  5 17:37 ..
-rwxrwxrwx    1 42949672 42949672       0 Sep  8 18:45 testfile

Not sure whats going on – the storage is on a NFS share.

[kolaente]

Are you able to upload attachments? If that doesn't work either and fails with the same error message, it's a permission problem.
The ls output looks like the folder is owned by root, the user you created the testfile does not seem to be the Vikunja user.

The user you're executing commands in the container is not necessarily the same user running the Vikunja binary. You can control the UID that user has, check out the docs for more details.

[NoLooseEnds]

I'll try to test some more, but the NFS share should be fully writable for anyone by the nature of MFD.

Thanks for your help so far 🙂

[NoLooseEnds]

It was clearly a permission error.

That's sorted now. And you were right, the app runs under the vikunja user while I tested under a root user.

Every board is now imported (yey! – some even with the Trello background image), and the logs seems fine as far as I can tell.

Unfortunately, attachments are still not imported and/or showing properly.

1. Some cards show that they have an attachment, with the correct filename, but list them as a 33-byte file. Downloading the file gives you a broken file. The file size also corresponds with some of the files in the list below.

2. Some Cards does not show any sign of attachments. I.e I have a recipe board where every card has an image, all cards and text have been imported fine, but none of them has an attachment.

So to summarize. It seems as every board gets imported, some images (ie. background), and some attachments get's an erroneous attachment – while others don't show that they have any.

I have tried to add attachments manually, and that works as expected.

I have the whole import as a debug-log, if you want it. I don't want to put it here since it's quite massive, and I'm unsure about privacy concerns.

List of files from the files directory:

➜  files ls -alh
total 194240
drwxrwxrwx+  2 1000     999          128 Sep  8 18:14 .
drwxrwxrwx+  4 1000     100            4 Sep  6 10:57 ..
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 10
-rwxrwxrwx+  1 1000     999        1018K Sep  8 18:14 100
-rwxrwxrwx+  1 1000     999        1018K Sep  8 18:14 101
-rwxrwxrwx+  1 1000     999        1018K Sep  8 18:14 102
-rwxrwxrwx+  1 1000     999        1018K Sep  8 18:14 103
-rwxrwxrwx+  1 1000     999        1018K Sep  8 18:14 104
-rwxrwxrwx+  1 1000     999        1018K Sep  8 18:14 105
-rwxrwxrwx+  1 1000     999        4.17M Sep  8 18:14 106
-rwxrwxrwx+  1 1000     999         609K Sep  8 18:14 107
-rwxrwxrwx+  1 1000     999         740K Sep  8 18:14 108
-rwxrwxrwx+  1 1000     999         297K Sep  8 18:14 109
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 11
-rwxrwxrwx+  1 1000     999         400K Sep  8 18:14 110
-rwxrwxrwx+  1 1000     999         400K Sep  8 18:14 111
-rwxrwxrwx+  1 1000     999         400K Sep  8 18:14 112
-rwxrwxrwx+  1 1000     999         400K Sep  8 18:14 113
-rwxrwxrwx+  1 1000     999        6.11M Sep  8 18:14 114
-rwxrwxrwx+  1 1000     999         408K Sep  8 18:14 115
-rwxrwxrwx+  1 1000     999         679K Sep  8 18:14 116
-rwxrwxrwx+  1 1000     999        53.8K Sep  8 18:14 117
-rwxrwxrwx+  1 1000     999         804K Sep  8 18:14 118
-rwxrwxrwx+  1 1000     999         255K Sep  8 18:14 119
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 12
-rwxrwxrwx+  1 1000     999        1.21M Sep  8 18:14 120
-rwxrwxrwx+  1 1000     999         354K Sep  8 18:14 121
-rwxrwxrwx+  1 1000     999         255K Sep  8 18:14 122
-rwxrwxrwx+  1 1000     999        7.03M Sep  8 18:14 123
-rwxrwxrwx+  1 1000     999        53.8K Sep  8 18:14 124
-rwxrwxrwx+  1 1000     999        53.8K Sep  8 18:14 125
-rwxrwxrwx+  1 1000     999        5.59M Sep  8 18:14 126
-rwxrwxrwx+  1 1000     999        1.47M Sep  8 18:14 127
-rwxrwxrwx+  1 1000     999        1.01M Sep  8 18:14 128
-rwxrwxrwx+  1 1000     999         507K Sep  8 18:14 129
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 13
-rwxrwxrwx+  1 1000     999        1.47M Sep  8 18:14 130
-rwxrwxrwx+  1 1000     999         255K Sep  8 18:14 131
-rwxrwxrwx+  1 1000     999         341K Sep  8 18:14 132
-rwxrwxrwx+  1 1000     999        53.8K Sep  8 18:14 133
-rwxrwxrwx+  1 1000     999         289K Sep  8 18:14 134
-rwxrwxrwx+  1 1000     999        53.8K Sep  8 18:14 135
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 14
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 15
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 16
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 17
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 18
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 19
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 20
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 21
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 22
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 23
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 24
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 25
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 26
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 27
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 28
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 29
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 30
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 31
-rwxrwxrwx+  1 1000     999           33 Sep  8 17:59 32
-rwxrwxrwx+  1 1000     999        1018K Sep  8 17:59 33
-rwxrwxrwx+  1 1000     999        1018K Sep  8 17:59 34
-rwxrwxrwx+  1 1000     999        1018K Sep  8 17:59 35
-rwxrwxrwx+  1 1000     999        1018K Sep  8 17:59 36
-rwxrwxrwx+  1 1000     999        1018K Sep  8 17:59 37
-rwxrwxrwx+  1 1000     999        1018K Sep  8 17:59 38
-rwxrwxrwx+  1 1000     999        1018K Sep  8 17:59 39
-rwxrwxrwx+  1 1000     999        1018K Sep  8 17:59 40
-rwxrwxrwx+  1 1000     999        1018K Sep  8 17:59 41
-rwxrwxrwx+  1 1000     999        1018K Sep  8 17:59 42
-rwxrwxrwx+  1 1000     999        4.17M Sep  8 17:59 43
-rwxrwxrwx+  1 1000     999         609K Sep  8 17:59 44
-rwxrwxrwx+  1 1000     999         740K Sep  8 17:59 45
-rwxrwxrwx+  1 1000     999         297K Sep  8 17:59 46
-rwxrwxrwx+  1 1000     999         400K Sep  8 17:59 47
-rwxrwxrwx+  1 1000     999         400K Sep  8 17:59 48
-rwxrwxrwx+  1 1000     999         400K Sep  8 17:59 49
-rwxrwxrwx+  1 1000     999         400K Sep  8 17:59 50
-rwxrwxrwx+  1 1000     999        6.11M Sep  8 17:59 51
-rwxrwxrwx+  1 1000     999         408K Sep  8 17:59 52
-rwxrwxrwx+  1 1000     999         679K Sep  8 17:59 53
-rwxrwxrwx+  1 1000     999        53.8K Sep  8 17:59 54
-rwxrwxrwx+  1 1000     999         804K Sep  8 17:59 55
-rwxrwxrwx+  1 1000     999         255K Sep  8 17:59 56
-rwxrwxrwx+  1 1000     999        1.21M Sep  8 17:59 57
-rwxrwxrwx+  1 1000     999         354K Sep  8 17:59 58
-rwxrwxrwx+  1 1000     999         255K Sep  8 17:59 59
-rwxrwxrwx+  1 1000     999        7.03M Sep  8 17:59 60
-rwxrwxrwx+  1 1000     999        53.8K Sep  8 17:59 61
-rwxrwxrwx+  1 1000     999        53.8K Sep  8 17:59 62
-rwxrwxrwx+  1 1000     999        5.59M Sep  8 17:59 63
-rwxrwxrwx+  1 1000     999        1.47M Sep  8 17:59 64
-rwxrwxrwx+  1 1000     999        1.01M Sep  8 17:59 65
-rwxrwxrwx+  1 1000     999         507K Sep  8 17:59 66
-rwxrwxrwx+  1 1000     999        1.47M Sep  8 17:59 67
-rwxrwxrwx+  1 1000     999         255K Sep  8 17:59 68
-rwxrwxrwx+  1 1000     999         341K Sep  8 17:59 69
-rwxrwxrwx+  1 1000     999        53.8K Sep  8 17:59 70
-rwxrwxrwx+  1 1000     999         289K Sep  8 17:59 71
-rwxrwxrwx+  1 1000     999        53.8K Sep  8 17:59 72
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 73
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 74
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 75
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 76
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 77
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 78
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 79
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 80
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 81
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 82
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 83
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 84
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 85
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 86
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 87
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 88
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 89
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 90
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 91
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 92
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 93
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 94
-rwxrwxrwx+  1 1000     999           33 Sep  8 18:14 95
-rwxrwxrwx+  1 1000     999        1018K Sep  8 18:14 96
-rwxrwxrwx+  1 1000     999        1018K Sep  8 18:14 97
-rwxrwxrwx+  1 1000     999        1018K Sep  8 18:14 98
-rwxrwxrwx+  1 1000     999        1018K Sep  8 18:14 99

[NoLooseEnds]

➜  files cat 86
unauthorized permission requested%

So something is weird with import. I checked the board settings on Trello, just to be sure, and I am an admin. My user also created all the cards.

I just sent you the debug log.

[kolaente]

Got it, will look into it.

[kolaente]

Fixed in c3e0e64. Please test with a new unstable release once the CI released one (in ~30 min).

Closing as resolved, feel free to ping me if you have issues with it.