scaleway / c14-cli Goto Github PK

Hello,
Is it safe to have this in source code ? https://github.com/online-net/c14-cli/blob/master/pkg/commands/login.go#L41

I'd love to be able to use this to create intensive type archives, instead of having to use the ui!

I get an error when I try to install the c14 command line on Debian Stable (Jessie/8) with only the "golang" package installed.

# go get -u github.com/online-net/c14-cli/cmd/c14
package github.com/online-net/c14-cli/cmd/c14
        imports github.com/BurntSushi/toml
        imports github.com/QuentinPerez/go-encodeUrl
        imports github.com/apex/log
        imports github.com/apex/log/handlers/text
        imports github.com/docker/docker/pkg/jsonlog
        imports github.com/docker/docker/pkg/jsonmessage
        imports github.com/docker/docker/pkg/term
        imports github.com/docker/go-units
        imports github.com/docker/docker/pkg/mflag
        imports github.com/docker/docker/pkg/mflag
        imports github.com/docker/docker/pkg/mflag: cannot find package "github.com/docker/docker/pkg/mflag" in any of:
        /usr/lib/go/src/pkg/github.com/docker/docker/pkg/mflag (from $GOROOT)
        /root/go/src/github.com/docker/docker/pkg/mflag (from $GOPATH)

It might be related to moby/moby@14712f9 where the whole mflag directory was seemingly removed from the master branch.

Another VM on Debian Testing doesn't raise an error, but already has the "docker.io" package installed (version 1.11.2~ds1-5)

What is value for the option parity on the command line c14 create? For standard, intensive or enterprise.

I was seeing strange behaviour with the c14-cli utility whenever I tried to have multiple buckets open and tried interacting with the newly created ones.

The online console was showing these archives with a ready temporary storage and was providing ssh authentication data that worked when entered manually, but the c14-cli utility would insist that the archive didn't exist.

Looking more closely at the source code I noticed that all of these commands were calling FindSafeUUIDFromArchive with useCache set to true, a quick and dirty patch, setting all of these calls to false resolved all of the issues I was facing.

The c14 upload command already seems to fall back to not using the cache automatically but the c14 bucket command does not, which I used to check if the bucket was ready, since bucket creation isn't instant even after the archive has been created using c14 create from what I could tell. At least I did get error messages of missing archives every so often with the upload command until I changed the script I had written to wait for the bucket to be returned with the c14 bucket command.

Generally the c14 bucket command seems to have been left in the dust a bit, since you have to use the uuid to refer to the archive rather than its given name like with all the other commands and the missing fallback to not using the cache.

That being said however I do believe the caching mechanism is not working as intended and is the issue at heart, since even after an entire day of waiting the command would still report the archive as missing, no matter what I tried, like running c14 ls or other commands.

I suspect that elements do not get added to the cache properly and are overwriting previous entries rather than being appended since the one archive that would work no matter what was always the oldest one, which was lowest in the ls command.

I have some problem with a really simple script, executed by cron.

Here is it, for testing :

#! /bin/bash c14 upload /tmp/2017-01-17-backup.tar.gz xxxx-xxxxx-xxxxx

I was logged as root and I setup this script using crontab -e.

On executing this script manually, everything is working fine, but when cron did it, i have this error on my mail :

FATAL[0000] You need to specified a name

Can you help me to solve this problem please ?

Thanks

Regards

I have multiple SSH keys added so I can use different key for different servers.

When I create new bucket using "c14 create", it just picks wrong key. I would like to specify which key to use for ssh connections or even better use current ~/.ssh/id_rsa.pub

I'm trying to get the create output to a variable so I can upload to it from a script, but the output is a mess:
$'\r'-$'\r'$'\r'\\$'\r'$'\r'\|$'\r'$'\r'/$'\r'$'\r'-$'\r'$'\r'-$'\r'$'\r'\\$'\r'$'\r'\|$'\r'$'\r'/$'\r'$'\r'-$'\r'$'\r'-$'\r'$'\r'\\$'\r'$'\r'\|$'\r'$'\r'/$'\r'$'\r'-$'\r'$'\r'-$'\r'$'\r'\\$'\r'$'\r'\|$'\r'$'\r'/$'\r'$'\r'-$'\r'$'\r'-$'\r'$'\r'\\$'\r'$'\r'\|$'\r'$'\r'/$'\r'$'\r'-$'\r'$'\r'-$'\r'$'\r'\\$'\r'$'\r'\ $'\r'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa

uuid was replaced with a's for potential security reasons.

This makes scripting an unnecessary pain, and is probably a bug.

Here's what Ask Ubuntu General Room has to say: It looks like a progress spinner of sorts, the output - Ah! Yes. And the idiots are printing it to stdout. - Ask them to print progress reports to stderr where they belong.

Hello
I've got some trouble about removing old archive. (older than a month)
When I run the command c14 remove I've got this warning

WARN[0000] xxxxx: Archive xxxxx not found

But when I'm using the web interface I can remove the concerned archive without any problem

What could be the problem?

Thanks
Matthieu

cannot find package "github.com/docker/docker/pkg/mflag"

Hello,

In my account, I have 100+ archives, each in one safe.

When using c14 ls, it only list the first 50 archives. How can I get the others ones ?

Thanks.

Usage

$ c14 vault create                                           # Creates new vault
375BE0D8-D294-4B25-9BE5-E08224151A62
$ c14 vault remove 375BE0D8-D294-4B25-9BE5-E08224151A62      # Removes vault
$ c14 vault download  375BE0D8-D294-4B25-9BE5-E08224151A62   # Download files from vault
$ c14 vault upload dir/ 375BE0D8-D294-4B25-9BE5-E08224151A62 # Upload files from dir to c14
$ c14 vault info 375BE0D8-D294-4B25-9BE5-E08224151A62        # Displays info/metadata about vault
$ c14 vault lock 375BE0D8-D294-4B25-9BE5-E08224151A62        # Lock vault to upload it to c14
$ c14 vault list                                             # Displays the list of vault
73B3514A-63EC-40BC-A7EB-A233AAEC97B9
$ c14 worker show                                            # Displays workers which upload the datas
82AFF89D-5B3C-430F-AA1F-164A9772FC54 total_size total_time
|--7DDFA7F4-F21B-4D2A-A7C2-25EE0C390640  file size time
\--3C5FF29D-B57C-4857-8DB4-83F436C6231E   file size time

$ c14 worker abort 82AFF89D-5B3C-430F-AA1F-164A9772FC54     # Abort a woker

$ c14 freeze dir/                                           # Creates a vault/ uploads dir/ and locks it
$ c14 watch dir/ 375BE0D8-D294-4B25-9BE5-E08224151A62       # Download the archive and syncs the data when they are modified
$ c14 wait 73B3514A-63EC-40BC-A7EB-A233AAEC97B9             # Waits until the data are transferred
$ c14 verify 73B3514A-63EC-40BC-A7EB-A233AAEC97B9           # Checks data checksum
$ c14 logs                                                  # Displays the history/errors
$ c14 stats                                                 # Stats about your account


/// BONUS
$ c14 cron every-week dir                                   # Makes an archive every-week

Architecture

// RPC Architecture:
┌────────────────────────────────────────────────────────────┐                        
│                                                            │                        
│                                                            │                        
│                                                            │                        
│              LOCAL                                         │                        
│                                                            │                        
│                                                            │                        
│                              ┌────────────────────────┐    │                        
│  ┌──────────────────┐        │      SERVER (RPC)      │    │                        
│  │                  │        │                        │    │         ┌─────────────┐
│  │                  │        │    ┌┬─┬─┬──┬────────┐  │    │         │             │
│  │                  │  API   │    ││ │ │  │        │  │    │         │             │
│  │ CLI (client RPC) │        │    ││ │ │  │ WORKERS│  │    │         │             │
│  │                  │───────▶│    ││ │ │  │        │──┼────┼────────▶│     C14     │
│  │                  │        │    ││ │ │  │        │  │    │         │             │
│  │                  │        │    └┴─┴─┴──┴────────┘  │    │         │             │
│  │                  │        │             ▲          │    │         │             │
│  └──────────────────┘        │             │          │    │         └─────────────┘
│                              └─────────────┼──────────┘    │                        
│                                           ││               │                        
│                                           ││               │                        
│                                           └┘               │                        
│                                                            │                        
└────────────────────────────────────────────────────────────┘                        

Reviewing the commands below:
There seems to be no command to download data back from C14 Archive

Commands:
create Create a new archive
files List the files of an archive
freeze Lock an archive
help Help of the c14 command line
login Log in to Online API
ls List the archives
rename Rename an archive
remove Remove an archive
unfreeze Unlock an archive
upload Upload your file or directory into an archive

The API exposes a feature to verify (scrub) an archive
We should implement it

/storage/c14/safe/{safe_id}/archive/{archive_id}/location/{location_id}/verify

The console allows the usage of a private token which has all permissions
Could be cool to allow the user to use it

Hello,

the documentation in regards to docker should be extended with the following information:

1. official image on docker hub - i worked around it by creating a fork and publishing it on titpetric/c14-cli,
2. Dockerfile including the VOLUME command, to declare a location needed for data, so it may be used with -v for persistence.

I currently have no idea about the second task and I'm going to have to read some code to find out where c14-cli stores it's data. Edit: the path is /root/.config/c14-cli. Code/help references some $CONFIG which is not an environment variable from what I can tell.

Can i create a file hosting web from c14 storage?

Hey, would it be possible to add a command that would return username, password, host and port for a safe deposit box with given UUID? Optionally, a flag could be used to also set ENV variables with returned values. Also, this could be an option for "create" command, returning everything you need to connected to a newly created safe deposit box.

This would greatly simplify using c14 with rsync, sshfs, davfs, etc.

Hi
When trying to upload via C14 CLI I get this error ..
root@vrayo:~# c14 upload /etc/ 6cd0ab4e-xxxx-4fe8-8207-0614fc5f494b
FATAL[0000] ssh: must specify HostKeyCallback
Host runs Ubuntu 16.04.2 LTS
Go version : go version go1.6.2 linux/amd64

Any idea on how to fix this ? :)

thanks

Gonzague

Starting last weekend I was getting 503 responses from the online.net api when I used c14. Especially when creating new safes. I created a ticket at online.net because I thought I was a problem with api backend. I was then told that this was causes because of rate limiting and that I was polling the api to often. Since I haven't changed anything in c14 with regard to that I'm not sure what caused this.

Is there something we can do in the client to prevent this?

The internal ticket id is #510273 if that helps.

Traces essentially looked like this:

./c14 --debug create --name=test6
 DEBUG[0000] [GET]: https://api.online.net/api/v1/user/key/ssh
 DEBUG[0000] [Response]: [200] [{"uuid_ref":"226e8a7f-b267-421c-9b96-d71bcb595135","description":"gitlab","fingerprint":"11:93:ff:c6:6c:aa:ad:f7:50:eb:89:90:d0:22:95:b3"}]
 DEBUG[0000] [POST]: https://api.online.net/api/v1/storage/c14/safe
 DEBUG[0000] [Response]: [201] {"uuid_ref":"bec8e55b-cd5f-46b4-a512-dfa51c1a3b3a"}
 DEBUG[0000] [GET]: https://api.online.net/api/v1/storage/c14/safe/bec8e55b-cd5f-46b4-a512-dfa51c1a3b3a
 DEBUG[0001] [Response]: [200] {"$ref":"\/api\/v1\/storage\/c14\/safe\/bec8e55b-cd5f-46b4-a512-dfa51c1a3b3a","uuid_ref":"bec8e55b-cd5f-46b4-a512-dfa51c1a3b3a","name":"test6_safe","status":"active","description":""}
 DEBUG[0001] [POST]: https://api.online.net/api/v1/storage/c14/safe/bec8e55b-cd5f-46b4-a512-dfa51c1a3b3a/archive
 DEBUG[0001] [Response]: [503] <html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx</center>
</body>
</html>

 FATAL[0001] /Users/kdummann/go/src/github.com/online-net/c14-cli/pkg/api/api.go:179: [503] <html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx</center>
</body>
</html>

/Users/kdummann/go/src/github.com/online-net/c14-cli/pkg/api/methods.go:186: CreateArchive
/Users/kdummann/go/src/github.com/online-net/c14-cli/pkg/api/helpers.go:56: CreateSSHBucketFromScratch:CreateArchive
/Users/kdummann/go/src/github.com/online-net/c14-cli/pkg/commands/create.go:97: Run:CreateSSHBucketFromScratch

Hey there, I'm a security researcher and I found this project by looking for projects using the InsecureIgnoreHostKey function on Github.

To fix issue #36 (a runtime error that broke the application on a new Go version), this code was introduced.

The error is actually created on purpose to make it clear that implementations needed to do host key checking. Without it, clients are vulnerable to "man in the middle" attacks, where attackers can get in between the user and the server and snoop commands and files or even inject their own. In the case of c14-cli, it seems a man in the middle server could forward the one time password to the server in order to log in as the user and inject commands.

More information is in the golang commit and the blog post by the person who found the issue in Go.

I would suggest adding the host key of the online.net SSH server to the c14-cli if that's possible, or otherwise maybe you can extend the Online web API to also send the trusted SSH host key for host key verification.

I hope you can use this information to improve the security for your users. :-)

I installed the C14 CLI on windows 10pro x64.

I logged in successfully then tried to upload a folder containing lots of subfolders.

It seems that the " \ " char is not considered as a directory separator on windows. So instead of creating subfolders, the tool creates a new folder with name 'Folder\Subfolder" on the same level.

1 picture is better than 1000 words of my poor english, see screenshot attached.

my go version : go version go1.7.1 windows/amd64

Hi,

Can you implement a command that can choose where we want to store an archive?

Thanks

When do we have to use c14 login? Each time by computer or each time we want to use a c14 command line?

There is 4 operations: Archiving, Unarchiving, Destruction and Verification.
What is the equivalent of these CLI operations?
I guess, Archiving is "c14 freeze".
I guess, Unarchiving is "c14 unfreeze".
I guess, Destruction is "c14 remove".
But, which one is for Verification?

Hello,

I'm new to go, and unless I'm mistaken, my version is above 1.6 :

go version go1.11.5 linux/amd64

when I run the go get as displayed in the README.md:

# go get -v -u github.com/scaleway/c14-cli/cmd/c14
github.com/scaleway/c14-cli (download)

No output is given, and then :

bash: c14: command not found

Is there a way to tshoot this ?

CentOS Linux release 7.6.1810 (Core)

golang.x86_64                         1.11.5-1.el7                   @epel
golang-bin.x86_64                     1.11.5-1.el7                   @epel
golang-src.noarch                     1.11.5-1.el7                   @ErwanPaillard 

GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)

Linux nas-centos 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Thanks,

Hello, the CLI was written in GoLang, this is nice, but is there a project to add ansible modules support ? Could be very very great to manage C14 backups like S3/Glacier in ansible

Thanks

As described on https://console.online.net/en/api/ > Authentication > OAuth2 for opensource apps

Request the following URL: GET https://console.online.net/oauth/v2/device/code?client_id=OUR_APPS_CLIENT_ID&new_credentials=yes

You should get an object that gives you a device_code, user_code, and verification_url. (The codes have a TTL of 30 minutes.)

Tell the user to go to the verification URL and enter the value of user_code, then allow your app.
While the user is busy doing that, poll the following URL until you don't have an error anymore: GET https://console.online.net/oauth/v2/device/credentials?client_id=OUR_APPS_CLIENT_ID&code=DEVICE_CODE

That URL should give you an object with client_id and client_secret. These are specific to the user, and can't be used with another user.

With those, you can now request a token: curl -X POST "https://console.online.net/oauth/v2/token" -d "client_id=THE_NEW_CLIENT_ID&client_secret=THE_NEW_CLIENT_SECRET&code=DEVICE_CODE&grant_type=http://oauth.net/grant_type/device/1.0"

I've install Go in a FreeBSD Jail and make command didn't make everything.

First, I've to hack Makefile because it didn't want ifneq:

make: "/root/go/src/github.com/online-net/c14-cli/Makefile" line 23: Missing dependency operator
make: "/root/go/src/github.com/online-net/c14-cli/Makefile" line 25: Need an operator
make: Fatal errors encountered -- cannot continue
make: stopped in /root/go/src/github.com/online-net/c14-cli

Then, it didn't compil at all because it don't understand variable $SOURCES

gofmt -w -s 
error: cannot use -w with standard input
*** Error code 2

Stop.
make: stopped in /root/go/src/github.com/online-net/c14-cli

But command work:

find . -type f -name "*.go" | grep -vE '^./vendor'
./cmd/c14/main.go
./pkg/commands/login.go
./pkg/commands/verify.go
./pkg/commands/command.go
./pkg/commands/help.go
./pkg/commands/freezze.go
./pkg/commands/upload.go
./pkg/commands/remove.go
./pkg/commands/ls.go
./pkg/commands/root.go
./pkg/commands/unfreeze.go
./pkg/commands/files.go
./pkg/commands/rename.go
./pkg/commands/create.go
./pkg/utils/ssh/ssh.go
./pkg/utils/configstore/store.go
./pkg/utils/pgbar/pgbar.go
./pkg/api/methods.go
./pkg/api/cache.go
./pkg/api/error.go
./pkg/api/helpers.go
./pkg/api/auth/auth.go
./pkg/api/api.go
./pkg/api/resources.go
./pkg/version/version.go

I've install Go with pkg:
#go version
go version go1.7.3 freebsd/amd64

Hi,
When using this cli in a script, it should be good to have a quiet or silent option to not output the progress bar but just the status at the end, because when uploading a big file, the output is very long.

Example during a proxmox backup :

102: 2018-01-25 02:40:45 INFO: Uploading vzdump-lxc-102-2018_01_25-00_46_51.tar.gz.gpg to C14
102: 2018-01-25 02:40:47 INFO: vzdump-lxc-102-2018_01_25-00_46_51.tar.gz.gpg [> ] 8.389MB/40.57GB
102: 2018-01-25 02:40:47 INFO: vzdump-lxc-102-2018_01_25-00_46_51.tar.gz.gpg [> ] 16.78MB/40.57GB
102: 2018-01-25 02:40:47 INFO: vzdump-lxc-102-2018_01_25-00_46_51.tar.gz.gpg [> ] 25.17MB/40.57GB
102: 2018-01-25 02:40:47 INFO: vzdump-lxc-102-2018_01_25-00_46_51.tar.gz.gpg [> ] 33.55MB/40.57GB
102: 2018-01-25 02:40:47 INFO: vzdump-lxc-102-2018_01_25-00_46_51.tar.gz.gpg [> ] 41.94MB/40.57GB
102: 2018-01-25 02:40:48 INFO: vzdump-lxc-102-2018_01_25-00_46_51.tar.gz.gpg [> ] 50.33MB/40.57GB
102: 2018-01-25 02:40:48 INFO: vzdump-lxc-102-2018_01_25-00_46_51.tar.gz.gpg [> ] 58.72MB/40.57GB
102: 2018-01-25 02:40:48 INFO: vzdump-lxc-102-2018_01_25-00_46_51.tar.gz.gpg [> ] 67.11MB/40.57GB
102: 2018-01-25 02:40:48 INFO: vzdump-lxc-102-2018_01_25-00_46_51.tar.gz.gpg [> ] 75.5MB/40.57GB
102: 2018-01-25 02:40:48 INFO: vzdump-lxc-102-2018_01_25-00_46_51.tar.gz.gpg [> ] 83.89MB/40.57GB
…

root@vrayo:~# c14 create --safe "MySafe" FATAL[0000] [409] : This name is already used
When trying to create an archive in a safe which already exists ( because c14 create without a safe name creates a new safe each time which I dont want) ... it fails :(

What is the expected behaviour ?
I would like to do c14 create --name "MyArchive" --description "A new archive" --safe "MySafe"`` where MySafe already exists and this adds an archive to this safe

thanks !

We should synchronize with the Online API team to allow the form fields to be prefilled using the URL query string, i.e:

Actual

$ c14 login
! You must provide a ClientID !

Please opens this link with your web browser: https://console.online.net/en/api/apps
You can create a new app with the following fields:

---
    App name:    c14
    Description: c14 cli
    Permissions:
        (c14)  Read/Write
        (user) Read/Write

---
Then copy the client_id here : XXXXXXXXXXXXXXXXXXXXXXXXX

Please opens this link with your browser: https://console.online.net/oauth/v2/device/usercode
Then copy paste the code XXXXXXXX (enter when is done) :

expected

➜  c14-cli git:(master) ./c14 login
! You must provide a ClientID !

Please opens this link with your web browser: https://console.online.net/en/api/apps/?app_name=c14&app_description=c14+cli&c14_permissions=rw&user_permissions=rw
Then copy the client_id here :XXXXXXXXXXXXXXXXXXXXXXXXX

Please opens this link with your browser: https://console.online.net/oauth/v2/device/usercode?code= XXXXXXXX