nytimes / gcs-helper Goto Github PK

Hi,

I was testing out the container, and I cannot seem to get a hole through to GCS with a service account credential file.

I have set GOOGLE_APPLICATION_CREDENTIALS to point to my json file. And I have verified that my key has access to the specified resources.

I modified the following function in proxy.go to give some more output:

func objectHandle(c *Config, client *storage.Client, r *http.Request) *storage.ObjectHandle {
	bucketName := c.BucketName
	objectName := strings.TrimLeft(r.URL.Path, "/")
	if c.Proxy.BucketOnPath {
		pos := strings.Index(objectName, "/")
		bucketName = objectName[:pos]
		objectName = objectName[pos+1:]
	}

	fmt.Printf("BucketName: %s ObjName: %s\n", bucketName, objectName)

	rc, err := client.Bucket(bucketName).Object(objectName).NewReader(context.Background())
	if err != nil {
		fmt.Println("Got error reading file from GCS:")
		fmt.Println(err)
	}
	slurp, err := ioutil.ReadAll(rc)
	rc.Close()
	if err != nil {
		fmt.Println("Got error reading byte stream:")
		fmt.Println(err)
	}
	fmt.Println("file contents:", slurp)

	return client.Bucket(bucketName).Object(objectName)
}

And i get the following output:

$ docker run -e "GCS_HELPER_BUCKET_NAME=my-bucket-name" -e "GOOGLE_APPLICATION_CREDENTIALS=/credentials.json" -p 8080:8080 -it t  
INFO[0000] Listening on [::]:8080...                    
BucketName: my-bucket-name ObjName: test_report/index.html
Got error reading file from GCS:
Get https://storage.googleapis.com/my-bucket-name/test_report/index.html: oauth2: cannot fetch token: 400 Bad Request
Response: {
  "error": "invalid_scope",
  "error_description": "Empty or missing scope not allowed."
}

Is this something anyone have seen before? I have never had to set any scopes before when I previously used service accounts as json files.

If anyone can point me in the right direction, I'll volunteer to help update the documentation accordingly. I tried googling around, but most things I found are talking about Compute Instance scopes which does not sound applicable.

Hey guys, there's a tag, v2 listed in the Docker Hub API but there's no manifest for it, which can mess with FluxCD (see fluxcd/flux#2517 as an example):

TOKEN=`curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:nytimes/gcs-helper:pull" | jq .token | tr -d \"`
curl  -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/nytimes/gcs-helper/tags/list
curl  -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/nytimes/gcs-helper/manifests/v2

If you have the time is there any chance you could remove the v2 tag from the registry please?

I have a private GCS bucket that is only accessible to staff members - we do that using the GCS-Helper proxy that is mapped to an internal hostname (GCS Proxy deployed to K8s).

Everything works great for the most part - one thing that does not is that even though I've set the GCS web config via:
gsutil web set -m index.html -e index.html gs://www.example.com

Requests to a folder like "/a" do not forward requests to "/a/index.html" and I just get a:

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Details>No such object: example.com/a</Details>
</Error>

Has anyone figured out how to get GCS-Helper working with default documents?

Would you be interested in taking a PR that would show the content of a folder instead of the current placeholder?
Our use-case is building a small file browser that shows sub-folder content artifacts.
It would be an opt-in enabled with ie. GCS_HELPER_FOLDER_LISTING=1 and something to control the max amount of files returned as to not return massive payloads.
Ideally we would like to avoid maintaining our own fork and just have it be a feature of gcs-helper
Thanks!

The gcs-helper closes the connection at around 30-40 MB transferred:

$ wget -v 127.0.0.1:8080/files/test 
--2020-02-21 19:43:05--  http://127.0.0.1:8080/files/test
Connecting to 127.0.0.1:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1186458112 (1.1G) [application/octet-stream]
Saving to: 'test'

test.1                                                                                                                               3%[============>                                                                                                                                                                                                                                                                                                                                                                                                                                         ]  34.13M  8.24MB/s    in 4.1s    

2020-02-21 19:43:10 (8.24 MB/s) - Connection closed at byte 35790848. Retrying.

I have reproduced the behavior with Chrome and curl. Strangely, the byte amount it closes at is not consistent.

Hi,

I was testing out the container, and I cannot seem to get a hole through to GCS with a service account credential file.

I have set GOOGLE_APPLICATION_CREDENTIALS to point to my json file. And I have verified that my key has access to the specified resources.

I modified the following function in proxy.go to give some more output:

func objectHandle(c *Config, client *storage.Client, r *http.Request) *storage.ObjectHandle {
	bucketName := c.BucketName
	objectName := strings.TrimLeft(r.URL.Path, "/")
	if c.Proxy.BucketOnPath {
		pos := strings.Index(objectName, "/")
		bucketName = objectName[:pos]
		objectName = objectName[pos+1:]
	}

	fmt.Printf("BucketName: %s ObjName: %s\n", bucketName, objectName)

	rc, err := client.Bucket(bucketName).Object(objectName).NewReader(context.Background())
	if err != nil {
		fmt.Println("Got error reading file from GCS:")
		fmt.Println(err)
	}
	slurp, err := ioutil.ReadAll(rc)
	rc.Close()
	if err != nil {
		fmt.Println("Got error reading byte stream:")
		fmt.Println(err)
	}
	fmt.Println("file contents:", slurp)

	return client.Bucket(bucketName).Object(objectName)
}

And i get the following output:

$ docker run -e "GCS_HELPER_BUCKET_NAME=my-bucket-name" -e "GOOGLE_APPLICATION_CREDENTIALS=/credentials.json" -p 8080:8080 -it t  
INFO[0000] Listening on [::]:8080...                    
BucketName: my-bucket-name ObjName: test_report/index.html
Got error reading file from GCS:
Get https://storage.googleapis.com/my-bucket-name/test_report/index.html: oauth2: cannot fetch token: 400 Bad Request
Response: {
  "error": "invalid_scope",
  "error_description": "Empty or missing scope not allowed."
}

Is this something anyone have seen before? I have never had to set any scopes before when I previously used service accounts as json files.

If anyone can point me in the right direction, I'll volunteer to help update the documentation accordingly. I tried googling around, but most things I found are talking about Compute Instance roles which does not sound applicable.

I'm trying to add bitrate value with sequence

Added bitrate struct in Sequence struct
type Sequence struct { Clips []Clip json:"clips"Bitrates Bitratejson:"bitrate"}

bitrate struct
type Bitrate struct { V stringjson:"v"A stringjson:"a"}

Append sequence to seqs variable
seqs = append(seqs, Sequence{ Clips: []Clip{{Type: "source", Path: "/" + obj.Bucket + "/" + obj.Name}}, Bitrates: {{"v": 900000, "a": 64000}}, })

When run it's give missing type in composite literal error
# github.com/NYTimes/gcs-helper/v3/vodmodule vodmodule/mapper.go:60:16: missing type in composite literal

Thanks for this useful app :)

Would it be possible to run this as a non-root container (can submit a PR if you want).

I'm trying this in k8s, and it works fine, but currently having to override GOPS_CONFIG_DIR, as well as set the securityContext to run as nobody.

Not sure there's any reason why this needs to run as root?

Thanks! :)

I'm trying to build the docker image and am having problems running it:

go build -o gcs-helper
docker build -t gcs-helper:latest .
docker run -t gcs-helper:latest
standard_init_linux.go:207: exec user process caused "exec format error"

I'm running on macOS, go version go1.14.2 darwin/amd64, docker 18.09.1

Hi

Can you provide some samples on how to use gcs-helper with Kaltura VOD?

best regards

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

github.com/NYTimes/gcs-helper/v3/handlers: cannot find module providing package github.com/NYTimes/gcs-helper/v3/handlers

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.