vdaas / vald Goto Github PK

Vulnerabilities

DepShield reports that this application's usage of golang.org/x:crypto:0.0.0-20181030102418-4d3f4d9ffa16 results in the following vulnerability(s):

• (CVSS 5.9) [CVE-2019-11840] Use of Insufficiently Random Values

Occurrences

golang.org/x:crypto:0.0.0-20181030102418-4d3f4d9ffa16 is a transitive dependency introduced by the following direct dependency(s):

• github.com/francoispqt:gojay:1.2.13
        └─ cloud.google.com:go:0.37.0
              └─ golang.org/x:build:0.0.0-20190111050920-041ab4dc3f9d
                    └─ golang.org/x:crypto:0.0.0-20181030102418-4d3f4d9ffa16

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Now in vald, all error definition is defined in internal/errors package.
But I don't think it is a good practice in golang, as the error should belongs to meaningful package instead of misleading errors package. Even in golang implementation it is definied under the corresponding package, for example:
https://golang.org/src/net/http/server.go :Line974

Is your feature request related to a problem? Please describe.:

When the option return error, currently the implementation will use the default value and continue the execution.
But the problem is that the users do not know the default value is used because theres is no error message is printed when error occurred.

For example:
https://github.com/vdaas/vald/blob/master/internal/servers/option.go#L58

Describe the solution you'd like:

1. Print out the error and terminate the program.
   Users may need to understand all the parameters and set it correctly.
   We provide helm chart with default values and users may not touch it if they don't understand the parameter.
2. Print out the error only
   Log the error and notify users that the default value is used. User may not see the log and the default value is used without any intention.

Describe alternatives you've considered:

Describe the solution you'd like:

I integrated all-contributors bot.
so I'd like to use it, this issue is not necessary

Describe alternatives you've considered:

Is your feature request related to a problem? Please describe.:

In my case, I want to disable internal DNS resolver, but I cannot do it now.

Describe the solution you'd like:

Add a option to disable internal DNS resolver.

Describe alternatives you've considered:

I use raw gRPC connection.

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

go: github.com/envoyproxy/[email protected]: reading github.com/envoyproxy/protoc-gen-validate/go.mod at revision v0.3.0: unknown revision v0.3.0

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

The project could not be analyzed because of build errors. Please review the error messages here. Another build will be scheduled within 24 hours. If the build is successful this issue will be closed, otherwise the error message will be updated.

_{This is an automated GitHub Issue created by Sonatype DepShield. GitHub Apps, including DepShield, can be managed from the Developer settings of the repository administrators.}

Describe the bug:

with invalid NGT backup files, pkg/agent/core/ngt/service/New blocks startup process of agent-ngt.

To Reproduce:

put invalid backup files in index_path and enable_in_memory_mode=false.

Expected behavior:

Maybe there are two options to fallback the application. 🤔

• restart pod and retry to load
• start daemon without backup data.

Environment:

• Go Version: 1.14.3
• Docker Version: 19.03.8
• Kubernetes Version: 1.18.2
• NGT Version: 1.11.5

Vulnerabilities

DepShield reports that this application's usage of golang.org/x:crypto:0.0.0-20190313024323-a1f597ede03a results in the following vulnerability(s):

• (CVSS 5.9) [CVE-2019-11840] Use of Insufficiently Random Values

Occurrences

golang.org/x:crypto:0.0.0-20190313024323-a1f597ede03a is a transitive dependency introduced by the following direct dependency(s):

• github.com/francoispqt:gojay:1.2.13
        └─ golang.org/x:crypto:0.0.0-20190313024323-a1f597ede03a

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Describe the bug:

It have been found the same bug in 2 different package (tls and cache) so just in case I create a bug report for it.
Using defaultOps as var in option.go may create potential bugs in the program as it share the same instance to all callers.

I suggest to change all the implementation of defaultOps to from var to func()? (like this commit)
db434cb

To Reproduce:

Expected behavior:

Environment:

• Go Version: 1.14.3
• Docker Version: 19.03.8
• Kubernetes Version: 1.18.2
• NGT Version: 1.11.5

We are considering using vald for an internal project. I had a couple of questions about vald:

• Can vald handle sparse vector efficiently? The vector we are working with contains from 0-1% of 1 value (binary vectors). The architecture page mentions "compressed vector" so maybe the compression used is already efficient according to the type of vector used?
• Have you already scaled vald to an index of 5-10 billion vectors?
• Can a single deployed vald instance deal with multiple types of indexes? Let's say one sparse binary of size 512 and another one continuous float16 of size 128. This setup example will have two separate indexes (associated with two different distance types) and a request could choose only of them.
• What type is allowed for the ID/UUID of the vectors? Ideally, we would be using a custom UUID made of ASCI characters of size 27.
• It's not clear from the doc and diagram whether the index vectors is stored and backup to a database (Cassandra, Scylla, etc) or on GCS.

Thanks. Vald looks like a really nice project!

The project could not be analyzed because of build errors. Please review the error messages here. Another build will be scheduled when a change to a manifest file^* occurs. If the build is successful this issue will be closed, otherwise the error message will be updated.

_{This is an automated GitHub Issue created by Sonatype DepShield. GitHub Apps, including DepShield, can be managed from the Developer settings of the repository administrators.}

_{* Supported manifest files are: pom.xml, package.json, package-lock.json, npm-shrinkwrap.json, Cargo.lock, Cargo.toml, main.rs, lib.rs, build.gradle, build.gradle.kts, settings.gradle, settings.gradle.kts, gradle.properties, gradle-wrapper.properties, go.mod, go.sum}

Describe the bug:

As described in the PR, there's a problem that Cassandra cluster doesn't start correctly if it is deployed by using k8s/external/cassandra/deploy task.
#948 (comment)

Maybe caused by the changes in cassandra:latest container.

It seems better to use k8ssandra or cassandra-operator like k8s/external/scylla/deploy task.

To Reproduce:

make k8s/external/cassandra/deploy

Expected behavior:

Start Cassandra correctly.

Should we use "vald" as K8s default namespaces?

Vulnerabilities

DepShield reports that this application's usage of github.com/gorilla:websocket:1.4.0 results in the following vulnerability(s):

• (CVSS 7.5) CWE-190: Integer Overflow or Wraparound

Occurrences

github.com/gorilla:websocket:1.4.0 is a transitive dependency introduced by the following direct dependency(s):

• github.com/cockroachdb:errors:0.0.0-00010101000000-000000000000
        └─ github.com/cockroachdb:sentry-go:0.3.999
              └─ github.com/kataras/iris:v12:12.0.1
                    └─ github.com/kataras:neffos:0.0.10
                          └─ github.com/gorilla:websocket:1.4.0

• sigs.k8s.io:controller-runtime:0.0.0-00010101000000-000000000000
        └─ k8s.io:apiextensions-apiserver:0.18.2
              └─ k8s.io:apiserver:0.18.2
                    └─ github.com/gorilla:websocket:1.4.0

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

go: github.com/envoyproxy/[email protected]: reading github.com/envoyproxy/protoc-gen-validate/go.mod at revision v0.3.0: unknown revision v0.3.0

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Vulnerabilities

DepShield reports that this application's usage of golang.org/x:crypto:0.0.0-20180904163835-0709b304e793 results in the following vulnerability(s):

• (CVSS 5.9) [CVE-2019-11840] Use of Insufficiently Random Values

Occurrences

golang.org/x:crypto:0.0.0-20180904163835-0709b304e793 is a transitive dependency introduced by the following direct dependency(s):

• github.com/kpango:gache:1.1.22
        └─ github.com/prologic:bitcask:0.3.4
              └─ github.com/spf13:viper:1.4.0
                    └─ github.com/prometheus:client_golang:0.9.3
                          └─ github.com/prometheus:common:0.4.0
                                └─ github.com/sirupsen:logrus:1.2.0
                                      └─ golang.org/x:crypto:0.0.0-20180904163835-0709b304e793

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

We suggest the following example test template of the testing guideline.
We suggest to use gotests to generate the test code, and we will modify the template of gotests to generate the following test code.

The example implementation is here.
https://github.com/vdaas/vald/tree/proposal/tls_test/internal/tls

Please note that:

1. The args struct will only be generated if target function accept arguments.
2. The fields struct will only be generated if target struct have fields.
3. For the option.go template, the comments will be generated with the source code, user need to decide which source code to use.

func Test_person_Hoge4(t *testing.T) {
	type args struct { 
		err error
	}
	type fields struct {
		name string
	}
	type want struct {
		want  string
		want1 string
		err   error
	}
	type test struct {
		name       string
		args       args
		fields     fields
		want       want
		checkFunc  func(want, string, string, error) error
		beforeFunc func(args)
		afterFunc  func(args)
	}
	defaultCheckFunc := func(w want, got string, got1 string, err error) error {
		if !errors.Is(err, w.err) {
			return errors.Errorf("got error = %v, want %v", err, w.err)
		}
		if !reflect.DeepEqual(got, w.want) {
			return errors.Errorf("got = %v, want %v", got, w.want)
		}
		if !reflect.DeepEqual(got1, w.want1) {
			return errors.Errorf("got = %v, want %v", got1, w.want1)
		}
		return nil
	}
	tests := []test{
		// TODO test cases
		{
			name:      "test_case_1",
			args:      args{},
			fields:    fields{},
			want:      want{},
			checkFunc: defaultCheckFunc,
		},
                 // TODO test cases
		func() test {
			return test {
				name:      "test_case_2",
				args:      args{},
				fields:    fields{},
				want:      want{},
				checkFunc: defaultCheckFunc,
			}
		}(),
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if tt.beforeFunc != nil {
				tt.beforeFunc(tt.args)
			}
			if tt.afterFunc != nil {
				defer tt.afterFunc(tt.args)
			}
			if tt.checkFunc == nil {
				tt.checkFunc = defaultCheckFunc
			}
			p := &person{
				name: tt.fields.name,
			}

			got, got1, err := p.Hoge4(tt.args.err)
			if err := tt.checkFunc(tt.want, got, got1, err); err != nil {
				t.Errorf("error = %v", err)
			}

		})
	}
}

Option.go template:

func TestWithAddr(t *testing.T) {
	type T = interface{}
	type args struct {
		addr string
		a    int
		b    int
		c    int
		d    int
	}
	type want struct {
		obj *T
		// Uncomment this line if the option returns an error, otherwise delete it
		// err error
	}
	type test struct {
		name string
		args args
		want want
		// Use the first line if the option returns an error. otherwise use the second line
		// checkFunc  func(want, *T, error) error
		// checkFunc  func(want, *T) error
		beforeFunc func(args)
		afterFunc  func(args)
	}

	// Uncomment this block if the option returns an error, otherwise delete it
	/*
	   defaultCheckFunc := func(w want, obj *T, err error) error {
	       if !errors.Is(err, w.err) {
	           return errors.Errorf("got error = %v, want %v", err, w.err)
	       }
	       if !reflect.DeepEqual(obj, w.obj) {
	           return errors.Errorf("got = %v, want %v", obj, w.obj)
	       }
	       return nil
	   }
	*/

	// Uncomment this block if the option do not returns an error, otherwise delete it
	/*
	   defaultCheckFunc := func(w want, obj *T) error {
	       if !reflect.DeepEqual(obj, w.obj) {
	           return fmt.Errorf("got = %v, want %v", obj, w.c)
	       }
	       return nil
	   }
	*/

	tests := []test{
                 // TODO test cases
		{
			name:      "test_case_1",
			args:      args{},
			want:      want{},
			// checkFunc: defaultCheckFunc,
		},
                 // TODO test cases
		func() test {
			return test {
				name:      "test_case_2",
				args:      args{},
				fields:    fields{},
				want:      want{},
				checkFunc: defaultCheckFunc,
			}
		}(),
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if tt.beforeFunc != nil {
				tt.beforeFunc(tt.args)
			}
			if tt.afterFunc != nil {
				defer tt.afterFunc(tt.args)
			}

			// Uncomment this block if the option returns an error, otherwise delete it
			/*
			   if tt.checkFunc == nil {
			       tt.checkFunc = defaultCheckFunc
			   }

			   got := WithAddr(tt.args.addr, tt.args.a, tt.args.b, tt.args.c, tt.args.d)
			   obj := new(T)
			   if err := tt.checkFunc(tt.want, obj, got(obj)); err != nil {
			       t.Errorf("error = %v", err)
			   }
			*/

			// Uncomment this block if the option do not returns an error, otherwise delete it
			/*
			   if tt.checkFunc == nil {
			       tt.checkFunc = defaultCheckFunc
			   }

			   got := WithAddr(tt.args.addr, tt.args.a, tt.args.b, tt.args.c, tt.args.d)
			   obj := new(T)
			   got(obj)
			   if err := tt.checkFunc(tt.want, obj); err != nil {
			       t.Errorf("error = %v", err)
			   }
			*/
		})
	}
}

Is there any way I can delete indexes. I created some for testing. I am using scyalla db. Are the indexes stored in db?

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

gomodules.xyz/jsonpatch/[email protected]: unrecognized import path "gomodules.xyz/jsonpatch/v2" (parse https://gomodules.xyz/jsonpatch/v2?go-get=1: no go-import meta tags ())

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

go: github.com/envoyproxy/[email protected]: reading github.com/envoyproxy/protoc-gen-validate/go.mod at revision v0.3.0: unknown revision v0.3.0

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Describe the bug:

The schema that will create the initializer job seems not corret

kubectl logs -f mysql-init-gprsh
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1072 (42000) at line 6: Key column 'meta' doesn't exist in table

To Reproduce:

Deploy helm chart vald and enable initializer

The job try to execute

CREATE TABLE IF NOT EXISTS `vald`.`backup_vector` (
    ->         `uuid` VARCHAR(255) NOT NULL,
    ->         `vector` BLOB NOT NULL,
    ->         `id` int NOT NULL AUTO_INCREMENT,
    ->         PRIMARY KEY (`uuid`),
    ->         UNIQUE INDEX `id_unique` (`id` ASC),
    ->         UNIQUE INDEX `meta_unique` (`meta` ASC)
    ->     )
    ->     ENGINE = InnoDB;

but the meta column doesn't exist and the index creation fail.

Expected behavior:

Environment:

• Go Version: 1.15.6
• Docker Version: 19.03.8
• Kubernetes Version: 1.19.2
• NGT Version: 1.12.3

Vulnerabilities

DepShield reports that this application's usage of golang.org/x:crypto:0.0.0-20181203042331-505ab145d0a9 results in the following vulnerability(s):

• (CVSS 5.9) [CVE-2019-11840] Use of Insufficiently Random Values

Occurrences

golang.org/x:crypto:0.0.0-20181203042331-505ab145d0a9 is a transitive dependency introduced by the following direct dependency(s):

• github.com/kpango:gache:1.1.22
        └─ github.com/prologic:bitcask:0.3.4
              └─ github.com/spf13:cobra:0.0.5
                    └─ github.com/spf13:viper:1.3.2
                          └─ golang.org/x:crypto:0.0.0-20181203042331-505ab145d0a9

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Related: #685

Currently, Agent-NGT requires AVX2 instructions support of the running node.
In the future, we will provide Agent-NGT without AVX using Docker tags like vX.X.X-noavx or something.

To disable AVX instructions, build options are needed to add like #685.
Agent-NGT without AVX instructions takes longer time to calculate distances.

All of the package document contains in the source code file, instead of separated doc.go file, which affect the readability of the source code.
Each package should contains a doc.go file to describe the package.

Also some package doc is missing, like https://github.com/vdaas/vald/tree/master/internal.

helm install vald vald/vald --values example/helm/values-scylla.yaml

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "Ingress" in version "networking.k8s.io/v1"

I tried to set the value of usev1beta1 to true there in the file - ./charts/vald/values.yaml but still getting the error.

Can someone please help?

Environment:

• Go Version: 1.16
• Docker Version: 19.03.8
• Kubernetes Version: 1.18.2
• NGT Version: 1.12.3

Vulnerabilities

DepShield reports that this application's usage of golang.org/x:crypto:0.0.0-20190308221718-c2843e01d9a2 results in the following vulnerability(s):

• (CVSS 5.9) [CVE-2019-11840] Use of Insufficiently Random Values

Occurrences

golang.org/x:crypto:0.0.0-20190308221718-c2843e01d9a2 is a transitive dependency introduced by the following direct dependency(s):

• github.com/kpango:gache:1.1.22
        └─ github.com/prologic:bitcask:0.3.4
              └─ github.com/spf13:viper:1.4.0
                    └─ golang.org/x:net:0.0.0-20190522155817-f3200d17e092
                          └─ golang.org/x:crypto:0.0.0-20190308221718-c2843e01d9a2
                    └─ google.golang.org:grpc:1.21.0
                          └─ golang.org/x:lint:0.0.0-20190313153728-d0100b6bd8b3
                                └─ golang.org/x:tools:0.0.0-20190311212946-11955173bddd
                                      └─ golang.org/x:net:0.0.0-20190311183353-d8887717615a
                                            └─ golang.org/x:crypto:0.0.0-20190308221718-c2843e01d9a2
                          └─ golang.org/x:net:0.0.0-20190311183353-d8887717615a
                                └─ golang.org/x:crypto:0.0.0-20190308221718-c2843e01d9a2
                          └─ golang.org/x:tools:0.0.0-20190311212946-11955173bddd
                                └─ golang.org/x:net:0.0.0-20190311183353-d8887717615a
                                      └─ golang.org/x:crypto:0.0.0-20190308221718-c2843e01d9a2
              └─ golang.org/x:exp:0.0.0-20190731235908-ec7cb31e5a56
                    └─ golang.org/x:mod:0.1.0
                          └─ golang.org/x:crypto:0.0.0-20190510104115-cbcb75029529
                                └─ golang.org/x:net:0.0.0-20190404232315-eb5bcb51f2a3
                                      └─ golang.org/x:crypto:0.0.0-20190308221718-c2843e01d9a2
                    └─ golang.org/x:tools:0.0.0-20190312151545-0bb0c0a6e846
                          └─ golang.org/x:net:0.0.0-20190311183353-d8887717615a
                                └─ golang.org/x:crypto:0.0.0-20190308221718-c2843e01d9a2

• google.golang.org:grpc:1.23.1
        └─ golang.org/x:lint:0.0.0-20190313153728-d0100b6bd8b3
              └─ golang.org/x:tools:0.0.0-20190311212946-11955173bddd
                    └─ golang.org/x:net:0.0.0-20190311183353-d8887717615a
                          └─ golang.org/x:crypto:0.0.0-20190308221718-c2843e01d9a2
        └─ golang.org/x:net:0.0.0-20190311183353-d8887717615a
              └─ golang.org/x:crypto:0.0.0-20190308221718-c2843e01d9a2
        └─ golang.org/x:tools:0.0.0-20190524140312-2c0ae7006135
              └─ golang.org/x:net:0.0.0-20190311183353-d8887717615a
                    └─ golang.org/x:crypto:0.0.0-20190308221718-c2843e01d9a2

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Is your feature request related to a problem? Please describe.:

The current implementation uses sync.Mutex and map to cache objects.
Removing the lock logic will improve performance.

Describe the solution you'd like:

Since locks and unlocks occur frequently, replacing them with sync.Map can be made lock-free, which is considered to speed up the process.

Describe alternatives you've considered:

Change from sync.Mutex and map to sync.Map to cache object.

Vulnerabilities

DepShield reports that this application's usage of golang.org/x:crypto:0.0.0-20190211182817-74369b46fc67 results in the following vulnerability(s):

• (CVSS 5.9) [CVE-2019-11840] Use of Insufficiently Random Values

Occurrences

golang.org/x:crypto:0.0.0-20190211182817-74369b46fc67 is a transitive dependency introduced by the following direct dependency(s):

• k8s.io:client-go:0.0.0-20191114101535-6c5935290e33
        └─ github.com/gophercloud:gophercloud:0.1.0
              └─ golang.org/x:crypto:0.0.0-20190211182817-74369b46fc67

• sigs.k8s.io:controller-runtime:0.0.0-00010101000000-000000000000
        └─ k8s.io:apiextensions-apiserver:0.0.0-20190918161926-8f644eb6e783
              └─ k8s.io:component-base:0.0.0-20191114102325-35a9586014f7
                    └─ k8s.io:client-go:0.0.0-20191114101535-6c5935290e33
                          └─ github.com/gophercloud:gophercloud:0.1.0
                                └─ golang.org/x:crypto:0.0.0-20190211182817-74369b46fc67
              └─ k8s.io:client-go:0.0.0-20191114101535-6c5935290e33
                    └─ github.com/gophercloud:gophercloud:0.1.0
                          └─ golang.org/x:crypto:0.0.0-20190211182817-74369b46fc67
              └─ k8s.io:apiserver:0.0.0-20191114103151-9ca1dc586682
                    └─ k8s.io:component-base:0.0.0-20191114102325-35a9586014f7
                          └─ k8s.io:client-go:0.0.0-20191114101535-6c5935290e33
                                └─ github.com/gophercloud:gophercloud:0.1.0
                                      └─ golang.org/x:crypto:0.0.0-20190211182817-74369b46fc67
                    └─ k8s.io:client-go:0.0.0-20191114101535-6c5935290e33
                          └─ github.com/gophercloud:gophercloud:0.1.0
                                └─ golang.org/x:crypto:0.0.0-20190211182817-74369b46fc67
        └─ k8s.io:client-go:0.0.0-20190918160344-1fbdaa4c8d90
              └─ github.com/gophercloud:gophercloud:0.1.0
                    └─ golang.org/x:crypto:0.0.0-20190211182817-74369b46fc67

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Describe the bug:

As @kpango reported, there are many missing boundary check on the slice. For example
https://github.com/vdaas/vald/blob/master/internal/core/ngt/ngt.go#L284

We may need to check the vec[0] is exists or not first.

One of the solution may be the following

if len(vec) == 0 {
// handle
}

ret := C.ngt_search_index_as_float(
	n.index,
	(*C.float)(&vec[0]),
         .....

But in some case we do not need to do the boundary check, for example:
https://github.com/vdaas/vald/blob/master/internal/info/info.go#L170

To Reproduce:

Expected behavior:

Environment:

• Golang Version: 1.13.5
• Docker Version: 19.03.5
• Kubernetes Version: 1.16.3
• NGT Version: 1.8.4

Describe the issue:

When the backup files become large, sometimes agent sidecar writes uncompleted backup files to object storage bucket.
This is because agent NGT does not write backup files at once.

Solution:

agent NGT writes backup metadata file after finished to write whole of the backup files.
agent sidecar watches metadata changes.

Environment:

• Go Version: 1.14.3
• Docker Version: 19.03.8
• Kubernetes Version: 1.18.2
• NGT Version: 1.11.5

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

go: github.com/envoyproxy/[email protected]: reading github.com/envoyproxy/protoc-gen-validate/go.mod at revision v0.3.0: unknown revision v0.3.0

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Vulnerabilities

DepShield reports that this application's usage of golang.org/x:crypto:0.0.0-20180820150726-614d502a4dac results in the following vulnerability(s):

• (CVSS 5.9) [CVE-2019-11840] Use of Insufficiently Random Values

Occurrences

golang.org/x:crypto:0.0.0-20180820150726-614d502a4dac is a transitive dependency introduced by the following direct dependency(s):

• sigs.k8s.io:controller-runtime:0.2.1
        └─ golang.org/x:crypto:0.0.0-20180820150726-614d502a4dac

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Describe the bug:

We implement the test code based on the code automatically generated from the template. However, when I execute the go test command, the error occurs in internal / errors. The details of the error are cyclic imports.

To Reproduce:

I think it's because the template uses errors.Errorf statically. Therefore, in case of errors in the same package, I want to put an implementation that removes the errors prefix in the template. It also fixed existing errors.

Expected behavior:

we expect that no occurs cyclic imports in the internal/errors.
and It also fixed existing errors.

Environment:

• Golang Version: 1.13.5
• Docker Version: 19.03.5
• Kubernetes Version: 1.16.3
• NGT Version: 1.8.4

Describe the bug:

If the operator.release.create field of the Vald-Helm-Operator Helm Chart is enabled, when an operator leader instance is shutting down, another operator will freeze after becomes the leader.
This may be caused by deadlocks between helm release and ValdHelmOperatorRelease resource.

To Reproduce:

Just deploy operator with the following command.

$ helm install vald-helm-operator-release vald/vald-helm-operator

and then, kill the operator instance

$ kubectl delete pods vald-helm-operator-xxx

workaround

when the operator.release.create is disabled, this bug doesn't happen even if i apply a ValdHelmOperatorRelease resource manually.
So, the default value should be disabled and revise the user guides.

I think Helm's improvements (helm/helm#7649) may be a fix of this issue, but it is coming in Helm v3.2.0 release. I'll try it after released.

Expected behavior:

the new instance of operator works correctly.

Environment:

• Vald Version: v0.0.28
• Operator SDK Version: v0.16.0
• Kubernetes Version: 1.17.0
• Helm Version: 3.0.1

Is your feature request related to a problem? Please describe.:

[feature request & discussion]

As we discussed in #945, I think similar problems will occur not only in our environment, but in many user environments and cloud providers in the future.

Describe the solution you'd like:

I thought we should set a rule for the Vald's k8s apiVersion support and support at least two apiVersions in our Helm Chart, internal/k8s components.
To discuss this, I bring up this issue, please let me know what you think.

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

go: github.com/envoyproxy/[email protected]: reading github.com/envoyproxy/protoc-gen-validate/go.mod at revision v0.3.0: unknown revision v0.3.0

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Issue

After Vald receive the SearchByVectorID request, Vald LB Gateway will forward the UUID to the Vald Agent to perform the search.

For example we have 4 Vald Agents:

When user perform the searchByVectorID where the vector ID matches the vector V2, only Agent 1 and 3 can return the result, but the most accurate result maybe stored in Agent 2 or Agent 4.

To Reproduce:

Expected behavior:

All the Vald Agent should be able to perform the search.

Environment:

• Golang Version: 1.13.5
• Docker Version: 19.03.5
• Kubernetes Version: 1.16.3
• NGT Version: 1.8.4

What is this proposal

All of the components in Vald have a functionality to export their metrics to Prometheus or other external monitoring system. (#131)
In this proposal, i'm going to discuss what metrics are needed for each components and how to implement them.

metrics considered to be added (including already implemented)

🌱 ... just an idea. need to be discussed
🍃 ... can be split into smaller metrics. need to be discussed

Commons

• gRPC server metrics (latency, receive bytes, send bytes) using ocgrpc interceptor
• memory metrics (heap, stack, sys, etc...)
• goroutine count
• GC count
• CGO call count
• gRPC connection rebalance cumulative count 🌱
• gRPC connected server count 🌱
  • maybe gRPCClient.clientCount in internal/net/grpc/client.go
• received request cumulative count 🌱
• completed request cumulative count 🌱 (achieved by ocgrpc)

Agent

• NGT indices
• NGT uncommitted indices
• NGT insert/delete vcache
• indexing flag
• indexing time total
• completed indexing process cumulative count #389
• received/completed search query request cumulative count 🌱
• received/completed insert request cumulative count 🌱
• received/completed update request cumulative count 🌱
• received/completed delete request cumulative count 🌱

Sidecar

• backup file size bytes
• completed backup cumulative count
• failed backup cumulative count
• time required to upload file (cumulative?)

Gateway

• received/completed search query request cumulative count 🌱
• received/completed insert request cumulative count 🌱
• received/completed update request cumulative count 🌱
• received/completed delete request cumulative count 🌱

Discoverer

• reconcile cumulative count 🌱

Backup Manager

Cassandra

• completed query cumulative count
• query latency

MySQL

• completed query cumulative count 🍃
• query latency

Compressor

• requested registerer jobs cumulative count
• completed registerer jobs cumulative count
• registerer jobs count in queue
• requested compressor jobs cumulative count
• completed compressor jobs cumulative count
• compressor jobs count in queue

Index Manager

• UUID count
• uncommitted UUID count
• indexing flag

Replication Manager

Agent

Controller

Meta

Cassandra

• completed query cumulative count 🍃
• query latency

Redis

• completed query cumulative count 🍃
• query latency

Is your feature request related to a problem? Please describe.:

Current Runner interface doesn't provide success completion.
In some cases, I want to terminate Runner with success.

Describe the solution you'd like:

Runner#Start returns (context.Context, <-chan error, error) and cancel in Start if needed.
And then, cancel parent context when catching children's context cancellation.
It can terminate process with success completion.

Describe alternatives you've considered:

Now I send SIGTERM to itself. I think it is worth than above.

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

go: github.com/envoyproxy/[email protected]: reading github.com/envoyproxy/protoc-gen-validate/go.mod at revision v0.3.0: unknown revision v0.3.0

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

related to #503, #556

Describe the issue:

currently, vald-agent-ngt pods have these containers:

• initContainers
  • agent-sidecar (initcontainer mode: download S3 backup data to volume)
• containers
  • agent-ngt
  • agent-sidecar (sidecar mode: upload S3 backup data)

agent-sidecar on initContainer mode may fail to complete to download backup data and it returns status code 0 (RST stream from remote host will cause this case). in this case, there may be fragments of backup data in the volume and they cause blocking of NGT startup (#503).
the ideal behavior of the pods on the status like this is retrying to download backup data. however, a failing status of a container doesn't trigger pod restarts.

if there's liveness probe server in the pods, it can trigger pod restarts.
however, agent-NGT has a postStop phase (it is executed after liveness probe killed) to save index. agent-sidecar has a postStop phase to upload index.
so, it is required to improve internal/servers/server to handle these problems.

Vulnerabilities

DepShield reports that this application's usage of github.com/gorilla:websocket:0.0.0-20170926233335-4201258b820c results in the following vulnerability(s):

• (CVSS 7.5) CWE-190: Integer Overflow or Wraparound

Occurrences

github.com/gorilla:websocket:0.0.0-20170926233335-4201258b820c is a transitive dependency introduced by the following direct dependency(s):

• github.com/cockroachdb:errors:0.0.0-00010101000000-000000000000
        └─ github.com/cockroachdb:sentry-go:0.3.999
              └─ github.com/kataras/iris:v12:12.0.1
                    └─ github.com/dgraph-io:badger:1.6.0
                          └─ github.com/spf13:cobra:0.0.5
                                └─ github.com/spf13:viper:1.3.2
                                      └─ github.com/coreos:etcd:3.3.10
                                            └─ github.com/gorilla:websocket:0.0.0-20170926233335-4201258b820c

• sigs.k8s.io:controller-runtime:0.0.0-00010101000000-000000000000
        └─ k8s.io:apiextensions-apiserver:0.18.2
              └─ k8s.io:apiserver:0.18.2
                    └─ go.etcd.io:etcd:0.0.0-20191023171146-3cf2f69b5738
                          └─ github.com/gorilla:websocket:0.0.0-20170926233335-4201258b820c
              └─ go.etcd.io:etcd:0.0.0-20191023171146-3cf2f69b5738
                    └─ github.com/gorilla:websocket:0.0.0-20170926233335-4201258b820c
              └─ github.com/spf13:cobra:0.0.5
                    └─ github.com/spf13:viper:1.3.2
                          └─ github.com/coreos:etcd:3.3.10
                                └─ github.com/gorilla:websocket:0.0.0-20170926233335-4201258b820c

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

High Vald team!

First of all, congrats for the great project!

I work for a major video game company. My team is involved in the development of an in-house NLP tech.

We have been following progress on NGT and Vald for the past few months and we’ve got very strong results in our benchmarks!

I was wondering if you could share any thought on when Vald will get its first official release? I can see you’re actively developing it at the moment, but I could not find any documentation to test it so far.

Also, could you add a few usage examples to the Readme file?

Thank you very much for the great work again!

Vulnerabilities

DepShield reports that this application's usage of golang.org/x:crypto:0.0.0-20181025213731-e84da0312774 results in the following vulnerability(s):

• (CVSS 5.9) [CVE-2019-11840] Use of Insufficiently Random Values

Occurrences

golang.org/x:crypto:0.0.0-20181025213731-e84da0312774 is a transitive dependency introduced by the following direct dependency(s):

• k8s.io:client-go:11.0.1-0.20190409021438-1a26190bd76a
        └─ golang.org/x:crypto:0.0.0-20181025213731-e84da0312774

• sigs.k8s.io:controller-runtime:0.0.0-00010101000000-000000000000
        └─ k8s.io:client-go:11.0.1-0.20190409021438-1a26190bd76a
              └─ golang.org/x:crypto:0.0.0-20181025213731-e84da0312774
        └─ k8s.io:apiextensions-apiserver:0.0.0-20190409022649-727a075fdec8
              └─ k8s.io:client-go:0.0.0-20190918200256-06eb1244587a
                    └─ golang.org/x:crypto:0.0.0-20181025213731-e84da0312774
              └─ k8s.io:apiserver:0.0.0-20190918200908-1e17798da8c1
                    └─ k8s.io:client-go:0.0.0-20190918200256-06eb1244587a
                          └─ golang.org/x:crypto:0.0.0-20181025213731-e84da0312774
                    └─ golang.org/x:crypto:0.0.0-20181025213731-e84da0312774

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

• create tutorial of deploy agent in another page of `get-started, update document spreadsheet
• [feednback] it is enough to write about how to split config.yaml from configmap.yaml and running the command:
  docker run -v path-to-config:/etc/server -it vdaas/vald-agent-ngt

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

go: github.com/envoyproxy/[email protected]: reading github.com/envoyproxy/protoc-gen-validate/go.mod at revision v0.3.0: unknown revision v0.3.0

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Describe the bug:

MetaCassandra GetMetas & GetMetasInverse returns nil when some key not found
It makes implicit meta key overwrite problem from gateway

To Reproduce:

Insert A B C D... data
then
MultiInsert [A B', C', D ...]
A & D will overwrite

Expected behavior:

not overwrite from Multiinsert

Environment:

• Golang Version: 1.13.5
• Docker Version: 19.03.5
• Kubernetes Version: 1.16.3
• NGT Version: 1.8.4

Describe the bug:

application crashes when res.StatusCode is nil.

currently, this line is used only in agent-sidecar.

To Reproduce:

start agent-sidecar with invalid URL for S3 endpoint.

Expected behavior:

don't crash.

Environment:

• Go Version: 1.14.3
• Docker Version: 19.03.8
• Kubernetes Version: 1.18.2
• NGT Version: 1.11.5

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

go: github.com/envoyproxy/[email protected]: reading github.com/envoyproxy/protoc-gen-validate/go.mod at revision v0.3.0: unknown revision v0.3.0

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.