tryvium-travels / memongo Goto Github PK

View Code? Open in Web Editor NEW

61.0 1.0 14.0 128 KB

Easily run an in-memory MongoDB server for your Go unit tests

License: MIT License

Go 99.77% Shell 0.23%

golang mongo unit-testing mongodb mocking test mongo-memory-server tryvium travel tryvium-travels

memongo's People

Contributors

Stargazers

Watchers

Forkers

lessthan3 dgunay recep fernferret josejuanmontiel daveteu jdkallio vinhphuctadang injectivelabs felipeparaujo dendromeda devnev artahc theleeeo

memongo's Issues

Issue with Github action

Hi, i have been trying to use this library with github actions and I am getting the following error:
did not find a mongod binary in the tar from https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2004-6.0.1.tgz

could this be a version issue or something else?

thanks in advance for the help

Invalid Mongo Server URI Port

func TestMain(m *testing.M) {
	opts := &memongo.Options{
		MongoVersion:   "7.0.2",
		MongodBin:      "/opt/homebrew/bin/mongod",
		Logger:         log.Default(),
		LogLevel:       2,
		Port:           27018,
		StartupTimeout: time.Second * 30,
	}
	if runtime.GOARCH == "arm64" {
		if runtime.GOOS == "darwin" {
			opts.DownloadURL = fmt.Sprintf("https://fastdl.mongodb.org/osx/mongodb-macos-x86_64-%s.tgz", MONGODB_VERSION)
		}
	}

	mongoServer, err := memongo.StartWithOptions(opts)
	if err != nil {
		log.Fatal(err)
	}
	defer mongoServer.Stop()

	os.Exit(m.Run())
}

mongoServer.URI always returns mongodb://localhost:0 instead of given port number

Edit:
I'm using the latest commit b8467444e40c09e725bd81372d236a5a948694f2 because of removal ephemeralForTest on mongodb version which I'm using

Could you please add support for centos?

Currently, RHEL 7.0 works but RHEL8.0 and centos are not supported. It would be great to add support for them.

Issue running it Github Actions using ubuntu-latest

Hi ya'll!

We are receiving this error from Github Actions when using the new version v0.8.0:

[memongo] [INFO]  Starting MongoDB with options &memongo.Options{ShouldUseReplica:false, Port:32841, CachePath:"/home/runner/.cache/memongo", MongoVersion:"4.2.1", DownloadURL:"https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-4.2.1.tgz", MongodBin:"", Logger:(*log.Logger)(nil), LogLevel:0, StartupTimeout:10000000000, Auth:false}
[memongo] [INFO]  mongod from https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-4.2.1.tgz does not exist in cache, downloading to /home/runner/.cache/memongo/mongodb-linux-x86_64-ubuntu2204-4_2_1_tgz_9ce45a6aea/mongod
--- FAIL: TestCreateMongoServer (0.23s)
    --- FAIL: TestCreateMongoServer/valid_server (0.23s)
        embedded_test.go:31: 
            	Error Trace:	/home/runner/work/bux/bux/tester/embedded_test.go:31
            	Error:      	Received unexpected error:
            	            	error intializing gzip reader from /tmp/430061428: gzip: invalid header
            	Test:       	TestCreateMongoServer/valid_server
FAIL

We do not get the same error when using v0.6.1 however.

Here is the section in our Github actions file:

  test:
    strategy:
      matrix:
        go-version: [1.18.x]
        os: [ubuntu-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - name: Install Go ${{ matrix.go-version }}
        uses: actions/setup-go@v3
        with:
          go-version: ${{ matrix.go-version }}
      - name: Checkout code
        uses: actions/checkout@v3 
   .......
   .......

You can see we are using the latest version of Ubuntu, but we are still getting an error with loading memongo.

We also referenced: #9 but again, using ubuntu-latest is not working.

Also, the error message has a spelling error: error intializing gzip vs error initializing gzip

Can't start memmongo with replicaset - panic with wrong error returned

When trying to start mongo with a replica set, we are getting a panic instead of an error due to:

memongo/memongo.go

Lines 176 to 181 in a32a48f

    
           err2 := replicaSetCommand.Run() 
        
           if err2 != nil { 
        
           	logger.Warnf("Error initiating replica: %v", err2) 
        
           	return nil, err 
        
           }

Where err is erroneously returned instead of err2.

But the actual problem lies a few lines above, where the existence of mongo cli tools is assumed to exist on the $PATH:

memongo/memongo.go

Lines 167 to 171 in a32a48f

    
           if opts.ShouldUseReplica { 
        
           	//nolint:gosec 
        
           	mongoCommand := fmt.Sprintf("mongo --port %d --retryWrites --eval \"rs.initiate()\"", opts.Port) 
        
           	//nolint:gosec 
        
           	replicaSetCommand := exec.Command("bash", "-c", mongoCommand)

Dependency on 'libssl.so.1.1' fails when running on latest Golang docker image

Error

Running on docker image golang:1.20.5 fails with the following debug message.

[memongo] [INFO]  Starting MongoDB with options &memongo.Options{ShouldUseReplica:false, Port:33511, CachePath:"/root/.cache/memongo", MongoVersion:"5.0.8", DownloadURL:"https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-debian11-5.0.8.tgz", MongodBin:"", Logger:(*log.Logger)(nil), LogLevel:2, StartupTimeout:60000000000, Auth:false}

[memongo] [INFO]  mongod from https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-debian11-5.0.8.tgz does not exist in cache, downloading to /root/.cache/memongo/mongodb-linux-x86_64-debian11-5_0_8_tgz_88eac89ebf/mongod

[memongo] [INFO]  finished downloading mongod to /root/.cache/memongo/mongodb-linux-x86_64-debian11-5_0_8_tgz_88eac89ebf/mongod in 2.845388665s

[memongo] [DEBUG] Using binary /root/.cache/memongo/mongodb-linux-x86_64-debian11-5_0_8_tgz_88eac89ebf/mongod

[memongo] [DEBUG] Starting mongod

[memongo] [DEBUG] Started mongod; starting watcher

[memongo] [DEBUG] Started watcher; waiting for mongod to report port number

[memongo] [DEBUG] [Mongod stderr] /root/.cache/memongo/mongodb-linux-x86_64-debian11-5_0_8_tgz_88eac89ebf/mongod: error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file: No such file or directory

panic: timed out waiting for mongod to start

Debugging

The "old" golang:1.20.4 have the correct libssl version.

>> docker run -it golang:1.20.4 find /usr/lib  -name 'libssl.so*'
/usr/lib/aarch64-linux-gnu/libssl.so.1.1

The newer golang:1.20.5 have changed the libssl version from 1.1 to 3.

docker run -it golang:1.20.5 find /usr/lib  -name 'libssl.so*'
/usr/lib/aarch64-linux-gnu/libssl.so.3

afero.Rename errors when copying mongod binary from /tmp mounted on different filesystem

My system has /tmp mounted on a different filesystem, so when afs.Rename is called to copy the mongod binary from /tmp to the cache, I get the following error:

panic: error writing mongod binary from /tmp/192458888 to /home/devin/.cache/memongo/mongodb-linux-x86_64-debian10-4_2_16_tgz_ffac481192/mongod: rename /tmp/192458888 /home/devin/.cache/memongo/mongodb-linux-x86_64-debian10-4_2_16_tgz_ffac481192/mongod: invalid cross-device link

This is because it is likely calling os.Rename, which cannot move files across filesystems. This can be worked around by copying the file and then deleting the original if the copy was successful.

Checksum failed for v0.8.0

Hi all contributors,
when I go mod tidy with github.com/tryvium-travels/memongo and github.com/tryvium-travels/memongo/memongolog, I faced this issue:

github.com/tryvium-travels/memongo/memongolog: github.com/tryvium-travels/[email protected]: verifying module: checksum mismatch
        downloaded: h1:+tXjXsGTz8uRpHNkO5OhgZYolaVHh3QJDoBweOTSnIk=
        sum.golang.org: h1:+2h0VpspWyRT6len83UcohbCcKarUBvUgXZj7ezhW2o=

, which means checksum for the package is modified, is it expected? Can you guys help check on the go sum server to update checksum?

The workaround method is to turn off the go sum db (go env -w GOSUMDB=off), but it's not safe, is it?

Does not work with CircleCI/Github actions?

When running on github actions I'm getting the following errors. Is this a limitation of memongo?

[memongo] [INFO]  Starting MongoDB with options &memongo.Options{ShouldUseReplica:false, Port:46361, CachePath:"/home/runner/.cache/memongo", MongoVersion:"4.0.5", DownloadURL:"https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2004-4.0.5.tgz", MongodBin:"", Logger:(*log.Logger)(nil), LogLevel:0, StartupTimeout:10000000000, Auth:false}
[83](https://github.com/myproject/server-go/runs/8247274161?check_suite_focus=true#step:5:84)
[memongo] [INFO]  mongod from https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2004-4.0.5.tgz does not exist in cache, downloading to /home/runner/.cache/memongo/mongodb-linux-x86_64-ubuntu2004-4_0_5_tgz_29b99839f9/mongod
[84](https://github.com/myproject/server-go/runs/8247274161?check_suite_focus=true#step:5:85)
{"level":"fatal","error":"error intializing gzip reader from /tmp/987266152: gzip: invalid header","time":"2022-09-08T10:39:18Z","message":"error intializing gzip reader from /tmp/987266152: gzip: invalid header"}

memongo doesn't properly set permissions for mongod

We're using memongo in Go tests. It appears the permissions on the binary in the Mongo tar.gz file are set to 644 (or -rw-r--r--) and it fails to execute. I'm not sure if the reason this isn't handled properly is that MongoDB used to distribute the binary with the permissions set to 744 in the past, or if this section of the _saveFile function isn't working correctly to run chmod on the file. But the result is that when memongo tries to start it in the cache directory it isn't executable.

I'm running the following command to perform go tests on a Fedora 39 system (RHEL compatible):

MEMONGO_DOWNLOAD_URL=https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel90-7.0.5.tgz go test -v ./...

The error I get with debug output is:

=== RUN   TestSeedCmdTestSuite
[memongo] [INFO]  Starting MongoDB with options &memongo.Options{ShouldUseReplica:false, Port:37917, CachePath:"/home/testuser/.cache/memongo", MongoVersion:"7.0.5", DownloadURL:"https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel90-7.0.5.tgz", MongodBin:"", Logger:(*log.Logger)(nil), LogLevel:2, StartupTimeout:10000000000, Auth:false}
[memongo] [DEBUG] mongod from https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel90-7.0.5.tgz exists in cache at /home/testuser/.cache/memongo/mongodb-linux-x86_64-rhel90-7_0_5_tgz_57613dff3a/mongod
[memongo] [DEBUG] Using binary /home/testuser/.cache/memongo/mongodb-linux-x86_64-rhel90-7_0_5_tgz_57613dff3a/mongod
[memongo] [DEBUG] Starting mongod
    seed_test.go:30: 
        	Error Trace:	/home/testuser/blah/cmd/seed_test.go:30
        	            	/home/testuser/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:154
        	            	/home/testuser/blah/cmd/seed_test.go:24
        	Error:      	Received unexpected error:
        	            	fork/exec /home/testuser/.cache/memongo/mongodb-linux-x86_64-rhel90-7_0_5_tgz_57613dff3a/mongod: permission denied

And the call to start the memongo server is:

mongoServer, err := memongo.StartWithOptions(&memongo.Options{MongoVersion: "7.0.5", LogLevel: memongolog.LogLevel(memongolog.LogLevelDebug)})

As a workaround I can manually run chmod 744 on the cached file after the initial failure. But this seems like it should be handled by memongo to properly set the permissions to executable.

Let me know if there's any more information I can provide.

Thanks,
-Matt

Problem with defaults are in diferent partitions

If i have /home and /tmp in different partitions

[memongo] [INFO] Starting MongoDB with options &memongo.Options{ShouldUseReplica:false, Port:33655, CachePath:"/home/jose/.cache/memongo", MongoVersion:"4.0.5", DownloadURL:"https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-debian92-4.0.5.tgz", MongodBin:"", Logger:(*log.Logger)(nil), LogLevel:0, StartupTimeout:10000000000, Auth:false}
[memongo] [INFO] mongod from https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-debian92-4.0.5.tgz does not exist in cache, downloading to /home/jose/.cache/memongo/mongodb-linux-x86_64-debian92-4_0_5_tgz_013dacf59f/mongod
2022/05/25 23:16:57 error writing mongod binary from /tmp/795252025 to /home/jose/.cache/memongo/mongodb-linux-x86_64-debian92-4_0_5_tgz_013dacf59f/mongod: rename /tmp/795252025 /home/jose/.cache/memongo/mongodb-linux-x86_64-debian92-4_0_5_tgz_013dacf59f/mongod: invalid cross-device link
Process 9141 has exited with status 1

Seem something like this works

gzip: invalid header

func startTestDbServer() string {
	// Start a MongoDB server and get a connection string for it
	opts := &memongo.Options{
		MongoVersion: "6.0.1",
	}
	if runtime.GOARCH == "arm64" {
		if runtime.GOOS == "darwin" {
			// Only set the custom url as workaround for arm64 macs
			opts.DownloadURL = "https://fastdl.mongodb.org/osx/mongodb-macos-x86_64-6.0.1.tgz"
		}
	}

	mongoServer, err := memongo.StartWithOptions(opts)
	if err != nil {
		log.Fatal(err)
	}

	// Connect to the in-memory server with a MongoDB client
	return mongoServer.URI()
}

Test Logs:

[memongo] [INFO]  Starting MongoDB with options &memongo.Options{ShouldUseReplica:false, Port:33755, CachePath:"/home/asimzahid/.cache/memongo", MongoVersion:"6.0.1", DownloadURL:"https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-6.0.1.tgz", MongodBin:"", Logger:(*log.Logger)(nil), LogLevel:0, StartupTimeout:10000000000, Auth:false}
[memongo] [INFO]  mongod from https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-6.0.1.tgz does not exist in cache, downloading to /home/asimzahid/.cache/memongo/mongodb-linux-x86_64-ubuntu2204-6_0_1_tgz_68b6ad5aea/mongod
2023/10/05 13:43:48 error intializing gzip reader from /tmp/110027803: gzip: invalid header, https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-6.0.1.tgz

Unable to resolve this issue.
Could anyone please help with understand this and resolve it?

Debian11 download spec is incorrect

Hi! We are getting issues in out CI/CD pipelines because of what looks like a copy-pasted line on

memongo/mongobin/downloadSpec.go

Line 295 in de2d8c1

if majorVersion >= 11 && versionGTE(mongoVersion, []int{4, 2, 1}) {

The oldest mongo version that has a download link is 5.0.8, not 4.2.1 as in the code.

Can we get a new release?

There have been a couple of PRs merged since the last release, including one that adds support for arm (apple silicon) processors. Can we get a new tag for it?

Mongo > v5.0.0 breaks the build and testing process

The new mongo client (from v5.0.0) is being split into multiple files and it does not have much sense to download the whole tarball when just the correct executable (server) has to be downloaded

issue running inside of docker

I am trying to use it inside of golang on docker, any idea how to achieve that?

Config:

	opts := &memongo.Options{
		MongoVersion: "5.0.0",
	}

	if runtime.GOARCH == "arm64" && runtime.GOOS == "darwin" {
		// Only set the custom url as workaround for arm64 macs
		opts.DownloadURL = "https://fastdl.mongodb.org/osx/mongodb-macos-x86_64-5.0.0.tgz"
	}

	mongoServer, err := memongo.StartWithOptions(opts)
	assert.NoError(t, err)

Dockerfile:

FROM golang:latest as builder

WORKDIR /app

COPY go.mod go.sum ./

RUN go mod download && go mod verify

COPY . .

ENV DOCKER_RUNNING=true

RUN CGO_ENABLED=0 GOARCH=amd64 GOOS=linux go test -failfast ./it

Getting the following error:

> [builder 8/9] RUN CGO_ENABLED=0 GOARCH=amd64 GOOS=linux go test -failfast ./it:
#16 28.82 GOARCH:  amd64
#16 28.82 GOOS:  linux
#16 28.82 [memongo] [INFO]  Starting MongoDB with options &memongo.Options{ShouldUseReplica:false, Port:41831, CachePath:"/root/.cache/memongo", MongoVersion:"5.0.0", DownloadURL:"https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-debian10-5.0.0.tgz", MongodBin:"", Logger:(*log.Logger)(nil), LogLevel:0, StartupTimeout:10000000000, Auth:false}
#16 28.82 [memongo] [INFO]  mongod from https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-debian10-5.0.0.tgz does not exist in cache, downloading to /root/.cache/memongo/mongodb-linux-x86_64-debian10-5_0_0_tgz_fe6e20c131/mongod
#16 28.82 [memongo] [INFO]  finished downloading mongod to /root/.cache/memongo/mongodb-linux-x86_64-debian10-5_0_0_tgz_fe6e20c131/mongod in 11.270371588s
#16 28.82 --- FAIL: TestE2ETestSuite (21.39s)
#16 28.82     testing.go:77: 
#16 28.82               Error Trace:    testing.go:77
#16 28.82                                                       e2e_test.go:66
#16 28.82                                                       suite.go:118
#16 28.82                                                       e2e_test.go:44
#16 28.82               Error:          Received unexpected error:
#16 28.82                               timed out waiting for mongod to start

nullpointer panic when running go test ./...

Hi,

first of all, thanks for keeping this project alive, it helps a lot. I have some issues when running tests in a CI pipeline (or just with go test ./... on my machine - sometimes (but not every time) the tests fail with a nil pointer exception in memongo.go:192:

panic: runtime error: invalid memory address or nil pointer dereference
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8c7ee6]

goroutine 1 [running]:
github.com/tryvium-travels/memongo.(*Server).Stop(0x0)
	//home/user/go/pkg/mod/github.com/tryvium-travels/[email protected]/memongo.go:192 +0x46
panic({0xc831c0, 0x144b6f0})
	/usr/lib/go/src/runtime/panic.go:1047 +0x266
github.com/tryvium-travels/memongo.(*Server).URI(...)
	//home/user/go/pkg/mod/github.com/tryvium-travels/[email protected]/memongo.go:181
myproject/pkg/store/mongo_test.TestMain(0x0)
	/home/user/myproject/src/pkg/store/mongo/mongo_test.go:39 +0x1ee
main.main()
	_testmain.go:101 +0x365

I tried downgrading to v0.3.2 yesterday, which worked - yesterday. Today I have the same error so downgrading does nothing I suppose. I cannot tackle this because as I said, sometimes it works.

System is (locally) an up-to-date arch linux box with mongod v4.4.12 installed, the tests are run with a func TestMain(m *testing.M), where the server gets started.

It all seems to work fine when running the tests directly (i.e. from the VSCode window).

I appreciate any hints on this. If you need more info, please say so. Thanks!

	err2 := replicaSetCommand.Run()
	if err2 != nil {
	logger.Warnf("Error initiating replica: %v", err2)

	return nil, err
	}

	if opts.ShouldUseReplica {
	//nolint:gosec
	mongoCommand := fmt.Sprintf("mongo --port %d --retryWrites --eval \"rs.initiate()\"", opts.Port)
	//nolint:gosec
	replicaSetCommand := exec.Command("bash", "-c", mongoCommand)