bacalhau-project / bacalhau

Compute over Data framework for public, transparent, and optionally verifiable computation

The Distributed Computation Framework ⚡
Compute Over Data (CoD)

Bacalhau is a platform for fast, cost efficient, and secure computation by running jobs where the data is generated and stored. With Bacalhau you can streamline your existing workflows without the need of extensive rewriting by running arbitrary Docker containers and WebAssembly (wasm) images as tasks.

Table of Contents

• Table of Contents
• Why Bacalhau?
• Getting started - Bacalhau in 1 minute
  • Learn more
• Documentation
• Developers guide
  • Running Bacalhau locally
  • Notes for Dev contributors
  • OpenAPI
  • Python Libraries
• Issues, feature requests, and questions
• Ways to contribute
• Open Source

Why Bacalhau?

• ⚡ Fast job processing: Jobs in Bacalhau are processed where the data was created and all jobs are parallel by default.
• 💰 Low cost: Reduce (or eliminate) ingress/egress costs since jobs are processed closer to the source. Take advantage of as well idle computation capabilities at the edge.
• 🔒 Secure: Data scrubbing and security can before migration to reduce the chance of leaking private information, and with a far more granular, code-based permission model.
• 🚛 Large-scale data: Bacalhau operates on a network of open compute resources made available to serve any data processing workload. With Bacalhau, you can batch process petabytes (quadrillion bytes) of data.

Getting started - Bacalhau in 1 minute

Go to the folder directory that you want to store your job results

Install the bacalhau client

curl -sL https://get.bacalhau.org/install.sh | bash

Submit a "Hello World" job

bacalhau docker run ubuntu echo Hello World

Download your result

bacalhau get 63d08ff0..... # make sure to use the right job id from the docker run command

For a more detailed tutorial, check out our Getting Started tutorial.

Learn more

• Understand Bacalhau Concepts
• Get an overview of the different usecases that you can use with Bacalhau.
• To see Bacalhau in action, check out the Bacalhau Examples
• You can check out this featured example video tutorial Text to image- Stable Diffusion GPU. You can watch more tutorials here

Documentation

📚 Read the Bacalhau docs guide here! 📚

The Bacalhau docs is the best starting point as it contains all the information to ensure that everyone who uses Bacalhau is doing so efficiently.

Developers guide

Running Bacalhau locally

Developers can spin up bacalhau and run a local demo using the devstack command.

Please see running_locally.md for instructions. Also, see debugging_locally.md for some useful tricks for debugging.

Notes for Dev contributors

Bacalhau's CI pipeline performs a variety of linting and formatting checks on new pull requests. To have these checks run locally when you make a new commit, you can use the precommit hook in ./githooks:

make install-pre-commit

# check if pre-commit works
make precommit

If you want to run the linter manually:

curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sudo sh -s -- -b /usr/local/go/bin
golangci-lint --version
make lint

The config lives in .golangci.yml

OpenAPI

OpenAPI v2 annotations sit by the endpoints in pkg/publicapi; these are built using swag, a Go converter for Swagger documentation. Find more details about the Swag annotations in their docs. The swagger specification is built automatically by the CI pipeline (see the build_swagger workflow) but you can trigger a local build with make swagger-docs.

The build parses the OpenAPI annotations as well as the markdown files in docs/swagger/ (containing long-form descriptions of the API endpoints), and generates the following swagger specification files:

• docs/docs.go
• docs/swagger.json
• docs/swagger.yaml

Note that the CI pipeline will open and automatically merge a pull request titled [circleci] Build swagger reference - this is an automatic commit ... containing only the updated spec files.

Python Libraries

We ship two Python Bacalhau libraries:

• bacalhau-apiclient wraps only the API endpoint calls and request/response models. It's autogenerated from the OpenAPI specification (see paragraph above). Read more about it in its readme.
• bacalhau-sdk is a high-level Bacalhau SDK that ships all the client-side logic (e.g. signing requests) needed to query the endpoints. Its examples folder contains code snippets to create, list and inspect jobs. Under the hood, it uses the bacalhau-apiclient to call the API. Please use this library in your projects. Read more about it in its readme.

Issues, feature requests, and questions

We are excited to hear your feedback!

• For issues and feature requests, please open a GitHub issue.
• For questions, give feedback or answer questions that will help other user product please use GitHub Discussions.
• To engage with other members in the community, join us in our slack community #bacalhau channel 🙋

Ways to contribute

All manner of contributions are more than welcome!

We have highlighted the different ways you can contribute in our contributing guide. You can be part of community discussions, development, and more.

Open Source

This repository contains the Bacalhau software, covered under the Apache-2.0, except where noted (any Bacalhau logos or trademarks are not covered under the Apache License, and should be explicitly noted by a LICENSE file.)

Bacalhau is a product produced from this open source software, exclusively by Expanso, Inc. It is distributed under our commercial terms.

Others are allowed to make their own distribution of the software, but they cannot use any of the Bacalhau trademarks, cloud services, etc.

We explicitly grant permission for you to make a build that includes our trademarks while developing Bacalhau software itself. You may not publish or share the build, and you may not use that build to run Bacalhau software for any other purpose.

We have borrowed the above Open Source clause from the excellent System Initiative

bacalhau's Issues

When running something that outputs something non-stdout, nothing gets fetched

https://github.com/filecoin-project/bacalhau/tree/main/examples

export file_path=./high_quality_picture.jpg
cid=$(IPFS_PATH=/tmp/bacalhau-ipfs3163549147 ipfs add -q $file_path)
sudo apt-get -y update && sudo apt-get install -y graphicsmagick && \
    gm convert high_quality_picture.jpg -quality 20% -colorspace Gray gray_scale.jpg

Every node should be able to coordinate with every other node (no jrpcports or jrpcservers necessary)

21. Encryption between clients and compute nodes to avoid snooping [was Placeholder milestone 2]

• FINAL DELIVERABLE: xxx (@lukemarsden to fill in)

11. Client is notified/can request results

• Reading them from where the compute nodes wrote them to
• Deliverable: client can act upon the above and get the results

19. Tests and CI setup

As we move towards moving the project towards production, have tests that run regularly.

• Green CI with integration tests
• Docker backend so that Bacalhau can run on GitHub Actions

Investigating approaches for nondeterministic jobs such as ML training

• FINAL DELIVERABLE: xxx (@lukemarsden to fill in)

12. Results aggregation

• Client considers results from everywhere
• More than one miner has run the job, present summary of multiple results
• Jobs are aggregated and evaluated
• Confidence score system in the client: filtering aggregated results step like kubectl –wait
• Deliverable: two compute nodes do the same job and the client is able to display status on both

REALLY need a way to sanity check a bash command before executing

Lots of times there's a syntax error before submitting. We need to check the bash script before submission.

ignore this

5. Filecoin integration design

• Deliverable: a design for how Bacalhau could integrate with Filecoin

3. Compute node selects from network

• A list of pending transactions (more like a transaction than saving a file)
• Deliverable: popping arriving messages into mempool per compute node

7. Job execution investigation: wasm

• Build a prototype of wasm execution backend
• Deliverable: a job can start a wasm process and execute some arbitrary python code provided by the user (for example)

17. Research possibility of partial verification to reduce energy usage and cost

• Sharding 100 shards of data I only need 3% of them to be verified
• And how miners would be penalised for lying
• This might relate to proof of space-time
• Deliverable: meetings with Protocol Labs experts in proof of spacetime, written conclusions on how we can approach confidence when not every job has been run multiple times

Creating/running ipfs requires larger UDP buffers

https://medium.com/@CameronSparr/increase-os-udp-buffers-to-improve-performance-51d167bb1360

We should detect this and fail.

Support running on an M1 mac

At least with docker mode, build an arm64 image.

8. Data can be consumed by the scheduled job

• Presenting the required data to the process execution environment
• Investigate ipfs fuse mounts in order to efficiently mount a dataset inside a firecracker vm running the docker container with the job in it (avoiding any copying)
• Deliverable: one of the two job execution environments can mount data into the user code (e.g. sed can run on data in ipfs!)

Executing "docker run" does not work

attempting to execute the following:

go run . --jsonrpc-port=39359 submit --commands "docker run hello-world"

Does not work with the following:

we are updating a job!:
&{JobId:d5d620a9-aed5-47c2-a807-c12dcaebe517 NodeId:QmeSWYJnbTYks3xqBnE9x7TTekmTnYKEX87bBdwcjXMq4c State:error Status:Error running the job: exit status 1
 Output:}

I'm sure it's some permissions thing - but we should make sure to have a clear(er) error output.

Timeouts when running tests

Not sure what's going on, but when I checkout master on the DO node, and do make test it runs forever and then times out.

That said, it runs fine in Github Actions.

Anything I can do to debug?

not respecting correct user?

Ran submit job:

go run . submit --cids=$cid --commands="grep -o 'W' /ipfs/$cid | wc -l" --jsonrpc-port=41923

Everything ran normally up to this point:

was selected because jobs CID are local:
 [Qmdd3GdLhQJENPLs6VyGByaZWytDCcjJezGUcWonEybrzu]
 Output:}
INFO[0001] Created VM with ID "8af9ab483799ac66" and name "d978308c-95d2-4193-99b6-a1c1f9fbd189a9fa74f4-e62f-46b8-87ae-28676ba25fd4"
INFO[0001] Networking is handled by "cni"
INFO[0001] Started Firecracker VM "8af9ab483799ac66" in a container with ID "ignite-8af9ab483799ac66"
INFO[0001] Waiting for the ssh daemon within the VM to start...
generating ED25519 keypair...done
peer identity: 12D3KooWLaE9mxuP57Pn4wC3JrKTnpmeqKWQJmYwB7RwjSbiYaqK
initializing IPFS node at /root/.ipfs
to get started, enter:

        ipfs cat /ipfs/QmQPeNsJPyVWPFDVHb77w8G42Fvo15z4bG2X8D2GhfbSXc/readme

2022-03-01T17:22:42.502Z        ERROR   provider.queue  queue/queue.go:124      Failed to enqueue cid: leveldb: closed
removed /dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN
removed /dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa
removed /dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb
removed /dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt
removed /ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
removed /ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
added /ip4/127.0.0.1/tcp/36815/p2p/12D3KooWPszx6ZjoexB2nMnWWxPCtxbnSv6S4tck61jPCDLGid5x

It APPEARS that ignite is taking over and re-creating a new ipfs stack?

18. Investigating approaches for nondeterministic jobs such as ML training

• Deliverable: written report on findings

16. Do research into payment, incentives and rewards, meet and discuss with experts, and make some recommendations for future work

• Deliverable: meetings with Protocol Labs experts in proof of spacetime, written conclusions on how we can adapt this or other algorithms to provide a payment layer for Bacalhau

Figure out a way to bulk all logging together by server (during debug)

Unfortunately right now, logging interleaves various servers/nodes together in output.

Eg

Starting: /home/user/go/bin/dlv dap --check-go-version=false --listen=127.0.0.1:43559 --log-dest=3 from /home/user/code/bacalhau/cmd/bacalhau
DAP server listening at: 127.0.0.1:43559
2022-03-12T20:53:44Z error layer=debugger can't find build-id note on binary
Type 'dlv help' for list of commands.
20h53m45.48	DEBUG	runtime/proc.go:6498	Log level from LOG_LEVEL_ENV_VAR: debug
Zap log level: debug
=== RUN   TestCommands
20h53m45.76	DEBUG	system/utils.go:16	IPFS Command: sudo [pkill ipfs]
20h53m46.78	DEBUG	internal/dev_stack.go:37	[---------------------]
20h53m46.78	INFO	internal/dev_stack.go:38		Creating Node #0
20h53m46.78	DEBUG	internal/dev_stack.go:39	[---------------------]
20h53m47.35	DEBUG	libp2p/libp2p.go:108	[Waiting for bacalhau libp2p context to finish.
]
20h53m47.36	DEBUG	internal/dev_stack.go:68	bacalhau multiaddress: /ip4/172.31.35.25/tcp/33843/p2p/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T
20h53m47.36	DEBUG	system/utils.go:16	IPFS Command: mkdir [-p /home/user/.bacalhau//tmp/bacalhau-ipfs718364386]
20h53m47.36	DEBUG	internal/jsonrpc_server.go:67	[Waiting for json rpc context to finish.]
20h53m47.37	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs init
20h53m47.37	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [init]
20h53m47.97	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs bootstrap rm --all
20h53m47.97	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [bootstrap rm --all]
20h53m48.20	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/37257
20h53m48.20	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [config Addresses.Gateway /ip4/0.0.0.0/tcp/37257]
20h53m48.44	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs config Addresses.API /ip4/0.0.0.0/tcp/44999
20h53m48.44	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [config Addresses.API /ip4/0.0.0.0/tcp/44999]
20h53m48.75	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs config Addresses.Swarm --json ["/ip4/0.0.0.0/tcp/40349"]

20h53m48.75	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [config Addresses.Swarm --json ["/ip4/0.0.0.0/tcp/40349"]]
20h53m49.07	DEBUG	ipfs/utils.go:80	Starting IPFS Daemon: IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs daemon
20h53m49.07	DEBUG	ipfs/utils.go:100	[waiting for ipfs context done
]
20h53m50.07	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs id
20h53m50.07	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [id]
20h53m50.32	DEBUG	internal/dev_stack.go:104	bacalhau multiaddress: /ip4/172.31.35.25/tcp/33843/p2p/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T

20h53m50.32	DEBUG	internal/dev_stack.go:105	ipfs multiaddress: /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2

20h53m50.32	DEBUG	internal/dev_stack.go:106	ipfs repo: /tmp/bacalhau-ipfs718364386

20h53m50.32	DEBUG	internal/dev_stack.go:116	[==== Complete]
20h53m50.32	DEBUG	internal/dev_stack.go:37	[---------------------]
20h53m50.32	INFO	internal/dev_stack.go:38		Creating Node #1
20h53m50.32	DEBUG	internal/dev_stack.go:39	[---------------------]
20h53m50.95	DEBUG	internal/dev_stack.go:68	bacalhau multiaddress: /ip4/172.31.35.25/tcp/43707/p2p/QmVwQH415Bi4XGtVkLyaeTJ8LWMEXBoSYChL7r9vz8YmSF
20h53m50.95	DEBUG	libp2p/libp2p.go:108	[Waiting for bacalhau libp2p context to finish.
]
20h53m50.99	DEBUG	internal/jsonrpc_server.go:67	[Waiting for json rpc context to finish.]
20h53m50.99	DEBUG	system/utils.go:16	IPFS Command: mkdir [-p /home/user/.bacalhau//tmp/bacalhau-ipfs150944631]
20h53m50.99	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs init
20h53m50.99	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [init]
20h53m51.70	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs bootstrap rm --all
20h53m51.70	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [bootstrap rm --all]
20h53m51.85	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs bootstrap add /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2
20h53m51.85	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [bootstrap add /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2]
20h53m52.16	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/39687
20h53m52.16	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [config Addresses.Gateway /ip4/0.0.0.0/tcp/39687]
20h53m52.29	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs config Addresses.API /ip4/0.0.0.0/tcp/45093
20h53m52.29	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [config Addresses.API /ip4/0.0.0.0/tcp/45093]
20h53m52.42	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs config Addresses.Swarm --json ["/ip4/0.0.0.0/tcp/38221"]

20h53m52.42	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [config Addresses.Swarm --json ["/ip4/0.0.0.0/tcp/38221"]]
20h53m52.73	DEBUG	ipfs/utils.go:80	Starting IPFS Daemon: IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs daemon
20h53m52.73	DEBUG	ipfs/utils.go:100	[waiting for ipfs context done
]
20h53m53.73	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs id
20h53m53.73	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [id]
20h53m53.84	DEBUG	internal/dev_stack.go:104	bacalhau multiaddress: /ip4/172.31.35.25/tcp/43707/p2p/QmVwQH415Bi4XGtVkLyaeTJ8LWMEXBoSYChL7r9vz8YmSF

20h53m53.84	DEBUG	internal/dev_stack.go:105	ipfs multiaddress: /ip4/127.0.0.1/tcp/38221/p2p/12D3KooWPGdapcvdoeVd6MGvf3aWWKGHKUUgctQJ952QQ9fzVo2A

20h53m53.84	DEBUG	internal/dev_stack.go:106	ipfs repo: /tmp/bacalhau-ipfs150944631

20h53m53.84	DEBUG	internal/dev_stack.go:116	[==== Complete]
20h53m53.84	DEBUG	internal/dev_stack.go:37	[---------------------]
20h53m53.84	INFO	internal/dev_stack.go:38		Creating Node #2
20h53m53.84	DEBUG	internal/dev_stack.go:39	[---------------------]
20h53m54.62	DEBUG	libp2p/libp2p.go:108	[Waiting for bacalhau libp2p context to finish.
]
20h53m54.63	DEBUG	internal/dev_stack.go:68	bacalhau multiaddress: /ip4/172.31.35.25/tcp/45821/p2p/QmRqsfviqXhQ6Kj6zoHEdQ4BSA9PvDEgxnkSHtFmKCVXNK
20h53m54.64	DEBUG	system/utils.go:16	IPFS Command: mkdir [-p /home/user/.bacalhau//tmp/bacalhau-ipfs4068152898]
20h53m54.64	DEBUG	internal/jsonrpc_server.go:67	[Waiting for json rpc context to finish.]
20h53m54.65	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs init
20h53m54.65	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [init]
20h53m54.98	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs bootstrap rm --all
20h53m54.98	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [bootstrap rm --all]
20h53m55.12	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs bootstrap add /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2
20h53m55.12	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [bootstrap add /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2]
20h53m55.27	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/45503
20h53m55.27	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [config Addresses.Gateway /ip4/0.0.0.0/tcp/45503]
20h53m55.39	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs config Addresses.API /ip4/0.0.0.0/tcp/45955

20h53m55.39	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [config Addresses.API /ip4/0.0.0.0/tcp/45955]
20h53m55.50	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs config Addresses.Swarm --json ["/ip4/0.0.0.0/tcp/43017"]
20h53m55.50	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [config Addresses.Swarm --json ["/ip4/0.0.0.0/tcp/43017"]]
20h53m55.63	DEBUG	ipfs/utils.go:80	Starting IPFS Daemon: IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs daemon
20h53m55.63	DEBUG	ipfs/utils.go:100	[waiting for ipfs context done
]
20h53m56.63	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs id
20h53m56.63	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [id]
20h53m56.76	DEBUG	internal/dev_stack.go:104	bacalhau multiaddress: /ip4/172.31.35.25/tcp/45821/p2p/QmRqsfviqXhQ6Kj6zoHEdQ4BSA9PvDEgxnkSHtFmKCVXNK

20h53m56.77	DEBUG	internal/dev_stack.go:105	ipfs multiaddress: /ip4/127.0.0.1/tcp/43017/p2p/12D3KooWMQQjoKrjwx8Ar5dGuXPsVhgVcekmGBpbdgpfvKbpbUsy

20h53m56.77	DEBUG	internal/dev_stack.go:106	ipfs repo: /tmp/bacalhau-ipfs4068152898
20h53m56.77	DEBUG	internal/dev_stack.go:116	[==== Complete]
20h53m56.77	DEBUG	internal/dev_stack.go:123	[Finished provisioning nodes.]
=== RUN   TestCommands/grep
20h53m58.77	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs add -Q ../../testdata/grep_file.txt

20h53m58.77	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [add -Q ../../testdata/grep_file.txt]
20h53m58.88	DEBUG	bacalhau/devstack_test.go:261	About to submit job:
cmd: grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX
20h53m58.89	DEBUG	libp2p/libp2p.go:329	Sending event: {"JobId":"e6fa8515-2907-4f2e-8cdf-56f0aab6d2a2","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"job_created","JobSpec":{"Engine":"","Commands":["grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX"],"Image":"","Cpu":0,"Memory":0,"Disk":0,"Inputs":[{"Engine":"ipfs","Cid":"QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX"}]},"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":null},"JobState":null}
20h53m58.89	DEBUG	internal/compute_node.go:54	Found new job to schedule: 
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}

20h53m58.89	DEBUG	internal/compute_node.go:112	Selecting for job with matching CID(s): [{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]

20h53m58.89	DEBUG	ipfs/utils.go:197	[Beginning to collect all refs in IPFS Repo.]
20h53m58.89	DEBUG	ipfs/utils.go:198	RepoPath {/tmp/bacalhau-ipfs718364386}
20h53m58.89	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs refs local

20h53m58.89	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [refs local]
20h53m58.89	INFO	bacalhau/submit.go:130	Submitted Job Id: e6fa8515-2907-4f2e-8cdf-56f0aab6d2a2

20h53m58.89	DEBUG	internal/compute_node.go:54	Found new job to schedule: 
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}

20h53m58.89	DEBUG	internal/compute_node.go:112	Selecting for job with matching CID(s): [{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]

20h53m58.89	DEBUG	ipfs/utils.go:197	[Beginning to collect all refs in IPFS Repo.]
20h53m58.89	DEBUG	ipfs/utils.go:198	RepoPath {/tmp/bacalhau-ipfs150944631}
20h53m58.89	DEBUG	internal/compute_node.go:54	Found new job to schedule: 
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}

20h53m58.89	DEBUG	internal/compute_node.go:112	Selecting for job with matching CID(s): [{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]

20h53m58.89	DEBUG	ipfs/utils.go:197	[Beginning to collect all refs in IPFS Repo.]
20h53m58.89	DEBUG	ipfs/utils.go:198	RepoPath {/tmp/bacalhau-ipfs4068152898}
20h53m58.89	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs refs local

20h53m58.89	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs refs local

20h53m58.89	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [refs local]
20h53m58.89	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [refs local]
20h53m58.90	DEBUG	system/utils.go:43	Error wait for results to be: Expected job to be complete, got [], pausing and trying again...

20h53m58.90	DEBUG	system/utils.go:43	Error wait for results to be: Expected job to be complete, got [], pausing and trying again...
20h53m59.00	DEBUG	ipfs/utils.go:204	[Finished collecting refs in IPFS Repo.]
20h53m59.00	DEBUG	ipfs/utils.go:208	Comparing CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) collecting to all refs in repo.
20h53m59.00	DEBUG	ipfs/utils.go:210	Total number of local refs: 23
20h53m59.00	DEBUG	ipfs/utils.go:216	CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) in local refs: false
20h53m59.00	INFO	internal/compute_node.go:128	No matching CIDs found on this server. Passing on job
20h53m59.00	DEBUG	internal/compute_node.go:67	We ignored a job because we didn't have the data: 
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
20h53m59.01	DEBUG	ipfs/utils.go:204	[Finished collecting refs in IPFS Repo.]
20h53m59.01	DEBUG	ipfs/utils.go:208	Comparing CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) collecting to all refs in repo.
20h53m59.01	DEBUG	ipfs/utils.go:210	Total number of local refs: 23
20h53m59.01	DEBUG	ipfs/utils.go:216	CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) in local refs: false
20h53m59.01	INFO	internal/compute_node.go:128	No matching CIDs found on this server. Passing on job
20h53m59.01	DEBUG	internal/compute_node.go:67	We ignored a job because we didn't have the data: 
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}
20h53m59.09	DEBUG	ipfs/utils.go:204	[Finished collecting refs in IPFS Repo.]
20h53m59.09	DEBUG	ipfs/utils.go:208	Comparing CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) collecting to all refs in repo.
20h53m59.09	DEBUG	ipfs/utils.go:210	Total number of local refs: 25
20h53m59.09	DEBUG	ipfs/utils.go:216	CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) in local refs: true
20h53m59.09	INFO	internal/compute_node.go:123	CID ([{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]) found on this server. Accepting job.
20h53m59.09	DEBUG	internal/compute_node.go:62	We are bidding on a job because the data is local! 
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}

20h53m59.09	DEBUG	libp2p/libp2p.go:329	Sending event: {"JobId":"e6fa8515-2907-4f2e-8cdf-56f0aab6d2a2","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"bid","JobSpec":null,"JobDeal":null,"JobState":{"State":"bidding","Status":"","Outputs":null}}
20h53m59.10	DEBUG	libp2p/libp2p.go:329	Sending event: {"JobId":"e6fa8515-2907-4f2e-8cdf-56f0aab6d2a2","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"bid_accepted","JobSpec":null,"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":["QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T"]},"JobState":{"State":"running","Status":"","Outputs":null}}
20h53m59.11	DEBUG	system/utils.go:16	IPFS Command: mkdir [-p /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T]
20h53m59.11	DEBUG	system/utils.go:54	Running system command: sudo [docker run --privileged -d --rm --name bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426 --entrypoint bash binocarlos/bacalhau-ignite-image:v1 -c tail -f /dev/null]
20h53m59.70	DEBUG	runtime/runtime.go:141	Script to run for job: sleep 2
grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX
sleep 2
20h53m59.70	DEBUG	system/utils.go:54	Running system command: sudo [docker cp /tmp/bacalhau-ignite-job.3212996715.sh bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426:/job.sh]
20h53m59.86	DEBUG	system/utils.go:54	Running system command: sudo [docker exec bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426 ipfs init]
20h53m59.90	DEBUG	system/utils.go:43	Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
20h54m00.43	DEBUG	system/utils.go:54	Running system command: sudo [docker exec bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426 ipfs bootstrap rm --all]
20h54m00.69	DEBUG	system/utils.go:54	Running system command: sudo [docker exec bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426 ipfs bootstrap add /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2]
20h54m00.95	DEBUG	runtime/runtime.go:201	Running system command: sudo [docker exec bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426 ipfs daemon --mount]
20h54m01.90	DEBUG	system/utils.go:43	Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
20h54m04.91	DEBUG	system/utils.go:43	Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
20h54m05.95	DEBUG	system/utils.go:28	Running system command: sudo [docker exec bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426 psrecord bash /job.sh --log /tmp/metrics.log --plot /tmp/metrics.png --include-children]
kiwi is delicious
20h54m08.91	DEBUG	system/utils.go:43	Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
Starting up command 'bash /job.sh' and attaching to process
Process finished (4.00 seconds)
/usr/local/lib/python3.8/dist-packages/psrecord/main.py:219: UserWarning: Attempting to set identical bottom == top == 0.0 results in singular transformations; automatically expanding.
  ax.set_ylim(0., max(log['cpu']) * 1.2)
20h54m11.90	DEBUG	runtime/runtime.go:249	Writing stdout to /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T/stdout.log
20h54m11.90	DEBUG	runtime/runtime.go:255	Writing stderr to /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T/stderr.log

20h54m11.90	INFO	runtime/runtime.go:261	Finished writing results of job (Id: f7a1a9e3-82cf-4689-b65d-49517876a426) to results folder (/home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T).
20h54m11.90	DEBUG	runtime/runtime.go:270	Copying files - Writing metrics.log to /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T/metrics.log

20h54m11.90	DEBUG	system/utils.go:16	IPFS Command: sudo [docker cp bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426:/tmp/metrics.log /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T/metrics.log]
20h54m12.07	DEBUG	runtime/runtime.go:270	Copying files - Writing metrics.png to /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T/metrics.png

20h54m12.07	DEBUG	system/utils.go:16	IPFS Command: sudo [docker cp bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426:/tmp/metrics.png /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T/metrics.png]
20h54m12.21	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs add -rq /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T
20h54m12.21	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [add -rq /home/user/.bacalhau/results/e6fa8515/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T]
20h54m12.39	DEBUG	system/utils.go:54	Running system command: sudo [docker rm -f bacalhaue6fa8515-2907-4f2e-8cdf-56f0aab6d2a2f7a1a9e3-82cf-4689-b65d-49517876a426]
20h54m12.49	DEBUG	runtime/runtime.go:214	Failed to run : exit status 137
20h54m12.74	INFO	internal/compute_node.go:86	Completed the job - results cid: QmdhfyBxpZEv7sor6vNbdJYz1prGih7uPNGuHwoBAYJ8f8
&{Id:e6fa8515-2907-4f2e-8cdf-56f0aab6d2a2 Owner:QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T Spec:0xc00017e070 Deal:0xc00161b4a0 State:map[QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T:0xc0029d8200]}
20h54m12.75	DEBUG	libp2p/libp2p.go:329	Sending event: {"JobId":"e6fa8515-2907-4f2e-8cdf-56f0aab6d2a2","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"results","JobSpec":null,"JobDeal":null,"JobState":{"State":"complete","Status":"Got job results cid: QmdhfyBxpZEv7sor6vNbdJYz1prGih7uPNGuHwoBAYJ8f8","Outputs":[{"Engine":"ipfs","Cid":"QmdhfyBxpZEv7sor6vNbdJYz1prGih7uPNGuHwoBAYJ8f8"}]}}
=== RUN   TestCommands/sed
20h54m41.73	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs add -Q ../../testdata/sed_file.txt
20h54m41.73	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [add -Q ../../testdata/sed_file.txt]
20h54m41.86	DEBUG	bacalhau/devstack_test.go:261	About to submit job:
cmd: sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG
20h54m41.86	DEBUG	libp2p/libp2p.go:329	Sending event: {"JobId":"172c0fc0-a270-4937-965d-ae1f53d8107a","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"job_created","JobSpec":{"Engine":"","Commands":["sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG"],"Image":"","Cpu":0,"Memory":0,"Disk":0,"Inputs":[{"Engine":"ipfs","Cid":"QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG"}]},"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":null},"JobState":null}

20h54m41.87	DEBUG	internal/compute_node.go:54	Found new job to schedule: 
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

20h54m41.87	DEBUG	internal/compute_node.go:112	Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]

20h54m41.87	DEBUG	ipfs/utils.go:197	[Beginning to collect all refs in IPFS Repo.]
20h54m41.87	DEBUG	ipfs/utils.go:198	RepoPath {/tmp/bacalhau-ipfs718364386}
20h54m41.87	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs718364386 ipfs refs local

20h54m41.87	INFO	bacalhau/submit.go:130	Submitted Job Id: 172c0fc0-a270-4937-965d-ae1f53d8107a

20h54m41.87	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [refs local]
20h54m41.87	DEBUG	internal/compute_node.go:54	Found new job to schedule: 
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
20h54m41.87	DEBUG	internal/compute_node.go:112	Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]

20h54m41.87	DEBUG	ipfs/utils.go:197	[Beginning to collect all refs in IPFS Repo.]
20h54m41.87	DEBUG	ipfs/utils.go:198	RepoPath {/tmp/bacalhau-ipfs4068152898}
20h54m41.87	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs4068152898 ipfs refs local

20h54m41.87	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [refs local]
20h54m41.87	DEBUG	internal/compute_node.go:54	Found new job to schedule: 
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

20h54m41.87	DEBUG	internal/compute_node.go:112	Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]

20h54m41.87	DEBUG	ipfs/utils.go:197	[Beginning to collect all refs in IPFS Repo.]
20h54m41.87	DEBUG	ipfs/utils.go:198	RepoPath {/tmp/bacalhau-ipfs150944631}
20h54m41.87	DEBUG	system/utils.go:43	Error wait for results to be: Expected job to be complete, got [], pausing and trying again...
20h54m41.87	DEBUG	ipfs/utils.go:24	ipfs command -->   IPFS_PATH=/tmp/bacalhau-ipfs150944631 ipfs refs local
20h54m41.87	DEBUG	system/utils.go:61	Running system command: /usr/local/bin/ipfs [refs local]
20h54m41.87	DEBUG	system/utils.go:43	Error wait for results to be: Expected job to be complete, got [], pausing and trying again...
20h54m41.99	DEBUG	ipfs/utils.go:204	[Finished collecting refs in IPFS Repo.]
20h54m41.99	DEBUG	ipfs/utils.go:208	Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
20h54m41.99	DEBUG	ipfs/utils.go:210	Total number of local refs: 23
20h54m41.99	DEBUG	ipfs/utils.go:216	CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: false
20h54m41.99	INFO	internal/compute_node.go:128	No matching CIDs found on this server. Passing on job
20h54m41.99	DEBUG	internal/compute_node.go:67	We ignored a job because we didn't have the data: 
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
20h54m42.00	DEBUG	ipfs/utils.go:204	[Finished collecting refs in IPFS Repo.]
20h54m42.00	DEBUG	ipfs/utils.go:208	Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
20h54m42.00	DEBUG	ipfs/utils.go:210	Total number of local refs: 34
20h54m42.00	DEBUG	ipfs/utils.go:216	CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: true
20h54m42.00	INFO	internal/compute_node.go:123	CID ([{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]) found on this server. Accepting job.
20h54m42.00	DEBUG	internal/compute_node.go:62	We are bidding on a job because the data is local! 
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

20h54m42.00	DEBUG	libp2p/libp2p.go:329	Sending event: {"JobId":"172c0fc0-a270-4937-965d-ae1f53d8107a","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"bid","JobSpec":null,"JobDeal":null,"JobState":{"State":"bidding","Status":"","Outputs":null}}
20h54m42.01	DEBUG	libp2p/libp2p.go:329	Sending event: {"JobId":"172c0fc0-a270-4937-965d-ae1f53d8107a","NodeId":"QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T","EventName":"bid_accepted","JobSpec":null,"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":["QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T"]},"JobState":{"State":"running","Status":"","Outputs":null}}
20h54m42.01	DEBUG	system/utils.go:16	IPFS Command: mkdir [-p /home/user/.bacalhau/results/172c0fc0/QmZc61Dq8Dz7hh7yA4jFZ3CBceypUGe1GctjH2jHcdF75T]
20h54m42.01	DEBUG	system/utils.go:54	Running system command: sudo [docker run --privileged -d --rm --name bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c --entrypoint bash binocarlos/bacalhau-ignite-image:v1 -c tail -f /dev/null]
20h54m42.02	DEBUG	ipfs/utils.go:204	[Finished collecting refs in IPFS Repo.]
20h54m42.02	DEBUG	ipfs/utils.go:208	Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
20h54m42.02	DEBUG	ipfs/utils.go:210	Total number of local refs: 23
20h54m42.02	DEBUG	ipfs/utils.go:216	CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: false
20h54m42.02	INFO	internal/compute_node.go:128	No matching CIDs found on this server. Passing on job
20h54m42.02	DEBUG	internal/compute_node.go:67	We ignored a job because we didn't have the data: 
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}
20h54m42.63	DEBUG	runtime/runtime.go:141	Script to run for job: sleep 2
sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG
sleep 2
20h54m42.63	DEBUG	system/utils.go:54	Running system command: sudo [docker cp /tmp/bacalhau-ignite-job.4225559538.sh bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c:/job.sh]
20h54m42.87	DEBUG	system/utils.go:54	Running system command: sudo [docker exec bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c ipfs init]
20h54m42.87	DEBUG	system/utils.go:43	Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
20h54m43.51	DEBUG	system/utils.go:54	Running system command: sudo [docker exec bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c ipfs bootstrap rm --all]
20h54m43.79	DEBUG	system/utils.go:54	Running system command: sudo [docker exec bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c ipfs bootstrap add /ip4/127.0.0.1/tcp/40349/p2p/12D3KooWMENvtysEkyjkwfVZaHL6R9rJRLeTCaZhHV4QVLkBmzC2]
20h54m44.11	DEBUG	runtime/runtime.go:201	Running system command: sudo [docker exec bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c ipfs daemon --mount]
20h54m44.88	DEBUG	system/utils.go:43	Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
20h54m47.88	DEBUG	system/utils.go:43	Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...
20h54m49.11	DEBUG	system/utils.go:28	Running system command: sudo [docker exec bacalhau172c0fc0-a270-4937-965d-ae1f53d8107a081a2ba6-b773-408a-b821-1aa4f840991c psrecord bash /job.sh --log /tmp/metrics.log --plot /tmp/metrics.png --include-children]
2021-01-01 05:54:00,LISBON,38.7305,-9.1502,11.0983,2.7888
2021-01-01 06:33:00,LISBON,38.7384,-9.1600,11.4359,3.9713
2021-01-01 12:54:00,LISBON,38.7482,-9.1376,14.3912,4.4965
2021-01-01 16:14:00,LISBON,38.7444,-9.1408,12.3883,4.0889
Detaching and terminating target process

To simplify debugging, we should understand/figure out how to group server information/output together.

When running image_processing example, got this error in the feed

2022/03/08 01:32:34 Starting ipfs daemon --mount inside the runtime failed with: exit status 137
13ff6ac3-85cd-4be7-9a20-5fe4216b71aa03887500-6c97-4228-a3d3-e36209cd0b31

Concern?

https://github.com/filecoin-project/bacalhau/tree/main/examples

report docker networking flakiness with ipfs upstream

Need to add the %s at the end because we don't have an elegant way to run without a cid (yet). Will fix later.

	// TODO:  This is stupid (for now) but need to add the %s at the end because we don't have an elegant way to run without a cid (yet). Will fix later.

https://github.com/filecoin-project/bacalhau/blob/d0aef41ca20deb14741a614eced28d92e3ed0f78/cmd/bacalhau/devstack_test.go#L345

13. Client can specify what parallelism they can specify for their jobs, and compare outputs

• Or, I want to wait until N compute nodes have run it, want to see a summary
• Deliverable: client can specify to run each job N times and the CLI displays the progress and compare the hashes of the outputs

14. Client confidence flag

• Client confidence flag
• Determine how many times the job gets run, what’s the requested number of copies?
• Deliverable: client can specify a confidence flag which is persisted for the job

, is removed as part of job submission

Submit:

awk -F','  '{x=38.7077507-; y=-9.1365919-; if(x^2+y^2<0.3^2) print}' /ipfs/$cid

Appears as:

awk -F' '  '{x=38.7077507-; y=-9.1365919-; if(x^2+y^2<0.3^2) print}' /ipfs/QmPVy9XiWRBrfvtiSBcCiWZSwSBZwVrG6oJo6eiar1qycP

Notice the lack of ,

Issue with "unterminated regex" when executing

When I execute the following command from the CLI, it works fine:

sed -n '/38.7[2-4]..,-9.1[3-7]../p' examples/gps-substitution/temperature_sensor_data.csv 

However, whene I execute the following, it doesn't work:

bin/bacalhau submit --cids=$cid --commands="sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/$cid" --jsonrpc-port=35203

With the following error:

sed: -e expression #1, char 12: unterminated address regex

Ideas?

Failed to parse multiaddr

Used a URL for peering with the following - this is a correct address, shouldn't fail.

Mar 19 11:28:23 ip-10-0-1-15 bacalhau[1611]: failed to parse multiaddr "/ip4/ip-10-0-1-78.eu-west-1.compute.internal/tcp/0/p2p/Qmdpeuf4BpyP51cm3W3ezazgLrMGy8rrJt9Bf3stbvByZ6": invalid value "ip-10-0-1-78.eu-west-1.compute.internal" for protocol ip4: failed to parse ip4 addr: ip-10-0-1-78.eu-west-1.compute.internal

Investigate ipfs fuse mounts in order to efficiently mount a dataset inside a firecracker vm running the docker container with the job in it (avoiding any copying)

Need a way to run locally

P0 - some way to run the command through the bacalhau CLI and execute against local for detecting syntax errors
P1 - some form of syntax checker

Results from many jobs aggregated

• FINAL DELIVERABLE: xxx (@lukemarsden to fill in)

Is this a threading issue?

When I execute the following command:

LOG_LEVEL=debug go test -run ^TestCommands$ github.com/filecoin-project/bacalhau/cmd/bacalhau/

The first (and only the first) test passes - though non-deterministically - e.g. could be any of the three randomly chosen.

Here's the log output:

[2207] DBG internal/dev_stack.go:116 > ==== Complete
[2207] DBG internal/dev_stack.go:123 > Finished provisioning nodes.
[2207] WRN bacalhau/devstack_test.go:213 >
========================================
Starting new job:
	name: awk
     cmd: awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/%s
    file: ../../testdata/awk_file.txt
========================================

[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [add -Q ../../testdata/awk_file.txt]
[2207] DBG bacalhau/devstack_test.go:275 > About to submit job:
cmd: awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"5b5a108a-3c17-4cef-a6be-277c42761788","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"job_created","JobSpec":{"Engine":"","Commands":["awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2\u003c0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG"],"Image":"","Cpu":0,"Memory":0,"Disk":0,"Inputs":[{"Engine":"ipfs","Cid":"QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG"}]},"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":null},"JobState":null}

[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]

[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs84900375}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]

[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs4201827199}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]

[2207] INF bacalhau/submit.go:130 > Submitted Job Id: 5b5a108a-3c17-4cef-a6be-277c42761788

[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs975119736}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [], pausing and trying again...

[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [], pausing and trying again...

[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 23
[2207] DBG ipfs/utils.go:214 > CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: false
[2207] INF internal/compute_node.go:137 > No matching CIDs found on this server. Passing on job
[2207] DBG internal/compute_node.go:74 > We ignored a job because we didn't have the data:
&{Engine: Commands:[awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 23
[2207] DBG ipfs/utils.go:214 > CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: false
[2207] INF internal/compute_node.go:137 > No matching CIDs found on this server. Passing on job
[2207] DBG internal/compute_node.go:74 > We ignored a job because we didn't have the data:
&{Engine: Commands:[awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 25
[2207] DBG ipfs/utils.go:214 > CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: true
[2207] INF internal/compute_node.go:132 > CID ([{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]) found on this server. Accepting job.
[2207] DBG internal/compute_node.go:65 > We are bidding on a job because the data is local!
&{Engine: Commands:[awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"5b5a108a-3c17-4cef-a6be-277c42761788","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"bid","JobSpec":null,"JobDeal":null,"JobState":{"State":"bidding","Status":"","Outputs":null}}

[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"5b5a108a-3c17-4cef-a6be-277c42761788","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"bid_accepted","JobSpec":null,"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":["QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua"]},"JobState":{"State":"running","Status":"","Outputs":null}}

[2207] DBG internal/compute_node.go:85 > BID ACCEPTED. Server (id: QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua) - Job (id: 5b5a108a-3c17-4cef-a6be-277c42761788)
[2207] DBG internal/compute_node.go:145 > Running job on node: QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG system/utils.go:83 > Enforcing creation of results dir: /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG internal/compute_node.go:174 > Ensured results directory created: /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] INF internal/compute_node.go:180 > Results directory for job id (5b5a108a-3c17-4cef-a6be-277c42761788) exists: /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG system/utils.go:54 > Running system command: sudo [docker run --privileged -d --rm --name bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562 --entrypoint bash binocarlos/bacalhau-ignite-image:v1 -c tail -f /dev/null]
[2207] DBG runtime/runtime.go:146 > Script to run for job: sleep 2
awk -F',' '{x=38.7077507-$3; y=-9.1365919-$4; if(x^2+y^2<0.3^2) print}' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG
sleep 2
 R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG system/utils.go:54 > Running system command: sudo [docker cp /tmp/bacalhau-ignite-job.1711282195.sh bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562:/job.sh]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562 ipfs init]
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...

[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562 ipfs bootstrap rm --all]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562 ipfs bootstrap add /ip4/127.0.0.1/tcp/34147/p2p/12D3KooWAc628yLorNsZC2XRV4zy5z7VGRNbE3nxo6u65F6VdKJd]
[2207] DBG runtime/runtime.go:208 > Running system ipfs daemon mount: /bin/sudo docker exec bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562 ipfs daemon --mount R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...

[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...

[2207] DBG system/utils.go:28 > Running system command: sudo [docker exec bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562 psrecord bash /job.sh --log /tmp/metrics.log --plot /tmp/metrics.png --include-children]
2021-01-01 00:00:00,LISBON,38.5940,-8.9303,5.9366,21.9316
2021-01-01 16:39:00,LISBON,38.6990,-8.9610,10.9692,15.2670
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...

/usr/local/lib/python3.8/dist-packages/psrecord/main.py:219: UserWarning: Attempting to set identical bottom == top == 0.0 results in singular transformations; automatically expanding.
  ax.set_ylim(0., max(log['cpu']) * 1.2)
Starting up command 'bash /job.sh' and attaching to process
Process finished (4.02 seconds)
[2207] DBG runtime/runtime.go:256 > Writing stdout to /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua/stdout.log R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG runtime/runtime.go:260 > Directory found: /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG runtime/runtime.go:266 > Expected folder /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua exists?: true R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG runtime/runtime.go:273 > Writing stderr to /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua/stderr.log
 R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] INF runtime/runtime.go:279 > Finished writing results of job (Id: 265f0663-6204-4da1-bc29-722bdb83c562) to results folder (/home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua). R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG runtime/runtime.go:288 > Copying files - Writing metrics.log to /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua/metrics.log
 R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG runtime/runtime.go:288 > Copying files - Writing metrics.png to /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua/metrics.png
 R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [add -rq /home/daaronch/.bacalhau/results/5b5a108a/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker rm -f bacalhau5b5a108a-3c17-4cef-a6be-277c42761788265f0663-6204-4da1-bc29-722bdb83c562]
[2207] DBG runtime/runtime.go:220 > Failed to run : exit status 137 R:5B5A108A-3C17-4CEF-A6BE-277C42761788
[2207] INF internal/compute_node.go:95 > Completed the job - results cid: QmReYEN87RiLa3DqFWLm5pvcW57zhxpjXh2P2zzpnqUfuF
&{Id:5b5a108a-3c17-4cef-a6be-277c42761788 Owner:QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua Spec:0xc002e60460 Deal:0xc0001f1e00 State:map[QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua:0xc0001f2980]}

[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"5b5a108a-3c17-4cef-a6be-277c42761788","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"results","JobSpec":null,"JobDeal":null,"JobState":{"State":"complete","Status":"Got job results cid: QmReYEN87RiLa3DqFWLm5pvcW57zhxpjXh2P2zzpnqUfuF","Outputs":[{"Engine":"ipfs","Cid":"QmReYEN87RiLa3DqFWLm5pvcW57zhxpjXh2P2zzpnqUfuF"}]}}

[2207] WRN bacalhau/devstack_test.go:213 >
========================================
Starting new job:
	name: grep
     cmd: grep kiwi /ipfs/%s
    file: ../../testdata/grep_file.txt
========================================

[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [add -Q ../../testdata/grep_file.txt]
[2207] DBG bacalhau/devstack_test.go:275 > About to submit job:
cmd: grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"d61f7f7f-4ab8-4488-ab3d-f25ffe43a159","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"job_created","JobSpec":{"Engine":"","Commands":["grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX"],"Image":"","Cpu":0,"Memory":0,"Disk":0,"Inputs":[{"Engine":"ipfs","Cid":"QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX"}]},"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":null},"JobState":null}

[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}

[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}

[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]

[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]

[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs975119736}
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs84900375}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}

[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]

[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs4201827199}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] INF bacalhau/submit.go:130 > Submitted Job Id: d61f7f7f-4ab8-4488-ab3d-f25ffe43a159

[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [], pausing and trying again...

[2207] WRN bacalhau/devstack_test.go:213 >
========================================
Starting new job:
	name: sed
     cmd: sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/%s
    file: ../../testdata/sed_file.txt
========================================

[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [add -Q ../../testdata/sed_file.txt]
[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 34
[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:214 > CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) in local refs: true
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) collecting to all refs in repo.
[2207] INF internal/compute_node.go:132 > CID ([{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]) found on this server. Accepting job.
[2207] DBG internal/compute_node.go:65 > We are bidding on a job because the data is local!
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}

[2207] DBG ipfs/utils.go:208 > Total number of local refs: 23
[2207] DBG ipfs/utils.go:214 > CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) in local refs: false
[2207] INF internal/compute_node.go:137 > No matching CIDs found on this server. Passing on job
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"d61f7f7f-4ab8-4488-ab3d-f25ffe43a159","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"bid","JobSpec":null,"JobDeal":null,"JobState":{"State":"bidding","Status":"","Outputs":null}}

[2207] DBG internal/compute_node.go:74 > We ignored a job because we didn't have the data:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}

[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"d61f7f7f-4ab8-4488-ab3d-f25ffe43a159","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"bid_accepted","JobSpec":null,"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":["QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua"]},"JobState":{"State":"running","Status":"","Outputs":null}}

[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 23
[2207] DBG ipfs/utils.go:214 > CID (QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX) in local refs: false
[2207] INF internal/compute_node.go:137 > No matching CIDs found on this server. Passing on job
[2207] DBG internal/compute_node.go:74 > We ignored a job because we didn't have the data:
&{Engine: Commands:[grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]}

[2207] DBG internal/compute_node.go:85 > BID ACCEPTED. Server (id: QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua) - Job (id: d61f7f7f-4ab8-4488-ab3d-f25ffe43a159)
[2207] DBG internal/compute_node.go:145 > Running job on node: QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG system/utils.go:83 > Enforcing creation of results dir: /home/daaronch/.bacalhau/results/d61f7f7f/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG internal/compute_node.go:174 > Ensured results directory created: /home/daaronch/.bacalhau/results/d61f7f7f/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] INF internal/compute_node.go:180 > Results directory for job id (d61f7f7f-4ab8-4488-ab3d-f25ffe43a159) exists: /home/daaronch/.bacalhau/results/d61f7f7f/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG system/utils.go:54 > Running system command: sudo [docker run --privileged -d --rm --name bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3 --entrypoint bash binocarlos/bacalhau-ignite-image:v1 -c tail -f /dev/null]
[2207] DBG bacalhau/devstack_test.go:275 > About to submit job:
cmd: sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG
[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"73e5c514-62f3-42c1-8257-52039dcbe73d","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"job_created","JobSpec":{"Engine":"","Commands":["sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG"],"Image":"","Cpu":0,"Memory":0,"Disk":0,"Inputs":[{"Engine":"ipfs","Cid":"QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG"}]},"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":null},"JobState":null}

[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]

[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs84900375}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]

[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs4201827199}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG internal/compute_node.go:57 > Found new job to schedule:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

[2207] INF bacalhau/submit.go:130 > Submitted Job Id: 73e5c514-62f3-42c1-8257-52039dcbe73d

[2207] DBG internal/compute_node.go:121 > Selecting for job with matching CID(s): [{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]

[2207] DBG ipfs/utils.go:195 > Beginning to collect all refs in IPFS Repo.
[2207] DBG ipfs/utils.go:196 > RepoPath {/tmp/bacalhau-ipfs975119736}
[2207] DBG system/utils.go:61 > Running system command: /usr/local/bin/ipfs [refs local]
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...

[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...

[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 23
[2207] DBG ipfs/utils.go:214 > CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: false
[2207] INF internal/compute_node.go:137 > No matching CIDs found on this server. Passing on job
[2207] DBG internal/compute_node.go:74 > We ignored a job because we didn't have the data:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 34
[2207] DBG ipfs/utils.go:214 > CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: true
[2207] INF internal/compute_node.go:132 > CID ([{ipfs QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]) found on this server. Accepting job.
[2207] DBG internal/compute_node.go:65 > We are bidding on a job because the data is local!
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"73e5c514-62f3-42c1-8257-52039dcbe73d","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"bid","JobSpec":null,"JobDeal":null,"JobState":{"State":"bidding","Status":"","Outputs":null}}

[2207] DBG libp2p/libp2p.go:328 > Sending event: {"JobId":"73e5c514-62f3-42c1-8257-52039dcbe73d","NodeId":"QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua","EventName":"bid_accepted","JobSpec":null,"JobDeal":{"Concurrency":1,"Confidence":1,"Tolerance":0.1,"AssignedNodes":["QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua"]},"JobState":{"State":"running","Status":"","Outputs":null}}

[2207] DBG ipfs/utils.go:202 > Finished collecting refs in IPFS Repo.
[2207] DBG ipfs/utils.go:206 > Comparing CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) collecting to all refs in repo.
[2207] DBG ipfs/utils.go:208 > Total number of local refs: 23
[2207] DBG ipfs/utils.go:214 > CID (QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG) in local refs: false
[2207] INF internal/compute_node.go:137 > No matching CIDs found on this server. Passing on job
[2207] DBG internal/compute_node.go:74 > We ignored a job because we didn't have the data:
&{Engine: Commands:[sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG] Image: Cpu:0 Memory:0 Disk:0 Inputs:[{Engine:ipfs Cid:QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG}]}

[2207] DBG internal/compute_node.go:85 > BID ACCEPTED. Server (id: QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua) - Job (id: 73e5c514-62f3-42c1-8257-52039dcbe73d)
[2207] DBG internal/compute_node.go:145 > Running job on node: QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG system/utils.go:83 > Enforcing creation of results dir: /home/daaronch/.bacalhau/results/73e5c514/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG internal/compute_node.go:174 > Ensured results directory created: /home/daaronch/.bacalhau/results/73e5c514/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] INF internal/compute_node.go:180 > Results directory for job id (73e5c514-62f3-42c1-8257-52039dcbe73d) exists: /home/daaronch/.bacalhau/results/73e5c514/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua
[2207] DBG system/utils.go:54 > Running system command: sudo [docker run --privileged -d --rm --name bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c --entrypoint bash binocarlos/bacalhau-ignite-image:v1 -c tail -f /dev/null]
[2207] DBG runtime/runtime.go:146 > Script to run for job: sleep 2
grep kiwi /ipfs/QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX
sleep 2
 R:D61F7F7F-4AB8-4488-AB3D-F25FFE43A159
[2207] DBG system/utils.go:54 > Running system command: sudo [docker cp /tmp/bacalhau-ignite-job.2044548828.sh bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3:/job.sh]
[2207] DBG runtime/runtime.go:146 > Script to run for job: sleep 2
sed -n '/38.7[2-4]..,-9.1[3-7]../p' /ipfs/QmfNxptvVcr9R5X3KqxENWieFzRCKvncezkhf2WddMqhiG
sleep 2
 R:73E5C514-62F3-42C1-8257-52039DCBE73D
[2207] DBG system/utils.go:54 > Running system command: sudo [docker cp /tmp/bacalhau-ignite-job.578337918.sh bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c:/job.sh]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3 ipfs init]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c ipfs init]
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...

[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3 ipfs bootstrap rm --all]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c ipfs bootstrap rm --all]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3 ipfs bootstrap add /ip4/127.0.0.1/tcp/34147/p2p/12D3KooWAc628yLorNsZC2XRV4zy5z7VGRNbE3nxo6u65F6VdKJd]
[2207] DBG system/utils.go:54 > Running system command: sudo [docker exec bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c ipfs bootstrap add /ip4/127.0.0.1/tcp/34147/p2p/12D3KooWAc628yLorNsZC2XRV4zy5z7VGRNbE3nxo6u65F6VdKJd]
[2207] DBG runtime/runtime.go:208 > Running system ipfs daemon mount: /bin/sudo docker exec bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3 ipfs daemon --mount R:D61F7F7F-4AB8-4488-AB3D-F25FFE43A159
[2207] DBG runtime/runtime.go:208 > Running system ipfs daemon mount: /bin/sudo docker exec bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c ipfs daemon --mount R:73E5C514-62F3-42C1-8257-52039DCBE73D
[2207] DBG system/utils.go:43 > Error wait for results to be: Expected job to be complete, got [running], pausing and trying again...

[2207] DBG ipfs/utils.go:102 > got to after closing ipfs daemon

[2207] DBG libp2p/libp2p.go:109 > Closing bacalhau libp2p daemon

[2207] DBG libp2p/libp2p.go:109 > Closing bacalhau libp2p daemon

[2207] DBG libp2p/libp2p.go:109 > Closing bacalhau libp2p daemon

[2207] DBG ipfs/utils.go:102 > got to after closing ipfs daemon

[2207] DBG ipfs/utils.go:102 > got to after closing ipfs daemon

[2207] DBG libp2p/libp2p.go:111 > Closed bacalhau libp2p daemon

[2207] DBG libp2p/libp2p.go:111 > Closed bacalhau libp2p daemon

[2207] DBG libp2p/libp2p.go:111 > Closed bacalhau libp2p daemon

[2207] DBG system/utils.go:28 > Running system command: sudo [docker exec bacalhaud61f7f7f-4ab8-4488-ab3d-f25ffe43a159459fab34-0a26-4857-9707-0ccf27e226b3 psrecord bash /job.sh --log /tmp/metrics.log --plot /tmp/metrics.png --include-children]
[2207] DBG system/utils.go:28 > Running system command: sudo [docker exec bacalhau73e5c514-62f3-42c1-8257-52039dcbe73d22d93b06-fe9b-4293-9841-b14d0311e53c psrecord bash /job.sh --log /tmp/metrics.log --plot /tmp/metrics.png --include-children]
--- FAIL: TestCommands (30.88s)
    --- FAIL: TestCommands/grep (0.11s)
        devstack_test.go:235:
            	Error Trace:	devstack_test.go:235
            	Error:      	Received unexpected error:
            	            	open /home/daaronch/.bacalhau/results/d61f7f7f/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua/stdout.log: no such file or directory
            	Test:       	TestCommands/grep
        devstack_test.go:237:
            	Error Trace:	devstack_test.go:237
            	Error:      	Should be true
            	Test:       	TestCommands/grep
        devstack_test.go:239:
            	Error Trace:	devstack_test.go:239
            	Error:      	Not equal:
            	            	expected: 1
            	            	actual  : 4
            	Test:       	TestCommands/grep
            	Messages:   	Count mismatch:
            	            	Expected: 4
            	            	Actual: 1
    --- FAIL: TestCommands/sed (6.15s)
        devstack_test.go:235:
            	Error Trace:	devstack_test.go:235
            	Error:      	Received unexpected error:
            	            	open /home/daaronch/.bacalhau/results/73e5c514/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua/stdout.log: no such file or directory
            	Test:       	TestCommands/sed
        devstack_test.go:237:
            	Error Trace:	devstack_test.go:237
            	Error:      	Should be true
            	Test:       	TestCommands/sed
        devstack_test.go:239:
            	Error Trace:	devstack_test.go:239
            	Error:      	Not equal:
            	            	expected: 1
            	            	actual  : 7
            	Test:       	TestCommands/sed
            	Messages:   	Count mismatch:
            	            	Expected: 7
            	            	Actual: 1

I SUSPECT this is something about after the job finishing, the system closes down the nodes.

For example, midway through the log, you can see:

[2207] INF internal/compute_node.go:180 > Results directory for job id (73e5c514-62f3-42c1-8257-52039dcbe73d) exists: /home/daaronch/.bacalhau/results/73e5c514/QmVn72EZAJu5xtzXeCbGTA4sTCLVcw57pHMa5p9FDBADua

Which is confirmed here - https://github.com/filecoin-project/bacalhau/blob/reformatting_logs_for_multi_threading/internal/compute_node.go#L176

However, by the time it gets to here: https://github.com/filecoin-project/bacalhau/blob/d0aef41ca20deb14741a614eced28d92e3ed0f78/internal/runtime/runtime.go#L245

The directory is gone. A bit stumped - my suspicion:

1. The directory is gone because ipfs daemon shut down somewhere (here? https://github.com/filecoin-project/bacalhau/blob/9b2ea198cdb570804a1884c404ac9d15393928f1/internal/ipfs/utils.go#L100 or here? https://github.com/filecoin-project/bacalhau/blob/9b2ea198cdb570804a1884c404ac9d15393928f1/internal/jsonrpc_server.go#L75 or here? https://github.com/filecoin-project/bacalhau/blob/9b2ea198cdb570804a1884c404ac9d15393928f1/internal/scheduler/libp2p/libp2p.go#L108)

(I think the last one is the most likely, since it's the newest)

1. OR the "job preparation" is not being done on the same node as the runtime, but this feels wrong. The same vm is being run.

20. Introduce Scheduler interface so that ConsensusLabs team can implement smart contract version of scheduler

Deliverable:

• interface for scheduler, with passing tests

Need to allow results directory to be an environment variable

Particularly for testing, but generally important

Confidence score system in the client: filtering aggregated results step kubectl –wait

• FINAL DELIVERABLE: xxx (@lukemarsden to fill in)

1. Basic libp2p based system for communicating “I want to run a job”

• Client submits to network
• ipfs job submit - where does it go?
• I.e. mempool for jobs
• Deliverable: There’s a binary you can invoke that listens and it can join a network of other servers on the network and a CLI that can trigger “I got a message” and everyone can get the message

4. Job execution investigation: docker in firecracker

• I’m going to run the job: how does that work? Firecracker/docker
• Build a prototype of firecracker/docker
• Deliverable: a job can start a firecracker vm and launch an arbitrary docker container – we can run sed! (and we can see the result in the docker logs)

15. Interpreting results in terms of confidence flag into some output

• Deliverable: calculate a % of agreeing hashes before we call a result the “majority” result in the client

Manually tee stdout and stderr when running jobs

We just got bitten by trying to write directly to the os.Stdout & std.Stderr streams from go-routines and this also just blocked the tests

The solution is to just return a buffer of the commands stdout & stderr and print that buffer to the console when the job has finished

It would be nice to have streaming output though - let's do this by reading the buffer from a go-routine whilst the command is running

IPFS inside the VM only connects to the network outside the VM by sheer luck

The IPFS instance inside the Ignite VMs is told to connect to:

added /ip4/127.0.0.1/tcp/35787/p2p/12D3KooWFwnmuDniA3VY18BdyJMLhJKfbE4ntDswNmRjuvFxbQXE

However, 127.0.0.1 on that port doesn't route to the private IPFS network outside the VM, however it does work because of some announcement trick that IPFS does. It's nice that it works by chance, but make it more robust by giving it not the 0'th address but instead, maybe all of them? Or the one that looks like an IP of the host that the VMs can route to.

We should bail out if we do not fit the execution profile of this machine. E.g., the below:

			// TODO: We should bail out if we do not fit the execution profile of this machine. E.g., the below:

https://github.com/filecoin-project/bacalhau/blob/77beeebedd5d85fbc313f25c3638088dc9b0bd2f/internal/compute_node.go#L57

6. Distributed data based job selection (orchestration)

• Job selection for a miner and execution, I’ve got a chunk of data so i’m eligible to consider this job
• Do I have this data?
• Do I want to execute it?
• Deliverable: A compute node knows what CIDs it has and therefore can pick work from the mempool

2. IPFS integration design

• Deliverable: a design for how Bacalhau will integrate with IPFS, libp2p as needed

9. Compute node writing results

• Presenting results
• Do we write the results back to ipfs, if not where do they go?
• Could involve libp2p stuff again - client can request results?
• Deliverable: processed data goes back into ipfs and can be seen there

Docker needs to open ports when running in DevStack (for non-Linux networking)

Repro:
Mac M1, fresh from main

Execute:

make test

Result:

===========================

[2207] INF bacalhau/submit.go:147 > Submitted Job Id: 06b7779c-87fe-4c0c-b4a0-6e399f92e58e

[2207] INF internal/compute_node.go:138 > CID ([{ipfs QmRyDNzrxwcL4ENNGyKLr9bf1FAs2unLKyngCb5tqaUpsX}]) found on this server. Accepting job.
[2207] INF internal/compute_node.go:192 > Results directory for job id (06b7779c-87fe-4c0c-b4a0-6e399f92e58e) exists: /Users/daaronch/.bacalhau/results/06b7779c/QmRhnE4qG72UGxXV9TLQ5g1vDRwbS3ZaHd7YjhaDPedsGx
[2207] ERR internal/compute_node.go:95 > ERROR running the job: Error connecting host ipfs into container:
        Error: exit status 1
        Output: Error: connect 12D3KooWDTMfZuHVaYRGRbHmgS6AMeoDKWCiMDDekcBBPSVbWuBx failure: failed to dial 12D3KooWDTMfZuHVaYRGRbHmgS6AMeoDKWCiMDDekcBBPSVbWuBx:
  * [/ip4/172.17.0.3/tcp/4001] dial tcp4 172.17.0.3:4001: connect: connection refused

&{Id:06b7779c-87fe-4c0c-b4a0-6e399f92e58e Owner:QmRhnE4qG72UGxXV9TLQ5g1vDRwbS3ZaHd7YjhaDPedsGx Spec:0x1400299b7a0 Deal:0x1400306b2c0 State:map[QmRhnE4qG72UGxXV9TLQ5g1vDRwbS3ZaHd7YjhaDPedsGx:0x14000a528c0]}

    devstack_test.go:233: 
                Error Trace:    devstack_test.go:233
                Error:          Received unexpected error:
                                open /Users/daaronch/.bacalhau/results/06b7779c/QmRhnE4qG72UGxXV9TLQ5g1vDRwbS3ZaHd7YjhaDPedsGx/stdout.log: no such file or directory
                Test:           TestCommands/grep
    devstack_test.go:235: 
                Error Trace:    devstack_test.go:235
                Error:          Should be true
                Test:           TestCommands/grep
    devstack_test.go:237: 
                Error Trace:    devstack_test.go:237
                Error:          Not equal: 
                                expected: 1
                                actual  : 4
                Test:           TestCommands/grep
                Messages:       Count mismatch:
                                Expected: 4
                                Actual: 1

22. Filecoin integration [was Placeholder milestone 1]

• FINAL DELIVERABLE: xxx (@lukemarsden to fill in)

10. Compute node broadcasts results

• Libp2p results broadcasting and receiving
• Deliverable: client that wanted the work to be done now knows that the work has been done and where it lives

bacalhau fetch results unclear

No response on execution - not clear anything happened.