m1 Pro macOS 12.1 Xcode 13.2.1 Build lotus v1.13.2

Error

╰─➤ make all rm -f lotus go build -ldflags="-X=github.com/filecoin-project/lotus/build.CurrentCommit=+git.f9db71ee1.dirty -extldflags=-framework OpenCL" -o lotus ./cmd/lotus github.com/filecoin-project/filecoin-ffi/generated ld: warning: ignoring file /opt/homebrew/lib/libhwloc.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64 ld: warning: ignoring file /Users/m1/workspace/filecoin-project/lotus/extern/filecoin-ffi/libfilcrypto.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64 Undefined symbols for architecture x86_64:

Currently we don't build the FFI on CI for ARM, but we should, to make sure we don't break it.

Behavior

The go-fil-markets team occasionally reports that generate a piece commitment will fail with a panic from the filecoin-ffi code: failed to build tree.

To Reproduce

The following test trivially reproduces the issue:

func TestDoesNotExhaustFileDescriptors(t *testing.T) {
	m := 500         // loops
	n := uint64(508) // quantity of piece bytes

	for i := 0; i < m; i++ {
		// create a temporary file over which we'll compute CommP
		file, err := ioutil.TempFile("", "")
		if err != nil {
			panic(err)
		}

		// create a slice of random bytes (represents our piece)
		b := make([]byte, n)

		// load up our byte slice with random bytes
		if _, err = rand.Read(b); err != nil {
			panic(err)
		}

		// write buffer to temp file
		if _, err := bytes.NewBuffer(b).WriteTo(file); err != nil {
			panic(err)
		}

		// seek to beginning of file
		if _, err := file.Seek(0, 0); err != nil {
			panic(err)
		}

		if _, err = GeneratePieceCID(abi.RegisteredProof_StackedDRG2KiBSeal, file.Name(), abi.UnpaddedPieceSize(n)); err != nil {
			panic(err)
		}

		if err = file.Close(); err != nil {
			panic(err)
		}
	}
}

Possibly related: #168

There's a bunch of calls from Go <-> FFI <-> Rust instead of just "one" call that does all transformations and check the sig in one call.
One concrete example is the hash to curve: Go first call the rust library to hash all messages to the curve, that means, the rust code creates a G2 point for each messages, and then marshal it bytes and give it back to Go. Then the Go lib passes these bytes back to rust again, but rust needs to unmarshal it back to a point ! You loose a bunch of time here - verifying a slice of bytes is a valid point often requires an expensive check - multiexp - not sure how its done here.

I do think it would be possible to just pass it all at once to Rust and let it verify by itself. If there is no other reason not to do it.

When rebuilding from source on mac, filecoin.h is not updated (and I think the .a files neither), even when running make clean. Running cargo clean in the rust folder fixes it, but means you are rebuilding from scratch everytime.

In the generated filecoin.h file
Incorrect number of brackets in "extern C" block

/* filecoin Header */

#ifdef __cplusplus
extern "C" {{
#endif


#ifndef FILECOIN_H
#define FILECOIN_H

Fix it, please

pkg-config --cflags -- /go/src/github.com/filecoin-project/lotus/extern/filecoin-ffi/generated/../filcrypto.pc /go/src/github.com/filecoin-project/lotus/extern/filecoin-ffi/generated/../filcrypto.pc /go/src/github.com/filecoin-project/lotus/extern/filecoin-ffi/generated/../filcrypto.pc /go/src/github.com/filecoin-project/lotus/extern/filecoin-ffi/generated/../filcrypto.pc /go/src/github.com/filecoin-project/lotus/extern/filecoin-ffi/generated/../filcrypto.pc

cargo tree --duplicates shows quite a few duplicate packages. We can likely reduce the code size significantly by fixing this.

A bunch of functions in proofs.go accept a miner ID parameter as an actor ID. This id is then turned into a "prover id" by creating an ID address and then extracting its payload. But in implementations, all call sites started out with an ID address in the first place and had to extract the actor ID from it in order to call these functions.

Change the functions to just accept an address, and perhaps add a precondition that it is of protocol address.ID in order to fail fast if it's not.

We get source code of lotus v1.13.0.

1. cd lotus directory and make, it succeeds
2. but cd lotus/extern/filecoin-ffi, then
   rm rust/Cargo.lock,
   make clean && FFI_BUILD_FROM_SOURCE=1 ./install-filcrypto
   it fails now.

Building on ClearLinux platform currently fails with /usr/bin/ld: cannot find -lOpenCL.

OpenCL path doesn't seem to be available via ClearLinux swupd package manager. Any idea how to go around this? ClearLinux does seem to have the best benchmarks with AMD processors.

Thanks for your help!

I am running macOS 10.15. When I run install-filcrypto, it pulls the prebuilt release for Darwin and my CPU features, but it's built for macOS 11. When I try to link against this libfilcrypto, I get scary warnings that look like this. Have not tried to run lotus linked like this yet, so I don't know if it'll end up breaking, but especially the first error gives me really bad possible-scary-memory-corruption energy!

ld: warning: could not create compact unwind for _blst_sha256_block_data_order: does not use RBP or RSP based frame
ld: warning: object file ($HOME/pl-work/lotus/extern/filecoin-ffi/libfilcrypto.a(sha256_x64.o)) was built for newer macOS version (11.0) than being linked (10.15)

Suggested fix (if this is even a real issue?): check macos prebuilt version and compare it to the local version somehow? figure out whether there could be any other potential issues with incorrect prebuilt release selection?

Running "cargo test" from the 'rust' sub-directory results in the following errors:

test proofs::api::tests::test_faulty_sectors_v1 ... FAILED
test proofs::api::tests::test_faulty_sectors_v1_1 ... FAILED
test proofs::api::tests::test_sealing_versions ... FAILED

failures:

---- proofs::api::tests::test_faulty_sectors_v1 stdout ----
thread 'proofs::api::tests::test_faulty_sectors_v1' panicked at 'assertion failed: (left == right)
left: [],
right: [42]: sector 42 should be faulty', src/proofs/api.rs:2644:13

---- proofs::api::tests::test_faulty_sectors_v1_1 stdout ----
thread 'proofs::api::tests::test_faulty_sectors_v1_1' panicked at 'assertion failed: (left == right)
left: [],
right: [42]: sector 42 should be faulty', src/proofs/api.rs:2644:13

---- proofs::api::tests::test_sealing_versions stdout ----
thread 'proofs::api::tests::test_sealing_versions' panicked at 'seal_commit_phase2 failed: "No cached parameters found for stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f [failure finding /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.params]\n\n

CI leak detector checks take an hour or more to run. We now run 2 configurations of it (both pairing and blst). We should look into ways to speed this up.

On an m1 Mac, the nightly-2020-10-05 toolchain can't be found for the aarch64-apple-darwin platform, so the build fails.

Changing rust/rust-toolchain to 1.50.0 makes things start to compile, but it looks like it hits this issue in the sha2-asm crate, since I get output like this:

error: failed to run custom build command for `sha2-asm v0.5.5`

Caused by:
  process didn't exit successfully: `/Users/yusef/work/repos/filecoin-ffi/rust/target/release/build/sha2-asm-cb66c0b7bf3bbc7f/build-script-build` (exit code: 1)
  --- stdout
  TARGET = Some("aarch64-apple-darwin")
  OPT_LEVEL = Some("3")
  HOST = Some("aarch64-apple-darwin")
  CC_aarch64-apple-darwin = None
  CC_aarch64_apple_darwin = None
  HOST_CC = None
  CC = None
  CFLAGS_aarch64-apple-darwin = None
  CFLAGS_aarch64_apple_darwin = None
  HOST_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = None
  running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-arch" "arm64" "-Wall" "-Wextra" "-march=armv8-a+crypto" "-c" "-o" "/Users/yusef/work/repos/filecoin-ffi/rust/target/release/build/sha2-asm-e5881292d436fa64/out/src/sha256_aarch64.o" "-c" "src/sha256_aarch64.S"
  cargo:warning=src/sha256_aarch64.S:64:2: error: ADR/ADRP relocations must be GOT relative
  cargo:warning= adrp x2, .K
  cargo:warning= ^
  cargo:warning=src/sha256_aarch64.S:64:2: error: unknown AArch64 fixup kind!
  cargo:warning= adrp x2, .K
  cargo:warning= ^
  cargo:warning=src/sha256_aarch64.S:65:2: error: unknown AArch64 fixup kind!
  cargo:warning= add x2, x2, :lo12:.K
  cargo:warning= ^
  exit code: 1

  --- stderr

I think that sha2-asm is pulled in by rust-fil-proofs, but that's as far as I have time to dig atm :)

We should merge the filecoin-ffi-tools crate into here, it is not used anywhere else.

Describe the bug
GPU gets error when starting on Snark processing after being scheduled on PC2 phase. Recovers after 4 minutes

To Reproduce

2020-08-07T23:33:32.275 INFO storage_proofs_core::compound_proof > vanilla_proof:finish
2020-08-07T23:33:32.308 INFO storage_proofs_core::compound_proof > snark_proof:start
2020-08-07T23:33:32.418 INFO bellperson::groth16::prover > Bellperson 0.9.2 is being used!
2020-08-07T23:34:30.234Z INFO markets loggers/loggers.go:18 storage event {"name": "ProviderEventDealPublished", "proposal CID": "bafyreibtp5t6ffsm5q5uodattf2xc6zs2nawi2nqdhv7knoww4dlqgi7rm", "state": "StorageDealStaged", "message": ""}
2020-08-07T23:34:30.316Z INFO rpc [email protected]/client.go:204 rpc output message buffer {"n": 2}
2020-08-07T23:34:30.397Z INFO sectors [email protected]/sealing.go:123 Adding piece for deal 14602
2020-08-07T23:34:30.954Z INFO rpc [email protected]/client.go:204 rpc output message buffer {"n": 3}
2020-08-07T23:35:12.446 DEBUG bellperson::gpu::locks > Acquiring priority lock...
2020-08-07T23:35:12.446 DEBUG bellperson::gpu::locks > Priority lock acquired!
2020-08-07T23:35:12.660 INFO bellperson::gpu::locks > GPU is available for FFT!
2020-08-07T23:35:12.660 DEBUG bellperson::gpu::locks > Acquiring GPU lock...
2020-08-07T23:35:12.660 DEBUG bellperson::gpu::locks > GPU lock acquired!
2020-08-07T23:35:12.840 INFO bellperson::gpu::fft > FFT: 1 working device(s) selected.
2020-08-07T23:35:12.840 INFO bellperson::gpu::fft > FFT: Device 0: GeForce GTX 1080 Ti
2020-08-07T23:35:12.840 INFO bellperson::domain > GPU FFT kernel instantiated!
2020-08-07T23:35:13.376 WARN bellperson::gpu::locks > GPU FFT failed! Falling back to CPU... Error: Ocl Error:

################################ OPENCL ERROR ###############################

Error executing function: clEnqueueNDRangeKernel("radix_fft")

Status error code: CL_MEM_OBJECT_ALLOCATION_FAILURE (-4)

Please visit the following url for more information:

https://www.khronos.org/registry/cl/sdk/1.2/docs/man/xhtml/clEnqueueNDRangeKernel.html#errors

#############################################################################

[Repeats six times then]

2020-08-07T23:38:39.215 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 2/8 of length 153391689
2020-08-07T23:39:04.780 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2020-08-07T23:39:04.780 DEBUG bellperson::gpu::locks > Acquiring GPU lock...
2020-08-07T23:39:04.780 DEBUG bellperson::gpu::locks > GPU lock acquired!
2020-08-07T23:39:05.020 INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0)
2020-08-07T23:39:05.020 INFO bellperson::gpu::multiexp > Multiexp: Device 0: GeForce GTX 1080 Ti (Chunk-size: 6167411)
2020-08-07T23:39:05.020 INFO bellperson::multiexp > GPU Multiexp kernel instantiated!

Expected behavior
I would like to see GPU cleanly engage in C2 Phase

Screenshots
Error description from reference:

CL_MEM_OBJECT_ALLOCATION_FAILURE if there is a failure to allocate memory for data store associated with image or buffer objects specified as arguments to kernel.

BEFORE
nvidia-smi

Fri Aug 7 23:28:26 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.57 Driver Version: 450.57 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:09:00.0 Off | N/A |
| 58% 80C P2 232W / 250W | 2003MiB / 11170MiB | 100% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 3913035 C lotus-miner 2001MiB |
+-----------------------------------------------------------------------------+

AFTER
nvidia-smi

Fri Aug 7 23:55:50 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.57 Driver Version: 450.57 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:09:00.0 Off | N/A |
| 55% 76C P2 228W / 250W | 3675MiB / 11170MiB | 100% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 3913035 C lotus-miner 3673MiB |
+-----------------------------------------------------------------------------+

Version (run lotus version):
lotus version

Daemon: 0.4.3+git.460723d2+api0.8.1
Local: lotus version 0.4.3+git.460723d2

Additional context
Running Ubuntu 20.04 server
GeForce GTX 1080 Ti running latest NVIDIA driver 450.57

Related ENV variables used when building Lotus:
FIL_PROOFS_MAXIMIZE_CACHING=1
BELLMAN_CUSTOM_GPU=GeForce GTX 1080 Ti:3584
FFI_BUILD_FROM_SOURCE=1
RUSTFLAGS=-C target-cpu=native -g
FIL_PROOFS_USE_GPU_TREE_BUILDER=1
RUST_BACKTRACE=full
RUST_LOG=debug
FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1

Implementations reasonably need to be able to generate deterministic keys for testing an other purposes.

Note this is already implemented in filecoin-project/bls-signatures#17 but lost in repo churn.

In particular, note that blst portable downloads are not optimized for specific hardware, so user's interested in best performance should be building from source.

Use the function VerifyAggregateSeals to verify the AggregateSealVerifyProofAndInfos structure (using 4 proofs), and report an error: 'invalid compression point'. what does this mean？Please answer my question .thanks!

I am using the commit: 3088d56d3d7787893008c3b86ac34f6621a2342c

Steps followed:

1. ./build.sh

Error

The log for build.sh is:

go clean -cache -testcache .
rm -rf filcrypto.h filcrypto.pc libfilcrypto.a .install-filcrypto
rm -f ./runner
cd rust && cargo clean && cd ..
warning: unused config key `net.get-fetch-with-cli` in `/Users/narurkar/.cargo/config`
warning: unused config key `net.get-fetch-with-cli` in `/Users/narurkar/.cargo/config`
    Updating crates.io index
warning: unused config key `net.get-fetch-with-cli` in `/Users/narurkar/.cargo/config`
    Updating crates.io index
     Ignored package `cbindgen v0.20.0` is already installed, use --force to override
WARN: Skipping filcrypto::convert_aggregation_inputs - (not `extern "C"`
WARN: Skip filcrypto::LOG_INIT - (not `pub`).
WARN: Skip filcrypto::LOG_INIT - (not `no_mangle`).
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
WARN: Missing `[defines]` entry for `target_os = "windows"` in cbindgen config.
go clean -cache -testcache .
./install-filcrypto
+ auth_header=()
+ '[' -n '' ']'
++ dirname ./install-filcrypto
+ cd .
+ rust_sources_dir=rust
++ jq -r '.[].check_cpu_for_feature | select(. != null)'
+ cpu_features_required_for_optimized_release='adx
sha_ni
sha2
sse2
avx2
avx
sse4_2
sse4_1'
+ main
++ get_release_flags
++ local __features=
++ [[ ! -f /proc/cpuinfo ]]
++ echo '[get_release_flags] no /proc/cpuinfo file; falling back to Darwin feature detection'
[get_release_flags] no /proc/cpuinfo file; falling back to Darwin feature detection
+++ sysctl -a
+++ grep machdep.cpu
+++ tr '[:upper:]' '[:lower:]'
+++ grep features
++ __features=
++ feature_map=("adx:+adx" "sha_ni:+sha" "sha2:+sha2" "sse2:+sse2" "avx2:+avx2" "avx:+avx" "sse4_2:+sse4.2" "sse4_1:+sse4.1")
++ target_features=
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c adx
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c sha_ni
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c sha2
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c sse2
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c avx2
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c avx
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c sse4_2
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in '${cpu_features_required_for_optimized_release[@]}'
+++ echo ''
+++ grep -c sse4_1
++ current_feature=0
++ '[' 1 = 0 ']'
++ echo ''
+ local __release_flags=
+ '[' 1 '!=' 1 ']'
+ echo '[install-filcrypto/main] building libfilcrypto from local sources (dir = rust)'
[install-filcrypto/main] building libfilcrypto from local sources (dir = rust)
+ build_from_source filcrypto rust ''
+ local __library_name=filcrypto
+ local __rust_sources_path=rust
+ local __release_flags=
++ git rev-parse HEAD
+ local __repo_sha1=3088d56d3d7787893008c3b86ac34f6621a2342c
+ local __repo_sha1_truncated=3088d56d3d778789
+ echo 'building from source @ 3088d56d3d778789'
building from source @ 3088d56d3d778789
++ command -v cargo
+ '[' -x /Users/narurkar/.cargo/bin/cargo ']'
++ command -v rustup
+ '[' -x /Users/narurkar/.cargo/bin/rustup ']'
+ pushd rust
~/RenProject/go-workspace/src/github.com/renproject/multichain/chain/filecoin/filecoin-ffi/rust ~/RenProject/go-workspace/src/github.com/renproject/multichain/chain/filecoin/filecoin-ffi
+ cargo --version
cargo 1.53.0-nightly (65d57e6f3 2021-04-04)
+ additional_flags=
++ uname -s
+ '[' Darwin = Darwin ']'
+ build=lipo
+ additional_flags='--targets x86_64-apple-darwin,aarch64-apple-darwin '
+ gpu_flags=,gpu
+ '[' '' == 0 ']'
+ use_multicore_sdr=multicore-sdr
+ '[' '' == 0 ']'
+ '[' '' == 1 ']'
+ '[' '' == 1 ']'
+ '[' '' == 1 ']'
+ additional_flags='--targets x86_64-apple-darwin,aarch64-apple-darwin  --no-default-features --features multicore-sdr,blst,gpu'
+ echo 'Using additional build flags: --targets x86_64-apple-darwin,aarch64-apple-darwin  --no-default-features --features multicore-sdr,blst,gpu'
Using additional build flags: --targets x86_64-apple-darwin,aarch64-apple-darwin  --no-default-features --features multicore-sdr,blst,gpu
+ '[' -n '' ']'
++ cat rust-toolchain
+ ./scripts/build-release.sh filcrypto nightly-2021-04-24 lipo '--targets x86_64-apple-darwin,aarch64-apple-darwin  --no-default-features --features multicore-sdr,blst,gpu'
+ main filcrypto nightly-2021-04-24 lipo '--targets x86_64-apple-darwin,aarch64-apple-darwin  --no-default-features --features multicore-sdr,blst,gpu'
+ [[ -z filcrypto ]]
+ [[ -z nightly-2021-04-24 ]]
+ [[ -z lipo ]]
++ mktemp
+ __build_output_log_tmp=/var/folders/7h/3_b66mg534x0k6rj4dlczh340000gn/T/tmp.erNUH915
+ trap '{ rm -f $__build_output_log_tmp; }' EXIT
+ local '__rust_flags=--print native-static-libs '
+ RUSTFLAGS='--print native-static-libs '
+ cargo +nightly-2021-04-24 lipo --release --targets x86_64-apple-darwin,aarch64-apple-darwin --no-default-features --features multicore-sdr,blst,gpu
+ tee /var/folders/7h/3_b66mg534x0k6rj4dlczh340000gn/T/tmp.erNUH915
warning: unused config key `net.get-fetch-with-cli` in `/Users/narurkar/.cargo/config`
[INFO  cargo_lipo::meta] Will build universal library for ["filcrypto"]
[INFO  cargo_lipo::lipo] Building "filcrypto" for "x86_64-apple-darwin"
warning: unused config key `net.get-fetch-with-cli` in `/Users/narurkar/.cargo/config`
   Compiling libc v0.2.99
   Compiling autocfg v1.0.1
   Compiling cfg-if v1.0.0
   Compiling proc-macro2 v1.0.28
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.74
   Compiling lazy_static v1.4.0
   Compiling serde_derive v1.0.127
error[E0463]: can't find crate for `core`
  |
  = note: the `x86_64-apple-darwin` target may not be installed

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: could not compile `lazy_static`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error[E0463]: can't find crate for `core`
  |
  = note: the `x86_64-apple-darwin` target may not be installed

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: build failed
[ERROR cargo_lipo] Failed to build "filcrypto" for "x86_64-apple-darwin": Executing "/Users/narurkar/.rustup/toolchains/nightly-2021-04-24-aarch64-apple-darwin/bin/cargo" "--color" "auto" "build" "-p" "filcrypto" "--target" "x86_64-apple-darwin" "--release" "--lib" "--no-default-features" "--features" "multicore-sdr,blst,gpu" finished with error status: exit status: 101
+ rm -f /var/folders/7h/3_b66mg534x0k6rj4dlczh340000gn/T/tmp.erNUH915
make: *** [.install-filcrypto] Error 1

Even with FFI_USE_GPU=0 rust-gpu-tools are compiled and fail (rightfully) with:

error: At least one of the features cuda or opencl must be enabled.

I guess rust-gpu-tools should be a conditional dependency.

It looks like we're using enums in FFI types in several places, but this is inherently unsafe as we can end up creating enums with arbitrary values and the rust type system will assume that this can't happen.

This can cause problems like:

1. Code checking said enum value may be optimized away.
2. If the enum has no zero variant but a zero is passed, rust may interpret Option<Enum> as None.
3. Really, whatever the rust compiler wants to do.

Detailed design: https://hackmd.io/@mvdan/Hy7iK0TEY

Note that the design just outlines the problem and the high level design of how it would be fixed. I've deliberately left the implementation details vague, e.g. what kind of HTTP server to use for the proxy redirects.

description

segmentation violation fatal when calling packSXXXX functions generated by c-for-go

reproduce

OS:

➜  go-gc git:(dev) uname -r
3.10.0-957.el7.x86_64
➜  go-gc git:(dev) cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)

hardware
Intel E5 with 16GiB ram

go version:

➜  go-gc git:(dev) go version
go version go1.13.4 linux/amd64

rust version:

➜  go-gc git:(dev) cargo --version
cargo 1.42.0 (86334295e 2020-01-31)
➜  go-gc git:(dev) rustc --version
rustc 1.42.0 (b8cedc004 2020-03-09)

fatal log:

unexpected fault address 0xe4760000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe4760000 pc=0x45f2baa]

goroutine 76997 [running]:
runtime.throw(0x5c12fb2, 0x5)
  /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc0271d1698 sp=0xc0271d1668 pc=0x4031372
runtime.sigpanic()
  /usr/local/go/src/runtime/signal_unix.go:401 +0x3de fp=0xc0271d16c8 sp=0xc0271d1698 pc=0x4046f2e
github.com/filecoin-project/filecoin-ffi/generated.packSFilPublicReplicaInfo(0xc03d112000, 0xad4, 0xad4, 0xe4760000)
  /Users/janedoe/Desktop/workload/lotus/extern/filecoin-ffi/generated/cgo_helpers.go:3633 +0x5a fp=0xc0271d1740 sp=0xc0271d16c8 pc=0x45f2baa
github.com/filecoin-project/filecoin-ffi/generated.FilVerifyPost(0xc181715e8c7ede93, 0x631fc98317b96c02, 0x78d1049e8f726a8b, 0x83b242221db1d9, 0x0, 0x0, 0x0, 0x6f, 0xc03d112000, 0xad4, ...)
  /Users/janedoe/Desktop/workload/lotus/extern/filecoin-ffi/generated/generated.go:642 +0x2c3 fp=0xc0271d1830 sp=0xc0271d1740 pc=0x45f6c83
github.com/filecoin-project/filecoin-ffi.VerifyPoSt(0xc039c945c0, 0x20, 0x20, 0xc02da858c0, 0x4, 0x4, 0xc000944680, 0x4, 0x4, 0xc03d0fa000, ...)
  /Users/janedoe/Desktop/workload/lotus/extern/filecoin-ffi/proofs.go:78 +0x323 fp=0xc0271d1a48 sp=0xc0271d1830 pc=0x46003f3
github.com/filecoin-project/sector-storage/ffiwrapper.verifyPost(0x6024d20, 0xc03c331080, 0xc039c945c0, 0x20, 0x20, 0xc02da858c0, 0x4, 0x4, 0xc000944680, 0x4, ...)
  /Users/janedoe/go/pkg/mod/github.com/filecoin-project/[email protected]/ffiwrapper/verifier_cgo.go:142 +0xfc fp=0xc0271d1b18 sp=0xc0271d1a48 pc=0x463b8ac
github.com/filecoin-project/sector-storage/ffiwrapper.proofVerifier.VerifyElectionPost(...)
  /Users/janedoe/go/pkg/mod/github.com/filecoin-project/[email protected]/ffiwrapper/verifier_cgo.go:129
github.com/filecoin-project/lotus/chain.(*Syncer).VerifyElectionPoStProof(0xc00204f480, 0x6024d20, 0xc03c331080, 0xc000941500, 0xc03c3526c0, 0x31, 0xc033da24b0, 0x403011a)
  /Users/janedoe/Desktop/workload/lotus/chain/sync.go:775 +0xcac fp=0xc0271d1f10 sp=0xc0271d1b18 pc=0x490c7cc
github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock.func6(0xc02638c780, 0x1)
  /Users/janedoe/Desktop/workload/lotus/chain/sync.go:644 +0x5b fp=0xc0271d1f70 sp=0xc0271d1f10 pc=0x49176cb
github.com/Gurpartap/async.Err.func1(0xc03c293200, 0xc03c301240, 0xc03c2fcc10)
  /Users/janedoe/go/pkg/mod/github.com/!gurpartap/[email protected]/error.go:29 +0x5b fp=0xc0271d1fc8 sp=0xc0271d1f70 pc=0x46f6afb
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0271d1fd0 sp=0xc0271d1fc8 pc=0x4061651
created by github.com/Gurpartap/async.Err
  /Users/janedoe/go/pkg/mod/github.com/!gurpartap/[email protected]/error.go:27 +0x8a

one can easily reproduce this situation with this demo
use make run and you'll see fatals like below soon enough

➜  go-gc git:(dev) make run
./run.sh
    Finished release [optimized] target(s) in 0.03s
rust target dir /mnt/rust/target
2020/04/08 09:10:22 20414
unexpected fault address 0x7efd85f68010
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7efd85f68010 pc=0x49b1a7]

goroutine 1 [running]:
runtime.throw(0x4d4191, 0x5)
	/home/dtynn/go/src/runtime/panic.go:774 +0x72 fp=0xc00004ddb0 sp=0xc00004dd80 pc=0x42bd72
runtime.sigpanic()
	/home/dtynn/go/src/runtime/signal_unix.go:401 +0x3de fp=0xc00004dde0 sp=0xc00004ddb0 pc=0x43e74e
github.com/dtynn/go-gc/gen.packSGogcPublicReplicaInfo(0xc0000a8000, 0x3d0900, 0x3d0900, 0x7efd85f68010)
	/home/dtynn/proj/go-gc/gen/cgo_helpers.go:196 +0x57 fp=0xc00004de50 sp=0xc00004dde0 pc=0x49b1a7
github.com/dtynn/go-gc/gen.GogcVerifyPost(0xc0000a8000, 0x3d0900, 0x3d0900, 0x3d0900)
	/home/dtynn/proj/go-gc/gen/gen.go:19 +0x8c fp=0xc00004de90 sp=0xc00004de50 pc=0x49b33c
main.main()
	/home/dtynn/proj/go-gc/main.go:44 +0x281 fp=0xc00004df60 sp=0xc00004de90 pc=0x49b831
runtime.main()
	/home/dtynn/go/src/runtime/proc.go:203 +0x21e fp=0xc00004dfe0 sp=0xc00004df60 pc=0x42d70e
runtime.goexit()
	/home/dtynn/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00004dfe8 sp=0xc00004dfe0 pc=0x455cd1
exit status 2
make: *** [run] Error 1

cause

seems to be caused by golang gc
Curious about runtime.KeepAlive usage

Can you describe the steps in detail? Thank you very much

When building on M1, rust lib compiles properly however CGO build fails with 170 duplicate symbols. (Builds properly when downloading the prebuilt version on Intel).

Any idea what I may be doing wrong? Thanks

$ cd /path/to/filecoin-ffi
$ go test

go: downloading github.com/filecoin-project/specs-actors v0.6.0
go: extracting github.com/filecoin-project/specs-actors v0.6.0
go: finding github.com/filecoin-project/specs-actors v0.6.0
[src/proofs/api.rs:1022] &s = "StackedDrg2KiBV1"
[src/proofs/api.rs:1022] &s = "StackedDrg8MiBV1"
[src/proofs/api.rs:1022] &s = "StackedDrg512MiBV1"
[src/proofs/api.rs:1022] &s = "StackedDrg32GiBV1"
[src/proofs/api.rs:1022] &s = "StackedDrg64GiBV1"
--- FAIL: TestProofsLifecycle (0.25s)
    proofs_test.go:126:
                Error Trace:    proofs_test.go:126
                                                        workflows.go:115
                                                        proofs_test.go:30
                Error:          Received unexpected error:
                                Cache is only available for 32GiB and 64GiB sectors
                                github.com/filecoin-project/filecoin-ffi.SealPreCommitPhase1
                                        /usr/local/services/lotus/extern/filecoin-ffi/proofs.go:291
                                github.com/filecoin-project/filecoin-ffi.WorkflowProofsLifecycle
                                        /usr/local/services/lotus/extern/filecoin-ffi/workflows.go:114
                                github.com/filecoin-project/filecoin-ffi.TestProofsLifecycle
                                        /usr/local/services/lotus/extern/filecoin-ffi/proofs_test.go:30
                                testing.tRunner
                                        /usr/local/services/go/src/testing/testing.go:909
                                runtime.goexit
                                        /usr/local/services/go/src/runtime/asm_amd64.s:1357
                Test:           TestProofsLifecycle
                Messages:       []
devices: [GeForce RTX 2080 SUPER]
FAIL
exit status 1
FAIL    github.com/filecoin-project/filecoin-ffi        38.334s

The following has been moved from a Slack conversation on May 7.

@dignifiedquire @icorderi @whyrusleeping @anorth

I think that the following is true:

Any change to the parameter version, regardless of whether it requires a code API change, is a breaking change (e.g. a verifying node which receives a v30 winning PoSt won't be able to verify it with their v29 verify_winning_post function) from the perspective of a libfilcrypto consumer today. Furthermore, there is no way for a node to link multiple versions of libfilcrypto when building their node (overlapping symbols from multiple copies of the Rust standard library, for example).

I think we should do the following:

The libfilcrypto library should expose a set of proofs functions (seal, verify, whatever) per parameter version, up to some defined number of parameter versions (let's say that we start out with N=3) which we all agree to support. I think that filecoin-ffi should import N versions (identified by Git commit) of the filecoin-proofs crate, and should provide parameter version-prefixed functions in the filcrypto.h file, e.g. v27_fil_verify_winning_post.

Why would we do this?

The protocol will map a range of epochs to a proofs protocol version (such as V1). The nodes will need to map one of these protocol versions (such as V1) to a set of libfilcrypto functions (such as v27_fil_verify_winning_post), and will also be responsible for knowing the current parameter version.

If we give the nodes the ability to pick (within a protocol version) a set of functions appropriate for their parameter version, we allow the nodes a migration path from one protocol version to the other as well as from one parameter version to the other. This approach would also allow for a more finely-grained bug fix-deployment pipeline (we could hotfix v42 while continuing to support v43).

Thoughts?

Lotus version 1.7.1-dev+mainnet+git.f97784ad4 with GPU2 enabled.

If there is NO BELLMAN_CPU_UTILIZATION - PreCommit2 lasts 20-22 min, works on GPU and nvtop/nvidia-smi shows load.

If BELLMAN_CPU_UTILIZATION=0 - PreCommit2 lasts 90+ min and workrs only on CPU (GPU idle).
Unexpected, because it is =0 by default. Looks like any nonempty value enables CPU workload.

If BELLMAN_CPU_UTILIZATION=0.875 - PreCommit2 lasts 90+ min and workrs only on CPU (GPU idle).

Cargo doesn't support ARM64 iOS！

Hi,
I found that when fvm apply the message, I can receive ExecTraceBytes

But when fvm apply the implicit message, it is missing, is this expected?

Hi,

I'm new to go, but it looks like lotus 1.11.0 (commit 283793f9435b12be03e0de43ac017eba3dc1ceac tag v1.11.0) refers to filecoin-ffi 0.30.4:

https://github.com/filecoin-project/lotus/blob/v1.11.0/go.mod#L29

github.com/filecoin-project/filecoin-ffi v0.30.4-0.20200910194244-f640612a1a1f

Should the version in this repository be bumped from 0.30.3?

In my (brief) experience, the only reliable way to build this project is to run make clean followed by FFI_BUILD_FROM_SOURCE=1 make cgo-gen. This is frustrating and time consuming.

I'd expect make to just "do the thing" but it looks like the makefile needs to be refactored to make this happen.

rust-toolchain is currently set to an older nightly build.
Since the project is compatible with Rust 1.40, why not set it to stable or remove it?

➜ filecoin-ffi git:(master) ✗ rustup show
Default host: x86_64-apple-darwin
rustup home: /Users/xx/.rustup

installed toolchains

stable-x86_64-apple-darwin
nightly-2020-03-19-x86_64-apple-darwin (default)
nightly-x86_64-apple-darwin
1.8.0-x86_64-apple-darwin

installed targets for active toolchain

aarch64-apple-ios
x86_64-apple-darwin

active toolchain

nightly-2020-03-19-x86_64-apple-darwin (default)
rustc 1.44.0-nightly (f509b26a7 2020-03-18)

➜ filecoin-ffi git:(master) ✗

Where should I configure aarch64-apple-ios and then build the static library to support aarch64-apple-ios..

Description

A build from source failure has been reported since the install script does not install the required cargo lipo tooling, whereas the CI scripting installs all required pre-reqs. This shouldn't affect systems that already have all of the pre-reqs, but shouldn't fail if they are not present.

the toFilPoStProofs function has cgo.NewPoStProof(pp, src[idx].ProofBytes). but it do't has free this new object

{"ExitCode":0,"ReturnVal":null,"GasUsed":22114968,"PenaltyHi":0,"PenaltyLo":0,"MinerTipHi":0,"MinerTipLo":0,"BaseFeeBurnHi":0,"BaseFeeBurnLo":0,"OverEstimationBurnHi":0,"OverEstimationBurnLo":0,"RefundHi":0,"RefundLo":0,"GasRefund":0,"GasBurned":0,"ExecTrace":null,"FailureInfo":""}

FvMachineExecuteMessage calls the implicit message and the result is empty

Currently we generate release for any commit, this should only happen for the master branch

There is an ask over in estuary for a windows binary. Would it be possible to have a windows release of filecoin-ffi?

go get . this library does not work:

# pkg-config --cflags  -- <...>go/pkg/mod/github.com/filecoin-project/fil
[email protected]/generated/../filcrypto.pc <...>/go/pkg/mod/github.com/filecoin-project/[email protected]
6500/generated/../filcrypto.pc <...>/go/pkg/mod/github.com/filecoin-projec
t/[email protected]/generated/../filcrypto.pc <...>/go/pkg/mod/github.com/filecoin-project/[email protected]
20ed16500/generated/../filcrypto.pc
Package ../../go/pkg/mod/github.com/filecoin-project/[email protected]
37-41b20ed16500/generated/../filcrypto.pc was not found in the pkg-config search path
.
Perhaps you should add the directory containing `<...>go/pkg/mod/github.c
om/filecoin-project/[email protected]/generated/../filc
rypto.pc.pc'
to the PKG_CONFIG_PATH environment variable
Package '<...>pkg/mod/github.com/filecoin-project/[email protected]
20200324233837-41b20ed16500/generated/../filcrypto.pc', required by 'virtual:world',
not found
Package '<...>go/pkg/mod/github.com/filecoin-project/[email protected]
20200324233837-41b20ed16500/generated/../filcrypto.pc', required by 'virtual:world',
not found
Package '<...>/go/pkg/mod/github.com/filecoin-project/[email protected]
20200324233837-41b20ed16500/generated/../filcrypto.pc', required by 'virtual:world',
not found
Package <...>/go/pkg/mod/github.com/filecoin-project/[email protected]
20200324233837-41b20ed16500/generated/../filcrypto.pc', required by 'virtual:world',
not found

i have got a problem while compile lotus
####################################################

error: failed to run custom build command for filcrypto v0.7.5 (/kingdom/lotus/extern/filecoin-ffi/rust)

Caused by:
process didn't exit successfully: /kingdom/lotus/extern/filecoin-ffi/rust/target/release/build/filcrypto-b21c34c7f0029451/build-script-build (exit status: 101)
--- stderr
thread 'main' panicked at 'Could not generate header: CargoMetadata("/kingdom/lotus/extern/filecoin-ffi/rust/Cargo.toml", Metadata(Output { status: ExitStatus(ExitStatus(25856)), stdout: "", stderr: "warning: Patch opencl3 v0.2.4 (https://github.com/vmx/opencl3?branch=filecoin-ffi-v9-tmp-fix#80f4520a) was not used in the crate graph.\nCheck that the patched package version and available features are compatible\nwith the dependency requirements. If the patch has a different version from\nwhat is locked in the Cargo.lock file, run cargo update to use the new\nversion. This may also occur with an optional dependency that is not enabled.\n Downloading crates ...\nwarning: spurious network error (2 tries remaining): [28] Timeout was reached (download of fuchsia-cprng v0.1.1 failed to transfer more than 10 bytes in 30s)\nwarning: spurious network error (2 tries remaining): [28] Timeout was reached (failed to download any data for aesni v0.10.0 within 30s)\n Downloaded hmac v0.10.1\n Downloaded hkdf v0.10.0\n Downloaded rand_hc v0.2.0\n Downloaded rand_hc v0.3.1\n Downloaded wasi v0.10.2+wasi-snapshot-preview1\n Downloaded output_vt100 v0.1.2\n Downloaded kernel32-sys v0.2.2\n Downloaded rand_core v0.4.2\n Downloaded redox_syscall v0.2.10\n Downloaded aesni v0.10.0\n Downloaded fuchsia-cprng v0.1.1\n Downloaded winapi v0.2.8\nwarning: spurious network error (2 tries remaining): [28] Timeout was reached (download of winapi v0.3.9 failed to transfer more than 10 bytes in 30s)\nwarning: spurious network error (2 tries remaining): [28] Timeout was reached (failed to download any data for paired v0.22.0 within 30s)\nwarning: spurious network error (2 tries remaining): [28] Timeout was reached (download of winapi-x86_64-pc-windows-gnu v0.4.0 failed to transfer more than 10 bytes in 30s)\nwarning: spurious network error (2 tries remaining): [28] Timeout was reached (failed to download any data for winapi-i686-pc-windows-gnu v0.4.0 within 30s)\nwarning: spurious network error (1 tries remaining): [28] Timeout was reached (download of paired v0.22.0 failed to transfer more than 10 bytes in 30s)\nwarning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for winapi v0.3.9 within 30s)\nwarning: spurious network error (1 tries remaining): [28] Timeout was reached (download of winapi-i686-pc-windows-gnu v0.4.0 failed to transfer more than 10 bytes in 30s)\nwarning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for winapi-x86_64-pc-windows-gnu v0.4.0 within 30s)\nerror: failed to download from https://crates.io/api/v1/crates/winapi/0.3.9/download\n\nCaused by:\n [28] Timeout was reached (download of winapi v0.3.9 failed to transfer more than 10 bytes in 30s)\n" }))', build.rs:9:10
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

• rm -f /tmp/tmp.z2matGUEqQ
  make[1]: *** [.install-filcrypto] Error 101
  make[1]: Leaving directory `/kingdom/lotus/extern/filecoin-ffi'
  make: *** [build/.filecoin-install] Error 2

In 142aa2f
I got

• cargo +nightly build --release --no-default-features --features pairing,gpu
• tee /var/folders/2j/5hjcsc6x1gg3t7gyzjt2z0cw0000gn/T/tmp.puwcJhUY
  Updating git repository https://github.com/filecoin-project/rust-fil-proofs
  error: failed to get filecoin-hashers as a dependency of package filecoin-proofs-api v6.1.0 (https://github.com/filecoin-project/rust-filecoin-proofs-api?branch=m1-portable#88804d20)
  ... which is depended on by filcrypto v0.7.5 (/Users/yuda/go/src/github.com/filecoin-project/lotus/extern/filecoin-ffi/rust)

Caused by:
failed to load source for dependency filecoin-hashers

Caused by:
Unable to update https://github.com/filecoin-project/rust-fil-proofs?branch=m1-portable#48c7ee80

Caused by:
object not found - no match for id (48c7ee80a247fa63bcb4c53676860085bb51f513); class=Odb (9); code=NotFound (-3)

• rm -f /var/folders/2j/5hjcsc6x1gg3t7gyzjt2z0cw0000gn/T/tmp.puwcJhUY
  make: *** [.install-filcrypto] Error 101

when I run "make lotus" in 'file coin-project/lotus/extern/filecoin-ffi*m1-portable'

on my macOS Big Sur
11.2.1
base on m1

There is a problem with the judgment here. If there is "tpr_shadow" in the CPU Flags, then according to this judgment, deleting the left and right strings will also be "sha", then optimized will be true, but this is the Intel CPU.

I can solve this problem by pulling request

Consider adding something like this:

92ec33e?diff=unified&w=1