Coder Social home page Coder Social logo

secretflow / yacl Goto Github PK

View Code? Open in Web Editor NEW
63.0 6.0 57.0 2.58 MB

YACL (Yet Another Common crypto library) is a C++ library that contains cryptgraphy, network and io modules which other SecretFlow code depends on.

License: Apache License 2.0

Starlark 5.63% C++ 93.98% C 0.39%
crypto cryptography cryptography-library dpf oblivious-transfer ot secure-computation vole

yacl's Introduction

YACL (Yet Another Common crypto Library)

CircleCI OpenSSF Scorecard

Yacl is a C++ library that contains common cryptgraphy, network and io modules which other SecretFlow code depends on. The crypto modules in Yacl implement many state-of-art secure computation protocols, including primitives like OT, VOLE, TPRE, and tools like PRG, RO. Check the full list of Yacl's supported algorithms in ALGORITHMS.md.

Supported platforms:

Linux x86_64 Linux aarch64 macOS x86_64 macOS Apple Silicon Windows x86_64 Windows WSL2 x86_64
yes yes yes1 yes no yes1
  1. Yacl has not been thoroughly tested on these platforms.

Repo Layout

  • base: some basic types and utils in yacl.
  • crypto: crypto algorithms without link.
  • kernels: crypto kernels that includes link with multi-thread support, i.e. OT, DPF.
  • io: a simple streaming-based io library.
  • link: a simple rpc-based MPI framework, providing the SPMD parallel programming capability.

Prerequisites

Build & UnitTest

# build as debug
bazel build //... -c dbg

# build as release
bazel build //... -c opt

# test
bazel test //...

# [optional] build & test with ASAN if you're not on MacOS
bazel build //... -c dbg --config=asan
bazel test //... --config=asan -c dbg

# [optional] build & test with ASAN on MacOS
bazel build //... -c dbg --config=macos-asan
bazel test //... --config=macos-asan -c dbg

License

See LICENSE and NOTICE.md

yacl's People

Contributors

6fj avatar 982945902 avatar anakinxc avatar candicepan avatar cryptographer63 avatar huocun-ant avatar iiiimp avatar jamie-cui avatar maths644311798 avatar oeqqwq avatar primummobile avatar renovate[bot] avatar usafchn avatar zhangwfjh avatar zr-182 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

yacl's Issues

Problems in yacl/io/rw

In the 91 line of csv_writer.cc:
YACL_THROW("unknow Schema::type {}", types[c]);
Note that types[c] is a variable of enum type. So it does not fit into the parameters of fmt.
I think int(types[c]) will be better. Similar problems occur in csv_reader.cc.

在 YACL 上支持新的 Codes: Expand-Accumulate Codes

此 ISSUE 为 隐语开源共建计划(SecretFlow Open Source Contribution Plan,简称 SF OSCP)第三期任务 ISSUE,欢迎社区开发者参与共建~
若有感兴趣想要认领的任务,但还未报名,辛苦先完成报名进行哈~

任务介绍

  • 任务名称:在 YACL 上支持新的 Codes: Expand-Accumulate Codes
  • 技术方向:YACL,密码学,编码理论
  • 任务难度:进阶🌟🌟

详细要求

由于目前 Silent-Extension 的 OTe 算法基于不同的 LPN 问题假设,目前 YACL 已实现基于 Primal LPN 的算法,因此需要探索、支持基于其他假设(例如 dual LPM)的算法,支持 EA Code 是实现该类算法的前置工作。

  • 安全性
    • 遵循论文中的协议实现
    • if possible,默认安全强度为 128 bit
  • 功能性:
    • 支持 Encode 函数
    • 支持相应参数调整
  • 代码规范:
    • C++ 代码需要遵循 Google C++ style guidelines,除了以下几条 exceptions
      • Exceptions are allowed and encouraged where appropriate.
      • Header guards should use #pragma once.
      • Adopt camelBack for function names.
      • Use fixed width integer types whenever possible.
      • Avoid using size_t on interface APIs.
    • 可使用 clangd 进行代码格式化
    • 可使用 cpplint 检查格式
  • 提交说明:关联该 issue 并提交代码至 https://github.com/secretflow/yacl/tree/main/yacl/crypto/tools

能力要求

  • 熟悉密码学或者编码理论
  • 熟悉 C++ 以及性能优化

操作说明

bazel build //... -c opt

bazel build //... -c opt编译后怎么找到构建的二进制文件执行呀。找了很久没找到

yacl通信实际测试问题

根据factory_test.cc测试的一部分,我改造到真实的两台机器上测试,感觉有些问题。
环境:两台机器都是Ubuntu系统,地址分别为172.18.0.2, 172.18.0.3,分别取编号(rank)为0,1。
rank是0的机器运行代码如下

//Mytest.cpp
#include <iostream>
#include <string>
#include <vector>
#include <map>
#include <memory>
#include <type_traits>
#include <variant>
#include <unistd.h>
#include <future>
#include <limits>
#include "fmt/format.h"
#include "gtest/gtest.h"
#include "yacl/link/context.h"
#include "yacl/link/link.h"
#include "yacl/link/factory.h"

class FactoryTest{
 public:
  FactoryTest()
  {
    static int desc_count = 0;
    contexts_.resize(2);
    yacl::link::ContextDesc desc;
    desc.id = fmt::format("world_{}", desc_count++);
    desc.brpc_retry_count = 20;
    desc.parties.push_back(yacl::link::ContextDesc::Party("alice", "172.18.0.2:63927"));
    desc.parties.push_back(yacl::link::ContextDesc::Party("bob", "172.18.0.3:63921"));
    auto create_brpc = [&](int self_rank) {
      contexts_[self_rank] = yacl::link::FactoryBrpc().CreateContext(desc, self_rank);
    };
    std::vector<std::future<void>> creates;
    creates.push_back(std::async(create_brpc, 0));
    for (auto& f : creates) {
      f.get();
    }
    std::cout << "Connect to Bob successfully\n";
  }

  void work()
  {
    auto test = [&](int self_rank)
    {
      int dst_rank = 1 - self_rank;
      this->contexts_[self_rank]->SendAsync(dst_rank, "Hello I am 0", "test");
      yacl::Buffer r = this->contexts_[self_rank]->Recv(dst_rank, "test");
      std::string r_str(r.data<const char>(), r.size());
      std::cout << self_rank << " Receive "  << r_str << '\n';
    };
    std::vector<std::future<void>> tests;
    tests.push_back(std::async(test, 0));
    for (auto& f : tests) {
      f.get();
    }
  }

  ~FactoryTest()
  {
    auto wait = [&](int self_rank) {
      contexts_[self_rank]->WaitLinkTaskFinish();
    };
    std::vector<std::future<void>> waits;
    waits.push_back(std::async(wait, 0));
    for (auto& f : waits) {
      f.get();
    }
  }
  std::vector<std::shared_ptr<yacl::link::Context>> contexts_;
};

int main() {
  FactoryTest F;
  sleep(2);
  F.work();
  return 0;
}

编号为1的机器的代码主要改了上面的self_rank的取值。由于是手工启动,测试时两台机器启动程序的时间可能会相差几秒,先启动1号机器的程序,再启动0号机器的。上面代码运行没有问题,0号机器输出

0 Receive Hello I am 1

1号机器输出

1 Receive Hello I am 0

但是代码中如果去掉sleep(2)语句,再测试时就会有以下报错,0号机器报错

I0924 02:51:37.530009 1192314 /repository/brpc-1.6.0/src/brpc/server.cpp:1127] Server[yacl::link::transport::internal::ReceiverServiceImpl] is serving on port=63927.
Connect to Bob successfully
I0924 02:51:56.632742 1192407 /repository/brpc-1.6.0/src/brpc/socket.cpp:2465] Checking Socket{id=0 addr=172.18.0.3:63921} (0x7fbacc067020)
terminate called after throwing an instance of 'yacl::IoError'
what(): [/repository/yacl/yacl/link/transport/channel.cc:351] Get data timeout, key=world_0:P2P-1:1->0
Stacktrace:
#0 yacl::link::transport::Channel::Recv()+0x4d68b8

Aborted (core dumped)

1号机器报错


[2023-09-24 02:51:55.515] [info] [default_brpc_retry_policy.cc:29] cntl ErrorCode '112', http status code '200', response header '', error msg '[E111]Fail to connect Socket{id=0 addr=172.18.0.2:63927} (0x0x7f8a34067000): Connection refused [R1][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R2][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R3][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R4][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R5][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R6][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R7][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R8][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R9][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R10][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R11][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R12][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R13][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R1
auto wait = [&](int self_rank) {
4][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R15][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R16][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R17][E112]Not connected
to 172.18.0.2:63927 yet, server_id=0 [R18][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R19][E112]Not connected to 172.18.0.2:63927 yet, server_id=0'
[2023-09-24 02:51:55.515] [info] [default_brpc_retry_policy.cc:75] aggressive retry, sleep=1000000us and retry
I0924 02:51:56.516082 769 /repository/brpc-1.6.0/src/brpc/socket.cpp:2465] Checking Socket{id=0 addr=172.18.0.2:63927} (0x7f8a34067000)
1 Receive Hello I am 0
I0924 02:51:56.516975 695 /repository/brpc-1.6.0/src/brpc/socket.cpp:2525] Revived Socket{id=0 addr=172.18.0.2:63927} (0x7f8a34067000) (Connectable)
[2023-09-24 02:51:56.522] [error] [channel.cc:98] SendImpl error [/repository/yacl/yacl/link/transport/brpc_link.cc:187] send, rpc failed=112, message=[E111]Fail to connect Socket{id=0 addr=172.18.0.2:63927}
(0x0x7f8a34067000): Connection refused [R1][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R2][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R3][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R4][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R5][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R6][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R7][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R8][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R9][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R10][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R11][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R12][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R13][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R14][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R15][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R16][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R17][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R18][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R19][E112]Not connected to 172.18.0.2:63927 yet, server_id=0 [R20][E112]Not connected to 172.18.0.2:63927 yet, server_id=0
Stacktrace:
#0 yacl::link::transport::BrpcLink::SendRequest()+0x4cb5cf
#1 (unknown)+0x7f8a34002da0

上面省略了一些[info]段落。1号机器确实输出了”1 Receive Hello I am 0”,但0号机器似乎没有收到消息。我确信1号机器程序启动后,0号机器的程序在5秒内启动。

A question about pull #163

In a recent update, you added using ValueType = T; in arg_k.h and changed T GetRequired(const SpiArgKey<T> &key) const { in arg_set.h to

auto GetRequired(const SpiArgKey<T> &key) const ->
typename SpiArgKey<T>::ValueType {

I can not see any problems of the original codes. What’s the advantage of this update?

YACL ECC 模块支持 FourQ 曲线

此 ISSUE 为 隐语开源共建计划(SecretFlow Open Source Contribution Plan,简称 SF OSCP)任务 ISSUE,欢迎社区开发者参与共建~
若有感兴趣想要认领的任务,但还未报名,辛苦先完成报名进行哈~

任务介绍

  • 任务名称:YACL 支持 FourQ 曲线
  • 技术方向:ECC 椭圆曲线密码学
  • 任务难度:热身🌟
    • 任务期望时间:2-3 周

详细要求

YACL ECC 模块位于 yacl/crypto/base/ecc 路径下, ecc_spi.h 是模块的总入口,所有椭圆曲线遵循相同的接口。目前 ECC 模块已经实现了大部分主流曲线,但是缺少 fourq,本任务需要在 ECC 模块中集成/实现 fourq 曲线。
关于 fourq 曲线更详细的介绍可参考此 paper https://eprint.iacr.org/2015/565.pdf
鉴于微软已经实现了 FourQlib,简单起见您可以直接集成此 Lib 到 YACL

能力要求

  • 精通 C++ 语言
  • 了解基本 git 操作
  • 了解椭圆曲线密码学(ECC)

操作说明

  • 请在 YACL ECC 模块 中新建一个文件夹存放您的代码
  • 集成方式可参考 ECC 下其它 Lib 的集成方法,一般由一个 xxxGroup 实现曲线主体功能;一个 factory 用于创建曲线实例;一个 test 类用于单测
  • 集成完成后请执行 bazel run -c opt //yacl/crypto/base/ecc/benchmark:benchmark -- --curve=ed25519 运行性能测试

Problems in mpint_field.cc and item.h

(1)
In mpint_field.cc,

void MPIntField::NegInplace(MPInt *x) const {
  if (x->IsZero()) {
    return;
  }

  WEAK_ENFORCE(IsInField(*x), "x is not a valid field element, x={}", *x);
  x->NegateInplace();
  AddInplace(x, mod_);
  x->DecrOne();
}

This function does not need x->DecrOne();.

(2)
In item.h,

template <typename T>
  absl::Span<const T> AsSpan() const {
…
static_assert(!std::is_same_v<bool, RawT>,
                    "Call AsSpan<bool> on a vector item is not allowed");
…

Why "Call AsSpan<bool> on a vector item is not allowed"?

编译错误

运行命令:

bazel build //... -c dbg --config=macos-asan

报错日志

Loading: 
Loading: 0 packages loaded
Analyzing: 184 targets (0 packages loaded, 0 targets configured)
DEBUG: Rule 'org_interconnection' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1669271462 +0800"
DEBUG: Repository org_interconnection instantiated at:
  /Users/dsy/Desktop/yacl/WORKSPACE:19:10: in <toplevel>
  /Users/dsy/Desktop/yacl/bazel/repositories.bzl:59:10: in yacl_deps
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/build_defs/repo/utils.bzl:233:18: in maybe
Repository rule git_repository defined at:
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
DEBUG: Rule 'simplest_ot' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1658802777 +0800"
DEBUG: Repository simplest_ot instantiated at:
  /Users/dsy/Desktop/yacl/WORKSPACE:19:10: in <toplevel>
  /Users/dsy/Desktop/yacl/bazel/repositories.bzl:51:10: in yacl_deps
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/build_defs/repo/utils.bzl:233:18: in maybe
Repository rule git_repository defined at:
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
INFO: Analyzed 184 targets (0 packages loaded, 0 targets configured).
INFO: Found 184 targets...
[0 / 18] [Prepa] Creating source manifest for //yacl/crypto/base/hash:ssl_hash_all_test
[26 / 45] Compiling yacl/crypto/base/hash/ssl_hash_all_test.cc; 1s darwin-sandbox ... (8 actions, 7 running)
[32 / 45] Compiling yacl/crypto/base/aead/sm4_mac.cc; 2s darwin-sandbox ... (8 actions, 7 running)
ERROR: /Users/dsy/Desktop/yacl/yacl/crypto/base/ecc/BUILD.bazel:27:16: Compiling yacl/crypto/base/ecc/ecc_spi.cc failed: (Aborted): wrapped_clang_pp failed: error executing command external/local_config_cc/wrapped_clang_pp -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -g '-std=c++11' 'DEBUG_PREFIX_MAP_PWD=.' -iquote . -iquote ... (remaining 55 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
yacl/crypto/base/ecc/ecc_spi.cc:95:105: error: use of undeclared identifier 'ex'
      SPDLOG_DEBUG("Ec lib {} do not support curve {}, msg={}, try next ...",perf_item.second, ec_name, ex.what());
                                                                                                        ^
1 error generated.
Error in child process '/usr/bin/xcrun'. 1
INFO: Elapsed time: 2.948s, Critical Path: 2.55s
INFO: 43 processes: 29 internal, 14 darwin-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

分析

问题出在“yacl/crypto/base/ecc/ecc_spi.cc:95:105: error: use of undeclared identifier 'ex'
      SPDLOG_DEBUG("Ec lib {} do not support curve {}, msg={}, try next ...",perf_item.second, ec_name, ex.what());”

当我修改这行代码为“SPDLOG_DEBUG("Ec lib {} do not support curve {}, msg=, try next ...",perf_item.second, ec_name);”
**可以正常通过**,这只是临时解决方法,请yacl团队评估。

The order of headers

The order of headers in /yacl/link/transport/interconnection_base.cc is not good.

#include "yacl/link/transport/interconnection_base.h"

will actually include "spdlog/spdlog.h". But

#include "fmt/ostream.h"

should be above "spdlog/spdlog.h". See also gabime/spdlog#2872 (comment)

使用bazel编译yacl发生错误

lgs@dell-Precision-7920-Tower:/data/lgs/yinyu/yacl$ bazel build //... -c dbg
Starting local Bazel server and connecting to it...
INFO: Analyzed 195 targets (117 packages loaded, 9791 targets configured).
INFO: Found 195 targets...

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.UnsupportedOperationException: /data/lgs/yinyu/yacl/bazel-bin (??????)
at com.google.devtools.build.lib.unix.NativePosixFiles.symlink(Native Method)
at com.google.devtools.build.lib.unix.UnixFileSystem.createSymbolicLink(UnixFileSystem.java:344)
at com.google.devtools.build.lib.vfs.Path.createSymbolicLink(Path.java:500)
at com.google.devtools.build.lib.vfs.FileSystemUtils.ensureSymbolicLink(FileSystemUtils.java:343)
at com.google.devtools.build.lib.vfs.FileSystemUtils.ensureSymbolicLink(FileSystemUtils.java:294)
at com.google.devtools.build.lib.buildtool.OutputDirectoryLinksUtils.createLink(OutputDirectoryLinksUtils.java:332)
at com.google.devtools.build.lib.buildtool.OutputDirectoryLinksUtils.createOutputDirectoryLinks(OutputDirectoryLinksUtils.java:143)
at com.google.devtools.build.lib.buildtool.ExecutionTool.createConvenienceSymlinks(ExecutionTool.java:695)
at com.google.devtools.build.lib.buildtool.ExecutionTool.handleConvenienceSymlinks(ExecutionTool.java:654)
at com.google.devtools.build.lib.buildtool.ExecutionTool.executeBuild(ExecutionTool.java:363)
at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:235)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:443)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:411)
at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:103)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:585)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:231)
at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:550)
at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:614)
at io.grpc.Context$1.run(Context.java:579)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
lgs@dell-Precision-7920-Tower:/data/lgs/yinyu/yacl$

满足requirements的安装要求,但是报错,编译不成功

Problems in Black box interconnection

(1) Are there any more explanations about “Black box interconnection transport layer protocol” than blackbox_service.proto?

(2) It seems that the difference between ChannelBrpcBlackBox and ChannelBrpc is: the former builds a channel via environment variables while the latter inputs information explicitly. What’s the main reason to write these two classes of channels instead of one?

(3) Many error codes are defined in blackbox_service_errorcode.h. But where are they set? For example, where is the code that sets “QueueFull”?

Hash algorithms

In yacl/crypto/base/hash/hash_interface.h, the following code means a recommendation of some hash algorithms.

enum class HashAlgorithm : int {
UNKNOWN,
// SHA-2 family of algorithms
SHA224 = 1,
SHA256 = 2,
SHA384 = 3,
SHA512 = 4,

SHA_1 = 5,

SM3 = 6,

BLAKE2B = 7,
BLAKE3 = 8
};

My first question is : Why not use SHA3 since SHA1 is not safe?

The second question comes from the following code in hmac.cc:

void Init_HMAC(HashAlgorithm hash_algo, ByteContainerView key,
HMAC_CTX* context) {
int res = 0;
switch (hash_algo) {
case HashAlgorithm::SHA224:
res =
HMAC_Init_ex(context, key.data(), key.size(), EVP_sha224(), nullptr);
break;
case HashAlgorithm::SHA256:
res =
HMAC_Init_ex(context, key.data(), key.size(), EVP_sha256(), nullptr);
break;
case HashAlgorithm::SHA384:
res =
HMAC_Init_ex(context, key.data(), key.size(), EVP_sha384(), nullptr);
break;
case HashAlgorithm::SHA512:
res =
HMAC_Init_ex(context, key.data(), key.size(), EVP_sha512(), nullptr);
break;
case HashAlgorithm::SHA_1:
res = HMAC_Init_ex(context, key.data(), key.size(), EVP_sha1(), nullptr);
break;
case HashAlgorithm::SM3:
res = HMAC_Init_ex(context, key.data(), key.size(), EVP_sm3(), nullptr);
break;
case HashAlgorithm::UNKNOWN:
default:
YACL_THROW("Unsupported hash algo: {}", static_cast(hash_algo));
break;
}

YACL_ENFORCE_EQ(res, 1, "Failed to HMAC_Init_ex.");
}

Why is BLAKE not written here?

Problems in yacl/link/context

(1) In yacl/link/context,
should Context.stats_ be a const std::shared_ptr?

(2) In

void SendAsync(size_t dst_rank, ByteContainerView value, std::string_view tag);

what does tag mean? It is not defined in secretflow’s interconnection protocol.

(3) If every party holds one context, then the following code implies that everyone should maintain the counter of every p2p-channel.

Context::Context(…)
…
for (size_t src = 0; src < world_size; ++src) {
    for (size_t dst = 0; dst < world_size; ++dst) {
      p2p_counter_[std::make_pair(src, dst)] = 0U;
    }
  }

调研:YACL 即将升级到 C++20 并放弃对老版本 C++ 的兼容

鉴于 C++20 带来了许多语言新特性,可以提高代码可读性,简化写法,提升 meta programing 能力,提升性能,加快编译速度等等,YACL/HEU/SPU/SCQL 计划从 C++17 全面迁移到 C++20。

目前 YACL 最低要求是 C++17,GCC > 11.2,升级后最低要求 C++20,GCC > 11.2

对使用者的影响

仅影响直接通用 C++ 调用 YACL 的用户,通过其它语言接口(Python SDK)的用户不受影响。

对于 C++ 使用者:YACL 升级 C++20 后直接依赖项目也需要在 20 模式下编译,其中 GCC 11.2 已经支持 C++20 语法,编译器不用变,需要做的是 GCC 编译参数把 -std=c++17 改为 -std=c++20

反馈

考虑到一些项目从 C++17 升到 20 后可能编译不过,如果您对升级 20 有什么疑问或需要技术支持,请留言;或者您确实必须依赖 C++17 版本,也请留言告诉我们

在 YACL 上支持新的 Codes: LDPC Codes

此 ISSUE 为 隐语开源共建计划(SecretFlow Open Source Contribution Plan,简称 SF OSCP)第三期任务 ISSUE,欢迎社区开发者参与共建~
若有感兴趣想要认领的任务,但还未报名,辛苦先完成报名进行哈~

任务介绍

  • 任务名称:在 YACL 上支持新的 Codes: LDPC Codes
  • 技术方向:YACL,密码学,编码理论
  • 任务难度:进阶🌟🌟

详细要求

由于目前 Silent-Extension 的 OTe 算法基于不同的 LPN 问题假设,目前 YACL 已实现基于 Primal LPN 的算法,因此需要探索、支持基于其他假设(例如 dual LPM)的算法,支持 LDPC Code 是实现该类算法的前置工作。

  • 安全性
    • 遵循论文中的协议实现
    • if possible,默认安全强度为 128 bit
  • 功能性:
    • 支持 Encode 函数
    • 支持相应参数调整
  • 代码规范:
  • C++ 代码需要遵循 Google C++ style guidelines,除了以下几条 exceptions
    • Exceptions are allowed and encouraged where appropriate.
    • Header guards should use #pragma once.
    • Adopt camelBack for function names.
    • Use fixed width integer types whenever possible.
    • Avoid using size_t on interface APIs.
  • 可使用 clangd 进行代码格式化
  • 可使用 cpplint 检查格式
  • 提交说明:关联该 issue 并提交代码至 https://github.com/secretflow/yacl/tree/main/yacl/crypto/tools

能力要求

  • 熟悉密码学或者编码理论
  • 熟悉 C++ 以及性能优化

操作说明

gcc version

In yacl/base/int128.h,

#if __cplusplus >= 202002L
#else
constexpr int128_t abs(int128_t x) { return x >= 0 ? x : -x; }
#endif

I test it using gcc 12.2.0 with __cplusplus = 201703. "int128_t abs(int128_t x)" is already written in stl.

The comment in ecc_spi.h

There are comments in ecc_spi.h as below:

// The h, cofactor.
// Cofactor is the number of non-overlapping subgroups of points, which
// together hold all curve points
virtual MPInt GetCofactor() const = 0;

Some websites also make similar mistakes of calling the non-overlapping subsets "subgroups". But the exact definition of "cofactor" is the number of cosets.
Equivalently, cofactor is the number of non-overlapping subsets of points. Of course, the subsets are divided according to a designated subgroup.

The definition can be checked in "Standards for Efficient Cryptography" and many books about elliptic curves.

Problems in yacl/math/mpint

(1)It seems that libtommath is keen on 64 bit platforms, defining MP_64BIT. But it may define MP_32BIT. So mp_digit is likely uint64_t. It is possible that mp_digit is uint32_t. MP_DIGIT_BIT may be 60, 31, 28 respectively.
In math/mpint/mp_int.cc,

template <>
void MPInt::Set(int64_t value) {
  MPINT_ENFORCE_OK(mp_grow(&n_, 2));
  mp_set_i64(&n_, value);
}

…

template <>
void MPInt::Set(uint128_t value) {
  MPINT_ENFORCE_OK(mp_grow(&n_, 3));
  mp_set_u128(&n_, value);
}


Should we judge whether the platform is 64 bit or 32 bit before we use mp_grow(&n_,int )? To be exact, judge the value of MP_DIGIT_BIT.

(2) Maybe the annotation has some errors.

* > RandomExactBits
*     - Generate an exact bit_size random number, the smb is not guaranteed

Should “smb” be “msb”?

In the annotation of SlowCustomPow, we usually write “base ** scalar” instead of “scalar ** base” as far as I know.

两方纵向woe任务中,brpc报method not found的错误

我在两方纵向woe任务中,我用两方的spu与自己一侧的grpc代理连接,然后两方代理之间通信,进而实现隔离网络下的通信问题。但是yacl中报错“Method not found”。grpc代理转发给spu的报文头信息是“content-type=application/grpc,accept=/,user-agent=brpc/1.0 curl/7.0”。这是什么原因了?

Problems in yacl/link/transport

You claim that the transport protocol is based on gRPC in www.secretflow.org.cn. But the code shows that it is based on bRPC. Why?

Another problem is the function
`
explicit ChannelBrpc(size_t self_rank, size_t peer_rank, Options options,
bool exit_if_async_error = true)
: ChannelBase(self_rank, peer_rank, exit_if_async_error),
options_(std::move(options)) {}

explicit ChannelBrpc(size_t self_rank, size_t peer_rank,
size_t recv_timeout_ms, Options options,
bool exit_if_async_error = true)
: ChannelBase(self_rank, peer_rank, recv_timeout_ms, exit_if_async_error),
options_(std::move(options)) {}
`
But ChannelBase does not have such interface.

Problems in yacl/link/transport

(1) You claim that the transport protocol is based on gRPC in www.secretflow.org.cn. But the code shows that it is based on bRPC. Why?

(2) Another problem is the function
`
explicit ChannelBrpc(size_t self_rank, size_t peer_rank, Options options,
bool exit_if_async_error = true)
: ChannelBase(self_rank, peer_rank, exit_if_async_error),
options_(std::move(options)) {}

explicit ChannelBrpc(size_t self_rank, size_t peer_rank,
size_t recv_timeout_ms, Options options,
bool exit_if_async_error = true)
: ChannelBase(self_rank, peer_rank, recv_timeout_ms, exit_if_async_error),
options_(std::move(options)) {}
`
But ChannelBase does not have such interface.

矩阵转置算法疑问

关于matrix_utils中的函数AvxTranspose128,我阅读了参考文献《An Extension of Eklundh's matrix transposition algorithm and its application in digital image processing》。但是里面的参数和Yacl实现中的参数完全对不上,请问代码中的iter,blockSizeShift等变量对应文献中哪些参数?或者实现不是参考这篇文献?

c++20 compilation error

In file included from external/com_github_fmtlib_fmt/include/fmt/format.h:48, from external/com_github_fmtlib_fmt/include/fmt/ostream.h:20, from external/yacl/yacl/crypto/base/mpint/mp_int.h:21, from external/yacl/yacl/crypto/base/ecc/ec_point.h:20, from external/yacl/yacl/crypto/base/ecc/ec_point.cc:15: external/com_github_fmtlib_fmt/include/fmt/core.h: In instantiation of 'constexpr fmt::v9::detail::value<Context> fmt::v9::detail::make_value(T&&) [with Context = fmt::v9::basic_format_context<fmt::v9::appender, char>; T = const yacl::crypto::MPInt&]': external/com_github_fmtlib_fmt/include/fmt/core.h:1777:29: required from 'constexpr fmt::v9::detail::value<Context> fmt::v9::detail::make_arg(T&&) [with bool IS_PACKED = true; Context = fmt::v9::basic_format_context<fmt::v9::appender, char>; fmt::v9::detail::type <anonymous> = fmt::v9::detail::type::custom_type; T = const yacl::crypto::MPInt&; typename std::enable_if<IS_PACKED, int>::type <anonymous> = 0]' external/com_github_fmtlib_fmt/include/fmt/core.h:1901:77: required from 'constexpr fmt::v9::format_arg_store<Context, Args>::format_arg_store(T&& ...) [with T = {const yacl::crypto::MPInt&, const yacl::crypto::MPInt&}; Context = fmt::v9::basic_format_context<fmt::v9::appender, char>; Args = {yacl::crypto::MPInt, yacl::crypto::MPInt}]' external/com_github_fmtlib_fmt/include/fmt/core.h:1918:31: required from 'constexpr fmt::v9::format_arg_store<Context, typename std::remove_cv<typename std::remove_reference<Args>::type>::type ...> fmt::v9::make_format_args(Args&& ...) [with Context = fmt::v9::basic_format_context<fmt::v9::appender, char>; Args = {const yacl::crypto::MPInt&, const yacl::crypto::MPInt&}]' external/com_github_fmtlib_fmt/include/fmt/core.h:3206:44: required from 'std::string fmt::v9::format(fmt::v9::format_string<T ...>, T&& ...) [with T = {const yacl::crypto::MPInt&, const yacl::crypto::MPInt&}; std::string = std::__cxx11::basic_string<char>; fmt::v9::format_string<T ...> = fmt::v9::basic_format_string<char, const yacl::crypto::MPInt&, const yacl::crypto::MPInt&>]' external/yacl/yacl/crypto/base/ecc/ec_point.cc:28:21: required from here external/com_github_fmtlib_fmt/include/fmt/core.h:1757:7: error: static assertion failed: Cannot format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api.html#udt

gcc version 11.3.0
-std=gnu++20

why not define namespace fmt { struct formatter<...>: ostream_formatter {};}

What is butil?

"yacl/io/kv/leveldb_kvstore.cc" uses a header from "butil". It is not a famous library. My questions are: What is "butil"? Where is "butil"?

The function log10 is strange

In yacl/base/int128.h, there is a strange sentence:
constexpr double log10(uint128_t x) { return 0.0; }
Why is it always 0? Or is it a bug?

在WSL:Ubuntu20.04下创建SPU设备出现问题

执行环境:WSL:Ubuntu20.04
版本:SecretFlow: 1.1.0b0
执行代码:spu_device = sf.SPU(aby3_config) (文档:教程:SPU基础)
打印日志:
(SPURuntime pid=19246) 2023-09-20 14:00:38.165 [info] [default_brpc_retry_policy.cc:DoRetry:52] socket error, sleep=1000000us and retry
(SPURuntime pid=19245) 2023-09-20 14:00:38.165 [info] [default_brpc_retry_policy.cc:DoRetry:52] socket error, sleep=1000000us and retry
(SPURuntime pid=19246) 2023-09-20 14:00:39.165 [info] [default_brpc_retry_policy.cc:LogHttpDetail:29] cntl ErrorCode '112', http status code '200', response header '', error msg '[E111]Fail to connect Socket{id=1 addr=127.0.0.1:56231} (0x0x4a73900): Connection refused [R1][E112]Not connected to 127.0.0.1:56231 yet, server_id=1'
(SPURuntime pid=19246) 2023-09-20 14:00:39.165 [info] [default_brpc_retry_policy.cc:DoRetry:75] aggressive retry, sleep=1000000us and retry
(SPURuntime pid=19245) 2023-09-20 14:00:39.165 [info] [default_brpc_retry_policy.cc:LogHttpDetail:29] cntl ErrorCode '112', http status code '200', response header '', error msg '[E111]Fail to connect Socket{id=1 addr=127.0.0.1:56231} (0x0x4cdfd00): Connection refused [R1][E112]Not connected to 127.0.0.1:56231 yet, server_id=1'
(SPURuntime pid=19245) 2023-09-20 14:00:39.165 [info] [default_brpc_retry_policy.cc:DoRetry:75] aggressive retry, sleep=1000000us and retry
(SPURuntime pid=19246) 2023-09-20 14:00:40.166 [info] [default_brpc_retry_policy.cc:LogHttpDetail:29] cntl ErrorCode '112', http status code '200', response header '', error msg '[E111]Fail to connect Socket{id=1 addr=127.0.0.1:56231} (0x0x4a73900): Connection refused [R1][E112]Not connected to 127.0.0.1:56231 yet, server_id=1 [R2][E112]Not connected to 127.0.0.1:56231 yet, server_id=1'
(SPURuntime pid=19246) 2023-09-20 14:00:40.166 [info] [default_brpc_retry_policy.cc:DoRetry:75] aggressive retry, sleep=1000000us and retry
(SPURuntime pid=19245) 2023-09-20 14:00:40.166 [info] [default_brpc_retry_policy.cc:LogHttpDetail:29] cntl ErrorCode '112', http status code '200', response header '', error msg '[E111]Fail to connect Socket{id=1 addr=127.0.0.1:56231} (0x0x4cdfd00): Connection refused [R1][E112]Not connected to 127.0.0.1:56231 yet, server_id=1 [R2][E112]Not connected to 127.0.0.1:56231 yet, server_id=1'
(SPURuntime pid=19245) 2023-09-20 14:00:40.166 [info] [default_brpc_retry_policy.cc:DoRetry:75] aggressive retry, sleep=1000000us and retry
(SPURuntime pid=19248) 2023-09-20 14:00:40.214 [info] [default_brpc_retry_policy.cc:DoRetry:69] not retry for reached rcp timeout, ErrorCode '1008', error msg '[E1008]Reached timeout=2000ms @127.0.0.1:34335'

不知道是具体哪块的问题,是WSL网络方面的问题还是其他情况,希望能解答一下,谢谢。

编译tpre_test时报错

g++ tpre_test.cc -I/home/sangge/yacl -o tpre_test

In file included from /home/sangge/yacl/yacl/math/mpint/mp_int.h:25,
from /home/sangge/yacl/yacl/crypto/base/ecc/ec_point.h:21,
from /home/sangge/yacl/yacl/crypto/primitives/tpre/capsule.h:23,
from /home/sangge/yacl/yacl/crypto/primitives/tpre/tpre.h:23,
from tpre_test.cc:15:
/home/sangge/yacl/yacl/base/int128.h:110:20: error: redefinition of ‘constexpr int128_t std::abs(int128_t)’
110 | constexpr int128_t abs(int128_t x) { return x >= 0 ? x : -x; }
| ^~~
In file included from /usr/include/c++/12/cstdlib:77,
from /usr/include/c++/12/ext/string_conversions.h:41,
from /usr/include/c++/12/bits/basic_string.h:3960,
from /usr/include/c++/12/string:53,
from /usr/include/c++/12/bits/locale_classes.h:40,
from /usr/include/c++/12/bits/ios_base.h:41,
from /usr/include/c++/12/ios:42,
from /usr/include/c++/12/ostream:38,
from /usr/include/c++/12/iostream:39,
from /home/sangge/yacl/yacl/crypto/primitives/tpre/tpre.h:17:
/usr/include/c++/12/bits/std_abs.h:85:3: note: ‘constexpr __int128 std::abs(__int128)’ previously defined here
85 | abs(__GLIBCXX_TYPE_INT_N_0 __x) { return __x >= 0 ? __x : -x; }
| ^~~
In file included from /home/sangge/yacl/yacl/crypto/base/ecc/any_ptr.h:22,
from /home/sangge/yacl/yacl/crypto/base/ecc/ec_point.h:20:
/home/sangge/yacl/yacl/math/mpint/mp_int.h: In member function ‘std::size_t std::hashyacl::math::MPInt::operator()(const yacl::math::MPInt&) const’:
/home/sangge/yacl/yacl/math/mpint/mp_int.h:392:22: error: ‘mp_hash’ was not declared in this scope; did you mean ‘hash’?
392 | MPINT_ENFORCE_OK(mp_hash(&x.n
, &h));
| ^~~~~~~
/home/sangge/yacl/yacl/base/exception.h:353:52: note: in definition of macro ‘YACL_ENFORCE_THAT_IMPL’
353 | ::yacl::enforce_detail::EnforceFailMessage r(condition);
| ^~~~~~~~~
/home/sangge/yacl/yacl/math/mpint/mp_int_enforce.h:26:3: note: in expansion of macro ‘YACL_ENFORCE_EQ’
26 | YACL_ENFORCE_EQ((MP_ERR), MP_OKAY, VA_ARGS)
| ^~~~~~~~~~~~~~~
/home/sangge/yacl/yacl/math/mpint/mp_int.h:392:5: note: in expansion of macro ‘MPINT_ENFORCE_OK’
392 | MPINT_ENFORCE_OK(mp_hash(&x.n
, &h));

Trivial case in compile_time_utils.h

compile_time_utils.h中有代码

template <>
constexpr uint32_t crc32<static_cast<size_t>(-1)>(
    [[maybe_unused]] const char* str) {
  return 0xFFFFFFFF;
}

这个特殊情况对应零多项式,应该返回0吧。

Memory management in channel

In yacl/link/transport/channel.cc, there is a function
'
void ChannelBase::SubmitSendTask(Message&& msg) {
auto btask = std::make_unique(this->shared_from_this(), std::move(msg));
bthread_t tid;
if (bthread_start_background(&tid, nullptr, SendTask::Proc, btask.get()) ==0) {
// bthread takes the ownership, release it.
static_cast(btask.release());
} else {
YACL_THROW("failed to push async sending job to bthread");
}
}
'

Note that this function creates a variable of “SendTask” type.
Will bRPC deconstruct the “SendTask” it receives?When will “SendTask” deconstruct?

An optimization in segment_tree.h

In yacl/utils/segment_tree.h, SegmentTree::GetSegments() gets a copy of its data member segments_. I think you can just use the copy constructor to implement the function instead of a for loop. For example,

std::vector<std::pair<T, T>> GetSegments() const {
std::vector<std::pair<T, T>> ret(segments_);
return ret;
}

Are there any reasons to use a for loop in SegmentTree::GetSegments()?

在 YACL 上支持基于 OT 的 Private Set Union 算法

此 ISSUE 为 隐语开源共建计划(SecretFlow Open Source Contribution Plan,简称 SF OSCP)第三期任务 ISSUE,欢迎社区开发者参与共建~
若有感兴趣想要认领的任务,但还未报名,辛苦先完成报名进行哈~

任务介绍

  • 任务名称:在 YACL 上支持基于 OT 的 Private Set Union 算法
  • 技术方向:YACL,密码学,PSU,OPRF
  • 任务难度:进阶🌟🌟

详细要求

  • 安全性:
    • 遵循论文中的协议实现
    • if possible,默认安全强度为 128 bit
  • 功能性:
    • 支持输入值为 uint128_t vector 的 PSU 基础计算
    • 支持相应参数调整
  • 代码规范:
  • C++ 代码需要遵循 Google C++ style guidelines,除了以下几条 exceptions
    • Exceptions are allowed and encouraged where appropriate.
    • Header guards should use #pragma once.
    • Adopt camelBack for function names.
    • Use fixed width integer types whenever possible.
    • Avoid using size_t on interface APIs.
  • 可使用 clangd 进行代码格式化
  • 可使用 cpplint 检查格式
  • 提交说明:关联该 issue 并提交代码至 https://github.com/secretflow/yacl/tree/main/yacl/crypto/primitives/sse

能力要求

  • 熟悉密码学以及隐私计算原语
  • 熟悉 C++ 以及性能优化

参考内容

OT调用问题

您好,我想使用这个库里的base ot实现两方之间的不经意传输,但是出现调用不成功问题。能否分享下如何单独调用ot,实现简单功能?

编译时报错:除以0

./yacl/base/dynamic_bitset.h:2294:20: error: division by zero is not a constant expression
2294 | constexpr size_t init_val_required_blocks = u128_bits_number / bits_per_block;

Problems in brpc_blackbox_link

In brpc_blackbox_link.cc line 197,

options.max_retry = options.max_retry;

I guess the right-hand-side should be options_. max_retry.

Moreover, in the function

brpc::ChannelOptions BrpcBlackBoxLink::GetChannelOption(const SSLOptions* ssl_opts)
...
auto retry_policy = std::make_unique<BlackboxRetryPolicy>( options_.retry_interval_ms, options_.aggressive_retry, push_wait_ms_);
...
retry_policy_ = std::move(retry_policy);

Why does yacl set the value of retry_policy_ in GetChannelOption? What's the relationship between retry_policy_ and GetChannelOption?

Do not need to reset in the constructor of Hmac

In yacl/crypto/base/hmac.cc, the constructor is

Hmac::Hmac(HashAlgorithm hash_algo, ByteContainerView key)
: hash_algo_(hash_algo),
key_(key.begin(), key.end()),
context_(CheckNotNull(HMAC_CTX_new())) {
Reset();
}

where Reset() is

Hmac& Hmac::Reset() {
YACL_ENFORCE_EQ(HMAC_CTX_reset(context_), 1);
Init_HMAC(hash_algo_, key_, context_);
return *this;
}

I don't think that we need to invoke Reset() in the constructor. My suggestion is that the following code is better:

Hmac::Hmac(HashAlgorithm hash_algo, ByteContainerView key)
: hash_algo_(hash_algo),
key_(key.begin(), key.end()),
context_(CheckNotNull(HMAC_CTX_new())) {
Init_HMAC(hash_algo_, key_, context_);
}

The reason is that HMAC_CTX_new() already invokes HMAC_CTX_reset().

在WSL:Ubuntu20.04下创建SPU设备出现问题

执行环境:WSL:Ubuntu20.04
版本:SecretFlow: 1.1.0b0
执行代码:spu_device = sf.SPU(aby3_config) (文档:教程:SPU基础)
打印日志:
(SPURuntime pid=19246) 2023-09-20 14:00:38.165 [info] [default_brpc_retry_policy.cc:DoRetry:52] socket error, sleep=1000000us and retry
(SPURuntime pid=19245) 2023-09-20 14:00:38.165 [info] [default_brpc_retry_policy.cc:DoRetry:52] socket error, sleep=1000000us and retry
(SPURuntime pid=19246) 2023-09-20 14:00:39.165 [info] [default_brpc_retry_policy.cc:LogHttpDetail:29] cntl ErrorCode '112', http status code '200', response header '', error msg '[E111]Fail to connect Socket{id=1 addr=127.0.0.1:56231} (0x0x4a73900): Connection refused [R1][E112]Not connected to 127.0.0.1:56231 yet, server_id=1'
(SPURuntime pid=19246) 2023-09-20 14:00:39.165 [info] [default_brpc_retry_policy.cc:DoRetry:75] aggressive retry, sleep=1000000us and retry
(SPURuntime pid=19245) 2023-09-20 14:00:39.165 [info] [default_brpc_retry_policy.cc:LogHttpDetail:29] cntl ErrorCode '112', http status code '200', response header '', error msg '[E111]Fail to connect Socket{id=1 addr=127.0.0.1:56231} (0x0x4cdfd00): Connection refused [R1][E112]Not connected to 127.0.0.1:56231 yet, server_id=1'
(SPURuntime pid=19245) 2023-09-20 14:00:39.165 [info] [default_brpc_retry_policy.cc:DoRetry:75] aggressive retry, sleep=1000000us and retry
(SPURuntime pid=19246) 2023-09-20 14:00:40.166 [info] [default_brpc_retry_policy.cc:LogHttpDetail:29] cntl ErrorCode '112', http status code '200', response header '', error msg '[E111]Fail to connect Socket{id=1 addr=127.0.0.1:56231} (0x0x4a73900): Connection refused [R1][E112]Not connected to 127.0.0.1:56231 yet, server_id=1 [R2][E112]Not connected to 127.0.0.1:56231 yet, server_id=1'
(SPURuntime pid=19246) 2023-09-20 14:00:40.166 [info] [default_brpc_retry_policy.cc:DoRetry:75] aggressive retry, sleep=1000000us and retry
(SPURuntime pid=19245) 2023-09-20 14:00:40.166 [info] [default_brpc_retry_policy.cc:LogHttpDetail:29] cntl ErrorCode '112', http status code '200', response header '', error msg '[E111]Fail to connect Socket{id=1 addr=127.0.0.1:56231} (0x0x4cdfd00): Connection refused [R1][E112]Not connected to 127.0.0.1:56231 yet, server_id=1 [R2][E112]Not connected to 127.0.0.1:56231 yet, server_id=1'
(SPURuntime pid=19245) 2023-09-20 14:00:40.166 [info] [default_brpc_retry_policy.cc:DoRetry:75] aggressive retry, sleep=1000000us and retry
(SPURuntime pid=19248) 2023-09-20 14:00:40.214 [info] [default_brpc_retry_policy.cc:DoRetry:69] not retry for reached rcp timeout, ErrorCode '1008', error msg '[E1008]Reached timeout=2000ms @127.0.0.1:34335'

不知道是具体哪块的问题,是WSL网络方面的问题还是其他情况,希望能解答一下,谢谢。

Problems in mcl_field

The problems are mainly about mcl_field in yacl/crypto/base/field.
(1)
class Field is defined in crypto/base/field/ field_spi.h. class GaloisField is defined in /math/galois_field/gf_spi.h. Why not combine the two definitions (classes) into one?
(2) In crypto/base/field/mcl/mcl_field.cc,

template <typename T_, size_t degree_>
std::string MclField<T_, degree_>::GetFieldName() const {
  return fmt::format("<MclField F_p^{}>", degree_,
                     Mpz2Mp(T_::BaseFp::getOp().mp));
}

Should the fmt sentence be fmt::format("<MclField F_{}^{}>", Mpz2Mp(T_::BaseFp::getOp().mp),degree_);?
(3)

template <typename T_, size_t degree_>
class MclField : public Field
…

Why does this template need “degree_”? I doubt if this parameter is provided by some interface of T_.
(4) In mcl_field.cc,

template <typename T_, size_t degree_>
FElement MclField<T_, degree_>::Rand() const {
  using BaseFp = typename T_::BaseFp;
  const auto per_size = (BaseFp::getOp().mp.getBitSize() + 7) / 8;

  auto ret = MakeShared<T_>();
  Buffer buf(per_size * degree_);
  BaseFp p;
  for (uint64_t i = 0; i < degree_; i++) {
    p.setByCSPRNG();
    p.serialize(buf.data<uint8_t>() + i * per_size, per_size);
  }

  CastAny<T_>(ret)->deserialize(buf.data<uint8_t>(), buf.size());
  return ret;
}

Can we use T_:: setByCSPRNG(); instead of T_:: BaseFp:: setByCSPRNG();?

(5) In mcl_field.cc,

#include "yacl/crypto/base/field/mcl/mcl_field.h"

#include "mcl/fp_tower.hpp"
#include "mcl/op.hpp"
#include "mcl_field.h"
…

Are the two files "mcl_field.h" the same?

macos m1 编译yacl报错

分支

main

macos

macos Ventura 13.2

clang -v

Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

clang++ -v

Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

gcc -v

Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

g++ -v

Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

bazel info

Starting local Bazel server and connecting to it...
bazel-bin: /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/execroot/yacl/bazel-out/darwin_arm64-fastbuild/bin
bazel-genfiles: /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/execroot/yacl/bazel-out/darwin_arm64-fastbuild/bin
bazel-testlogs: /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/execroot/yacl/bazel-out/darwin_arm64-fastbuild/testlogs
character-encoding: file.encoding = ISO-8859-1, defaultCharset = ISO-8859-1
command_log: /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/command.log
committed-heap-size: 322MB
execution_root: /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/execroot/yacl
gc-count: 6
gc-time: 18ms
install_base: /var/tmp/_bazel_dsy/install/dc2159ad4e237743745cc81e45bf4748
java-home: /private/var/tmp/_bazel_dsy/install/dc2159ad4e237743745cc81e45bf4748/embedded_tools/jdk
java-runtime: OpenJDK Runtime Environment (build 11.0.10+9-LTS) by Azul Systems, Inc.
java-vm: OpenJDK 64-Bit Server VM (build 11.0.10+9-LTS, mixed mode) by Azul Systems, Inc.
max-heap-size: 4294MB
output_base: /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c
output_path: /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/execroot/yacl/bazel-out
package_path: %workspace%
**release: release 5.4.1**
repository_cache: /var/tmp/_bazel_dsy/cache/repos/v1
server_log: /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/java.log.macbook-pro.dsy.log.java.20230424-232120.88617
server_pid: 88617
used-heap-size: 55MB
workspace: /Users/dsy/Desktop/yacl

编译命令

 bazel build //... -c opt

报错日志

DEBUG: Rule 'org_interconnection' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1669271462 +0800"
DEBUG: Repository org_interconnection instantiated at:
  /Users/dsy/Desktop/yacl/WORKSPACE:19:10: in <toplevel>
  /Users/dsy/Desktop/yacl/bazel/repositories.bzl:59:10: in yacl_deps
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/build_defs/repo/utils.bzl:233:18: in maybe
Repository rule git_repository defined at:
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
ERROR: /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/com_github_brpc_brpc/BUILD.bazel:228:13: in objc_library rule @com_github_brpc_brpc//:macos_lib: Expected action_config for 'objc++-compile' to be configured
ERROR: /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/com_github_brpc_brpc/BUILD.bazel:228:13: in objc_library rule @com_github_brpc_brpc//:macos_lib: 
Traceback (most recent call last):
        File "/virtual_builtins_bzl/common/objc/objc_library.bzl", line 127, column 124, in _objc_library_impl
        File "/virtual_builtins_bzl/common/objc/compilation_support.bzl", line 316, column 50, in _register_compile_and_archive_actions
        File "/virtual_builtins_bzl/common/objc/compilation_support.bzl", line 422, column 74, in _cc_compile_and_link
        File "/virtual_builtins_bzl/common/objc/compilation_support.bzl", line 193, column 29, in _compile
Error in compile: Expected action_config for 'objc++-compile' to be configured
ERROR: /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/com_github_brpc_brpc/BUILD.bazel:228:13: Analysis of target '@com_github_brpc_brpc//:macos_lib' failed
INFO: Repository remote_java_tools instantiated at:
  /DEFAULT.WORKSPACE.SUFFIX:392:6: in <toplevel>
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/build_defs/repo/utils.bzl:233:18: in maybe
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/build_defs/repo/http.bzl:355:31: in <toplevel>
INFO: Repository 'remote_java_tools' used the following cache hits instead of downloading the corresponding file.
 * Hash '2eede49b2d80135e0ea22180f63df26db2ed4b795c1c041b25cc653d6019fbec' for https://mirror.bazel.build/bazel_java_tools/releases/java/v11.7.1/java_tools-v11.7.1.zip
If the definition of 'remote_java_tools' was updated, verify that the hashes were also updated.
INFO: Repository com_github_dltcollab_sse2neon instantiated at:
  /Users/dsy/Desktop/yacl/WORKSPACE:19:10: in <toplevel>
  /Users/dsy/Desktop/yacl/bazel/repositories.bzl:39:35: in yacl_deps
  /Users/dsy/Desktop/yacl/bazel/repositories.bzl:321:10: in _com_github_dltcollab_sse2neon
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/build_defs/repo/utils.bzl:233:18: in maybe
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/build_defs/repo/http.bzl:355:31: in <toplevel>
INFO: Repository remotejdk11_macos_aarch64 instantiated at:
  /DEFAULT.WORKSPACE.SUFFIX:123:6: in <toplevel>
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/build_defs/repo/utils.bzl:233:18: in maybe
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/jdk/remote_java_repository.bzl:48:17: in remote_java_repository
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db19c/external/bazel_tools/tools/build_defs/repo/http.bzl:355:31: in <toplevel>
ERROR: Analysis of target '//yacl/link/transport:channel_brpc_test' failed; build aborted: 
INFO: Elapsed time: 9.120s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (109 packages loaded, 9059 targets configured)
    Fetching ...est_ot; Cloning f40a33a37e3ff8cd81655c35237c177e358dc5b1 of https://github.co\
m/secretflow/simplest-ot.git 7s
    Fetching ..._tools; Extracting /private/var/tmp/_bazel_dsy/4f9993a0cd77836fcc6fa6eaf86db1\
9c/external/remote_java_tools/temp241924901069986556/java_tools-v11.7.1.zip
    Fetching https://mirror.bazel.build/.../zulu11.50.19-ca-jdk11.0.12-macosx_aarch64.tar.gz

为什么把context的析构工作交给上层?

首先context使用默认析构函数,那么析构的关键在于Channel的析构。Channel的析构函数如下,

~Channel() override {
if (!send_thread_stopped_.load()) {
SPDLOG_WARN(
"Channel destructor is called before WaitLinkTaskFinish, try "
"stop send thread");
try {
WaitAsyncSendToFinish();
} catch (const std::exception& e) {
SPDLOG_ERROR("Stop send thread err {}", e.what());
if (exit_if_async_error_) {
exit(-1);
}
}
}
}

为什么Channel析构时不直接调用WaitLinkTaskFinish()?或者Context在析构时调用WaitLinkTaskFinish()?按照目前的用法,Context析构前还要调用WaitLinkTaskFinish(),逻辑上感觉有些冗余。

sm4_drbg的一些问题

(1) 在sm4_drbg中看到《软件随机数设计指南》- 征求意见稿,请问这个文件在哪里下载?
(2) sm4_drbg为什么可以不用Nonce?
(3) 在sm4_drbg的函数Sm4Drbg::Instantiate有一句std::memset(v_.data(), 0, key_.size());,虽然目前v_和key_是一样长,但是这样看起来写不太妥,可以考虑改成

std::memset(v_.data(), 0, v_.size());

TempFile in LeveldbKVStore

In yacl/io/kv/leveldb_kvstore.cc, there is a constructor as the following:
//
LeveldbKVStore::LeveldbKVStore(bool is_temp, const std::string &file_path)
: is_temp_(is_temp) {
leveldb::Options options;
options.create_if_missing = true;

std::string db_path = file_path;
if (db_path.empty()) {
butil::TempFile temp_file;
db_path = std::string(temp_file.fname());
}

leveldb::DB *db_ptr = nullptr;
leveldb::Status db_status = leveldb::DB::Open(options, db_path, &db_ptr);
YACL_ENFORCE(db_status.ok(), "leveldb open failed, msg: {}",
db_status.ToString());
db_.reset(db_ptr);
path_ = db_path;
is_open_ = true;
}
//

If file_path is empty, then temp_file creates a temporary file. Then its destructor unlinks the temporary file, causing the deletion of the file. Then leveldb::DB::Open can not succeed. What is the role that temp_file plays?

在YACL上支持恶意Beaver三元组和shuffle算法

In YACL, all existing algorithms are under the semi-honest model.

My question is: is it possible to implement a Beaver Triple and shuffling protocols in the malicious setting? We need those protocols as primitives to design a new protocol and use secretflow to test our protocol for benchmarking to publish academic papers.

The necessary knowledge that might be used in this issue includes cryptography, oblivious transfer, and secret shared shuffle.
And let's consider the computational security parameter is $\kappa=128$.

Some related works are provided as follows.
Ferret: Fast extension for correlated OT with small communication. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security.
Secret-shared shuffle. ASIACRYPT 2020

采用Context类实现两方同步传输

怎么在两方使用yacl/link/context文件中的Send Recv函数实现同步传输,需要多次传输二维的unsigned char数组数据,在这遇到了困难,希望作者给一些思路,谢谢

Bugs in yacl/kv/io/

Here is a problem in yacl/kv/io/. In LeveldbKVStore::~LeveldbKVStore(), observe the sentence
SPDLOG_INFO("Delete tmp file:{} exception {}", path_, e.what());
When there are more than one parameters, SPDLOG_INFO does not accept const char*.

And in LeveldbKVStore::Get, the sentence
SPDLOG_ERROR("Get key: {}, error:", key, db_status.ToString());
misses a "{}"

在 YACL 上支持新的对称可搜索加密算法

此 ISSUE 为 隐语开源共建计划(SecretFlow Open Source Contribution Plan,简称 SF OSCP)任务 ISSUE,欢迎社区开发者参与共建~
若有感兴趣想要认领的任务,但还未报名,辛苦先完成报名进行哈~

任务介绍

  • 任务名称:在 YACL 上支持新的对称可搜索加密算法
  • 技术方向:YACL,密码学,编码理论
  • 任务难度:进阶🌟🌟
  • 任务期望时间:4-5 周

详细要求

  • 安全性
    • 遵循论文中的协议实现
    • if possible,默认安全强度为 128 bit
  • 功能性:
    • 支持基本可对称搜索加密功能
    • 支持简单的 boolean query
  • 代码规范:
  • C++ 代码需要遵循 Google C++ style guidelines,除了以下几条 exceptions
    • Exceptions are allowed and encouraged where appropriate.
    • Header guards should use #pragma once.
    • Adopt camelBack for function names.
    • Use fixed width integer types whenever possible.
    • Avoid using size_t on interface APIs.
  • 可使用 clangd 进行代码格式化
  • 可使用 cpplint 检查格式
  • 提交说明:关联该 issue 并提交代码至 https://github.com/secretflow/yacl/tree/main/yacl/crypto/primitives/sse

能力要求

  • 熟悉密码学以及对称可搜索加密
  • 熟悉 C++ 以及性能优化

参考内容

在 YACL 上支持 Verifiable Shamir Secret Sharing

此 ISSUE 为 隐语开源共建计划(SecretFlow Open Source Contribution Plan,简称 SF OSCP)第二期任务 ISSUE,欢迎社区开发者参与共建~
若有感兴趣想要认领的任务,但还未报名,辛苦先完成报名进行哈~

任务介绍

  • 任务名称:在 YACL 上支持 Verifiable Shamir Secret Sharing
  • 技术方向:YACL,密码学,MPC
  • 任务难度:进阶🌟🌟

详细要求

  • 安全性:
    • 遵循论文中的协议实现
    • if possible,默认安全强度为 128 bit
  • 功能性:
    • 支持 Encode 函数
    • 支持相应参数调整
  • 代码规范:
  • C++ 代码需要遵循 Google C++ style guidelines,除了以下几条 exceptions
    • Exceptions are allowed and encouraged where appropriate.
    • Header guards should use #pragma once.
    • Adopt camelBack for function names.
    • Use fixed width integer types whenever possible.
    • Avoid using size_t on interface APIs.
  • 可使用 clangd 进行代码格式化
  • 可使用 cpplint 检查格式
  • 提交说明:关联该 issue 并提交代码至 https://github.com/secretflow/yacl/tree/main/yacl/crypto/primitives/shamir

能力要求

  • 熟悉密码学或者编码理论
  • 熟悉 C++ 以及性能优化

操作说明

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.