Coder Social home page Coder Social logo

buckyos / cyfs Goto Github PK

View Code? Open in Web Editor NEW
2.0K 2.0K 275.0 13.23 MB

CYFS is the next-generation technology to build real Web3 by upgrading the basic protocol of Web (TCP/IP+DNS+HTTP),is short for CYberFileSystem. https://www.cyfs.com/, cyfs://cyfs/index_en.html.

Home Page: https://www.cyfs.com/

License: BSD 2-Clause "Simplified" License

JavaScript 0.27% Rust 99.53% C 0.01% Makefile 0.01% Python 0.02% Shell 0.04% RenderScript 0.01% Objective-C 0.04% Java 0.08%
blockchain cryptography named-data-networking named-object-networking ndn non p2p protocol rust web3

cyfs's People

Contributors

alexsunxl avatar dependabot[bot] avatar glen0125 avatar jing-git avatar jinquantianxia avatar lurenpluto avatar photosssa avatar song0125 avatar streetycat avatar tracy101 avatar waterflier avatar weiqiushi avatar winevery avatar zwong91 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cyfs's Issues

Windows bash environment cross compile cyfs_raptorq error

After compiling the linux version of the cyfs project using windows bash environment , and then using windows cmd environment to compile the cyfs project, an error will appear in the compilation of the cyfs_raptorq module

error[E0786]: found invalid metadata files for crate `cyfs_raptorq`
  --> CYFS\src\component\cyfs-bdt\src\ndn\chunk\encode\raptor.rs:10:5
   |
10 | use cyfs_raptorq::{
   |     ^^^^^^^^^^^^
   |
   = note: invalid metadata version found: \\?  CYFS\src\target\release\deps\libcyfs_raptorq.rlib

After deleting the "target" directory, windows cmd compile pass

Optimized compilation speed

Speed up the compilation of ood service

channel.rs moved from cyfs-base to new project cyfs-version
Move all logging-related code from cyfs-debug to the new project cyfs-log
keep the old dead-related code in cyfs-debug

This way, every time you change the version number, channel, etc. and recompile, you don't need to compile cyfs-base, cyfs-bdt, cyfs-lib and other libraries that take a long time, but only the final binary library, which can greatly speed up the compilation speed.


加快ood service的编译速度

channel.rs从cyfs-base里移出到新建工程cyfs-version
所有log相关的代码从cyfs-debug里移出到新建工程cyfs-log
cyfs-debug里保留原有的check相关代码

这样,每次修改版本号重编译的时候,就不需要再编cyfs-base, cyfs-bdt, cyfs-lib等需要很长时间的库了,而只需要编译最终的二进制库,可以大大加快编译速度

gateway unwrap

Problem
[2022-09-08 12:42:08.567990 +00:00] INFO [component/cyfs-debug/src/panic/panic.rs:32] thread 'async-std/runtime' panicked at 'called Option::unwrap() on a None value': component/cyfs-stack/src/interface/http_ws_listener.rs:88

To Reproduce
vood batch and concurrent op_env

Expected behavior
gateway work well

System information
Intel Xeon Processor (Skylake, IBRS) 2 Core
RAM: 4GB
OS: Ubuntu20.4

Additional context

[2022-09-08 12:42:08.567524 +00:00] INFO [component/cyfs-debug/src/panic/panic.rs:31] thread 'async-std/runtime' panicked at 'called Option::unwrap() on a None value': component/cyfs-stack/src/interface/http_ws_listener.rs:88
0: 0x0000564ec338d390
1: 0x0000564ec337b690
2: 0x0000564ec31ea950
3: 0x0000564ec3926000
4: 0x0000564ec3925f30
5: 0x0000564ec3924710
6: 0x0000564ec3925cd0
7: 0x0000564ec1dfa110
8: 0x0000564ec1df9fc0
9: 0x0000564ec23da310
10: 0x0000564ec2e52ed0
11: 0x0000564ec2e97c80
12: 0x0000564ec2e4d2c0
13: 0x0000564ec2ed9a30
14: 0x0000564ec38ba180
15: 0x0000564ec38babc0
16: 0x0000564ec38b88f0
17: 0x0000564ec38af430
18: 0x0000564ec38ae980
19: 0x0000564ec38b02d0
20: 0x0000564ec392b840
21: 0x00007f152b671530
22: 0x00007f152b44112d
23: 0x0000000000000000
[2022-09-08 12:42:08.567990 +00:00] INFO [component/cyfs-debug/src/panic/panic.rs:32] thread 'async-std/runtime' panicked at 'called Option::unwrap() on a None value': component/cyfs-stack/src/interface/http_ws_listener.rs:88
0: cyfs_debug::panic::manager::PanicManager::start::{{closure}}
1: std::panicking::rust_panic_with_hook
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:702:17
2: std::panicking::begin_panic_handler::{{closure}}
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:586:13
3: std::sys_common::backtrace::__rust_end_short_backtrace
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys_common/backtrace.rs:138:18
4: rust_begin_unwind
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:584:5
5: core::panicking::panic_fmt
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:142:14
6: core::panicking::panic
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:48:5
7: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
8: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
9: std::thread::local::LocalKey::with
10: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
11: async_task::raw::RawTask<F,T,S>::run
12: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
13: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
14: async_io::driver::block_on
15: async_global_executor::threading::thread_main_loop
16: std::sys_common::backtrace::__rust_begin_short_backtrace
17: core::ops::function::FnOnce::call_once{{vtable.shim}}
18: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/alloc/src/boxed.rs:1951:9
<alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/alloc/src/boxed.rs:1951:9
std::sys::unix::thread::Thread::new::thread_start
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys/unix/thread.rs:108:17
19: start_thread
20: clone

文檔建議

強迫症提示:建議所有關鍵字統一規範大小寫書寫風格,中文和英文字符串之間增加空格來達到更有好的可讀性,典型的案例如:

舊:订阅基于VPS实现的立即可用的VirtualOOD, VirtualOOD是门槛最低开通速度最快的OOD方式,相信会是大部分普通用户第一次的选择。

新:订阅基于 VPS 实现的立即可用的 Virtual OOD,Virtual OOD 是门槛最低开通速度最快的 OOD 方式,相信会是大部分普通用户第一次的选择。

另外,很多操作步驟沒有命令行實例,如:身份生成后,复制<save_path>/ood.desc和<save_path>/ood.sec两个文件到OOD机器的${cyfs_root}/etc/desc下,并重命名为device.desc和device.sec,這裡可以考慮用 cp 命令或者 cp 後再使用 mv 命令的完整提示。

App-Manager support for OOD standby mode

In the standby mode of OOD, App-Manager should not start the app service, and should not modify the root-state (because it is read-only mode), and when the OOD working mode is switched, all app services need to be stopped

A more concise solution is that when the mode is switched, the OOD-daemon is responsible for stopping the App-Manager, and the App-Manager needs to stop all currently running app services under the --stop command.

Hello CYFS Guide Feedback

Describe
This issue is to discuss the Hello CYFS Guide. If you have any issues with or feedback on the document, please leave a comment here.

Note: this is for feedback on the document, not on CYFS or on the CYFS changes.

Thank you for your efforts!

Stream send data content is incorrect

Describe the bug

(1)LN connect RN second success
(2)LN use stream send 1M data to RN
(3)RN resp 1M data to LN,
LN recv data content is incorrect, the content [0..8] is remark data length ,RN send [0..8] length is 1048576 ,LN recv length is 5941175098377794146
To Reproduce

    pub async fn send_file(&mut self, size: u64) -> Result<HashValue, BuckyError> {
        let mut hashs = Vec::<HashValue>::new();
        let mut send_buffer = Vec::new();
        send_buffer.resize(PIECE_SIZE, 0u8);
        let mut gen_count = PIECE_SIZE;
        let mut size_need_to_send = size + 8;
        if gen_count as u64 > size_need_to_send {
            gen_count = size_need_to_send as usize;
        }
        send_buffer[0..8].copy_from_slice(&size_need_to_send.to_be_bytes());
        Self::random_data(send_buffer[8..].as_mut());

        loop {
            let hash = hash_data(&send_buffer[0..gen_count]);
            hashs.push(hash);

            let _ = self.stream.write_all(&send_buffer[0..gen_count]).await.map_err(|e| {
                log::error!("send file failed, e={}",&e);
                e
            });
            size_need_to_send -= gen_count as u64;

            if size_need_to_send == 0 {
                break;
            }

            gen_count = PIECE_SIZE;
            if gen_count as u64 > size_need_to_send {
                gen_count = size_need_to_send as usize;
            }
            Self::random_data(send_buffer[0..].as_mut());
        }

        let mut total_hash = Vec::new();
        for h in hashs.iter() {
            total_hash.extend_from_slice(h.as_slice());
        }
        let hash = hash_data(total_hash.as_slice());

        log::info!("send file finish, hash={:?}", &hash);

        Ok(hash)
    }
pub async fn recv_file(&mut self) -> Result<(u64, HashValue), BuckyError> {
        let mut hashs = Vec::<HashValue>::new();
        let mut recv_buffer = Vec::new();
        recv_buffer.resize(PIECE_SIZE, 0u8);
        let mut piece_recv: usize = 0;
        let mut file_size: u64 = 0;
        let mut total_recv: u64 = 0;
        loop {
            let len = self.stream.read(recv_buffer[piece_recv..].as_mut()).await.map_err(|e| {
                log::error!("recv failed, e={}", &e);
                e
            })?;
            if len == 0 {
                log::error!("remote close");
                return Err(BuckyError::new(BuckyErrorCode::ConnectionReset, "remote close"));
            }
            piece_recv += len;
            total_recv += len as u64;

            if file_size == 0 {
                if piece_recv < FILE_SIZE_LEN {
                    continue;
                }
                let mut b = [0u8; FILE_SIZE_LEN];
                b.copy_from_slice(&recv_buffer[0..FILE_SIZE_LEN]);
                file_size = u64::from_be_bytes(b);
                log::info!("=====================================file_size={}", file_size);
            }

            if file_size > 0 {
                if total_recv == file_size || piece_recv == PIECE_SIZE {
                    let recv_hash = hash_data(&recv_buffer[0..piece_recv].as_ref());
                    hashs.push(recv_hash);
                }

                if total_recv == file_size {
                    log::info!("=====================================recv finish");
                    break;
                }
            }

            if piece_recv == PIECE_SIZE {
                piece_recv = 0;
            }
        }

        let mut total_hash = Vec::new();
        for h in hashs.iter() {
            total_hash.extend_from_slice(h.as_slice());
        }
        let hash = hash_data(total_hash.as_slice());
        log::info!("recv file finish, hash={:?}", &hash);
        Ok((file_size, hash))
    }

Device add bdt_version field

For the sake of bdt protocol version compatibility, an additional field needs to be added to indicate the bdt protocol version of the protocol stack used by the current device.

Device body content needs to add a field as follows:
bdt_version: Option<u8>

Compatibility between old and new binary data layers needs to be considered

danger for `import-people`

Exporting identities to the developer's computer is dangerous, and we may need a tool that authorizes publishing; exporting identities is only an option that is convenient for the development phase.

NDN Cache Node support in BDT

interest package routing semantic

  • foward interest package to another device;
  • reply interest with another source;
  • reply interest with waiting;

scheduling upload tasks

  • resource usage statistics for upload tasks
  • resource quota for upload tasks, when reaches, reply waiting to all incoming interests

service proof of traffic for cache node

  • embed service proof fields in NDN packages

SN Online Timeout

Describe the bug
cyfs stack wait sn online 30s timeout
run open bdt satck 166498 times success , timeout 237 times
To Reproduce

let stack = Stack::open(
            local, 
            key, 
            params).await?;

        let acceptor = stack.stream_manager().listen(0).unwrap();

        let peer_impl = unsafe {
            &mut *(Arc::as_ptr(&self.0) as *mut PeerImpl)
        };
        peer_impl.lazy_components = Some(LazyComponents {
            stack, 
            acceptor
        });
let begin_time = system_time_to_bucky_time(&std::time::SystemTime::now());
                    let mut local = peer.get_stack().local();
                    log::info!("on create succ, local: {}", local.desc().device_id());
                    let mut result = 0 ;
                    if sn.len()>0{
                        log::info!("peer sn list len={},wait for sn online",sn.len());
                        let result = match future::timeout(Duration::from_secs(30), peer.get_stack().net_manager().listener().wait_online()).await {
                            Err(_) => {
                                log::error!("sn online timeout");
                                1000
                            },
                            Ok(_) => {
                                log::info!("sn online success");
                                0
                            }
                        };
                    }
                    let online_time = system_time_to_bucky_time(&std::time::SystemTime::now()) - begin_time;

brief information in object-id

Now to save any information, we need to construct a special object, which is too complicated when the amount of information is small. We can design an object, and the information is directly encapsulated in "Object-Id", such as: number, short byte stream, short string etc.

BDT packet version campatible

  • Version field in handshake packets;
  • Version property on tunnel;
  • Semantic session implemention differs from versioned tunnel;

Window ood-installer 重新启动导致的gateway.toml被覆盖的bug

Describe the bug

CYFSOOD-x86-64-1.0.0.505-nightly安装包。WIN10专业版,安装配置完成后,再次重新启动机器,不能正确启动服务导致CYFS_browser中,info不能正确的显示OOD的信息。同时会每隔30秒启动一次服务(应该是cyfs-drive-server.exe)
To Reproduce

1,确保本机没有安装mogodb
2,下载 diy-ood-window版CYFSOOD-x86-64-1.0.0.505-nightly安装包,安装
3,使用https://youtrack.buckycloud.com/articles/SHARES-A-2/CYFS开发环境搭建 (注意host修改)中提供的 cyfs身份创建工具 中的,gateway.toml,替换 c:\cyfs\etc\gateway\gateway.toml 这个文件。此文件中,数据库配置为sqlite
4,关联账号
5,打开browser,可以看到ood的info信息是正确的
6,关机重新启动,ood-installer 又被自动执行。
7,c:\cyfs\etc\gateway\gateway.toml 这个文件被重新覆盖,其中的数据库配置又被修改为mongodb。
8,启动browser,不能正确显示ood 信息,同时会不停的启动某个服务(一闪而过)。

Expected behavior

再次重启启动,运行ood-installer ,不应该覆盖被修改的配置文件
System information

ftp 服务上下载 的CYFSOOD-x86-64-1.0.0.505-nightly安装包

OS:win10

Additional context

OOD-daemon supports read-only mode of gateway service

OOD-daemon needs to deal with the ood working mode of the gateway, mainly in the case of readonly in the current standby-ood mode, the app-manager service needs to be stopped, including the monitoring mechanism of dynamic switching, etc.

NDN get_data improvements and optimizations for local data access

  1. Integrate the existing get_data support with ChunkManager, ndc and tracker, which is consistent with the achievements of the BDT layer
  2. Optimize the reading mechanism of local file chunks, improve the performance of delayed reading of multiple blocks and the reading part with a range

[Congestion Control] The performance problem of BDT TCP file transmission in high latency and high packet loss

【Problem Description】
Multinational network Shanghai, China DCFS <-> US VOOD BDT only uses TCP protocol and the transmission fails
Network environment: 20-30% packet loss, 150ms delay, 30MB bandwidth, BDT transmission rate of 20-50KB/s, there is a small probability of transmission failure

【Modify system configuration optimization plan】

Linux kernel configuration to enable TCP congestion control can definitely improve performance
vim /etc/sysctl.conf

net.core.default_qdisc = fq 
net.ipv4.tcp_congestion_control = bbr

【Lab Verification】
0% packet loss 0ms delay bandwidth 30MB speed 3MB/s
0% packet loss 150ms delay bandwidth 30MB speed 1MB/s
20% packet loss 150ms delay bandwidth 30MB speed 500KB/s

NOC/NON/ACL redesign and refactoring

Mainly include the following two core changes

  1. Refactor the underlying structure and implementation of NOC, introduce permission design and GC mechanism support, etc.
  2. Refactor the NON interface to work with the new NOC and authority mechanism

BDT support scale out for NDN request

  • create download session for data piece returned from a channel which is different from where interest is sent;
  • add a from field to Interest packet: if interest is transmit from one device to another device, this field points to orign device where interest if sent from;

Improve Perf module

Improve Perf module

Goal

  • Multi report target, Default local_cache of the local cyfs stack
  • Use local cache to store stat information
  • Easy to use
  • Start using the Perf module inside OOD service and CYFS stack

Super frame datagram

  • support super frame datagram: send/receive datagrams beyound UDP MTU;
  • support plaintext datagram;

GlobalState op-env adds new methods support

On the basis of the existing interface, the following new interfaces and features are added

update

Unlike commit, update can be called multiple times. If something changes internally, it will try to commit and update the global state, returning the latest root
update does not affect the holding status of op-env on lock, so the lock on the specified path can continue to be valid (until commit or abort)

get_current_root

Get the root state of the current op-env, SingleOpEnv and PathOpEnv will return the corresponding values

SingleOpEnv -> root object

PathOpEnv -> (root, revision, global_root)

It should be noted that for PathOpEnv, calling get_current_root will make the op-env immediately bind the corresponding global-state snapshot (but lock will not)

reset

For SingleOpEnv, reset the current iterator

NDN refactor for new ACL and load logic

Loading logic of dir+inner-path

dir is the same as objectmap, the loading is changed to the full local protocol stack, that is, dir+inner_path is loaded from the target protocol stack, and the subdir, chunk and file that a dir depends on must be completely in the same protocol stack, and the internal nodes are no longer support complicated search across protocol stacks (consistent with objectmap loading and operation behavior)

NDN

ndn loads data, supports three types

  • chunk_id
  • file_id
  • dir_id+inner-path

The new ACL needs to match the following fields

  • referer_object
  • req_path

Several key designs

  • The minimum granularity of permission control is the file_id/dir_id corresponding to the referer_object, or the chunk_id/file_id/dir_id in the direct request (if the referer_object is empty), finer granularity is not supported
  • A request supports at most one referer_object to specify the association with the request target; or if no referer_object is specified, multiple referer_objects are no longer supported
  • Bridge target_object_id and req_path through referer_object to determine the permission of target_object_id, so the permission determination is basically the following two modes
    • referer_object + target_object_id + req_path triple
    • target_object_id + req_path 2-tuple

for chunk-id

Load a chunk data directly, check the permissions as follows

  1. referer_object is empty
    Then use req_path+chunk_id directly for verification, that is, require chunk_id to be linked to root_state
  2. referer_object = file_id
    First check if chunk_id exists in file.chunklist
    Then use req_path+file_id for verification, that is, the referenced file_id is required to be linked to root_state
  3. referer_object = dir_id + inner_path
    First get the corresponding file_id/dir_id through dir_id + inner_path
    Check if chunk_id is in file.chunklist or embedded chunk in dir
    Then use req_path+file_id or req_path + dir_id for verification, that is, the referenced file_id/dir_id is required to be linked to root_state

for file_id

Load a file data directly, verify the permissions as follows

  1. referer_object is empty
    Then use req_path+file_id directly for verification, that is, file_id is required to be linked to root_state
  2. referer_object = dir_id + inner_path
    First get the corresponding file_id through dir_id + inner_path
    Check if chunk_id is in file.chunklist
    Then use req_path + dir_id for verification, that is, the referenced dir_id is required to be linked to root_state and meet the permission requirements

For dir_id+inner_path

Load a file corresponding to the internal path of dir, and the permission check is as follows
The referer_object parameter is not accepted in this mode, and the dir_id itself is used directly
Use req_path + dir_id for verification, that is, the referenced dir_id is required to be linked to root_state and meet the permission requirements

CYFS DSG Cache Service

DSG Contract support cache service:

  • Cache storage fields;
  • Global cache storage option for cache node;

Prepare cached chunk list:

  • DSG cache aggregation objects: traffic proof, chunk traffic;
  • Update chunk list;

Proxy node service proof

implement service proof to support a OOD running as proxy node, to help devices behind NAT which failed to play hold-punching, or speed up link between devices

Refactor cyfs-stack front-end support for protocol layer

The purpose of the task
In order to better and systematically improve the runtime layer's support for the protocol, solve some leftover problems and inconsistent errors in the past historical evolution, and better support the o/r/a protocol

The core changes are as follows

  • Add the Front module, deconstruct the existing translator processing mechanism, and uniformly handle the o/r/a protocol
  • The non/ndn/global-state processing mechanism of cyfs-stack strips the support for runtime protocol layers such as mode and format, and the function is more explicit and pure

Spend time
About one to two weeks, including testing

Standby ood failed to sync chunk data

Describe the bug

the standby ood just sync the object_map ,not sync chunk data
To Reproduce

(1) use SharedCyfsStack ndn_service put a chunk to master ood
(2)master ood's root_state add a ObjectMap of chunkId
(3) waitting for standby ood sync data complete from master OOD
(4) check standby ood can use SharedCyfsStack ndn_service get the chunk
(5)the standby ood root_state revision is sync ,but get chunk data failed, ndn get_chunk from local but not found!
Err(err: (4, ndn get_chunk from local but not found! id=7EYaFPU2ry4McmHUJtxz1iDKEyPfkSaoDz7AFuxiDFJv, undefined))
Expected behavior

(1) the standby ood root_state revision is sync , the standby ood should get chunk from NOC

Improve the event mechanism of zone role switching

At present, zone role switching is a built-in event of role manager, which can only be used in the same process of CyfsStack. It needs to be switched to rely on the standard RouterEvent mechanism to provide cross-process access mode. For example, ood-daemon/app-manager will also pay attention to this event.

StateStorage implementation

Add the implementation of the basic data structure based on ObjectMap to realize the support of data storage and access such as FriendList

Add additional sync support to DirObject

Due to the particularity and complexity of DirObject, there are multiple layers of Object nesting and Chunk interleaving. Therefore, the synchronization mechanism needs to be considered additionally and corresponding implementations must be added.

SharedCyfsStack.util().build_dir_from_object_map parameter input req_path will cause the request to fail

Describe the bug

build_dir_from_object_map filling in req_path will cause the request to fail

To Reproduce

let dir_obj_resp1 = await stack.util().build_dir_from_object_map({
common: {
req_path: "qaTest",
target : stack.local_device_id().object_id,
dec_id: ZoneSimulator.APPID,
flags: 0,
},
object_map_id: get_ret1.unwrap().object.object_id,
dir_type: cyfs.BuildDirType.Zip,
});

result:

Error: err: (4, request not handled: method=POST, path=/util/build_dir_from_object_map/qaTest, undefined)
let dir_obj_resp1 = await stack.util().build_dir_from_object_map({
common: {
req_path: "",
target : stack.local_device_id().object_id,
dec_id: ZoneSimulator.APPID,
flags: 0,
},
object_map_id: get_ret1.unwrap().object.object_id,
dir_type: cyfs.BuildDirType.Zip,
});

result:

{"ok":true,"err":false,"val":{"file_id":"95RvaS5aUeMomNs3LBR89YwU6b3Aj7Tm5aQJPsJfH8SC"}}

Expected behavior
req_path is an optional parameter in UtilOutputRequestCommon, it should be able to be filled in successfully

Need ood-daemon-stop.sh on DIYOOD

The purpose of the feature
I want stop the ood service and backup the /cyfs/data/, now I killed service process manually

Describe the solutions to the feature you'd like
I need start/stop scripts~

Add mechanism to clear handlers based on dec

Due to the persistence mechanism of the handler, after dec modifies the handler, if you forget to remove the old handler, there will be residues that will affect the handler mechanism. So consider adding an interface for clearing the handler based on dec granularity. After dec starts or when necessary, you can call the clearing history version once.

The parameters of A link should be maintained after transform.

Describe the bug
The parameters will be removed when I open an A link html file.

To Reproduce
For example, I will open the following link:
cyfs://a/9tGpLNna8UVtPYCfV1LbRN2Bqa5G9vRBKhDhZiWjd7wA/index.html?hello=world
once I open it, the link where transform to like this :
http://127.0.0.1:38090/o/5hLXAcNtoFKLkvPEmdMFDJ9NjpCWJxdSZU6uKXPkR6m9/95RvaS5cWCTCKejHojmHtauJTQwL7FZ1AMeLLba1Dk1w/index.html?dec_id=9tGpLNna8UVtPYCfV1LbRN2Bqa5G9vRBKhDhZiWjd7wA
which means I've lost my parameter.

Expected behavior
In fact, the result should be like this:
http://127.0.0.1:38090/o/5hLXAcNtoFKLkvPEmdMFDJ9NjpCWJxdSZU6uKXPkR6m9/95RvaS5cWCTCKejHojmHtauJTQwL7FZ1AMeLLba1Dk1w/index.html?dec_id=9tGpLNna8UVtPYCfV1LbRN2Bqa5G9vRBKhDhZiWjd7wA?hello=world

Add list method to op-env

Now op-env supports iterator mode, which has better support for the data structure of large mode, but for ObjectMap in most cases, it is more convenient and efficient to use list directly

NDN Transfer Group basis

NDN Transfer Group is designed for named data transfer sessions, which may devices request some same named datas at same time; each transfer group could have different strategy for requested named data feature differs, BDT has to design transfer group basis to support making these strategies:

  • more fields on existing NDN package(e.g. interest package), or new defined NDN package to support transfer group member communication;
  • design basic roles in transfer group: tracker, hub, seeder etc.
  • design basic actions in transfer group member.

Datagram broadcasting

  • enable DHT routing map on BDT stack;
  • implement datagram broadcasting with DHT;
  • filter broadcating with area code in device object;

BDT-Stream Direct failure between BDT LAN devices

【Test Node EP】
PC_0015_0 L4tcp192.168.1.145:38549
PC_0016_0 L4tcp192.168.1.142:45531
【Steps】

  1. PC_0015_0, PC_0016_0 do not use SN
  2. PC_0015_0 -> PC_0016_0 ,remote_desc contain EP L4tcp192.168.1.142:45531
  3. PC_0016_0 -> PC_0015_0 , remote_desc contain EP L4tcp192.168.1.145:38549
    【expected outcome】
  4. The connection is successful
    【actual results】
    1.The connection is failure . if not SN,Direct just can use WAN address

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.