what advantage and disadvantage in mtcp and seastar about f-stack

[beacer]

Let me try to ask for mTCP part. I'm not f-stack staff, just have some idea for mTCP and f-stack.

1. To some extent, mTCP is not a fully TCP/IP stack, it has ARP/IP/Routing but need hard coding in config file. Actually the ARP/IP/Routing implementation is too simple to be a production ready stack (In my opinion). Further more, it's not support UDP/IGMP/IPv6/VLAN/VxLAN/Tunnel/Bonding/Tools/..., while some user cases may need them. All of above means workloads and bugs.
   So if we need a sophisticated stack, user mode FreeBSD/Linux is a better choices.
2. mTCP is really quick, but not as good as f-stack/FreeBSD.
   After integrated mTCP with Nginx, we found it's much faster than native Nginx. However, mTCP has dedicated thread for mTCP/DPDK (RunMainLoop) besides user application (like Nginx/Lighttpd). Two threads means lock for mTCP API and internal thread, and they shares CPU core. perf flame graph shows about 1/2 CPU time is consumed by mTCP thread while 1/2 left for app(Nginx). Yes, we can adjust the ration, but ...
   That may be the reason f-stack is faster.
3. To use mTCP, some effort is needed to integrated with Nginx, the original project is only integrated with lighttpd. Particularly，Nginx is multi-process model (master/workers) while mTCP is mult-thread model.

[jfb8856606]

F-Stack glued DPDK,FreeBSD TCP/IP stack,Posix API,Coroutine API and applications to a full user space development kit.

1. F-Stack and mTCP developed by pure C. Seastar developed by C++14,supported Future-Promise, Shared-nothing，message-passing etc.
2. All these projects do packet I/O via DPDK. mTCP modified the DPDK source code.
3. F-Stack ported FreeBSD TCP/IP stack to user space, the FreeBSD TCP/IP stack is complex(compared with Seastar or mTCP), advantageous, functional, stable，and there are many tools can be used，such as sysctl, ifconfig, netstat etc. Seastar's native stack and mTCP's user-level stack developed according to the RFC，they are simple and efficient. However, there are many problems in the WAN environment.
4. F-Stack and mTCP's API are posix-like, the existing applications can be ported easily. When use Seastar, you need to rewrite your applications with C++14.
5. F-Stack and Seastar supported coroutine API, you don't need to process complex asynchronous events. mTCP does not have a similar interface.
6. Seastar has a memory manage system(like tcmalloc), supported zero copy when using hugepages, and user space I/O scheduler via libaio. F-Stack and mTCP have not supported that.
7. They all feature high performance, especially the performance of short TCP connection(see CPS test, Connections Per Second). The performance of L4/L7 applications is in the same order of magnitude.

[gpstrive]

1. mTCP is optimized for short tcp flows， which is not suitable for persistent connection
2. mTCP is not robust and buggy（we have tested it with dpdk）
3. f-stack is multi-process model and you can't run multiple different apps at the same time

[whl739]

@tigerjibo

2. What the detail next development plan?

See our roadmap.
What we have done:

NIC offload: checkksum(IP/TCP/UDP/SCTP), TSO, VLAN).
User space tools: sysctl, ifconfig.

3. Is there any operating system limitation?

We support Linux. And we only tested on the kernel version 3.10+, we didn't test on 2.6.

[tigerjibo]

Thanks for all reply!
But What about are the other questions?
2. What the detail next development plan?
3. Is there any operating system limitation?

[tigerjibo]

@gpstrive. For now mtcp has some buggy, but I think mtcp is still the excellent project based on DPDK.
This is very simple and support the virtual net interface for controlling and configure the data plane netcard. It is very nice to develop the particular business,which is not the full stack.

[tigerjibo]

@whl739 OK, thx!

[isyes]

@jfb8856606 I think seastar do not support co-routine API, seastar thread is cooperative, I think not the same with co-routine.