libFenrir/src/Architecture.md

Architecture

The current architecture is based on tokio. In the future we might want something more general purpose (and no-std), but good enough for now.

Tokio has its own thread pool, and we spawn one async loop per listening socket.

Then we spawn our own thread pool. These threads are pinned to the cpu cores. We spawn one async worker per thread, making sure it remains pinned to that core. This is done to avoid a lot of locking and multithread syncronizations.

We do connection sharding on the connection id.

Future

I randomly came across this can easily explains most problems with multithreaded scaling and work queues:
https://tokio.rs/blog/2019-10-scheduler

What we want to do is:

• ebpf BBR packet pacing
  • need to support non-ebpf BBR for mobile ios, too
• connection sharding
  • directly in ebpf?
  • based on connection id
  • no cross-core talk
• implement connection id change
  • any enpoint can ask the other to change the connection id
    • once ACK is received, takes effect, old connection id dropped?
  • lets us rebalance the work on cores, without synchronization
  • can just use one work queue for core, without work stealing

Problems

How to integrate application thread pool with libFenrir?

The application/service will obviously have to do some work on its own

This means that it will have either:

• its own thread pool
• our same threadpool

Handling the connection on one thread, only to have the work done elsewhere will incurr in cross-cpu talk, cache invalidations, synchronization techniques, etc

So we should find a way to either:

• use the same threadpool as the application
• sync work threads with the application threadpool

Do remember that all of this is not just Rust, but it will have a C interface, which means that it will be used by other languages, too.

This alone probably means that we should run our own tokio runtime