libFenrir/src/Architecture.md

Architecture

For now we will keep things as easy as possible, not caring about the performance.

This means we will use only ::tokio and spawn one job per connection

Future

I randomly came across this can easily explains most problems with multithreaded scaling and work queues:
https://tokio.rs/blog/2019-10-scheduler

What we want to do is:

• one thread per core
• thread pinning
• ebpf BBR packet pacing
  • need to support non-ebpf BBR for mobile ios, too
• connection sharding
  • directly in ebpf?
  • based on connection id
  • no cross-core talk
• implement connection id change
  • any enpoint can ask the other to change the connection id
    • once ACK is received, takes effect, old connection id dropped?
  • lets us rebalance the work on cores, without synchronization
  • can just use one work queue for core, without work stealing

Problems

How to integrate application thread pool with libFenrir?

The application/service will obviously have to do some work on its own

This means that it will have either:

• its own thread pool
• our same threadpool

Handling the connection on one thread, only to have the work done elsewhere will incurr in cross-cpu talk, cache invalidations, synchronization techniques, etc

So we should find a way to either:

• use the same threadpool as the application
• sync work threads with the application threadpool

Do remember that all of this is not just Rust, but it will have a C interface, which means that it will be used by other languages, too.

This alone probably means that we should run our own tokio runtime