59 lines
1.9 KiB
Markdown
59 lines
1.9 KiB
Markdown
# Architecture
|
|
|
|
The current architecture is based on tokio.
|
|
In the future we might want something more general purpose (and no-std),
|
|
but good enough for now.
|
|
|
|
Tokio has its own thread pool, and we spawn one async loop per listening socket.
|
|
|
|
Then we spawn our own thread pool.
|
|
These threads are pinned to the cpu cores.
|
|
We spawn one async worker per thread, making sure it remains pinned to that core.
|
|
This is done to avoid *a lot* of locking and multithread syncronizations.
|
|
|
|
We do connection sharding on the connection id.
|
|
|
|
|
|
# Future
|
|
|
|
I randomly came across this can easily explains most problems with multithreaded
|
|
scaling and work queues:
|
|
https://tokio.rs/blog/2019-10-scheduler
|
|
|
|
What we want to do is:
|
|
* ebpf BBR packet pacing
|
|
* need to support non-ebpf BBR for mobile ios, too
|
|
* connection sharding
|
|
* directly in ebpf?
|
|
* based on connection id
|
|
* no cross-core talk
|
|
* implement connection id change
|
|
* any enpoint can ask the other to change the connection id
|
|
* once ACK is received, takes effect, old connection id dropped?
|
|
* lets us rebalance the work on cores, without synchronization
|
|
* can just use one work queue for core, without work stealing
|
|
|
|
## Problems
|
|
|
|
### How to integrate application thread pool with libFenrir?
|
|
|
|
The application/service will obviously have to do some work on its own
|
|
|
|
This means that it will have either:
|
|
* its own thread pool
|
|
* our same threadpool
|
|
|
|
Handling the connection on one thread, only to have the work done elsewhere
|
|
will incurr in cross-cpu talk, cache invalidations, synchronization techniques, etc
|
|
|
|
So we should find a way to either:
|
|
* use the same threadpool as the application
|
|
* sync work threads with the application threadpool
|
|
|
|
Do remember that all of this is not just Rust, but it will have a C interface,
|
|
which means that it will be used by other languages, too.
|
|
|
|
This alone probably means that we should run our own tokio runtime
|
|
|
|
|