52 lines
1.6 KiB
Markdown
52 lines
1.6 KiB
Markdown
|
# Architecture
|
||
|
|
|
||
|
|
For now we will keep things as easy as possible, not caring about the performance.
|
||
|
|
|
||
|
|
This means we will use only ::tokio and spawn one job per connection
|
||
|
|
|
||
|
|
|
||
|
|
# Future
|
||
|
|
|
||
|
|
I randomly came across this can easily explains most problems with multithreaded
|
||
|
|
scaling and work queues:
|
||
|
|
https://tokio.rs/blog/2019-10-scheduler
|
||
|
|
|
||
|
|
What we want to do is:
|
||
|
|
* one thread per core
|
||
|
|
* thread pinning
|
||
|
|
* ebpf BBR packet pacing
|
||
|
|
* need to support non-ebpf BBR for mobile ios, too
|
||
|
|
* connection sharding
|
||
|
|
* directly in ebpf?
|
||
|
|
* based on connection id
|
||
|
|
* no cross-core talk
|
||
|
|
* implement connection id change
|
||
|
|
* any enpoint can ask the other to change the connection id
|
||
|
|
* once ACK is received, takes effect, old connection id dropped?
|
||
|
|
* lets us rebalance the work on cores, without synchronization
|
||
|
|
* can just use one work queue for core, without work stealing
|
||
|
|
|
||
|
|
## Problems
|
||
|
|
|
||
|
|
### How to integrate application thread pool with libFenrir?
|
||
|
|
|
||
|
|
The application/service will obviously have to do some work on its own
|
||
|
|
|
||
|
|
This means that it will have either:
|
||
|
|
* its own thread pool
|
||
|
|
* our same threadpool
|
||
|
|
|
||
|
|
Handling the connection on one thread, only to have the work done elsewhere
|
||
|
|
will incurr in cross-cpu talk, cache invalidations, synchronization techniques, etc
|
||
|
|
|
||
|
|
So we should find a way to either:
|
||
|
|
* use the same threadpool as the application
|
||
|
|
* sync work threads with the application threadpool
|
||
|
|
|
||
|
|
Do remember that all of this is not just Rust, but it will have a C interface,
|
||
|
|
which means that it will be used by other languages, too.
|
||
|
|
|
||
|
|
This alone probably means that we should run our own tokio runtime
|
||
|
|
|
||
|
|
|