Notes.md

Scope

There are numerous things that we come across while dealing with networking at such a level as Crust. This file is meant to document all such learnings/findings which are not so apparant. They can be about tricky differences in behaviours on various platforms that we see either in the libraries we use (e.g. Mio) or in sockets and so on. Otherwise we might lose/forget such findings and commit same errors in future while coding something new or refactoring existing code. Some of these may be due to the behaviour at the time of writing. In such a case pls update this if the behaviour has changed and the point is no longer valid.

Details

• While obtaining a TcpListener from net2::TcpBuilder via listen() API, keep the listener backlog at 100.
  • The value is chosen with this as reference. For e.g. keeping the value as 1, we see that during stress potentially valid connections get dropped in the background while the code is processing an accept() from a peer. Since backlog of 1 will imply keeping exactly one ongoing TCP 3-way handshake all the rest will be refused in the meantime untill we finish processing currently accepted stream and call accept() again. This will inturn lead to frequent false events to Routing about failed peers and lead to unnecessary tunneling.
  • After setting it to 100, the readiness handler for a readable() event will be fired only once in PollOpt::edge() (even if there are more than 1 connection in queue which have completed the 3-way handshake) until an accept() has been performed and then a new connection is detected. So either loop and accept() to empty the queue (what the code currently does) or reregister as readable() after each accept() (slower).
• Peer A and B are connected. If A closes connection gracefully, B gets a 0 byte read denoting EOF during blocking reads on Linux and OS X. However on Windows, the blocking read return with an error instead of a 0-byte read.
• mio::Evented (e.g. a socket) cannot be re-registered with a different token without first deregistering it on Windows - it will lead to an error. Linux and OS X allow this behaviour.
• When mio::TcpListener fires readable() event and before we do an accept on it the peer client closes the socket, we still get a Ok(Some(mio::TcpStream)) on calling accept (not None or Error). However any operation on it will result in detection of hup(). This is the behaviour on Linux. Behaviour on other platforms are yet to be found out. Please update this point when found out.
• Do not consider the address obtained by consulting a STUN as being nat_restricted (as was done previously). If the port is manually forwarded and socket is bound to it, then it is not nat-restricted. Since this cannot be determined by code at runtime, entire notion of nat_restricted has been done away with.