Easy network code in C# – [Part 1]

So you want to add some sort of networking functionality to your C# application?
Then this little blog-post is for you!
It doesn’t even matter if you’d consider yourself a noob because I’m about to make network code real easy for you.

Whenever you need performance (maybe you’re making a game or some other real-time thing) you almost definitely want a binary network protocol.
That means HTTP and all the other dumb text-based “web” stuff is out! 😛

What we need is an easy way to directly send binary representations of our objects TCP or UDP.

What do we want exactly?

Let’s say we have an object like this and want to send it over internet:

class Person
{
	public string Name;
	public int Age;
	public List<Person> Friends = new List<Person>();
}

Starting with the final usage (because that’s where we want to end up), what is the most comfortable way you can imagine to send/receive that over the internet?
Maybe something like this:

Client Server
var person = new Person { Name = "Bob" };

net.Send(person);
object obj = await net.ReceiveObject();

if(obj is Person p)
	Log($"Received Person '{p.Name}'");

Is that all???

Well the usage looks nice.
But what if whatever solution we come up with is slow? That’d be a real downer.

So in order for our network code to be truly considered compfortable there are some more requirements:

  • Easy to integrate
    I don’t want to annotate my classes with any attributes or spend much time preparing my previously written code.

  • Efficient!
    Whatever gets sent over the wire should be as small as possible!
    For example sending type-names (as strings) is a no go, no wasted bandwidth.

  • Robust
    I want to be able to throw pretty much any object into the network and have it arrive at the other side.
    No artificial limitations.

  • Reference loops and polymorphism
    Absolutely basic features, but very rarely supported unfortunately.

    Quick explanation what those two things are (Click to expand)
    • A reference loop is super simple: It’s when an object (A) references another object (B), which then references object A again.
      Example: Two ‘Person’ objects that have each other has a Friend.
      Why is that a problem for many serializers?
      Because they don’t track what objects they’ve already serialized, so you get a StackOverflowException

    • Polymorphism is simple as well.
      A scary sounding word for a very simple concept, it basically just means “inheritance”.
      One example would be: object obj = 5;, the variable is of type object and the actual type inside is int.
      Another example of an actual scenario would be: You have an interface ISpell { ... } (or an abstract base-class) and some classes that implement it.
      If you now want to send a List<ISpell> over the network many system will have trouble with that (but they really shouldn’t!).
      So basically whenever the variable-type isn’t exactly equal to the object-type.

Now that’s what I’d deem a “compfortable networking solution“.

How to do it?

In the super-comfy approach we imagined we send and receive objects.
But the basic network primitives in .NET (Socket / NetworkStream) want bytes, not objects.

It turns out networking is (for the most part) actually a serialization problem.

There are a lot of serializers out there already for various scenarios.
And not only that, most of them have active communities, and a huge amount of work went into making them!

Alright, so we’ll just take some existing serializer. Problem solved! Right?

Spoiler alert: No, obviously not.

(Click to expand)
If you want to know why I think no existing serializer can help us, click here.

Existing serializers

  • JSON
    Whenever my data is small and I want to modify it by hand in a text editor JSON is my first choice.
    Small settings files for desktop applications are a perfect fit for example (but there are many more).
    The JSON format is (obviously) text-based, so naturally its output is very large (compared to a binary encoding), which immediately disqualifies it from our little networking adventure.
    JSON is awesome, but it’s the completely wrong tool for the job here.

  • MsgPack
    There is a pretty well known C# library (called MessagePack-CSharp) that implements the ‘MsgPack’-format.
    It’s a binary format, turning objects into byte[], and the name even suggests that it was specifically made for “packing messages”.
    That sounds exactly like what we want!
    However there are some huge down-sides that you only encounter once you try something beyond some very simple scenarios…
    What are those downsides?

    • No support at all for “reference loops”.
      What is a reference loop? Sounds like something rare, but it’s actually extremely common.
      A reference loop is when you can find a “path” from one object back to itself in some way.
      For example if you have two Persons that have each other as a friend.
      Or even more simple: An object that simply contains a reference to itself.
      If you have something like that, you’ll get a nice StackOverflowException.
    • Barely any support for inheritance (aka polymorphism).
      If you plan to use MsgPack then your List<object> won’t make.
      There are some cases where you can add some annotations to your classes but in most cases you’re out of luck.
      MsgPack sounds promissing initially, but just those two limitations alone already become a total roadblock for most projects.
      Unless your objects are stupidly simple it just won’t work.
      And then there are a lot more problems as well:
    • Having to manually annotate every single class and field with attributes
    • Encoding is sub-optimal in some cases (resulting in larger than necceary binary size)
    • Version tolerance is fully manual (wtf). You have to manually manage some internal IDs whenever you add/remove/rename a field in one of your classes…
      Doing these changes even makes the binary bigger because it somehow results in “holes” in the binary.
  • BinaryFormatter
    BinaryFormatter is part of the .NET standard libraries, so anyone can use it.
    Feature-wise it’s mostly complete, but slow, incredibly slow actually.
    It’s output is binary, but it encodes so much extra data that it is entirely unusable for any networking purposes.
    I don’t think I’d ever advise anyone to use it. Even the text-based JSON would be the better choice in many cases.
    So downsides:

    • Huge output
    • Incredibly slow serialization and deserialization
    • Pretty much ZERO control over how things get serialized
      We’re starting to scrape the bottom of the barrel here…
  • Protobuf / Flatbuffers
    Those two were initially invented by Google (not for C#, but people have made some libraries).
    Extremely tedious to use, so pretty much the anti-thesis of the “comfortable” we’re looking for.
    Maybe we could forgive that, but they also have no support for reference loops, polymorphism.
    Serializing/Deserializing those two formats is fast, but that’s the only thing they got going for them.

What do we want from a serializer that’d work for us?

  • It must somehow embed the type of the object, but preferably not as a string because that’s too large.
    In situations where the object type is the same as the field type, the serializer should actually not write any type information at all!

    • Actually it would be nice if we could give it a list of types that it can expect, and it would somehow use that information to do things even more efficiently
  • Would be cool if it would “learn” from the data we give it and optimize itself.

  • In order to reduce the amount of setup we have to do ourselves the serializer should assume some things already, but it must also give us an easy way to change things if we want to!

  • It definitely must support reference loops and polymorphism.

  • When we send an object (A) to the other side, and then later we send another object (B) which has a reference to the previously sent object, then the serializer on the receiving side should be able to somehow make that work. Either doing it automatically, or giving us a way to easily link up the objects ourselves. If we had the choice between both options that’d be even better.

  • It would be cool if there would be some sort of compatibility mode, so two different versions of our program could still communicate with each other.

All of that together seems to be a pretty hefty list of requirements…

Ceras

I was searching for a serializer that does those things for years.
After all there is no reason it couldn’t be done in theory, it’s just that nobody seemed to have invested the time to actually do it!

While I was searching for that holy-grail of serializers, I either had to abuse JSON a bit or write some manual serialization code (yea it’s as boring as it sounds).
Then recently I worked on a project where none of those old workarounds would help – writing a better serializer was literally the only way.

Thus Ceras (https://github.com/rikimaru0345/Ceras) was created!
An open-source binary serializer with so many features that I literally had to move the feature-list to a separate page!

Important!
I’m not saying Ceras is better than other serializers in general! It is better at certain tasks (just like any other serializer is tailored to some specific tasks). All the serializers I listed have their place, and it’s up to you to decided when to use what.

In the most simple words: Just because there’s a new awesome hammer, doesn’t mean you don’t need a screwdriver anymore!
Right tool for the job and all that…

How does it look like in action? How does Ceras help? (Demo Project)

I made an example project, and for the impatient here’s a little screenshot so you know what you can expect:

Ceras example code

As you can imagine posting code on a blog only works well for small snippets.
Also what’s the point if you can’t run it yourself, mess around with it, etc… 😛

> Link to Example Project

In ‘Part 2‘ of this post we’ll explore how the demo works, what we can do with it, and more importantly what we can’t do with it and how to successfully solve those problems using Ceras.

Then, in an upcoming post I’m working on, we’ll see how Ceras can help in an entirely different scenario (splitting objects).

To use Ceras in your own project:
1. Install the package from NuGet
2. Instantiate it var ceras = new CerasSerializer();
3. And then just serialize objects: var bytes = ceras.Serialize(person);

Easy network code in C# – [Part 1]

5 thoughts on “Easy network code in C# – [Part 1]

  1. That’s something I was trying to find for years without luck 🙂
    This is veeeery promising, I can’t wait to put my hands on it. I will give you feedback once I’m done with testing 🙂

    P.S. what about the size of the bytes[] created based on an object, comparing to e.g. json serialization?

    1. Hey, thanks for the nice comment! 🙂

      > what about the size of the bytes[] created based on an object, comparing to e.g. json serialization?

      Since Ceras is a binary serializer the resulting byte array is **much** smaller than a Json string.

      Json is a text-based format that has a very different intention/goal: having output that is easily readable and editable by humans in a text editor. (It’s only that Json is commonly abused for situations where it isn’t the best choice).

      Also, if you have any feedback feel free to join the official discord server here:
      https://discord.gg/FGaCX4c

      I really appreciate all feedback (especially if something doesn’t work, so I can fix it :P)

      1. Hello Riki,

        Thanks for the answer 🙂

        That’s good that it’s much smaller, I was asking about the size because like ordinary .NET serializers I suppose you save some metadata about the types, inheritance etc. so it is more information to serialize. .NET binary serializer saves a lot of data and the size of the (in my case) files was too big. I think it was even bigger than just plain json files (in which there is additional metadata).

        All in all, it’s really great you’ve started this project, I will take a closer look soon 🙂

  2. Is it possible to send an object and wait for a response? (yeah I probable should RTFM.. I know, but I’d rather ask to let you know someone else is about to use this. I love the send part! just need to recieve back (I don’t want to make each client that sends also a server…

    1. Hi, sorry for answering so late, I rarely check the comments here.
      The client doesn’t have to be a server to receive data back, you can just call net.ReceiveObject() on the client as well to receive any objects the server has sent.

      The client already receives any messages here: https://github.com/rikimaru0345/Ceras/blob/master/samples/NetworkExample/Client.cs#L77-L79

      But maybe that’s not what you meant?
      Anyway, feel free to join the discord server (link on the github page) to discuss it further as comments are really slow 😛

Leave a Reply

Your email address will not be published.