samedi 29 juillet 2017

System design for a client server architecture

I am trying to design and implement a solution for the following problem I am facing in one of my projects.

There are n (say 30) clients that send me data points of the form {timestamp, object}, where ‘timestamp’ comes in strictly ascending order.

I (server) need to listen for them and process the aggregated data from all clients for each time stamp in the following cases:

  1. I hear from all 30 clients for a single timestamp.
  2. I time out waiting to hear from a subset of the clients. This time out starts after I receive data for a time stamp for the first time.
  3. A subset of clients send data of timestamp higher than the one I need to process for (in this case I should process before the time out for whatever data I aggregated).

Is there a better way than maintaining bitmap of 30 clients for each timestamp, using background threads to continuously check the bitmaps and receive messages from forked listener process if higher timestamp is seen for any client? I would like something that is fast because the amount of data to be received in one hour is around 200 GB.

Any suggestions are appreciated, I don't have much experience in C++ systems programming. Can ZeroMQ or Apache Kafka be used here? I don't know much about any of them, I am willing to learn and implement; I just need some guidance. Please comment if you need any more details about anything, I apologise for missing anything.

Aucun commentaire:

Enregistrer un commentaire