Xing's Weekly Tech Reports

As part of the effort for re-writing data synchronizer in golang, I started looking at existing memcached libraries in golang. The best one I found is this: https://github.com/rainycape/memcache Compared to the most popular library:  https://github.com/bradfitz/gomemcache It utilizes memcached binary protocol to achieve extreme performance. The benchmark result can be found in readme file. I was not aware of the binary protocol for memcached, although I do use it on a daily basis. So I looked it up in the release notes and turns out this is supported as early as 1.4.0 (released in 2009) Going through our tech stack where memcached is used, none of the accessors are using binary protocol at all. A seems-like-easy win is to simply replace command based protocol to binary protocol for all components of the game. Started looking into that immediately and ran into a big blocker. Php code base uses Memcache extension instead of Memcached extesnion, and only the latter suppo...

For those of you who don't know, terraform is a tool to manage infrastructure with code: https://www.terraform.io/ It's a revolutional idea for server ops people. On a high level, your code (in this case, tf files) is a 1-to-1 representation of your infrastructure. Any change needed for the infrastructure will be simply translated into a change in code. After you apply the change, bam, your infrastructure is updated. I'm not a pro on terraform yet but here are something I read through out the week to get a good understanding of the tool and provide me with enough knowledge to play with it:  - A comprehensive guid to terraform https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca  - Guid on official website https://www.terraform.io/intro/getting-started/install.html Since we have an existing infrastructure running on AWS, we need to migrate those to be managed by terraform. A small project I finished this week is to migrate one of our dev...

We rolled out some fixes on hardening our auth funnel, rejecting a lot of bots from our game. As retaliation, these bots starting scanning our endpoints and hitting all of them with a lot of request trying to bring the server down. Most of our endpoints were robust enough to handle that, but they managed to find one which is not robust enough, and caused me a lot of pain. Thanks to our investment in instrumentation & monitoring, we caught the issue quickly and identified the root cause being bot attaching that end point. Rolled out 2 steps fix: 1. a bandaid to stop bleeding. basically a simply "if" check to match the particular id bots are using and kill the request right away. 2. adding a new rate limitor based on the client id that's provided in the request. The rest of the week was spend on doing some simple coding test with golang.  Golang has always been on my radar and I haven't got a chance to use it after finish reading the book. So I decided...

Spend the week improve the instrumentation of the server.  The java based socket server hosts room based MMO game and we did not have transaction flow instrumentation in place before. A simple image would explain how amazing the result is. transaction flows by response over time The tool used is datadog, which is a one-stop shop for instrumenting & monitoring infrastructure as well as application.  As described by the image, multiple flows are covered in this dashboard (title partially wiped out). Light blue stands for number of requests received, light purple for successful responses and dark blue for failed ones. Good news is for the majority of the flows, we're responding with successful responses. Bad news is for a particular flow, the number of responses does not match the number of requests, meaning the handling of the some requests are interrupted due to exceptions. Also something else surfaced after I visualized the flows: Unnatural traf...