Xing's Weekly Tech Reports

  In the past week I've been working on mainly one thing: migrating our existing repositories from bitbucket to github.   Git is the first source control tool I used and has become dominant over the past few years. To be honest I'm surprised to see my project uses mercurial.    The task of migration is fairly simple: fast-export will do the heavy lifting for you. Here 's an article on how to do it. The reason why it took me a lot of time is: my new project has baked a lot of secrets / keys inside source code and now it's going to be more public than only accessible by my group.   After I started digging into these secrets, I've discovered more issues:  - configuration management and secret management are done with the same configuration file  - configuration and source code is not a perfect 1:1 map. With a lot of copy-paste, many sections of configuration are not even being used in code  - deployment is a mess. some repos have dedicated (...

For the past week I've been working on backing up S3 bucket data. A lot of the things I've heard of multiple times but I haven't personally done before. So last week was quite a lot of learning on fundamentals of AWS. Started off by setting up data synchronization with AWS CLI, everything seems straightforward except for one undocumented thing, which surfaces with an seemingly somewhat unrelated error message. I've spent a lot of time on that and eventually figured out exactly why it happened. And given I wasn't able to find any related answer for that, I've written it down to help people who run into the same issue, which can be found here: http://xingdu.blogspot.com/2017/10/aws-s3-transferring-data-across-accounts.html The rest of the week was spent on playing assessing options to drive the synchronize call. Played a bit with AWS Lambda and really liked the idea of being "serverless": no need to worry about deployment, no need to worry about sc...

  First 3 weeks at new job. A lot of time spent on on-boarding and didn't get much achieved. Also I got interrupted pretty often by ripples from relocation.    Good news is these random interruptions are coming to an end. I've renewed a bunch of documents which are expiring at around the same time and settled down at my new apartment with all service activated. I can't really see any big interruption in the coming few months.   At work, I spent ~3 days just to set up developing environment due to legacy code base as well as the fact that this is a very small & stable team. The last engineer they hired was a few years ago.    Knowing that there should be more engineering joining the team soon, I spent another 3 days going through all the engineer on-boarding process to simplify things.    The document used for engineer development environment set up has been reduced from 12 pages to 6, and I've created a setup shell script for each proj...

  Backfilling those weeks from mid October 2017.   I accepted an job offer in early August and sent my resignation to my team. It's been a great 4yr run, and I enjoyed every minute staying at work, especially in the past 15months since I started working on infrastructure.    It's a good fit for my skillset and I'm very interested in getting outside my comfort zone to learn new stuff.   During these last 7 weeks I was not given much work, but I delivered a lot of things until my last day at work.   I was supposed to do knowledge transfer but given my expertise in our server & infrastructure, it's a better use of my time to make a few last bits of contributions.   It's kinda hard to recall exactly everything I did in these 7 weeks and I'll list the things I can remember and am proud of.   A lot of time were actually spent on getting prepared for the new gig: it does require relocation so I need to deal with packing & mov...