Week 10/29/2017 - 11/12/2017 Continuation: Repository migration & Deployment process improvement
Past 2 weeks I spent most of my time continue doing the repository migration and deployment process improvement.
Old deployment process:
- not fully automated and require manual process
- deployment key / SSH based
- remote server has history to source
- a lot of dead / unnecessary steps / scripts involved
- duplicated scripts with poor reusability
I spend a good amount of time walking through all the services we need to deploy and studied all the steps involved (documented once and undocumented once). With a good understanding of what needs to be done, I started stripping off dead script and replacing old scripts with new ones: data driven and no duplication. In addition I created new scripts to cover all to the services that need to be deployed so that deployment can be 100% automated.
The only big issue I ran into is not technical: the corporate github (enterprise) setting has SSH disabled, allowing only HTTPS, and to get around that I need to submit a request to the team who manages that and ask for server instance ip white-listing. IP based white-listing is not going to work for me if I need to auto-scale my cluster (unless I use elastic IP and choose to destroy before creating new instance). So I've chosen to use a dedicated deploying user (read-only permission), and use the OAuth token for deployment. Unfortunately this needs to go through a request to the same team as well so right now the automated deployment pipeline is using an OAuth token of my own account.
With all of these problems solved (at least prepared solution), I started the tedious process of migrating repos one at a time:
- convert mercurial to git
- insert sub-module for deployment
- create new jenkins jobs that uses new deployment scripts
- deploy to dev and test
- deploy to prod and test, if anything goes wrong, run deployment with old repo & old job immediately.
- move on to next repo.
Thank god this is close to completion. Major repos have all been migrated (13? 15?) and I'll have to do this a few more times for the remaining minor repos next week. And after that I'll be moving on to doing system level work!
Old deployment process:
- not fully automated and require manual process
- deployment key / SSH based
- remote server has history to source
- a lot of dead / unnecessary steps / scripts involved
- duplicated scripts with poor reusability
I spend a good amount of time walking through all the services we need to deploy and studied all the steps involved (documented once and undocumented once). With a good understanding of what needs to be done, I started stripping off dead script and replacing old scripts with new ones: data driven and no duplication. In addition I created new scripts to cover all to the services that need to be deployed so that deployment can be 100% automated.
The only big issue I ran into is not technical: the corporate github (enterprise) setting has SSH disabled, allowing only HTTPS, and to get around that I need to submit a request to the team who manages that and ask for server instance ip white-listing. IP based white-listing is not going to work for me if I need to auto-scale my cluster (unless I use elastic IP and choose to destroy before creating new instance). So I've chosen to use a dedicated deploying user (read-only permission), and use the OAuth token for deployment. Unfortunately this needs to go through a request to the same team as well so right now the automated deployment pipeline is using an OAuth token of my own account.
With all of these problems solved (at least prepared solution), I started the tedious process of migrating repos one at a time:
- convert mercurial to git
- insert sub-module for deployment
- create new jenkins jobs that uses new deployment scripts
- deploy to dev and test
- deploy to prod and test, if anything goes wrong, run deployment with old repo & old job immediately.
- move on to next repo.
Thank god this is close to completion. Major repos have all been migrated (13? 15?) and I'll have to do this a few more times for the remaining minor repos next week. And after that I'll be moving on to doing system level work!
Comments
Post a Comment