Conclusion - CI/CD Series

In this series, we have covered how to implement CI/CD (continuous integration, continuous deployment) using DevOps (development and operations). We have shown how to go from a simple change in the code, to adding tests, building an assembly, planning the deployment, and eventually apply the changes to our live environment:

1. Introduction
   
2. Base Application
   
3. Unit Tests
   
4. Assembly
   
5. PATs
   
6. Plan
   
7. Apply

Future enhancements to this pipeline could include:

• Separate DEV and PROD environments
• DEV deploy on pushes to master
• PROD deploy on specific tag/release cycles
• Performance tests for the REST service
• Ability to destroy the resources using the Terraform destroy command
• Run PATs against service deployed in the DEV environment
• Currently, PATs are only run against the service running in-memory

Introduction - CI/CD Series

Over the last several years, the terms CI/CD (continuous integration, continuous deployment) and DevOps (development and operations) have grown to be known throughout most of the software industry. The implication of these practices is that no longer must software wait for a "release ritual" to push a new version to production. Nor must software be developed by one team and then "handed over" to a production support team. 

Instead, now a development team manages the software from end-to-end:

• Requirements
• Code changes
• Testing
• Build and package artifacts
• Deploy to production
• Support

Also, software has seen a much quicker release cycles. It is even possible to have pipelines that perform releases and deployments upon every committed code change.

Throughout the next few posts, we will build a fully automated CI/CD pipeline for a simple REST service. The series will be broken down into individual posts:

1. Introduction (this post)
2. Base application
3. CI for testing
4. Dockerizing the application
5. Product acceptance testing (PATs)
6. CD for deployment

The tools we will be working with throughout this project will be:

• sbt
  
• akka-http
  
• ScalaTest
  
• GitHub
  
• Docker
  
• EDIT 07/07/2020: Docker ended up not being used as Herkou was able to deploy the application from the source directly.
• Cucumber
  
• Terraform
  
• Heroku
  

SCGet Status Update

Recently both Josh and I have been busy with soundcloud-get (scget) and it has been coming along quite nicely. We feel confident that it is currently at a stable place and are looking into how to release a python project and tag it as version 1.0.

The current command line options that scget supports are:

• -u <song_url>- Downloads a song from the specified song url. This parameter can be provided multiple times.
• -a <artist_url> - Downloads all songs from the specified artist url.
• -o <folder> - Folder to download all songs to. If not provided or doesn't exist, all songs are downloaded to the current directory.
• -m - If provided, downloads are multi-threaded.
• -st <track_name> - Searches for tracks that are similar to the provided name and prints their titles.
• -sa <artist_name> - Searches for artists that are similar to the provided name and prints their names.
• -l <number> - Limits the number of results pulled back from either the search track or search artist queries. If not provided, each search defaults to 20 results.

Once we figure out how to release this project we do have a few ideas of what still needs to come for this project:

• Ways for the user to choose songs/artists to download from the provided search results.
• Being able to provide a yaml configuration instead of having so many command line arguments.
• Add a default download directory structure instead of just the current directory.
• Thread pooling for multi-threaded downloads to better manage the many download threads.
• Track down Unicoder Coercion errors that happen on occasion.

Though, for now it is all working fine and as expected for almost all cases which is why we want to release it now and then tackle these other issues.

Remember you can always watch the progress of this project on our github page.

SCGet

I currently started to work on a new side project, scget. It's a command line tool that will allow users to easily download songs from SoundCloud. SoundCloud does allow downloads from the browser, but that requires one to navigate to the page, click download on each song they want, and specify an output folder. This command line tool is going to be simple, yet powerful alternative to downloading from the browser.

Some of the options that have already been thought of are:

• Download a single song

 scget <trackUrl>  

• Download multiple songs

 scget <trackUrl1> <trackUrl2> ... <trackUrlN>  

• Download all songs from a particular artist

 scget -dj <userUrl>  

• And of course, the ability to specify a download folder

 scget <trackUrl> -o <folder>  

Obviously this would be easier than having to navigate the webpage because you could just queue up a bunch of urls to download and then hit go. It will do all of the processing behind the scenes while you get back to what you were doing before.

For this project, I am working in collaboration with Josh Branchaud on this project because not only are we great friends, but we both love SoundCloud. You can follow our progress on this command line tool on our github page.

P.S. - If you have any suggestions about other functionality that should be included in this tool feel free to leave it in a comment.