Table of Contents
URL: https://www.progressiverobot.com/how-to-use-the-cloud-provider-plugin-for-knife-to-manage-droplets-in-chef/
Introduction
Chef is a configuration management system that allows you to build and manage your infrastructure in a controlled and repeatable way. With Chef, you can ensure that your infrastructure decisions are all centrally documented and can be reimplemented easily at any time.
In previous articles, we have discussed basic Chef terminology, how to install the software(with Chef 12 or Chef 11), how to work with cookbooks, and how to work with roles and environments.
In this article, we will discuss how to use a plugin for knife, the Chef configuration tool, to work with your cloud servers. Using this tool, we can create infrastructure droplets and configure them easily from within our Chef system.
We will assume that you have installed and configured your server and workstation, and that you know how to create and bootstrap new nodes.
Install the Knife the cloud provider Plugin
Before we can begin working with cloud servers from within our Chef system, we need to install the the cloud provider plugin for the knife tool.
We can do this simply by installing a gem.
With the way our workstation is configured, if we were to type in:
gem install knife-the cloud provider
We would get an error like this:
Fetching: knife-the cloud provider-0.4.0.gem (100%)
ERROR: While executing gem … (Gem::FilePermissionError)
You don't have write permissions into the /opt/chef/embedded/lib/ruby/gems/1.9.1 directory.
Similarly, if we run this command with sudo, we will get a command not found:
sudo gem install knife-the cloud provider
***
sudo: gem: command not found
The problem is that we have specified the path to our Chef executables within our user's environment, but we do not have the privileges necessary to execute the command correctly.
We can get around this by using the full path to the Chef gem executable:
sudo /opt/chef/embedded/bin/gem install knife-the cloud provider
This will install the the cloud provider knife plugin, allowing you to take advantage of the included functionality.
Configure the Knife Plugin with your the cloud provider Credentials
—
In order to use the knife plugin, you need to give knife authorization to use your cloud account. This can be done using your account's API key and Client ID.
In your the cloud provider control panel, click on the API link in the left-hand navigation menu. You should see the "Client ID" of your account here:
You will need this value for your configuration file in a moment.
If you have already generated an API key, you will need to collect that value from another application that is using it to connect to your cloud account. It is not possible to retrieve the current API value from the interface, as a security measure.
If you have not generated an API key yet, or if you have lost or do not need your old key anymore, you can generate a new key by typing the "Generate New Key" button:
Once you generate a new key, you will need to copy the value generated before leaving the screen. As mentioned above, you will not be able to access this value after leaving this page, so you will either have to generate a new value or retrieve your API key from an application that is already using it.
Once you have both the Client ID and the API key, you are ready to configure knife to use these values.
On your workstation, navigate to the Chef configuration directory within your Chef repo:
cd ~/chef-repo/.chef
We will be editing the knife.rb file located within:
nano knife.rb
***
log_level :info
log_location STDOUT
node_name 'station1'
client_key '/home/demo/chef-repo/.chef/station1.pem'
validation_client_name 'chef-validator'
validation_key '/home/demo/chef-repo/.chef/chef-validator.pem'
chef_server_url 'https://chef_server.com:443'
syntax_check_cache_path '/home/demo/chef-repo/.chef/syntax_check_cache'
cookbook_path [ '/home/demo/chef-repo/cookbooks' ]
At the bottom of the file, we need to add our credentials, like this:
log_level :info
log_location STDOUT
node_name 'station1'
client_key '/home/demo/chef-repo/.chef/station1.pem'
validation_client_name 'chef-validator'
validation_key '/home/demo/chef-repo/.chef/chef-validator.pem'
chef_server_url 'https://chef_server.com:443'
syntax_check_cache_path '/home/demo/chef-repo/.chef/syntax_check_cache'
cookbook_path [ '/home/demo/chef-repo/cookbooks' ]
knife[:the cloud provider_client_id] = 'your_client_id'
knife[:the cloud provider_api_key] = 'your_api_key'
Save and close the file when you are done.
Now, we can test to see if knife can connect to the the cloud provider servers by asking for a list of our currently active droplets. Make sure you are in your Chef repo before running the knife command:
cd ~/chef-repo
knife the cloud provider droplet list
***
ID Name Size Region IPv4 Image Status
111111 irssi 512MB New York 1 111.111.111.111 479972 (N/A) active
222222 try 4GB New York 2 222.222.222.222 1575388 (Ubuntu-Init) active
333333 nftables 4GB New York 2 333.333.333.333 308287 (Debian 7.0 x64) active
4444444 snmp 4GB New York 2 444.444.444.444 1575388 (Ubuntu-Init) active
5555555 node 4GB New York 2 555.555.555.555 1575388 (Ubuntu-Init) active
If you get a list of your current droplets, you've connected successfully and your authentication is working.
Configure SSH Keys on your Chef Workstation
—
Now that you can connect to your cloud account using the knife tool, we need to go one step further to help automate the droplet creation process. We need to create an SSH key for our Chef workstation to embed in the droplets we will be created.
On our Workstation computer, we should create a new SSH key by typing:
ssh-keygen
Press enter through the prompts to accept the default values. This will create a hidden subdirectory in your user's home directory called .ssh with a public and private key inside.
Change into this directory right now by typing:
cd ~/.ssh
ls
***
id_rsa id_rsa.pub known_hosts
The file that we need right now is the id_rsa.pub file. You can view the contents of this file by typing:
cat id_rsa.pub
***
sh-rsa AAAAB3NzaC1yc2EAAAAxAQABAAABAQCv60WjxoM39LgPDbiW7ne3gu18q0NIV…………………lANpl5qmhDQ+GS/sO…………mHWL2irjuB9xBXr00+44vSd2q/vtXdGXhdSMTf4/XK17fjKSG/9y3yD6nml6q9XgQxx9Vf/IKaKdlK0hbC1ds0+8/83PTb9dF3L7hf3Ch/ghvj5++twJFdFeG+VI7EDuKNA4zL8C5FdY…………………XIIeZvZ/z9Kp…………………nyiIuEAmn8fbnosWcsovw0IS1Hz6HsjYo4bu/gA82LWt3sdRUBZ/7ZsVD3ELip
Back in the the cloud provider control panel, click on the "SSH Keys" link in the left-hand menu. Click on the "Add SSH Key" button in the upper-right corner:
On the next page, create a name for your new key and then paste the contents of the file into the area given:
Click on the "Create SSH Key" button below when you are finished.
You now should have the ability to embed the SSH key from your Chef server into new droplets. This should allow you to create droplets from within the interface and then login to configure your server with the cookbooks and roles that you've created, all in a single step.
Working with the Knife Plugin
—
Let's begin working with the knife plugin.
We can start by once again querying for the droplets associated with your cloud account:
knife the cloud provider droplet list
***
ID Name Size Region IPv4 Image Status
111111 irssi 512MB New York 1 111.111.111.111 479972 (N/A) active
222222 try 4GB New York 2 222.222.222.222 1575388 (Ubuntu-Init) active
333333 nftables 4GB New York 2 333.333.333.333 308287 (Debian 7.0 x64) active
4444444 snmp 4GB New York 2 444.444.444.444 1575388 (Ubuntu-Init) active
5555555 node 4GB New York 2 555.555.555.555 1575388 (Ubuntu-Init) active
This gives us an overview of our droplets.
The knife plugin is rather simple, but contains all of the pieces you need to control the droplets within your account, perhaps with the exception of DNS functionality. The main functionality is creating and destroying droplets with the information that you can get from querying.
Querying for Information Using the Knife Plugin
—
We can find out all of the information we may need for our create and destroy commands using different knife queries.
To find out which standard the cloud provider images that we have available to use as base images, we can type this. The -G flag stands for "global":
knife the cloud provider image list -G
***
ID Distribution Name Global
361740 Arch Linux Arch Linux 2013.05 x32 +
350424 Arch Linux Arch Linux 2013.05 x64 +
1602 CentOS CentOS 5.8 x32 +
1601 CentOS CentOS 5.8 x64 +
376568 CentOS CentOS 6.4 x32 +
. . .
This will give all of the standard the cloud provider images, including all one-click applications that you can use to deploy a new droplet. The ID column is the item that you need to pay attention to in order to deploy from the correct image.
Similarly, if you want a list of your snapshots, backups, etc., you can type the same command without the -G flag:
knife the cloud provider image list
***
ID Distribution Name Global
11xxxxx Ubuntu Dokku –
15xxxxx Ubuntu Ubuntu-Init –
15xxxxx Ubuntu Ubuntu-LAMP –
15xxxxx Ubuntu Ubuntu-WP –
You can use the image IDs from this list as base images as well for increased flexibility.
To find out which regions are available to deploy your droplet, type:
knife the cloud provider region list
***
ID Name
1 New York 1
2 Amsterdam 1
3 San Francisco 1
4 New York 2
5 Amsterdam 2
For a list of sizes to deploy your droplet, you can type:
knife the cloud provider size list
***
ID Name
63 1GB
62 2GB
64 4GB
65 8GB
61 16GB
. . .
To list the SSH keys that we can embed within an image, we can issue this command:
knife the cloud provider sshkey list
***
ID Name
11111 Home key
22222 Chef key
Creating and Destroying Droplets
—
We now know how to get all of the information necessary to create and destroy droplets.
We can start with destroying a droplet, since destruction always requires less effort than creation.
All we need to destroy a droplet is the droplet's ID. This is available in the first column of the droplet listing:
knife the cloud provider droplet list
***
ID Name Size Region IPv4 Image Status
111111 irssi 512MB New York 1 111.111.111.111 479972 (N/A) active
222222 try 4GB New York 2 222.222.222.222 1575388 (Ubuntu-Init) active
333333 nftables 4GB New York 2 333.333.333.333 308287 (Debian 7.0 x64) active
4444444 snmp 4GB New York 2 444.444.444.444 1575388 (Ubuntu-Init) active
5555555 node 4GB New York 2 555.555.555.555 1575388 (Ubuntu-Init) active
Once you have the ID of the droplet you wish to delete, you can use it by typing:
knife the cloud provider droplet destroy 111111
This should queue the droplet for deletion.
In order to create a droplet, you will need more information. Any of the fields that you usually need to fill out in the the cloud provider control panel will need to be entered as parameters into this command.
The command will look like this:
knife the cloud provider droplet create --server-name name_for_server --image image_id --location region_id --size size_id --ssh-keys ssh_key_ids --bootstrap --run-list "nodes_run_list"
For instance, to create an x86_64 Ubuntu 12.04 instance called "hello" on a 1G droplet in the NY2 region with an SSH key of 1111, we can type this:
knife the cloud provider droplet create --server-name hello --image 1505447 --location 4 --size 63 --ssh-keys 22222
Make sure that you are passing in the SSH key of your Chef workstation so that you will be able to connect to it is online.
If we want to bootstrap the droplet as it is created and pass a run_list to install and configure some software, we can add the additional parameters. Here, we can tell it to configure our new server as a web server:
knife the cloud provider droplet create --server-name hello --image 1505447 --location 4 --size 63 --ssh-keys 22222 --bootstrap --run-list "role[web_server]"
This will apply all of the recipes and attributes inherent in the role just as if we had edited the node's run_list after creating it.
Conclusion
—
Using the the cloud provider knife plugin, you can easily spin up new infrastructure resources as you require them. For instance, if your database is getting more traffic than it previously was, it is trivial to spin up additional database servers to manage that load.
By including functionality to create new VPS instances in the same environment where you manage your infrastructure, it becomes easy to scale out your infrastructure with well-designed cookbooks, roles, and attributes. Scaling can simply become a question of when instead of how.
