Saturday, June 25, 2016

Using Cloudstack with Powershell

If you are trying to work with Powershell with a Cloudstack environment through the REST API, you will soon find that there aren't that many solutions available. For Linux/Python environments there is Cloudmonkey which is excellent, but there is no native "VMware PowerCLI" like tool available from either Apache or Citrix, and even googling for Cloudstack and Powershell give basically just two Github projects:

  • A project that was last edited 3 years ago on Github from fifthecho
  • A project on Github from one of the engineers of Schuberg Philis, called psCloudstack

There is a project from Exoscale as well, but that is basically the Github contents from fifthecho, with perhaps some additions to it, and also from 2013.

The fifthecho project consists of a number of Powershell functions spread over a few.ps1 files, and a Cloudstack module that is called by every ps1 file to give the core functionality to talk to the Cloudstack API. Unfortunately it is not actively maintained, and all the files have an expired signature giving you an error when you try to execute the Powershell scripts. You can get around this by doing:

Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process

Basically for the Powershell session you are in, you are allowing everything, so it is not very elegant. Next to this, the Powershell scripts that are delivered only cover a small fraction of what is possible through the API (10 or so out of almost 250), and the way it is programmed is a bit dated, and I didn't feel like spending ages to create an extensive library myself, so I will let that be.

Then there is psCloudstack. I saw a video about this module a while ago from the creator himself, Hans van Veen, and I was impressed by the capability. This Powershell module takes a different approach: Instead of having a bunch of Powershell commands that talk to the API, it lists all the API commands (by virtue of the "listApis" command in the Cloudstack API library), and builds actual Powershell functions from them, complete with get-help functionality. These functions reside in memory, and will need to be "re-built" whenever you close the Powershell window. The elegance of this is that whenever you have a new version of Cloudstack, you have the complete API set to your disposal.

I had issues with getting the module to work though, but since it was the most comprehensive Cloudstack module for Powershell, I wanted to make it work, so I took some time to troubleshoot what happens. I'll go over installation first:

To download the module, go to the psCloudstack Github page, and download the files.

Create a module directory if you don't have one:

$PSModulePath = $Env:PSModulePath -split ";" | Select -Index ([int][bool]$Global)
New-Item -ItemType Directory $PSModulePath\psCloudstack

Then unzip all psCloudstack files into that directory (especially the .psm1 and psd1 file). Windows recognizes things that are downloaded from the Internet so you need to rightclick the psm1 and psd1 file and select Properties, then click the Unblock button.

If you have a Windows 10 machine, or have WMF5 installed, you can do it with this script:

$PSModulePath = $Env:PSModulePath -split ";" | Select -Index ([int][bool]$Global)
New-Item -ItemType Directory $PSModulePath\psCloudstack
Invoke-WebRequest -Uri "" -OutFile $PSModulePath\psCloudstack\PSCloudstack.psm1
Invoke-WebRequest -Uri "" -OutFile $PSModulePath\psCloudstack\PSCloudstack.psd1
Unblock-File $PSModulePath\psCloudstack\PSCloudstack.psd1
Unblock-File $PSModulePath\psCloudstack\PSCloudstack.psm1

So, now you have psCloudstack on your machine. Time to add a configuration. If you list the commands from the module you get the following list:

PS C:\Users\Akos\Documents\WindowsPowerShell\Modules> Get-Command -Module pscloudstack

CommandType     Name                                               Version    Source                            
-----------     ----                                               -------    ------                            
Function        Add-CSConfig                                       3.2.1      pscloudstack                      
Function        Connect-CSManager                                  3.2.1      pscloudstack                      
Function        Convert-CSConfig                                   3.2.1      pscloudstack                      
Function        Get-CSConfig                                       3.2.1      pscloudstack                      
Function        Invoke-CSApiCall                                   3.2.1      pscloudstack                      
Function        Remove-CSConfig                                    3.2.1      pscloudstack                      
Function        Set-CSConfig                                       3.2.1      pscloudstack                      
Function        Start-CSConsoleSession                             3.2.1      pscloudstack                      

Most commands in the module have to do with the configuration, but we just need Add-CSConfig, so let's look how we should use it:
PS C:\Users\Akos\Documents\WindowsPowerShell\Modules> get-help Add-CSConfig

    Adds connection configuration information to the configuration file
    Add-CSConfig [[-Zone] ] [-Server]  [[-SecurePort] ] [[-UnsecurePort] ] 
    [-Apikey]  [-Secret]  [-UseSSL] [[-ConfigFile] ] []
    This function is used to add connection configuration information to the configuration file.

PS C:\Users\Akos\Documents\WindowsPowerShell\Modules> get-help Add-CSConfig -Examples

    Adds connection configuration information to the configuration file
    -------------------------- EXAMPLE 1 --------------------------
    PS C:\># Create/Update the content of the default config file
    C:\PS> Add-CSConfig -Server -Api xxxxxxx -Secret yyyyyyyyy

This, unfortunately, is where the first "bug" occurs for me, when trying to execute this command. When executing this command without the -UseSSL switch you get an error:

Add-CSConfig -Server -Apikey xxx -Secret yyy
Add-CSConfig : Cannot process argument transformation on parameter 'UseSSL'. Cannot convert value "System.String" to type "System.Management.Automation.SwitchParameter". Boolean parameters accept only Boolean values and numbers, such as $True, $False, 1 or 0.

When try use it with the -UseSSL (and without the "https://" prefix, there is no feedback, so let's try to connect:
Add-CSConfig -Server -Apikey xxx -Secret yyy -UseSSL

PS C:\Users\Akos\Documents\WindowsPowerShell\Modules> Connect-CSManager
Welcome to psCloudstack V3.2.1, ...
API Call Error: 
                    The application has encountered an unknown error.
                    Our technical staff have been automatically notified and will be looking into this with the u
tmost urgency.
  psCloudstack-version displaytext                                                                                
-------------------- -----------                                                                                
3.2.1                ...                                                                                        

Whoops, what happened?.. It was time to start looking at the module itself, because something was happening, but I didn't know what. I see that by default, the config file is put in $env:LocalAppData (a.k.a. C:\Users\<username>\AppData\Local ), so when I look at that file, I see unusual ports being used (8080 and 8096):

Instead of trying to redo the Add-CSConfig, I decided to edit the file directly, putting 443 an 80. Now lets try again:

PS C:\Users\Akos\Documents\WindowsPowerShell\Modules> Connect-CSManager
Welcome to psCloudstack V3.2.1, ...
API Call Error: 
                    The application has encountered an unknown error.
                    Our technical staff have been automatically notified and will be looking into this with the u
tmost urgency.
  psCloudstack-version displaytext                                                                                
-------------------- -----------                                                                                
3.2.1                ...                                                                                        

More troubleshooting was needed. I will spare you the gritty details, but eventually I used the fifthecho module to create a connectionstring that I knew worked, and compared it to the connectionstring that the psCloudstack module produced, and I found what was the issue for me:

In line 1036, there is the following statement:

$baseUrl = "{0}://{1}:{2}/client/{3}?{4}" -f $Protocol, $InputObject.Server, $Port, $InputObject.Type, $InputObject.Command

which creates something like https://url:443/client/api?convertedstringwhichholdsapicall.

In the mean time, the Cloudstack implementation from Interoute VDC wants:


So by changing the way the string is connecting, the problem should be solved. Comment out the line, and put the line below under it.

#$baseUrl = "{0}://{1}:{2}/client/{3}?{4}" -f $Protocol, $InputObject.Server, $Port, $InputObject.Type, $InputObject.Command
$baseUrl = "{0}://{1}?{2}" -f $Protocol, $InputObject.Server, $InputObject.Command

Now let's try it once more  (do a "remove-module psCloudstack" first to reload the module):

PS C:\Users\Akos\Documents\WindowsPowerShell\Modules> Connect-CSManager
Welcome to psCloudstack V3.2.1, ...generating 252 api functions for you

Yeay, success! Now when you do Get-Command -Module psCloudstack you'll see a whole bunch of functions. These functions are the actual API calls that you can make to the Cloudstack environment:

PS C:\Users\Akos\Documents\WindowsPowerShell\Modules> Get-Command -Module psCloudstack

CommandType     Name                                               Version    Source                            
-----------     ----                                               -------    ------                            
Function        activateProject                                    3.2.1      PSCloudstack                      
Function        addAccountToProject                                3.2.1      PSCloudstack                      
Function        Add-CSConfig                                       3.2.1      PSCloudstack                      
Function        addIpToNic                                         3.2.1      PSCloudstack                      
Function        addNicToVirtualMachine                             3.2.1      PSCloudstack                      
Function        addVpnUser                                         3.2.1      PSCloudstack                      
Function        archiveEvents                                      3.2.1      PSCloudstack                      
Function        assignCertToLoadBalancer                           3.2.1      PSCloudstack                      
Function        assignToGlobalLoadBalancerRule                     3.2.1      PSCloudstack                      
Function        assignToLoadBalancerRule                           3.2.1      PSCloudstack                      
Function        assignVirtualMachine                               3.2.1      PSCloudstack                      
Function        associateIpAddress                                 3.2.1      PSCloudstack                      
Function        attachIso                                          3.2.1      PSCloudstack                      
Function        attachVolume                                       3.2.1      PSCloudstack                      
Function        authorizeSecurityGroupEgress                       3.2.1      PSCloudstack                      
Function        authorizeSecurityGroupIngress                      3.2.1      PSCloudstack                      
Function        changeServiceForRouter                             3.2.1      PSCloudstack                      
Function        changeServiceForVirtualMachine                     3.2.1      PSCloudstack                      
Function        configureInternalLoadBalancerElement               3.2.1      PSCloudstack                      
Function        configureVirtualRouterElement                      3.2.1      PSCloudstack                      
Function        Connect-CSManager                                  3.2.1      PSCloudstack                      
Function        Convert-CSConfig                                   3.2.1      PSCloudstack      

252 modules are created. Note that your Cloudstack provider could deny access to some API calls.

Example to list and create VM's:
PS C:\Users\Akos\Documents\WindowsPowerShell\Modules> listVirtualMachines |select name, templatename

name     templatename      
----     ------------      
server01 ubuntu1404_minimal

Awesome! and when you do Get-Help on one of these functions, you see actual help: Super stuff from Hans van Veen. He is maintaining this module, and even while I was writing this blogpost, he updated the module again. Unfortunately, the most recent update he made will give you an error, as it is trying to use some API calls that have been disallowed from usage by the Interoute VDC API (which is using Cloudstack), but the functions are still created and work.

I may want to modify the module so that the functions don't just live in memory (and you first need to run Connect-CSManager), but are in a module of its own, and I may want to modify the connection functionality, so it "just works" (at least with the Cloudstack environment that I have at my disposal).

Somewhere in another post I will also add some examples to use this excellent module.

Thursday, June 16, 2016

Desired State Configuration - Part 1

I have been diving into Powershell Desired State Configuration lately, as the technology is now a few years old and a little more mature, especially with the coming of Powershell 5. Many in the industry named this technology the technology that you should learn last year or the year before that already and not to mention that Microsoft keeps hammering on it that you should learn about it.

So what is Powershell DSC? In its core it is a configuration management platform. It allows for systems and applications to be installed and configured by means of a configuration document, and can manage configuration drift. Configuration management tools are not new, and have been around for a while with other tools like Puppet, Chef, Ansible, Salt and others.They serve the same purpose with varying features, but are mostly catered to Linux platforms. The same way that Powershell was an answer to having a proper commandline automation tool for Windows that supported its .Net and object oriented nature, DSC is Microsoft's answer to Configuration Management. But Microsoft went further by making DSC a platform that other tools use. So instead of competing against companies like Pupped and Chef, Microsoft has actively engaged with these companies so they could use the technology to better automatically configure Windows systems and applications.

Powershell and DSC inventor Jeffrey Snover describes DSC as captain Picard on the Starship Enterprise ordering Riker to go to that distant planet by saying "Make it so number one", and Riker executes that order: Picard desires something, and Riker "makes it so". DSC is basically the same thing inside your computer: With DSC, you desire something to be in a certain state through a configuration file and the local agent (called the Local Configuration Manager a.k.a. LCM) makes it so. This LCM is not all knowing in itself, but it uses things called DSC Resources to do the tasks you want it to do. These resources are now rapidly being developed by both Microsoft as well as the community around it, but if you can't find the thing you want to have configured, you can create them yourself. The main source for resources is the Powershell Gallery, and Powershell 5 has got the PowerShellGet module which now lets you install modules straight from this repository, just like you would use apt-get from a linux repository.

What makes DSC so cool?
  • You can deploy a system complete with applications and settings, nuke it and deploy the exact same machine (which makes devs happy because your dev matches your prod environment exactly)
  • Your OS/application is as you documented it in your configuration file, and configuration drift can be corrected by DSC
  • The DSC configuration is a Powershell based script, so you can use Powershell for variables and such, and do troubleshooting with it too
  • DSC can be used not only for Windows systems, but is slowly making its way to hardware devices too (e.g. Cisco Nexus)
  • There's linux support too, but as far as I've seen it, that's a bit of a work in progress. Cool nonetheless. 
  • Puppet/Chef/Salt/Ansible support the use of the DSC resources!
  • Windows Nano Server doesn't support Group Policies, but it supports DSC instead
There are more cool things about it, and DSC resources are added daily to make it even cooler. In another post I will go over the more technical details.