Recursion with Python

I had a bug that was difficult to trace down. I had a double list that I removed some of the elements using the remove() method, however, not all of the elements were removed. In fact, the code was removing only every other element. The bug turned out to be the call to remove() would shorten the list by one, and thus cause a skipping effect. For example:

some_lst = [['a','b'],['c','d'],['e','f']]

for i in some_lst:
  if re.match(regexToMatch, i):
     some_lst.remove(i)

I needed to rewind the list if remove() was called. At first, I thought this would be perfect for recursion. As it turned out, with the amount of data I needed the algorithm to search through, recursion was not a the best solution.

A recursive algorithm is one that calls itself. These functions will additionally have a decrementing counter or meet some condition to exit the recursive call. Having an exit condition is required since the function calls itself, otherwise, the function will loop until the system runs out of memory. This is a fast algorithm, however, it is not suited for all conditions. For example, small datasets recursion is best with regards to performance, but as the data grows, performance decreases quickly. This is because memory is consumed and the process incurs a context switch for every function call. The code below worked great on a small subset of data, but did not scale to what was needed for the project.

import re

def remove_processes(the_list: list):
    for (hostname, process) in the_list:
        for line in exclude_processes:
            try:
                if re.search(line.strip(), process):
                    the_list.remove([hostname,process])
                    remove_processes(theList)
            except ValueError:
                continue
    return the_list

Eventually, I replaced the for loop with while() and a counter. When a match is found, count is subtracted by 1 so every element in theList is evaluated. This also removes any duplicates.

def remove_processes(the_list):
    count = 0
    while count <= len(the_list) - 1:
        hostname = the_list[count][0]
        process = the_list[count][1]
        count += 1
    try:
        if re.search(exclude_process, process):
            the_list.remove([hostname, process])
            # reduce the count by 1 which resolves skipping
            # every other element
            count -= 1
    except ValueError:
        continue

    return the_list

Using Recursion with JSON

Here is another example of recursion. recurs() searches for k in a JSON document. If key isn’t found, it continues to the next element until all elements are searched. Once key is found, the function updates the value to “changed”. I used this function to remove passwords in a JSON document so it could be sent externally.

def recurs(k: str, json_doc: dict):
    """
    Takes a string, k and searches through json_doc
    :param k: search string
    :param json_doc: JSON / dictionary to search through for k
    """
    for i in json_doc.keys():
        if isinstance(json_doc[i], list):
            for p in range(len(json_doc[i])):
                json_doc[i][p][k] = 'changed'
        elif isinstance(json_doc[i], str):
           if k in json_doc:
               json_doc[k] = 'changed'
        elif isinstance(json_doc[i], dict):
           json_doc[i][k] = 'changed'
        try:
            for v in json_doc[i].values():
                if isinstance(v, dict):
                   return recurs(k, v)
        except AttributeError:
            pass
    return json_doc

Using Kitchen and Docker to Test Chef Cookbooks

Chef Cookbooks

Chef is a powerful configuration management tool that automates system builds, configures software and ensures configurations are kept up-to-date. A recipe is the most basic type of configuration for Chef. A recipe installs, configures or starts a service, updates a configuration file or anything else that is required to configure a node. Cookbooks group multiple recipes in a collection. Chef executes these cookbooks on multiple nodes which could range from a few systems to thousands. Given the scale at which Chef operates, changes to the cookbooks should be rigorously tested prior to deploying them on live systems. This is where kitchen plays an important role. With kitchen, we can spin up a Docker container, test the cookbook changes in a container and then remove the container once the test is complete.

Components of a Cookbook

The main components of a cookbook are recipes, tests and spec directories. The format of a recipe takes is a resource declaration, name of the resource, and an action. Develop the Cookbook Test Cases section of this article covers test cases.

resource name do
    action :some_action
end

Note: Anyone with shell scripting experience will naturally end the resource definition with done and not end. This will generate an unexpected end-of-input error.

FATAL: Stacktrace dumped to /root/.chef/local-mode-cache/cache/chef-stacktrace.out
FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
ERROR: /root/cookbooks/kitchen_test/recipes/default.rb:14: syntax error, unexpected end-of-input, expecting keyword_end
FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Creating a Cookbook

For our mongodb example cookbook, we will use two resources: package and service. The package resource will install or remove the software. The service resource will enable, start or disable a service from the system. You will notice on line 6 that we are using a Ruby list and combining two actions, enable and start.

Please note: this cookbook is an example only to showcase using kitchen. The Chef Supermarket has a cookbook available which provides much more functionality. You can download the cookbook from here.

The chef command below will create the cookbook structure. If chef isn’t installed on the system, you can download it as part of the ChefDK found here. Chef is deprecating the knife cookbook create command in favor of using the chef command.

root@ip-172-36-61-107:~/cookbooks# chef generate cookbook mongodb_server
Generating cookbook mongodb_server
- Ensuring correct cookbook file content
- Committing cookbook files to git
- Ensuring delivery configuration
- Ensuring correct delivery build cookbook content
- Adding delivery configuration to feature branch
- Adding build cookbook to feature branch
- Merging delivery content feature branch to master

Your cookbook is ready. Type `cd mongodb_server` to enter it.

There are several commands you can run to get started locally developing and testing your cookbook.
Type `delivery local --help` to see a full list.

Why not start by writing a test? Tests for the default recipe are stored at:

test/recipes/default_test.rb

If you'd prefer to dive right in, the default recipe can be found at:

recipes/default.rb

The chef generate command creates the directory structure below. Note: the default_test.rb has some sample content, so be sure to replace this content as needed.

root@ubuntu-chef:~/cookbooks# tree mongodb_server
mongodb_server
├── Berksfile
├── chefignore
├── LICENSE
├── metadata.rb
├── README.md
├── recipes
│   └── default.rb
├── spec
│   ├── spec_helper.rb
│   └── unit
│       └── recipes
│           └── default_spec.rb
└── test
    └── integration
        └── default
            └── default_test.rb

This is the cookbook we will use for the kitchen tests. Create the recipe file under mongodb_server/recipes and name it mongodb.rb. The “rb” is the extention for Ruby and is used for all recipes and kitchen tests.

package 'mongodb-server' do
        action :install
end

package 'mongodb-dev' do
        action :install
end


service 'mongodb' do
        action [ :enable, :start ]
end

Update the default.rb file in the mongodb_server/recipe to include the mongodb.rb file.

include_recipe 'mongodb_server::mongodb'

Execute the chef-client command to run the recipe locally. Alternatively, if you have setup Hosted Chef, you may omit “–local-mode” from the command below.

root@ip-172-36-61-107:~/cookbooks# chef-client --local-mode -r "recipe[mongodb_server]"
[0000-00-00T00:00:00+00:00] WARN: No config file found or specified on command line, using command line options.
Starting Chef Client, version 12.13.37
resolving cookbooks for run list: ["mongodb_server"]
Synchronizing Cookbooks:
  - mongodb_server (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 3 resources
Recipe: mongodb_server::mongodb
  * apt_package[mongodb-server] action install (up to date)
  * apt_package[mongodb-dev] action install (up to date)
  * service[mongodb] action enable (up to date)
  * service[mongodb] action start (up to date)

Running handlers:
Running handlers complete
Chef Client finished, 0/4 resources updated in 01 seconds

Testing Chef Cookbooks with Kitchen

The Chef Development Kit (ChefDK) includes the kitchen command to validate changes to a cookbook . With the kitchen-docker driver, kitchen automatically creates Docker containers, applies the cookbook and then destroys the container. In order to install Docker, please follow the instructions here.

We have to initialize the environment using the command kitchen init. Since we are using the Docker driver, we pass the “-D docker” argument so Chef will use docker instead of the default vagrant. kitchen will automatically download and install the Docker gem package.

root@ubuntu-chef:~/cookbooks/mongodb_server# kitchen init -D docker
      create  .kitchen.yml
      create  chefignore
Successfully installed docker-0.4.0
Parsing documentation for docker-0.4.0
Done installing documentation for docker after 0 seconds
1 gem installed

Configuring .kitchen.yml File

The init phase creates the configuration file named .kitchen.yml. You can leave the default .kitchen.yaml, or add or remove any other items required for testing with the cookbook. We removed the CentOS platform since we are only using Ubuntu in this example.

We are using the default verifier named Inspec. The suites heading, describes what we are wanting to test. We are only testing the default recipe, but in a real-world cookbook, we may define different recipes to support the cookbook. Underneath the suites -> verifier subheading lists the directory where the test cases reside. The test cases will not execute since Chef will check a different location.

---
driver:
  name: docker

provisioner:
  name: chef_zero
  # You may wish to disable always updating cookbooks in CI or other testing environments.
  # For example:
  #   always_update_cookbooks: <%= !ENV['CI'] %>
  always_update_cookbooks: true

verifier:
  name: inspec

platforms:
  - name: ubuntu-14.04

suites:
  - name: default
    run_list:
      - recipe[mongodb_server::default]
    verifier:
      inspec_tests:
        - test/recipes/default
    attributes:

After running kitchen init, you can see the list of instances available by running list. Below is only one instance since we removed Centos from the .kitchen.yml file.

root@ubuntu-chef:~/cookbooks/mongodb_server# kitchen list
Instance             Driver  Provisioner  Verifier  Transport  Last Action    Last Error
default-ubuntu-1404  Docker  ChefSolo     Busser    Ssh        <Not Created>  <None>

Now create the virtual instance used for testing with the kitchen create command. Note: this command usually takes 3-5 minutes to complete.

root@ubuntu-chef:~/cookbooks/mongodb_server# kitchen create
-----> Starting Kitchen (v1.20.0)
-----> Creating <default-ubuntu-1404>...
       Sending build context to Docker daemon  201.2kB
       Step 1/17 : FROM ubuntu:14.04
       14.04: Pulling from library/ubuntu
       324d088ce065: Pulling fs layer
       2ab951b6c615: Pulling fs layer
       9b01635313e2: Pulling fs layer
       04510b914a6c: Pulling fs layer
       83ab617df7b4: Pulling fs layer
       04510b914a6c: Waiting
       83ab617df7b4: Waiting
       9b01635313e2: Verifying Checksum
       9b01635313e2: Download complete

We can use docker commands to see the images and containers kitchen created. If you need to check the container, use the command docker exec -it exec -it 83043dc787e8 /bin/bash

root@ubuntu-chef:~/cookbooks/mongodb_server# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                   NAMES
b6948b229272        71cdf1224f9a        "/usr/sbin/sshd -D -…"   2 minutes ago       Up 2 minutes        0.0.0.0:32768->22/tcp   defaultubuntu1604-chefuser-ubuntuchef-9fphugrq


root@ubuntu-chef:~/cookbooks/mongodb_server# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
<none>              <none>              71cdf1224f9a        2 minutes ago       216MB
ubuntu              14.04               0b1edfbffd27        3 days ago          113MB

The next step is to install and run the cookbook in the virtual instances created from the step above using the converge sub-command. This command may take several minutes to complete.

root@ubuntu-chef:~/cookbooks/mongodb_server# kitchen converge
-----> Starting Kitchen (v1.21.1)
-----> Converging <default-ubuntu-1404>...
       Preparing files for transfer
       Preparing dna.json
       Preparing current project directory as a cookbook
       Removing non-cookbook files before transfer
       Preparing solo.rb
-----> Installing Chef Omnibus (install only if missing)
       Downloading https://omnitruck.chef.io/install.sh to file /tmp/install.sh
       Trying wget...
       Download complete.
       ubuntu 16.04 x86_64
       Getting information for chef stable  for ubuntu...

Develop the Cookbook Test Cases

After running kitchen converge, we can run test cases using the verify sub-command. Below is an example test script which verifies the state of Mongo. The script checks that the default mongodb port is in a LISTEN state, the mongodb-server package is installed, a user is defined as mongodb, /etc/mongodb.conf exists and finally checks if the service is running.

describe port(27017) do
  it { should be_listening }
end

describe package('mongodb-server') do
               it { should be_installed}
end

describe user('mongodb') do
    it { should exist }
end

describe file('/etc/mongodb.conf') do
    it { should be_file }
end


describe service('mongodb') do
    it { should be_running }
end

Use the verify argument to execute the test cases written in the previous step.

root@ubuntu-chef:~/cookbooks/mongodb_server# kitchen verify
-----> Starting Kitchen (v1.20.0)
-----> Verifying <default-ubuntu-1404>...
       Loaded tests from {:path=>".root.cookbooks.mongodb_server.test.recipes.default"} 

Profile: tests from {:path=>"/root/cookbooks/recipes/test/recipes/default"} (tests from {:path=>".root.cookbooks.mongodb_server.test.recipes.default"})
Version: (not specified)
Target:  ssh://kitchen@localhost:32769

  Port 27017
     ✔  should be listening
  System Package mongodb-server
     ✔  should be installed
  User mongodb
     ✔  should exist
  File /etc/mongodb.conf
     ✔  should be file
  Service mongodb
     ✔  should be running

Test Summary: 5 successful, 0 failures, 0 skipped
       Finished verifying <default-ubuntu-1404> (0m0.24s).
-----> Kitchen is finished. (0m1.50s)

The last step uses the destroy sub-command which removes the Docker container.

root@ubuntu-chef:~/cookbooks/mongodb_server# kitchen destroy
-----> Starting Kitchen (v1.20.0)
-----> Destroying <default-ubuntu-1404>...
       UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
       root                27070               19304               0                   May03               ?                   00:00:00            /usr/sbin/sshd -D -o UseDNS=no -o UsePAM=no -o PasswordAuthentication=yes -o UsePrivilegeSeparation=no -o PidFile=/tmp/sshd.pid
       usbmux              34586               27070               0                   09:19               ?                   00:00:18            /usr/bin/mongod --unixSocketPrefix=/var/run/mongodb --config /etc/mongodb.conf run
       41f85bcf65e1707d152d2e8c351ed4289d4e65d355328acc1e4914457d5193d4
       41f85bcf65e1707d152d2e8c351ed4289d4e65d355328acc1e4914457d5193d4
       Finished destroying <default-ubuntu-1404> (0m1.88s).
-----> Kitchen is finished. (0m2.93s)

Python – Working with Lists

One of the most common data types in Python is the list. A list is basically an array in other languages, however with a list, you can mix different data types in the same list. You can test this in the interactive shell:

>>> a_list = ['dog', 'cat', 1, 3, 1000]
>>> print(a_list)
['dog', 'cat', 1, 3, 1000]
>>> type(a_list[3])
<class 'int'>
>>> type(a_list[1])
<class 'str'>

Working with Python Lists

After declaring a list, we will need to either add data, delete data or assign data to another variable. In order do accomplish these tasks, we will need to use the append() method to add data, pop() or remove() to delete data, and subset the list to retrieve elements or assign to another variable.

Lists are subsetted by using the brackets ([]), a positional number and / or using a colon (:). Using a colon will allow a subset range.

Note: list elements start at 0 not 1.

>>> a_list = ['dog', 'cat', 'bat']
>>> b_str = a_list[0] # Take the first element and assign it to b_str
>>> b_str
'dog'
>>> type(b_str)
<class 'str'>
>>> a_list[-1] # Take the last element from the list 
'bat'
>>> a_list[-2] # Take the second to last element from the list
'cat'
>>> type(a_list[0]) # Lists can contain ints, strings, dictionaries or other lists
<class 'str'>
>>> type(a_list[-1])
<class 'int'>

Using a range with list elements sometimes is prone to defects in code. The number before the colon is the starting point, and the number after is the position to end minus 1. For example, a_list[1:4] starts at element 1 and ends at element 3, not 4. If you’ve developed in other languages, this will take some time to acclimate to Python’s way of list subscripting.

>>> a_list
['cat', 1, 3, 1000]
>>> a_list[0:3] # Take the first element through the second element
['cat', 1, 3]
>>> a_list[1:-1] # Take the second element through the second to last element
'cat', 1, 3]

Here is an example of a list containing another list and dictionary. Lists containing other lists is common in JSON or RESTful development, so becoming familiar with the syntax is important as you develop more complex or web-enabled applications. We still can call dictionary methods like keys() or values().

>>> b_list = ['this', 'new' 'list']
>>> a_list.append(b_list)
>>> a_list
['cat', 1, 3, 1000, ['this', 'newlist']]

>>> my_dct = {'language': 'Python'}
>>> a_list.append(my_dct)
>>> a_list
['cat', 1, 3, 1000, ['this', 'newlist'], {'language': 'Python'}]
>>> a_list[-1].keys() # We can call dictionary methods  
dict_keys(['language'])
>>> a_list[-1].values()
dict_values(['Python'])

Simple Merge

Here are some examples of using Python lists by merging two lists into one. This simple example appends the values from second_list onto first_list.

firstList = ['dog', 'cat', 'tiger', 'rhnio']
secondList = ['2 x 18g', '250m swap']

for line in secondList:
   firstList.append(line)

Merge Lists Based on A Condition

The following code merges two lists, but will insert the second list after finding a specific string, in this case a hostname. outerList.pop(0) removes the first element from the list, and then inserts the remaining list into firstList.

firstList = ['dog', 'cat', '2', '500', 'daeo', 'DL580', '8', '128']
secondList = [['dog', '2 x 18g', '250m swap'], ['daeo', '4 x 146g', '16g swap']]

try:
    for outerList in secondList:
        systemName = outerList.pop(0)
        for innerList in outerList:
            indexPos = firstList.index(systemName)
            firstList.insert(indexPos + 1, innerList)
except ValueError as err:
    print('ValueError: {}'.format(err))