Ansible is a great tool for automation; it has many ways to parse data from one form to another. For example, ansible ships with Jina2
filters and a bunch of ansible filters(as an extension to Jinja2 filters) for parsing the data. Using jinaj2 filters and ansible provided filters, we can do a lot of things; however, sometimes we have to deal with a non-standard data format(see example below), and parsing that data using ansible is impossible or very complex.
Thankfully, ansible is extendable, allowing us to write our custom filter plugin. The custom filter plugins are python programs( to be specific, python functions) that perform the desired operations. This post will show the usage and steps to create a filter plugin.
When to create a custom filter plugin?
We should try to leverage the plugins provided by jinja2 and ansible to parse any data; however, if they are not sufficient in some rare cases, we should write our filter plugin.
How is a filter used inside the playbook?
Any filter in the playbook is called by piping it to the input. Here is one example: we are using a filter called “upper” and piping it to a variable to transform the contents of the variable to upper case.
---
- hosts: localhost
gather_facts: False
vars:
myvar: 'hello there'
tasks:
- debug:
msg: "Input data is '{{ myvar }}', converted to upper case {{ myvar | upper }}"
The above playbook would create the output like the below:
TASK [debug] *************************************************************************************************************************************
ok: [localhost] => {
"msg": "Input data is 'hello there', converted to upper case HELLO THERE"
}
Jinja2 filters | Additional filters provided by ansible | custom filters | |
Example(s) | lower() list() title() replace() first() upper() | regex_replace regex_escape regex_search basename | We will see how to create one and use it. |
Documentation: | https://jinja.palletsprojects.com/en/3.1.x/templates/#list-of-builtin-filters | https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/filter/core.py |
Writing own custom filter
We often have to deal with a complex data format that is not easily parsable using ansible. Although, we are confident that we can quickly parse the data with our knowledge of python. In those cases, we should write our custom filter. The following is the example showing an input file with non-standard data as input, and we want to convert it to a friendlier format for further processing.
Sample input data
For example, one of the applications running on a node creates the logs in the following format, and I need to print them on the terminal in a better format. The problem is that the log file is neither JSON, YAML, or any standard Syslog format. As you can see below, the data is in a non-standard format. It makes the data non-standard data and parsing it will be a bit tricky using existing filters provided by ansible.
Objective?
Write a custom plugin to convert the logs shown in the below file into a one-liner log for a better look and further processing.
cat /tmp/sample.log
2022-09-01T00:01:05 MAJOR VM-1
CPU usage above threshold, current usage is 90%
2022-09-01T00:01:06 CRITICAL VM-1
MEMORY usage above threshold, current usage is 16Gi
2022-09-01T00:01:10 CRITICAL VM-1
Node going for reboot, marked as failed
2022-09-01T00:01:11 CRITICAL VM-0
Mate is unavailable, VM-0 is not Master
Step-1: Create your parsing logic and put it in python code
Assuming we will have a playbook file called example.yml, you need first to create a file inside a directory called “filter_plugins” you may give any name to the file. The directory “filter_plugins” must* be at the same level as the playbook(example.yml).
* Alternatively, you can define the “filter_plugins” variable in the ansible.conf file with any custom location.
#expected directory structure
# example.yml is the playbook name
# filter_plugins is the directory we created to write our custom filter plugin.
# my_filters.py is the file with out custom plugins(python functions)
tree
.
├── example.yml
└── filter_plugins
└── my_filters.py
1 directory, 2 files
vi filter_plugins/my_filters.py
import sys
# this is returning the filter name to function mapping.
# so if you create any mode functions(AKS filters) then you will have to
# add those to the returned dictornary.
# see example here https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/filter/core.py
class FilterModule(object):
def filters(self):
return {
'parse_logs': self.parse_logs
}
def parse_logs(self, supplied_file):
with open(supplied_file, 'r') as f:
lines = f.read().split('\n\n')
fixed_logs = []
for item in lines:
fixed_logs.append(item.replace('\n',' ').strip())
fixed_logs[:] = [x for x in fixed_logs if x]
return fixed_logs
Step-2: Create the playbook using the filter
---
- hosts: localhost
gather_facts: False
vars:
my_log_file: /tmp/sample.log
tasks:
- debug:
msg: "{{ my_log_file |parse_logs }}"
Step-3: Review the output of the playbook
PLAY [localhost] *********************************************************************************************************************************
TASK [debug] *************************************************************************************************************************************
ok: [localhost] => {
"msg": [
"2022-09-01T00:01:05 CRITICAL VM-1 CPU usage above threshold, current usage is 90%",
"2022-09-01T00:01:06 CRITICAL VM-1 MEMORY usage above threshold, current usage is 4Gi",
"2022-09-01T00:01:10 CRITICAL VM-1 Node going for reboot, marked as failed",
"2022-09-01T00:01:11 CRITICAL VM-0 Mate is unavailable, VM-0 is not Master"
]
}
PLAY RECAP ***************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Summary
Of course, in this post, I have shown a silly example, but the possibilities are endless as the filters are written in python, so you can do anything that python can do using the filters.
Filter plugins provide a clean and robust way of transforming data from files or variables into our desired format.