CFengine 3.12 new features 'missing_ok' and multiple 'augments'

At SURFsara we use CFEngine on our National Compute Cluster (LISA) and other systems as our configuration management tool. With the release of CFEngine 3.12  I want to highlight 2 new features,  namely:
  1.  missing_ok
  2.  multiple augments
We use these 2 new features heavily in our framework in combination with my open source library
cf_surfsara_lib.  This library aims to be a central repository for configuring services, eg: ssh. For configuring the services we use JSON as data format and it is easily to override the default values via JSON. Pre CFEngine 3.12 there is only one strategy possible:
  1. def.json : Setting the initial  value
  2. def.cf  :  Override the initial value
Due to this restriction we have a def.json  for all hosts even when the difference between them are small.  For minor tweaks we use def.cf  with the aid of classes. As with CFEngine version 3.12 you can define multiple JSON files  with the aid of the multiple augments feature.  In our environment we have implemented the following strategy in def.json:
{
   "augments": [
        "$(sys.inputdir)/json.d/global.json",
        "$(sys.inputdir)/json.d/os.json",
        "$(sys.inputdir)/json.d/key.json",
        "$(sys.inputdir)/json.d/ip.json"
    ]
}
 

We only have these definitions in our def.json  so we can easily create a local JSON file with local paths for testing and running automatic checks. Another  issue is that all the JSON files must be present else an error is reported.  This issue has been reported:
Due to this restriction we must have a default JSON file for all defined JSON files in def.json.  In our cluster we have defined roles  for this we use the CFEngine key, eg: all compute nodes in our cluster share the same CFEngine key. That is the reason why key.json  is not the last JSON file being used in our def.json.  For copying the JSON files we use variables that are set by CFEngine, namely:
  • os.json  is copied via the CFEngine variable: sys.flavor
  • key.json  is copied via the  CFEngine variable: connecton.id 
  • ip.json : is copied via the CFEngine variable: connection.ip

We do not care if the os.json, key.json  or ip.json files are present on the policy server.  For this I wrote a patch and introduced the missing_ok feature.   When this is set then CFEngine will consider the promise kept even if the file is missing.  This is how we use this two new features in CFengine at SURFsara.  Below I will share the update.conf  and cfserverd.conf  code. It would be nice for the feature if we can specify another JSON merge option then override. The discussion for other merge options is in ticket:
Another feature that I have proposed is CFE-2790: autoloading JSON file(s).  With this feature I can just drop JSON file(s) in a directory and it will be load just like the autorun bundle feature.

 UPDATE def.json setup

bundlle agent update_def_json()
{
    methods:
       any::
            "" usebundle => update_json_file("global.json", "global_def_json", "global_def_json");
            "" usebundle => update_json_file("os.json", "$(sys.flavor)_def_json", "default_os_def_json");
            "" usebundle => update_json_file("key.json", "key_def_json", "default_key_def_json");
            "" usebundle => update_json_file("ip.json", "ip_def_json", "default_ip_def_json");
}

bundle agent update_json_file(file, shortcut, failback_shortcut)
{
    vars:
        "json_file" string => "$(update_init.json_dir)/$(file)";

    files:
        any::
            "$(json_file)"
                comment => "def json file copy for $(shortcut), missing_ok: true",
                copy_from => update_def_json("$(shortcut)", "true"),
                perms  => update_perms("600"),
                move_obstructions   => "true",
                action => update_immediate;

            "$(json_file)"
                comment => "default def json file copy for $(failback_shortcut), missing_ok: false",
                copy_from => update_def_json("$(failback_shortcut)", "false"),
                perms  => update_perms("600"),
                move_obstructions   => "true",
                action => update_immediate,
                classes => update_results("$(failback_shortcut)");

    reports:
        any::
            "$(this.bundle): Failed to fetch: $(json_file)"
                ifvarclass => or( canonify("$(failback_shortcut)_failed") );
}
 

body copy_from update_def_json(from, missing)
{
    source => "$(from)";
    compare => "hash";
    copy_backup => "false";
    missing_ok => "$(missing)";
    protocol_version => "2";
    servers => { @(update_init.servers) };
}
 

CFSERVERD def.json setup

bundle server access_rules()
{
### Global.json
            "/data/cfengine3/cmdb/global.json"
                comment => "def.json for all hosts",
                shortcut => "global_def_json",
                admit_ips => { "@(g.acl_admit_ips)" };

### Key def.json       
            "/data/cfengine3/cmdb/key/$(connection.key).json"
                comment => "host key def.json",
                shortcut => "key_def_json",
                admit_keys => { "$(connection.key)" };
           
            "/data/cfengine3/cmdb/key/default.json"
                comment => "default host key def.json, failback",
                shortcut => "default_key_def_json",
                admit_ips => { "@(g.acl_admit_ips)" };
### End Key def.json

### IP def.json          
            "/data/cfengine3/cmdb/ip/$(connection.ip).json"                                                                                                                   
                comment => "host ip def.json",
                shortcut => "ip_def_json",
                admit_ips => { "$(connection.ip)" };

            "/data/cfengine3/cmdb/ip/default.json"
                comment => "default host ip def.json, failback",
                shortcut => "default_ip_def_json",
                admit_ips => { "@(g.acl_admit_ips)" };
### End IP def.json

### OS def.json
            "/data/cfengine3/cmdb/os/debian_9.json"
                comment => "debian stretch  def.json",
                shortcut => "debian_9_def_json",
                admit_ips => { "@(g.acl_admit_ips)" };

            "/data/cfengine3/cmdb/os/debian_8.json"
                comment => "debian jessie  def.json",
                shortcut => "debian_9_def_json",
                admit_ips => { "@(g.acl_admit_ips)" };

            "/data/cfengine3/cmdb/os/debian_7.json"
                comment => "debian jessie  def.json",
                shortcut => "debian_9_def_json",
                admit_ips => { "@(g.acl_admit_ips)" };

            "/data/cfengine3/cmdb/os/centos_6.json"
                comment => "centos 6 def.json",
                shortcut => "centos_6_def_json",
                admit_ips => { "@(g.acl_admit_ips)" };

            "/data/cfengine3/cmdb/os/default.json"
                comment => "default host os def.json, failback",
                shortcut => "default_os_def_json",
                admit_ips => { "@(g.acl_admit_ips)" };
### End OS def.json
}





Comments

Post a Comment

Popular Posts