Introduction
Puppet is a software configuration management solution to manage IT infrastructure. One of the first things to be learnt about Puppet is its domain-specific language – the Puppet-DSL – and the concepts that come with it.
Users can organize their code in classes and modules and use pre-defined resource types to manage resources like files, packages, services and others.
The most commonly used types are part of the Puppet core, implemented in Ruby. Composite resource types may be defined via Puppet-DSL from already known types by the Puppet user themself, or imported as part of an external Puppet module which is maintained by external module developers.
It so happens that Puppet users can stay within the Puppet-DSL for a long time even when they deal with Puppet on a regular basis.
The first time I had a glimpse into this topic was when Debian Stable was shipping Puppet 5.5, which was not too long ago. The Puppet 5.5 documentation includes a chapter on custom types and provider development respectively, but to me they felt incomplete and lacking self contained examples. Apparently I was not the only one feeling that way, even though Puppet’s gem documentation is a good overview of what is possible in principle.
Gary Larizza’s blog post was more than ten years ago. I had another look into the documentation for Puppet 7 on that topic recently, as this is the Puppet version in current’s Debian Stable.
The Puppet 5.5 way to type & provider development is now called the low level method, and its documentation has not changed significantly. However, Puppet 6 upwards recommends a new method to create custom types & providers via the so-called Resource-API, whose documentation is a major improvement compared to the low-level method’s. The Resource-API is not a replacement, though, and has several documented limitations.
Nevertheless, for the remaining blog post, we will re-prototype a small portion of file
s functionality using the low-level method, as well as the Resource-API, namely the ensure
and content
properties.
Preparations
The following preparations are not necessary in an agent-server setup. We use bundle
to obtain a puppet
executable for this demo.
demo@85c63b50bfa3:~$ cat > Gemfile <<EOF source 'https://rubygems.org' gem 'puppet', '>= 6' EOF
demo@85c63b50bfa3:~$ bundle install Fetching gem metadata from https://rubygems.org/........ Resolving dependencies... ... Installing puppet 8.10.0 Bundle complete! 1 Gemfile dependency, 17 gems now installed. Bundled gems are installed into `./.vendor`
demo@85c63b50bfa3:~$ cat > file_builtin.pp <<EOF $file = '/home/demo/puppet-file-builtin' file {$file: content => 'This is madness'} EOF
demo@85c63b50bfa3:~$ bin/puppet apply file_builtin.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.01 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-builtin]/ensure: defined content as '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e' Notice: Applied catalog in 0.04 seconds demo@85c63b50bfa3:~$ sha256sum /home/demo/puppet-file-builtin 0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e /home/demo/puppet-file-builtin
Keep in mind the state change information printed by Puppet.
Low-level prototype
Custom types and providers that are not installed via a Gem need to be part of some Puppet module, so they can be copied to Puppet agents via the pluginsync mechanism.
A common location for Puppet modules is the modules
directory inside a Puppet environment. For this demo, we declare a demo
module.
Basic functionality
Our first attempt is the following type definition for a new type we will call file_llmethod
. It has no documentation or validation of input values.
# modules/demo/lib/puppet/type/file_llmethod.rb Puppet::Type.newtype(:file_llmethod) do newparam(:path, namevar: true) {} newproperty(:content) {} newproperty(:ensure) do newvalues(:present, :absent) defaultto(:present) end end
We have declared a path
parameter that serves as the namevar
for this type – there cannot be other file_llmethod
instances managing the same path
. The ensure
property is restricted to two values and defaults to present
.
The following provider implementation consists of a getter and a setter for each of the two properties content
and ensure
.
# modules/demo/lib/puppet/provider/file_llmethod/ruby.rb Puppet::Type.type(:file_llmethod).provide(:ruby) do def ensure File.exist?(@resource[:path]) ? :present : :absent end def ensure=(value) if value == :present # reuse setter self.content=(@resource[:content]) else File.unlink(@resource[:path]) end end def content File.read(@resource[:path]) end def content=(value) File.write(@resource[:path], value) end end
This gives us the following:
demo@85c63b50bfa3:~$ cat > file_llmethod_create.pp <<EOF $file = '/home/demo/puppet-file-lowlevel-method-create' file {$file: ensure => absent} -> file_llmethod {$file: content => 'This is Sparta!'} EOF demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_llmethod_create.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.01 seconds Notice: /Stage[main]/Main/File_llmethod[/home/demo/puppet-file-lowlevel-method-create]/ensure: defined 'ensure' as 'present' demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_llmethod_create.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.01 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-lowlevel-method-create]/ensure: removed Notice: /Stage[main]/Main/File_llmethod[/home/demo/puppet-file-lowlevel-method-create]/ensure: defined 'ensure' as 'present' demo@85c63b50bfa3:~$ cat > file_llmethod_change.pp <<EOF $file = '/home/demo/puppet-file-lowlevel-method-change' file {$file: content => 'This is madness'} -> file_llmethod {$file: content => 'This is Sparta!'} EOF demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_llmethod_change.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-lowlevel-method-change]/ensure: defined content as '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e' Notice: /Stage[main]/Main/File_llmethod[/home/demo/puppet-file-lowlevel-method-change]/content: content changed 'This is madness' to 'This is Sparta!' demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_llmethod_change.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-lowlevel-method-change]/content: content changed '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2' to '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e' Notice: /Stage[main]/Main/File_llmethod[/home/demo/puppet-file-lowlevel-method-change]/content: content changed 'This is madness' to 'This is Sparta!'
Our custom type already kind of works, even though we have not implemented any explicit comparison of is- and should-state. Puppet does this for us based on the Puppet catalog and the property getter return values. Our defined setters are also invoked by Puppet on demand, only.
We can also see that the ensure
state change notice is defined 'ensure' as 'present'
and does not incorporate the desired content
in any way, while the content
state change notice shows plain text. Both tell us that the SHA256 checksum from the file_builtin.pp
example is already something non-trivial.
Validating input
As a next step we add validation for path
and content
.
# modules/demo/lib/puppet/type/file_llmethod.rb Puppet::Type.newtype(:file_llmethod) do newparam(:path, namevar: true) do validate do |value| fail "#{value} is not a String" unless value.is_a?(String) fail "#{value} is not an absolute path" unless File.absolute_path?(value) end end newproperty(:content) do validate do |value| fail "#{value} is not a String" unless value.is_a?(String) end end newproperty(:ensure) do newvalues(:present, :absent) defaultto(:present) end end
Failed validations will look like these:
demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules --exec 'file_llmethod {"./relative/path": }' Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds Error: Parameter path failed on File_llmethod[./relative/path]: ./relative/path is not an absolute path (line: 1) demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules --exec 'file_llmethod {"/absolute/path": content => 42}' Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds Error: Parameter content failed on File_llmethod[/absolute/path]: 42 is not a String (line: 1)
Content Checksums
We override change_to_s so that state changes include content checksums:
# modules/demo/lib/puppet/type/file_llmethod.rb
require 'digest'
Puppet::Type.newtype(:file_llmethod) do
newparam(:path, namevar: true) do
validate do |value|
fail "#{value} is not a String" unless value.is_a?(String)
fail "#{value} is not an absolute path" unless File.absolute_path?(value)
end
end
newproperty(:content) do
validate do |value|
fail "#{value} is not a String" unless value.is_a?(String)
end
define_method(:change_to_s) do |currentvalue, newvalue|
old = "{sha256}#{Digest::SHA256.hexdigest(currentvalue)}"
new = "{sha256}#{Digest::SHA256.hexdigest(newvalue)}"
"content changed '#{old}' to '#{new}'"
end
end
newproperty(:ensure) do
define_method(:change_to_s
) do |currentvalue, newvalue| if currentvalue == :absent should = @resource.property(:content).should digest = "{sha256}#{Digest::SHA256.hexdigest(should)}" "defined content as '#{digest}'" else super(currentvalue, newvalue) end end newvalues(:present, :absent) defaultto(:present) end end
The above type definition yields:
demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_llmethod_create.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-lowlevel-method-create]/ensure: removed Notice: /Stage[main]/Main/File_llmethod[/home/demo/puppet-file-lowlevel-method-create]/ensure: defined content as '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2' demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_llmethod_change.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-lowlevel-method-change]/content: content changed '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2' to '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e' Notice: /Stage[main]/Main/File_llmethod[/home/demo/puppet-file-lowlevel-method-change]/content: content changed '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e' to '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2'
Improving memory footprint
So far so good. While our current implementation apparently works, it has at least one major flaw. If the managed file already exists, the provider stores the file’s whole content in memory.
demo@85c63b50bfa3:~$ cat > file_llmethod_change_big.pp <<EOF $file = '/home/demo/puppet-file-lowlevel-method-change_big' file_llmethod {$file: content => 'This is Sparta!'} EOF demo@85c63b50bfa3:~$ rm -f /home/demo/puppet-file-lowlevel-method-change_big demo@85c63b50bfa3:~$ ulimit -Sv 200000 demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_llmethod_change_big.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds Notice: /Stage[main]/Main/File_llmethod[/home/demo/puppet-file-lowlevel-method-change_big]/ensure: defined content as '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2' Notice: Applied catalog in 0.02 seconds demo@85c63b50bfa3:~$ dd if=/dev/zero of=/home/demo/puppet-file-lowlevel-method-change_big seek=8G bs=1 count=1 1+0 records in 1+0 records out 1 byte copied, 8.3047e-05 s, 12.0 kB/s demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_llmethod_change_big.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds Error: Could not run: failed to allocate memory
Instead, the implementation should only store the checksum so that Puppet can decide based on checksums if our content=
setter needs to be invoked.
This also means that the Puppet catalog’s content
needs to be checksummed by munge before it it processed by Puppet’s internal comparison routine. Luckily we also have access to the original value via shouldorig.
# modules/demo/lib/puppet/type/file_llmethod.rb require 'digest' Puppet::Type.newtype(:file_llmethod) do newparam(:path, namevar: true) do validate do |value| fail "#{value} is not a String" unless value.is_a?(String) fail "#{value} is not an absolute path" unless File.absolute_path?(value) end end newproperty(:content) do validate do |value| fail "#{value} is not a String" unless value.is_a?(String) end munge do |value| "{sha256}#{Digest::SHA256.hexdigest(value)}" end # No need to override change_to_s with munging end newproperty(:ensure) do define_method(:change_to_s) do |currentvalue, newvalue| if currentvalue == :absent should = @resource.property(:content).should "defined content as '#{should}'" else super(currentvalue, newvalue) end end newvalues(:present, :absent) defaultto(:present) end end
# modules/demo/lib/puppet/provider/file_llmethod/ruby.rb Puppet::Type.type(:file_llmethod).provide(:ruby) do ... def content File.open(@resource[:path], 'r') do |file| sha = Digest::SHA256.new while chunk = file.read(2**16) sha << chunk end "{sha256}#{sha.hexdigest}" end end def content=(value) # value is munged, but we need to write the original File.write(@resource[:path], @resource.parameter(:content).shouldorig[0]) end end
Now we can manage big files:
demo@85c63b50bfa3:~$ ulimit -Sv 200000 demo@85c63b50bfa3:~$ dd if=/dev/zero of=/home/demo/puppet-file-lowlevel-method-change_big seek=8G bs=1 count=1 1+0 records in 1+0 records out 1 byte copied, 9.596e-05 s, 10.4 kB/s demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_llmethod_change_big.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds Notice: /Stage[main]/Main/File_llmethod[/home/demo/puppet-file-lowlevel-method-change_big]/content: content changed '{sha256}ef17a425c57a0e21d14bec2001d8fa762767b97145b9fe47c5d4f2fda323697b' to '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2'
Ensure it the Puppet way
There is still something not right. Maybe you have noticed that our provider’s content
getter attempts to open a file unconditionally, and yet the file_llmethod_create.pp
run has not produced an error. It seems that an ensure
transition from absent
to present
short-circuits the content
getter, even though we have not expressed a wish to do so.
It turns out that an ensure
property gets special treatment by Puppet. If we had attempted to use a makeitso
property instead of ensure
, there would be no short-circuiting and the content
getter would raise an exception.
We will not fix the content
getter though. If Puppet has special treatment for ensure
, we should use Puppet’s intended mechanism for it, and declare the type ensurable:
# modules/demo/lib/puppet/type/file_llmethod.rb require 'digest' Puppet::Type.newtype(:file_llmethod) do ensurable newparam(:path, namevar: true) do validate do |value| fail "#{value} is not a String" unless value.is_a?(String) fail "#{value} is not an absolute path" unless File.absolute_path?(value) end end newproperty(:content) do validate do |value| fail "#{value} is not a String" unless value.is_a?(String) end munge do |value| "{sha256}#{Digest::SHA256.hexdigest(value)}" end end end
With ensurable
the provider needs to implement three new methods, but we can drop the ensure
accessors:
# modules/demo/lib/puppet/provider/file_llmethod/ruby.rb Puppet::Type.type(:file_llmethod).provide(:ruby) do def exists? File.exist?(@resource[:name]) end def create self.content=(:dummy) end def destroy File.unlink(@resource[:name]) end def content File.open(@resource[:path], 'r') do |file| sha = Digest::SHA256.new while chunk = file.read(2**16) sha << chunk end "{sha256}#{sha.hexdigest}" end end def content=(value) # value is munged, but we need to write the original File.write(@resource[:path], @resource.parameter(:content).shouldorig[0]) end end
However, now we have lost the SHA256 checksum on file creation:
demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_llmethod_create.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-lowlevel-method-create]/ensure: removed Notice: /Stage[main]/Main/File_llmethod[/home/demo/puppet-file-lowlevel-method-create]/ensure: created
To get it back, we replace ensurable
by an adapted implementation of it, which includes our previous change_to_s
override:
newproperty(:ensure, :parent => Puppet::Property::Ensure) do defaultvalues define_method(:change_to_s) do |currentvalue, newvalue| if currentvalue == :absent should = @resource.property(:content).should "defined content as '#{should}'" else super(currentvalue, newvalue) end end end
demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_llmethod_create.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-lowlevel-method-create]/ensure: removed Notice: /Stage[main]/Main/File_llmethod[/home/demo/puppet-file-lowlevel-method-create]/ensure: defined content as '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2'
Our final low-level prototype is thus as follows.
Final low-level prototype
# modules/demo/lib/puppet/type/file_llmethod.rb # frozen_string_literal: true require 'digest' Puppet::Type.newtype(:file_llmethod) do newparam(:path, namevar: true) do validate do |value| fail "#{value} is not a String" unless value.is_a?(String) fail "#{value} is not an absolute path" unless File.absolute_path?(value) end end newproperty(:content) do validate do |value| fail "#{value} is not a String" unless value.is_a?(String) end munge do |value| "{sha256}#{Digest::SHA256.hexdigest(value)}" end end newproperty(:ensure, :parent => Puppet::Property::Ensure) do defaultvalues define_method(:change_to_s) do |currentvalue, newvalue| if currentvalue == :absent should = @resource.property(:content).should "defined content as '#{should}'" else super(currentvalue, newvalue) end end end end
# modules/demo/lib/puppet/provider/file_llmethod/ruby.rb # frozen_string_literal: true Puppet::Type.type(:file_llmethod).provide(:ruby) do def exists? File.exist?(@resource[:name]) end def create self.content=(:dummy) end def destroy File.unlink(@resource[:name]) end def content File.open(@resource[:path], 'r') do |file| sha = Digest::SHA256.new while (chunk = file.read(2**16)) sha << chunk end "{sha256}#{sha.hexdigest}" end end def content=(_value) # value is munged, but we need to write the original File.write(@resource[:path], @resource.parameter(:content).shouldorig[0]) end end
Resource-API prototype
According to the Resource-API documentation we need to define our new file_rsapi
type by calling Puppet::ResourceApi.register_type
with several parameters, amongst which are the desired attributes, even ensure
.
# modules/demo/lib/puppet/type/file_rsapi.rb require 'puppet/resource_api' Puppet::ResourceApi.register_type( name: 'file_rsapi', attributes: { content: { desc: 'description of content parameter', type: 'String' }, ensure: { default: 'present', desc: 'description of ensure parameter', type: 'Enum[present, absent]' }, path: { behaviour: :namevar, desc: 'description of path parameter', type: 'Pattern[/\A\/([^\n\/\0]+\/*)*\z/]' }, }, desc: 'description of file_rsapi' )
The path
type
uses a built-in Puppet data type. Stdlib::Absolutepath
would have been more convenient but external data types are not possible with the Resource-API yet.
In comparison with our low-level prototype, the above type definition has no SHA256-munging and SHA256-output counterparts. The canonicalize
provider feature looks similar to munging, but we skip it for now.
The Resource-API documentation tells us to implement a get
and a set
method in our provider, stating
The get method reports the current state of the managed resources. It returns an enumerable of all existing resources. Each resource is a hash with attribute names as keys, and their respective values as values.
This demand is the first bummer, as we definitely do not want to read all files with their content and store it in memory. We can ignore this demand – how would the Resource-API know anyway.
However, the documented signature is def get(context) {...}
where context
has no information about the resource we want to manage.
This would have been a show-stopper, if the simple_get_filter provider feature didn’t exist, which changes the signature to def get(context, names = nil) {...}
.
Our first version of file_rsapi
is thus the following.
Basic functionality
# modules/demo/lib/puppet/type/file_rsapi.rb require 'puppet/resource_api' Puppet::ResourceApi.register_type( name: 'file_rsapi', features: %w[simple_get_filter], attributes: { content: { desc: 'description of content parameter', type: 'String' }, ensure: { default: 'present', desc: 'description of ensure parameter', type: 'Enum[present, absent]' }, path: { behaviour: :namevar, desc: 'description of path parameter', type: 'Pattern[/\A\/([^\n\/\0]+\/*)*\z/]' }, }, desc: 'description of file_rsapi' )
# modules/demo/lib/puppet/provider/file_rsapi/file_rsapi.rb require 'digest' class Puppet::Provider::FileRsapi::FileRsapi def get(context, names) (names or []).map do |name| File.exist?(name) ? { path: name, ensure: 'present', content: filedigest(name), } : nil end.compact # remove non-existing resources end def set(context, changes) changes.each do |path, change| if change[:should][:ensure] == 'present' File.write(path, change[:should][:content]) elsif File.exist?(path) File.delete(path) end end end def filedigest(path) File.open(path, 'r') do |file| sha = Digest::SHA256.new while chunk = file.read(2**16) sha << chunk end "{sha256}#{sha.hexdigest}" end end end
The desired content
is written correctly into the file, but we have again no SHA256 checksum on creation as well as unnecessary writes, because the checksum from get
does not match the cleartext from the catalog:
demo@85c63b50bfa3:~$ cat > file_rsapi_create.pp <<EOF $file = '/home/demo/puppet-file-rsapi-create' file {$file: ensure => absent} -> file_rsapi {$file: content => 'This is Sparta!'} EOF demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_rsapi_create.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.03 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-rsapi-create]/ensure: removed Notice: /Stage[main]/Main/File_rsapi[/home/demo/puppet-file-rsapi-create]/ensure: defined 'ensure' as 'present' Notice: Applied catalog in 0.02 seconds demo@85c63b50bfa3:~$ cat > file_rsapi_change.pp <<EOF $file = '/home/demo/puppet-file-rsapi-change' file {$file: content => 'This is madness'} -> file_rsapi {$file: content => 'This is Sparta!'} EOF demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_rsapi_change.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.03 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-rsapi-change]/content: content changed '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2' to '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e' Notice: /Stage[main]/Main/File_rsapi[/home/demo/puppet-file-rsapi-change]/content: content changed '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e' to 'This is Sparta!' Notice: Applied catalog in 0.03 seconds
Hence we enable and implement the canonicalize
provider feature.
Canonicalize
According to the documentation, canonicalize
is applied to the results of get
as well as to the catalog properties. On the one hand, we do not want to read the file’s content into memory, on the other hand cannot checksum the file’s content twice.
An easy way would be to check whether the canonicalized
call happens after the get
call:
def canonicalize(context, resources) if @stage == :get # do nothing, is-state canonicalization already performed by get() else # catalog canonicalization ... end end def get(context, paths) @stage = :get ... end
While this works for the current implementation of the Resource-API, there is no guarantee about the order of canonicalize
calls. Instead we subclass from String
and handle checksumming internally. We also add some state change calls to the appropriate context
methods.
Our final Resource-API-based prototype implementation is:
# modules/demo/lib/puppet/type/file_rsapi.rb # frozen_string_literal: true require 'puppet/resource_api' Puppet::ResourceApi.register_type( name: 'file_rsapi', features: %w[simple_get_filter canonicalize], attributes: { content: { desc: 'description of content parameter', type: 'String' }, ensure: { default: 'present', desc: 'description of ensure parameter', type: 'Enum[present, absent]' }, path: { behaviour: :namevar, desc: 'description of path parameter', type: 'Pattern[/\A\/([^\n\/\0]+\/*)*\z/]' } }, desc: 'description of file_rsapi' )
# modules/demo/lib/puppet/provider/file_rsapi/file_rsapi.rb # frozen_string_literal: true require 'digest' require 'pathname' class Puppet::Provider::FileRsapi::FileRsapi class CanonicalString < String attr_reader :original def class # Mask as String for YAML.dump to mitigate # Error: Transaction store file /var/cache/puppet/state/transactionstore.yaml # is corrupt ((/var/cache/puppet/state/transactionstore.yaml): Tried to # load unspecified class: Puppet::Provider::FileRsapi::FileRsapi::CanonicalString) String end def self.from(obj) return obj if obj.is_a?(self) return new(filedigest(obj)) if obj.is_a?(Pathname) new("{sha256}#{Digest::SHA256.hexdigest(obj)}", obj) end def self.filedigest(path) File.open(path, 'r') do |file| sha = Digest::SHA256.new while (chunk = file.read(2**16)) sha << chunk end "{sha256}#{sha.hexdigest}" end end def initialize(canonical, original = nil) @original = original super(canonical) end end def canonicalize(_context, resources) resources.each do |resource| next if resource[:ensure] == 'absent' resource[:content] = CanonicalString.from(resource[:content]) end resources end def get(_context, names) (names or []).map do |name| next unless File.exist?(name) { content: CanonicalString.from(Pathname.new(name)), ensure: 'present', path: name } end.compact # remove non-existing resources end def set(context, changes) changes.each do |path, change| if change[:should][:ensure] == 'present' File.write(path, change[:should][:content].original) if change[:is][:ensure] == 'present' # The only other possible change is due to content, # but content change transition info is covered implicitly else context.created("#{path} with content '#{change[:should][:content]}'") end elsif File.exist?(path) File.delete(path) context.deleted(path) end end end end
It gives us:
demo@85c63b50bfa3:~$ cat > file_rsapi_create.pp <<EOF $file = '/home/demo/puppet-file-rsapi-create' file {$file: ensure => absent} -> file_rsapi {$file: content => 'This is Sparta!'} EOF demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_rsapi_create.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.03 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-rsapi-create]/ensure: removed Notice: /Stage[main]/Main/File_rsapi[/home/demo/puppet-file-rsapi-create]/ensure: defined 'ensure' as 'present' Notice: file_rsapi: Created: /home/demo/puppet-file-rsapi-create with content '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2' demo@85c63b50bfa3:~$ cat > file_rsapi_change.pp <<EOF $file = '/home/demo/puppet-file-rsapi-change' file {$file: content => 'This is madness'} -> file_rsapi {$file: content => 'This is Sparta!'} EOF demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_rsapi_change.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.03 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-rsapi-change]/content: content changed '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2' to '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e' Notice: /Stage[main]/Main/File_rsapi[/home/demo/puppet-file-rsapi-change]/content: content changed '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e' to '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2' demo@85c63b50bfa3:~$ cat > file_rsapi_remove.pp <<EOF $file = '/home/demo/puppet-file-rsapi-remove' file {$file: ensure => present} -> file_rsapi {$file: ensure => absent} EOF demo@85c63b50bfa3:~$ bin/puppet apply --modulepath modules file_rsapi_remove.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.03 seconds Notice: /Stage[main]/Main/File[/home/demo/puppet-file-rsapi-remove]/ensure: created Notice: /Stage[main]/Main/File_rsapi[/home/demo/puppet-file-rsapi-remove]/ensure: undefined 'ensure' from 'present' Notice: file_rsapi: Deleted: /home/demo/puppet-file-rsapi-remove
Thoughts
The apparent need for a CanonicalString
masking as String
makes it look like we are missing something. If the Resource-API only checked data types after canonicalization, we could make get
return something simpler than CanonicalString
to signal an already canonicalized value.
The Resource-API’s default demand to return all existing resources simplifies development when one had this plan anyway. The low-level version of doing so is often a combination of prefetch and instances.
The administration of a large number of servers can be quite tiresome without a central configuration management. This article gives a first introduction into the configuration management tool, Puppet.
Introduction
In our daily work at the Open Source Support Center we maintain a large number of servers. Managing larger clusters or setups means maintaining dozens of machines with an almost identical configuration and only slight variations, if any. Without central configuration management, making small changes to the configuration would mean repeating the same step on all machines. This is where Puppet comes into play.
As with all configuration management tools, Puppet uses a central server which manages the configuration. The clients query the server on a regular basis for new configuration via an encrypted connection. If a new configuration is found, it is imported as the server instructs: the client imports new files, modifies rights, starts services and executes commands, whatever the server says. The advantages are obvious:
- Each configuration change is done only once, regardless of the actual number of maintained servers. Unnecessary – and pretty boring – repetition is avoided, lucky us!
- The configuration is streamlined for all machines, which makes it much easier to maintain.
- A central infrastructure makes it easier to quickly get an overview about the setup – “running around” is not necessary anymore.
- Last but not least, a central configuration tree enables you to incorporate a simple version control of your configuration: for example, playing back the configuration “PRE-UPDATE” on all machines of an entire setup only takes a couple of commands!
Technical workflow
Puppet consists of a central server, called “Puppet Master”, and the clients, called “Nodes”. The nodes query the master for the current configuration. The master responds with a list of configuration and management items: files, services which have to be running, commands which need to be executed, and so on – the possibilities are practically endless:
- The master can hand over files which the node copies to a defined place – if it does not already exist.
- The node is asked to check certain file and directory permissions and to correct them if necessary.
- Depending upon the operating system, the node checks the state of services and starts or stops them. It can also check for installed packages and if they are up to date.
- The master can force the node to execute arbitrary commands
- etc.
Of course, in general all tasks can be fulfilled by handing over files from the master to the client. However, in more complex setups this kind of behaviour is not easily arranged, nor does it simplify the setup. Puppet’s strength is that it facilitates abstract system tasks (restart services, ensure installed packages, add users, etc.), regardless of the actual changed files in the background. You can even use the same statement in Puppet to configure different versions of Linux or Unix.
Installation
First, you need the master, the center of all the configuration you want to manage: apt-get install puppetmaster
Puppet expects that all machines in the network have FQDNs – but that should be the case anyway in a well maintained network.
Other machines become a node by installing the Puppet client: apt-get install puppet
Puppet, main configuration
The Puppet nodes do not need to be configured – they will check for a machine called Puppet in the local network. As long as that name points to the master you do not have to do anything else.
Since the master provides files to the nodes, the internal file server must be configured accordingly. There are different solutions for the internal file server, depending on the needs of your setup. For example, it might be better for your setup to store all files you provide to the nodes on one place, and the actual configuration you provide to the nodes somewhere else. However, in our example we keep the files and the configuration for the nodes close, as it is outlined in Puppet’s Best Practice Guide and in the Module Configuration part of the Puppet documentation. Thus, it is enough to change the file /etc/puppet/fileserver.conf to:
[modules] allow 192.168.0.1/24 allow *.credativ.de
Configuration of the configuration – Modules
Puppet’s way of managing configuration is to use sets of tasks grouped by topic. For example, all tasks related to SSH should go into the module “ssh”, while all tasks related to apache should be placed in the module “apache” and so on. These sets of tasks are called “Modules” and are the core of Puppet – in a perfect Puppet setup everything is defined in modules! We will explain the structure of a SSH module to highlight the basics and ideas behind Puppet’s modules. We will also try to stay close to the Best Practise Guide to make it easier to check back against the Puppet documentation.
Please note, however, that this example is an example: in a real world setup the SSH configuration would be a bit more dynamic, but we focused on simple and easy-to-understand methods.
The SSH module
We have the following requirements:
- The package open-ssh must be installed and be the newest version.
- Each node’s sshd_config file has to be the same as the one saved on the master.
- In the event that the sshd_config is changed on any node, the sshd service should be restarted.
- The user credativ needs to have certain files in his/her directory $HOME/.ssh.
To comply with these requirements we start by creating some necessary paths:
mkdir -p /etc/puppet/modules/ssh/manifests mkdir -p /etc/puppet/modules/ssh/files
The directory “manifests” contains the actual configuration instructions of the module and the directory “files” provides the files we hand over to the clients.
The instructions themselves are written down in init.pp in the “manifests” directory. The set of instructions to fulfil aims 1 – 4 are grouped in a so called “class”. For each task a “class” has one subsection, a type. So in our case we have four types, one for each aim:
class ssh{ package { "openssh-server": ensure => latest, } file { "/etc/ssh/sshd_config": owner => root, group => root, mode => 644, source => "puppet:///ssh/sshd_config", } service { ssh: ensure => running, hasrestart => true, subscribe => File["/etc/ssh/sshd_config"], } file { "/home/credativ/.ssh": path => "/home/credativ/.ssh", owner => "credativ", group => "credativ", mode => 600, recurse => true, source => "puppet:///ssh/ssh", ensure => [directory, present], } }
Each type is another task and calls another action on the node:
package
Here we make sure that the package openssh-server is installed in the newest version.
file
A file on the node is compared with the version on the server and overwritten if necessary. Also, the rights are adjusted.
service
Well, as the name says, this deals with services: in our case the service must be running on the node. Also, in case the file /etc/ssh/sshd_config is modified, the service is restarted automatically.
file
Here we have again the file type, but this time we do not compare a file, but an entire directory.
As mentioned above, the files and directories you configured so that the server provides them to the nodes must be available in the directory /etc/puppet/modules/ssh/files/.
Nodes and modules
We now have three parts: the master, the nodes and the modules. The next step is to tell the master which nodes are related to which modules. First, you must tell the master that this module exists in /etc/puppet/manifests/modules.pp:
import "ssh"
Next, you need to modify /etc/puppet/manifests/nodes.pp. This specifies which module is loaded for which node, and which modules should be loaded as default in the event that a node does not have a special entry. The entries for the nodes support inheritance.
So, for example, to have the module “rsyslog” ready for all nodes but the module “ssh” only ready for the node “external” you need the following entry:
node default { include rsyslog } node 'external' inherits default { include ssh }
Puppet is now configured!
Certificates – secured communication between nodes and master
As mentioned above, the communication between master and node is encrypted. But that implies you have to verify the partners at least once. This can be done after a node queries the master for the first time. Whenever the master is queried by an unknown node it does not provide the default configuration but instead puts the node on a waiting list. You can check the waiting list with the command: # puppetca --list
To verify a node and incorporate it into the Puppet system you need to verify it: # puppetca --sign external.example.com
The entire process is explained in more detail in the puppet doceumentation.
Closing words
The example introduced in this article is very simple – as I noted, a real world example would be more complex and dynamic. However, it is a good way to start with Puppet, and the documentation linked throughout this article will help the willing reader to dive deeper into the components of Puppet.
We, here at credativ’s Open Source Support Center have gained considerable experience with Puppet in recent years and really like the framework. Also, in our day to day support and consulting work we see the market growing as more and more customers are interested in the framework. Right now, Puppet is in the fast lane and it will be interesting to see how more established solutions like cfengine will react to this competition.
This post was originally written by Roland Wolters.