Puppet is a software configuration management solution to manage IT infrastructure. One of the first things to be learnt about Puppet is its domain-specific language – the Puppet-DSL – and the concepts that come with it.<\/p>\n
Users can organize their code in classes and modules and use pre-defined resource types to manage resources like files<\/a>, packages<\/a>, services<\/a> and others.<\/p>\n The most commonly used types are part of the Puppet core, implemented in Ruby. Composite resource types may be defined via Puppet-DSL from already known types by the Puppet user themself, or imported as part of an external Puppet module which is maintained by external module developers.<\/p>\n It so happens that Puppet users can stay within the Puppet-DSL for a long time even when they deal with Puppet on a regular basis.<\/p>\n The first time I had a glimpse into this topic was when Debian Stable was shipping Puppet 5.5, which was not too long ago. The Puppet 5.5 documentation includes a chapter on custom types<\/a> and provider development<\/a> respectively, but to me they felt incomplete and lacking self contained examples. Apparently I was not the only one feeling that way<\/a>, even though Puppet’s gem documentation<\/a> is a good overview of what is possible in principle.<\/p>\n Gary Larizza’s blog post was more than ten years ago. I had another look into the documentation for Puppet 7 on that topic<\/a> recently, as this is the Puppet version in current’s Debian Stable.<\/p>\n The Puppet 5.5 way to type & provider development is now called the low level method<\/a>, and its documentation has not changed significantly. However, Puppet 6 upwards recommends a new method to create custom types & providers<\/a> via the so-called Resource-API<\/a>, whose documentation<\/a> is a major improvement compared to the low-level method’s. The Resource-API is not a replacement, though, and has several documented limitations<\/a>.<\/p>\n Nevertheless, for the remaining blog post, we will re-prototype a small portion of The following preparations are not necessary in an agent-server setup. We use Keep in mind the state change information printed by Puppet.<\/p>\n Custom types and providers that are not installed via a Gem need to be part of some Puppet module, so they can be copied to Puppet agents via the pluginsync mechanism<\/a>.<\/p>\n A common location for Puppet modules is the Our first attempt is the following type definition for a new type we will call We have declared a The following provider implementation consists of a getter and a setter for each of the two properties This gives us the following:<\/p>\n Our custom type already kind of works, even though we have not implemented any explicit comparison of is- and should-state. Puppet does this for us based on the Puppet catalog and the property getter return values. Our defined setters are also invoked by Puppet on demand, only.<\/p>\n We can also see that the As a next step we add validation<\/a> for Failed validations will look like these:<\/p>\n We override change_to_s<\/a> so that state changes include content checksums:<\/p>\n The above type definition yields:<\/p>\n So far so good. While our current implementation apparently works, it has at least one major flaw. If the managed file already exists, the provider stores the file’s whole content in memory.<\/p>\n Instead, the implementation should only store the checksum so that Puppet can decide based on checksums if our This also means that the Puppet catalog’s Now we can manage big files:<\/p>\n There is still something not right. Maybe you have noticed that our provider’s It turns out that an We will not fix the With However, now we have lost the SHA256 checksum on file creation:<\/p>\n To get it back, we replace Our final low-level prototype is thus as follows.<\/p>\n According to the Resource-API documentation<\/a> we need to define our new The In comparison with our low-level prototype, the above type definition has no SHA256-munging and SHA256-output counterparts. The The Resource-API documentation<\/a> tells us to implement a The get method reports the current state of the managed resources. It returns an enumerable of all existing resources<\/b>. Each resource is a hash with attribute names as keys, and their respective values as values.<\/p><\/blockquote>\n This demand is the first bummer, as we definitely do not want to read all files with their content and store it in memory. We can ignore this demand – how would the Resource-API know anyway.<\/p>\n However, the documented signature is This would have been a show-stopper, if the simple_get_filter<\/a> provider feature didn’t exist, which changes the signature to Our first version of The desired Hence we enable and implement the According to the documentation, An easy way would be to check whether the While this works for the current implementation of the Resource-API, there is no guarantee about the order of Our final Resource-API-based prototype implementation is:<\/p>\n It gives us:<\/p>\n The apparent need for a The Resource-API’s default demand to return all existing resources simplifies development when one had this plan anyway. The low-level version of doing so is often a combination of prefetch<\/a> and instances<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" Introduction Puppet is a software configuration management solution to manage IT infrastructure. One of the first things to be learnt about Puppet is its domain-specific language – the Puppet-DSL – and the concepts that come with it. Users can organize their code in classes and modules and use pre-defined resource types to manage resources like […]<\/p>\n","protected":false},"author":20,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_improvement_type_select":"improve_an_existing","_thumb_yes_seoaic":false,"_frame_yes_seoaic":false,"seoaic_generate_description":"","seoaic_improve_instructions_prompt":"","seoaic_rollback_content_improvement":"","seoaic_idea_thumbnail_generator":"","thumbnail_generated":false,"thumbnail_generate_prompt":"","seoaic_article_description":"","seoaic_article_subtitles":[],"footnotes":""},"categories":[2053,1885],"tags":[1808],"class_list":["post-8543","post","type-post","status-publish","format-standard","hentry","category-automation","category-howtos-en","tag-puppet-en"],"acf":[],"yoast_head":"\nfile<\/code>s functionality using the low-level method, as well as the Resource-API, namely the ensure<\/code> and content<\/code> properties.<\/p>\nPreparations<\/h3>\n
bundle<\/code> to obtain a puppet<\/code> executable for this demo.<\/p>\ndemo@85c63b50bfa3:~$ cat > Gemfile <<EOF\r\nsource 'https:\/\/rubygems.org'\r\n\r\ngem 'puppet', '>= 6'\r\nEOF\r\n<\/pre>\n
demo@85c63b50bfa3:~$ bundle install\r\nFetching gem metadata from https:\/\/rubygems.org\/........\r\nResolving dependencies...\r\n...\r\nInstalling puppet 8.10.0\r\nBundle complete! 1 Gemfile dependency, 17 gems now installed.\r\nBundled gems are installed into `.\/.vendor`\r\n<\/pre>\n
demo@85c63b50bfa3:~$ cat > file_builtin.pp <<EOF\r\n$file = '\/home\/demo\/puppet-file-builtin'\r\nfile {$file: content => 'This is madness'}\r\nEOF\r\n<\/pre>\ndemo@85c63b50bfa3:~$ bin\/puppet apply file_builtin.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.01 seconds\r\nNotice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-builtin]\/ensure: defined content as '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e'\r\nNotice: Applied catalog in 0.04 seconds\r\n\r\ndemo@85c63b50bfa3:~$ sha256sum \/home\/demo\/puppet-file-builtin\r\n0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e \/home\/demo\/puppet-file-builtin\r\n<\/pre>\nLow-level prototype<\/h3>\n
modules<\/code> directory inside a Puppet environment<\/a>. For this demo, we declare a demo<\/code> module.<\/p>\nBasic functionality<\/h4>\n
file_llmethod<\/code>. It has no documentation or validation of input values.<\/p>\n# modules\/demo\/lib\/puppet\/type\/file_llmethod.rb\r\n\r\nPuppet::Type.newtype(:file_llmethod) do\r\n newparam(:path, namevar: true) {}\r\n newproperty(:content) {}\r\n newproperty(:ensure) do\r\n newvalues(:present, :absent)\r\n defaultto(:present)\r\n end\r\nend\r\n<\/pre>\npath<\/code> parameter that serves as the namevar<\/code> for this type – there cannot be other file_llmethod<\/code> instances managing the same path<\/code>. The ensure<\/code> property is restricted to two values and defaults to present<\/code>.<\/p>\ncontent<\/code> and ensure<\/code>.<\/p>\n# modules\/demo\/lib\/puppet\/provider\/file_llmethod\/ruby.rb\r\n\r\nPuppet::Type.type(:file_llmethod).provide(:ruby) do\r\n def ensure\r\n File.exist?(@resource[:path]) ? :present : :absent\r\n end\r\n\r\n def ensure=(value)\r\n if value == :present\r\n # reuse setter\r\n self.content=(@resource[:content])\r\n else\r\n File.unlink(@resource[:path])\r\n end\r\n end\r\n\r\n def content\r\n File.read(@resource[:path])\r\n end\r\n\r\n def content=(value)\r\n File.write(@resource[:path], value)\r\n end\r\nend\r\n<\/pre>\n
demo@85c63b50bfa3:~$ cat > file_llmethod_create.pp <<EOF\r\n$file = '\/home\/demo\/puppet-file-lowlevel-method-create'\r\nfile {$file: ensure => absent} ->\r\nfile_llmethod {$file: content => 'This is Sparta!'}\r\nEOF\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_llmethod_create.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.01 seconds\r\nNotice: \/Stage[main]\/Main\/File_llmethod[\/home\/demo\/puppet-file-lowlevel-method-create]\/ensure: defined 'ensure' as 'present'\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_llmethod_create.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.01 seconds\r\nNotice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-lowlevel-method-create]\/ensure: removed\r\nNotice: \/Stage[main]\/Main\/File_llmethod[\/home\/demo\/puppet-file-lowlevel-method-create]\/ensure: defined 'ensure' as 'present'\r\n\r\ndemo@85c63b50bfa3:~$ cat > file_llmethod_change.pp <<EOF\r\n$file = '\/home\/demo\/puppet-file-lowlevel-method-change'\r\nfile {$file: content => 'This is madness'} ->\r\nfile_llmethod {$file: content => 'This is Sparta!'}\r\nEOF\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_llmethod_change.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds\r\nNotice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-lowlevel-method-change]\/ensure: defined content as '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e'\r\nNotice: \/Stage[main]\/Main\/File_llmethod[\/home\/demo\/puppet-file-lowlevel-method-change]\/content: content changed 'This is madness' to 'This is Sparta!'\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_llmethod_change.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds\r\nNotice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-lowlevel-method-change]\/content: content changed '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2' to '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e'\r\nNotice: \/Stage[main]\/Main\/File_llmethod[\/home\/demo\/puppet-file-lowlevel-method-change]\/content: content changed 'This is madness' to 'This is Sparta!'\r\n<\/pre>\nensure<\/code> state change notice is defined 'ensure' as 'present'<\/code> and does not incorporate the desired content<\/code> in any way, while the content<\/code> state change notice shows plain text. Both tell us that the SHA256 checksum from the file_builtin.pp<\/code> example is already something non-trivial.<\/p>\nValidating input<\/h4>\n
path<\/code> and content<\/code>.<\/p>\n# modules\/demo\/lib\/puppet\/type\/file_llmethod.rb\r\n\r\nPuppet::Type.newtype(:file_llmethod) do\r\n newparam(:path, namevar: true) do\r\n validate do |value|\r\n fail \"#{value} is not a String\" unless value.is_a?(String)\r\n fail \"#{value} is not an absolute path\" unless File.absolute_path?(value)\r\n end\r\n end\r\n\r\n newproperty(:content) do\r\n validate do |value|\r\n fail \"#{value} is not a String\" unless value.is_a?(String)\r\n end\r\n end\r\n\r\n newproperty(:ensure) do\r\n newvalues(:present, :absent)\r\n defaultto(:present)\r\n end\r\nend\r\n<\/pre>\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules --exec 'file_llmethod {\".\/relative\/path\": }'\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds\r\nError: Parameter path failed on File_llmethod[.\/relative\/path]: .\/relative\/path is not an absolute path (line: 1)\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules --exec 'file_llmethod {\"\/absolute\/path\": content => 42}'\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds\r\nError: Parameter content failed on File_llmethod[\/absolute\/path]: 42 is not a String (line: 1)\r\n<\/pre>\nContent Checksums<\/h4>\n
# modules\/demo\/lib\/puppet\/type\/file_llmethod.rb\r\n\r\nrequire 'digest'\r\n\r\nPuppet::Type.newtype(:file_llmethod) do\r\n newparam(:path, namevar: true) do\r\n validate do |value|\r\n fail \"#{value} is not a String\" unless value.is_a?(String)\r\n fail \"#{value} is not an absolute path\" unless File.absolute_path?(value)\r\n end\r\n end\r\n\r\n newproperty(:content) do\r\n validate do |value|\r\n fail \"#{value} is not a String\" unless value.is_a?(String)\r\n end\r\n define_method(:change_to_s) do |currentvalue, newvalue|\r\n old = \"{sha256}#{Digest::SHA256.hexdigest(currentvalue)}\"\r\n new = \"{sha256}#{Digest::SHA256.hexdigest(newvalue)}\"\r\n \"content changed '#{old}' to '#{new}'\"\r\n end\r\n end\r\n\r\n newproperty(:ensure) do\r\n define_method(:change_to_s<\/code>) do |currentvalue, newvalue| if currentvalue == :absent should = @resource.property(:content).should digest = \"{sha256}#{Digest::SHA256.hexdigest(should)}\" \"defined content as '#{digest}'\" else super(currentvalue, newvalue) end end newvalues(:present, :absent) defaultto(:present) end end<\/pre>\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_llmethod_create.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds\r\nNotice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-lowlevel-method-create]\/ensure: removed\r\nNotice: \/Stage[main]\/Main\/File_llmethod[\/home\/demo\/puppet-file-lowlevel-method-create]\/ensure: defined content as '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2'\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_llmethod_change.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds\r\nNotice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-lowlevel-method-change]\/content: content changed '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2' to '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e'\r\nNotice: \/Stage[main]\/Main\/File_llmethod[\/home\/demo\/puppet-file-lowlevel-method-change]\/content: content changed '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e' to '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2'\r\n<\/pre>\nImproving memory footprint<\/h4>\n
demo@85c63b50bfa3:~$ cat > file_llmethod_change_big.pp <<EOF\r\n$file = '\/home\/demo\/puppet-file-lowlevel-method-change_big'\r\nfile_llmethod {$file: content => 'This is Sparta!'}\r\nEOF\r\n\r\ndemo@85c63b50bfa3:~$ rm -f \/home\/demo\/puppet-file-lowlevel-method-change_big\r\ndemo@85c63b50bfa3:~$ ulimit -Sv 200000\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_llmethod_change_big.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds\r\nNotice: \/Stage[main]\/Main\/File_llmethod[\/home\/demo\/puppet-file-lowlevel-method-change_big]\/ensure: defined content as '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2'\r\nNotice: Applied catalog in 0.02 seconds\r\n\r\ndemo@85c63b50bfa3:~$ dd if=\/dev\/zero of=\/home\/demo\/puppet-file-lowlevel-method-change_big seek=8G bs=1 count=1\r\n1+0 records in\r\n1+0 records out\r\n1 byte copied, 8.3047e-05 s, 12.0 kB\/s\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_llmethod_change_big.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds\r\nError: Could not run: failed to allocate memory\r\n<\/pre>\ncontent=<\/code> setter needs to be invoked.<\/p>\ncontent<\/code> needs to be checksummed by munge<\/a> before it it processed by Puppet’s internal comparison routine. Luckily we also have access to the original value via shouldorig<\/a>.<\/p>\n# modules\/demo\/lib\/puppet\/type\/file_llmethod.rb\r\n\r\nrequire 'digest'\r\n\r\nPuppet::Type.newtype(:file_llmethod) do\r\n newparam(:path, namevar: true) do\r\n validate do |value|\r\n fail \"#{value} is not a String\" unless value.is_a?(String)\r\n fail \"#{value} is not an absolute path\" unless File.absolute_path?(value)\r\n end\r\n end\r\n\r\n newproperty(:content) do\r\n validate do |value|\r\n fail \"#{value} is not a String\" unless value.is_a?(String)\r\n end\r\n munge do |value|\r\n \"{sha256}#{Digest::SHA256.hexdigest(value)}\"\r\n end\r\n # No need to override change_to_s with munging\r\n end\r\n\r\n newproperty(:ensure) do\r\n define_method(:change_to_s) do |currentvalue, newvalue|\r\n if currentvalue == :absent\r\n should = @resource.property(:content).should\r\n \"defined content as '#{should}'\"\r\n else\r\n super(currentvalue, newvalue)\r\n end\r\n end\r\n newvalues(:present, :absent)\r\n defaultto(:present)\r\n end\r\nend\r\n<\/pre>\n# modules\/demo\/lib\/puppet\/provider\/file_llmethod\/ruby.rb\r\n\r\nPuppet::Type.type(:file_llmethod).provide(:ruby) do\r\n ...\r\n\r\n def content\r\n File.open(@resource[:path], 'r') do |file|\r\n sha = Digest::SHA256.new\r\n while chunk = file.read(2**16)\r\n sha << chunk\r\n end\r\n \"{sha256}#{sha.hexdigest}\"\r\n end\r\n end\r\n\r\n def content=(value)\r\n # value is munged, but we need to write the original\r\n File.write(@resource[:path], @resource.parameter(:content).shouldorig[0])\r\n end\r\nend\r\n<\/pre>\ndemo@85c63b50bfa3:~$ ulimit -Sv 200000\r\ndemo@85c63b50bfa3:~$ dd if=\/dev\/zero of=\/home\/demo\/puppet-file-lowlevel-method-change_big seek=8G bs=1 count=1\r\n1+0 records in\r\n1+0 records out\r\n1 byte copied, 9.596e-05 s, 10.4 kB\/s\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_llmethod_change_big.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds\r\nNotice: \/Stage[main]\/Main\/File_llmethod[\/home\/demo\/puppet-file-lowlevel-method-change_big]\/content: content changed '{sha256}ef17a425c57a0e21d14bec2001d8fa762767b97145b9fe47c5d4f2fda323697b' to '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2'\r\n<\/pre>\nEnsure it the Puppet way<\/h4>\n
content<\/code> getter attempts to open a file unconditionally, and yet the file_llmethod_create.pp<\/code> run has not produced an error. It seems that an ensure<\/code> transition from absent<\/code> to present<\/code> short-circuits the content<\/code> getter, even though we have not expressed a wish to do so.<\/p>\nensure<\/code> property gets special treatment by Puppet. If we had attempted to use a makeitso<\/code> property instead of ensure<\/code>, there would be no short-circuiting and the content<\/code> getter would raise an exception.<\/p>\ncontent<\/code> getter though. If Puppet has special treatment for ensure<\/code>, we should use Puppet’s intended mechanism for it, and declare the type ensurable<\/a>:<\/p>\n# modules\/demo\/lib\/puppet\/type\/file_llmethod.rb\r\n\r\nrequire 'digest'\r\n\r\nPuppet::Type.newtype(:file_llmethod) do\r\n ensurable\r\n\r\n newparam(:path, namevar: true) do\r\n validate do |value|\r\n fail \"#{value} is not a String\" unless value.is_a?(String)\r\n fail \"#{value} is not an absolute path\" unless File.absolute_path?(value)\r\n end\r\n end\r\n\r\n newproperty(:content) do\r\n validate do |value|\r\n fail \"#{value} is not a String\" unless value.is_a?(String)\r\n end\r\n munge do |value|\r\n \"{sha256}#{Digest::SHA256.hexdigest(value)}\"\r\n end\r\n end\r\nend\r\n<\/pre>\nensurable<\/code> the provider needs to implement three new methods, but we can drop the ensure<\/code> accessors:<\/p>\n# modules\/demo\/lib\/puppet\/provider\/file_llmethod\/ruby.rb\r\n\r\nPuppet::Type.type(:file_llmethod).provide(:ruby) do\r\n def exists?\r\n File.exist?(@resource[:name])\r\n end\r\n\r\n def create\r\n self.content=(:dummy)\r\n end\r\n\r\n def destroy\r\n File.unlink(@resource[:name])\r\n end\r\n\r\n def content\r\n File.open(@resource[:path], 'r') do |file|\r\n sha = Digest::SHA256.new\r\n while chunk = file.read(2**16)\r\n sha << chunk\r\n end\r\n \"{sha256}#{sha.hexdigest}\"\r\n end\r\n end\r\n\r\n def content=(value)\r\n # value is munged, but we need to write the original\r\n File.write(@resource[:path], @resource.parameter(:content).shouldorig[0])\r\n end\r\nend\r\n<\/pre>\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_llmethod_create.pp Notice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds Notice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-lowlevel-method-create]\/ensure: removed Notice: \/Stage[main]\/Main\/File_llmethod[\/home\/demo\/puppet-file-lowlevel-method-create]\/ensure: created\r\n<\/pre>\n
ensurable<\/code> by an adapted implementation of it, which includes our previous change_to_s<\/code> override:<\/p>\nnewproperty(:ensure, :parent => Puppet::Property::Ensure) do\r\n defaultvalues\r\n define_method(:change_to_s) do |currentvalue, newvalue|\r\n if currentvalue == :absent\r\n should = @resource.property(:content).should\r\n \"defined content as '#{should}'\"\r\n else\r\n super(currentvalue, newvalue)\r\n end\r\n end\r\nend\r\n<\/pre>\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_llmethod_create.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.02 seconds\r\nNotice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-lowlevel-method-create]\/ensure: removed\r\nNotice: \/Stage[main]\/Main\/File_llmethod[\/home\/demo\/puppet-file-lowlevel-method-create]\/ensure: defined content as '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2'\r\n<\/pre>\nFinal low-level prototype<\/h4>\n
# modules\/demo\/lib\/puppet\/type\/file_llmethod.rb\r\n\r\n# frozen_string_literal: true\r\n\r\nrequire 'digest'\r\n\r\nPuppet::Type.newtype(:file_llmethod) do\r\n newparam(:path, namevar: true) do\r\n validate do |value|\r\n fail \"#{value} is not a String\" unless value.is_a?(String)\r\n fail \"#{value} is not an absolute path\" unless File.absolute_path?(value)\r\n end\r\n end\r\n\r\n newproperty(:content) do\r\n validate do |value|\r\n fail \"#{value} is not a String\" unless value.is_a?(String)\r\n end\r\n munge do |value|\r\n \"{sha256}#{Digest::SHA256.hexdigest(value)}\"\r\n end\r\n end\r\n\r\n newproperty(:ensure, :parent => Puppet::Property::Ensure) do\r\n defaultvalues\r\n define_method(:change_to_s) do |currentvalue, newvalue|\r\n if currentvalue == :absent\r\n should = @resource.property(:content).should\r\n \"defined content as '#{should}'\"\r\n else\r\n super(currentvalue, newvalue)\r\n end\r\n end\r\n end\r\nend\r\n<\/pre>\n# modules\/demo\/lib\/puppet\/provider\/file_llmethod\/ruby.rb\r\n\r\n# frozen_string_literal: true\r\n\r\nPuppet::Type.type(:file_llmethod).provide(:ruby) do\r\n def exists?\r\n File.exist?(@resource[:name])\r\n end\r\n\r\n def create\r\n self.content=(:dummy)\r\n end\r\n\r\n def destroy\r\n File.unlink(@resource[:name])\r\n end\r\n\r\n def content\r\n File.open(@resource[:path], 'r') do |file|\r\n sha = Digest::SHA256.new\r\n while (chunk = file.read(2**16))\r\n sha << chunk\r\n end\r\n \"{sha256}#{sha.hexdigest}\"\r\n end\r\n end\r\n\r\n def content=(_value)\r\n # value is munged, but we need to write the original\r\n File.write(@resource[:path], @resource.parameter(:content).shouldorig[0])\r\n end\r\nend\r\n<\/pre>\nResource-API prototype<\/h3>\n
file_rsapi<\/code> type by calling Puppet::ResourceApi.register_type<\/code> with several parameters, amongst which are the desired attributes, even ensure<\/code>.<\/p>\n# modules\/demo\/lib\/puppet\/type\/file_rsapi.rb\r\n\r\nrequire 'puppet\/resource_api'\r\n\r\nPuppet::ResourceApi.register_type(\r\n name: 'file_rsapi', \r\n attributes: {\r\n content: {\r\n desc: 'description of content parameter',\r\n type: 'String' \r\n },\r\n ensure: {\r\n default: 'present',\r\n desc: 'description of ensure parameter',\r\n type: 'Enum[present, absent]'\r\n },\r\n path: {\r\n behaviour: :namevar,\r\n desc: 'description of path parameter',\r\n type: 'Pattern[\/\\A\\\/([^\\n\\\/\\0]+\\\/*)*\\z\/]'\r\n },\r\n },\r\n desc: 'description of file_rsapi'\r\n)\r\n<\/pre>\npath<\/code> type<\/code> uses a built-in Puppet data type. Stdlib::Absolutepath<\/code> would have been more convenient but external data types are not possible with the Resource-API yet.<\/p>\ncanonicalize<\/code> provider feature looks similar to munging, but we skip it for now.<\/p>\nget<\/code> and a set<\/code> method in our provider, stating<\/p>\ndef get(context) {...}<\/code> where context<\/code> has no information about the resource we want to manage.<\/p>\ndef get(context, names = nil) {...}<\/code>.<\/p>\nfile_rsapi<\/code> is thus the following.<\/p>\nBasic functionality<\/h4>\n
# modules\/demo\/lib\/puppet\/type\/file_rsapi.rb\r\n\r\nrequire 'puppet\/resource_api'\r\n\r\nPuppet::ResourceApi.register_type(\r\n name: 'file_rsapi',\r\n features: %w[simple_get_filter],\r\n attributes: {\r\n content: {\r\n desc: 'description of content parameter',\r\n type: 'String' \r\n },\r\n ensure: {\r\n default: 'present',\r\n desc: 'description of ensure parameter',\r\n type: 'Enum[present, absent]'\r\n },\r\n path: {\r\n behaviour: :namevar,\r\n desc: 'description of path parameter',\r\n type: 'Pattern[\/\\A\\\/([^\\n\\\/\\0]+\\\/*)*\\z\/]'\r\n },\r\n },\r\n desc: 'description of file_rsapi'\r\n)\r\n<\/pre>\n# modules\/demo\/lib\/puppet\/provider\/file_rsapi\/file_rsapi.rb\r\n\r\nrequire 'digest'\r\n\r\nclass Puppet::Provider::FileRsapi::FileRsapi\r\n def get(context, names)\r\n (names or []).map do |name|\r\n File.exist?(name) ? {\r\n path: name,\r\n ensure: 'present',\r\n content: filedigest(name),\r\n } : nil\r\n end.compact # remove non-existing resources\r\n end\r\n\r\n def set(context, changes)\r\n changes.each do |path, change|\r\n if change[:should][:ensure] == 'present'\r\n File.write(path, change[:should][:content])\r\n elsif File.exist?(path)\r\n File.delete(path)\r\n end\r\n end\r\n end\r\n\r\n def filedigest(path)\r\n File.open(path, 'r') do |file|\r\n sha = Digest::SHA256.new\r\n while chunk = file.read(2**16)\r\n sha << chunk\r\n end\r\n \"{sha256}#{sha.hexdigest}\"\r\n end\r\n end\r\nend\r\n<\/pre>\ncontent<\/code> is written correctly into the file, but we have again no SHA256 checksum on creation as well as unnecessary writes, because the checksum from get<\/code> does not match the cleartext from the catalog:<\/p>\ndemo@85c63b50bfa3:~$ cat > file_rsapi_create.pp <<EOF\r\n$file = '\/home\/demo\/puppet-file-rsapi-create'\r\nfile {$file: ensure => absent} ->\r\nfile_rsapi {$file: content => 'This is Sparta!'}\r\nEOF\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_rsapi_create.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.03 seconds\r\nNotice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-rsapi-create]\/ensure: removed\r\nNotice: \/Stage[main]\/Main\/File_rsapi[\/home\/demo\/puppet-file-rsapi-create]\/ensure: defined 'ensure' as 'present'\r\nNotice: Applied catalog in 0.02 seconds\r\n\r\ndemo@85c63b50bfa3:~$ cat > file_rsapi_change.pp <<EOF\r\n$file = '\/home\/demo\/puppet-file-rsapi-change'\r\nfile {$file: content => 'This is madness'} ->\r\nfile_rsapi {$file: content => 'This is Sparta!'}\r\nEOF\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_rsapi_change.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.03 seconds\r\nNotice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-rsapi-change]\/content: content changed '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2' to '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e'\r\nNotice: \/Stage[main]\/Main\/File_rsapi[\/home\/demo\/puppet-file-rsapi-change]\/content: content changed '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e' to 'This is Sparta!'\r\nNotice: Applied catalog in 0.03 seconds\r\n<\/pre>\ncanonicalize<\/code> provider feature.<\/p>\nCanonicalize<\/h4>\n
canonicalize<\/code> is applied to the results of get<\/code> as well as to the catalog properties. On the one hand, we do not want to read the file’s content into memory, on the other hand cannot checksum the file’s content twice.<\/p>\ncanonicalized<\/code> call happens after the get<\/code> call:<\/p>\ndef canonicalize(context, resources)\r\n if @stage == :get\r\n # do nothing, is-state canonicalization already performed by get()\r\n else\r\n # catalog canonicalization\r\n ...\r\n end\r\nend\r\n\r\ndef get(context, paths)\r\n @stage = :get\r\n\r\n ...\r\nend\r\n<\/pre>\n
canonicalize<\/code> calls. Instead we subclass from String<\/code> and handle checksumming internally. We also add some state change calls to the appropriate context<\/code> methods.<\/p>\n# modules\/demo\/lib\/puppet\/type\/file_rsapi.rb\r\n\r\n# frozen_string_literal: true\r\n\r\nrequire 'puppet\/resource_api'\r\n\r\nPuppet::ResourceApi.register_type(\r\n name: 'file_rsapi',\r\n features: %w[simple_get_filter canonicalize],\r\n attributes: {\r\n content: {\r\n desc: 'description of content parameter',\r\n type: 'String'\r\n },\r\n ensure: {\r\n default: 'present',\r\n desc: 'description of ensure parameter',\r\n type: 'Enum[present, absent]'\r\n },\r\n path: {\r\n behaviour: :namevar,\r\n desc: 'description of path parameter',\r\n type: 'Pattern[\/\\A\\\/([^\\n\\\/\\0]+\\\/*)*\\z\/]'\r\n }\r\n },\r\n desc: 'description of file_rsapi'\r\n)\r\n<\/pre>\n# modules\/demo\/lib\/puppet\/provider\/file_rsapi\/file_rsapi.rb\r\n\r\n# frozen_string_literal: true\r\n\r\nrequire 'digest'\r\nrequire 'pathname'\r\n\r\nclass Puppet::Provider::FileRsapi::FileRsapi\r\n class CanonicalString < String\r\n attr_reader :original\r\n\r\n def class\r\n # Mask as String for YAML.dump to mitigate\r\n # Error: Transaction store file \/var\/cache\/puppet\/state\/transactionstore.yaml\r\n # is corrupt ((\/var\/cache\/puppet\/state\/transactionstore.yaml): Tried to\r\n # load unspecified class: Puppet::Provider::FileRsapi::FileRsapi::CanonicalString)\r\n String\r\n end\r\n\r\n def self.from(obj)\r\n return obj if obj.is_a?(self)\r\n return new(filedigest(obj)) if obj.is_a?(Pathname)\r\n\r\n new(\"{sha256}#{Digest::SHA256.hexdigest(obj)}\", obj)\r\n end\r\n\r\n def self.filedigest(path)\r\n File.open(path, 'r') do |file|\r\n sha = Digest::SHA256.new\r\n while (chunk = file.read(2**16))\r\n sha << chunk\r\n end\r\n \"{sha256}#{sha.hexdigest}\"\r\n end\r\n end\r\n\r\n def initialize(canonical, original = nil)\r\n @original = original\r\n super(canonical)\r\n end\r\n end\r\n\r\n def canonicalize(_context, resources)\r\n resources.each do |resource|\r\n next if resource[:ensure] == 'absent'\r\n\r\n resource[:content] = CanonicalString.from(resource[:content])\r\n end\r\n resources\r\n end\r\n\r\n def get(_context, names)\r\n (names or []).map do |name|\r\n next unless File.exist?(name)\r\n\r\n {\r\n content: CanonicalString.from(Pathname.new(name)),\r\n ensure: 'present',\r\n path: name\r\n }\r\n end.compact # remove non-existing resources\r\n end\r\n\r\n def set(context, changes)\r\n changes.each do |path, change|\r\n if change[:should][:ensure] == 'present'\r\n File.write(path, change[:should][:content].original)\r\n if change[:is][:ensure] == 'present'\r\n # The only other possible change is due to content,\r\n # but content change transition info is covered implicitly\r\n else\r\n context.created(\"#{path} with content '#{change[:should][:content]}'\")\r\n end\r\n elsif File.exist?(path)\r\n File.delete(path)\r\n context.deleted(path)\r\n end\r\n end\r\n end\r\nend\r\n<\/pre>\ndemo@85c63b50bfa3:~$ cat > file_rsapi_create.pp <<EOF\r\n$file = '\/home\/demo\/puppet-file-rsapi-create'\r\nfile {$file: ensure => absent} ->\r\nfile_rsapi {$file: content => 'This is Sparta!'}\r\nEOF\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_rsapi_create.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.03 seconds\r\nNotice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-rsapi-create]\/ensure: removed\r\nNotice: \/Stage[main]\/Main\/File_rsapi[\/home\/demo\/puppet-file-rsapi-create]\/ensure: defined 'ensure' as 'present'\r\nNotice: file_rsapi: Created: \/home\/demo\/puppet-file-rsapi-create with content '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2'\r\n\r\ndemo@85c63b50bfa3:~$ cat > file_rsapi_change.pp <<EOF\r\n$file = '\/home\/demo\/puppet-file-rsapi-change'\r\nfile {$file: content => 'This is madness'} ->\r\nfile_rsapi {$file: content => 'This is Sparta!'}\r\nEOF\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_rsapi_change.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.03 seconds\r\nNotice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-rsapi-change]\/content: content changed '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2' to '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e'\r\nNotice: \/Stage[main]\/Main\/File_rsapi[\/home\/demo\/puppet-file-rsapi-change]\/content: content changed '{sha256}0549defd0a7d6d840e3a69b82566505924cacbe2a79392970ec28cddc763949e' to '{sha256}823cbb079548be98b892725b133df610d0bff46b33e38b72d269306d32b73df2'\r\n\r\ndemo@85c63b50bfa3:~$ cat > file_rsapi_remove.pp <<EOF\r\n$file = '\/home\/demo\/puppet-file-rsapi-remove'\r\nfile {$file: ensure => present} ->\r\nfile_rsapi {$file: ensure => absent}\r\nEOF\r\n\r\ndemo@85c63b50bfa3:~$ bin\/puppet apply --modulepath modules file_rsapi_remove.pp\r\nNotice: Compiled catalog for 85c63b50bfa3 in environment production in 0.03 seconds\r\nNotice: \/Stage[main]\/Main\/File[\/home\/demo\/puppet-file-rsapi-remove]\/ensure: created\r\nNotice: \/Stage[main]\/Main\/File_rsapi[\/home\/demo\/puppet-file-rsapi-remove]\/ensure: undefined 'ensure' from 'present'\r\nNotice: file_rsapi: Deleted: \/home\/demo\/puppet-file-rsapi-remove\r\n<\/pre>\nThoughts<\/h3>\n
CanonicalString<\/code> masking as String<\/code> makes it look like we are missing something. If the Resource-API only checked data types after canonicalization, we could make get<\/code> return something simpler than CanonicalString<\/code> to signal an already canonicalized value.<\/p>\n