ldap integration unuseable, toml requires native extension

[joernott]

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: 6.4.3
• Ruby: 2.5.3p105
• Distribution: RHEL 7
• Module version: 6.1.0

How to reproduce (e.g Puppet code you use)

Define a LDAP configuration. Puppet runs will fail due to the missing toml gem on the puppet server. Try to install the gem on the puppet server to solve this issue.

What are you seeing

Installation of the toml gem fails as it requires a native extension to be installed.

What behaviour did you expect instead

The puppet module should not rely on gems which require native extensions r even better not require the gem at all.

Output log

Any additional information you'd like to impart

I also tried installing the toml-rb gem instead (as this is recommended e.g. for the puppetserver dashboards. But that requires another gem"citrus" which in turn needs a native extension.

I have worked around this issue by creating the toml file by using a file resource and an epp template.

[bastelfreak]

Hi @joernott ,
We documented it in the README.md:
https://github.com/voxpupuli/puppet-grafana#toml-note

Do you have a better way to implement this, without the gem?

[joernott]

In our wrapper (which does the data setuip as well, I commented out the ldap_cfg parameter for grafana and added the following:

    if $ldap_cfg {
      file { '/etc/grafana/ldap.toml': # working around broken toml support requiring native librarys on puppet master
        ensure  => 'file',
        owner   => 'grafana',
        group   => 'grafana',
        mode    => '0640',
        content => epp('company_grafana/ldap.toml.epp', $ldap_cfg)

      }
    }

The template is pretty simple (see attached file):

<% $servers.each |$server| {%>
[[servers]]
bind_dn = "<%= $server['bind_dn']  %>"
bind_password = "<%= $server['bind_password'] %>"
host = "<%= $server['host'] %>"
port = <%= $server['port'] %>
search_base_dns = [<%= $server['search_base_dns'].map|$sbd| {"\"${sbd}\""}.join(',')%>]
search_filter = "<%= $search_filter %>"
ssl_skip_verify = <%= $ssl_skip_verify %>
use_ssl = <%= $use_ssl %>

[servers.attributes]
<%- $server['attributes'].each|$k,$v| { -%>
<%= $k %> = "<%= $v %>"
<%- } -%>

<%- $server['group_mappings'].each|$gm| { -%>
[[servers.group_mappings]]
grafana_admin = <%= $gm['grafana_admin']  %>
group_dn = "<%= $gm['group_dn'] %>"
org_role = "<%= $gm['org_role']  %>"

<%- } -%>
<%- } -%>

[joernott]

The point is that the things you write in the documentation don't work. The gem requires a native extension. puppetserver gem install toml fails as jruby doesn't properly support native extensions.

[bastelfreak]

@joernott on which puppetserver/jruby version are you on? puppetserver gem install toml --no-document works fine on puppetserver 6.11.0 / jruby 9.2.11.1.

[joernott]

@bastelfreak We use Puppet Enterprise 2019.1 and we are currently updating to 2019.5. As puppetserver 6.11.0 is quite new, I doubt, that it will be a part of 2019.1. I will check, once I am back in the office.

[joernott]

According to /opt/puppetlabs/server/ezbake.manifest, the jruby version is 9.2.0.0-1 and the puppet enterprise version is 2019.1.1 according to /opt/puppetlabs/server/pe_version.

[lucasmaurice]

@joernott did you find any solutions ?

[bastelfreak]

I created a fresh PE 2019 VM and the installation works fine for me:

[root@localhost ~]# puppetserver gem install toml --no-document
Fetching toml-0.3.0.gem
Fetching parslet-2.0.0.gem
Successfully installed parslet-2.0.0
Successfully installed toml-0.3.0
2 gems installed
[root@localhost ~]# puppetserver ruby --version
jruby 9.2.17.0 (2.5.8) 2021-03-29 84d363da97 OpenJDK 64-Bit Server VM 11.0.13-internal+8-adhoc.root.openjdk on 11.0.13-internal+8-adhoc.root.openjdk +jit [linux-x86_64]
[root@localhost ~]# rpm -qa | grep puppet
pe-puppetdb-6.19.1-1.el8.noarch
puppet-tools-release-1.0.0-6.el8.noarch
puppet-agent-6.25.1-1.el8.x86_64
pe-puppetdb-termini-6.19.1-1.el8.noarch
puppet-bolt-3.21.0-1.el8.x86_64
pe-puppet-enterprise-release-2019.8.9.0-2.pe.el8.noarch
pe-puppetserver-2019.8.3.95-1.el8.noarch
[root@localhost ~]#

@lucasmaurice which issue do you see?

[joernott]

@lucasmaurice I provided the toml file myself with a simple file resource using a template (see above). That eliminated the need for a native extension completely.

[alexjfisher]

Stdlib 8.1.0 introduced a to_toml function I might consider switching to. Stdlib 8.x had some other breaking changes though which, IMO, still need to be fixed. Perhaps I'll see if we can use the stdlib function if it's available.

[MartyEwings]

@alexjfisher id be interested in switching over to using Stdlib to_toml instead of requiring the GEM ( and contributing) if that was something we could entertain. It would make offline installations easier of some of our downstream use cases

[bastelfreak]

@MartyEwings this would be an awesome contribution!

[jerome2710]

I created a fresh PE 2019 VM and the installation works fine for me:

[root@localhost ~]# puppetserver gem install toml --no-document
Fetching toml-0.3.0.gem
Fetching parslet-2.0.0.gem
Successfully installed parslet-2.0.0
Successfully installed toml-0.3.0
2 gems installed
[root@localhost ~]# puppetserver ruby --version
jruby 9.2.17.0 (2.5.8) 2021-03-29 84d363da97 OpenJDK 64-Bit Server VM 11.0.13-internal+8-adhoc.root.openjdk on 11.0.13-internal+8-adhoc.root.openjdk +jit [linux-x86_64]
[root@localhost ~]# rpm -qa | grep puppet
pe-puppetdb-6.19.1-1.el8.noarch
puppet-tools-release-1.0.0-6.el8.noarch
puppet-agent-6.25.1-1.el8.x86_64
pe-puppetdb-termini-6.19.1-1.el8.noarch
puppet-bolt-3.21.0-1.el8.x86_64
pe-puppet-enterprise-release-2019.8.9.0-2.pe.el8.noarch
pe-puppetserver-2019.8.3.95-1.el8.noarch
[root@localhost ~]#

@lucasmaurice which issue do you see?

I did the same, however the Puppetserver still fails;

INFO	 -> Deploying environment /etc/puppetlabs/code/environments/production
INFO	 -> Environment production is now at c62982ba52db73fd636e363b3bc36a7cabd78091
INFO	 -> Using Puppetfile '/etc/puppetlabs/code/environments/production/Puppetfile'
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/archive
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/augeasproviders_core
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/augeasproviders_sysctl
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/elasticsearch
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/letsencrypt
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/grafana
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/nginx
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/postfix
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/prometheus
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/firewall
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/git
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/java
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/motd
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/ssh
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/sudo
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/alternatives
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/elastic_stack
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/mysql
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/apt
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/logrotate
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/concat
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/inifile
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/mailalias_core
INFO	 -> Deploying module to /etc/puppetlabs/code/environments/production/modules/stdlib
ERROR	 -> Command exited with non-zero exit code:
Command: /opt/puppetlabs/bin/puppet generate types --environment production --environmentpath /etc/puppetlabs/code/environments --config /etc/puppetlabs/puppet/puppet.conf
Stdout:
Notice: Generating Puppet resource types.
Stderr:
Error: Failed to load custom type 'grafana_ldap_config' from '/etc/puppetlabs/code/environments/production/modules/grafana/lib/puppet/type/grafana_ldap_config.rb': cannot load such file -- toml
Exit code: 1
ERROR: Job failed: exit code 1

How can we fix this? I am not actually using LDAP, I just want to install Grafana.

[bastelfreak]

Did you read my comment I posted 4 years ago? #205 (comment)

[joernott]

@bastelfreak, I solved it by not using the feature at all. I created the toml file, using a simple file resource and a template, see #205 (comment).

That solved the issue for us.

[jerome2710]

Did you read my comment I posted 4 years ago? #205 (comment)

@bastelfreak Yes, I did. The readme actually suggests the same what comment #1034610151 describes and what I executed. But the Puppetserver still fails.

@joernott suggestion is not applicable for me, I think? I don't actually want to use LDAP nor did I enable it in the config.

[bastelfreak]

#374 I updated the README.md to explain the toml process a bit better. tl;dr the "generate types" option uses the puppet agent ruby, not the puppetserver ruby. So you need to install the gem for the puppet-agent as well.

[jerome2710]

@bastelfreak thank you, the deployment does succeed now.

As we are using a repository to manage our Puppet Code, which gets deployed to our Puppet server through r10k when pushing changes, I don't believe I can run puppet ... commands (I'm quite new to Puppet). Therefore, I added a manifest/puppetserver.domain.tld.pp file with the following code:

node 'puppetserver.domain.tld' {
  package { ['toml']:
    ensure => 'installed',
    provider => puppetserver_gem
  }

  package { ['toml']:
    ensure => 'installed',
    provider => puppet_gem
  }
}