Installing and running Consul on CoreOS

I’m new to Fedora CoreOS but actually wrote some of the original documentation for installing on physical machines in 2014 when it was just CoreOS)and I am working on an FCCT config to run HashiCorp Consul on EC2 with Terraform. Since I am using the provided Fedora CoreOS AMI and Terraform, I am trying to keep everything in Terraform and not build a custom AMI to keep everything simple.

My FCCT is performing the following tasks:

  1. Uses the storage.files to download the binary from HashiCorp releases
  2. Runs a container to unzip the Consul file as FCOS does not have unzip available
  3. Sets some permissions (maybe this should be setting up SELinux properly)
variant: fcos
version: 1.2.0
passwd:
  users:
    - name: core
      groups: [sudo, docker]
      ssh_authorized_keys:
        - "thekey"
    - name: consul
      system: true
      home_dir: /etc/consul.d
      shell: /bin/false
systemd:
  units:
    - name: install-consul.service
      enabled: true
      contents: |
        [Unit]
        Before=consul.service
        After=network-online.target docker.service
        ConditionPathExists=!/usr/local/bin/consul

        [Service]
        Type=oneshot
        ExecStart=/usr/local/bin/install-consul.sh
        RemainAfterExit=yes

        [Install]
        WantedBy=consul.service
    - name: consul.service
      enabled: true
      contents: |
        [Unit]
        Description="HashiCorp Consul - A service mesh solution"
        Documentation=https://www.consul.io/
        Requires=network-online.target
        After=network-online.target
        Wants=install-consul.service
        ConditionFileNotEmpty=/etc/consul.d/consul.hcl

        [Service]
        Type=notify
        ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/
        ExecReload=/bin/kill --signal HUP $MAINPID
        KillMode=process
        KillSignal=SIGTERM
        Restart=on-failure
        LimitNOFILE=65536

        [Install]
        WantedBy=multi-user.target
storage:
  directories:
    - path: /etc/consul.d
      user:
        name: consul
      group:
        name: consul
    - path: /opt/consul
      user:
        name: consul
      group:
        name: consul
    - path: /opt/bootstrap
  files:
    - path: /opt/bootstrap/consul.zip
      contents:
        source: https://releases.hashicorp.com/consul/${consul_version}/consul_${consul_version}_linux_amd64.zip
    - path: /usr/local/bin/install-consul.sh
      mode: 0755
      contents:
        inline: |
          #!/bin/bash
          docker run --rm -v /opt/bootstrap:/tmp:z --workdir=/tmp alpine:latest unzip consul.zip
          mv /opt/bootstrap/consul /usr/local/bin/
          chown root:root /usr/local/bin/consul
          chmod 755 /usr/local/bin/consul
          # should not be doing this
          setenforce 0
    - path: /etc/consul.d/consul.hcl
      contents:
        inline: |
          datacenter = "${datacenter}"
          server = true
          bootstrap_expect = ${bootstrap_expect}
          data_dir = "/opt/consul/data"
          advertise_addr = "{{GetInterfaceIP \"ens5\"}}"
          client_addr = "0.0.0.0"
          log_level = "INFO"
          ui_config {
            enabled = true
          }

However, I am not able to get the systemd unit to run properly as it displays the following in journalctl:

consul.service: Failed at step EXEC spawning /usr/local/bin/consul: Permission denied

I can get it to run by running setenforce 0 as part of the install-consul.service script, but obviously this is not the best solution.

Is there a better way to set this up? I realize that FCOS is really designed for containers, but is there something I am missing in the documentation? I realize I could move this into a container image and pull it, but the FCCT makes it really easy with Terraform to really keep IAAC and deployments light.

I am touching up on SELinux but is there a way to use SELinux to determine that labels I should be setting/using?

2 Likes

Are there AVCs from selinux in the log?

To be honest, I’m not sure where to check for those. I primarily use Ubuntu but switching to CoreOS for this project. I assumed journalctl would show those. Needless to say I still have a lot of SELinux to learn.

You are not alone with that :slight_smile: I found some breadcrumbs here:
Getting started with SELinux :: Fedora Docs

See the Note under “Changing SELinux states and modes :: Fedora Docs
sudo sealert -a /var/log/audit/audit.log

In Portuguese = AVC ( Acidente Vascular Cerebral ) who got a stroke ? :stuck_out_tongue_winking_eye:

Technically “Access Vector Cache denial”. It’s the main time sysadmins usually see it, so it’s AVC for short.

1 Like

Also, I was using mv instead of cp in the script, which changed the selinux label causing the permissions issue. It’s up and running now! Also gave me a reason to deep dive into selinux (it’s been a long time).

1 Like

Is there a recommended way to look at AVC messages in coreos? I’m assuming it’s journalctl but most references are saying audit and messages logs.

Really excited about CoreOS, keep up the great work!

1 Like

You should be able to find them in the journal. If they are not there, then it probably mean that they are being skipped from auditing. See 8.3.2. Possible Causes of Silent Denials Red Hat Enterprise Linux 6 | Red Hat Customer Portal

For your use case, I would suggest building a container with a Containerfile in Quay adding the consul binary on top of a base image and then running that image via podman (or Docker) on boot, potentially with host network namespace.

This way you can easily rebuild the container and update each instance instead of having to manually update the binary or re-provision the node.

Yep. Something like journalctl -b 0 --grep 'avc: denied' should work. A little less ideal but easier to remember is something like journalctl -b 0 | grep AVC | grep denied.

Thanks @jasonmccallister for using FCOS!