Collecting logs from salt-managed nodes without ssh access

Posted September 8, 2021 by Adrian Wyssmann ‐ 3 min read

We manage our baremetal kubernetes nodes with salt and for security purposes, there is no direct ssh access. The only way to access them is using salt-master. This can be challenging if you want to get some files from the nodes, but it's possible.

My use case was, that for an issue with the Rancher logging I have to run the a log-collector script on all nodes. But I only have access via saltstack and not direct ssh access. So these are the steps performed

  1. Ensure the script rancher2_logs_collector.sh

  2. Then you can run salt-cp to copy the file to all nodes

    [[email protected] ~]$ sudo salt-cp -L 'server0068,server0069,server0070,server0071,server0072,server0073' ./rancher2_logs_collector.sh /tmp
    [sudo] password for saltuser:
    server0068:
        ----------
        /tmp/rancher2_logs_collector.sh:
            True
    server0069:
        ----------
        /tmp/rancher2_logs_collector.sh:
            True
    server0070:
        ----------
        /tmp/rancher2_logs_collector.sh:
            True
    server0071:
        ----------
        /tmp/rancher2_logs_collector.sh:
            True
    server0072:
        ----------
        /tmp/rancher2_logs_collector.sh:
            True
    server0073:
        ----------
        /tmp/rancher2_logs_collector.sh:
            True
    
  3. Make the file executable using cmd.run:

    [[email protected] ~]$ sudo salt -L 'server0068,server0069,server0070,server0071,server0072,server0073' cmd.run 'chmod u+x /tmp/rancher2_logs_collector.sh'
    server0071:
    server0073:
    server0068:
    server0072:
    server0070:
    server0069:
    
  4. I use the same then to execute the script:

    [[email protected] ~]$ sudo salt -L 'server0068,server0069,server0070,server0071,server0072,server0073' cmd.run '/tmp/rancher2_logs_collector.sh'
    server0069:
        2021-09-08 06:43:11: Created /tmp/tmp.SfrWsQ6vR6
        2021-09-08 06:43:11: Detecting available commands... renice ionoice
        2021-09-08 06:43:11: Detecting OS... centos 7
        2021-09-08 06:43:11: Detecting k8s distribution... rke
        2021-09-08 06:43:11: Detecting init type... systemd
        2021-09-08 06:43:11: Collecting system info
        cp: cannot stat '/run/systemd/resolve/resolv.conf': No such file or directory
        2021-09-08 06:43:30: Collecting network info
        2021-09-08 06:43:34: Collecting docker info
        2021-09-08 06:43:37: Collecting rancher logs
        2021-09-08 06:43:37: Collecting k8s component logs
        2021-09-08 06:43:40: Collecting system pod logs
        2021-09-08 06:43:50: Collecting nginx-proxy info
        2021-09-08 06:43:50: Collecting k8s directory state
        2021-09-08 06:43:50: Collecting k8s certificates
        2021-09-08 06:43:50: Collecting rke etcd info
        2021-09-08 06:43:50: Collecting etcdctl output
        2021-09-08 06:43:52: Collecting system logs from /var/log
        cp: omitting directory '/var/log/sa'
        2021-09-08 06:43:52: Collecting system logs from journald
        2021-09-08 06:43:56: Created /tmp/server0069-2021-09-08_06_43_52.tar.gz
        2021-09-08 06:43:56: Removing /tmp/tmp.SfrWsQ6vR6a
    ...
    

Now that we have the files e.g. /tmp/server0069-2021-09-08_06_43_52.tar.gz on the minions comes the tricky part, we have to copy the files back from the minions to the master.

The solution is to use cp module which allows to “push” files from the minions to the master. But first check your /etc/salt/master.d/master.conf cause

Since this feature allows a minion to push a file up to the master server it is disabled by default for security purposes. To enable, set file_recv to True in the master configuration file, and restart the master.

Once this is done, you can do this:

[[email protected] ~]$ sudo salt -L 'server0068,server0069,server0070,server0071,server0072,server0073' cp.push /tmp/*.tar.gz
server0072:
    False
server0069:
    False
server0070:
    False
server0071:
    false
server0073:
    false
server0068:
    false

At least I thought so but apparently wildcards are not supported, so I have to execute the command for each file, so let’s check the filenames…

[[email protected]] ~]$ sudo salt -L 'server0068,server0069,server0070,server0071,server0072,server0073' cmd.run "ls /tmp/ | grep server"
server0072:
    server0072-2021-09-08_06_43_45.tar.gz
server0073:
    server0073-2021-09-08_06_43_47.tar.gz
server0069:
    server0069-2021-09-08_06_43_52.tar.gz
server0071:
    server0071-2021-09-08_06_43_48.tar.gz
server0068:
    server0068-2021-09-08_06_43_48.tar.gz
server0070:
    server0070-2021-09-08_06_43_47.tar.gz

… and then run this for each file, for example

[[email protected] ~]$ sudo salt 'server0068' cp.push /tmp/server0069-2021-09-08_06_43_52.tar.gz
server0068:
    True

At last you should have all files on your salt master in /var/cache/salt/master/minions/saltminion-id:

[[email protected] ~]$ find  /var/cache/salt/master/minions/ -name "devs*.tar.gz"
/var/cache/salt/master/minions/server0069/files/tmp/server0069-2021-09-08_06_43_52.tar.gz
/var/cache/salt/master/minions/server0068/files/tmp/server0068-2021-09-08_06_43_48.tar.gz
/var/cache/salt/master/minions/server0070/files/tmp/server0070-2021-09-08_06_43_47.tar.gz
/var/cache/salt/master/minions/server0071/files/tmp/server0071-2021-09-08_06_43_48.tar.gz
/var/cache/salt/master/minions/server0072/files/tmp/server0072-2021-09-08_06_43_45.tar.gz
/var/cache/salt/master/minions/server0073/files/tmp/server0073-2021-09-08_06_43_47.tar.gz

Hope that help somebody else with a similar setup.