Collecting logs from salt-managed nodes without ssh access

We manage our baremetal kubernetes nodes with salt and for security purposes, there is no direct ssh access. The only way to access them is using salt-master. This can be challenging if you want to get some files from the nodes, but it's possible.

My use case was, that for an issue with the Rancher logging I have to run the a log-collector script on all nodes. But I only have access via saltstack and not direct ssh access. So these are the steps performed

1. Ensure the script rancher2_logs_collector.sh

2. Then you can run salt-cp to copy the file to all nodes

   [[email protected] ~]$ sudo salt-cp -L 'server0068,server0069,server0070,server0071,server0072,server0073' ./rancher2_logs_collector.sh /tmp
   [sudo] password for saltuser:
   server0068:
       ----------
       /tmp/rancher2_logs_collector.sh:
           True
   server0069:
       ----------
       /tmp/rancher2_logs_collector.sh:
           True
   server0070:
       ----------
       /tmp/rancher2_logs_collector.sh:
           True
   server0071:
       ----------
       /tmp/rancher2_logs_collector.sh:
           True
   server0072:
       ----------
       /tmp/rancher2_logs_collector.sh:
           True
   server0073:
       ----------
       /tmp/rancher2_logs_collector.sh:
           True
   

3. Make the file executable using cmd.run:

   [[email protected] ~]$ sudo salt -L 'server0068,server0069,server0070,server0071,server0072,server0073' cmd.run 'chmod u+x /tmp/rancher2_logs_collector.sh'
   server0071:
   server0073:
   server0068:
   server0072:
   server0070:
   server0069:
   

4. I use the same then to execute the script:

   [[email protected] ~]$ sudo salt -L 'server0068,server0069,server0070,server0071,server0072,server0073' cmd.run '/tmp/rancher2_logs_collector.sh'
   server0069:
       2021-09-08 06:43:11: Created /tmp/tmp.SfrWsQ6vR6
       2021-09-08 06:43:11: Detecting available commands... renice ionoice
       2021-09-08 06:43:11: Detecting OS... centos 7
       2021-09-08 06:43:11: Detecting k8s distribution... rke
       2021-09-08 06:43:11: Detecting init type... systemd
       2021-09-08 06:43:11: Collecting system info
       cp: cannot stat '/run/systemd/resolve/resolv.conf': No such file or directory
       2021-09-08 06:43:30: Collecting network info
       2021-09-08 06:43:34: Collecting docker info
       2021-09-08 06:43:37: Collecting rancher logs
       2021-09-08 06:43:37: Collecting k8s component logs
       2021-09-08 06:43:40: Collecting system pod logs
       2021-09-08 06:43:50: Collecting nginx-proxy info
       2021-09-08 06:43:50: Collecting k8s directory state
       2021-09-08 06:43:50: Collecting k8s certificates
       2021-09-08 06:43:50: Collecting rke etcd info
       2021-09-08 06:43:50: Collecting etcdctl output
       2021-09-08 06:43:52: Collecting system logs from /var/log
       cp: omitting directory '/var/log/sa'
       2021-09-08 06:43:52: Collecting system logs from journald
       2021-09-08 06:43:56: Created /tmp/server0069-2021-09-08_06_43_52.tar.gz
       2021-09-08 06:43:56: Removing /tmp/tmp.SfrWsQ6vR6a
   ...
   

Now that we have the files e.g. /tmp/server0069-2021-09-08_06_43_52.tar.gz on the minions comes the tricky part, we have to copy the files back from the minions to the master.

The solution is to use cp module which allows to “push” files from the minions to the master. But first check your /etc/salt/master.d/master.conf cause

Since this feature allows a minion to push a file up to the master server it is disabled by default for security purposes. To enable, set file_recv to True in the master configuration file, and restart the master.

Once this is done, you can do this:

[[email protected] ~]$ sudo salt -L 'server0068,server0069,server0070,server0071,server0072,server0073' cp.push /tmp/*.tar.gz
server0072:
    False
server0069:
    False
server0070:
    False
server0071:
    false
server0073:
    false
server0068:
    false

At least I thought so but apparently wildcards are not supported, so I have to execute the command for each file, so let’s check the filenames…

[[email protected]] ~]$ sudo salt -L 'server0068,server0069,server0070,server0071,server0072,server0073' cmd.run "ls /tmp/ | grep server"
server0072:
    server0072-2021-09-08_06_43_45.tar.gz
server0073:
    server0073-2021-09-08_06_43_47.tar.gz
server0069:
    server0069-2021-09-08_06_43_52.tar.gz
server0071:
    server0071-2021-09-08_06_43_48.tar.gz
server0068:
    server0068-2021-09-08_06_43_48.tar.gz
server0070:
    server0070-2021-09-08_06_43_47.tar.gz

… and then run this for each file, for example

[[email protected] ~]$ sudo salt 'server0068' cp.push /tmp/server0069-2021-09-08_06_43_52.tar.gz
server0068:
    True

At last you should have all files on your salt master in /var/cache/salt/master/minions/saltminion-id:

[[email protected] ~]$ find  /var/cache/salt/master/minions/ -name "devs*.tar.gz"
/var/cache/salt/master/minions/server0069/files/tmp/server0069-2021-09-08_06_43_52.tar.gz
/var/cache/salt/master/minions/server0068/files/tmp/server0068-2021-09-08_06_43_48.tar.gz
/var/cache/salt/master/minions/server0070/files/tmp/server0070-2021-09-08_06_43_47.tar.gz
/var/cache/salt/master/minions/server0071/files/tmp/server0071-2021-09-08_06_43_48.tar.gz
/var/cache/salt/master/minions/server0072/files/tmp/server0072-2021-09-08_06_43_45.tar.gz
/var/cache/salt/master/minions/server0073/files/tmp/server0073-2021-09-08_06_43_47.tar.gz

Hope that help somebody else with a similar setup.