Collecting logs from salt-managed nodes without ssh access

We manage our baremetal kubernetes nodes with salt and for security purposes, there is no direct ssh access. The only way to access them is using salt-master. This can be challenging if you want to get some files from the nodes, but it's possible.

My use case was, that for an issue with the Rancher logging I have to run the a log-collector script on all nodes. But I only have access via saltstack and not direct ssh access. So these are the steps performed

1. Ensure the script rancher2_logs_collector.sh

2. Then you can run salt-cp to copy the file to all nodes

   [[email protected] ~]$ sudo salt-cp -L 'devs0468,devs0469,devs0470,devs0471,devs0472,devs0473' ./rancher2_logs_collector.sh /tmp
   [sudo] password for saltuser:
   devs0468:
       ----------
       /tmp/rancher2_logs_collector.sh:
           True
   devs0469:
       ----------
       /tmp/rancher2_logs_collector.sh:
           True
   devs0470:
       ----------
       /tmp/rancher2_logs_collector.sh:
           True
   devs0471:
       ----------
       /tmp/rancher2_logs_collector.sh:
           True
   devs0472:
       ----------
       /tmp/rancher2_logs_collector.sh:
           True
   devs0473:
       ----------
       /tmp/rancher2_logs_collector.sh:
           True
   

3. Make the file executable using cmd.run:

   [[email protected] ~]$ sudo salt -L 'devs0468,devs0469,devs0470,devs0471,devs0472,devs0473' cmd.run 'chmod u+x /tmp/rancher2_logs_collector.sh'
   devs0471:
   devs0473:
   devs0468:
   devs0472:
   devs0470:
   devs0469:
   

4. I use the same then to execute the script:

   [[email protected] ~]$ sudo salt -L 'devs0468,devs0469,devs0470,devs0471,devs0472,devs0473' cmd.run '/tmp/rancher2_logs_collector.sh'
   devs0469:
       2021-09-08 06:43:11: Created /tmp/tmp.SfrWsQ6vR6
       2021-09-08 06:43:11: Detecting available commands... renice ionoice
       2021-09-08 06:43:11: Detecting OS... centos 7
       2021-09-08 06:43:11: Detecting k8s distribution... rke
       2021-09-08 06:43:11: Detecting init type... systemd
       2021-09-08 06:43:11: Collecting system info
       cp: cannot stat '/run/systemd/resolve/resolv.conf': No such file or directory
       2021-09-08 06:43:30: Collecting network info
       2021-09-08 06:43:34: Collecting docker info
       2021-09-08 06:43:37: Collecting rancher logs
       2021-09-08 06:43:37: Collecting k8s component logs
       2021-09-08 06:43:40: Collecting system pod logs
       2021-09-08 06:43:50: Collecting nginx-proxy info
       2021-09-08 06:43:50: Collecting k8s directory state
       2021-09-08 06:43:50: Collecting k8s certificates
       2021-09-08 06:43:50: Collecting rke etcd info
       2021-09-08 06:43:50: Collecting etcdctl output
       2021-09-08 06:43:52: Collecting system logs from /var/log
       cp: omitting directory '/var/log/sa'
       2021-09-08 06:43:52: Collecting system logs from journald
       2021-09-08 06:43:56: Created /tmp/devs0469-2021-09-08_06_43_52.tar.gz
       2021-09-08 06:43:56: Removing /tmp/tmp.SfrWsQ6vR6a
   ...
   

Now that we have the files e.g. /tmp/devs0469-2021-09-08_06_43_52.tar.gz on the minions comes the tricky part, we have to copy the files back from the minions to the master.

The solution is to use cp module which allows to “push” files from the minions to the master. But first check your /etc/salt/master.d/master.conf cause

Since this feature allows a minion to push a file up to the master server it is disabled by default for security purposes. To enable, set file_recv to True in the master configuration file, and restart the master.

Once this is done, you can do this:

[[email protected] ~]$ sudo salt -L 'devs0468,devs0469,devs0470,devs0471,devs0472,devs0473' cp.push /tmp/*.tar.gz
devs0472:
    False
devs0469:
    False
devs0470:
    False
devs0471:
    false
devs0473:
    false
devs0468:
    false

At least I thought so but apparently wildcards are not supported, so I have to execute the command for each file, so let’s check the filenames…

[[email protected]] ~]$ sudo salt -L 'devs0468,devs0469,devs0470,devs0471,devs0472,devs0473' cmd.run "ls /tmp/ | grep devs"
devs0472:
    devs0472-2021-09-08_06_43_45.tar.gz
devs0473:
    devs0473-2021-09-08_06_43_47.tar.gz
devs0469:
    devs0469-2021-09-08_06_43_52.tar.gz
devs0471:
    devs0471-2021-09-08_06_43_48.tar.gz
devs0468:
    devs0468-2021-09-08_06_43_48.tar.gz
devs0470:
    devs0470-2021-09-08_06_43_47.tar.gz

… and then run this for each file, for example

[[email protected] ~]$ sudo salt 'devs0468' cp.push /tmp/devs0469-2021-09-08_06_43_52.tar.gz
devs0468:
    True

At last you should have all files on your salt master in /var/cache/salt/master/minions/saltminion-id:

[[email protected] ~]$ find  /var/cache/salt/master/minions/ -name "devs*.tar.gz"
/var/cache/salt/master/minions/devs0469/files/tmp/devs0469-2021-09-08_06_43_52.tar.gz
/var/cache/salt/master/minions/devs0468/files/tmp/devs0468-2021-09-08_06_43_48.tar.gz
/var/cache/salt/master/minions/devs0470/files/tmp/devs0470-2021-09-08_06_43_47.tar.gz
/var/cache/salt/master/minions/devs0471/files/tmp/devs0471-2021-09-08_06_43_48.tar.gz
/var/cache/salt/master/minions/devs0472/files/tmp/devs0472-2021-09-08_06_43_45.tar.gz
/var/cache/salt/master/minions/devs0473/files/tmp/devs0473-2021-09-08_06_43_47.tar.gz

Hope that help somebody else with a similar setup.