Collecting logs from salt-managed nodes without ssh access
Posted September 8, 2021 by Adrian Wyssmann ‐ 3 min read
We manage our baremetal kubernetes nodes with salt and for security purposes, there is no direct ssh access. The only way to access them is using salt-master. This can be challenging if you want to get some files from the nodes, but it's possible.
My use case was, that for an issue with the Rancher logging I have to run the a log-collector script on all nodes. But I only have access via saltstack and not direct ssh access. So these are the steps performed
-
Ensure the script
rancher2_logs_collector.sh
-
Then you can run
salt-cp
to copy the file to all nodes[[email protected] ~]$ sudo salt-cp -L 'server0068,server0069,server0070,server0071,server0072,server0073' ./rancher2_logs_collector.sh /tmp [sudo] password for saltuser: server0068: ---------- /tmp/rancher2_logs_collector.sh: True server0069: ---------- /tmp/rancher2_logs_collector.sh: True server0070: ---------- /tmp/rancher2_logs_collector.sh: True server0071: ---------- /tmp/rancher2_logs_collector.sh: True server0072: ---------- /tmp/rancher2_logs_collector.sh: True server0073: ---------- /tmp/rancher2_logs_collector.sh: True
-
Make the file executable using
cmd.run
:[[email protected] ~]$ sudo salt -L 'server0068,server0069,server0070,server0071,server0072,server0073' cmd.run 'chmod u+x /tmp/rancher2_logs_collector.sh' server0071: server0073: server0068: server0072: server0070: server0069:
-
I use the same then to execute the script:
[[email protected] ~]$ sudo salt -L 'server0068,server0069,server0070,server0071,server0072,server0073' cmd.run '/tmp/rancher2_logs_collector.sh' server0069: 2021-09-08 06:43:11: Created /tmp/tmp.SfrWsQ6vR6 2021-09-08 06:43:11: Detecting available commands... renice ionoice 2021-09-08 06:43:11: Detecting OS... centos 7 2021-09-08 06:43:11: Detecting k8s distribution... rke 2021-09-08 06:43:11: Detecting init type... systemd 2021-09-08 06:43:11: Collecting system info cp: cannot stat '/run/systemd/resolve/resolv.conf': No such file or directory 2021-09-08 06:43:30: Collecting network info 2021-09-08 06:43:34: Collecting docker info 2021-09-08 06:43:37: Collecting rancher logs 2021-09-08 06:43:37: Collecting k8s component logs 2021-09-08 06:43:40: Collecting system pod logs 2021-09-08 06:43:50: Collecting nginx-proxy info 2021-09-08 06:43:50: Collecting k8s directory state 2021-09-08 06:43:50: Collecting k8s certificates 2021-09-08 06:43:50: Collecting rke etcd info 2021-09-08 06:43:50: Collecting etcdctl output 2021-09-08 06:43:52: Collecting system logs from /var/log cp: omitting directory '/var/log/sa' 2021-09-08 06:43:52: Collecting system logs from journald 2021-09-08 06:43:56: Created /tmp/server0069-2021-09-08_06_43_52.tar.gz 2021-09-08 06:43:56: Removing /tmp/tmp.SfrWsQ6vR6a ...
Now that we have the files e.g. /tmp/server0069-2021-09-08_06_43_52.tar.gz
on the minions comes the tricky part, we have to copy the files back from the minions to the master.
The solution is to use cp module
which allows to “push” files from the minions to the master. But first check your /etc/salt/master.d/master.conf
cause
Since this feature allows a minion to push a file up to the master server it is disabled by default for security purposes. To enable, set
file_recv
toTrue
in the master configuration file, and restart the master.
Once this is done, you can do this:
[[email protected] ~]$ sudo salt -L 'server0068,server0069,server0070,server0071,server0072,server0073' cp.push /tmp/*.tar.gz
server0072:
False
server0069:
False
server0070:
False
server0071:
false
server0073:
false
server0068:
false
At least I thought so but apparently wildcards are not supported, so I have to execute the command for each file, so let’s check the filenames…
[[email protected]] ~]$ sudo salt -L 'server0068,server0069,server0070,server0071,server0072,server0073' cmd.run "ls /tmp/ | grep server"
server0072:
server0072-2021-09-08_06_43_45.tar.gz
server0073:
server0073-2021-09-08_06_43_47.tar.gz
server0069:
server0069-2021-09-08_06_43_52.tar.gz
server0071:
server0071-2021-09-08_06_43_48.tar.gz
server0068:
server0068-2021-09-08_06_43_48.tar.gz
server0070:
server0070-2021-09-08_06_43_47.tar.gz
… and then run this for each file, for example
[[email protected] ~]$ sudo salt 'server0068' cp.push /tmp/server0069-2021-09-08_06_43_52.tar.gz
server0068:
True
At last you should have all files on your salt master in /var/cache/salt/master/minions/saltminion-id
:
[[email protected] ~]$ find /var/cache/salt/master/minions/ -name "devs*.tar.gz"
/var/cache/salt/master/minions/server0069/files/tmp/server0069-2021-09-08_06_43_52.tar.gz
/var/cache/salt/master/minions/server0068/files/tmp/server0068-2021-09-08_06_43_48.tar.gz
/var/cache/salt/master/minions/server0070/files/tmp/server0070-2021-09-08_06_43_47.tar.gz
/var/cache/salt/master/minions/server0071/files/tmp/server0071-2021-09-08_06_43_48.tar.gz
/var/cache/salt/master/minions/server0072/files/tmp/server0072-2021-09-08_06_43_45.tar.gz
/var/cache/salt/master/minions/server0073/files/tmp/server0073-2021-09-08_06_43_47.tar.gz
Hope that help somebody else with a similar setup.