Sosreports

A sosreport is an advanced tool used for deep Linux system troubleshooting. It essentially takes a snapshot of a user's machine and puts it into a package that can looked through or sent to another user or machine. The report includes things like log files, system configuration files, device memory, storage, cpu info & much more.

This way you can look through someone's sosreport instead of intrusively sshing into their machine. All important system information is all in the sosreport.

The official man page for 'sos' says this:

   sos  is  a  diagnostic data collection utility, used by system administrators, support representa‐
   tives, and the like to assist in troubleshooting issues with a system or group of systems.

   The most well known function is  sos report or sos report as it was previously known.

   An sos archive is typically requested by support organizations to collect  baseline  configuration
   and system data from which to begin the troubleshooting process.

Let's take a look at a sosreport generated from one of my VMs named 'feminine-rat' (don't ask me why it's named that lol)

You can see it looks like any other Linux directory and can be explored the same way as any Linux file system.

Here are some places to look in a sosreport for troubleshooting info:

• /var/logs contains all the logs you need for a system
• /var/crash contains helpful crash logs
• /etc contains all of the configuration files for a system and services running on the system.
• /boot contains boot configurations and info
• use 'cat free' for quick memory info
• use 'cat ip_addr' for quick network info
• use 'cat uname' for quick kernel and linux distro info

Generating a sosreport

You can install sos with the command:

sudo apt install sosreport

You can generate a sosreport for your system with command:

sudo sos report --all-logs --batch

This will create a .tar.xz package under the /tmp directory. Go ahead and move this with cp command to another directory to unpack (I'd recommend creating a designated 'sosreport' directory).

Unpacking a sosreport

To see all of these important logs and configuration files you'll need to unpack this .tar.xz file. Make sure you are in a designated directory for this sosreport to avoid making a mess.

Before unpacking you may want to also run:

sudo chown <yourusername> yoursosreportname.tar.xz

To unpack this .tar.xz file you've just generated run the command:

tar -xf yoursosreportname.tar.xz

Great! Now you can cd into your sosreport and have a look around.

xsos & hotsos

xsos and hotsos are additional command line tools you can install that will look through a sosreport and output your desired information.

You can think of xsos as your quick system information tool. While hotsos you will often use to find quick info on Canonical specific software issues (like Ceph, LXD, Maas, & Openstack). Hotsos also has a nifty 'known bugs' tool that will show you any bugs your system might have with current versions of packages and/or OS version.

xsos

You can install xsos with the following command:

sudo snap install xsos

You can use xsos by running:

xsos <yoursosreportname>

Common xsos flags to use for specific info:

-a → all

-k → kernel info

-c → CPU info

-m → memory & swap

-s → storage

-n → networking

Here's an example of what an xsos output looks like with the -n network flag for my VM 'feminine-rat' sosreport:

hotsos

You can install hotsos just the same way you did xsos with command:

sudo snap install xsos

To use hotsos type command:

hotsos <yoursosreportname>

Some helpful plugin flags include:

--lxd → LXD info

--maas → MAAS info

--ceph → ceph info

--kernel → kernel info & potential issues

--mysql → mysql info & potential issues

--landscape → landscape info

--system → system info & potential issues

Here's an example output using hotsos and --kernel flag:

xsos vs hotsos

In a lot ways xsos and hotsos serve similar purposes. They both look through a sosreport and output info in an easy to read way.

The simplest way I look at the difference is xsos is best used (and has better visual representations) for getting basic system info like CPU, memory, and storage. Take for example the very pleasing way xsos outputs memory info for a sosreport:

It's also helpful to mention you can just use xsos on your own system without a sosreport. Just run the command by itself without a sosreport and xsos will scan your own system.

In contrast hotsos must be run with an actual sosreport. However, a benefit to hotsos is that you can use it on a .tar.xz file without having to unpack it yourself.

The main use for hotsos is to dig deeper into specific services or softwares running on your system like LXD or mysql. Hotsos also does a better job giving feedback on potential issues/conflicts with current versions of something on your system.

So there you go. xsos and hotsos won't always give you a perfect solution right away but they will give you all the system info you need to start troubleshooting an issue without having to ssh into someone's machine.