SteelCentral AppInternals CLI throwing "IOError: [Errno 28] No space left on device" error after an upgrade to 10.18.57

Categories: SteelCentral AppInternals (AppInternals Xpert)
Solution Number: S33434

Issue

We have SteelCentral AppInternals Analysis server installation which was upgraded to version 10.18.57. Now we are unable to access the appliance web UI and when we log into CLI we are presented with below error

Logged from file Logging.py, line 20
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file clish_output.py, line 127
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 883, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 843, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file clish_output.py, line 149


How do we resolve the issue and bring the appliance back online again?

Solution

The issue here is because of a missing symlink. The symlink is for
/var/log/appinternals to /mnt/data/log'

In cases where you run into the issue mentioned above, you will be missing the above symlink and the root partition (/) is probably filled. This is what might be causing the error in the CLI and also disabling the web UI access.

To fix the issue please  do the following steps
  1. Log into the appliance as root
  2. Stop all services including Web UI and auth service
  3. Execute the below commands

    rm -f /var/log/appinternals
    mkdir /var/log/appinternals
     
    cp -pr /mnt/data/log/* /var/log/appinternals/
    rm -rf /mnt/data/log

     
    We can then copy the logs to /mnt/data using the --preserve=all option that should also copy over the SELinux context
     
    mkdir /mnt/data/log
    cp -r --preserve=all /var/log/appinternals/* /mnt/data/log/
     
    rm -rf /var/log/appinternals
    ln -s /mnt/data/log /var/log/appinternals

  4. Start all services back
This should move all the data to appropriate location and also recreate the symlink which was missing. This is a known bug and has been fixed in upcoming release.

Please also take a look at this internal only article which might help clearing up more space

**Please open a technical support case if you run into any issues or have any questions

Environment

SteelCentral AppInternals Analysis server 10.x
NOTICE: Riverbed® product names have changed. Please refer to the Product List for a complete list of product names.
Last Modified: 2019-08-06
Can't find an answer? Create a case