Category Archives: Computer Servicing

Ubuntu – Fix “No Video Signal” Issue on Emu

An issue with Emu cropped up a few weeks ago that was seemingly caused by upgrading from Ubuntu 16.04 to 18.04.

However, the problems only seemed related to using Emu via the GUI; users could still use Emu as a headless computer via SSH.

Today, I was upgrading some packages and noticed two things:

  1. When initially logging in to Emu. 
    
sam@swoose:~$ ssh emu
Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-57-generic x86_64)

* Documentation:  https://help.ubuntu.com
* Management:     https://landscape.canonical.com
* Support:        https://ubuntu.com/advantage

0 packages can be updated.
0 updates are security updates.

New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

You have mail.
Last login: Tue Oct  2 07:30:32 2018 from 128.95.149.75

    This is showing that Emu is still running Ubuntu 16.04, not 18.04 as presumed!

  2. An error in the GRUB config generation process when upgrading packages.


run-parts: executing /etc/kernel/postrm.d/zz-update-grub 4.4.0-134-generic /boot/vmlinuz-4.4.0-134-generic
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.4.0-137-generic
Found initrd image: /boot/initrd.img-4.4.0-137-generic
Found linux image: /boot/vmlinuz-4.4.0-135-generic
Found initrd image: /boot/initrd.img-4.4.0-135-generic
Found linux image: /boot/vmlinuz-4.4.0-57-generic
Found initrd image: /boot/initrd.img-4.4.0-57-generic
Found linux image: /boot/vmlinuz-4.4.0-53-generic
Found initrd image: /boot/initrd.img-4.4.0-53-generic
error: syntax error.
error: Incorrect command.
error: syntax error.
Syntax error at line 98
Syntax errors are detected in generated GRUB config file.
Ensure that there are no errors in /etc/default/grub
and /etc/grub.d/* files or please file a bug report with
/boot/grub/grub.cfg.new file attached.
done
Processing triggers for libc-bin (2.23-0ubuntu10) ...

These two bits of information led me to believe the problem wasn’t that the system upgrade to 18.04 was incompatible with these old Apple Xserve hardware (since the upgrade didn’t actually get implemented) and instead was that the upgrade might have been initiated, but aborted, which modified the GRUB configuration file(s), breaking the GUI; much like the problem I previously addressed earlier this summer.

When I fixed the display/GUI issues with Emu and Roadrunner earlier this summer, I noted that the /etc/default/grub files on each of the computers were slightly different, despite the fact that these two computers should be identical. So, I replaced the /etc/default/grub file on Emu with the file from Roadrunner and rebooted Emu.

Contents of /etc/default/grub file on Emu/Roadrunner, for future reference:


# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Voila! Emu now has a functional display/GUI again!

Ubuntu – Fix “No Video Signal” Issue on Emu/Roadrunner

Both Apple Xserves (Emu/Roadrunner) running Ubuntu (16.04LTS) experienced the same issue – the monitor would indicate “No Video Signal”, would go dark, and wasn’t responsive to keyboard/mouse movements. However, you could ssh into both machines w/o issue.

Although having these machines be “headless” (i.e. with no display) is usually fine, it’s not ideal for a couple of reasons:

  1. Difficult to use for other lab members who aren’t as familiar with SSH – specifically if they would want to use a Jupyter Notebook remotely (this would require setting up a tunnel to their own computer).

  2. Can’t use Remmina Remote Desktop until a user has physically logged in from the Ubuntu login screen at least once, in order to launch Remmina.

The second aspect was the major impetus in me finally being motivated to deal with this. Accessing these computers via remote desktop is much easier to manage long-running Jupyter Notebooks instead of relying on an SSH tunnel. The tunnel greatly limits my access to the Jupyter Notebook outside of the computer that has the tunnel set up.

Well, this led me down a horrible rabbit hole of Linux stuff that I won’t get fully in to (particularly, since I didn’t understand most of it and can’t remember all the crazy stuff I read/tried).

However, here’s the gist:

  1. Needed to edit /etc/default/grub

  2. After editing, needed to update grub config file: sudo update-grub

Despite the fact that both machines are (or, should be) identical, I did not get the same results. The edits I made to the /etc/default/grub file on Emu worked immediately. The edits were:

  1. Add nomodeset to this (this is the edited line) line (this seemed to be the most common suggestion for fixing the “No Video Signal” issue):

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nomodeset"

  1. Comment out this line (this line was triggering an error/warning about writing the config file when running the update-grub command):

#GRUB_HIDDEN_TIMEOUT=0

For some reason, Roadrunner did not take kindly to those changes and it took a long time to resolve, ending with changing permissions on ~/.Xauthority back to their original permissions (they got altered when I ran some command – sudo startx or something) to get out of a login loop.

Regardless, both are fixed, both can be used when physically sitting at the computer, and both can be accessed remotely using Remmina!

Ubuntu Installation – Convert Apple Xserve “bigfish” to Ubuntu

Due to hardware limitations on the Apple Xserves we have, we can’t use drives >2TB in size. “Bigfish” was set up to be RAID’d and, as such, has three existing HDDs installed.

We wanted to upgrade the HDD size and convert over to Linux (Ubuntu) so that we could utilize the Linux operating system for some of our bioinformatics programs that won’t run on OSX.

I installed Ubuntu 16.04LTS to the SSD boot drive (128GB) and installed three, 2TB HDDs. However, it cannot detect the HDDs due to the Apple hardware RAID controller! Searching the internet has revealed that this is a commonly encountered issue with RAID’d Apple Xserves and Linux installs.

I haven’t come across a means by which to remedy this. Will likely have to install an OS X version in order to make this computer usable. Although, that won’t limit us too terribly in regards to program usage. Most programs will run fine on OSX.

Computing – Owl Partially Restored

Heard back from Synology and they indicated I should click the “Repair” option to fix the System Partition Failed error message seen previously.

I did that and our data is now accessible again. However, all the user account info, scheduled tasks (e.g. Glacier backups, notebook backup script), IP configurations, mail configurations, etc. have all been reset.

I downloaded/installed the various packages needed to have the server accessible via the web and configured the IP address settings.

Have a note out to Synology to see if the configurations can be restored somehow. Once I hear back, we’ll get user accounts re-established.

Below is a chronological set of screen caps of the repair process:

 

Our data is still here! This is before performing the “Repair” operation, btw. It seems it just required some time to re-populate directory structure.

 

 

 

 

Still getting a “degraded” error message, but all drives appear normal. However, Disk 3 in the DX513 is not showing; possible cause for “degraded” status?

 

 

 

 

Set up manual IP settings by expanding the “LAN 1″ connection.

Troubleshooting – Synology NAS (Owl) Down After Update

TL;DR – Server didn’t recover after firmware update last night. “Repair” is an option listed in the web interface, but I want to confirm with Synology what will happen if/when I click that button…

The data on Owl is synced here (Google Drive): UW Google Drive

However, not all of Owl was fully synced at the time of this failure, so it seems like a decent amount of data is not accessible. Inaccessible data is mostly from individual user directories.

All high-throughput sequencing is also backed up to Amazon Glacier, so we do have all of that data.

 

Here is what happened, in chronological order:

 

  1. Updated DSM via web interface in “Update & Restore”. Did NOT perform manual install.
  2. System became inaccessible via web interface and Synology Assistant.
  3. The physical unit showed blue, flashing power light and green flashing LAN1 light.
  4. No other lights were illuminated (this includes no lights for any of the drive bays).
  5. The attached expansion unit (DX513) showed steady blue power light, steady green lights on all drive bays, and steady green eSATA light.
  6. I powered down both units via the DS1812+ power button.
  7. I turned on the both units via the DS1812+ power button.
  8. Both units returned to their previous status and were still inaccessible via the web interface and Synology Assistant.
  9. I powered down both units via the DS1812+ power button.
  10. I removed all drives from both units.
  11. I turned on the both units via the DS1812+ power button.
  12. I connected to the DS1812+ via Synology Assistant. A message indicated “No Hard Disk Found on 1812+”.
  13. I powered down both units via the DS1812+ power button.
  14. I added a single HDD to the DS1812+.
  15. I turned on the both units via the DS1812+ power button.
  16. I connected to the DS1812+ via Synology Assistant. I was prompted to install the latest DSM. I followed the steps and created a new admin account. Now the system shows 7 drives in the DS1812+ with a message: “System Partition Failed; Healthy”. Disk 1 shows a “Normal” status; this is the disk that I used to re-install DSM in Step 14. Additionally, the system shows one unused disk in the DX513.
  17. I powered down both units via the web interface.
  18. I removed Disk 1 from DS1812+.
  19. I turned on the both units via the DS1812+ power button.
  20. The DS1812+ returns to its initial state as described in Step 3.
  21. I powered down both units via the DS1812+ power button.
  22. I returned Disk 1 to its bay.
  23. I turned on the both units via the DS1812+ power button.
  24. There’s an option to “Repair” the Volume, but I’m not comfortable doing so until I discuss the in/outs of this with Synology. Submitted a tech support ticket with Synology.

Below are pictures of the entire process, for reference.

 

Server status when I arrived to lab this morning.

 

Pulled the HDDs from both units, in an attempt to be able to connect via Synology Assistant.

 

Units w/o HDDs.

 

No HDDs in units made the server detectable via Synology Assistant, but it indicates “Not installed” in the “Status” column…

 

Successfully connected, but the DS1812+ indicates no HDDs installed.

 

 

Added a single HDD back to the DS1812+. Notice, the drive light is green and the “Status” light is amber. This is actually an improvement over what I saw when I arrived.

 

Added back a single HDD to the DS1812+ and now have this setup menu.

 

I’m prompted to install the Synology DSM.

 

Installing DSM. This “Formatting system partition” message might be related to the eventual error message that I see (“System Partition Failed”) after this is all set up…

 

 

 

 

 

 

 

 

Prompted to create an admin account. This does not bode well, since this is behaving like a brand new installation (i.e. no record of the previous configuration, users, etc.).

 

Continuing set up.

 

All set up…

 

 

Added all the HDDs back and detected via Synology Assistant.

 

This shows that there are no other users – i.e. previous configuration is not detected.

 

After putting all the HDDs back in, got this message after logging in.

 

Looking at the Storage info in DSM; seems bad.

 

 

Physically, the drives all look fine (green lights on all drive bays), despite the indication in the DSM about “System Partition Failed” for all of them (except Disk 1). The expansion unit’s bay lights are actually all green, but were actively being read at the time of picture (i.e. flashing) so the image didn’t capture all of them being green. Amber light on expansion unit reflects what was seen in the DSM – the middle drive is “Not initialized”. Note, the drive is actually inserted, but the handle has been released.

 

This is how I left the system. Notice that after rebooting, the expansion unit no longer shows that “Not initialized” message for Disk 3. Instead, Disk 3 is now detected as installed, but not used…

 

Computer Management – Additional Configurations for Reformatted Xserves

Sean got the remaining Xserves configured to run independently from the master node of the cluster they belonged to and installed OS X 10.11 (El Capitan).

The new computer names are Ostrich (formerly node004) and Emu (formerly node002).

 

He enabled remote screen sharing and remote access for them.

Sean also installed a working hard drive on Roadrunner and got that back up and running.

I went through this morning and configured the computers with some other changes (some for my user account, others for the entire computer):

  • Renamed computers to reflect just the corresponding bird name (hostnames had been labeled as “bird name’s Xserve”)

  • Created srlab user accounts

  • Changed srlab user accounts to Standard instead of Administrative

  • Created steven user account

  • Turned on Firewalls

  • Granted remote login access to all users (instead of just Administrators)

  • Installed Docker Toolbox

  • Changed power settings to start automatically after power failure

  • Added computer name to login screen via Terminal:

sudo defaults write /Library/Preferences/com.ap\ple.loginwindow LoginwindowText "TEXT GOES HERE"
  • Changed computer HostName via Terminal so that Terminal displays computer name:
sudo scutil --set HostName "TEXT GOES HERE"

  • Installed Mac Homebrew (I don’t know if installation of Homebrew is “global” – i.e. installs for all users)

  • Used Mac Homebrew to install wget

  • Used Mac Homebrew to install tmux

Data Management – Modify Eagle/Owl Cloud Sync Account

Re-examining our backup options for our two Synology servers (Eagle & Owl), I realized that they were both backing up to the just my account on UW’s unlimited Google Drive storage.

The desired backup was to go to our shared UW account, so that others in the lab would have access to the backups.

Strangely, I could not add the shared UW account (srlab) to my list of Google accounts. In order to verify the shared UW account with Google, I had to connect to the servers’ web interfaces in a private browsing session and then I was able to provide the correct user account info/permissions.

Anyway, it’s all going to our shared UW account now.

 

SELECT GOOGLE DRIVE AS THE SYNC PROVIDER:

 

 

 

 

SHARED UW ACCOUNT IS NOT A CHOICE:

 

 

TRY “ADD ACCOUNT”:

 

BUT ADD ACCOUNT DOESN’T WORK (DROP-DOWN MENU DOESN’T OFFER SRLAB AS A CHOICE)”

 

 

 

REPEAT STEPS, BUT CONNECT TO SYNOLOGY VIA PRIVATE BROWSING SESSION AND IT’S GOOD TO GO:

 

 

SET LOCAL AND REMOTE FOLDERS:

 

 

CONFIRMATION THAT IT’S SET UP:

 

 

AND, IT’S RUNNING:

 

Data Management – Synology Cloud Sync to UW Google Drive

After a bit of a scare this weekend (Synology DX513 expansion unit no longer detected two drives after a system update – resolved by powering down the system and rebooting it), we revisited our approach for backing up data.

Our decision was to utilize UW Google Drive, as it provides unlimited storage space!

Synology has an available app for syncing data to/from Google Drive, so I set up both Owl (Synology DS1812+) and Eagle (Synology DS413) to sync all of their data to a shared UW Google Drive account. This should provide a functional backup solution for the massive amounts of data we’re storing and it will simplify tracking where/what is backed up where. Now, instead of wondering if certain directories are backed up via CrashPlan or Backblaze or Time Backup to another Synology server, we know that everything is backed up to Google Drive.

 

 

 

 

 

Server HDD Failure – Owl

Noticed that Owl (Synology DS1812+ server) was beeping.

I also noticed, just like the last time we had to replace a HDD in Owl, that I didn’t receive a notification email… As it turns out, this time the reason no notification email was received was due to the fact that I had changed my UW password and we use my UW account for authorizing usage of the UW email server through Owl. So, the emails Owl’s been trying to send have failed because the authorization password was no longer valid… Yikes!

Anyway, I’ve updated the password on Owl for using the UW email servers and swapped out the bad drive with a backup drive we keep on hand for just such an occasion. See the first post about this subject for a bit more detail on the process of swapping hard drives.

 

Unfortunately, the dead HDD is out of warranty, however we already have another backup drive on-hand.

 

Below are some screen caps of today’s incident:

 

 

 

 

 

 

 

Notice the empty slot in the graphical representation of the disk layout, as well as the “Available Slots” showing 1.

 

 

 

 

After replacing the HDD (but before the system has rebuilt the new HDD), the empty slot is now represented as a green block and the “Available Slots” is now zero and “Unused Disks” is now 1.