Tag Archives: Synology

Computing – Owl Partially Restored

Heard back from Synology and they indicated I should click the “Repair” option to fix the System Partition Failed error message seen previously.

I did that and our data is now accessible again. However, all the user account info, scheduled tasks (e.g. Glacier backups, notebook backup script), IP configurations, mail configurations, etc. have all been reset.

I downloaded/installed the various packages needed to have the server accessible via the web and configured the IP address settings.

Have a note out to Synology to see if the configurations can be restored somehow. Once I hear back, we’ll get user accounts re-established.

Below is a chronological set of screen caps of the repair process:

 

Our data is still here! This is before performing the “Repair” operation, btw. It seems it just required some time to re-populate directory structure.

 

 

 

 

Still getting a “degraded” error message, but all drives appear normal. However, Disk 3 in the DX513 is not showing; possible cause for “degraded” status?

 

 

 

 

Set up manual IP settings by expanding the “LAN 1″ connection.

Troubleshooting – Synology NAS (Owl) Down After Update

TL;DR – Server didn’t recover after firmware update last night. “Repair” is an option listed in the web interface, but I want to confirm with Synology what will happen if/when I click that button…

The data on Owl is synced here (Google Drive): UW Google Drive

However, not all of Owl was fully synced at the time of this failure, so it seems like a decent amount of data is not accessible. Inaccessible data is mostly from individual user directories.

All high-throughput sequencing is also backed up to Amazon Glacier, so we do have all of that data.

 

Here is what happened, in chronological order:

 

  1. Updated DSM via web interface in “Update & Restore”. Did NOT perform manual install.
  2. System became inaccessible via web interface and Synology Assistant.
  3. The physical unit showed blue, flashing power light and green flashing LAN1 light.
  4. No other lights were illuminated (this includes no lights for any of the drive bays).
  5. The attached expansion unit (DX513) showed steady blue power light, steady green lights on all drive bays, and steady green eSATA light.
  6. I powered down both units via the DS1812+ power button.
  7. I turned on the both units via the DS1812+ power button.
  8. Both units returned to their previous status and were still inaccessible via the web interface and Synology Assistant.
  9. I powered down both units via the DS1812+ power button.
  10. I removed all drives from both units.
  11. I turned on the both units via the DS1812+ power button.
  12. I connected to the DS1812+ via Synology Assistant. A message indicated “No Hard Disk Found on 1812+”.
  13. I powered down both units via the DS1812+ power button.
  14. I added a single HDD to the DS1812+.
  15. I turned on the both units via the DS1812+ power button.
  16. I connected to the DS1812+ via Synology Assistant. I was prompted to install the latest DSM. I followed the steps and created a new admin account. Now the system shows 7 drives in the DS1812+ with a message: “System Partition Failed; Healthy”. Disk 1 shows a “Normal” status; this is the disk that I used to re-install DSM in Step 14. Additionally, the system shows one unused disk in the DX513.
  17. I powered down both units via the web interface.
  18. I removed Disk 1 from DS1812+.
  19. I turned on the both units via the DS1812+ power button.
  20. The DS1812+ returns to its initial state as described in Step 3.
  21. I powered down both units via the DS1812+ power button.
  22. I returned Disk 1 to its bay.
  23. I turned on the both units via the DS1812+ power button.
  24. There’s an option to “Repair” the Volume, but I’m not comfortable doing so until I discuss the in/outs of this with Synology. Submitted a tech support ticket with Synology.

Below are pictures of the entire process, for reference.

 

Server status when I arrived to lab this morning.

 

Pulled the HDDs from both units, in an attempt to be able to connect via Synology Assistant.

 

Units w/o HDDs.

 

No HDDs in units made the server detectable via Synology Assistant, but it indicates “Not installed” in the “Status” column…

 

Successfully connected, but the DS1812+ indicates no HDDs installed.

 

 

Added a single HDD back to the DS1812+. Notice, the drive light is green and the “Status” light is amber. This is actually an improvement over what I saw when I arrived.

 

Added back a single HDD to the DS1812+ and now have this setup menu.

 

I’m prompted to install the Synology DSM.

 

Installing DSM. This “Formatting system partition” message might be related to the eventual error message that I see (“System Partition Failed”) after this is all set up…

 

 

 

 

 

 

 

 

Prompted to create an admin account. This does not bode well, since this is behaving like a brand new installation (i.e. no record of the previous configuration, users, etc.).

 

Continuing set up.

 

All set up…

 

 

Added all the HDDs back and detected via Synology Assistant.

 

This shows that there are no other users – i.e. previous configuration is not detected.

 

After putting all the HDDs back in, got this message after logging in.

 

Looking at the Storage info in DSM; seems bad.

 

 

Physically, the drives all look fine (green lights on all drive bays), despite the indication in the DSM about “System Partition Failed” for all of them (except Disk 1). The expansion unit’s bay lights are actually all green, but were actively being read at the time of picture (i.e. flashing) so the image didn’t capture all of them being green. Amber light on expansion unit reflects what was seen in the DSM – the middle drive is “Not initialized”. Note, the drive is actually inserted, but the handle has been released.

 

This is how I left the system. Notice that after rebooting, the expansion unit no longer shows that “Not initialized” message for Disk 3. Instead, Disk 3 is now detected as installed, but not used…

 

Data Management – Modify Eagle/Owl Cloud Sync Account

Re-examining our backup options for our two Synology servers (Eagle & Owl), I realized that they were both backing up to the just my account on UW’s unlimited Google Drive storage.

The desired backup was to go to our shared UW account, so that others in the lab would have access to the backups.

Strangely, I could not add the shared UW account (srlab) to my list of Google accounts. In order to verify the shared UW account with Google, I had to connect to the servers’ web interfaces in a private browsing session and then I was able to provide the correct user account info/permissions.

Anyway, it’s all going to our shared UW account now.

 

SELECT GOOGLE DRIVE AS THE SYNC PROVIDER:

 

 

 

 

SHARED UW ACCOUNT IS NOT A CHOICE:

 

 

TRY “ADD ACCOUNT”:

 

BUT ADD ACCOUNT DOESN’T WORK (DROP-DOWN MENU DOESN’T OFFER SRLAB AS A CHOICE)”

 

 

 

REPEAT STEPS, BUT CONNECT TO SYNOLOGY VIA PRIVATE BROWSING SESSION AND IT’S GOOD TO GO:

 

 

SET LOCAL AND REMOTE FOLDERS:

 

 

CONFIRMATION THAT IT’S SET UP:

 

 

AND, IT’S RUNNING:

 

Data Management – Synology Cloud Sync to UW Google Drive

After a bit of a scare this weekend (Synology DX513 expansion unit no longer detected two drives after a system update – resolved by powering down the system and rebooting it), we revisited our approach for backing up data.

Our decision was to utilize UW Google Drive, as it provides unlimited storage space!

Synology has an available app for syncing data to/from Google Drive, so I set up both Owl (Synology DS1812+) and Eagle (Synology DS413) to sync all of their data to a shared UW Google Drive account. This should provide a functional backup solution for the massive amounts of data we’re storing and it will simplify tracking where/what is backed up where. Now, instead of wondering if certain directories are backed up via CrashPlan or Backblaze or Time Backup to another Synology server, we know that everything is backed up to Google Drive.

 

 

 

 

 

Server HDD Failure – Owl

Noticed that Owl (Synology DS1812+ server) was beeping.

I also noticed, just like the last time we had to replace a HDD in Owl, that I didn’t receive a notification email… As it turns out, this time the reason no notification email was received was due to the fact that I had changed my UW password and we use my UW account for authorizing usage of the UW email server through Owl. So, the emails Owl’s been trying to send have failed because the authorization password was no longer valid… Yikes!

Anyway, I’ve updated the password on Owl for using the UW email servers and swapped out the bad drive with a backup drive we keep on hand for just such an occasion. See the first post about this subject for a bit more detail on the process of swapping hard drives.

 

Unfortunately, the dead HDD is out of warranty, however we already have another backup drive on-hand.

 

Below are some screen caps of today’s incident:

 

 

 

 

 

 

 

Notice the empty slot in the graphical representation of the disk layout, as well as the “Available Slots” showing 1.

 

 

 

 

After replacing the HDD (but before the system has rebuilt the new HDD), the empty slot is now represented as a green block and the “Available Slots” is now zero and “Unused Disks” is now 1.

 

Data Analysis – Identification of duplicate files on Eagle

Recently, we’ve been bumping into our storage limit on Eagle (our Synology DS413):

 

Being fairly certain that there’s a significant amount of large datasets that is duplicated throughout Eagle, I ran a program on Linux called “fslint”. It searches for duplicates files based on a few parameters and is smart enough to be able to compare files with different filenames that share the same file contents!

I decided to check for duplicate files in the Eagle/archive folder and the Eagle/web folder. Initially, I tried searching for duplicates across all of Eagle, but after a week of running I got tired of waiting for results and ran the analysis on those two directories independently. As such, there is a possibility that there are more duplicates (consuming even more space) across the remainder of Eagle that have not been identified. However, this is a good starting point.

Here are the two output files from the fslint analysis:

 

To get a summary of the fslint output, I tallied the total amount of duplicates files that were >100MB in size. This was performed in a Jupyter notebook (see below):
Notebook Viewer: 20160114_wasted_space_synologies.ipynb
Jupyter (IPython) Notebook File: 20160114_wasted_space_synologies.ipynb

 

Here are the cleaned output files from the fslint analysis:

 

Summary

There are duplicates of files (>100MB in size) that are consuming at least 730GB!

Since the majority of these files exist in the Eagle/web folder, careful consideration will have to be taken in determining which duplicates (if any) can be deleted since it’s highly possible that there are notebooks that link to some of the files. Regardless, this analysis shows just how space is being consumed by the presence of large, duplicate files; something to consider for future data handling/storage/analysis with Eagle.

Data Storage – Synology DX513

Running a bit low on storage on Owl (Synology DS1812+) and we will be receiving a ton of data in the next few months, so we purchased a Synology DX513. It’s an expansion unit designed specifically for seamlessly expanding our existing storage volume on Owl.

Installed 5 x 8TB Seagate HDDs and connected to Owl with the supplied eSATA cable.

Now, we just need to wait (possibly days) for the full expansion to be completed.

Uninterruptible Power Supplies (UPS)

A new UPS we installed this week for our qPCR machine (Opticon2 – BioRad) to handle power surges and power outages doesn’t seem to be working properly. With the qPCR machine (and computer and NanoDrop1000) plugged into the “battery” outlets on the UPS, this is what happens when the Opticon goes through a heating cycle:

The UPS becomes overloaded when the Opticon is in a heating cycle.

 

And, sometimes, that results in triggering a fault, shutting everything off in the middle of a qPCR run:

Fault message indicating unit overload.

 

This is supremely lame because having a battery backup is a great way to prevent the qPCR machine from shutting off when a power outage occurs!

 

I switched the Opticon (and computer and NanoDrop1000) to the outlets that are solely for surge protection. Check out what happens when I run the qPCR machine now:

Opticon plugged in to surge protection outlet while in heating cycle. Notice that output load is 0%.

 

So, I guess we’ll settle for at least having the surge protection aspect of things.

 

While handling this UPS issue, I realized that the two Synology servers we have possess a built-in UPS monitor. So, I connected the USB cables to/from each of the UPS that each server is plugged into and enabled UPS shutdown in the Synology Diskstation Management (DSM):

 

Eagle

 

Owl

 

Now, both Synology units will enter Safe Mode when the UPS they’re connected to reaches a low battery status. This will help minimize data loss/corruption during the next extended power outage we experience.

Server Email Notifications Fix – Eagle

The system was previously set to use Steven’s Comcast SMTP server. Sending a test email from Eagle failed, indicating authentication failure. I changed this to use the University of Washington’s email server for outgoing messages. Here’s how…

In the Synology Disk Station Manager (DSM):

Control Panel > Notifications

  • Service provider: Custom SMTP Server
  • SMTP server: smtp.washington.edu
  • SMTP port: 587
  • Username: myUWnetID@uw.edu
  • Password: myUWpassword

Server HDD Failure – Owl

We had our first true test of the Synology RAID redundancy with our Synology 1812+ server (Owl). One of the hard drives (HDD) failed. All of the other drives were fine, the data was intact and we had a new replacement HDD on hand. However, there was one shortcoming: no email notification of the drive failure. Luckily, the Synology server is next to Steven’s office and he could hear an audible beeping alerting him to the fact that something was wrong. In any case, the email notifications have been fixed and a replacement hard drive was added to the system. Here’s how these things were accomplished.

 

Fix email notifications

The system was previously set to use Steven’s Comcast SMTP server. Sending a test email from Owl failed, indicating authentication failure. I changed this to use the University of Washington’s email server for outgoing messages. Here’s how…

In the Synology Disk Station Manager (DSM):

Control Panel > Notifications

  • Service provider: Custom SMTP Server
  • SMTP server: smtp.washington.edu
  • SMTP port: 587
  • Username: myUWnetID@uw.edu
  • Password: myUWpassword

Interesting note, there’s a “Push Service” tab in the “Notifications” window. This allows you to have Synology send emails to email addresses when the server has an issue. This eliminates the need for the SMTP settings shown above which may not be easy to find and/or understand for a given email service provider. The “Push Service” appears to be much simpler and more user friendly to set up.

 

Hot Swap HDD

We’ve kept a backup HDD on hand for just this occasion, so the HDD failure wasn’t too concerning. Here’re the steps I followed to swap the HDD and have the Synology system initialize/build the new HDD:

 

Remove the dead HDD and put the new HDD in.

 

 

Initialize/build/repair the new HDD.

In Synology DSM:

Storage Manger > Volume

Notice, there should be eight drives listed, but since one has died, only seven are shown:

 

 

 

 

 

 

 

That’s it! Easy breezy!

I’ve checked with Seagate on the dead HDD and it is still under warranty. Will get that returned and also purchase a new backup drive to have on hand.