Optimize Your Synology NAS for Downloading

I recently bought a NAS so my data is safe & available, with the benefit of being low power / noise / heat. I've considered Netgear, QNAP, but decided to go for a Synology as it was affordable, still had a big community, decent reviews & Time Machine support.

I wanted 4 bays so that I could use RAID5 and only lose 25% space on fault tolerance instead of (RAID1) 50%. Synology has 2 offerings in the 4-bay home-user range this year: the DS411+ (fast) and the DS411j (slow).

I figured as long as it can blast a bandwidth adequate for 1080p over my network, I'd save myself some money (300$vs 600$), heat and power consumption that come with the more powerful + version.

However now that it's here I want it do download from newsgroups and am running into performance issues with my junior edition.

No worries. With a little bit of hacking you can squeeze just enough performance out of this thing to make sense of it all.

Here's how I turned my budget NAS that's mediocre at 8 things into a more powerful one that's good at 3 things: downloading / file serving / backups.

Warning

This article assumes you're somewhat skilled in Linux. By applying these suggestions you could seriously mess up your Disk Station.

I'm doing this on a DS411j running DSM 3.0. Your mileage may vary.

Downloading

In an earlier article I described how to install SABnzbd. After testdriving it for a while I was never able to get it to download above 3MB/s (2 average). Where as nzbget (the program used by Synology's own Download Station), peaks at 8MB/s (6 average).

Although I really like that SABnzbd automatically unpacks your downloads, these speed differences made me decide to go back to nzbget. The j is just not powerful enough to do SABnzbd at these speeds, and I can write auto-unpackers myself.

Optimal Config

I found that optimal speeds can be reached by letting your Synology download with 8 connections on 1 single download. With these settings the load reaches 11, so don't expect your NAS to do anything else while it's busy. But at least you're saturating your connection.

If you want it to multitask, limit it to 1 connection on 1 single download at any time, but you won't see it peak beyond 2MB/s.

If you use it for torrents as well, you don't want 1 slow torrent blocking the rest of the queue. In that case, set it to 2 to 3 connections with 2 to 4 threads each for optimal downloading.

Turn Off Unused Protocols

Decide on 1 file-sharing protocol (I chose Mac File service cause all my systems speak it and use Time Machine, but SMB/Windows is typically the right choice). Disable the rest in your configuration panel, saving a few precious MBs of RAM.

This is all just done from your web-interface.

SSH Access

Before you can do any hacking on your Synology, turn on SSH access in the web-interface's control panel. You can now type: ssh root@<nas ip>. Followed by sh. The root password is the same as admin password.

AppStore :)

Get your hands on ipkg, which is like your Synology's secret AppStore. From here on, it's much easier to install cool additional software.

Turn Off Media Indexers to Free Up CPU & Memory

When I logged in to see what was eating up my NAS' resources, I saw a lot of processes running that I don't need such as thumbnail generators and media indexers (ffmpeg & convert). They were endlessly consuming 100% CPU, leaving nothing for my other tasks.

Any currently available NAS is a terrible media streamer. And that's ok, just get yourself an AC Ryan ($80) or Boxee Box ($250) to do that instead and dedicate your NAS to less tasks.

In my case that meant killing off all these wannabe media processes that are eating up your poor handheld CPU with 128MB RAM (every MB we'll save from this point forward counts to faster download speeds :)

So if you don't use the Photo/Media/iTunes station and would like more power for other tasks, consider turning off indexers:

1. Turn off all services in the bottom configuration panel (iTunes, everything except Download Station, unless you're going to use SABnzbd for this)

2. Login as root via SSH and stop all indexing by pasting:

/usr/syno/etc/rc.d/S??synoindexd.sh stop
/usr/syno/etc/rc.d/S??synomkflvd.sh stop
/usr/syno/etc/rc.d/S??synomkthumbd.sh stop
killall -9 convert
killall -9 ffmpeg
# If you don't use Download Station (but e.g. SABnzbd instead):
# /usr/syno/etc/rc.d/S??pgsql.sh stop

3. Make sure they won't restart on your next reboot by pasting:

chmod a-x /usr/syno/etc/rc.d/S??synoindexd.sh
chmod a-x /usr/syno/etc/rc.d/S??synomkflvd.sh
chmod a-x /usr/syno/etc/rc.d/S??synomkthumbd.sh
# If you don't use Download Station (but e.g. SABnzbd instead):
# chmod a-x /usr/syno/etc/rc.d/S??pgsql.sh

Hint) After a DSM firmware upgrade, you need to repeat these steps.

Custom Cleanup & Rename Script Cause SAB Is Too Slow

Building your own cleanup scripts can be fun (and risky). If you want to get into it, you'll need some system tools at your disposal.

Here's what I cooked up to take care of my downloads:

#!/opt/bin/bash
# @todo: Don't delete parent dir if Dir == Root
# @todo: Root = $1 - But what about series!

set +x
export PATH="/opt/bin:/opt/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/syno/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/syno/bin:/usr/syno/sbin:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/syno/bin:/usr/syno/sbin:/usr/local/bin:/usr/local/sbin"

# Locking
LockFile="/volume1/downloads/nas_is_unpacking.lock"
if [ -f "${LockFile}" ]; then
	echo "Lockfile still exists: ${LockFile}. Aborting"
	exit 0
fi
trap "{ rm -f ${LockFile} ; exit 255; }" EXIT
date > ${LockFile}

echo "Running ${0} on $(date)"

Root="/volume1/downloads"
Home="$(pwd)"
Purged=""

# Downloadstation
echo "Looking for downloadstation tasks..."
Prevdir=""
find ${Root}/_queue -mmin +5 -iname '*.nzb' -o -iname '*.torrent' |sort | while read File; do
	Dir="$(dirname "$File")"
	if [ "${Prevdir}" != "${Dir}" ]; then
		cd "${Dir}"
		echo ""
		echo "= $(pwd)"
		echo "================================================================================================"
	fi
	# Process the first par file in this directory thanks to |sort
	/opt/bin/downloadstation add "${File}"
	if [ $? -eq 0 ]; then
		echo "Successfully added ${File}; purging file"
		rm -f "${File}"
	else
		echo "Unable to add ${File}"
	fi
	Prevdir=$Dir
done
cd "${Home}"

# PAR
echo "Looking for files to repair..."
Prevdir=""
find ${Root} -mmin +5 -iname '*.par2' |sort | while read File; do
	Dir="$(dirname "$File")"
	if [ "${Prevdir}" != "${Dir}" ]; then
		cd "${Dir}"
		echo ""
		echo "= $(pwd)"
		echo "================================================================================================"
		# Process the first par file in this directory thanks to |sort
		par2 r "${File}"
		if [ $? -eq 0 ]; then
			echo "Successfully repaired; purging par files"
			rm -f *.par2
			rm -f *.PAR2
		else
			echo "Unable to repair; purging entire directory"
			Purged="${Purged}${Dir}\n"
			cd ..
			rm -rf "${Dir}"
		fi
	fi
	Prevdir=$Dir
done
cd "${Home}"

# RAR
echo "Looking for rar files to unpack..."
Prevdir=""
find ${Root} -mmin +5 -iname '*.rar' |sort | while read File; do
	Dir="$(dirname "$File")"
	if [ "${Prevdir}" != "${Dir}" ]; then
		cd "${Dir}"
		echo ""
		echo "= $(pwd)"
		echo "================================================================================================"
		# Process the first rar file in this directory thanks to |sort
		unrar e -y -o+ -p- "${File}"
		if [ $? -eq 0 ]; then
			echo "Successfully unpacked; purging rar files"
			rm -f *.rar
			rm -f *.r[0-9][0-9]
			rm -f *.s[0-9][0-9]
			rm -f *.t[0-9][0-9]
		else
			echo "Unable to unpack; purging entire directory"
			Purged="${Purged}${Dir}\n"
			cd ..
			rm -rf "${Dir}"
		fi
	fi
	Prevdir=$Dir
done
cd "${Home}"

# 7zip
echo "Looking for 7zip files to unpack..."
Prevdir=""
find ${Root} -mmin +5 -iname '*.7z.001' |sort | while read File; do
	Dir="$(dirname "$File")"
	if [ "${Prevdir}" != "${Dir}" ]; then
		cd "${Dir}"
		echo ""
		echo "= $(pwd)"
		echo "================================================================================================"
		# Process the first 7zip file in this directory thanks to |sort
		7z x "${File}"
		if [ $? -eq 0 ]; then
			echo "Successfully unpacked; purging rar files"
			rm -f *.7z.[0-9][0-9][0-9]
		else
			echo "Unable to unpack; purging entire directory"
			Purged="${Purged}${Dir}\n"
			cd ..
			rm -rf "${Dir}"
		fi
	fi
	Prevdir=$Dir
done
cd "${Home}"

# Move 1 Dir Up & Rename to Parent Dir
echo "Looking for files to clean..."
Prevdir=""
find ${Root} -mmin +5 -iname '*.mkv' -o -iname '*.avi' |sort | while read File; do
	Dir="$(dirname "$File")"
	Parent="$(dirname "$Dir")"
	if [ "${Prevdir}" != "${Dir}" ]; then
		cd "${Dir}"
		echo ""
		echo "= $(pwd)"
		echo "================================================================================================"
		rm -f *.1 2> /dev/null
		rm -f *.2 2> /dev/null
		rm -f *.nzb 2> /dev/null
		rm -f *.nfo 2> /dev/null
		rm -f *.par2_hellanzb_dupe0 2> /dev/null
		rm -f *.sfv 2> /dev/null
		rm -f *.srr 2> /dev/null
		rm -f *.segment000[0-9] 2> /dev/null
		# in the middle
		rm -f *[.-][Ss][Aa][Mm][Pp][Ll][Ee][.-]*.{mkv,avi,mpg,srs} 2> /dev/null
		# at the end
		rm -f *[.-][Ss][Aa][Mm][Pp][Ll][Ee].{mkv,avi,mpg,srs} 2> /dev/null
		# at the beginning
		rm -f [Ss][Aa][Mm][Pp][Ll][Ee][.-]*.{mkv,avi,mpg,srs} 2> /dev/null
		# complete
		rm -f [Ss][Aa][Mm][Pp][Ll][Ee].{mkv,avi,mpg,srs} 2> /dev/null

		# Synology media thumbs
		rm -rf @eaDir
	fi
	Prevdir=$Dir
done
cd "${Home}"

# Move Lonely Files 1 Dir Up & Rename to Parent Dir
echo "Looking for lonely files to promote 1 directory up..."
Prevdir=""
find ${Root} -mmin +5 -iname '*.mkv' -o -iname '*.avi' -o -iname '*.ts' |sort | while read File; do
	Dir="$(dirname "$File")"
	Parent="$(dirname "$Dir")"
	if [ "${Prevdir}" != "${Dir}" ]; then
		cd "${Dir}"

		if [ "$(ls -l |grep -v 'total ' |wc -l)" = "1" ]; then
			Basedir="$(basename "${Dir}")"
			Newname="$(echo "${Basedir}")"
			Ext=${File##*.}
			Newname="${Newname}.${Ext}"

			#cmd="mv \"${File}\" \"${Parent}/${Newname}\" && rmdir \"${Dir}\""
			mv "${File}" "${Parent}/${Newname}" && rmdir "${Dir}"
			echo "promoted: ${Parent}/${Newname}"
		fi
	fi
	Prevdir=$Dir
done
cd "${Home}"

## TV Episodes
# Please Use FileBot Instead. Much Better Results.
#if [ "${1}" = "tvnamer" ]; then
#	echo "Looking for tv episodes to rename..."
#	tvnamer -r --batch /volume1/video/series
#fi

# REPORT
if [ -n "${Purged}" ]; then
	echo ""
	echo "Had to purge these directories cause they were damaged beyond repair:"
	echo -e "${Purged}"
fi

echo "Done"

It runs every 15 minutes by cron, will remove broken downloads, unpack complete downloads, move lonely files 1 directory up, delete a bunch of unwanted extensions, etc. It makes a few assumptions (e.g. downloads must be in /volume1/downloads), so be sure to only use it for inspiration.

It's a work in progress, and improvements are more than welcome.

Downloadstation CLI

To have your Synology scan a directory for new download tasks, you can use Downloadstation CLI.

$ ipkg install python24 py-pgsql py24-mx-base
$ curl https://downloadstation.jroene.de/downloadstation -ko /opt/bin/downloadstation \
 && chmod a+x $_

With the command

$ downloadstation add $nzbfile

The download will be added to the queue. If you use an adaptation of my unpacker script, it will already automatically scan /volume1/downloads/_queue for any new torrent or nzb task.

Tools

These programs may take up a little bit of space, but won't be active in memory until you call upon them (except for cron), so feel free to install without performance loss:

$ ipkg install vim bash bash-completion less rsync mtr \
  sudo tshark htop openssl mlocate perl ack hdparm sysstat dstat \
  bzip2 unrar unzip zlib p7zip wget

$ curl https://raw.github.com/timkay/solo/master/solo -ko /usr/bin/solo \
 && chmod a+x $_

Optionally do ipkg install clamav so you can run clamscan on freshly downloaded files and check them for viruses (I decided not to).

Renaming Files

There's a neat program called tvnamer that will rename all your TV series files.

Install:

$ ipkg install python25 py25-setuptools git \
 && cd /volume1/@tmp \
 && git clone https://github.com/dbr/tvnamer.git \
 && cd tvnamer \
 && python setup.py install \
 && ln -s /opt/local/bin/tvnamer /usr/bin/tvnamer \

Use:

$ tvrenamer -r /volume1/video/tv

FileBot is even better but requires a GUI.

Crontab

Crontab works slightly different than on more high-level Operating Systems.

Here's how to edit your crontab:

$ $EDITOR /etc/crontab

Every job needs a user prefix. e.g. root:

*/15 * * * * root /usr/bin/solo -port=1111 /volume1/video/unpacker.sh 1>&2 > /volume1/@tmp/unpacker.log

When you're done editing the new crontab, reload it by executing:

$ /usr/syno/etc.defaults/rc.d/S??crond.sh stop
$ /usr/syno/etc.defaults/rc.d/S??crond.sh start

Tmux or Screen

If you start programs from within tmux, you can close your SSH session without killing it. You can check back later on it with tmux attach || tmux.

This makes it perfect to run cleanup/rename scripts in while you're still experimenting and need to check up on them regularly.

Tmux similar to screen, but I think it's a bit easier to deal with (just tmux attach || tmux is all).

However screen is a lot easier to install thanks to ipkg, so pick your poison.

Screen

$ ipkg install screen

Tmux

$ ipkg install libevent optware-devel ncurses-dev

# https://forum.synology.com/enu/viewtopic.php?f=90&t=30132
$ mkdir /opt/arm-none-linux-gnueabi/lib_disabled \
 && mv /opt/arm-none-linux-gnueabi/lib/libpthread* /opt/arm-none-linux-$ gnueabi/lib_disabled \
 && cp /lib/libpthread.so.0 /opt/arm-none-linux-gnueabi/lib/ \
 && cd /opt/arm-none-linux-gnueabi/lib/ \
 && ln -s libpthread.so.0 libpthread.so \
 && ln -s libpthread.so.0 libpthread-2.5.so

$ cd /volume1/@tmp \
 && wget https://sunet.dl.sourceforge.net/project/tmux/tmux/tmux-1.4/tmux-1.4.tar.gz \
 && tar -zxvf tmux-1.4.tar.gz \
 && cd tmux-1.4 \
 && export CC=gcc \
 && export CFLAGS="-L /opt/lib -I  /opt/include/ncurses" \
 && ./configure --prefix=/opt # prefix is not supported. So we'll need some symlinks \
 && make # This will take a while \
 && make install \
 && ln -s /opt/lib/libevent-1.4.so.2 /usr/lib/libevent-1.4.so.2 \
 && ln -s /opt/share/terminfo/* /usr/share/terminfo/