Disaster Recover & Remote Studio Sync
Information: This article was originally published as part of a longer Blog post on the Broadcast Radio Blog
Aim: To ensure all vital information is copied to a backup medium to allow you to recover your full system and get back ‘on air’ as soon as possible in the event that something really bad happens.
We will not be going into the deep technical details in this article as it can very quickly get very complex, and most solutions will be unique to your individual setup but this guide should help you to decide what method will work best for your station.
At a minimum, you should be backing up all your important data to a second location on the same site but ideally you should be backing up data to an off site location just in case the ‘disaster’ is something that can affect everything at your primary site (like a fire or computer virus).
Good practice would be to have a couple of large USB drives that are kept ‘off site’ and rotated so that you always have a couple of options.
Backing up to a fixed external site is also a great option if you can, but cloud backup can be expensive for the amount of data a typical Myriad 5 Playout system would generate. The other consideration with a fixed ‘off site’ backup (cloud or another site) is the recovery time. You can set your backup to only copy incremental changes but should you need to use it to recover, it might take some time to download all your data back from a remote or cloud based solution.
Things to consider:
- How much you need to backup.
- How regularly you need to backup.
- What mechanism or tools to use.
- How long it will take you to recover the data and get back on air!
How Much To Backup
This is at least an easy one to answer. If you want to create a complete backup of your system that you can use to recover after a full system loss you will basically need to backup everything.
So that includes:
- Your complete Myriad 5 data folder (including all data files, audio files etc) - The easiest thing is to just backup the entire Myriad folder.
- SQL Database - In Myriad 5, almost everything (other than the audio files themselves) is in the main SQL database, this includes all the settings as well as your MediaWall information, Clocks, Rules and any scheduled content in the Log. So even if you have a complete copy of your Myriad data folder, without the Database you will not be able to easily get back on air. Thankfully it is fairly easy to setup a script to generate an 'SQL Backup' file that can be used to 'restore' your data into a new SQL database.
Tip: Not sure what folder you should be backing up? In Myriad Playout, click on the Settings Menu > Applications Settings, look on the Data Locations tab and make a note of the folder listed in the Location of data files field. This folder contains everything Myriad 5 needs (including all your media files) except for the SQL Database.
To backup your SQL Database you will need to run the Microsoft SQL Server Management Studio (SSMS) application (if you have not installed it then you need to!). Once you have SSMS running, connect to your SQL server and expand the Databases branch in the left hand 'explorer'. Locate the Database you want to backup (again you can use the Application Settings in Myriad 5 to find out which one), right click on the database , choose Tasks > Backup and follow the instructions. You will find this process is well documented online.
How Regularly You Need To Backup
The temptation is to perform a backup very regularly so that your backup is as up to date as possible but the potential danger is that you stand a good chance of replicating issues to your backup before you know that they are issues.
Let's say you update your backup every hour. If the problem is that someone has deleted all of your Songs, it is possible that this would be replicated to your backup data before you realise they've done it. Performing a backup every few days or even once a week gives you time to find problems before they are replicated to your backup data, but of course means that a small amount of new content may not have been backed up yet.
What Mechanism To Use
Here is where it gets a lot more personal! There are many ways you can replicate the key data to your chosen backup device or location. There are simple tools (like Robocopy) that are built into Windows that can be effective for creating backup data within your network or you can use 3rd party applications like Dropbox or OneDrive to create replicas of your data at remote locations.
Some of the options include:
- Robocopy - A simple 'one way' tool that is capable of copying any changed data to different network location or drive. This is simple to setup and manage but only works within a network (or VPN) and is usually only used in a single direction (copies master data to a slave location).
- 3rd Party Folder Replication Tools (such as Dropbox and One Drive) - Probably the best known tool for replicating content from one site to another is Dropbox (although OneDrive works in a similar way and is built into Windows). It is possible to use DropBox to replicate your entire Myriad folder to a remote location and you could even run Myriad at the remote location 'looking' at the folder being updated.
- DFS Replication - The 'enterprise' solution for folder replication is to use Microsoft DFS Replication. This requires each site to have at least one Windows Server. The advantage of DFS replication is that it is truly two way making it possible to have studios in different locations that each run from local data that is synchronised in close to real time.
But the above three options will only get you half way because if you are not replicating your SQL database, you will not have all the data you need to reinstate your Myriad 5 Playout system.
To make sure you have the SQL Database to compliment your data, you also need to implement one of the following:
- Backup Your SQL Database To A File - This can be a scripted task that will take a 'snapshot' of the data in your SQL Database that can be used to 'restore' into a remote SQL database when required. The normal process is that you script a task that creates a 'backup file' and then use whichever file replication (outlined above) you prefer to ensure that the backup file is copied to your backup location along with all the other data. If you need to utilise your backup data, you will first need to 'restore' the backup file into the SQL database you want to use. Unfortunately this cannot be done while Myriad is running (as it is access the live database) so this solution is fine for creating a complete data set that you can use if you need to but is not suitable for synchronising a remote Myriad playout system (that is actively running).
- SQL Publisher / Subscriber - As with the DFS solution above, this is the 'enterprise' level solution that allows near to 'real-time' replication between SQL Databases at remote sites. To use this method you will need SQL Standard at your Publisher site (SQL Express is ok at the Subscriber sites). The remote sites can be set to be Subscribers of the Publisher site with changes made at any site be replicated in as close to real-time as the network connection will allow. When combined with DFS replication, this provides a virtually real-time synchronisation between sites. When we setup the GMG network in the UK, we used this method to synchronise broadcast centres in Manchester, London, Cardiff, Yorkshire and Glasgow such that any studio in any site could be used to deliver local or network content so that the group could use the site most convenient for the 'on air talent'. Please note that such a setup requires a strong infrastructure and IT knowledge and is not covered by our standard technical support, please contact us for more details on options.
For backup and disaster recovery, the first method should be the one that most stations consider!
How Long Will It Take To Get Back On Air
We often get asked about cloud backup and on the face of it, backing up all your data to the cloud seems like a great plan as it virtually guarantees that you data will be safe as most large provides (AWS, Azure etc) replicate data across many servers and offer a high level of availability.
The main issue is that because Myriad 5 Playout is a complex, real-time playout system, it will normally require data to be accessible on your local network in order to provide an acceptable level of reliability and performance. So if the worst happens and you are relying on backup data to get you back on air, the last thing you want is to spend 24 hours downloading 2TB of data from your cloud service provider!
That said, a fully automated backup in the cloud will provide a safe and secure solution that is virtually guaranteed to be safe.
By contrast, a low cost USB drive offers a great backup solution that has the advantage that you can run Myriad directly from it in an emergency but you need to ensure that is kept up to date and if it is physically in your building, it may be of little use if the reason you need it is because your build has burnt down!
The Recommended Solution For Disaster Recovery
Taking into consideration all of the points above, we would recommend a two tier approach to ensuring your data is adequately backed up for disaster recover.
- Use a USB External hard drive to create a local backup. You can use robocopy (or similar) to copy the whole Myriad File Structure onto this drive as well as a scheduled task to export an SQL Database backup using script. Ideally you would have two USB drives in rotation and change them weekly so that one is always stored 'off site'. That way you always have a fairly up to date copy of everything one site but you also have a complete backup 'off site' that is only a week out of date at the most.
- Setup an 'off site' backup that could be at a different site or cloud based. You can use a file replication technology such as Dropbox to replicate the Myriad file structure. Remember that you also need to script regular SQL Database backups and ensure these are also replicated to the 'off site' backup along with everything else.
We would recommend every station implement at least step one to ensure that they are covered should the need arise.