Checking database backups has always been one of the main concerns of DBAs. With Oracle is quite easy with a central RMAN catalog, but with other databases doing it with few effort can be a great challenge.
Some years ago I developed a little framework to control all SQLServer databases. This framework was based on Linux (strange but true!), bash, freetds, sqsh and flat configuration files. It’s still doing well its work, but not all SQLServer DBAs can deal with complex bash scripting, so a customer of mines asked me if I was able to rewrite it with a language Microsoft-like.
So I decided to go for a PowerShell script in conjunction with a couple of tables for the configuration and the data, and a simple TSQL script to provide HTML reporting. I have to say, I’m not an expert on PowerShell, but it’s far from being as flexible as other programming languages (damn, comparing to perl, python or php they have in common only the initial ‘P’). However I managed to do something usable.
The principle
This is quite simple: the PowerShell script looks up for the list of instance in a reference table, then it sequentially connect to and retrieves the data:
- recovery mode
- status
- creation time
- last full backup
- last log backup
This data is merged into a table on the central repository. Finally, a TSQL script do some reporting.
Custom classes in powershell
One of the big messes with PowerShell is the lack of the definition for custom classes, this is a special mess if we consider that PowerShell is higly object-oriented. To define your own classes to work with, you have to define them in another language (C# in this example):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
Add-Type @' using System; public class DatabaseBackup { public string instanceName; public string databaseName; public string recoveryMode; public string status; public DateTime creationTime; public DateTime lastFull; public DateTime lastLog; private TimeSpan diff; public double lastFullTotalHours () { diff = DateTime.Now - lastFull; return Math.Round(diff.TotalHours,2); } public double lastLogTotalHours () { diff = DateTime.Now - lastLog; return Math.Round(diff.TotalHours,2); } } '@ |
For better code reading, I’ve put this definition in a separate file (DatabaseBackup.ps1).
The query that retrieves the data…
Actually I use this query to get the information:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
$query_bck_database = "select [Database]=d.name,LastFull,LastTran,GetDate=getdate(), RecoveryMode=DATABASEPROPERTYEX(d.name, 'Recovery'), CreationTime=d.crdate, Status=DATABASEPROPERTYEX(d.name, 'Status') from master.dbo.sysdatabases d left outer join (select database_name, LastFull=max(backup_finish_date) from msdb.dbo.backupset where type = 'D' and backup_finish_date <= getdate() group by database_name ) b on d.name = b.database_name left outer join (select database_name, LastTran=max(backup_finish_date) from msdb.dbo.backupset where type ='L' and backup_finish_date <= getdate() group by database_name ) c on d.name = c.database_name where d.name <> 'Tempdb' order by [LastFull]"; |
I’ve also put this snippet in a separate file queries.ps1 to improve readability.
The tables
The first table (DB_Servers) can be as simple as a single column containing the instances to check. This can be any other kind of source like a corporate CMDB or similar.
The second table will contain the data collected. Off course it can be expanded!
The third table will contain some rules for managing exceptions. Such exceptions can be useful if you have situations like “all databases named northwind should not be checked”. I’ll show some examples in the next post.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
CREATE TABLE [dbo].[DB_Servers] ( [DB_Instance] [nvarchar](40) NOT NULL, CONSTRAINT [PK_DB_Servers] PRIMARY KEY (DB_Instance) ) GO CREATE TABLE [dbo].[DB_Status]( [InstanceName] [varchar](50) NOT NULL, [DatabaseName] [varchar](50) NOT NULL, [RecoveryMode] [varchar](12) NULL, [DatabaseStatus] [varchar](15) NULL, [CreationTime] [datetime] NULL, [LastFull] [datetime] NULL, [LastLog] [datetime] NULL, [LastUpdate] [datetime] NULL, PRIMARY KEY CLUSTERED ([InstanceName] ASC,[DatabaseName] ASC) ) GO CREATE TABLE [dbo].[DB_Backup_Exceptions]( [InstanceName] [varchar](50) NOT NULL, [DatabaseName] [varchar](50) NOT NULL, [LastFullHours] [int] NULL, [LastLogHours] [int] NULL, [Description] [varchar](250) NULL, [BestBefore] [datetime] NULL, PRIMARY KEY CLUSTERED ([InstanceName] ASC,[DatabaseName] ASC) ) GO |
The main code
Change this to whatever you want…
1 2 3 4 5 |
set-location "K:\TOOLS\CHECK_SQL\" ### environment to insert results/get lists $serverName = "SERVER01\MSSQL01" $databaseName = "Tools" |
This initializes the files explained earlier:
1 2 3 4 5 |
## initialise a file with some variables containing queries (to offload the script) . .\queries.ps1 ## initialise a class to better manage database backups as objects . .\DatabaseBackup.ps1 |
This adds the required snap-in to query sqlserver
1 2 3 4 5 6 |
if ( (Get-PSSnapin -Name sqlserverprovidersnapin100 -ErrorAction SilentlyContinue) -eq $null ) { Add-PsSnapin sqlserverprovidersnapin100 } if ( (Get-PSSnapin -Name sqlservercmdletsnapin100 -ErrorAction SilentlyContinue) -eq $null ) { Add-PsSnapin sqlservercmdletsnapin100 } |
The following function will, given the instance, do the following:
- Get the data in a ResultSet
- Instantiate an instance of the DatabaseBackup class (the one we defined in the external file) for each row
- Return an array of DatabaseBackup objects with all the data ready to be processed
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
function getDatabaseBackups ([String]$instance) { Write-Output " Instance: $instance" $databases = invoke-sqlcmd -Query $query_bck_database -Server $instance $i = 0 foreach ( $database in $databases ) { $dbbck = new-object DatabaseBackup $dbbck.instanceName = $instance $dbbck.databaseName = $database.Database $dbbck.recoveryMode = $database.RecoveryMode $dbbck.creationTime = $database.CreationTime $dbbck.status = $database.Status if ( -not ( $database.IsNull("LastFull") ) ) { $dbbck.lastFull = $database.LastFull } else { $dbbck.lastFull = "01.01.1900 00:00:00" } if ( -not ( $database.IsNull("LastTran") ) ) { $dbbck.lastLog = $database.LastTran } else { $dbbck.lastLog = "01.01.1900 00:00:00" } [DatabaseBackup[]]$databasebackups += $dbbck } return $databasebackups } |
This is the real “main” of the script, connecting to the central instance and getting the list of the instances to check:
1 2 3 4 5 6 7 8 |
$Connection = New-Object System.Data.SQLClient.SQLConnection $Connection.ConnectionString ="Server=$serverName;Database=$databaseName;trusted_connection=true;" $Connection.Open() $Command = New-Object System.Data.SQLClient.SQLCommand $Command.Connection = $Connection $instances = invoke-sqlcmd -Query "select [name]=db_instance from db_servers" -ServerInstance $serverName -Database $databasename |
Finally, for each instance we have to check, we trigger the function that collects the data and we insert the results in the central repository (I’m using a merge to update the existent records).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
foreach ( $instance in $instances ) { $databasebackups = getDatabaseBackups ($instance.name); $databasebackups[1..($databasebackups.length-1)] | foreach { $_ | select-object instanceName,databaseName $Command.CommandText = "MERGE DB_Status as target USING ( select '$($_.instanceName )','$($_.databaseName )','$($_.recoveryMode )','$($_.status )','$($_.creationTime)','$($_.lastFull)','$($_.lastLog)') as source (InstanceName, DatabaseName, RecoveryMode, DatabaseStatus, CreationTime, LastFull, LastLog) ON (source.InstanceName=target.InstanceName and source.DatabaseName=target.DatabaseName) WHEN MATCHED THEN UPDATE SET RecoveryMode = source.RecoveryMode, DatabaseStatus = source.DatabaseStatus, CreationTime = source.CreationTime, LastFull = source.LastFull, LastLog = source.LastLog, LastUpdate=getdate() WHEN NOT MATCHED THEN INSERT (InstanceName, DatabaseName, RecoveryMode, DatabaseStatus, CreationTime, LastFull, LastLog, LastUpdate) VALUES (source.InstanceName, source.DatabaseName,source.RecoveryMode,source.DatabaseStatus, source.CreationTime, source.LastFull,source.LastLog,getdate() ); " $Command.ExecuteNonQuery() | out-null } Remove-Variable databasebackups } $Connection.Close() |
How to use it
- Create the tables and insert your instances in the table db_servers.
- Put the three files (Collect_Backup_Data.ps1,queries.ps1 and DatabaseBackup.ps1) in a directory, modify the instance name and db name in Collect_Backup_Data.ps1
- Schedule the main script using the SQLAgent as a Operating system (CmdExec):
1 |
powershell.exe -file "K:\TOOLS\CHECK_SQL\Collect_Backup_Data.ps1" |
- You can’t use the internal powershell of SQLServer because it’s not full compatible with powershell 2.0.
- Check that the table db_status is getting populated
Limitations
- The script use Windows authentication, assuming you are working with a centralized domain user. If you want to use the SQL authentication (example if you are a multi-tenant managed services provider) you need to store your passwords somewhere…
- This script is intended to be used with single instances. It should works on clusters but I haven’t tested it.
- Check the backup chain up to the tape library. Relying on the information contained in the msdb is not a reliable monitoring solution!!
In my next post we’ll see how to generate HTML reports via email and manage exceptions.
Hope you’ll find it useful.
Again PLEASE, if you improve it, kindly send me back a copy or blog it and post the link in the comments!
Next: SQLServer centralized backup monitoring with PowerShell and TSQL (2/2)
Cheers
—
Ludo
Latest posts by Ludovico (see all)
- New views in Oracle Data Guard 23c - January 3, 2024
- New in Data Guard 21c and 23c: Automatic preparation of the primary - December 22, 2023
- Does FLASHBACK QUERY work across incarnations or after a Data Guard failover? - December 13, 2023
Hi Ludovic
Thank you very much for sharing your work. I find it simple to follow and am able to make modifications. These are the minor modifications I made to get it to work for me:
–>Delete importing of sql snapins (did not need as am running PSv4)
–>Had to include the full name of table as just table name itself gave the error “Invalid error” $instances = Invoke-Sqlcmd -Query “Select DBInstance from Tools.dbo.TbleDBServers;”
Keep up the great work.
Thanks for your comment 🙂
Thanks Ludovico. I have added differential backups to all the scripts. How do I share it with you?
Thanks,
Khan
Message
Executed as user: NT Service\SQLSERVERAGENT. Add-PSSnapin : No snap-ins have been registered for Windows PowerShell version 2. At D:\DBA_Database_Monitering\Collect_Backup_Data.ps1:17 char:17 + Add-PsSnapin <<<< sqlserverprovidersnapin100 + CategoryInfo : InvalidArgument: (sqlserverprovidersnapin100:Str ing) [Add-PSSnapin], PSArgumentException + FullyQualifiedErrorId : AddPSSnapInRead,Microsoft.PowerShell.Commands.Ad dPSSnapinCommand Add-PSSnapin : No snap-ins have been registered for Windows PowerShell version 2. At D:\DBA_Database_Monitering\Collect_Backup_Data.ps1:20 char:17 + Add-PsSnapin <<<< sqlservercmdletsnapin100 + CategoryInfo : InvalidArgument: (sqlservercmdletsnapin100:Strin g) [Add-PSSnapin], PSArgumentException + FullyQualifiedErrorId : AddPSSnapInRead,Microsoft.PowerShell.Commands.Ad dPSSnapinCommandThe term 'invoke-sqlcmd' is not recognized as the name of a cmdlet, function, s cript file, or operable program. Check the spelling of the name, or if a path w as included, verify that the path is correct and try again. At D:\DBA_Database_Monitering\Collect_Backup_Data.ps1:62 char:27 + $instances = invoke-sqlcmd <<<< -Query "select [name]=db_instance from db_se rvers" -ServerInstance $serverName -Database $databasename + CategoryInfo : ObjectNotFound: (invoke-sqlcmd:String) [], Comma ndNotFoundException + FullyQualifiedErrorId : CommandNotFoundException The term 'invoke-sqlcmd' is not recognized as the name of a cmdlet, function, s cript file, or operable program. Check the spelling of the name, or if a path w as included, verify that the path is correct and try again. At D:\DBA_Database_Monitering\Collect_Backup_Data.ps1:27 char:29 + $databases = invoke-sqlcmd <<<< -Query $query_bck_database -Server $insta nce + CategoryInfo : ObjectNotFound: (invoke-sqlcmd:String) [], Comma ndNotFoundException + FullyQualifiedErrorId : CommandNotFoundException Exception setting "creationTime": "Cannot convert null to type "System.DateTime "." At D:\DBA_Database_Monitering\Collect_Backup_Data.ps1:36 char:12 + $dbbck. <<<< creationTime = $database.CreationTime + CategoryInfo : InvalidOperation: (:) [], RuntimeException + FullyQualifiedErrorId : PropertyAssignmentException You cannot call a method on a null-valued expression. At D:\DBA_Database_Monitering\Collect_Backup_Data.ps1:38 char:33 + if ( -not ( $database.IsNull <<<< ("LastFull") ) ) { + CategoryInfo : InvalidOperation: (IsNull:String) [], RuntimeExc eption + FullyQualifiedErrorId : InvokeMethodOnNull You cannot call a method on a null-valued expression. At D:\DBA_Database_Monitering\Collect_Backup_Data.ps1:43 char:33 + if ( -not ( $database.IsNull <<<< ("LastTran") ) ) { + CategoryInfo : InvalidOperation: (IsNull:String) [], RuntimeExc eption + FullyQualifiedErrorId : InvokeMethodOnNull instanceName databaseName ———— ———— Exception calling "ExecuteNonQuery" with "0" argument(s): "The conversion of a varchar data type to a datetime data type resulted in an out-of-range value. The statement has been terminated." At D:\DBA_Database_Monitering\Collect_Backup_Data.ps1:85 char:29 + $Command.ExecuteNonQuery <<<< () | out-null + CategoryInfo : NotSpecified: (:) [], MethodInvocationException + FullyQualifiedErrorId : DotNetMethodException. Process Exit Code 0. The step succeeded.
Ludovico,
I am getting below error when trying run the job in SQL 2012 agent , can you please help , its not getting any data in DB_Status.
Another nice script. i am planning to add differential backups in to this script.
🙂 If you manage to do it, please share the code! It was in my plans but finally the customer I’m working for right now doesn’t plan to use incrementals (not yet!).
Ludovico,
Absolutely fantastic solution and very useful script. Thanks a bunch for sharing it.
I was using SSIS package for the Backup monitoring solution for my company.
After saw your blog, I have implemented it and it’s really quick in pooling the servers.
Also, your blog helped me as a staircase for powershell learning.
May I request you to blog about your other DBA solutions using powershell ?
Thank you Anandan!
I don’t have many solutions, this is the biggest one I’ve done on my own. If I’ll find the time I will post other snipplets…
—
Ludo
Pingback: SQLServer centralized backup monitoring with PowerShell and TSQL (2/2) | DBA survival BLOG