Development

Using LINQ to Objects for recursion

David Jade — Fri, 23 May 2008 20:06:22 GMT

One of the best new features of .NET 3.5 is LINQ. It's great for accessing data from databases like SQL Server but it's far more useful than just that. LINQ to objects is the killer feature in .NET 3.5 and it can change the way you write code for the better.

I am finding more cases where I can use LINQ for general data manipulation to simplify the code that I write. Coupled with Extension Methods it's a really powerful new way to write algorithms over your data types.

One of my favorite new tricks is to use LINQ to make recursion over a tree of objects simpler. There are many cases where this type of pattern is possible. For instance when working with controls in ASP.NET or Windows Forms each Control has a property named Controls, which is a collection of its child controls. If you need to perform some operation over all of them you can gather them all in just one LINQ statement similar to this:

ChildControls = from nested in 
                myform.Controls.Cast<System.Web.UI.Control>().Descendants(
                    c => c.Controls.Cast<System.Web.UI.Control>())
                select nested;

The above code is a lot simpler to express than the recursion code that would have been necessary to work with each of the nested controls. And if you need to filter the child controls returned you can just add a LINQ Where clause (or any of the other LINQ operators).

Note: The Cast<T> function above is necessary since the Controls property return a non-generic IEnumerable. Cast<T> simple wraps the non-generic IEnumerable and returns a typed IEnumerable<T>.

The trick that makes this work is a function called Descendants, which is an extension method that I wrote. You can use it with any IEnumerable<T> collection to descend into those collections. It will descend into those collections by way of a Func<T, IEnumerable<T>> lambda that you supply. In the above example I pass it a lambda function that tells it how to retrieve the nested child controls.

Here is that extension method:

static public class LinqExtensions
{
    static public IEnumerable<T> Descendants<T>(this IEnumerable<T> source, 
                                                Func<T, IEnumerable<T>> DescendBy)
    {
        foreach (T value in source)
        {
            yield return value;

            foreach (T child in DescendBy(value).Descendants<T>(DescendBy))
            {
                yield return child;
            }
        }
    }
}

You could just as easily write a lambda function to descend into a tree structure or any other data structure as long as it supports implemented IEnumerable<T>. Since you can supply the function you need to descend into the data structures, it makes Descendants a generic way to traverse into nested data structures.

del.icio.us Tags: .NET, LINQ, C#

Vista's 'virtualized' folders

David Jade — Mon, 31 Mar 2008 22:05:03 GMT

With Windows Vista Microsoft has changed where user account profile and program data gets stored. Instead of storing all profile data under "c:\Documents and Settings\" as they did with Windows XP, things are now split between "c:\Users\" and "c:\ProgramData\" (on a standard Vista installation). In order for older programs that hard-coded these directories names (instead of using the proper API to find them) will continue work, Microsoft has used features of the NTFS file system, specially junction points and symbolic links, to create virtual folders that look like the older locations but in reality point to the new locations. This virtualization scheme is pretty complex and for the most part hidden from regular standard users, but as a programmer and system administrator it is sometimes handy to know how these folders have been constructed.

I have created this diagram that maps out these junction points and symbolic links to their physical directories. I have also include information on the virtualized folders that get used when programs attempt to store data in 'c:\Program Files\' or 'c:\Windows\'.

Click to download this diagram as a PDF file

del.icio.us Tags: Windows,Vista,FileSystem,Virtualized,UserSettings

An intelligent backup system for Windows, part 3

David Jade — Tue, 21 Nov 2006 19:34:08 GMT

This is the third in a series of articles about a new backup process I have implemented for my home network. In the previous article I covered a mirror backup process that maintains a storage-efficient backup history. In this article I'll cover the tools I used and the issues I had to overcome while using them.

Common tools and a not so common use of them

Once I had decided to create a backup system that creates space-conserving mirror backups by leveraging NTFS hard links, I set out to make a simple prototype. It occurred to me that I already had a very good tool for copying data around, a free tool called robocopy from the Windows sources kit. Robocopy is a very powerful file copying tool that can be configured in a multitude of ways, including the ability to copy files in backup mode, a special mode of file access that can be used to bypass file security for the purposes of backing up files. It is also faster and more reliable than the file copy tools that come with Windows and has a very good set of options to control which files to copy. However robocopy know nothing about the process of creating hard links to previous versions of files. This step I would have to do myself.

In searching for information on how to create hard links, it wasn't long before I ran across references to the fsutil tool that is included in Windows XP and Windows Server 2003. Using this tool you can creating NTFS hard links from the command line.

Together with robocopy and a bit of creative CMD scripting, I was able to throw together a prototype that could create mirror backups while hard linking to the files that had not changed since the previous backup just like rsync did. I started by duplicating the directory structure of the old backup by using robocopy to copy just the directories. Next I used fsutil to hard linked copies of the previous backup files into the new directories. I did this by traversing the old backup directories and using fsutil to create hard links to each of the older files. Then I used robocopy to generate a list of the files that had changed since the last backup, including files that were no longer present. From that listing I then deleted those files from the newly created mirror backup. Finally, I used robocopy to copy over just the newer files into the new mirror backup. While it wasn't the most efficient method, it worked pretty well but it had one important limitation: fsutil only works on local disks. It was also a pretty hacky bit of CMD script since I had to do string manipulation to create the hard links. I had considered re-writing the whole process in C# but then something else popped on my radar.

PowerShell, isn't that some sort of new gasoline?

It was about this time that Microsoft released RC2 of PowerShell (which as just recently gone RTM). PowerShell is Microsoft's new administrative scripting language for the future. Besides be a very good replacement for command shell scripting and VBScript, it is also the new foundation of the management tools for the next version of Microsoft Exchange. It is an amazingly powerful scripting language, easily learned, easily extended, and is easily the more important tool I have learned in a long time.

PowerShell is different from other scripting languages because it is based on the concept of pipelining objects. Many scripting languages, including the native Windows shell, support pipeling text data from command to command. PowerShell is different however in that it pipelines complete .NET objects instead of just textual data. As full .NET objects, each object in the pipeline has state, properties, and methods. They can be passed as parameters to functions, extended dynamically, coerced into other types, and placed back into the pipeline. Functions in PowerShell can also be treated as objects allowing you to do some types of functional programming tasks that are not easily done in other .NET languages. It is a very powerful idea and my brief description doesn't even scratch the surface of the power that lies within PowerShell. It is all still very new to me but already I am finding many uses for it.

Tip: Here's a PowerShell gotcha to keep in mind. Every expression in PowerShell that produces output places that output in the pipeline. This can lead to pretty weird debugging issue if you aren't careful. I had more than one case where a function was returning more than I wanted because I was calling a command that placed things in the pipeline without realizing it. There are two ways to avoid this however. One is to assign the output of commands to a variable and the other is to redirect the output to $null (i.e. do-something > $null).

PowerShell's object pipeline nature along with the rich set of built-in commands knows as cmdlets, makes for a perfect system for doing administrative computer tasks. There are cmdlets for accessing PowerShell providers such as the file system and the registry, accessing WMI object, COM objects, and the full 2.0 .NET framework. I've seen examples of everything from a simple file parsing scripts to a simple but complete HTTP server written in PowerShell in just a few lines of code. To me it appeared to be the perfect language for scripting a new backup process. However PowerShell does not offer support for creating NTFS hard links either. For this I would need to extend PowerShell.

Extending PowerShell through custom C# objects and P/Invoke

Starting with Windows XP there is a new API for creating hard links, CreateHardlink. In previous versions of Windows, creating hard links was somewhat of a black art. You had to use the complex and sparsely documented Win32 Backup API's. It could be done and there are examples of how to do it out there, but it was not for the faint of heart. The CreateHardlink API however solves that, making it almost trivial to create hard links on NTFS. Furthermore, unlike fsutil the CreateHardlink API fully supports creating hard links on remote network NTFS drives. PowerShell cannot easily call native API's on its own though. To do that, you need to extend PowerShell with a bit on .NET code.

PowerShell is very easy to extend. You can write complete cmdlets', objects that fully plug into the PowerShell pipeline framework or you can just create simple .NET objects that can be created and invoked thanks to PowerShell's ability to access the .NET framework.

Using C# and a bit of P/Invoke it was almost trivial to solve the problem of not being able to create hard links in PowerShell (and .NET) by writing a simple object that called the Win32 CreateHardlink API. Once that was done, I could easily create my new .NET object in PowerShell and use it to create all the hard links that I wanted. Now I could create a more complete backup script from the ground up using PowerShell.

If you'd like to access the CreateHardlink API in PowerShell or .NET, here is a C# code snippet to help you. Simply create a new class in a .DLL and add this method. I added this method as a static member since it does not require any state from the class. This also makes it very easy to call from PowerShell.

[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Auto)]
internal static extern int CreateHardLink(string lpFileName, 
    string lpExistingFileName, IntPtr lpSecurityAttributes);

static public void CreateHardlink(string strTarget, string strSource)
{
    if(CreateHardLink(strTarget, strSource, IntPtr.Zero) == 0)
    {
        throw new System.ComponentModel.Win32Exception(Marshal.GetLastWin32Error());
    }
}

To call this code from PowerShell, you simply load the .NET assembly and then call that static method on your class. Note that this will throw an exception if it fails so make sure you have a PowerShell trap handler somewhere in your script.

# load the custom .NET assembly
[System.Reflection.Assembly]::LoadFrom('YourLibrary.dll')

# create a hard link
[YourLibraryName.YourClass]::CreateHardlink($Target, $Source) > $null

Whoops, that file is in use

There was still one more issue to tackle before I could write a robust backup system, accessing files that are in use. Starting with Windows XP Microsoft introduced a new system for accessing files that are currently in use on Windows systems, the Volume Shadow Copy Service (VSS for short, but not to be confused with Microsoft's VSS source control system).

One of the ideas behind VSS is that when requested, the OS will make a read-only copy of the drive, a snapshot frozen in time, available to a backup program. Other programs can continue to change the original disk files but this shadow copy, or snapshot will remain frozen and completely accessible to the program that created it. Furthermore when a backup program requests that a shadow copy is to be created, the OS can coordinate with shadow copy providers to ensure that the data on the disk is in a consistent state before the shadow copy is created. This further ensures that the files that the backup program has access are in a consistent enough state on the disk to be backed up. This is especially useful for files that are either always open or always changing like the system registry, user profiles, Exchange, or SQL databases. Once the backup program is finished with this temporary read-only shadow copy, it then releases it and it disappears from the system. By using the VSS system backup programs can gain access to every file on the drive even if they are exclusively in use by other programs. For me it was essential to use VSS with any backup process I implemented.

There were a few tough problems though. On windows XP these VSS snapshots are very temporary in that they only exist for as long as you hold a reference to them via COM. Once released, they auto-delete themselves. And unlike VSS on Windows Server 2003 they cannot be exposed as a drive letter for easy access. You have to access them via the native NT kernel's method of addressing NT namespace objects, the GLOBALROOT namespace. On XP when you ask the VSS service to create a snapshot, what you get is a NT GLOBALROOT path that looks like this:
\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1. Unfortunately this is something that not even the native Windows command shell fully understands and if you try and access it from PowerShell or .NET you'll get an exception telling you that you really shouldn't be accessing internal NT paths in .NET. To solve this I would need another bit of custom code to extend PowerShell.

VSHADOW.EXE and exposing a snapshot as a drive letter

VSHADOW is a sample tool that is part of the VSS SDK. It is a command line interface to the VSS API. By using this tool you can create and release VSS snapshots at will. It even has a way around the COM auto-destruction of snapshots on Windows XP by allowing you to call an external program once the snapshot has been created so that you can access the snapshot while VSHADOW is still keeping it alive. It will even create a set of environment variables for you to let you know the names of the GLOBALROOT shadow copies that it has created. This still didn't solve my problem of not being able to access them via PowerShell though (or robocopy for that matter) but having this source code was a good start.

All physical devices in Windows like hard drives exist in the GLOBALROOT namespace. It is only through device name mapping that we can access them via their friendly DOS names like C:, D:, etc.... Normally the OS creates these device mapping automatically at start up or whenever a new device has been connected. VSS snapshots however don't automatically get recognized and mapped. Mapping a friendly name to a VSS snapshot has to be done by directly using the win32 DefineDeviceCreate API. By using this API you can create and remove DOS device mappings to VSS snapshots on the fly even on Windows XP. But since VSS snapshots are temporary you have to manage them carefully or the system could become unstable.

Creating a VSS snapshot and mapping it to DOS device names is well beyond what I wanted to try to do in C#. Luckily for me, the VSHADOW C++ source code was written in a very reusable manor and I could easily reuse it by wrapping a COM object around it.

The not so nice COM interop experience with .NET 2.0

Creating a snapshot is not the simplest of procedures. You have to query for the list of VSS writers, map them against the target volume, determine which ones to include in the process, and finally request that the snapshot be created. You have to hold on to the VSS COM interface to keep the snapshot alive on XP for the duration of its use. When you are done, you have to release it in a controlled manor or the VSS system can completely degrade and require a system restart to recover from in most cases. It is also not the fastest process either, something that would come back to bite me later. However the VSHADOW source which is written in C++, was written in such a way that it made it very easy to turn into a COM object using ATL. It was as simple as creating a new ATL COM object project in Visual Studio and including the core VSHADOW sources file into the project. Once I had it building as a COM object it didn't take me long to put a .NET friendly interface on this new COM object that exposed methods to create and destroy VSS snapshots as well as map them to DOS device names.

PowerShell has native support for create and calling COM objects that is even easier than in other .NET languages. There is no need to create .NET interop classes, you just dynamically creating the COM object and use it much like you would in VBScript. Once I created my new VSS COM object it was trivial to create VSS snapshots on the fly and map them to DOS device names using PowerShell. With my new VSS COM object I now had complete access to VSS snapshots from any tool that could access a standard drive. It has some limitations but for this backup process it works very well.

Releasing the VSS snapshot in PowerShell however was another story. There is no clean way that I can find to force a created COM object to be released in PowerShell. You have to wait for the .NET garbage collector to do its thing which is usually not until the PowerShell process is exiting. My new COM object had its clean-up code in the COM object's Release method so that when it was released it would clean up the VSS state in the proper way, ensuring that the system remained stable. Unfortunately for me relying on a COM object's Release method to work during the .NET shutdown process proved to be one huge headache.

After many, many hours of debugging and not really believing what I was seeing I finally had to accept what was going on. From what I was seeing and from the research I have done it is my understanding that Finalizers in .NET, which are called when an object is being destroyed and which are also responsible for calling a COM object's Release method in PowerShell, are not guaranteed to complete when a process shuts down. Usually this is not a problem as the process is going away anyway. It is a problem however when you have native resources to release.

What I was seeing and not believing for literally hours and hours was that in the middle of my COM object's Release method the PowerShell process would just exit normally. No exceptions, no faults, nothing - just poof it's gone. And every time that it did this it would leave the VSS system in such a state that the machine had to be restarted because I was never given the chance to properly execute the VSS clean-up code which can be a lengthy process. It seems that the PowerShell shutdown process was timing out my clean-up code. It was a complete mess and still one that I cannot believe is acceptable but apparently to the folks who created .NET it is (you can read about it here in way more detail than anyone should have to know. Just search for "timeout" and "watchdog" on that page). The thought that external native code can have the plugged pulled just blows me away.

The fix was rather simple once I realized that I cannot count on my COM object's Release method to always complete. I had to move all critical clean-up code and put it in a public method that my PowerShell script would always call. Luckily PowerShell has pretty decent error handling and it wasn't too hard to ensure that I always called the clean-up method on my COM object before PowerShell normally terminates. I'm still not thrilled about this though. I would have preferred that my COM object be allowed to clean up after itself as necessary.

The moral of this story is that you are responsible for all complex clean-up even when calling native code. Don't depend on the .NET framework to always play nice.

Now that I had this behind me I had all the pieces that I needed: a robust file copy tool, a powerful scripting language, the ability to create hard links, and full access to Volume Shadow Copy snapshots.

In part four I'll cover the process overview and implementation details of creating the intelligent mirror backup process that I choose to be the foundation of my new backup strategy.

del.icio.us tags: Windows, PowerShell, backup, VSS, rsync

An intelligent backup system for Windows, part 2

David Jade — Wed, 15 Nov 2006 04:59:30 GMT

This is the second in a series of articles about a new backup process I have implemented for my home network. In the first article I covered some background information and why I choose a non-traditional backup process for my network. In this article I'll give an overview of mirror backups, their problems, and the ways in which they can be improved. I'll also talk about rsync and why for me, it was not the right tool for creating backups on Windows.

A more intelligent mirror backups process

Mirror backups have several advantages over the traditional backup process. Like a full backup they are a complete snapshot in time of the data being backed up but since they are typically stored on randomly accessible media (i.e. a disk or online), accessing any part of the backup set is easily accomplished. However, also like a full backup you usually have to copy each file to create the mirror even if that file has not changed since the last backup. This makes storing backup history both a time consuming process and an inefficient use of storage space. You can of course just copy the changed files into an existing mirror backup but then you lose any file change history from the last backup set.

While researching backup solutions I came across a UNIX utility called rsync. Rsync is a tool that was designed to efficiently create file set mirrors. It has a large and flexible set of features including its own network protocol that can efficiently transfer just the differences between versions of a file when creating a new mirror. The most appealing feature to me however is its ability to conserve storage space while still creating complete mirror backups. It does this by leveraging a feature called hard links which are found on modern files systems (including the Windows NTFS file system).

Hard links are a method by which the operating system separates the storage of the file's actual data from the file's directory entry. In other words, a file in a directory is really just a named pointer to the file's physical data which is stored somewhere else on the hard disk. Every file stored on a hard disk is stored in this way. Most of the time, there is a one-to-one mapping with each physical file's data having only one named directory entry or file name. However, some file systems allow you to create additional named references to the file's physical data. These reference can exist in the same directory or in any other directory on the file system. Furthermore, they can have the same or a completely different file name. When this is done they share all aspects of the actual data, including the file's attributes such as security settings, creation time, and the last modified time. If you change the file's data or attributes via one reference then all the other references will reflect those changes immediately. You can even delete the original reference to the data and the others will continue to live on. It is only after all references that point to the physical data have been deleted that the actual file data is deleted.

What makes rsync really special is that it can leverage this very powerful file system feature when creating mirror backups. It does this be examining the files being backed up, comparing each file against the copy in the previous mirror backup. Then for each file that it finds to be identical, instead of copying that unchanged file into the new mirror backup it simply creates a new hard link to the file as it exists in the previous mirror backup. This is much, much faster than re-copying the file itself but it essentially does the same thing. It makes the unchanged version of the file available in the newly created mirror backup. Furthermore it is also very space efficient since the file's data is still only stored once physically but it is accessible from every mirror backup that it is hard linked into.

By leveraging hard links, it also allows for another interesting feature. You can physically delete older mirror backups without affecting any of the new mirror backups that may be referencing unchanged data with these backups. Files that have other references will stay alive in those newer mirrors while older files that only have a reference in the mirror being deleted will be destroyed. This makes managing the amount of backup history to keep very easy; you can just delete older mirrors when you no longer want the history they contain.

Together these two features effectively allow you to break the traditional full/incremental backup cycle forever while keeping the best parts of each. You can have several full mirror backups available to preserve file history with no physical duplication of data for files that remain unchanged while still running a process that only copies newer files like an incremental backup. It is the best of both full backups and incremental backups combined in one logical backup process. You can read more about using rsync to create hard linking mirror backups here.

If rsync is for UNIX, what's this got to do with backing up Windows?

Rsync is a very UNIX-y tool and I am pretty firmly in the Windows camp. However you can still run rsync on Windows using the cygwin system. Cygwin is a software translation layer that was developed to allow certain types of UNIX programs to work on Windows systems. It has been around for a long time and is very functional and reliable. I used cygwin quite successfully to play with rsync on both Windows XP and Windows Server 2003. However for backing up data from Windows' NTFS file system there are some issues.

Windows' NTFS file system supports a wide array for features including security attributes, NTFS alternative data streams, sparse files, encrypted file, and much more. Cygwin and thus rsync know nothing about these additional file attributes since the translation layer was designed to make Windows seem like just another UNIX system. What this means is, that rsync can only copy the aspects of the files that it knows about, i.e. the file's regular data. So if you use rsync to make mirror backups on Windows you will essentially lose all of this other and sometimes important file and directory information (you also lose this information when simply copying files to a CD/DVD as well). This information however is usually preserved when doing traditional backups or when copying files between NTFS locations including across Windows networks. What was needed was a way for rsync to preserve this information on Windows.

I debated for quite some time on whether or not to dive in and start hacking away at the rsync source code to try to teach it more about the Windows NTFS file system and this additional file data. I even went so far as to patch cygwin to teach it about Windows' GLOBALROOT device paths, a feature that was essential in order to use Windows' Volume Shadow Copy Service (VSS) with rsync (this patch is now part of the cygwin CVS source, Btw) . However in the end I decided it would probably be much more effort to update rsync than it would be to write my own backup process. Rsync has a lot of features that I do not need and even though my new backup process was likely to be missing some of the features that I really liked about rsync, my new process would still contain the core method by which rsync creates space-conserving mirror backups.

This decision to start fresh on my own backup tool coincided with the time that I had started to play with another tool, Microsoft's recently released PowerShell scripting language. It wasn't long before I realized the PowerShell was a perfect fit for this type of problem.

In part three I'll cover the tools I used to implemented an intelligent mirror backup process in Windows using robocopy, Microsoft's PowerShell scripting language, C#, the Volume Shadow Copy Service, C++ and a little bit of COM interop.

del.icio.us tags: Windows, PowerShell, backup, VSS, rsync

PowerTips for PowerShell

David Jade — Thu, 05 Oct 2006 20:23:08 GMT

I'm certainly not a PowerShell expert but I have been finding my way around it lately. Once I got a few interesting scripts put together and saved to a file the next thing I ran up against was how do I launch these without loading them into the interactive environment by hand? By default when you install PowerShell it associates script files (.ps1 files) with notepad. Great for editing, not so great if you want to execute them. My guess is that after the 'Monad/PowerShell virus' story a year back or so that Microsoft got a little too freaked out to just let these things launch when clicked on. Unfortunately this decision also makes it more difficult to schedule a PowerShell script in the task scheduler too. To top it all off it's not quite as simple as just passing your script as a command line argument to PowerShell either. There are two separate steps necessary to enable you to launch PowerShell scripts from the Windows shell.

First you have to set up a file association in the Windows shell to change the default behavior for .ps1 files. However the PowerShell command line doesn't know what to do with a script file. By default it simple takes statements to execute. We can leverage this to our advantage though and construct a statement to cause our script file to be loaded and executed. That statement looks something like this:

powershell -command "& 'MyScript.psa1' "

If you update the file association with this you can then launch script files from the command line or by clicking on them. You can use the Tools | Folder Options dialog to do this but why not use PowerShell instead? Here are two simple lines of code to update your system registry to tell it how to execute .ps1 script files.

Note: This of course updates your system registry so you should back things up first.

new-item Registry::hkey_classes_root\microsoft.powershellscript.1\shell

new-item Registry::hkey_classes_root\microsoft.powershellscript.1\shell\open

new-item Registry::hkey_classes_root\microsoft.powershellscript.1\shell\open
\command -value ('"' + $PSHOME + '\powershell.exe" -command "& ''%1''"')

If you run these two PowerShell lines and then try and click on a .ps1 script file you'll see that we're not quite there yet. PowerShell has an execution policy that by default is set to "Restricted." In restricted mode no scripts are allowed to execute, only interactive commands may be executed. By using the PowerShell "set-executionpolicy" cmdlet you can change it to something more sensible like RemoteSigned which allows all locally generated scripts to execute but will only allow downloaded scripts to execute if they have been signed by a trusted source. In PowerShell you can execute the "help about_signing" command for more info on this.

Once these two steps are done you can now launch PowerShell scripts directly without having to start up the interactive environment first. In addition you can also use this method to schedule PowerShell scripts in the task scheduler.

del.icio.us tags: Windows, PowerShell

How to easily script any Windows application

David Jade — Sat, 30 Sep 2006 04:35:50 GMT

Many larger applications having scripting capabilities built in to allow you to automate repetitive tasks but what do you do if you need to script a repetitive task for an application that doesn't have a scripting or macro language built in? You could use VBScript and the SendKeys function. This works for simple things but it isn't always robust enough. What if you need to do something more complex, for instance: launch an application, wait for the main window to open, resize the window, send some keystrokes, and then click on a specific UI element? Unfortunately VBScript and its set of built-in functions don't always cut it when you need to move beyond SendKeys.

When your needs go beyond what VBScript can do natively you might want to take a look at AutoIt. AutoIt is a small suite of tools, the main component being a stand-alone scripting language. What sets AutoIt apart from other scripting languages is that it is targeted at GUI application scripting and its built-in set of functions enable the finer level of control lacking in VBScript and other scripting languages. The AutoIt scripting language is very VB-like and easy to pick up if you're already familiar with VB. AutoIt has a rich feature set including (taken from the AutoIt web site):

Provide a general-purpose scripting language for all Windows versions
Simulate keystrokes (supports most keyboard layouts)
Simulate mouse movements and clicks
Move, resize and manipulate windows
Interact directly with "controls" on a window (set/get text from edit controls, check boxes and radio buttons, select items in drop-down lists, etc.)
Create complex user interfaces (GUI's)
Work with the clipboard to cut/paste text items
Provide a scriptable RunAs function for Windows 2000/XP/2003

However what is really interesting to me is that AutoIt also includes a scriptable COM object essentially making its core application scripting capabilities available to any other COM capable languages including VBA, VBScript, PowerShell, or .Net. The COM interface on this object seems well thought out and is very easy to use. I started using AutoIt for this feature as I needed to automate an application from within Outlook's VBA scripting. Using the AutoIt COM object I was able to write Outlook VBA code to control another application including starting the application, waiting for its main window to launch, and intelligently sending data to the right parts of its UI.

And best of all, AutoIt is free. It can be found here.

Note: PowerShell is on the horizon and by all accounts it should be a very powerful tool but it also seems to lack some of this finer level of control for scripting GUI applications. You can however access the AutoIt COM object as well as the full .Net library.

del.icio.us tags: windows, PowerShell, AutoIt, automation, scripting

How to hyperlink to Microsoft Outlook messages

David Jade — Sun, 03 Sep 2006 01:11:22 GMT

First, a little context on why I wanted to do this in the first place. If you just want to get the source code then just scroll down to the "The source code".

A fair number of the things that make it on to my ToDo list are messages that I receive in Outlook. I've been using Outlook's flags to mark these items for follow up so that I could keep track of them. I use a different colored flag that denotes its own meaning as defined by me. This way I can flag an item using a action type (ToDo, Deferred, Waiting For, etc...). This has worked fairly well for me but once I had a large number of items flagged in Outlook (did I mention that I am a procrastinator?) it didn't really help me keep track of the things I needed to keep track of in any meaningful way. All I could even see was the complete list of flagged items but I couldn't break it down further into projects or context.

To overcome this limitation I've recently started managing my ToDo list outside of Outlook in a program called MyLifeOrganized (aka MLO) . MLO allows you to drag/drop Outlook items into MLO's task list. When you do this it creates a new MLO task using the subject of the dropped Outlook item for the task name. However MLO does something else really smart when you drag/drop an Outlook item. It not only put the text of the Outlook item into the notes, it will also create a hyperlink that will open up the original Outlook item when clicked. It does this by using Outlook's URL syntax which looks something like this:

Outlook:<entry_id> where <entry_id> is an Outlook Entry ID

Windows naturally understands this form of URLs. If you click on one it will cause Outlook to open the item referenced. This has allowed me to take Outlook items and create MLO items simply by dragging them to the MLO task list. In most cases this feature in MLO does exactly what I want, take an Outlook message that I need to follow up on and place it into my ToDo list. However sometimes this isn't exactly what I want. Sometimes I just want to place a link to Outlook items in the notes of an existing MLO item. Unfortunately MLO doesn't support this but there is a way to do it if you willing to do a little macro work in Outlook.

Update 6/4/2007: If you are using Office 2007 then you will probably need to enable the Outlook URL protocol handler so that hyperlinks to mail messages work.To do this requires editing the registry. You simple need to create these registry keys (substituting your installation paths of course):

HKEY_CLASSES_ROOT

outlook
(Default) = URL:Outlook Folders
URL Protocol= ""

DefaultIcon
(Default) = "C:\PROGRA~1\MICROS~2\OFFICE12\OUTLLIB.DLL,-9403"

shell

open

command
(Default) = "C:\PROGRA~1\MICROS~2\OFFICE12\OUTLOOK.EXE" /select "%1"

The source code

Below is the source for a two VB macros that can be added to Outlook. These macros will loop over all of the currently selected messages, getting the subject and Outlook EntryID for each message. With these two pieces of information they then build a string of text with the message's subject and its Outlook URL, each on their own line. It extends this text string for each selected message and places the resulting text string on the clipboard. The end result is one block of text that contains the message subject followed by the Outlook URL for each selected message. This text can then be pasted into any document that understands hyperlinks. This includes the notes of MLO items as well as all of the other Microsoft Office applications. This macro will work with multiple items selected in the main Outlook window as well as from the opened window of a single Outlook message. To use it, simple invoke the CopyItemIDs() macro. You can bind this macro to a menu or toolbar button for easier access within Outlook.

Note: I should mention that if you are using Microsoft Exchange server, the message Entry ID can change on you and break any existing Outlook URLs. This unfortunately always happens if you move a message to another folder so if you plan on using this, only invoke this macro after you have moved the message to a new folder.

Update: There is just one more thing you must do before you run this script. You need to include a reference to FM20.dll, which is the Forms 2.0 library. This will allow you to use the DataObject to manipulate the clipboard. Thanks to 'Some Guy' who pointed this omission out.

Sub CopyItemIDs()
    Dim myOLApp As Application
    Dim myNameSpace As NameSpace
    Dim currentMessage As MailItem
    Dim ClipBoard As String
    Dim DataO As DataObject
    
    ' Housekeeping: set up the macro environment
    Set myOLApp = CreateObject("Outlook.Application")
    Set myNameSpace = myOLApp.GetNamespace("MAPI")
    
    ' Figure out if the active window is a list of messages or one message
    ' in its own window
    On Error GoTo QuitIfError    ' But if there's a problem, skip it
    Select Case myOLApp.ActiveWindow.Class
        ' The active window is a list of messages (folder); this means there
        ' might be several selected messages
        Case olExplorer
            ' build the clipboard string
            For Each currentMessage In myOLApp.ActiveExplorer.Selection
                ClipBoard = GetMsgDetails(currentMessage, ClipBoard)
            Next
             
        ' The active window is a message window, meaning there will only
        ' be one selected message (the one in this window)
        Case olInspector
            ' build the clipboard string
            ClipBoard = GetMsgDetails(myOLApp.ActiveInspector.CurrentItem, _
                                         ClipBoard)
        ' can't handle any other kind of window; anything else will be ignored
    End Select
    
QuitIfError:       ' Come here if there was some kind of problem
    Set myOLApp = Nothing
    Set myNameSpace = Nothing
    Set currentMessage = Nothing

    Set DataO = New DataObject
    DataO.Clear
    DataO.SetText ClipBoard
    DataO.PutInClipboard
    
    Set DataO = Nothing

End Sub

Function GetMsgDetails(Item As MailItem, Details As String) As String

    If Details <> "" Then
        Details = Details + vbCrLf
    End If
    Details = Details + Item.Subject + vbCrLf
    Details = Details + "Outlook:" + Item.EntryID + vbCrLf

    GetMsgDetails = Details

End Function

del.icio.us tags: Windows, VBA, automation, Outlook, scripting, productivity, GTD