While at Citrix Synergy in Barcelona this week, I attended the Citrix Personal vDisk deep dive session. The session was interesting and informative but there was a mention of the inventory and scanning piece of the personal vDisk suite that really got me asking myself “what if?”.

From my understanding of the presentation, when you add a revision to the golden image, Personal vDisk scan’s both images then compares these items to the personal vDisk in an attempt to figure out which bits belong in the vDisk and which bits belong in the base image.

If you’ve read my previous blog post on golden image management with PVS (questionable assumptions and why I don’t trust people), you know I have a great fear with auditing and control of this image. Without having to read the old article, it basically translated to “Provisioning server is great, but I don’t trust people to audit and document the changes they have made to the golden images”.

While sitting in this session, I had another “lightbulb moment” . If the Personal vDisk has baked in technology that audits the changes to the golden image layer and registry, could it be extracted from personal vDisk? If so, wouldn’t this give you a granular view of changes to the golden image from point to point? I.E. a list of changes between snapshots (MCS) or versions (PVS)?

The more I think of it, the better this idea sounds. Imagine having a catalog of changes, searchable for file or registry key names that would help you track back changes, or even view changes made to the golden image to be reviewed before or after you seal the image? This technology would work well with Citrix Provisioning server, XenClient and Machine Creation Services, delivering a matrix of changes to the golden image.

I can’t see wrapping a gui around this auditing as being a challenge, this is Citrix we’re talking about! and as Citrix has mostly adopted Microsofts vhd file type, it would be a single image type to scan.

For me, this would address my concerns with moving most implementations from automated installs, to snapshot mechanisms while still achieving auditing and a deep view of the changes to the file system.

So Citrix, please consider this approach, it would be an immediate value add and put your image management head and shoulders above your competition.

So what do you the readers think? Would this give you more confidence of changes by others? Do you see this technology and a post change report as an extra safe guard on change management?

Note: This is part three, have a read of part one and two.

Hello there, and thank you for dropping back for part 3…

I suppose I should start with the disappointing news that I have yet to test this option for VDI in a box. And despite Aaron Parker’s suggestions it wasn’t due to lack of inspiration, it was down to lack of time! This series has gathered allot of interest from both community and storage vendors alike, and I feel I should set the record straight before I got any further:

1. This isn’t a production idea, you would be crazy to use this idea in a live environment.
2. Throughout this entire project, we’re focusing on pooled stateless. Stateful desktops would be a separate post entirely.
3. This wasn’t an attack on products in this market space, merely a fresh view on an old problem.
4. If i had the skills or funds necessary to run this project to a production solution, I wouldn’t have posted it. I would already be hard at work creating a reasonably priced product!

Now that my declarations are out of the way, I’d first like to talk about the moral of the story. This isn’t an unfamiliar expression:

IOPS mitigation is not about read IOPS it’s about WRITE IOPS!

VMware, Citrix and Microsoft have similar but very different solutions for read IOPS negotiation. Similar in the sense that they try to negate storage read IOPS. But the key difference with XenServer is the local disk cache via Intellicache has the out of box functionality to cache the majority of read’s to local disk (think SSD*) without the baked in soft limit of 512 MB for Microsoft HyperV and VMware respectively.

Long story short, VMware and Microsoft’s solution is about 512mb of recognizable read IOPS negation un-tuned, but enabled. Of course this value can be tuned upwards, but the low entry point of cache would suggest, at least to me, that tuning up will have an upsetting affect on the host.

This to me is why IntelliCache has the upperhand in the (value add product) VDI space for read IOPS negation and they even throw in the Hypervisor as part of your XenDesktop licensing, so win win, but what about those pesky write IOPS?

Let’s look at Citrix for a moment!

If we review article 1 and article 2 for a moment, the half baked idea formed by Barry Schiffer, Ingmar Verheij, Kees Baggerman, Remko Weijnen and I, when combined with Citrix IntelliCache turned out to be a phenomenal combination to create an IOPS avoidance product. But the key point we had in the back of our heads was, Citrix provisioning server already has this driver!

Citrix provisioning server has a RAM cache driver, with configurable size baked into the Provisioning Services product. This driver works in a very similar way to the EWF driver, just without the API and flexibility of a Microsoft driver.

There are customers of Citrix out there using RAM caching with XenApp who I spoke to, they have assigned a large (8gb+ cache, 3-4gb of cache utilised day to day) to negotiate the chance of a spillover, but it could still happen and this is an acceptable risk to their implementation. I struggled to find a XenDesktop customer using this method who would talk about their implementation.

But with XenDesktop in mind, using an overly large cache size to avoid spillover just doesn’t scale the way a consolidated Server Based Computing model would… Which leaves us back in this same dilemma, I’d like to cache in RAM, but I don’t want to risk users losing data when they perform a task we have not factored for.

So with all this in mind, lets talk about my final hurdle I faced with this project.

RAM cache flood and my half baked solution:

The Microsoft EWF driver suffered the same problem as the Citrix provisioning server when the cache filled, i.e. it froze or outright bug checked if you tried to reference a file put into the cache before you filled it with other information.

Jeff Silverman has done a great article on what happens in this event with the Citrix Provisioning Services RAM cache and I urge you to go read it so I don’t have to repeat it! Go now, I’ll wait right here for you to get back!

Read that article? Ok good, let’s continue.

To combat this scenario with the Windows EWF driver, I created a very rough around the edges windows service using polling (Sorry Remko) to check the size of the cache periodically via the EWFAPI and write the cache out to disk.

The function in the EWFAPI called EwfCommitAndDisableLive allowed me to dump the ram cache to disk and subvert the cache, going straight to disk when this event had occurred. By using this routine the ram cache simply disables itself and allows pass-through to disk at this point.

With this in mind, I tuned my service to spill to disk when the cache was greater than 80% of ram available, not in use by the operating system when the system booted, This worked well to a point, but the key failure to this approach became apparent when you opened a large number of applications and they struggled to occupy the space available around the cache.

My second attempt however, proved much more fruitful where I monitored for free system bytes in memory and if this dropped below a certain threshold the EWF drive began it’s dump routine to disk. Once the dump routine was complete, ram cleared and the writes had been committed to storage where the disk continued to use it for the remainder of the session. Once this spill over had occurred, a notification bubble fired in the user session warning them of the degradation of service and they should log off saving work at the next convenient moment…

Voila! no blue screen, spill to disk and user notification of degradation.

This wasn’t fool proof, it was very rough, it didn’t handle files being written larger than the RAM cache, but In my opinion, it satisfied the biggest fear and business case against using the Citrix Provisioning server ram caching, the ram cache flood scenario. I was happy with the results and it scaled well to the size of my lab, it showed what is possible with a RAM filter driver and it allowed me to prove my point before I started to poke the hornets nest of the storage market. So let’s park the EWF story for now and focus on my favorite subject, Citrix technologies.

Note: I won’t be making this service and driver solution publicly available, it’s not a production ready solution and Microsoft would sue the pants off me. I’m sure you’ll understand why, but if you want more information drop me an email.

The next part’s of this blog are all theoretical, but I know certain people I want to listen, are listening (Hi Kenneth, hope you are well ).

Negating the negated. Lets talk about that spill over.

But what about that Spill over event? by using a spill over from ram to disk, we create a situation where we could change a steady, slightly reliable IOPS per from each desktop, to a situation where, “couldn’t a collection of spillovers at the same time cause your storage to becoming swamped with IOPS?”

Absolutely, and I’ve been thinking about this quite a bit… But there’s another hidden potential here even in a RAM spill over scenario…

With a little bit of trickery couldn’t we also negate this spillover event with a simple provisioning job?

With a bit of time spent thinking about this scenario, this is what I came up with…

Why doesn’t the controller, for XenApp or XenDesktop (PVS / DDC) copy or (my personal preference) create a local, blank differencing disk when a VM boots?

The hypervisor could be completely agnostic at this point, we could spill over to local disk, keeping write IOPS completely away from the shared storage? even in a spill over event?

This approach (in this half baked enthusiasts view) would negate the negated… But don’t take my word for it, lets go for some theoretical solutions.

Solution A: Implementing a spill over routine with the Citrix provisioning server driver.

So with my success with the Microsoft driver, I got myself thinking how easy would this be to do utilizing the Citrix Provisioning Services driver? Without access to the code, I’m going to be making slightly risky statements here, but I have allot of faith in Citrix that they could make this happen.

From everyone I have spoken to about this idea, they all see the value in the ability to spill out of RAM… So Citrix, please make it so. Below are some idea’s for deployment methods assuming Citrix do march down this route and the pro’s and con’s I see that live with each scenario.

Bear in mind, I’m not a storage guy, a full time developer or a software architect, I’m just an enthusiast that see’s potential, so drop your comments below as to what you think!

Machine Creation Services.

Idea A: Provisioning services improved RAM caching, Machine Creation services and Intellicache on XenServer.

Utilizing XenServer and MCS we could leverage the intellicache to negate the read IOPS as we proved in my own EWF driver testing, but still deliver on that spill over mechanism allowing continuation of service.

Pros:

• Read IOPS negation.
• RAM caching to remove write IOPS (bar a spill over).
• Continuation of service in a cache flood scenario.

Cons:

• Limited to XenServer.
• Over provisioning of RAM necessary per desktop.
• RAM spillover will result in a large amount of IOPS to the shared storage.

Idea B: Provisioning services improved RAM caching, Machine Creation services and Intellicache on XenServer… With local copy!

Same benefits as previous, but now, we have zero reliance on the shared storage when the VM is up (except for ID disk actions).

Pros:

• Read IOPS negation.
• RAM caching to remove write IOPS.
• Uses local resources in a spill over.
• Continuation of service in a cache flood scenario.

Cons:

• Limited to XenServer.
• Over provisioning of RAM necessary per desktop.

So that’s MCS in a nutshell with per VM caching, I think this solution has so much potential I can’t believe it’s not been done, but I digress. So lets park that topic for now and move on to Citrix Provisioning Services.

Citrix Provisioning Services:

So lets look at the “favorite of many” technology.

Idea A: Citrix Provisioning Services and Improved RAM caching driver.

In a pure Provisioning services environment, we would force our read IOPS via the lan, instead of storage protocol but still deliver a spill back to disk to allow continuation of service.

Pros:

• Hypervisor agnostic.
• RAM caching to remove write IOPS (bar a spill over).
• Continuation of service in a cache flood scenario.
• Potentially no shared storage needed, at all, if caching on the PVS server.

Cons:

• Read IOPS aren’t really negated, they’re just forced over another technology.
• Over provisioning of RAM necessary per desktop.
• RAM spillover will result in a large amount of IOPS to the shared storage / pvs server.

Idea B: Citrix Provisioning Services and Improved RAM caching driver.. With local copy!

Taking the above benefits, but with the gain of utilizing local storage in the spillover event.

Pros:

• Hypervisor agnostic.
• RAM caching to remove write IOPS.
• Uses local resources in a spill over.
• Continuation of service in a cache flood scenario.
• Potentially no shared storage needed, at all.

Cons:

• Read IOPS aren’t really negated, they’re just forced over another technology.
• Over provisioning of RAM necessary per desktop.

So lets Review:

And there we have it, 4 solutions to IOPS negation utilizing the Provisioning server RAM caching driver, with a little bit of a modification to deliver a robust solution to RAM caching.

The copy and creation of differencing disks again would deliver additional benefits to leverage the hardware you put into each Hypervisor without the Shared Storage investment.

Win Win, but is it?

There’s an oversight here:

There’s a niggle here that’s been bothering me for some time and you’ll probably note I mentioned it as a CON to most of the solutions above… I’m going to lay it out on the table in complete honesty…

“Isn’t over provisioning RAM on a per desktop basis a waste of resource? Wouldn’t it be better if we could share that resource across all VM’s on a Hypervisor basis?”

You betcha! If we are assigning out (for argument sake) 1gb of ram cache per VM, that RAM is locked into that VM and if another machine spills, the free RAM in other desktops is wasted.

You would be completely insane not to reserve this RAM per VM, if an over-commit for the VM’s is reached this RAM will merely spill out to a page file type arrangement, negating all your benefits.

Ultimately, assigning RAM in this way could be seen as wasteful in the grand scheme of things…

But there are other ways to skin this cat!

So this leads me on to something I was also considering which popped up in a twitter conversation recently. RAM disk’s and Hypervisors.

Current storage vendors will create a storage pool, consisting of ram inside a VM, per hosting hypervisor for local storage to allow good IOPS leveraging that VM. This VM can perform additional RAM optimizations like compression, de-duplication and sharing of similar pages to reduce the count.

This approach is super clever, but, In my humble opinion (please don’t kill me vendors), this approach is wasteful. Running a VM as a storage repository per host has an overhead to run this virtual machine and takes away from the agnostic solution that is possible…

What If the hypervisor provides the RAM disk?

So excluding ESXi for a second, as getting a RAM disk into that platform would require VMware to allow access to the stack. Lets look at Hyper-V (2012) and XenServer for a second…

With Unix and Windows platforms, RAM disks have been available for years. They were once a necessity and a number of vendors still provide them for high performance IO environments.

Lets say (for arguments sake) Citrix and Microsoft decide to provide a snap-in to their hypervisor to allow native RAM disks (or a vendor writes one themselves!) and maybe, they even decide to provide RAM compression, Spill over to local disk, dedupe, and page sharing on this volume from the hypervisor stack…

Wouldn’t this extend provide all the benefits we’ve spoken about, without the need for a VM per host? And using Thin Provisioning allow all desktops to share the large pool of RAMdisk available?

Yes, yes, it would.

Above is an example of how I would see this working.

So random pictures are fine and all, but what about the read IOPS negation technologies? and what about combining these with XenServer or Hyper-V?

XenServer and IntelliCache:

Well there you have it now, all the benefits of a per VM filter, leveraging intellicache for reads and spilling out to local disk… RESULT!

Pro’s:

• Read IOPS negation
• Write IOPS negation
• No Shared Storage required for running VM’s
• Shared pool of RAM for all desktop’s to use.

Con’s:

• A small one, but no migration of VM’s.

Provisioning server?

and again, all the benefits of a per VM filter, reads redirected via lan and spilling out to local disk… RESULT!

Pro’s:

• Write IOPS negation
• No Shared Storage required for running VM’s
• Shared pool of RAM for all desktop’s to use.

Con’s:

• A small one, but no migration of VM’s.
• No real read IOPS negation.

And HyperV + CSV Cache?

Well here’s an accumulation of my thoughts:

So lets just talk for a second about what we are seeing… Utilizing a RAM disk with spill over, and copied vhd’s on boot we are removing the need for shared storage completely and hosting the IOPS from the hypervisor, natively without the need for additional VM’s.

And see that little icon down the bottom? Yep, that’s right, live migration from host to host thanks to Microsofts Shared Nothing Live Migration!

Pro’s:

• Some write IOPS negation
• No Shared Storage required for running VM’s
• Liv migration.
• Shared pool of RAM for all desktop’s to use.
• write back to local disk.

Con’s:

• No full read IOPS negation.

Review.

I’m sure there’s loads of reading in here, and there will be tons of thought and questions after this blog post. This has been in my head for roughly 9 months now and it feels victorious to finally get it all down on paper.

At the end of the day, Ram Caching is the way of the future, you pay no maintenance on it, you keep your IOPS off your storage and with a little magic from Microsoft, or particularly Citrix. You could see these benefits.

If / When these technologies make it out to you, you could quite happily stick with Machine Creation services, leverage intellicache and your shared storage requirement could be a cheap NAS from any vendor. Provisioning services also see’s allot of benefits from this approach, but the real creme de la creme is in that intellicache feature.

The key functionality to these approaches is simple:

• Cache RAM
• Spill to disk
• Keep the differences on local storage if it spills.

One last thing?

In true Mark Templeton fashion, you know I have a one last thing. This actually hit me today while writing this post and to be honest, I’m so amazed by the potential of this idea I’m going to build it in the lab this weekend.

But until then, a teaser…

What if I told you there was already an overlay, freely available baked into a Microsoft technology that would allow you to overlay local storage in Hyper-V with RAM. A powerful API attached to it, and the ability to write specific files to the local disk, subverting the overlay only when needed?

• RAM disk type technology? > yep
• already available? > yep
• Powerful API? > yep
• already can spill to disk? > yep

Yep Yep Yep! check back in a few days.

(If you know what it is already, keep it to yourself until I’ve gotten out of the lab!)

What do you think?

It doesn’t take a half baked enthusiast like me to see this potential and I’d be really eager to hear your comments on these approaches. If you would prefer to keep your comments offline, you can reach me on andrew@andrewmorgan.ie.

With great pleasure I’m announcing the general availability of ThinKiosk 3.1. Quite a bit of change under the hood and some nice features added to match.

New features:

VMware View enhanced support:

VMware View has gotten some love in this update, A big thanks to Jarian Gibson for the help.

You can now enforce end of session options for VMware view:

You can also now choose to wipe the last users details from the Kiosk between View sessions:

FTP policy management:

With ThinKiosk 3.1, you no longer are tied to manage the thinkiosk devices by Group Policy or local registry settings, you can now also use an ftp server with a shared xml configuration file:

Just configure a Device as you would like it to appear, unlock the admin menu and you can export the configuration to xml:

Then move it to your ftp server!

Encryption:

The unlock password in group policy can now be encrypted to save it appearing in plain text to anyone capable of viewing the policy. ThinKiosk 3.1 ships with a password encryption tool you can use to encrypt your password.

You can also test reversing the password to plain text to make sure you get it right before applying it en-mass and locking yourself out!

This encryption functionality has now been added to both the offline configuration tool:

And by default the FTP password will be encrypted too!

Battery Awareness:

ThinKiosk is now aware of batteries in laptop devices and will report their status.

When the battery begins to run out, ThinKiosk will throw a warning in the foreground as below:

You can additionally disable this functionality with the offline configuration tool.

Pre launch Citrix Receiver:

A rare issue seen with the latest versions of the receiver was a bit of a hang, pause or complete lock up as receiver came to life. To combat this, you can now choose to early launch the receiver for Citrix, allowing it to gracefully start up in the background before the user requires it.

Early launch process:

A number of customers needed to have third party software launched as soon as ThinKiosk started each day. I’ve now added the ability to early launch a process 

You can also choose to launch this process as hidden, away from the user.

Browser navigation buttons:

ThinKiosk can now act as a locked down browser by adding back and forward buttons.

AM / PM clock:

This feature was asked for quite a few times, so now you can set the clock to 12 hour.

Debug Mode:

A fully fledged debug window has been added to help timing issues. The debug menu can be accessed via command line (-debug) or via the admin menu in ThinKiosk.

Zorder awareness:

In rare situations (and I’ve been unable to reproduce it) ThinKiosk can jump above the citrix session when a log off of the web interface happens or during the login process.

Zorder awareness will tell ThinKiosk to send itself to the back of the Zorder when the browser finishes rendering. It will also display a hide button, which will send ThinKiosk to  the back in this rare event.

Please use this setting as a troubleshooting tool, not a production setting. If this setting fixes the issue for you, please drop me an email and I’ll write it in. As I’ve been unable to reproduce this issue, it’s a bit rough around the edges.

Citrix Storefront timeout screen:

ThinKiosk is now aware of the timeout screen and will automagically redirect back to the login screen if it see’s it.

Hide ThinKiosk when a desktop is active:

If you wish to outright hide ThinKiosk while a desktop is active, you can now do so!

Even More sites:

Support for up to 20 sites has been added, thanks Martijn!

Sticky Home Page:

A request came through to allow the home page always be site 1, this has now been included.

Bug Fixes:

• support for environment variables in custom tools and prelaunch commands. (thanks Nathan).
• Offline config tool not setting password correctly.
• VB Powerpack accidentally bundled with ThinKiosk 3.0
• In process launch mode, power options were intermittently being applied.

And it’s still free!

ThinKiosk development has taken quite some time and it takes time to support you via email. If you use ThinKiosk in your environment or appreciate the savings its made for you, please consider making a donation or paying for enterprise support to help me keep this project alive… I would really appreciate it as it will allow me to invest in better development tools to make the product look and feel even better!

Here’s one that used to bother me alot. The problem usually went as follows:

“Your XenApp servers have very high disk queue’s and IO”

“What’s causing it?”

“dunno…”

With Server 2008, the task manager’s resource monitor feature will help you find these items. But in server 2003 this was a perilous task. The specific details for disk io per process are stored in performance monitor under each specific process running. Trying to analyse each process was a massive pain, but powershell can do some very clever work to help alleviate this!

I wrote two quick functions which act similar to “top” in linux for giving an on screen view, updating at interval of what exactly is creating IO activity. These two functions are:

get-IODataBytes:

Get-IODataOperations

The code for these functions are below:

function get-iodatabytes{
    $result=(get-counter -counter "\Process(*)\IO Data Bytes/sec" -ea 0).countersamples | ? {$_.cookedvalue -gt 0} | select instancename,@{Name="SessionID";Expression={if ($_.path.contains("#")){($_.path.split("#)"))[1]}else{"0"}}},@{Name="IO Data Bytes/sec";Expression={[math]::Round($_.cookedvalue,0)}},@{Name="IO Data KBytes/sec";Expression={[math]::Round($_.cookedvalue / 1024,0)}} | sort -Descending "IO Data Bytes/sec" | ft
    $currentqueue=(((get-counter -counter "\PhysicalDisk(0 C:)\Current Disk Queue Length" -ea 0).countersamples) | select cookedvalue).cookedvalue
    clear
    write-warning "Hit [CTRL] + [C] to exit live capture"
    write-host "Current Disk queue: $currentqueue"
    return $Result
}

FUnction get-IODataOperations {
    $result=(get-counter -counter "\Process(*)\IO Data Operations/sec" -ea 0).countersamples | ? {$_.cookedvalue -gt 0} | select instancename,@{Name="SessionID";Expression={if ($_.path.contains("#")){($_.path.split("#)"))[1]}else{"0"}}},@{Name="IO Data Operations/sec";Expression={[math]::Round($_.cookedvalue,0)}} | sort -Descending "IO Data Operations/sec" | ft
    $currentqueue=(((get-counter -counter "\PhysicalDisk(0 C:)\Current Disk Queue Length" -ea 0).countersamples) | select cookedvalue).cookedvalue
    clear
    write-warning "Hit [CTRL] + [C] to exit live capture"
    write-host "Current Disk queue: $currentqueue"
    return $Result
}

if you wish to loop one of these functions, simply use the following code:

while ($true){
get-iodataoperations
start-sleep 1
}

So this is a situation you should all be aware of in an SBC / VDI environment, despite all warnings, you’ve redirected folders to your network drive and your file servers are screaming in agony?

Having been in this situation recently, I needed to audit and report on the types of files open on the file server, my hunch was a certain select number of users were running applications (like *gulp* lotus notes) from the network share.

Disappointed with the powershell scripts on the interwebs, I decided to write my own function to perform this task:

function get-openfiles{
param(
    $computername=@($env:computername),
    $verbose=$false)
    $collection = @()
foreach ($computer in $computername){
    $netfile = [ADSI]"WinNT://$computer/LanmanServer"

        $netfile.Invoke("Resources") | foreach {
            try{
                $collection += New-Object PsObject -Property @{
        		  Id = $_.GetType().InvokeMember("Name", 'GetProperty', $null, $_, $null)
        		  itemPath = $_.GetType().InvokeMember("Path", 'GetProperty', $null, $_, $null)
        		  UserName = $_.GetType().InvokeMember("User", 'GetProperty', $null, $_, $null)
        		  LockCount = $_.GetType().InvokeMember("LockCount", 'GetProperty', $null, $_, $null)
        		  Server = $computer
        		}
            }
            catch{
                if ($verbose){write-warning $error[0]}
            }
        }
    }
    Return $collection
}

The function above (get-openfiles) has been written to accept an array of servers to the command line and it will return the following items:

• The ID of the open file.

• The server it’s open from.

• The username who has the file open.

• The amount of locks the file has.

A couple of quick examples for using this command are below:


Retrieving open files from server1:






get-openfiles -computername server1 | select server,itempath,lockcount



Retrieve a count of open files that end with the nsf file type (Lotus Notes):






(get-open files -computername server1,server2 | ? {$_.itempath -like "*.nsf*"}).count()



Retrieve a report of total open files on a number of file servers:



get-openfiles -computername server1,server2,server3,server4,server5 | group -property server

Here’s a really strange and interesting issue I faced late last week that resulted in a few head scratching moments and late nights.

An Issue began two weeks ago intermittently in a XenApp 4.5 farm used for hosted desktops, intermittently NonPagedPool bytes would shoot through the roof, the event logs would become inundated with event 333 errors and the servers would lock up completely.

The running sessions could no longer open new applications, performance was extremely poor and RDP’ing to the XenApp server would result in an RPC error. Disconnecting the sessions remotely would also result in an RPC error or TSAdmin was completely incapable of connecting to the server. We had no choice but to dump the servers using NMI and pray for a crash dump.

No changes had been made to the environment in a number of weeks and the last change to the environment was a “Citrix Ready” printer driver from Lexmark. As the days progressed the issue got worse and worse with more servers going offline day by day. Although we did initially catch a number of crash dumps, we hit a bad run of luck with most of them being corrupt on restart.

By day six, 9 servers went offline throughout the day, I was pulled in to assist resolve this massive issue.

I fired up the windows debugging tools and managed to get a look at a crash dump fresh from a locked up server.

Using !vm i pulled the virtual memory statistics at the point of the crash:





So we had a serious non paged pool leak as we suspected, but what exactly was chewing up all that nonpaged?

Running !poolused 2, i was able to drill down into the drivers using nonpagedpool and view which driver tag was using the largest amount of the pool as below:





reviewing the list, i was immediately alarmed by the amount of ram in use by the “Ica” pool tag. Having reviewed 100′s of memory dumps I had never seen the Ica pool tag listed in the top 20, never mind using 99721328 bytes (~95mb).

The Ica pool tag is fairly obvious as to who owns it, but just to be on the safe side and to drill down to the owning driver, I ran the following command on the drivers folder to find references to the “Ica” pool tag.



findstr /m /l Ica *.sys



So we got quite a few hits off the Ica pool tag. Quite a number of the above drivers are Microsoft’s, which is not suprising in the grand scheme of things as we all know the origination of the RDP protocol.

So with a little more information to hand, I set about googling this chain of events to see if it’s either documented, or hotfixed. A search yielded quite alot of articles including a private hotfix and a Rollup pack.

Drilling down into the technotes to see if I could find a potential cause for this issue, I was left a little wanting with the lack of information available:

Servers with Hotfix Rollup Pack 6 installed can run out of non-paged pool memory in the ICA Pool tag. The issue occurs when users log on to the server and its frequency increases with the amount of time since the most recent restart and the number of logons since the restart.

What irked me here, was the lack of information and the fact that these servers had been running HFRP 6 for roughly 18 months with no issues similar to this.



Why all of a sudden are we losing servers all over the place to such an issue?



I dug further into the hotfix notes with help from my good friend and all round cool Citrite James Denne, the hotfix specifically noted:

When a server is in low memory condition the <Redacted>() spins in an infinite loop by constantly allocating an OUTBUF and trying to write it on the stack. This problem happen when system is going in and out in low memory condition.

So there’s a great explanation of the issue from the horses mouth, but again there was a niggling problem in the back of my head…



These servers weren’t spinning in and out of low memory, our pool usage reporting would have caught this?



I was satisfied to see a hotfix was available, but in the back of my head I was concerned about the change that may have caused this issue, it’s still unclear what is causing this low memory condition to spin the infinite loop and why we couldn’t see the low memory scenario before it happens. Being an massive issue, we had to make a quick turn around here. We had a choice of going to HFRP 7 or using the private hotfix available. I chose the private hotfix, for two reasons:

• Mass Deploying a roll up pack to fix one problem is like tapping a nail in with a sledge hammer.
• My experience with HotFix Rollup Packs is they fix your issues, but introduce at least one new one.

We took all the servers offline for emergency maintenance that night and cautiously waited for the morning to come and see if our issue was resolved.



and so we patiently waited…



Once hotfixed and rebooted, we arrived at the office early to watch as the user sessions began to filter in to the Farm. All was quiet for the first hour or so, but then the phones started.

once the user load hit 15-16 users per XenApp session, a number of servers began to again log a number of eventlog 333′s as below:






Dammit.



Frantically we connected to the console of a server, to check the paged pool states but again no alerts on pagepool size? as below the ICA pool tag was nowhere to be seen:





And the ica tag was at a much more respectable / expected value as below:





Our next clue came in the form of the following, when users were logging in they were getting the following error:






So we’ve fixed our Ica memory leak, now what the hell is happening?



If memory usage for the pools are ok but we’re still getting errors about flushing to the registry, and now new user profiles can’t load their profiles, my hunch was there had to be something wrong with the registry hives…

I used command prompt to open the “Documents and Settings” folder and ran the following command:

dir /s /a ntuser.dat

With a quick glance, i found the following:





The “Citrix Print Manager Service” user account had a registry hive of over 50mb? What in the name of superman is hiding in that registry hive?

To rectify the issue immediately, we stopped the above print manager service and forced the hive to be unloaded with delprof. Once we had done this, the user profiles began to load again on each affected server. But we’re now unable to use client redirected printing.



To regedit!



I mounted the registry of a profile that had failed to delete and drilled down to see what all the fuss was about. As this was  now firmly in the printing land, I went looking for keys to match the Lexmark driver change from a number of weeks ago.

What I found was extremely interesting, for each client redirected printer ever mapped with the Lexmark driver, we had an entry under both the PCLPlugin and PSPlugIn keys:





Although this was a really questionable practice from lexmark, I examined the keys under each entry for the PclPlugin key and they consisted of just a few small binary files of which were no more than a KB or two.

Upon looking at the same keys under PSPlugin, I found a new key, called GDL. This GDL key was absolutely massive and there was one for each and every time a client printer had been redirected using the Lexmark V2 driver.





I exported both both the users hive, and the psplugin key to text and the comparison is below:





The GDL key itself was over 3mb per redirected printer!?!?:



So there we have it.



The route cause was as follows:

This Lexmark driver has a weird tendency to store an unbelievable amount of crap in the registry for users.

The Citrix print manager service also suffers this faith when it maps a redirected printer.

As more and more users were testing in production (GRRRR!) / beginning to use a new printing solution on a customer site, this registry file began to grow and grow ultimately flooding the maximum registry size of 30% of the paged pool ram.

As the registry hive size was growing out of control, the Ica driver ran into a low memory situation and ultimately caused the infinite loop.

The Ica loop and nonpaged saturation was masking the printer driver bloat in the registry.

As the days went on, more and more servers began to saturate the Maximum registry size and go offline.

Corrective actions:

• Enforce a policy to not allow native drivers, in any way, shape or form when redirecting printers where possible.
• Obtain the latest driver from Lexmark is you have lexmark printers.
• Give lexmark an earful for not testing their drivers.

Lessons Learned:

• Don’t test things in production.
• Don’t trust a vendor with “Citrix Ready”, it’s up to them to test these things and they regularly don’t.
• Create a monitor for registry size (perfmon > system > % Registry quota in use)
• install the debugging tools on the XenApp 4.5 servers as this issue is going to become more prevalent. *

* This isn’t going to get any better.

As Vendors move further and further towards 64 bit architectures they can and will forget about the extremely restrictive memory sizes available in 32 bit versions of windows, 64bit windows has so much memory available for the pools they can be as sloppy as they want without much concern.

Server 2003, Windows XP and XenApp 4.5′s death bells are knelling and have been for some time.

You are going to see pagepool’s floods and other such nasties more and more in the coming months before you finally decommission your old server 2003 environment. My advice to you is to:

• get very comfortable with the following tools:
  • PoolMon.
  • Process explorer.
  • Windows debugging tools.
• Have a good read of the following article: 333, read it, read it again.
• Never be afraid to have a look at a dump file yourself.
• Throw an issue at every vendor possible during troubleshooting, it’s in their interest to prove it’s not their software at fault.
• Understand your pagepool sizes and limitations.
• Never trust a printer driver.
• Never, ever, ever trust a Vendor to behave accordingly or follow the Citrix Ready standards.
• If you absolutely, positively need to run something in server 2003 or XP, consider using XenDesktop hosted apps to isolate the problem to a singular kernel away from the bulk of your task workers.

Here’s a fun little customisation if you grow tired of the green bubbles of gloom.





The background above is a png file, with the following dimensions:

• Height: 2048
• Width: 1056

So if you want to replace this file, go find your replacement picture and ensure your picture is of a similar enough size.

Once you have a png file with similar enough dimensions, open the finder application, open the applications folder and right click the Citrix Receiver app, choose “Show Package Contents”.

Browse down to: contents > resources





In this folder, you will find a file “backgroundImage_big_b.png”, before you start, rename this file to back it up.

Now simply copy your replacement file into this folder, using the above name:





And that’s it! You’ve now got a lovely custom Citrix Receiver:





PS: I wouldn’t try to do this with windows, the file is an embedded resource and would require resource hacker to change the file.

A little irk of mine with Windows 7 and server 2008 R2 was the Devices and Printers interface. This mix of peripherals is fine for standard desktops, but in SBC / VDI the devices list generally contained items you didn’t want users seeing, or ejecting for that matter!

Not happy with the Irk, and still on my app developing buzz, i decided to write SBC Printers:

SBC-Printers is a simple little .net 4 application, leveraging WMI for printer enumeration and control.Because SBC Printers is an executable, it can published as a XenApp application. Sbc Printers can also be installed as the default printers interface on the start menu:

So really your users won’t know the difference or care for that matter!

SBC-Printers also comes with securable options for adding or deleting local printers:

The display of add or delete can be controlled via the settings file in the installation directory:

Installation:

1. Download the following MSI
2. Install the MSI to the default directory.

To restrict the standard printers dialog from users, but leaving it accessible to administrators:

• Browse to c:\program files (x86)\SBC-Printers\bin

• run the powershell script below, make sure to run it as an administrator!

That’s it, once the Powershell script runs. it removes the users access to the registry classes giving them access to the standard devices and printers interface. Which means we’re now ready to provision SBC-Printers to replace it.

Provisioning the replacement to the user:

Now just import the userkey.reg into the users profile on login, you can do this via your user profile manager of choice, or use Group Policy preferences.

That’s it!

As you can see I haven’t streamlined the install process too much, this is mostly down to the simplicity of the tool. If you like SBC-Printers but would like a better installer, just drop me a comment below.

Roll back:

if you need to restore the standard interface, uninstall SBC-Printers then add the (local computer\users) group back to the following registry keys ACL:

•  HKCR\software\classes\CLSID\{A8A91A66-3A7D-4424-8D24-04E180695C7A}
• HKCR\software\Wow6432Node\CLSID\{A8A91A66-3A7D-4424-8D24-04E180695C7A}

Hi Guys and Gals. I’m currently fighting the good fight with Microsoft support and require your help and backing in order to close down a long standing bug in the Windows Explorer Shell.

As you are all aware, hiding the c: drive and restricting access has been a utility we use frequently in shared computing and VDI environments. Restricting this functionality removes views of the shared drive from users and adds a layer of security and complexity* to ensure the users in question have access to only what they need in order to do their jobs day to day.

*I’m not looking to argue the merit of doing this either, it really depends on the business case or environment to dictate whether this option is set. I’m NOT saying it should be done in every case.

We all know it’s not fool proof, there are certain ways for users to circumvent this layer and I particularly don’t want to discuss them here to give potential devious users a landing page for idea’s!

The problem:

Prior to windows Vista, when you hide the c: drive and an application requests access to a c: drive folder, be it from an “open save dialog” or otherwise, Windows detects this event knows that the folder is restricted and merely redirects them to the desktop which they can see then browse to where they wish to open or save a document. This has worked fine to memory since windows server 2000.

But with the changes to Windows Vista’s windows explorer, repeating the above steps will result in the following annoying, unnecessary and interrupting error message “This operation has been cancelled due to.. bla bla blah”:

This issue can be easily recreated, simply hide and restrict the c: drive, then click start > run > browse… bang.

The more annoying problem here, is after the error message, windows simply redirects back to visible folder. In most cases this is the documents library. So the error message is simply poping up then reverting to the functionality seen in previous operating systems.

So to review:

• Issue introduced in Vista / 2008 and above.
• error message displays.
• Previous redirect functionality is still there and occurs after ok is pressed.

To Microsoft!

Being a pedantic individual, along with my colleague we brought this to Microsoft support and somehow lost months in the conversation as follows:

1. Microsoft then redirected us to RES Software.
2. Who (although very nice about it) sent us back to Microsoft.
3. At which point I got involved.

Now with the correct audience and suitable severity, this problem has been identified as “introduced in Windows Vista” as an “Added Security feature“. How an annoying pop up box, masking previous functionality is a security feature is anyones guess, but it’s bloody annoying…

We have raised this as a bug and have requested Microsoft to fix it. The change in question was deemed as large change or substantial change due to WIndows explorer being used by all of the operating systems and basically told, without significant backing, this change wont be implemented.

Bureaucracy and broken policies, yes but that doesn’t help my customer.

Here’s where I need you:

In order to bolster this change and fix an issue in our beloved operating systems for Server Based Computing and VDI environments I need to hear from you and your customers to confirm they have had this issue, or currently face the issue and wish for a fix.

• If you are a customer and suffer this issue, email me.
• If you are a consultant and have customers with this issue, email me.
• If you or your customer have enterprise support with Microsoft, I ESPECIALLY want to hear from you.

What’s in it for you?

Microsoft have provided us a work around, as a process that watches window messages and suppresses this dialog box when it occurs. If you get in touch, I’ll recompile this application with Microsofts permission and pass it on to you for use in your environment while we get “The Man” to fix it!

This fix is a bit of hack, as it’s scraping window messages but it’s light weight and scalable. Use this process for now and I’ll provide you with updates on a fix as and when I get them.

How do you contact me?

Please drop me and email on andrew{at}andrewmorgan{dot}ie with the following information:

• Customer name:
• Affected users:
• Has enterprise support: (yes/no)

Once I have that information, I’ll send you back an executable via dropbox and keep you updated on the call process. This information is merely going to be fed straight to Microsoft with my personal guarantee of confidentiality. No funny business.

If you can’t share customer information, but have suffered this issue in the past, no problem! Please comment on this blog post the number of seats that were affected and roughly how many times you’ve seen it.

That’s it!

Thanks for entertaining my request for help and hopefully you too want to get this issue fixed as much as I.

Everyone having a Good Citrix Synergy week? Some great new products announced! Ready for more announcements?

Great!

After 5 months of coffee, tears of frustration and hair pulling we’re absolutely delighted, thrilled and relieved to announce ThinKiosk 4.0 is nearly ready. Complete with my new partner in crime Remko Weijnen (I’ve been saying ‘we’ for ages, now you know who… awesome eh?) we’ve worked some long nights to get this version out the door.

With that out of the way, we’re proud to announce some of the new features coming in 4.0. Bear in mind this is just a preview, the final features and details of the product are still being hammered out, but below is a taster of some of the functionality you can expect to see shortly.

Back to the drawing board:

ThinKiosk 4.0 is a complete rewrite and refactor of ThinKiosk. It’s built on the 4.0 .Net framework which has brought a lot of simplicity and new features to our tool-set. ThinKiosk 4.0 was built with three main aims:

• Enterprise Ready.
• Fool Proof.
• Secure by Design.

With ThinKiosk 4.0, your setup time will go from days to minutes. Out of the box, ThinKiosk is ready for the following technologies without any local machine tuning:

• Citrix XenDesktop / XenApp.
• Citrix VDI in a Box.
• VMware View.
• Microsoft Remote Desktop Services.

For the exact details of each of these optimizations, follow the subsequent blog posts / documentation.

New Look and Feel:

Without further ado, lets start with the new look and feel:

ThinKiosk 4.0 has also been built on the industry leading graphical interface DevExpress giving us a really shiny, professional and sleek interface. Finally giving us an Interface we can be proud to put on your desktops.

ThinKiosk’s interface has been further improved giving you an Applications tab for Publishing desktops for VMware View, Microsoft Remote Desktop services or Citrix Desktops via ICA file or local applications.

This Applications tab has been modelled after the windows 8 Metro err, I mean Windows 8 UI. This provides a similar look and feel to the new Windows start menu and it really breathes new life into old hardware. With this tab, you can publish shortcuts to VDI Desktops or local applications making it a one stop shop for applications.

You can flick from one tab to another easily, or disable the one you do not wish to use.

It’s all about the customization!

Beauty is in the eye of the beholder right? Agreed!

Themes:

 ThinKiosk 4.0 will ship with over 8 themes and wallpapers, customization of the splash screen, buttons… everything!

The Applications tab can also be completely customized to your tastes:

Lock down:

As with Previous versions of ThinKiosk, every button and object in ThinKiosk can be locked down to exactly what you wish, for example here’s a stripped back browser session:

Or a stripped back application window:

Anyway… Enough about the appearance, Lets talk tech!

Introducing the new ThinKiosk Broker Service and Management console:

The ThinKiosk Broker, Management Console and ThinKiosk clients use an all new ThinKiosk TCP protocol (I never ever, ever want to see a tcp socket again for as long as I live, writing this protocol was a killer!) to allow you to centrally manage, catalog and report on your ThinKiosk devices. The protocol is lightening fast and secure by design.

This new framework will form a long blog post itself, but some quick fire information is below:

• Complete off domain management.
• Auto device registration, just point ThinKiosk at the broker and it will check in and download the default profile.
• Remote Control / Shadowing of end point devices via the console.
• Device Grouping for profiling multiple devices or creating an organisation structure.
• Remote actions (power off, restart, update).
• Device Reporting.
• No Enterprise database software necessary.
• Audit logging.

Unlike other Thin Client protocols and software, ThinKiosk does not accept any inbound connections, in user or system context. Removing the ability to hijack thin clients… which is all too possible with certain vendors!

The console is simple, and quick to navigate:

Installation of the broker takes roughly 5 minutes and is ready to serve your Devices as soon as you configure the default profile.

New Profile Handler:

The ThinKiosk client has received an overhaul and with it we’ve streamlined the profile. ThinKiosk no longer requires group policies or the clunky offline config tool, we have a new profile system based on XML files with a fitting profile editor to match:

No more configuring 5 group policies for one url, the new policy manager is clean, self explanatory, full of new functionality and uses the same interface whether you are using the ThinKiosk management console or modifying the local profile.

If you want to still use group policy to deploy configuration? No problem! just drop the file on the client via group policy preferences!

And the Client!

Lets talk about the 4.0 client.

Supported platforms:

Windows XP – Windows 8

Browser Ahoy!

ThinKiosk is now a fully fledged browser, complete with address bar. If you want to allow your users to browse around, now you can.

Browser improvements:

The ThinKiosk 4.0 browser will:

• Supress scripting errors.
• Allow you to add your sites to the trusted sites via policy.
• Auto tunes the browser for VDI portals.
• Auto circumvent silly SSL untrusted or mismatched errors (great for POC’s *cough* VDI in a Box *cough*)
• ThinKiosk now runs as an Internet explorer executable. No more flicking between iexplore.exe and thinkiosk.exe.

VDI Improvements:

Now to the nuts and bolts!

Local login pass through:

Now that you have the ability to add direct VDI connections. ThinKiosk will handle the log in experience and pass the credentials to the responsible technology:

This integration allows ThinKiosk to better manage the desktop experience and provide your users with a single login pane rather than the recurrent login screens you can experience with Microsoft / Citrix file connections.

These connection files can also be auto launched, to remove that pesky click first thing each day.

Citrix Technologies:

• Log off screen redirection for Web interface, storefront and VDI in a box.
• Log off the web portal when a desktop launches for the above platforms.
• Support for Adding ICA file connections.
• Auto configuration of Single sign on from local pc to remote desktop. (Nightmare previously).
• VDI in a Box auto browser tuning for compatibility.
• Optionally disable the Citrix Desktop viewer (CDviewer.exe).

VMware View:

• Support for publishing multiple pool connections
• Support for publishing multiple direct desktop connections.
• Support for PassThrough.
• Disables Certificate checking by default for quick POC’s.
• Pass through ctrl alt del / Windows + l (more on this later).

Microsoft Remote Desktop Services:

• Support for publishing multiple connections.
• Support for 2012 RDS and VDI.
• SSL Certificate warning suppression.
• Support for login once.

Improved local application handling:

ThinKiosk 4.0 has an improved local application engine, When you add an application to the Applications tab, it will automatically pull in the icon window and you can also specify to launch apps but hide them (think run key entries). If ThinKiosk is restarted via admin task, it’s smart enough to know not to relaunch them.

Environment variables for paths and arguments are fully supported and i’ve also added a variable for 32bit program files paths… I always wondered why Microsoft didn’t do this, but I digress.

Windows secure keystroke blocking and passthrough:

You asked… (and asked and asked and asked and asked). It’s done, with ThinKiosk 4.0 you will be able to block CTRL + Alt + Del, [Windows] + [L] etc.

Pass through of these keystrokes to the remote desktop is available for VMware View already and will be coming shortly after 4.0 for Citrix and Microsoft connections.

Group Policy Lockdown:

By default when you install ThinKiosk 4.0, it will arm the PC with the most restrictive policies via the local group policy engine, disabling access to all admin utilities and even local disks. This lockdown can be tuned or turned off via policy if required.

ThinKiosk performs privileged actions via the ThinKiosk Machine service which installs as part of the installation.

Auto log in account:

ThinKiosk will ship with it’s own user account for fast deployment. This account will be created on the local machine and gives you a quick an easy method to manage local accounts on non domain joined PC.

The accounts password is synchronized with the ThinKiosk unlock password you specify.

This account is completely optional and you can turn it off or substitute it with a domain account of your choice.

ThinKiosk will also manage the Windows Shell replacement policy itself via policy, so no more mucking around with local group policy or registry keys.

ThinKiosk also now encrypts the auto login account using LSA.

Active Setup:

With ThinKiosk as shell, you can now run Active Setup with ThinKiosk’s improved Active Setup Async.

Active setup Async is a utility we have implemented into ThinKiosk that will perform active setup 60% faster than standard Microsoft active setup via a threading and queuing engine, the end result is active setup support ( for example: HDX flash redirection) with a much faster (and prettier)  interface.

Start up Script:

ThinKiosk can now implement the local group policy engines start-up script to allow you to manage off domain PC’s. With the start-up script, you can install software, updates, disable services, uninstall software, delete files, profiles… anything!

The only limitation here is your own imagination or scripting abilities.

If the latter is a concern? worry not, we’ll be creating a scripting library where ThinKiosk enthusiasts can share and collaborate on similar tasks.

Local session control:

ThinKiosk 4.0 offers you the ability to control local volume, printers, screen saver and even background color.

Improved debug logging:

ThinKiosk logs everything, every action, command, hiccup… everything.

If something isn’t quite working as expected, chances are the debugging window will announce in triumphant glory exactly what is broken!

Redundant profile management:

ThinKiosk takes a copy of it’s profile on each check in to an FTP server or Broker server.

In the event of the server being offline ThinKiosk attempts five times to connect before failing back to the local profile allowing your users to continue working without an outage.

If the broker server becomes available again throughout the day, ThinKiosk will check back in to allow management but will not disturb the user.

And so much more!

I’m not going to go on and on, but as you can see… It’s awesome!

Check back in a few weeks for the release as we ready the build.

While working with a customer recently on a sharefile implementation, I set about creating a SAML / Active Directory single sign on deployment. Configuring ADFS and SAML were complete unknowns to me so I set about documenting the process end to end for future reference.

The end result of this activity will allow you to login to sharefile using a native account (think Guest) or an active directory account (think internal user).

What you will need in order to follow this guide:

• An enterprise Sharefile account.
• A local domain.
• An active directory service account. (standard user rights are fine)
• A windows 2012 server to host ADFS (windows 2008 r2 is fine, but you’ll need to install ADFS 2.0 manually).
• This windows server must be accessible via https (443) from the internet. (Netscaler SSL works fine).
• An external trusted certificate for the web server hosting saml (e.g. adfs.yourdomain.com). For this walk through, I’ll assume you have already done this. *
• A copy of the Sharefile User Management Tool.
• About 2-3 hours spare.

* for this, generate a server certificate and import it into the local machines personal certificates.

Steps:

• Installing Active Directory Federated Services.
• Configuring Federated Services.
• Configuring Sharefile for SAML.
• Syncing Active Directory users with Sharefile.
• Testing the saml login.
• COnfiguring the Split Login Page
• Configuring AD FS for google chrome authentication

This blog post is crazy long, 50 pages+. For a pdf, go here.

Credits:

• Juliano from Citrix for this lovely, but a bit lacking blog post: http://blogs.citrix.com/2012/03/19/saml-authentication-with-sharefile-using-ad-fs-2-0/
• The Superstars on the sharefile support team.

Installing Active Directory Federated Services:

Log into your server 2012 machine and fire up server manager. From server manager choose add role:

Choose Role Based or feature based installation, then choose next.

Select the local server and choose next:

Select Active Directory Federation Services:

Accept the Pre Requisits by pressing Add Features, then click next.

On the Features page, don’t select anything additional, just select next:

For ShareFIle, we only require the Federation service, choose this and select next:

On the Web Server role Services page, select the defaults and choose next:

Click to Install:

Assuming all goes to plan, the install should finish correctly. Click Close.

Now head over to the start menu, and open up AD FS Management:

Configuring Active Directory Federated Services:

On the landing page, select the Configuration Wizard:

Choose Create a new Federation Service:

Select New federation server farm:

If you are using a wildcard certificate like me, you’ll need to specify the hostname below. This hostname is the internet facing name of this SAML service:

Specify an active directory service account. A standard user, password never expires is fine. The installer will set the SPN for you.

Review the summary (ya right, click next so fast your mouse flies across the room).

Assuming all goes to plan, your screen should be all green and wonderful, click close.

Creating ShareFile SAML trust:

Now that we’re up and running, under ADFS > Trust Relationships, right click Add Relying Party Trust..

Skip the welcome screen, choose Start

Choose enter data about the relying party manually:

Add a Display name and optionally a description, then choose Next.

Choose an AD FS Profile, then Next.

Ignore the certificate tab, click Next

Click Enable support for the SAML 2.0 WebSSO protocol and enter the url to your sharefile instance with /saml/acs appended to the end. e.g. https://yourcompany.sharefile.com/saml/acs

You can also verify this url by going to your sharefile instance > admin > configure single sign on:

extra points if you spot this idiots spelling mistake below.

Add the trust identifier you wish, I would recommend a the lowercase string the same as the sharefile instance name. Make note of what you enter here for later.

Select Permit all users to access this relying party, then choose Next.

On the Ready to Add Trust page, click Next:

On the Finish page, select Open the Edit Claim Rules dialog for this relying party when the wizard closes, then click finish.

On the Issuance transform Rules, first we need to tell ADFS what kind of incoming credentials to expect. Click Add Rule.

Select Send LDAP Attributes as Claims, select next.

• Add a descriptive name, e.g. AD to SAML Email.
• Select Active Directory as the attribute store.
• Select Email Addresses as the Attribute
• Select E-Mail Address as the outgoing claim type.
• Click finish to add this rule.

Now we add an additional rule to transform the claim. Click Add again.

Select Transform an Incoming Claim as the claim rule template then click next.

• Configure a descriptive name, e.g. Email to NameID.
• Select E-Mail Address as the Incoming claim type.
• Select Name ID as the Outgoing claim type.
• Select Email as the Outgoing name ID format.
• Click Finish.

Click Finish, we’re done with the rules.

Now right click on your newly created relying party and choose properties.

On The advanced tab, configure the secure hash algorithm to SHA-1.

Click Apply and Ok.

In the ADFS MMC, browse to AD FS > Services > Certficates. Right Click the Token Signing Certificate and choose View Certificate.

On the Details Tab of the certificate, click Copy to File

Select Base-64 encoded X.509 (.CER) and click Next.

Browse to save the certificate to the desktop and name it output.cer

Once Saved, make your way back to your desktop, right click the file and choose Open with…

Select Notepad.

select the contents of the file and copy it to the clipboard:

Configuring Sharefile for SAML integration:

Log into your sharefile instance as an administrator. Choose the Admin Tab, then click Configure Single Sign On.

On the single sign on tab, configure your details as below:

• Use the saved string we mentioned earlier for both Sharefile issuer / Entity ID and Your IDP Issuer / Entity ID.
• configure the login url and log off url as the url to your adfs server as follows:
• Login: https://adfs.yourdomain.com/adfs/ls/
• Log off: https://adfs.yourdomain.com/adfs/ls/?wa=wsignout1.0

Paste the certificate details taken from the ADFS server into the following dialog (clearing its contents first) then click save.

Save again and we should be good to go.

Configuring the User Sync Tool to sync your active directory users:

Grab a copy of the Sharefile User Management tool from the citrix website (current version as of this document is 1.5.10.0)

Run the installer and choose the default options:

On the desktop, launch the ShareFile User Management Tool:

On the login page, enter your sharefile login details:

Note, if it bugs out, just close and open it again.

Enter your domain details, then click connect.

Select the users tab, then browse active directory to find a user you wish to test with. Select the OU containing the user, then click Add Rule

Configure any advanced features your may require on the rule, then click close.

Once created, head over to the rules tab. Highlight the rule you have created and press Commit Now.

You should receive notification this has been complete:

Once this is done, your new user should appear in ShareFile:

Testing ShareFile SAML authentication.

In a browser, head to https://yourdomain.sharefile.eu/saml/login

enter your credentials, e.g. domain\username

Assuming all has gone to plan, you should now be welcomed to sharefile:

If you click log off, you should now also be greeted by your saml service:

Enabling the Split Login page:

<borat> Great success </borat> But how do we enable the fancy split login page I showed you earlier?

email support@sharefile.com and they lovely ladies and gentlemen will walk you through the process. at most it’ll take 10 mins between emails.

Optional: Configuring SAML to work with Google chrome:

By default saml won’t work with chrome. Here’s how to fix it.

open the start menu and open IIS (or start > run > inetmgr).

In IIS manager, browse down to the login site (sites > default website > adfs > ls). Double Click Authentication:

Right Click Windows Authentication, Choose Advanced Settings.

Under Extended Protection, Select Off.

FAQ:

1: When logging into SAML and logging off again, if I don’t close the browser, the user can log in again without credentials.

A: Yep! ask support@sharefile.com about that chestnut.

2: when logging into sharefile for tablet / phone, how should I authenticate?

A: user@domainname and password. Other combinations wouldn’t work for me anyway.

3: How can I tell the difference between a normal user, and a SAML user?

A: browse the user as an administrator, if the user is NOT a saml user, you will get an option to email the password:

Consequently, I’ve not tried reseting the password on a saml user. I wouldn’t imagine it will end well, so don’t do it

Thinkiosk Version 4.0 is the culmination of 9 months hard work, rebuilding ThinKiosk in a new development style to include the enterprise features many of you requested, adding a management server, secure key redirection technologies, local group policy control and a number of other features. After weeks of rigorous testing we’re delighted to announce the availability of ThinKiosk version 4… Today!

With the release of Version 4.0 we’re lifting the cloak on the company we’ve setup in order to support and further develop ThinKiosk, ThinScale Technology. We’ve set up ThinScale as a little software company to publish applications to the virtualisation community, tackling the smaller issues and annoyances we face day to day as consultants and administrators. More clever little products are in the pipeline, but for now enough about the company!

ThinKiosk Versions:

The largest change around ThinKiosk 4.0 is the version introduction. ThinKiosk will ship in two editions, Enterprise edition and Community edition. Remko and I took a look at the product back in October last year and identified area’s that the project needed investment in order to reach and fulfill it’s full potential. We also noted that a number of customers really wanted the support and functionality offered by a professional product. After much deliberation we took the decision at that point to invest the time and resources into the product to ensure it fulfils it’s potential, this in turn justified the need for a chargeable Enterprise product.

ThinKiosk Community Edition.

• The community edition is free and will always remain free, we want to make sure the community will always have the benefit of the product.
• The Community edition is still one of the most powerful Windows alternatives on the market, including paid for products.
• The Community edition is an extremely powerful piece of software with one or two limitations in comparison to the Enterprise product.
• The Community edition will receive functionality from the enterprise edition over time.

We’re extremely proud of the community edition and we do recommend it if you do not require the functionality of the Enterprise Version.

Enterprise Edition.

ThinKiosk Enterprise Edition will include all the current functionality you know and use in ThinKiosk, along with loads of additional features and benefits. The enterprise version of ThinKiosk delivers far more value than the competitor products and from a functionality perspective beats them hands down even in its first release.

An exact side by side comparison can be found along with pricing and details on the ThinScale Licensing page.

Some of the New goodies are listed below!

Central Management:

ThinKiosk 4.0 new central management server. With this central management console, you can:

• Manage off domain machines.
• Push updates.
• Perform remote power commands.
• Remote Control end users.
• Report on your current ThinKiosk hardware.
• and much more.

MagicFilter:

Allow me to introduce our new ‘dynamic key pass-through technology’ MagicFilter. Magic filter will now block local Ctrl + Alt + Del and windows + L keystrokes and “magically” send them on to the remote desktop environment as if the user is working locally. This gives the user an immersive, native feeling desktop experience from the ThinKiosk client.

We are extremely proud to say we are the only Windows Thin Client vendor on the market who can do this.

Integrated Browser:

ThinKiosk 4.0 is a fully fledged browser, so you can allow your users access to web resources without compromising on security. You can layer in as many bookmarks as you like to the browser or you can simply allow the users to browse the sites they wish via the address bar.

And so much more!

I covered a lot of the functionality previews back in April in the feature teaser.

Want to learn more?

Remko and I will be doing a webinar with the good folks over in www.xenappblog.com next week, sign up to hear our story and get some insider information on the product road map!

And without further ado:

I’ve taken enough of your time for now, to jump right in click the download button below and we’ll send you everything you need to get started.

Recently I read a post from XD Tipster on how to convert Png files into icons and use them for XenDesktop and Storefront… A very interesting piece, but a bit convoluted and long winded for my liking. I didn’t like the idea of the two website hops to get this information into XenDesktop format… So I decided to write two utilities:

Iconizer:

Iconizer Converts png files (with transparency supported) to an Ico file format , then in turn converts it To a Base 64 String.

You can send the data to the clipboard or import directly into XenDesktop if you have the powershell tools available.

It’s very simple, I wont bore you with the details, just convert and import. then map with powershell:

Reverse Iconizer:

I’m sure you can guess, takes the massive string of information stored in base64 and gives you a visual representation.

An example command line of how to do this is below:

Hey wait?

Why didn’t you integrate both of these?

Well it seems .net and Powershell have a limit on the data (string length) it can pull out of the pipeline. The default Citrix icon is close to 20,000 characters and results in you being unable to pull this data from powershell directly to .net. WIth great help from http://www.jonathanmedd.net/  we found that, yep the console does seem to have a roughly 8k char limit… Sure I could parse it to a file or the clipboard, but that was messy and frankly, I really couldn’t be arsed.

If you are up to the challenge I’ve got the source code for forward and reverse of the icon data below. I’ve also got a half assed attempt at creating a list… So fill your boots and take up the challenge if you wish.

Download:

As with most of my utilities the download links and source code are below, and a few icons to get you going:

Download utilities.

Download Source Code.

I recently delivered a presention to the Dutch Citrix User Group and E2EVC on the new technology release by Citrix called ‘Local App Access’.

In this post you will find the presentation deck and two utilities I have written for this technology to help empower the user to configure settings.

Synopsis:

As I mentioned in my presentation, this technology is really cool, but it needs work. For a 1.0 it’s very promising but we need to use it in anger and log the bugs with Citrix to get them fixed. This technology  alike Citrix remote PC is not a silver bullet, but it is a very useful utility in your toolbox for concentrating on the low hanging fruit during a migration.

Don’t let a single user or application in a department hold up user migrations by using this technology to keep the application local until you have time to come back to it.

Question: “You mentioned there’s a work around for getting ‘local app access’ to work without requiring desktop viewer?”

Yes, I’m a complete eejit, in both sessions I told you I would show you a way to get around this…. Then completely forgot! To get this working without needing desktop viewer, rename the cdviewer.exe executable in the ica client program folder to something else!

Presentation:

Click to Download.

Reverse seamless VDI helper:

with the reverse seamless VDI helper tool, you can present this application to users In their virtual desktop to allow them to manage which applications are presented to their virtual desktop without having to lead the user through the registry.

Click to Download.

Revere Seamless local desktop helper:

with the reverse seamless local desktop helper tool, you can distribute this tool out to your users to control which folders from which shortcuts are brought up to the virtual desktop.

click to Download.

Source code:

Because life is about education, here’s the source code if you want to expand it yourself:

Click to Download

When checking the bandwidth requirement of multimedia sites, checking how much additional bandwidth video conferencing is going to require or even troubleshooting WAN capacity issues, it’s extremely useful to have a visible interpretation of realtime bandwidth consumption from your virtual desktop.

I wrote a tool quite some time ago called watcher2 while troubleshooting a similar issue. I finally took the time to refactor that tool for use with XenApp 6.5 , XenDesktop and VMware View and they are finally available to download! Both watcher utilities also include a latency counter which was a request that came in over and over.

HDX and PCOIP watcher by default dock to the top of the screen and can be moved left or right as below:

They can now also be completely un docked:

How do they work?

The tool finds your username in the performance monitor counters for session bandwidth, once it finds this entry it reads your performance monitor data once every second and reports on it.

In the case of PCOIP watcher, it reads the PCOIP counters from performance monitor.

what do the values mean?

All values are in either Kilobits per second or Megabits per second.

In = Traffic from the client to the virtual, this may spike during large copy / paste jobs,web cams or copying data from a usb key to the session:
Out = Traffic from the virtual desktop to the client, mainly audio or video traffic causes this to spike.
Latency = The delay between your client and the virtual desktop.

Can I Configure it?

Two thresholds are available, a yellow warning and a red warning, currently . These default values can be written to  HKCU\software\sessionmonitor or HKLM\software\sessionmonitor. E.G:

Do they have any dependencies?

.net framework 3.5

if you are running XenApp 6.5 or XenDesktop 5.6, ensure you have the latest hot-fixes installed or the counters may be incorrect.

How do I launch it?

Allow the user to run it manually, or place the executable in their start-up folder or login script.

Where Can I download it?

Here:

What’s coming next:

• Native Microsoft RDP Counters.
• Realtime graphs and recording.
• source code is available on request.

With the release of XenDesktop / XenApp 7.5, Citrix Storefront has brought back a very sought after feature, Single sign on for local credentials to the storefront site!

Citrix Storefront SSO can be the default configuration or a choice can be given to the user if you select more than one authentication type as below:

Desktop appliance site: (Slight deviation, bear with me).

An interesting addition to storefront in 2.5 is a desktop appliance site is installed by default. Richard covers what a desktop appliance site really well in this article for the current release of storefont here. It’s worth noting the desktop appliance site is running the older storefront code base and does not currently support single sign on, strangely.

Back on topic!

Below is a quick guide on how to get it working and any interesting features along the way, I’ve broken this piece down into three parts:

XenDesktop Delivery controller configuration:

on each delivery controller accessible by the storefront site, run the following two commands:

Client Configuration:

(Shawn Bass did alot of the hardwork here for me, so a thank you for that!)

when installing the client, you can enable the single sign on features with the following command line:

CitrixReceiver.exe /includeSSON /ENABLE_SSON=Yes /silent STORE0="Store;https://yourservername.yourdomain.com/Citrix/Store/discovery;on;Store"

Once this is complete, add the storefront url to the trusted sites for the user, then add the following setting to the trusted sites zone:

Once complete, open group policy on the local machine (or active directory group policy) and import the icaclient.adm file, the typical path is below for convenience:

x86:

C:\Program Files\Citrix\ICA Client\Configuration\icaclient.adm

x64:

C:\Program Files (x86)\Citrix\ICA Client\Configuration\icaclient.adm

Once you have imported this adm file, configure the following values in the LOCAL MACHINE configuration*

*the policies dont work in user mode, oddly.

Configure the authentication policy:

Configure the web interface authentication ticket settings also:

Now reboot the machine and log in, ensuring SSONSVR.exe is running in task manager.

Storefront Configuration:

I’m going to go ahead and assume you’ve already installed storefront, so lets start from there.

Make your way down to the ‘Authentication’ tab choose add/remove methods and select domain pass-through as an authentication type:

Note the warning, the receiver for web will also need some configuration, so that’s our next step:

Make your way down to your ‘receiver for web’ tab and select ‘Choose Authentication Methods’:

As you can see above, domain pass-through is now an option, with a nice little warning:

Note: if you don’t want SSO to be optional, don’t publish additional authentication types on this storeweb.

Testing:

The quickest way to test is to go right ahead now and use the storefront in anger, but if you’re the cautious type Storefront 2.5 includes a subdirectory called DomainPassthroughAuth/test.aspx. if you browse to this site from a configured machine, you should see the following screen.

if you are prompted as below, or see any of the following errors, go back a few steps and check what you missed:

and the following error’s mean you’ve gotten the configuration wrong on the client side:

and that’s it, happy sso’ing!

While at Citrix Synergy in Barcelona this week, I attended the Citrix Personal vDisk deep dive session. The session was interesting and informative but there was a mention of the inventory and scanning piece of the personal vDisk suite that really got me asking myself “what if?”.

From my understanding of the presentation, when you add a revision to the golden image, Personal vDisk scan’s both images then compares these items to the personal vDisk in an attempt to figure out which bits belong in the vDisk and which bits belong in the base image.

If you’ve read my previous blog post on golden image management with PVS (questionable assumptions and why I don’t trust people), you know I have a great fear with auditing and control of this image. Without having to read the old article, it basically translated to “Provisioning server is great, but I don’t trust people to audit and document the changes they have made to the golden images”.

While sitting in this session, I had another “lightbulb moment” . If the Personal vDisk has baked in technology that audits the changes to the golden image layer and registry, could it be extracted from personal vDisk? If so, wouldn’t this give you a granular view of changes to the golden image from point to point? I.E. a list of changes between snapshots (MCS) or versions (PVS)?

The more I think of it, the better this idea sounds. Imagine having a catalog of changes, searchable for file or registry key names that would help you track back changes, or even view changes made to the golden image to be reviewed before or after you seal the image? This technology would work well with Citrix Provisioning server, XenClient and Machine Creation Services, delivering a matrix of changes to the golden image.

I can’t see wrapping a gui around this auditing as being a challenge, this is Citrix we’re talking about! and as Citrix has mostly adopted Microsofts vhd file type, it would be a single image type to scan.

For me, this would address my concerns with moving most implementations from automated installs, to snapshot mechanisms while still achieving auditing and a deep view of the changes to the file system.

So Citrix, please consider this approach, it would be an immediate value add and put your image management head and shoulders above your competition.

So what do you the readers think? Would this give you more confidence of changes by others? Do you see this technology and a post change report as an extra safe guard on change management?

Note: This is part three, have a read of part one and two.

Hello there, and thank you for dropping back for part 3…

I suppose I should start with the disappointing news that I have yet to test this option for VDI in a box. And despite Aaron Parker’s suggestions it wasn’t due to lack of inspiration, it was down to lack of time! This series has gathered allot of interest from both community and storage vendors alike, and I feel I should set the record straight before I got any further:

1. This isn’t a production idea, you would be crazy to use this idea in a live environment.
2. Throughout this entire project, we’re focusing on pooled stateless. Stateful desktops would be a separate post entirely.
3. This wasn’t an attack on products in this market space, merely a fresh view on an old problem.
4. If i had the skills or funds necessary to run this project to a production solution, I wouldn’t have posted it. I would already be hard at work creating a reasonably priced product!

Now that my declarations are out of the way, I’d first like to talk about the moral of the story. This isn’t an unfamiliar expression:

IOPS mitigation is not about read IOPS it’s about WRITE IOPS!

VMware, Citrix and Microsoft have similar but very different solutions for read IOPS negotiation. Similar in the sense that they try to negate storage read IOPS. But the key difference with XenServer is the local disk cache via Intellicache has the out of box functionality to cache the majority of read’s to local disk (think SSD*) without the baked in soft limit of 512 MB for Microsoft HyperV and VMware respectively.

Long story short, VMware and Microsoft’s solution is about 512mb of recognizable read IOPS negation un-tuned, but enabled. Of course this value can be tuned upwards, but the low entry point of cache would suggest, at least to me, that tuning up will have an upsetting affect on the host.

This to me is why IntelliCache has the upperhand in the (value add product) VDI space for read IOPS negation and they even throw in the Hypervisor as part of your XenDesktop licensing, so win win, but what about those pesky write IOPS?

Let’s look at Citrix for a moment!

If we review article 1 and article 2 for a moment, the half baked idea formed by Barry Schiffer, Ingmar Verheij, Kees Baggerman, Remko Weijnen and I, when combined with Citrix IntelliCache turned out to be a phenomenal combination to create an IOPS avoidance product. But the key point we had in the back of our heads was, Citrix provisioning server already has this driver!

Citrix provisioning server has a RAM cache driver, with configurable size baked into the Provisioning Services product. This driver works in a very similar way to the EWF driver, just without the API and flexibility of a Microsoft driver.

There are customers of Citrix out there using RAM caching with XenApp who I spoke to, they have assigned a large (8gb+ cache, 3-4gb of cache utilised day to day) to negotiate the chance of a spillover, but it could still happen and this is an acceptable risk to their implementation. I struggled to find a XenDesktop customer using this method who would talk about their implementation.

But with XenDesktop in mind, using an overly large cache size to avoid spillover just doesn’t scale the way a consolidated Server Based Computing model would… Which leaves us back in this same dilemma, I’d like to cache in RAM, but I don’t want to risk users losing data when they perform a task we have not factored for.

So with all this in mind, lets talk about my final hurdle I faced with this project.

RAM cache flood and my half baked solution:

The Microsoft EWF driver suffered the same problem as the Citrix provisioning server when the cache filled, i.e. it froze or outright bug checked if you tried to reference a file put into the cache before you filled it with other information.

Jeff Silverman has done a great article on what happens in this event with the Citrix Provisioning Services RAM cache and I urge you to go read it so I don’t have to repeat it! Go now, I’ll wait right here for you to get back!

Read that article? Ok good, let’s continue.

To combat this scenario with the Windows EWF driver, I created a very rough around the edges windows service using polling (Sorry Remko) to check the size of the cache periodically via the EWFAPI and write the cache out to disk.

The function in the EWFAPI called EwfCommitAndDisableLive allowed me to dump the ram cache to disk and subvert the cache, going straight to disk when this event had occurred. By using this routine the ram cache simply disables itself and allows pass-through to disk at this point.

With this in mind, I tuned my service to spill to disk when the cache was greater than 80% of ram available, not in use by the operating system when the system booted, This worked well to a point, but the key failure to this approach became apparent when you opened a large number of applications and they struggled to occupy the space available around the cache.

My second attempt however, proved much more fruitful where I monitored for free system bytes in memory and if this dropped below a certain threshold the EWF drive began it’s dump routine to disk. Once the dump routine was complete, ram cleared and the writes had been committed to storage where the disk continued to use it for the remainder of the session. Once this spill over had occurred, a notification bubble fired in the user session warning them of the degradation of service and they should log off saving work at the next convenient moment…

Voila! no blue screen, spill to disk and user notification of degradation.

This wasn’t fool proof, it was very rough, it didn’t handle files being written larger than the RAM cache, but In my opinion, it satisfied the biggest fear and business case against using the Citrix Provisioning server ram caching, the ram cache flood scenario. I was happy with the results and it scaled well to the size of my lab, it showed what is possible with a RAM filter driver and it allowed me to prove my point before I started to poke the hornets nest of the storage market. So let’s park the EWF story for now and focus on my favorite subject, Citrix technologies.

Note: I won’t be making this service and driver solution publicly available, it’s not a production ready solution and Microsoft would sue the pants off me. I’m sure you’ll understand why, but if you want more information drop me an email.

The next part’s of this blog are all theoretical, but I know certain people I want to listen, are listening (Hi Kenneth, hope you are well :)).

Negating the negated. Lets talk about that spill over.

But what about that Spill over event? by using a spill over from ram to disk, we create a situation where we could change a steady, slightly reliable IOPS per from each desktop, to a situation where, “couldn’t a collection of spillovers at the same time cause your storage to becoming swamped with IOPS?”

Absolutely, and I’ve been thinking about this quite a bit… But there’s another hidden potential here even in a RAM spill over scenario…

With a little bit of trickery couldn’t we also negate this spillover event with a simple provisioning job?

With a bit of time spent thinking about this scenario, this is what I came up with…

Why doesn’t the controller, for XenApp or XenDesktop (PVS / DDC) copy or (my personal preference) create a local, blank differencing disk when a VM boots?

The hypervisor could be completely agnostic at this point, we could spill over to local disk, keeping write IOPS completely away from the shared storage? even in a spill over event?

This approach (in this half baked enthusiasts view) would negate the negated… But don’t take my word for it, lets go for some theoretical solutions.

Solution A: Implementing a spill over routine with the Citrix provisioning server driver.

So with my success with the Microsoft driver, I got myself thinking how easy would this be to do utilizing the Citrix Provisioning Services driver? Without access to the code, I’m going to be making slightly risky statements here, but I have allot of faith in Citrix that they could make this happen.

From everyone I have spoken to about this idea, they all see the value in the ability to spill out of RAM… So Citrix, please make it so. Below are some idea’s for deployment methods assuming Citrix do march down this route and the pro’s and con’s I see that live with each scenario.

Bear in mind, I’m not a storage guy, a full time developer or a software architect, I’m just an enthusiast that see’s potential, so drop your comments below as to what you think!

Machine Creation Services.

Idea A: Provisioning services improved RAM caching, Machine Creation services and Intellicache on XenServer.

Utilizing XenServer and MCS we could leverage the intellicache to negate the read IOPS as we proved in my own EWF driver testing, but still deliver on that spill over mechanism allowing continuation of service.

Pros:

• Read IOPS negation.
• RAM caching to remove write IOPS (bar a spill over).
• Continuation of service in a cache flood scenario.

Cons:

• Limited to XenServer.
• Over provisioning of RAM necessary per desktop.
• RAM spillover will result in a large amount of IOPS to the shared storage.

Idea B: Provisioning services improved RAM caching, Machine Creation services and Intellicache on XenServer… With local copy!

Same benefits as previous, but now, we have zero reliance on the shared storage when the VM is up (except for ID disk actions).

Pros:

• Read IOPS negation.
• RAM caching to remove write IOPS.
• Uses local resources in a spill over.
• Continuation of service in a cache flood scenario.

Cons:

• Limited to XenServer.
• Over provisioning of RAM necessary per desktop.

So that’s MCS in a nutshell with per VM caching, I think this solution has so much potential I can’t believe it’s not been done, but I digress. So lets park that topic for now and move on to Citrix Provisioning Services.

Citrix Provisioning Services:

So lets look at the “favorite of many” technology.

Idea A: Citrix Provisioning Services and Improved RAM caching driver.

In a pure Provisioning services environment, we would force our read IOPS via the lan, instead of storage protocol but still deliver a spill back to disk to allow continuation of service.

Pros:

• Hypervisor agnostic.
• RAM caching to remove write IOPS (bar a spill over).
• Continuation of service in a cache flood scenario.
• Potentially no shared storage needed, at all, if caching on the PVS server.

Cons:

• Read IOPS aren’t really negated, they’re just forced over another technology.
• Over provisioning of RAM necessary per desktop.
• RAM spillover will result in a large amount of IOPS to the shared storage / pvs server.

Idea B: Citrix Provisioning Services and Improved RAM caching driver.. With local copy!

Taking the above benefits, but with the gain of utilizing local storage in the spillover event.

Pros:

• Hypervisor agnostic.
• RAM caching to remove write IOPS.
• Uses local resources in a spill over.
• Continuation of service in a cache flood scenario.
• Potentially no shared storage needed, at all.

Cons:

• Read IOPS aren’t really negated, they’re just forced over another technology.
• Over provisioning of RAM necessary per desktop.

So lets Review:

And there we have it, 4 solutions to IOPS negation utilizing the Provisioning server RAM caching driver, with a little bit of a modification to deliver a robust solution to RAM caching.

The copy and creation of differencing disks again would deliver additional benefits to leverage the hardware you put into each Hypervisor without the Shared Storage investment.

Win Win, but is it?

There’s an oversight here:

There’s a niggle here that’s been bothering me for some time and you’ll probably note I mentioned it as a CON to most of the solutions above… I’m going to lay it out on the table in complete honesty…

“Isn’t over provisioning RAM on a per desktop basis a waste of resource? Wouldn’t it be better if we could share that resource across all VM’s on a Hypervisor basis?”

You betcha! If we are assigning out (for argument sake) 1gb of ram cache per VM, that RAM is locked into that VM and if another machine spills, the free RAM in other desktops is wasted.

You would be completely insane not to reserve this RAM per VM, if an over-commit for the VM’s is reached this RAM will merely spill out to a page file type arrangement, negating all your benefits.

Ultimately, assigning RAM in this way could be seen as wasteful in the grand scheme of things…

But there are other ways to skin this cat!

So this leads me on to something I was also considering which popped up in a twitter conversation recently. RAM disk’s and Hypervisors.

Current storage vendors will create a storage pool, consisting of ram inside a VM, per hosting hypervisor for local storage to allow good IOPS leveraging that VM. This VM can perform additional RAM optimizations like compression, de-duplication and sharing of similar pages to reduce the count.

This approach is super clever, but, In my humble opinion (please don’t kill me vendors), this approach is wasteful. Running a VM as a storage repository per host has an overhead to run this virtual machine and takes away from the agnostic solution that is possible…

What If the hypervisor provides the RAM disk?

So excluding ESXi for a second, as getting a RAM disk into that platform would require VMware to allow access to the stack. Lets look at Hyper-V (2012) and XenServer for a second…

With Unix and Windows platforms, RAM disks have been available for years. They were once a necessity and a number of vendors still provide them for high performance IO environments.

Lets say (for arguments sake) Citrix and Microsoft decide to provide a snap-in to their hypervisor to allow native RAM disks (or a vendor writes one themselves!) and maybe, they even decide to provide RAM compression, Spill over to local disk, dedupe, and page sharing on this volume from the hypervisor stack…

Wouldn’t this extend provide all the benefits we’ve spoken about, without the need for a VM per host? And using Thin Provisioning allow all desktops to share the large pool of RAMdisk available?

Yes, yes, it would.

Above is an example of how I would see this working.

So random pictures are fine and all, but what about the read IOPS negation technologies? and what about combining these with XenServer or Hyper-V?

XenServer and IntelliCache:

Well there you have it now, all the benefits of a per VM filter, leveraging intellicache for reads and spilling out to local disk… RESULT!

Pro’s:

• Read IOPS negation
• Write IOPS negation
• No Shared Storage required for running VM’s
• Shared pool of RAM for all desktop’s to use.

Con’s:

• A small one, but no migration of VM’s.

Provisioning server?

and again, all the benefits of a per VM filter, reads redirected via lan and spilling out to local disk… RESULT!

Pro’s:

• Write IOPS negation
• No Shared Storage required for running VM’s
• Shared pool of RAM for all desktop’s to use.

Con’s:

• A small one, but no migration of VM’s.
• No real read IOPS negation.

And HyperV + CSV Cache?

Well here’s an accumulation of my thoughts:

So lets just talk for a second about what we are seeing… Utilizing a RAM disk with spill over, and copied vhd’s on boot we are removing the need for shared storage completely and hosting the IOPS from the hypervisor, natively without the need for additional VM’s.

And see that little icon down the bottom? Yep, that’s right, live migration from host to host thanks to Microsofts Shared Nothing Live Migration!

Pro’s:

• Some write IOPS negation
• No Shared Storage required for running VM’s
• Liv migration.
• Shared pool of RAM for all desktop’s to use.
• write back to local disk.

Con’s:

• No full read IOPS negation.

Review.

I’m sure there’s loads of reading in here, and there will be tons of thought and questions after this blog post. This has been in my head for roughly 9 months now and it feels victorious to finally get it all down on paper.

At the end of the day, Ram Caching is the way of the future, you pay no maintenance on it, you keep your IOPS off your storage and with a little magic from Microsoft, or particularly Citrix. You could see these benefits.

If / When these technologies make it out to you, you could quite happily stick with Machine Creation services, leverage intellicache and your shared storage requirement could be a cheap NAS from any vendor. Provisioning services also see’s allot of benefits from this approach, but the real creme de la creme is in that intellicache feature.

The key functionality to these approaches is simple:

• Cache RAM
• Spill to disk
• Keep the differences on local storage if it spills.

One last thing?

In true Mark Templeton fashion, you know I have a one last thing. This actually hit me today while writing this post and to be honest, I’m so amazed by the potential of this idea I’m going to build it in the lab this weekend.

But until then, a teaser…

What if I told you there was already an overlay, freely available baked into a Microsoft technology that would allow you to overlay local storage in Hyper-V with RAM. A powerful API attached to it, and the ability to write specific files to the local disk, subverting the overlay only when needed?

• RAM disk type technology? > yep
• already available? > yep
• Powerful API? > yep
• already can spill to disk? > yep

Yep Yep Yep! check back in a few days.

(If you know what it is already, keep it to yourself until I’ve gotten out of the lab!)

What do you think?

It doesn’t take a half baked enthusiast like me to see this potential and I’d be really eager to hear your comments on these approaches. If you would prefer to keep your comments offline, you can reach me on andrew@andrewmorgan.ie.

Here’s one that used to bother me alot. The problem usually went as follows:

“Your XenApp servers have very high disk queue’s and IO”

“What’s causing it?”

“dunno…”

With Server 2008, the task manager’s resource monitor feature will help you find these items. But in server 2003 this was a perilous task. The specific details for disk io per process are stored in performance monitor under each specific process running. Trying to analyse each process was a massive pain, but powershell can do some very clever work to help alleviate this!

I wrote two quick functions which act similar to “top” in linux for giving an on screen view, updating at interval of what exactly is creating IO activity. These two functions are:

get-IODataBytes:

Get-IODataOperations

The code for these functions are below:

[sourcecode language="powershell"]
function get-iodatabytes{
$result=(get-counter -counter "Process(*)IO Data Bytes/sec" -ea 0).countersamples | ? {$_.cookedvalue -gt 0} | select instancename,@{Name="SessionID";Expression={if ($_.path.contains("#")){($_.path.split("#)"))[1]}else{"0"}}},@{Name="IO Data Bytes/sec";Expression={[math]::Round($_.cookedvalue,0)}},@{Name="IO Data KBytes/sec";Expression={[math]::Round($_.cookedvalue / 1024,0)}} | sort -Descending "IO Data Bytes/sec" | ft
$currentqueue=(((get-counter -counter "PhysicalDisk(0 C:)Current Disk Queue Length" -ea 0).countersamples) | select cookedvalue).cookedvalue
clear
write-warning "Hit [CTRL] + [C] to exit live capture"
write-host "Current Disk queue: $currentqueue"
return $Result
}

FUnction get-IODataOperations {
$result=(get-counter -counter "Process(*)IO Data Operations/sec" -ea 0).countersamples | ? {$_.cookedvalue -gt 0} | select instancename,@{Name="SessionID";Expression={if ($_.path.contains("#")){($_.path.split("#)"))[1]}else{"0"}}},@{Name="IO Data Operations/sec";Expression={[math]::Round($_.cookedvalue,0)}} | sort -Descending "IO Data Operations/sec" | ft
$currentqueue=(((get-counter -counter "PhysicalDisk(0 C:)Current Disk Queue Length" -ea 0).countersamples) | select cookedvalue).cookedvalue
clear
write-warning "Hit [CTRL] + [C] to exit live capture"
write-host "Current Disk queue: $currentqueue"
return $Result
}

[/sourcecode]

if you wish to loop one of these functions, simply use the following code:

[sourcecode language="powershell"]
while ($true){
get-iodataoperations
start-sleep 1
}
[/sourcecode]

Lets get right to it!

Warm up your labs or fire up your golden images ladies and gents, we’re delighted to announce ThinIO’s brief public beta will begin today!

This project has taught us some really interesting things about Windows IO, how Windows behaves and how the hypervisor and storage can behave. This project really felt like a David vs. Goliath task as we (members of our community with a desire to simplify this issue) attempted to tackle one of the largest issues in our industry, storage bottlenecks and Windows desktops.

What’s really unique about our approach is there are no hardware lead times, no architecture changes needed and no external dependencies. ThinIO can be installed in seconds and the benefits are seen immediately.

We’ve spent countless hours testing, tuning, retesting and even more tuning. We’re extremely happy with the results. This public beta will serve as an opportunity for you to really kick the tyres and believe the hype in what we’ve built while we’re putting together the final touches to release the product in the coming weeks.

During this time, we found achieving positive and consistent IO negation boils down to a number of items:

• cutting down on the volume of IOPS sent to the storage.
• Reducing the data transferred (MB/sec) to and from the storage.
• Intelligently cutting down on peak IO, such as boot and user logon.

In the coming days we’re going drill down into these categories in more depth. But as a quick overview, here’s a baseline (top) and ThinIO (bottom) session comparison of a Windows 8.1 desktop login, 1 hour Login VSI medium workload and log off with just 350 mb of cache for ThinIO:

Keep an eye out for the coming blog posts, but in the mean time, the ThinIO beta is available to download here now! Go forth and have fun.

Until next time,

A