Monday, March 3, 2014

System State Preservation (Deep Freeze, Fortres 101 and Clean Slate)

When I talk of system state preservation, I mean you want a system that is treated kind of like a turn-key kiosk. A locked down workstation, if you will.

I worked in a school environment for several years, and if there's one thing students love to do, it's destroy things. I've seen some outrageously entitled behavior before, such as the destruction of phones because certain teens have Mommies and Daddies that will buy them another after carelessly spider-webbing the touch screen. But some still manage to take it a level higher when they are using property that isn't theirs, such as school computers. I mean...yikes.

Sometimes it's not even malicious, it's just stupid. I won't get into the configuration and management issues involved in the politics of how much to allow installed on systems or user privilege levels that enables installation of software. Suffice it to say that when you manage an environment where you have hundreds of users and only a few staff to oversee them, there are compromises made.

That led to my experience with Deep Freeze from Faronics. Kind of neat, really. It seems to work by taking a snapshot of the system state when you "freeze" the workstation, then every disk write is written as a diff from the snapshot. Reboot, and the system rolls back to the snapshot.

Pro? It felt great to delete most of the Windows folder and still have the machine boot. Very cathartic. Oh, and it let students destroy the data on machines without the effect being permanent. We also had a kind of control console that gave a status of workstations in the school network, and we could remotely freeze and thaw systems from that. Or reboot them. Remotely update the Deep Freeze files by pushing updates. That sort of thing. Still...good to destroy Windows on a bad day.

We had a situation once where malware was spreading on several systems. From the console we told all the machines to reboot at the same time...the malware was gone. Changes students made that they didn't save to their network home drives or a USB drive? Well, that went away too. Listen to the messages we send out or you lose stuff. Whoops.

Cons? Anything automated for updates, like Windows update, would get very, very confused. Push out an update, poll the workstation and it's updated, next day...it's not updated anymore. Same goes for antivirus. Oh, sure, there was a function built into Deep Freeze so you can define a "maintenance period", where the machines would reboot and for a period of several hours the machine was "thawed," meaning changes were allowed and permanent until it was in a "frozen" mode again. The problem is that this was not always reliable. Sometimes Windows Update didn't finish in the allotted time. You know what happens if you (automatically) reboot when it's still updating or something was stuck? Yeah, whoops.

Or Deep Freeze would be in a mode where it was thawed until X boot cycles. You know that thing Windows will do where it says, "Oh, stuff is in use. Let's reboot, and at startup we'll update those files!," then it sets a reboot flag and restarts? Only if the bad timing fairy showed up, that reboot means the system is frozen. So it reboots, Deep Freeze freezes, Windows does the update, then reboots automatically to the frozen-in-need-of-updates-at-startup image, and Windows proceeds to update and then reboot again and this continues ad infinitum, requiring some fun hoops to jump through to disable the stuck-cycle because by the way, Deep Freeze was specifically made to prevent kids in schools from screwing with the image.

And of course there were glitches. We occasionally had systems that would report themselves as thawed, but the management console would say it was still frozen. And vice-versa. It would usually take a reboot cycle or two via the console to convince the systems they should revert to the desired state.

Overall Deep Freeze worked well to create a computer that wasn't easily broken by the students. We could create directories and open registry hives to full access for users, so they didn't get errors when trying to use the machines. Keeping systems patched to the latest level wasn't quite as pressing since any malware finding its way to the workstation would disappear at reboot. Relying on this mode of protection, of course, meant that the systems were extremely vulnerable when the systems were thawed for maintenance, since we gave up trying to get antivirus to behave properly using Deep Freeze.

If you have an environment where you are trying to prevent people from actively destroying your system configurations, or accidentally destroying them, Deep Freeze works. It shines in a "lab" environment where you want sets of systems with a heterogeneous configuration that stays intact when you are too grossly understaffed to properly monitor them.

 Fast forward to today. I have a request to configure a small number of systems for use as a sort of communication appliance. We aren't trying to lock out configuration changes for the sake of locking people out, or fear that people will intentionally try to reconfigure the systems. Instead the aim is to lock systems into a simplified configuration, so they can be used with minimal training or worry they'll make permanent configuration changes that will confuse other users or break functionality. Additionally, we want to make it hard for people to accidentally misuse systems in a way that will allow someone access to documents with sensitive data.

Deep Freeze could work, but it would still require a lot of extra configuration to try to lock systems down into a simplified interface. I was looking for something that locked down system configurations but also could manipulate the interface to restrict the number of options people had when using the computers. I needed turnkey "appliances."

After a little digging around I discovered there seems the be relatively few players that managed to get traction in this field. There are several small applications available similar to Deep Freeze, according to the Wikipedia page, but none that seem to be really popular.

So I decided to test a combination of two products called Fortres 101 and Clean Slate, both made by the same company. Presumably this would mean they would work in a complementary fashion (indeed, when I got the install CD's, the CD appears to have several of their products available and differentiated only by the licensing key, reminiscent of the Windows install where you get every version of Windows from Home to Ultimate, but the version you actually activate depends on the activation key entered in setup.)

Fortres 101 is aimed at securing the computer; I can configure it to hide the Start menu, prevent the system tray icons from working, prevent access to particular folders and/or allow particular applications to run. It also allows for a Kiosk mode, where the computer will launch, log in, and run a particular application as the interface. There's a long list of options that can be allowed and disallowed, along with options for security to apply to just users or users and administrators alike (I wonder how hard it is to accidentally lock out all access to the computer?)

Clean Slate is more directly analogous to Deep Freeze; when active, changes to the computer are reverted to a pre-enabled state.

I alluded earlier that the installs for all the Fortresgrand software was included on the application install CD; my impression is that their products are made to be modular enough to be standalone, but intended to be used all together as you purchase licenses. When I activated Fortres and Clean Slate, launching the administrator application...the interface for modifying the configuration to the protection programs...both appear on the same application, just integrated into a tree structure on the left side of the configuration program. By reading through various options, I eventually created a state where the computers seemed to act close to the envisioned appliances in functionality.

But there were problems.

The kiosk mode was supposed to allow the system to boot up, log in as a user, and present a program as the interface. I tried getting it to launch a web page for menus to launch other programs, but that didn't work; I deactivated kiosk mode, but it still logs in as the regular local user at startup. I like that it does this automatically, but why is it doing it, when the kiosk mode tree is not enabled? Is it a feature that, behind the scenes, is treated as another feature in a list of available options, while the UI presents it as a part of a branch in the option tree like it's a "related function" for kiosk configuration?

Or is something broken?

These machines, when worked on for maintenance, be accessed remotely. The configuration process was therefore carefully done through accessing the administrative drive share and RDP. Even though I had it set not to enforce rules for administrators...and I'm a domain administrator on the network, and when I had restrictions in place for users the interface was clearly different between my login and the local user login...I couldn't copy files to the machine's drives when accessing the system as me. From the description, I should have been able to do this; I'm an administrator, after all. And it must see me differently, since user interface restrictions didn't apply when I logged in and I could clearly see they weren't enforced.

So why couldn't I copy files?

Then there were obvious bug-related issues. A big one; the Start menu, which as a user was supposed to give me just options to log off or reboot rather than actually present a menu, started giving me a message that Explorer had crashed. It relaunched Explorer and the menu was now appearing for users. After giving that error message over several restarts, it stopped giving that error, but now the user gets a the Start menu when clicked, and I haven't figured out how to get it back to the previous setting yet.

I don't know what mechanisms are used to manipulate the OS. I infer certain details from observed behavior, but I think the companies keep things quiet to keep their implementations proprietary. For example, I remember using GPO many moons ago to prevent access to certain drive letters. Then I ran a file manager...perhaps it was File Manager from a previous version of Windows, my memory is foggy...and the locked out drives appeared. Apparently GPO was just bit-flipping settings in the registry where Windows Explorer checked what it was supposed to do; GPO wasn't affecting the operating system itself.

Deep Freeze allows changes at the drive level; you could create a "thawspace" to save data across boots, or specify which drives to freeze if you want to leave a drive persistent across boots. Fortres and Clean Slate seem to allow specification of drives you can save to, for selective persistence; if that's the case, it would seem to follow that Deep Freeze is monitoring drives and erasing a diff-like image of changes from your session while Clean Slate is doing something more along the lines of monitoring file and directory access. Someone even made a comment to this effect on a blog but I don't know how they gleaned this information, so I kind of take it with a grain of salt. I wonder if it's inaccurate though, given that the website makes a claim that sounds like Windows Updates work according while F101/CS is active.

From http://www.fortresgrand.com/products/cls/cls.htm
Everything in my interactions with Deep Freeze indications a snapshot-like behavior, while the information on the Fortresgrand site seems consistent with some kind of logging of file changes...journaling...behavior that rolls back a restart or logoff.

So does Fortresgrand work using a combination of registry changes and policy enforcement from its own driver(s)? Or is it entirely self contained? If I had enough time I might be tempted to try digging deeper into the inner workings. Part of me is a little worried that this journaling model isn't as secure as the snapshot-diff method. What dictates that a file is a critical update allowing a system update? Or what if something is incompletely or improperly labeled, allowing a partial alteration, leading to system compromise or corruption?

That leads me to the big head scratcher while trying to use our test prototype appliance systems. I configure the systems then hand them off to a project manager to test out, as he has the ultimate vision for how these are to function. He hands the prototype back with some notes on tweaks, which I try to implement and repeat until he gives the final okay and I move to the next milestone, create the system image to clone out.

The past several weeks, I've been nailed by Windows Updates that wouldn't apply. It seemed quite random, but there were 2 or 3 (the number would change depending on the reboot, but usually that was an MS anti-malware update tacked on to the 2 that kept re-listing and failing. I have no direct proof that F101/CS was the direct cause, but there was something definitely corrupted. 

I disabled F101. I triple checked CS was disabled. I tried installing from safe mode. I even uninstalled F101/CS, after carefully saving the settings to exported files since there was no way I could reproduce my settings (fun fact: a file identification utility thinks F101/CS uses SQLite 3.x in the back-end) and re-ran updates, but still they failed.

There is a "system imaging" mode that was discussed in a blog entry by someone having trouble installing a service pack; apparently the system imaging label is actually a way to tell F101/CS to disable the driver at reboot. It didn't fix his problem, and it didn't fix my problem either, unfortunately.

I eventually got on the right track to a solution thanks to someone who wrote up a similar problem in their blog; corrupt and missing files in the update manifests. I've definitely not run into this problem before, and it's certainly possible that this was coincidental with my other file access oddities while trying to update and configure the system with F101/CS running. But could F101/CS have corrupted something in the update process? It's unfair to blame it without direct evidence, but I think it's fair to suspect it had a hand in it.

I was a little more suspicious of its behavior when I reinstalled F101/CS. They both asked for my license, and I was afraid that reinstallation would eat up a total of 2 of X licenses despite this being a reinstall on the same hardware (I hated the prospect of having to call the company and explain we needed the licenses back...)

But nope. It knew the installs were reinstalls. It informed me I had X-1 licenses left.

More than that, it had my previous settings still set. All my custom settings. The uninstallation didn't fully erase the application, apparently. I'm always a little antsy when dealing with applications that don't fully uninstall when I tell them to.

In Conclusion...

The intention of the security settings and state preservation wasn't to lock people out, it was to create a system that defaulted to behavior that made it easier to use for a certain set of applications and discouraged accidentally leaving sensitive documents on "public" systems. 

There's a lot of settings that allow such customization in F101/CS. I'm a little leery of what appears to be the underlying implementation, and the access quirks I hit when trying to copy files was a little annoying. I am also troubled by the corrupted manifests in Windows Updates.

If I were trying to use system state preservation in a lab environment, I think I'd stick to Deep Freeze. If Windows had better support for configuring a system to be locked down into a pseudo-kiosk mode, I'd probably have used Deep Freeze to round out securing these systems. Instead I'm relying on the hope that F101/CS would step on each others toes less than, say, combining F101 with Deep Freeze. 

If you've run Deep Freeze, Fortres 101 and/or Clean Slate, I'd love to hear your experiences. Were the issues I ran into an anomaly? Am I right to suspect something a little wonky going on with the implementation? Anyone have insights on how these applications work behind the scenes? Please do share in the comments...

Next...I hope I won't run into too many problems trying to image these systems...

No comments:

Post a Comment